deobfuscation - Preventing reverse engineering with binary code and secret key

Question

I am working on a software program that has to be deployed on private cloud server of a client, who has root access. I can communicate with the software through a secure port.

I want to prevent client from reverse engineering my program, or at least make it "hard enough". Below is my approach:

1. Write code in Go and compile the software into binary code (may be with obfuscation)
2. Make sure that program can only be initiated with secret key that can be sent through the secure port. The secret key can be changing depending on time.
3. Every time I need to start/stop the program, I can send commands with the secret keys through the secured port.

I think this approach can prevent a root user from either:

1. Using a debugger to reverse engineer my code

2. Running the program repeatedly to check outputs

My question is: What are the weak spots of this design? How can a root user attack it?

Answer 1

I want to prevent client from reverse engineering my program,

You can't prevent this fully when software runs on hardware you don't own. To run the software, the CPU must see all instructions of the program, and they will be stored in the computer memory.

https://softwareengineering.stackexchange.com/questions/46434/how-can-software-be-protected-from-piracy

Code is data. When the code is runnable, a copy of that data is un-protected code. Unprotected code can be copied.

Peppering the code with anti-piracy checks makes it slightly harder, but hackers will just use a debugger and remove them. Inserting no-ops instead of calls to "check_license" is pretty easy.

(answers in https://softwareengineering.stackexchange.com/questions/46434 may be useful for you)

Hardware owner controls the OS and memory, he can dump everything.

or at least make it "hard enough".

You can only make it a bit harder.

Write code in Go and compile the software into binary code (may be with obfuscation)

IDA will decompile any machine code. Using native machine code is a bit stronger than bytecode (java or .NET or dex)

Make sure that program can only be initiate with secret key that can be sent through the secure port. The secret key can be changing depending on time.

If the copy of the same secret key (keys) is in code or memory of the program, user may dump it and simulate your server. If part of your code, or part of data needed for code to run is stored encrypted, and deciphered with such external key, user may either eavesdrop the key (after it will be decoded from SSL but before it will be used to decrypt secret part of code), or dump decrypted code/data from the memory (It is very easy to see new executable code created in memory even with default preinstalled tools like strace in Linux, just search for all mmaps with PROT_EXEC flags)

Every time I need to start/stop the program, I can send commands with the secret keys through the secured port.

This is just a variation of online license/antipiracy check ("phone home")

I think this approach can prevent a root user to: use a debugger to reverse engineer my code, or

No, he can start debugger at any time; but you can make it a bit harder to use interactive debugger, if the program communicates with your server often (every 5 seconds). But if it communicates so often it is better to move some part of computations to your server; this part will be protected.

And he still can use non-interactive debuggers, tracing tools and memory dumping. Also he can run program in virtual machine, wait until online check is done (using tcpdump and netstat to monitor network traffic), then do live snapshot of the VM (there are several variants to enable "live migration" of VM; only short pause may be recorded by your program if it has external timing), continue to run the first copy online, and take snapshot for offline debugging (with all keys and decrypted code in it).

run the program repeatedly to check outputs

Until he cracks the communications...