By using a manifest, you inform the OS you complete program needs the requested privileges.
What you describe though, is a program with only one code path that actually needs these privileges, and another that does not.
In this case, you would be better off programmatically requesting privileges from your program code, instead of through the manifest (which is handled by the OS before starting your actual application).
The problem is this isn't provided (as far as my quick google shows), so an alternative would be to write a wrapper (or keep the logic inside the same application somehow) that starts without permissions, then tries to run the actual program with elevated permissions, and if that fails, runs the actual program without elevated permissions.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…