x86 - does NEG instruction in assembly language sets the Overflow flag

Question

I want to know if the NEG instruction affects the overflow flag too. I know that it negates the value of a variable, but couldn't find out whether it affects the Overflow flag or not.

Answer 1

If you want to know what instructions do, consult the reference manuals.

The essential reference, namely the Intel instruction set manual says this about the NEG instruction:

Flags Affected
The CF flag set to 0 if the source operand is 0; otherwise it is set to 1.
The OF, SF, ZF, AF, and PF flags are set according to the result. 

So it is clear that the NEG instruction sets the O flag; therefore it affects the O flag, which is the OP's original question. And it does so every time it is executed. (People should not confuse "didn't change" from "not set").

That particular reference manual doesn't provide a specific algorithm to indicate when O is set to zero or one. However, Intel CPUs are 2's complement machines. The Subtract instruction has the exact same verbiage. NEG X is equivalent to (0 SUBTRACT X). So NEG should set the O bit according to "overflow" for (0 SUBTRACT X); this will set O when X is 0x8000000.

Inspecting the Intel Basic Archiecture Manual, we find this description of the OF bit:

OF (bit 11) Overflow flag
— Set if the integer result is too large a positive number or too small a
  negative number (excluding the sign-bit) to fit in the destination operand;
  cleared otherwise. This flag indicates an overflow condition for signed-integer
 (two’s complement) arithmetic

confirming our understanding.