c - Linux syscalls and errno

Question

Context: I am trying to write a small C program with inline asm that should run under Linux on an x86_64 system and being compiled with gcc in order to better understand how syscalls work under Linux.

My question is: How are error numbers returned from a syscall (e.g. write) in this environment? I understand that when I use a library such as glibc, it takes care of saving the resulting error code in the global errno variable. But where is the error number stored when I call a syscall directly through inline assembler? Will it be stored inside a separate register, or will it be encoded in %rax?

Let take the write syscall on linux as an example:

When call write then after the syscall returns I find it stores 0xfffffffffffffff2 inside %rax, do I need to somehow extract the error code from that?

If I have the error code number, where should I look to identify the actual error that occured? Lets say I get the number 5 returned, which header file do I need to consult to find the corresponding symbolic error name.

I am calling the write syscall like this:

asm ("mov $1,%%rax;"
     "mov $1,%%rdi;"
     "mov %1,%%rsi;"
     "mov %2,%%rdx;"
     "syscall;"
     "mov %%rax,%0;"
     : "=r" (result)
     : "r" (msg), "r" (len)
     : "%rdx", "%rsi", "%rax", "%rdi" /* EDIT: this is needed or else the registers will be overwritten */
    );

with result, msg and len defined like so:

long result = 0;
char* msg = "Hello World
";
long len = 12;

Answer 1

The linux syscall's convention is that they encode both the possible error code and the return value for successful call in the return value. It's just glibc or other C libraries's wrappers that they will set errno to the error code returned by the underline syscall, and the wrapper will return -1. Take the write as an example, the kernel does the error processing similar to this:

ssize_t write(int fd, ...) {
    if (fd is not valid)
         return -EBADF;
    return do_write(...);
}

So as you can see, the error code is just in the return value, and depending on the semantics, there is always a way to check if the syscall succeeded or not by comparing it to a value not possible for successful operation. For most syscalls, like the write one, that means check if it is negative.