On 11/22/20, Millar, Curtis (Data61, Kensington NSW)
If a thread attempts to perform a syscall directly rather than via the interface library, this shouldn't be a security issue although it may be a correctness one (if you intend to allow applications to make syscalls directly). You shouldn't rely on the shared library within the binary to enforce security requirements.
Yes, I'm well aware that it's far from sufficient to enforce security requirements. UX/RT will manage capabilities through its VFS layer, which will be part of the root server rather than a library, and processes won't get any capability unless they are authorized. System call origin limits would just make certain attacks a bit more difficult. The other reason I am wanting them is because I want to eliminate any possibility of badly-written programs making direct syscalls, rather than using the transport layer (the seL4 API isn't stable, and I don't want to expose raw IPC to user programs at all, since the transport layer will already map onto it quite closely). UX/RT is meant for real-world general-purpose installations, so backwards compatibility is very important, and allowing user programs to make raw system calls would make that more difficult.
If you need to be able to emulate the Linux system call ABI for the level of compatibility you are attempting to achieve, this should be feasible with the VCPUs present in seL4. You don't necessarily need to provide an entire linux VM within the VCPU, but a minimal guest supervisor mode would be sufficient to trap all calls made from the linux user-level and redirect them to the host system using the seL4 API.
I would think that would incur significant overhead. I want to implement the syscall handling as a library rather than a server, with the only server-like thing involved with syscalls being the trap handler thread, which as I said should replace traps with calls to a jump table that makes normal function calls (for Linux syscalls that have direct equivalent functions in UX/RT, the equivalent will just be called directly, and those that don't will be implemented in the compatibility library). Am I right that there is no way for VCPUs to make seL4 syscalls directly without going through the associated VMM thread? I don't see any way to do that in the manual. I don't want every single Linux system call to have to trap into the system call handler, only to have the system call handler trap into the kernel again to communicate with the underlying server. Linux programs under UX/RT should be treated as closely to native UX/RT ones as possible and should have identical performance, since UX/RT's API functions will be quite close to Linux even without the compatibility layer. There needs to be some way to patch out the Linux syscall traps so that no traps occur other than those that a native UX/RT program would make.