-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Wed, Aug 31, 2022 at 10:07:52PM -0500, Eric Jacobs wrote:
I was looking into sel4cp (https://github.com/BreakawayConsulting/sel4cp) for some ideas about designing high-performance, high-security systems with seL4 and I had some questions about how it interfaces with devices.
I saw that there's an ethernet driver (example/tqma8xqp1gb/ethernet/eth.c) which is structured as a PD with two shared memory ring buffers (TX and RX) and associated notifications, without using IPC/ppcall at all in the API.
Insofar as IPC/ppcall is the cornerstone of performance in seL4,
At least on high-performance processors, IPC performance is generally dominated by Spectre mitigations. On x86, those cost thousands of cycles for every context switch. You don’t want to disable them because if you do, you have an insecure system.
I wonder if there could be or should be a role for IPC/ppcall. Does using IPC in place of notifications here violate the "don't use IPC for synchronization" rule? I guess I'm not too clear on what the advantage of notifications is over IPC for things like shared memory buffers.
Notifications allow for untrusted drivers, reduce the rate of context switches (which are very expensive due to Spectre), and can operate between different CPU cores. They also allow batching of operations, which is a huge win from a CPU cache perspective.
I think my ideal goal would be something like a IPC-based API where one can pass in a (limited) number of scatter-gather vectors, plus some metadata (offloading parameters, priority, etc.), and could benefit from the fastpath. This would enable a high-performance stack that could take advantage of things like zero-copy buffering where applicable.
Sadly, it turns out that zero-copy is usually incompatible with security. The usual rule I have found is that one must make a copy whenever data crosses a security boundary and must be processed. Otherwise, TOCTOU vulnerabilities are almost inevitable. One can and should only copy data that will actually be accessed, but sometimes this will be all of it. A network stack might be able to get away with only copying the packet headers, but if it needs to perform encryption or authentication in software, it will need to copy the packet body as well. If applications and the NIC are both untrusted, two copies will be necessary. - -- Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab