[seL4] Re: sel4cp and device driver API's

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Thu, Sep 01, 2022 at 03:32:15AM +0000, Gernot Heiser wrote:

Hi Eric,

...

On 1 Sep 2022, at 13:07, Eric Jacobs wrote:

I was looking into sel4cp (https://github.com/BreakawayConsulting/sel4cp) for some ideas about designing high-performance, high-security systems with seL4 and I had some questions about how it interfaces with devices.

I saw that there's an ethernet driver (example/tqma8xqp1gb/ethernet/eth.c) which is structured as a PD with two shared memory ring buffers (TX and RX) and associated notifications, without using IPC/ppcall at all in the API.

Insofar as IPC/ppcall is the cornerstone of performance in seL4, I wonder if there could be or should be a role for IPC/ppcall. Does using IPC in place of notifications here violate the "don't use IPC for synchronization" rule? I guess I'm not too clear on what the advantage of notifications is over IPC for things like shared memory buffers.

You certainly should not use IPC just for synchronisation, but if you have a two-way control flow then it becomes a question of one IPC vs two Signals.

...

I think my ideal goal would be something like a IPC-based API where one can pass in a (limited) number of scatter-gather vectors, plus some metadata (offloading parameters, priority, etc.), and could benefit from the fastpath. This would enable a high-performance stack that could take advantage of things like zero-copy buffering where applicable.

More generally, I also wonder how IPC fits in with bidirectional devices if we follow the strict priority-based call restrictions (recommended in seL4; required in sel4cp). If the device can both send and receive data, then it seems it has to be both a high-priority PD (to be called) and a low-priority PD (to call in to the stack), assuming that we are avoiding the use of blocking-style turnaround API's (such as read, recv, etc. - I feel those are best left at the application layers.)

You identify some of the relevant issues ;-)

We have been looking at some of this in detail for our high-performance driver framework (see https://trustworthy.systems/projects/TS/drivers/, although that page is presently devoid of technical detail). We are preparing a release which is scheduled to happen at the time of the seL4 Summit in 5 weeks’ time. There will be an extensive report describing design and implementation.

While the basic framework is in place and performs well (it outperforms Linux without even trying too hard…) there are a number of questions that still need further research, and are unlikely to be resolved by the time of the initial release. One of them is whether drivers should be active (synchronising with notifications only) or passive PDs (using PPCs). There are a bunch of tradeoffs to consider, and we need a fair amount of experimental work to settle it. The good news is that the effect the choice has on driver as well as client implementation is minimal (a few lines changed, possibly zero on the client side).

I **strongly** recommend active drivers, for the following reasons: 1. While seL4 can perform context switches with incredible speed, lots of hardware requires very expensive flushing to prevent Spectre attacks. On x86, for instance, one must issue an Indirect Branch Predictor Barrier (IPBP), which costs thousands of cycles. Other processors, such as ARM, also require expensive flushing. In the general case, it is not possible to safely avoid these flushes on current hardware. Therefore, context switches must be kept to a minimum, no matter how good the kernel is at doing them. 2. Passive drivers only make sense if the drivers are trusted. In the case of e.g. a USB or network drivers, this is a bad assumption: both are major attack vectors and drivers ported from e.g. Linux are unlikely to be hardened against malicious devices. 3. A passive driver must be on the same core as its caller, since cross-core PPC is deprecated. An active driver does not have this restriction. Active drivers can therefore run concurrently with the programs they serve, which is the same technique used by Linux’s io_uring with SQPOLL. 4. Linux’s io_uring has the application submit a bunch of requests into a ring buffer and then tell the kernel to process these requests. The kernel processes the requests asynchronously and writes completions into another ring buffer. Userspace can use poll() or epoll() to wait for a completion to arrive. This is a fantastic fit for active drivers: the submission syscall is replaced by a notification, and another notification is used to for wakeup on completion. - -- Sincerely, Demi Marie Obenour (she/her/hers) Invisible Things Lab -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEdodNnxM2uiJZBxxxsoi1X/+cIsEFAmMQSFkACgkQsoi1X/+c IsHwiQ/8DnhbDrKNPSWaUs6f/SHI2li1ufOx1OwQx9fGRi2Lu1zhgs3CJko0z6Tu W6NqfzbqCYeh840a74XyQqjx/m1YzP5L7PQp5l45Aub5EzgrEOSs8x7m6cfUKvju LJ7Y86MBsxqUQVvspAHLC+n5JygmDBridqqWmSBC7kEYGLJ+WfgvJQ8xE+MMyCD7 BoVr/apC0IAydtC3rfu8OVIm4c4vRqq0FcYRDXQQDGLoybUP3Vxn8hQs67upKLRK nJX4v9q0L2gi7Wtq5si6E2FafCT8Jf5jSaFRVmyc4Ql1IZQcrdybmoiU8KmfkUxz x/RBGKnxGFRHpfH7eiVV+L4rKnnhUG3u0wXkCaGz9qfmqJP0Mbyh0HrKVpr5pS9D IBpMCdEwAQ3I+xetCLBd7I0cblnHne/uKhsetjyJgKgV2odZzdkGqaQr6B0lKDCN HIJf8kwsxUucje7ugMttJBiQQeAIP1iWV71l4GYshC22IuXhwEWgS/TkkVpqpeOW EWMYdMHrDw+kSKCgawUnV3HDnhYDW9xWuWfzi+xjEpp0Porx33i9a2SPgZzYmy08 UYvxxfHHwQm2CuRkqMxJoYy/N2OGhxxziUqgMswGf6g0szDNKMnesgrP1e0JkXRX AD5qDV3SAsJApwWsE9jntytl5SNN+NklEMZ3TdL/Db13d3SE38w= =EJdh -----END PGP SIGNATURE-----