IMHO, the biggest problem with the Raspberry Pi is that most of its I/O
(even the NIC) is connected through USB. A serial port is available
(albeit without level shifters). So, running the seL4test should be
possible without a USB stack, but as soon as you want to access a
keyboard, network or filesystem, you'll need one.

You should keep in mind that seL4 is a microkernel. So the USB stack and drivers that depend on that stack should all be implemented as processes/tasks running on top of seL4.

I know it is not cool to mention Minix here, but it and Plan9 suggest ideas as to how the layers above seL4 could be structured.

As far as I can understand, seL4 is not an operating system, it is a microkernel upon which one or more operating systems could be built. The key reason to use seL4 would be due to it providing capabilities based isolation between the layers that run on top of it and thus preventing rogue processes from interfering with other processes.