About MSI-X interrupts on x86
Hi all, here Albert I’m working on a SuperMicro E300-9D server and trying to passthrough an Intel X557-AT2 NIC to a Linux 6.1 guest. The .camkes configuration seems to be right, here are the configs for the device (the device is correctly detected on the VM): vm2.vm_ioports = []; vm2.pci_devices = [ { "name":"outer_eth", "bus":0xb5, "dev":0x0, "fun":1, "irq":"outer_eth", "memory":[ {"paddr":0xf9000000, "size":0x01000000, "page_bits":12}, // BAR0 {"paddr":0xfb000000, "size":0x00008000, "page_bits":12}, // BAR3 ] } ]; vm2.vm_irqs = [ {"name":"outer_eth", "ioapic":0, "source":0xb, "level_trig":0, "active_low":1, "dest":12} ]; I’m seeing the following errors during boot and when bringing the interface up: [ 2.040786] i40e 0000:00:01.0: MAC address: 3c:ec:ef:d2:88:85 [ 2.090786] i40e 0000:00:01.0: MSI-X vector reservation failed: -38 [ 2.109960] i40e 0000:00:01.0: MSI-X not available, trying MSI [ 2.115660] i40e 0000:00:01.0: MSI init failed - -38 [ 2.121656] i40e 0000:00:01.0: MSI-X and MSI not available, falling back to Legacy IRQ [ 2.220802] i40e 0000:00:01.0 eth1: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None [ 2.290774] i40e 0000:00:01.0: Features: PF-id[1] VSIs: 66 QP: 1 VxLAN Geneve PTP VEPA ... # ifconfig eth1 up [ 451.042525] i40e 0000:00:01.0 eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5770 ms [ 451.052764] i40e 0000:00:01.0 eth1: tx_timeout: VSI_seid: 391, Q 0, NTC: 0x0, HWB: 0x1, NTU: 0x1, TAIL: 0x1, INT: 0x0 [ 451.065876] i40e 0000:00:01.0 eth1: tx_timeout recovery level 2, txqueue 0 [ 456.082601] i40e 0000:00:01.0 eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 10810 ms My thought is that this NIC does not work reliably with legacy IRQ mode. My question is: Is there any implementation of MSI-X interrupts for x86 in the seL4 VMM, or plans to support it? If such support exists, I would like to contribute, as I plan to share my project once it’s finished. Thanks in advance for any guidance or suggestions. Albert
Hi Albert, We only have kernel support for MSI, but not MSI-X. We are planning to add support for MSI-X after merging a vtd patch and fixing a current kernel issue with handling multiple IOAPICs. They are our priority and should hopefully be resolved soon. Regards, Jade. ________________________________ From: Alberto Lange via Devel <devel@sel4.systems> Sent: Monday, September 1, 2025 8:02 AM To: devel@sel4.systems <devel@sel4.systems> Subject: [seL4] About MSI-X interrupts on x86 Hi all, here Albert I’m working on a SuperMicro E300-9D server and trying to passthrough an Intel X557-AT2 NIC to a Linux 6.1 guest. The .camkes configuration seems to be right, here are the configs for the device (the device is correctly detected on the VM): vm2.vm_ioports = []; vm2.pci_devices = [ { "name":"outer_eth", "bus":0xb5, "dev":0x0, "fun":1, "irq":"outer_eth", "memory":[ {"paddr":0xf9000000, "size":0x01000000, "page_bits":12}, // BAR0 {"paddr":0xfb000000, "size":0x00008000, "page_bits":12}, // BAR3 ] } ]; vm2.vm_irqs = [ {"name":"outer_eth", "ioapic":0, "source":0xb, "level_trig":0, "active_low":1, "dest":12} ]; I’m seeing the following errors during boot and when bringing the interface up: [ 2.040786] i40e 0000:00:01.0: MAC address: 3c:ec:ef:d2:88:85 [ 2.090786] i40e 0000:00:01.0: MSI-X vector reservation failed: -38 [ 2.109960] i40e 0000:00:01.0: MSI-X not available, trying MSI [ 2.115660] i40e 0000:00:01.0: MSI init failed - -38 [ 2.121656] i40e 0000:00:01.0: MSI-X and MSI not available, falling back to Legacy IRQ [ 2.220802] i40e 0000:00:01.0 eth1: NIC Link is Up, 1000 Mbps Full Duplex, Flow Control: None [ 2.290774] i40e 0000:00:01.0: Features: PF-id[1] VSIs: 66 QP: 1 VxLAN Geneve PTP VEPA ... # ifconfig eth1 up [ 451.042525] i40e 0000:00:01.0 eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5770 ms [ 451.052764] i40e 0000:00:01.0 eth1: tx_timeout: VSI_seid: 391, Q 0, NTC: 0x0, HWB: 0x1, NTU: 0x1, TAIL: 0x1, INT: 0x0 [ 451.065876] i40e 0000:00:01.0 eth1: tx_timeout recovery level 2, txqueue 0 [ 456.082601] i40e 0000:00:01.0 eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 10810 ms My thought is that this NIC does not work reliably with legacy IRQ mode. My question is: Is there any implementation of MSI-X interrupts for x86 in the seL4 VMM, or plans to support it? If such support exists, I would like to contribute, as I plan to share my project once it’s finished. Thanks in advance for any guidance or suggestions. Albert _______________________________________________ Devel mailing list -- devel@sel4.systems To unsubscribe send an email to devel-leave@sel4.systems This email and any files transmitted with it may contain confidential information. If you believe you have received this email or any of its contents in error, please notify me immediately by return email and destroy this email. Do not use, disseminate, forward, print or copy any contents of an email received in error.
Hi Jade, Thanks for the quick response. I’ve seen some code related to MSI in the kernel, but I’m not sure if it’s fully implemented in the CAmkES VMM. I verified that with the Ethernet device on Linux baremetal, using legacy IRQs I can still achieve throughput close to 1 Gbit/s, so performance isn’t a problem. I haven’t found any documentation on this, so I’m currently using the legacy IRQ configuration in CAmkES: vm2.vm_irqs = [ {"name":"outer_eth", "ioapic":0, "source":0xb, "level_trig":0, "active_low":1, "dest":12} ]; This seems to be correctly injected into the VM, but I cannot ping, and I get a NETDEV WATCHDOG message. Any hints or guidance you could share would be greatly appreciated. Thanks! Albert
Albert, DornerWorks has gotten MSIs working in the x86 CAmkES VM. Unfortunately, we can't yet release our work, as it needs to be OPSEC'd, and that's a slow process. Configuring the MSIs with the seL4 Kernel is part of the problem, but the other issue is that the PCI Bus is virtualized, and MSIs are initialized by the hardware resources via PCI reads and writes. The VMM intercepts those reads, and since MSI capabilities are masked in the VMM, the guest will never try to initialize them. There is an existing and likely outdated PR in seL4_projects_libs that you can reference that shows how to deal with MSIs in the VMM. https://github.com/seL4/seL4_projects_libs/pull/86/commits/554a3fd22b878e8cb... I didn't write that code, but I am available for help if you need any. Thanks! Chris -----Original Message----- From: Alberto Lange via Devel <devel@sel4.systems> Sent: Tuesday, September 2, 2025 11:04 AM To: devel@sel4.systems Subject: [seL4] Re: About MSI-X interrupts on x86 CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Jade, Thanks for the quick response. I’ve seen some code related to MSI in the kernel, but I’m not sure if it’s fully implemented in the CAmkES VMM. I verified that with the Ethernet device on Linux baremetal, using legacy IRQs I can still achieve throughput close to 1 Gbit/s, so performance isn’t a problem. I haven’t found any documentation on this, so I’m currently using the legacy IRQ configuration in CAmkES: vm2.vm_irqs = [ {"name":"outer_eth", "ioapic":0, "source":0xb, "level_trig":0, "active_low":1, "dest":12} ]; This seems to be correctly injected into the VM, but I cannot ping, and I get a NETDEV WATCHDOG message. Any hints or guidance you could share would be greatly appreciated. Thanks! Albert _______________________________________________ Devel mailing list -- devel@sel4.systems To unsubscribe send an email to devel-leave@sel4.systems
Hi Chris, Thanks for your help, that link was really useful. I didn’t know there was work already done on a SuperMicro machine. Right now, I’m struggling to get things working with legacy IRQs. The only configuration where the NIC showed any signs of life was this (by signs of life I mean a very poor ping with high delay and what looks like polling behavior): vm2.vm_irqs = [ {"name":"outer_eth", "ioapic":0, "source":0xb, "level_trig":1, "active_low":1, "dest":12} ]; This is for the 10Gb NIC on a SuperMicro SYS-E300-9D. What worries me is the ISSUES section in that example you shared: The SuperMicro machine has 5 IOAPICs. The gigabit and 10Gb NICs are on IOAPIC 1; the code for handling anything other than the 0th IOAPIC seems buggy. Right now the VM does not get any interrupts. https://github.com/andybui01/camkes-vm-examples/blob/andyb/supermicro_msi/ap... Are you aware of this kind of IOAPIC problem? Thanks a lot! Albert
participants (3)
-
Alberto Lange -
Chris Guikema -
Jingyao Zhou