RPi4: DTS memory device and vm_minimal
Hi, I have been attempting to get the vm_minimal application running on the Raspberry Pi 4 platform, and I have managed to get to the point where I need to define the memory regions for the kernel and virtual machine. The issue is that I do not know what the correct values (if any) would be needed in the overlay-rpi4.dts that I have created. From what I have found to this point, the Pi platform implements a memory system that means it is not possible to define memory regions in a DTS file. These memory regions are user-defined and are not known until runtime. I think I know what the answer will be to this; is there a way to overcome the oddities of the RPi4 memory setup when attempting to create a seL4 CAmkES VM linux image? Thanks, Ben Turner Senior Engineer Roke Manor Research Ltd Tel: +44 (0)1794 833721 ben.turner@roke.co.ukmailto:ben.turner@roke.co.uk ________________________________________ Roke Manor Research Limited, Romsey, Hampshire, SO51 0ZN, United Kingdom.Part of the Chemring Group. Registered in England & Wales. Registered No: 00267550 http://www.roke.co.uk _______________________________________ The information contained in this e-mail and any attachments is proprietary to Roke Manor Research Limited and must not be passed to any third party without permission. This communication is for information only and shall not create or change any contractual relationship. ________________________________________
Hi Ben, The kernel requires knowledge of the platform layout at configuration/compilation time. My understanding is that on a platform like the RPI the RAM available for the CPU is shared with the GPU and while isn't statically defined, is still user controlled by a boot config file. Changes to the config file that change the amount of ram available to the kernel would require a reconfiguration of the kernel. You can take a look at how this is handled for the qemu-arm-virt platform, where the qemu instance can have an arbitrary amount of memory: https://github.com/seL4/seL4/blob/master/src/plat/qemu-arm-virt/config.cmake.... The build scripts update the dts based on how much memory is available. If you don't want to continuously change the kernel build upon changes to the RPI4's memory layout, you could fix a minimum size of memory that the kernel has as a memory node in the device tree. Any additional address ranges will still get given to userlevel as device untyped memory but will only be able to be used as frames and won't be able to be used for kernel objects that the kernel writes to, such as CNodes or page tables. Userlevel could then dynamically interpret the memory layout based on the supplied device tree and still use the memory that the GPU isn't using. The camkes-arm-vm/camkes also requires knowledge of what RAM will be at configuration/compilation time in order to statically allocate memory resources also.
Hi Kent,
Thanks for the reply. I think your last point is the issue that I was having. I had almost everything working for the RPi4, however when I attempted to build the CAmkES project I could never seem to provide enough memory even though it compiled correctly. This meant that during boot I would get several untyped-retype errors, the number of which would depend on how much memory I declared in the overlay-dts.
My efforts were focused on simply attempting to get the vm_minimal project working, but due to the memory above 1GB being treated as highmem I had no way of addressing the rest of this at compile time, so I was stuck with the initial 1GB which just didn't seem to be enough to run the vm_minimal project. This did surprise me, and I wonder if you have any insight into the requirements of the vm_minimal image and why this might have happened?
Thanks,
Ben
________________________________________
Roke Manor Research Limited, Romsey, Hampshire, SO51 0ZN, United Kingdom.Part of the Chemring Group.
Registered in England & Wales. Registered No: 00267550
http://www.roke.co.uk
_______________________________________
The information contained in this e-mail and any attachments is proprietary to Roke Manor Research Limited and
must not be passed to any third party without permission. This communication is for information only and shall
not create or change any contractual relationship.
________________________________________
From: Mcleod, Kent (Data61, Kensington NSW)
Hi Ben, Getting the memory configurations right can become pretty annoying as there are several different ways the memory can be insufficient and given that some of the "Untyped Retype: Insufficient memory" emitted are spurious it can quickly become pretty impossible to debug it all. (There are some pending kernel patches that should remove the spurious kernel error messages). The memory requirements for a VM component in the vm_minimal app configuration come from a series of different constraints: - The Ram device that gets given to the guest needs to be directly mapped if the guest has access to any DMA capable devices and there isn't an IOMMU available to support a virtual-physical address space. - The VMM needs enough untyped objects to create all of the additional kernel objects required to support creating the guest. These get used for creating threads, paging structures, VCPU objects etc. - In addition, camkes implicitly allocates the untyped objects required for setting up the VMMs address space and backing data for its code and data program segments. - All of these objects are initially allocated by the capdl-loader-app which has to be able to allocate everything from the initial untyped objects that the kernel provides to it at boot time. - The kernel creates the initial untypeds based on the static description of Ram it has, minus any memory that it uses for its own static memory usage. So when you try and run an app you can see an allocation failure it could be due to: - There isn't enough memory available in the system for the ELFLoader to load the kernel and user programs into. The system will crash before the kernel even has a chance to start. - The kernel doesn't have enough memory available to create the initial objects required to finish initializing it and the first user task (the capdl-loader-app). The kernel init would crash at this point. - Then when the capdl-loader-app runs, the untypeds that it has been given may not be enough to fit all of the objects that it needs to create into. The loader would print an error and crash at this point. - Then once the camkes VM component starts it might run out of memory while trying to setup the VM if it hasn't been configured properly. Or it might not have been given the untypeds corresponding to the exact Ram range it requires for creating the guest's Ram device. - Then once the guest is started, it may run out of Ram while trying to bring up Linux. And then Linux will print its own error message (or everything will appear to hang if this occurs before Linux has started writing to a console). Debugging these failures requires knowing which memory setting to change based on what part is failing. - vm0.simple_untyped24_pool = 12; is the camkes configuration used to give 12 * 2^24 Untyped objects to the vm component. If it is running out of memory while trying to setup the VM then increasing this is required. - vm0.untyped_mmios = ["0x40000000:28"]; // RAM is the configuration for giving the component the exact region of memory from [0x40000000, 0x40000000 + 2^28). If this can't be given to the component then the capdl-loader-app would print an error and stop loading the system. If this happens, then that means either the kernel isn't making the right untypeds available at start up or that some other camkes component is trying to use the same memory. If the error is because of the first case, then we usually turn to creating a reserved region description in the device tree file for that platform so that the kernel doesn't use the memory and it is therefore ensured to be in the initial set of untypeds passed to memory. If there is less memory available in the system than what the camkes components are requesting, then the above configuration options need to be reduced if memory can't be unlocked from elsewhere somehow. On a system with at least 1GiB of memory available then it is more common that these options are too low or way too high. I'm not quite sure what you mean about the 1GB highmem issue sorry. Are you able to elaborate more? Kent.
participants (2)
-
Mcleod, Kent (Data61, Kensington NSW)
-
Turner, Ben