Hi, my seL4 Linux VM box just crashed when trying to access a memory address. Basically I simply wanted to use "devmem" tool to just print the content of a specific address in memory and looked like seL4 was unable to handle the guest OS mmap() correctly. Below the console log: # ls /proc/283/maps /proc/283/maps # cat /proc/283/maps 00400000-00532000 r-xp 00000000 00:02 6023 /bin/lighttpd 00541000-00543000 r--p 00131000 00:02 6023 /bin/lighttpd 00543000-00546000 rw-p 00133000 00:02 6023 /bin/lighttpd 00546000-0054d000 rw-p 00000000 00:00 0 0bd0b000-0bd2d000 rw-p 00000000 00:00 0 [heap] 0bd2d000-0bd91000 rw-p 00000000 00:00 0 [heap] ffff82084000-ffff82085000 r--p 00000000 00:00 0 [vvar] ffff82085000-ffff82086000 r-xp 00000000 00:00 0 [vdso] ffffe6c58000-ffffe6c79000 rw-p 00000000 00:00 0 [stack] # devmem 0xffffe6c58000 8 _utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffe6c58000 map_page@mapping.c:68 Failed to map page at address 0xffffe6c58000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xe6c58000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace -------- Pagefault from [Linux]: read fault @ PC: 0x414c38 IPA: 0xffffe6c58000, FSR: 0x93020004 Context: x0: 0xffff9cfe6000 x1: 0x0 x2: 0x0 x3: 0x1 x4: 0x3 x5: 0xffffe6c58000 x6: 0x5 x7: 0x20 x8: 0xde x9: 0xffffe539cf54 x10: 0xffff9cfe8050 x11: 0x1 x12: 0x436 pc: 0x414c38 x14: 0xffffffff sp: 0xffffe539bb70 spsr: 0x60000000 x13: 0x1 x15: 0x3 x16: 0xffff9cf486c8 x17: 0x4c3870 x18: 0x0 x19: 0x8 x20: 0x0 x21: 0x4c3000 x22: 0xffffe539bd98 x23: 0xffffe6c58000 x24: 0x0 x25: 0x0 x26: 0x0 x27: 0x0 x28: 0x0 x29: 0xffffe539bc60 x30: 0x414bac m-------- main_continued@main.c:1134 Failed to run VM Halting... My three concerns here are: 1) Why this guest mmap() is crashing the VM? Memory address is valid (just guest user space process stack). 2) Is it possible to have some kind of "dump and reboot VM" when a VM and/or guest OS crashes? 3) Are there any plans to formally verify the seL4's VM camkes code? Thank you!
FYI, this is a sample strace of devmem in a standard Linux: root@dev:/home/hugo/seL4/utilidades_sel4webserver# strace devmem2 7ffc8402e000 execve("/usr/bin/devmem2", ["devmem2", "7ffc8402e000"], 0x7ffc8a099a68 /* 21 vars */) = 0 (...) write(1, "/dev/mem opened.\n", 17/dev/mem opened. ) = 17 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f161b205000 write(1, "Memory mapped at address 0x7f161"..., 41Memory mapped at address 0x7f161b205000. ) = 41 write(1, "Value at address 0x7 (0x7f161b20"..., 48Value at address 0x7 (0x7f161b205007): 0x60E0F0 ) = 48 munmap(0x7f161b205000, 4096) = 0 close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ root@dev:/home/hugo/seL4/utilidades_sel4webserver# El mié, 30 jun 2021 a las 19:33, Hugo V.C. (<skydivebcn@gmail.com>) escribió:
Hi,
my seL4 Linux VM box just crashed when trying to access a memory address. Basically I simply wanted to use "devmem" tool to just print the content of a specific address in memory and looked like seL4 was unable to handle the guest OS mmap() correctly. Below the console log:
# ls /proc/283/maps /proc/283/maps # cat /proc/283/maps 00400000-00532000 r-xp 00000000 00:02 6023 /bin/lighttpd 00541000-00543000 r--p 00131000 00:02 6023 /bin/lighttpd 00543000-00546000 rw-p 00133000 00:02 6023 /bin/lighttpd 00546000-0054d000 rw-p 00000000 00:00 0 0bd0b000-0bd2d000 rw-p 00000000 00:00 0 [heap] 0bd2d000-0bd91000 rw-p 00000000 00:00 0 [heap] ffff82084000-ffff82085000 r--p 00000000 00:00 0 [vvar] ffff82085000-ffff82086000 r-xp 00000000 00:00 0 [vdso] ffffe6c58000-ffffe6c79000 rw-p 00000000 00:00 0 [stack]
# devmem 0xffffe6c58000 8 _utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffe6c58000 map_page@mapping.c:68 Failed to map page at address 0xffffe6c58000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xe6c58000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace -------- Pagefault from [Linux]: read fault @ PC: 0x414c38 IPA: 0xffffe6c58000, FSR: 0x93020004 Context: x0: 0xffff9cfe6000 x1: 0x0 x2: 0x0 x3: 0x1 x4: 0x3 x5: 0xffffe6c58000 x6: 0x5 x7: 0x20 x8: 0xde x9: 0xffffe539cf54 x10: 0xffff9cfe8050 x11: 0x1 x12: 0x436 pc: 0x414c38 x14: 0xffffffff sp: 0xffffe539bb70 spsr: 0x60000000 x13: 0x1 x15: 0x3 x16: 0xffff9cf486c8 x17: 0x4c3870 x18: 0x0 x19: 0x8 x20: 0x0 x21: 0x4c3000 x22: 0xffffe539bd98 x23: 0xffffe6c58000 x24: 0x0 x25: 0x0 x26: 0x0 x27: 0x0 x28: 0x0 x29: 0xffffe539bc60 x30: 0x414bac m-------- main_continued@main.c:1134 Failed to run VM Halting...
My three concerns here are:
1) Why this guest mmap() is crashing the VM? Memory address is valid (just guest user space process stack). 2) Is it possible to have some kind of "dump and reboot VM" when a VM and/or guest OS crashes? 3) Are there any plans to formally verify the seL4's VM camkes code?
Thank you!
Hello Hugo, I'm afraid some misconceptions about the devmem2 tool lead this. On Wed, Jun 30, 2021 at 19:38:44 CEST, Hugo V.C. wrote:
root@dev:/home/hugo/seL4/utilidades_sel4webserver# strace devmem2 7ffc8402e000 execve("/usr/bin/devmem2", ["devmem2", "7ffc8402e000"], 0x7ffc8a099a68 /* 21 vars */) = 0 (...)
This command line instructs devmem2 to read physical (logical) memory address 7 because it misses the 0x prefix. See below for the result.
write(1, "/dev/mem opened.\n", 17/dev/mem opened. ) = 17 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f161b205000 write(1, "Memory mapped at address 0x7f161"..., 41Memory mapped at address 0x7f161b205000. ) = 41 write(1, "Value at address 0x7 (0x7f161b20"..., 48Value at address 0x7 (0x7f161b205007): 0x60E0F0 ) = 48
The tool read address 0x7.
# ls /proc/283/maps [...] ffffe6c58000-ffffe6c79000 rw-p 00000000 00:00 0 [stack]
# devmem 0xffffe6c58000 8
In this case the 0x prefix there but I expect the requested memory address is outside of the available memory (and also does not correspong to any MMIO region). So, make sure to call devmem2 with the "physical address of a page frame" and not with some virtual-memory mapping from proc-maps. The following line should be expected if the guest OS accesses guest-physical addresses without backing memory in the VMM.
_utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffe6c58000 map_page@mapping.c:68 Failed to map page at address 0xffffe6c58000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xe6c58000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace [...] My three concerns here are:
1) Why this guest mmap() is crashing the VM? Memory address is valid (just guest user space process stack).
This is because you requested the guest OS to access invalid physical memory (with root privileges). Regards -- Christian Helmuth Genode Labs https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi Christian, let me clarify that the strace I provided is from my host machine (a common Linux distro) which provides "devmem2" that do not requires "0x" before the address. In fact, if you try with "0x" you get this argument error: root@dev:/home/hugo# devmem2 0x7ffc8402e000 /dev/mem opened. Error at line 75, file devmem2.c (22) [Invalid argument] Instead, if you pass the argument without "0x" tehn looks like it works: root@dev:/home/hugo# devmem2 7ffc8402e000 /dev/mem opened. Memory mapped at address 0x7fcea478e000. Value at address 0x7 (0x7fcea478e007): 0x60E0F0 I really don't know why the devmem2 says "Value at address 0x7" even if later it clarifies that the address is "0x7fcea478e007". In fact, in all the strace you can see the address handled is "0x7fcea478e007" and not "0x7": root@dev:/home/hugo# strace devmem2 7ffc8402e000 execve("/usr/bin/devmem2", ["devmem2", "7ffc8402e000"], 0x7fff973ff2a8 /* 21 vars */) = 0 brk(NULL) = 0x55a891f9e000 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffffb764240) = -1 EINVAL (Argumento inválido) access("/etc/ld.so.preload", R_OK) = 0 openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 close(3) = 0 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=102488, ...}) = 0 mmap(NULL, 102488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbca43a7000 close(3) = 0 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360q\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\t\233\222%\274\260\320\31\331\326\10\204\276X>\263"..., 68, 880) = 68 fstat(3, {st_mode=S_IFREG|0755, st_size=2029224, ...}) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbca43a5000 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\t\233\222%\274\260\320\31\331\326\10\204\276X>\263"..., 68, 880) = 68 mmap(NULL, 2036952, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fbca41b3000 mprotect(0x7fbca41d8000, 1847296, PROT_NONE) = 0 mmap(0x7fbca41d8000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7fbca41d8000 mmap(0x7fbca4350000, 303104, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x7fbca4350000 mmap(0x7fbca439b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7fbca439b000 mmap(0x7fbca43a1000, 13528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fbca43a1000 close(3) = 0 arch_prctl(ARCH_SET_FS, 0x7fbca43a6540) = 0 mprotect(0x7fbca439b000, 12288, PROT_READ) = 0 mprotect(0x55a8900ec000, 4096, PROT_READ) = 0 mprotect(0x7fbca43ee000, 4096, PROT_READ) = 0 munmap(0x7fbca43a7000, 102488) = 0 openat(AT_FDCWD, "/dev/mem", O_RDWR|O_SYNC) = 3 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0 brk(NULL) = 0x55a891f9e000 brk(0x55a891fbf000) = 0x55a891fbf000 write(1, "/dev/mem opened.\n", 17/dev/mem opened. ) = 17 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7fbca43ed000 write(1, "Memory mapped at address 0x7fbca"..., 41Memory mapped at address 0x7fbca43ed000. ) = 41 write(1, "Value at address 0x7 (0x7fbca43e"..., 48Value at address 0x7 (0x7fbca43ed007): 0x60E0F0 ) = 48 munmap(0x7fbca43ed000, 4096) = 0 close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ as you can see mmap() here is taking the address I passed by argument "mmap(NULL, 102488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbca43a7000". An later all the mmap() calls are using the address as passed by argument (0x7fbca43a7000). Finally you can see the write() call taking the right address too: write(1, "Value at address 0x7 (0x7fbca43e"..., 48Value at address 0x7 (0x7fbca43ed007): 0x60E0F0 Instead, in the seL4 Linux VM guest, the devmem tool (notice, "devmem" and not "devmem2") will complain if you don't use the "0x": # devmem ffffd506e000 BusyBox v1.31.0 (2019-10-24 15:10:08 AEDT) multi-call binary. Usage: devmem ADDRESS [WIDTH [VALUE]] Read/write from physical address ADDRESS Address to act upon WIDTH Width (8/16/...) VALUE Data to be written and if you try to use the "0x", then you get the crash: # devmem 0xffffd506e000 _utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffd506e000 map_page@mapping.c:68 Failed to map page at address 0xffffd506e000 with cap 396174, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xd506e000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace -------- Pagefault from [Linux]: read fault @ PC: 0x414c70 IPA: 0xffffd506e000, FSR: 0x93820004 Context: x0: 0xffff7ae65000 x1: 0x0 (...) x29: 0xfffff00c0c30 x30: 0x414bac m-------- main_continued@main.c:1134 Failed to run VM Halting... Now, regarding the address passed to the tool "devmem/devmem2" in both cases was taken from the output of mapped memory of a user space process in both examples. In the case of seL4 Linux guest the mapped memory for the user process "lighttpd" is: # cat /proc/282/maps 00400000-00532000 r-xp 00000000 00:02 6023 /bin/lighttpd 00541000-00543000 r--p 00131000 00:02 6023 /bin/lighttpd 00543000-00546000 rw-p 00133000 00:02 6023 /bin/lighttpd 00546000-0054d000 rw-p 00000000 00:00 0 070b1000-070d3000 rw-p 00000000 00:00 0 [heap] ffff7fda4000-ffff7fda5000 r--p 00000000 00:00 0 [vvar] ffff7fda5000-ffff7fda6000 r-xp 00000000 00:00 0 [vdso] ffffd506e000-ffffd508f000 rw-p 00000000 00:00 0 [stack] So the procedure to provide address to devmem2 (in the host) and to devmem (in the seL4 Linux guest) was the same: the address of the beginning of stack of the (user space) process as showed by /proc/[pid]/maps. So I still don't see it clearly as in my host I can directly use the memory address provided by /proc/[pid]/maps. In example, for a user level process I get this stack addresses: root@dev:/home/hugo# cat /proc/2686/maps 564153d3e000-564153d44000 r--p 00000000 08:05 22023015 /usr/bin/update-notifier (...) 7ffc9bf79000-7ffc9bf9a000 rw-p 00000000 00:00 0 [stack] and I can feed it direcctly to devmem2 tool: root@dev:/home/hugo# devmem2 7ffc9bf79000 /dev/mem opened. Memory mapped at address 0x7fb56541c000. Value at address 0x7 (0x7fb56541c007): 0x60E0F0 but I can't do the same in my seL4 Linux guest with devmem as it crashes. In both cases I'm root, in both cases I took the stack address from /proc/[pid]/maps and in both cases I used a correct syntax (devmem and devmem2 accepts different arguments as showed before). So,...what's going on here? El jue, 1 jul 2021 a las 8:31, Christian Helmuth (< christian.helmuth@genode-labs.com>) escribió:
Hello Hugo,
I'm afraid some misconceptions about the devmem2 tool lead this.
On Wed, Jun 30, 2021 at 19:38:44 CEST, Hugo V.C. wrote:
root@dev:/home/hugo/seL4/utilidades_sel4webserver# strace devmem2 7ffc8402e000 execve("/usr/bin/devmem2", ["devmem2", "7ffc8402e000"], 0x7ffc8a099a68 /* 21 vars */) = 0 (...)
This command line instructs devmem2 to read physical (logical) memory address 7 because it misses the 0x prefix. See below for the result.
write(1, "/dev/mem opened.\n", 17/dev/mem opened. ) = 17 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f161b205000 write(1, "Memory mapped at address 0x7f161"..., 41Memory mapped at address 0x7f161b205000. ) = 41 write(1, "Value at address 0x7 (0x7f161b20"..., 48Value at address 0x7 (0x7f161b205007): 0x60E0F0 ) = 48
The tool read address 0x7.
# ls /proc/283/maps [...] ffffe6c58000-ffffe6c79000 rw-p 00000000 00:00 0 [stack]
# devmem 0xffffe6c58000 8
In this case the 0x prefix there but I expect the requested memory address is outside of the available memory (and also does not correspong to any MMIO region). So, make sure to call devmem2 with the "physical address of a page frame" and not with some virtual-memory mapping from proc-maps.
The following line should be expected if the guest OS accesses guest-physical addresses without backing memory in the VMM.
_utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffe6c58000 map_page@mapping.c:68 Failed to map page at address 0xffffe6c58000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xe6c58000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace [...] My three concerns here are:
1) Why this guest mmap() is crashing the VM? Memory address is valid (just guest user space process stack).
This is because you requested the guest OS to access invalid physical memory (with root privileges).
Regards -- Christian Helmuth Genode Labs
https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth _______________________________________________ Devel mailing list -- devel@sel4.systems To unsubscribe send an email to devel-leave@sel4.systems
Hi Hugo, On Thu, Jul 01, 2021 at 10:28:10 CEST, Hugo V.C. wrote:
let me clarify that the strace I provided is from my host machine (a common Linux distro) which provides "devmem2" that do not requires "0x" before the address.
I'm very aware of the fact that your strace was recorded with an ordinary Linux distro, but still devmem2 needs the 0x prefix to translate your command-line argument (speak string) into an integer value.
In fact, if you try with "0x" you get this argument error:
root@dev:/home/hugo# devmem2 0x7ffc8402e000 /dev/mem opened. Error at line 75, file devmem2.c (22) [Invalid argument]
It seems you're lucky because your Linux kernel does not access arbitrary physical addresses but warns you about 0x7ffc8402e000 beeing an invalid argument for mmap("/dev/mem") [1].
Instead, if you pass the argument without "0x" tehn looks like it works:
root@dev:/home/hugo# devmem2 7ffc8402e000 /dev/mem opened. Memory mapped at address 0x7fcea478e000. Value at address 0x7 (0x7fcea478e007): 0x60E0F0
Sorry no, it does not work as you expect it to work. Please consider devmem2 translates your address argument like follows [2]. #include <stdio.h> #include <stdlib.h> int main() { printf("i0=%lx\n", strtoul("7fff", 0, 0)); } If you try this, you will see that strtoul stops parsing the number on the occurrence of the first f character because this is no decimal digit.
I really don't know why the devmem2 says "Value at address 0x7" even if later it clarifies that the address is "0x7fcea478e007". In fact, in all the strace you can see the address handled is "0x7fcea478e007" and not "0x7":
Please see above because your argument 7ffc8402e000 is parsed to 0x7. And please, consider the difference between virtual-memory addresses and physical-memory addresses.
as you can see mmap() here is taking the address I passed by argument "mmap(NULL, 102488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbca43a7000". An later all the mmap() calls are using the address as passed by argument (0x7fbca43a7000). Finally you can see the write() call taking the right address too: write(1, "Value at address 0x7 (0x7fbca43e"..., 48Value at address 0x7 (0x7fbca43ed007): 0x60E0F0
Would you agree that the following numbers are quite different 7ffc8402e000, 0x7, and 0x7fbca43ed007? Only 0x7 is a physical address in my opinion. For information on how to retrieve page-frame numbers from virtual-memory maps please consider [3] which I discovered on Stack OVerflow [4]. [1] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [2] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [3] https://github.com/dwks/pagemap [4] https://unix.stackexchange.com/q/624746/334627 Greets -- Christian Helmuth Genode Labs https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi Christian, ok, I updated my kernel to the latest available version (just in case it was something already patched at Linux kernel). Then I managed to have a strace in my seL4 Linux guest. Here you have the output: # cat /proc/284/maps 00400000-00532000 r-xp 00000000 00:02 6023 /bin/lighttpd 00541000-00543000 r--p 00131000 00:02 6023 /bin/lighttpd 00543000-00546000 rw-p 00133000 00:02 6023 /bin/lighttpd 00546000-0054d000 rw-p 00000000 00:00 0 3ab85000-3aba7000 rw-p 00000000 00:00 0 [heap] ffff9f205000-ffff9f206000 r--p 00000000 00:00 0 [vvar] ffff9f206000-ffff9f207000 r-xp 00000000 00:00 0 [vdso] ffffc7931000-ffffc7952000 rw-p 00000000 00:00 0 [stack] # ./strace devmem 0xffffc7931000 execve("/sbin/devmem", ["devmem", "0xffffc7931000"], [/* 11 vars */]) = 0 readlinkat(AT_FDCWD, "/proc/self/exe", "/bin/busybox", 4096) = 12 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd5000 openat(AT_FDCWD, "/lib//libc.so.0", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0755, st_size=510632, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd4000 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\300Q\1\0\0\0\0\0"..., 4096) = 4096 mmap(NULL, 679936, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6b1c000 mmap(0xffffa6b1c000, 502012, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xffffa6b1c000 mmap(0xffffa6ba7000, 5244, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x7b000) = 0xffffa6ba7000 mmap(0xffffa6ba9000, 99848, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffffa6ba9000 close(3) = 0 munmap(0xffffa6bd4000, 4096) = 0 newfstatat(AT_FDCWD, "/lib/ld-uClibc.so.0", {st_mode=S_IFREG|0755, st_size=33864, ...}, 0) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd4000 mprotect(0x4c2000, 4096, PROT_READ) = 0 mprotect(0xffffa6ba7000, 4096, PROT_READ) = 0 mprotect(0xffffa6bd8000, 4096, PROT_READ) = 0 set_tid_address(0xffffa6bd40d0) = 331 set_robust_list(0xffffa6bd40e0, 24) = 0 rt_sigaction(SIGRTMIN, {0xffffa6b78ae8, [], SA_SIGINFO}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0xffffa6b78ba8, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 getuid() = 0 openat(AT_FDCWD, "/dev/mem", O_RDONLY|O_SYNC) = 3 mmap(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0xffffc7931000) = 0xffffa6bd3000 _utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffc7931000 map_page@mapping.c:68 Failed to map page at address 0xffffc7931000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xc7931000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace -------- Pagefault from [Linux]: read fault @ PC: 0x414c70 IPA: 0xffffc7931000, FSR: 0x93820004 Context: x0: 0xffffa6bd3000 x1: 0x0 x2: 0x0 x3: 0x1 x4: 0x3 x5: 0xffffc7931000 x6: 0xf x7: 0x20 x8: 0xde x9: 0xffffeb09bf54 x10: 0xffffa6bd5050 x11: 0x1 x12: 0x436 pc: 0x414c70 x14: 0xffffffff sp: 0xffffeb09b820 spsr: 0x60000000 x13: 0x1 x15: 0x3 x16: 0xffffa6b356c8 x17: 0x4c3870 x18: 0x0 x19: 0x20 x20: 0x0 x21: 0x4c3000 x22: 0xffffeb09ba40 x23: 0xffffc7931000 x24: 0x0 x25: 0x0 x26: 0x0 x27: 0x0 x28: 0x0 x29: 0xffffeb09b910 x30: 0x414bac m-------- main_continued@main.c:1134 Failed to run VM Halting... Clearly here I can see devmem is not taking "0x" as argument but the full address passed in the argument. And what I see is the last system call is: mmap(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0xffffc7931000) = 0xffffa6bd3000 before seL4 VM crashes. El jue, 1 jul 2021 a las 14:18, Christian Helmuth (< christian.helmuth@genode-labs.com>) escribió:
Hi Hugo,
On Thu, Jul 01, 2021 at 10:28:10 CEST, Hugo V.C. wrote:
let me clarify that the strace I provided is from my host machine (a common Linux distro) which provides "devmem2" that do not requires "0x" before the address.
I'm very aware of the fact that your strace was recorded with an ordinary Linux distro, but still devmem2 needs the 0x prefix to translate your command-line argument (speak string) into an integer value.
In fact, if you try with "0x" you get this argument error:
root@dev:/home/hugo# devmem2 0x7ffc8402e000 /dev/mem opened. Error at line 75, file devmem2.c (22) [Invalid argument]
It seems you're lucky because your Linux kernel does not access arbitrary physical addresses but warns you about 0x7ffc8402e000 beeing an invalid argument for mmap("/dev/mem") [1].
Instead, if you pass the argument without "0x" tehn looks like it works:
root@dev:/home/hugo# devmem2 7ffc8402e000 /dev/mem opened. Memory mapped at address 0x7fcea478e000. Value at address 0x7 (0x7fcea478e007): 0x60E0F0
Sorry no, it does not work as you expect it to work. Please consider devmem2 translates your address argument like follows [2].
#include <stdio.h> #include <stdlib.h>
int main() { printf("i0=%lx\n", strtoul("7fff", 0, 0)); }
If you try this, you will see that strtoul stops parsing the number on the occurrence of the first f character because this is no decimal digit.
I really don't know why the devmem2 says "Value at address 0x7" even if later it clarifies that the address is "0x7fcea478e007". In fact, in all the strace you can see the address handled is "0x7fcea478e007" and not "0x7":
Please see above because your argument 7ffc8402e000 is parsed to 0x7. And please, consider the difference between virtual-memory addresses and physical-memory addresses.
as you can see mmap() here is taking the address I passed by argument "mmap(NULL, 102488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbca43a7000". An later all the mmap() calls are using the address as passed by argument (0x7fbca43a7000). Finally you can see the write() call taking the right address too: write(1, "Value at address 0x7 (0x7fbca43e"..., 48Value at address 0x7 (0x7fbca43ed007): 0x60E0F0
Would you agree that the following numbers are quite different 7ffc8402e000, 0x7, and 0x7fbca43ed007? Only 0x7 is a physical address in my opinion.
For information on how to retrieve page-frame numbers from virtual-memory maps please consider [3] which I discovered on Stack OVerflow [4].
[1] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [2] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [3] https://github.com/dwks/pagemap [4] https://unix.stackexchange.com/q/624746/334627
Greets -- Christian Helmuth Genode Labs
https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth _______________________________________________ Devel mailing list -- devel@sel4.systems To unsubscribe send an email to devel-leave@sel4.systems
Ok, from "https://busybox.net/downloads/busybox-1.31.0.tar.bz2" which is the version packed in the "demo" seL4 VM Linux guest, looks like devmem is not working like you say (devmem2). Look at this: /* devmem ADDRESS [WIDTH [VALUE]] */ // TODO: options? // -r: read and output only the value in hex, with 0x prefix // -w: write only, no reads before or after, and no output // or make this behavior default? // Let's try this and see how users react. /* ADDRESS */ if (!argv[1]) bb_show_usage(); errno = 0; target = bb_strtoull(argv[1], NULL, 0); /* allows hex, oct etc */ that's the first thing that was confusing me (different behavior from devmem2). On the other side, still need to understand why an mmap() is crashing the guest/VM from user space (root is user space). This is insane to me, so I suspect may I have enabled/disabled something in the kernel config to allow this. All this is independent to the fact that, as you pointed out, devmem reads physical addresses and /proc/pid/maps shows virtual addresses thus I need to translate them in some way. My fault :( Thank you for your help! El jue, 1 jul 2021 a las 16:59, Hugo V.C. (<skydivebcn@gmail.com>) escribió:
Hi Christian,
ok, I updated my kernel to the latest available version (just in case it was something already patched at Linux kernel). Then I managed to have a strace in my seL4 Linux guest. Here you have the output:
# cat /proc/284/maps 00400000-00532000 r-xp 00000000 00:02 6023 /bin/lighttpd 00541000-00543000 r--p 00131000 00:02 6023 /bin/lighttpd 00543000-00546000 rw-p 00133000 00:02 6023 /bin/lighttpd 00546000-0054d000 rw-p 00000000 00:00 0 3ab85000-3aba7000 rw-p 00000000 00:00 0 [heap] ffff9f205000-ffff9f206000 r--p 00000000 00:00 0 [vvar] ffff9f206000-ffff9f207000 r-xp 00000000 00:00 0 [vdso] ffffc7931000-ffffc7952000 rw-p 00000000 00:00 0 [stack] # ./strace devmem 0xffffc7931000 execve("/sbin/devmem", ["devmem", "0xffffc7931000"], [/* 11 vars */]) = 0 readlinkat(AT_FDCWD, "/proc/self/exe", "/bin/busybox", 4096) = 12 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd5000 openat(AT_FDCWD, "/lib//libc.so.0", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0755, st_size=510632, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd4000 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\300Q\1\0\0\0\0\0"..., 4096) = 4096 mmap(NULL, 679936, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6b1c000 mmap(0xffffa6b1c000, 502012, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xffffa6b1c000 mmap(0xffffa6ba7000, 5244, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x7b000) = 0xffffa6ba7000 mmap(0xffffa6ba9000, 99848, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffffa6ba9000 close(3) = 0 munmap(0xffffa6bd4000, 4096) = 0 newfstatat(AT_FDCWD, "/lib/ld-uClibc.so.0", {st_mode=S_IFREG|0755, st_size=33864, ...}, 0) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa6bd4000 mprotect(0x4c2000, 4096, PROT_READ) = 0 mprotect(0xffffa6ba7000, 4096, PROT_READ) = 0 mprotect(0xffffa6bd8000, 4096, PROT_READ) = 0 set_tid_address(0xffffa6bd40d0) = 331 set_robust_list(0xffffa6bd40e0, 24) = 0 rt_sigaction(SIGRTMIN, {0xffffa6b78ae8, [], SA_SIGINFO}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0xffffa6b78ba8, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0 getuid() = 0 openat(AT_FDCWD, "/dev/mem", O_RDONLY|O_SYNC) = 3 mmap(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0xffffc7931000) = 0xffffa6bd3000 _utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0xffffc7931000 map_page@mapping.c:68 Failed to map page at address 0xffffc7931000 with cap 396187, error: 1 sel4utils_map_page_pd@vspace.c:153 Error mapping pages, bailing: 1 map_vm_memory_reservation@guest_memory.c:477 Failed to map address 0xc7931000 into guest vm vspace vm_map_reservation@guest_memory.c:510 Failed to map vm reservation: Error when mapping into VM's vspace -------- Pagefault from [Linux]: read fault @ PC: 0x414c70 IPA: 0xffffc7931000, FSR: 0x93820004 Context: x0: 0xffffa6bd3000 x1: 0x0 x2: 0x0 x3: 0x1 x4: 0x3 x5: 0xffffc7931000 x6: 0xf x7: 0x20 x8: 0xde x9: 0xffffeb09bf54 x10: 0xffffa6bd5050 x11: 0x1 x12: 0x436 pc: 0x414c70 x14: 0xffffffff sp: 0xffffeb09b820 spsr: 0x60000000 x13: 0x1 x15: 0x3 x16: 0xffffa6b356c8 x17: 0x4c3870 x18: 0x0 x19: 0x20 x20: 0x0 x21: 0x4c3000 x22: 0xffffeb09ba40 x23: 0xffffc7931000 x24: 0x0 x25: 0x0 x26: 0x0 x27: 0x0 x28: 0x0 x29: 0xffffeb09b910 x30: 0x414bac m-------- main_continued@main.c:1134 Failed to run VM Halting...
Clearly here I can see devmem is not taking "0x" as argument but the full address passed in the argument. And what I see is the last system call is:
mmap(NULL, 4096, PROT_READ, MAP_SHARED, 3, 0xffffc7931000) = 0xffffa6bd3000
before seL4 VM crashes.
El jue, 1 jul 2021 a las 14:18, Christian Helmuth (< christian.helmuth@genode-labs.com>) escribió:
Hi Hugo,
On Thu, Jul 01, 2021 at 10:28:10 CEST, Hugo V.C. wrote:
let me clarify that the strace I provided is from my host machine (a common Linux distro) which provides "devmem2" that do not requires "0x" before the address.
I'm very aware of the fact that your strace was recorded with an ordinary Linux distro, but still devmem2 needs the 0x prefix to translate your command-line argument (speak string) into an integer value.
In fact, if you try with "0x" you get this argument error:
root@dev:/home/hugo# devmem2 0x7ffc8402e000 /dev/mem opened. Error at line 75, file devmem2.c (22) [Invalid argument]
It seems you're lucky because your Linux kernel does not access arbitrary physical addresses but warns you about 0x7ffc8402e000 beeing an invalid argument for mmap("/dev/mem") [1].
Instead, if you pass the argument without "0x" tehn looks like it works:
root@dev:/home/hugo# devmem2 7ffc8402e000 /dev/mem opened. Memory mapped at address 0x7fcea478e000. Value at address 0x7 (0x7fcea478e007): 0x60E0F0
Sorry no, it does not work as you expect it to work. Please consider devmem2 translates your address argument like follows [2].
#include <stdio.h> #include <stdlib.h>
int main() { printf("i0=%lx\n", strtoul("7fff", 0, 0)); }
If you try this, you will see that strtoul stops parsing the number on the occurrence of the first f character because this is no decimal digit.
I really don't know why the devmem2 says "Value at address 0x7" even if later it clarifies that the address is "0x7fcea478e007". In fact, in all the strace you can see the address handled is "0x7fcea478e007" and not "0x7":
Please see above because your argument 7ffc8402e000 is parsed to 0x7. And please, consider the difference between virtual-memory addresses and physical-memory addresses.
as you can see mmap() here is taking the address I passed by argument "mmap(NULL, 102488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbca43a7000". An later all the mmap() calls are using the address as passed by argument (0x7fbca43a7000). Finally you can see the write() call taking the right address too: write(1, "Value at address 0x7 (0x7fbca43e"..., 48Value at address 0x7 (0x7fbca43ed007): 0x60E0F0
Would you agree that the following numbers are quite different 7ffc8402e000, 0x7, and 0x7fbca43ed007? Only 0x7 is a physical address in my opinion.
For information on how to retrieve page-frame numbers from virtual-memory maps please consider [3] which I discovered on Stack OVerflow [4].
[1] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [2] https://github.com/hackndev/tools/blob/7ed212230f8fbb1da3424a15ee88de3279bf9... [3] https://github.com/dwks/pagemap [4] https://unix.stackexchange.com/q/624746/334627
Greets -- Christian Helmuth Genode Labs
https://www.genode-labs.com/ · https://genode.org/ https://twitter.com/GenodeLabs · /ˈdʒiː.nəʊd/
Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth _______________________________________________ Devel mailing list -- devel@sel4.systems To unsubscribe send an email to devel-leave@sel4.systems
participants (2)
-
Christian Helmuth -
Hugo V.C.