Hi
The root cause of this problem has been found, when Guest OS do the virtio console driver probe, if virtqueue's desc addr allocated by alloc_pages_exact() in the setup_vq() function is >= 0x100000000 (it is common when the memory of VM allocated exceeds 3G), at first It will be shifted right 12 bits by Guest OS(turn into a value >= 0x100000), and then use iowrite32(iobase + VIRTIO_PCI_QUEUE_PFN) to notify Host, after unsigned int type data be shifted left 12 bits by Host, the higest bits 0x1 of virtqueue's desc addr will be lost.
I have submitted a PR for the amendment, waiting for the review and merge into the main branch. https://github.com/seL4/seL4_projects_libs/pull/43
BR. Allen
发件人: Han JingLong 韩景龙 发送时间: 2022年3月21日 20:27 收件人: 'devel@sel4.systems' devel@sel4.systems 主题: Report "Failed to touch ram region" error using virtio_console when assigned for Arm-VM more than 3G memory
Hi
I had make virtio_console work well on my environment(armv8, RAM is 8G) based on vm_multi with Linux Guest OS(512M) and Android Guest OS(<= 3G) But I meet an error(vm_ram_touch@guest_ram.c:157mailto:vm_ram_touch@guest_ram.c:157 Failed to touch ram region: Not registered RAM region) when assign for a VM(Android Guest OS) more than 3G memory.
1. Some logs are added for debugging this issue： +++ b/seL4_projects_libs/libsel4vmmplatsupport/src/drivers/virtio_emul.c @@ -86,6 +86,7 @@ static int emul_io_out(virtio_emul_t *emul, unsigned int offset, unsigned int si assert(size == 4); int queue = emul->virtq.queue; emul->virtq.queue_pfn[queue] = value; + printf("[hjl] queue_size:%d desc:%p\n", emul->virtq.queue_size[queue], (void *)(uintptr_t)(value << 12)); vring_init(&emul->virtq.vring[queue], emul->virtq.queue_size[queue], (void *)(uintptr_t)(value << 12), VIRTIO_PCI_VRING_ALIGN); break;
+++ b/seL4_projects_libs/libsel4vmmplatsupport/src/drivers/virtio_emul_helpers.c @@ -25,10 +25,43 @@ static int write_guest_mem(vm_t *vm, uintptr_t phys, void *vaddr, size_t size, s
int vm_guest_write_mem(vm_t *vm, void *data, uintptr_t address, size_t size) { + bool is_inram = false; + vm_mem_t *guest_memory = &vm->mem; + for (int i = 0; i < guest_memory->num_ram_regions; i++) { + if (guest_memory->ram_regions[i].start <= address && + guest_memory->ram_regions[i].start + guest_memory->ram_regions[i].size >= address + size) { + /* We are within a ram region*/ + is_inram = true; + } + } + + if (!is_inram) { + printf("[hjl] write mem addr:%p size:0x%lx\n", address, size); + for (int index = 0; index < guest_memory->num_ram_regions; index++) { + printf("[hjl] write mem ram_regions start:%p size:0x%lx\n", guest_memory->ram_regions[index].start, guest_memory->ram_regions[index].size); + } + } + return vm_ram_touch(vm, address, size, write_guest_mem, data); }
int vm_guest_read_mem(vm_t *vm, void *data, uintptr_t address, size_t size) { + bool is_inram = false; + vm_mem_t *guest_memory = &vm->mem; + for (int i = 0; i < guest_memory->num_ram_regions; i++) { + if (guest_memory->ram_regions[i].start <= address && + guest_memory->ram_regions[i].start + guest_memory->ram_regions[i].size >= address + size) { + /* We are within a ram region*/ + is_inram = true; + } + } + + if (!is_inram) { + printf("[hjl] read mem addr:%p size:0x%lx\n", address, size); + for (int index = 0; index < guest_memory->num_ram_regions; index++) { + printf("[hjl] read mem ram_regions start:%p size:0x%lx\n", guest_memory->ram_regions[index].start, guest_memory->ram_regions[index].size); + } + } return vm_ram_touch(vm, address, size, read_guest_mem, data); }
2. The detailed log is as follows: [hjl] queue_size:128 desc:0x8eb6000 [hjl] queue_size:128 desc:0x8eb8000 [hjl] read mem addr:0x8eb8802 size:0x2 [hjl] read mem ram_regions start:0x80000000 size:0x80000 [hjl] read mem ram_regions start:0x80080000 size:0x17cda00 [hjl] read mem ram_regions start:0x8184da00 size:0xa5b2600 [hjl] read mem ram_regions start:0x8be00000 size:0x83ae98 [hjl] read mem ram_regions start:0x8c63ae98 size:0x29c5168 [hjl] read mem ram_regions start:0x8f000000 size:0x26d35 [hjl] read mem ram_regions start:0x8f026d35 size:0xc0fd92cb vm_ram_touch@guest_ram.c:157mailto:vm_ram_touch@guest_ram.c:157 Failed to touch ram region: Not registered RAM region
3. devices.camkes configuration reference is as follows: #define VM1_RAM_BASE 0x80000000 #define VM1_RAM_SIZE 0xD0000000 #define VM1_DTB_ADDR 0x8F000000 //VM1_RAM_BASE + 0xF000000 #define VM1_INITRD_ADDR 0x8BE00000 //VM1_DTB_ADDR - VM1_INITRD_MAX_SIZE
vm1.untyped_mmios = [ "0x80000000:28", // RAM "0x90000000:28", // RAM "0xa0000000:28", // RAM "0xb0000000:28", // RAM
"0xc0000000:28", // RAM "0xd0000000:28", // RAM "0xe0000000:28", // RAM "0xf0000000:28", // RAM
"0x100000000:28", // RAM "0x110000000:28", // RAM "0x120000000:28", // RAM "0x130000000:28", // RAM
"0x140000000:28", // RAM /* "0x150000000:28", // RAM "0x160000000:28", // RAM "0x170000000:28", // RAM
"0x180000000:28", // RAM "0x190000000:28", // RAM "0x1a0000000:28", // RAM "0x1b0000000:28", // RAM */ ];
4. Present situation: At present, I tracked and found that When the Guest OS's virtio_console driver do the probe, it created the vring and iowrite to notify VMM of the virtqueue desc 's physical address in setup_vq()
/* create the vring */ static void *vring_alloc_queue(struct virtio_device *vdev, size_t size, dma_addr_t *dma_handle, gfp_t flag) { ...... void *queue = alloc_pages_exact(PAGE_ALIGN(size), flag); if (queue) { phys_addr_t phys_addr = virt_to_phys(queue); *dma_handle = (dma_addr_t)phys_addr;
/* activate the queue */ iowrite32(virtqueue_get_desc_addr(vq) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
I'm very confused why is the physical memory allocated for virtqueue desc not within the effective memory range of VM when assign the memory to it exceeds 3G?
BR. Allen 邮件免责申明 Email Disclaimer
本邮件仅供本邮件指定收件人使用，其所载内容可能因含有保密信息或其它原因而不得披露。除本公司及本邮件指定收件人外，任何人不得公开、传播、分发、复制、印刷或使用本邮件之任何部分或其所载之任何内容。如您误收到本邮件，请立即通知本公司，并将原始邮件、附件及其所有复本从系统中删除，切勿使用。 This email is for the use of the designated receivers only，and the content is not allowed to be disclosed due to the confidential information or other reasons. Except for the Company and the designated receivers of this email, no one shall disclose, disseminate, distribute, copy, print or use any part of this email or any content contained therein. If you receive this email by mistake, please notify the Company immediately, and delete the original email, attachments and all copies from the system. Do not use it.
网络通信可能含有计算机病毒或其它缺陷，可能无法准确和/或及时送达其它系统，亦可能受阻而不为本公司或本邮件指定收件人所知。本公司对此类错误或遗漏以及任何因使用本邮件而引致之任何损失概不承担责任。 Network communication may contain computer viruses or other defects, which may not be delivered to other systems accurately and / or in time, or may be blocked by the Company or the designated receivers of this email. The Company shall not be liable for such errors or omissions and for any loss arising from this email.
本邮件所载任何内容仅作为业务层面交流与参考，除非明确说明，本公司不对邮件所载内容之准确性、完整性或公平性等承担任何法律责任。 Any contents contained in this email are only for the purpose of business communication and reference only. Unless explicitly stated otherwise, the Company shall not assume any legal responsibility for the accuracy, completeness or fairness of the content contained in the email.
本邮件指定收件人应特别注意：本邮件所载任何内容不构成本公司对本邮件指定收件人和/或其所属商业实体的任何要约、要约邀请或承诺，任何权利义务皆以双方签字盖章的书面文件为准。除经本公司以签字盖章的书面文件确认外，收件人和/或其所属商业实体不得以本邮件所载任何内容作为其向本公司主张任何权利或利益的正式依据。 The designated receivers should pay special attention to the fact that nothing contained in this email shall constitute an offer, invitation or acceptance by the Company to the designated receivers of this email and/or its affiliated business entities, and any rights and obligations are subject to the written documents signed and sealed by both parties. Except from the written document signed ,sealed and confirmed by the Company, the receivers and / or its affiliated business entity shall not rely on anything contained in this email as the formal basis for claiming any rights or interests to the Company.