Thank you, Chris. Now I understand. I was confused, because in kernel/include/plat/zynqmp/plat/machine.h these interrupts are commented out Leonid -----Original Message----- From: Chris Guikema <Chris.Guikema@dornerworks.com> Sent: Thursday, October 3, 2019 3:44 PM To: Leonid Meyerovich <lmeyerovich@i-a-i.com>; Shen, Yanyan (Data61, Kensington NSW) <Yanyan.Shen@data61.csiro.au>; devel@sel4.systems Subject: RE: [seL4] Zynq UltraScale+ locks up after hours running Hello Leonid, I'm assuming you're pointing to code from src/arch/arm/kernel/boot.c? If that’s the case, then you're right, every interrupt in the system is turned off initially. The kernel then turns on some PPIs for timer (and if in hypervisor mode, vgic maintenance). If you're running an SMP system, the irq_remote_call_ipi and irq_reschedule_ipi (interrupts 0 and 1) are turned back on such that cores can communicate with each other. See the links below. https://github.com/seL4/seL4/blob/8234026c1fb827abee75731b2575ddb741687eff/i... https://github.com/seL4/seL4/blob/7798b4767dcb7bcd3699f191f685435cfa645518/s... Chris -----Original Message----- From: Devel <devel-bounces@sel4.systems> On Behalf Of Leonid Meyerovich Sent: Thursday, October 3, 2019 2:51 PM To: Shen, Yanyan (Data61, Kensington NSW) <Yanyan.Shen@data61.csiro.au>; devel@sel4.systems Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I forgot to mention: my board is zcu102, 4 cores A53 -----Original Message----- From: Devel <devel-bounces@sel4.systems> On Behalf Of Leonid Meyerovich Sent: Thursday, October 3, 2019 2:40 PM To: Shen, Yanyan (Data61, Kensington NSW) <Yanyan.Shen@data61.csiro.au>; devel@sel4.systems Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running Hi Everybody, I am not sure this is related to my previous question/e-mail, but I have collected some more data about an interrupts in my system. I observe a lot of interrupts with ID=0, and later with ID=1 (see below) Both IDs 0 and 1 are initialized as inactive in while booting all cores init_irqs(cap_t root_cnode_cap) { irq_t i; for (i = 0; i <= maxIRQ; i++) { setIRQState(IRQInactive, i); } COULD SOMEBODY EXPLAIN WHAT THESE INTERRUPTS MEANS AND WHY I AM GETTING THEM. Thank you, Leonid I have printed interrupt statistic from inside of handleInterrupt(irq_t irq) function. It has a following structure: void handleInterrupt(irq_t irq) { switch (intStateIRQTable[irq]) { case IRQSignal: { ... } case IRQTimer: timerTick(); resetTimer(); if ((++tickCount % 200) == 0) { printInt(0); printf("- - - \n"); printInt(1); printf("- - - \n"); printInt(2); printf("- - - \n"); printInt(3); printf("- - - - - - - \n"); cleanIntStat(); } break; #ifdef ENABLE_SMP_SUPPORT case IRQIPI: handleIPI(irq, true); break; #endif /* ENABLE_SMP_SUPPORT */ case IRQReserved: #ifdef CONFIG_IRQ_REPORTING //printf("Received reserved IRQ: %d", (int)irq); #endif handleReservedIRQ(irq); break; case IRQInactive: /* * This case shouldn't happen anyway unless the hardware or * platform code is broken. Hopefully masking it again should make * the interrupt go away. */ maskInterrupt(true, irq); #ifdef CONFIG_IRQ_REPORTING printf("Received disabled IRQ: %d\n", (int)irq); #endif break; default: /* No corresponding haskell error */ fail("Invalid IRQ state"); } ackInterrupt(irq); } printInt function prints all collected interrupts for one core in the following format: ---irq stat: cpu=0 (2) 26_50_50 0_258_258 the number inside parentheses is the number of interrupt for particular time interval - 1 sec, ignore it Every interrupt statistic: ID_(enter counter)_(exit_counter) each counter is for 1 sec interval ---irq stat: cpu=0 (2) 26_50_50 0_258_258 - - - ---irq stat: cpu=1 (2) 26_50_50 0_258_258 - - - ---irq stat: cpu=2 (2) 26_50_50 0_258_258 - - - ---irq stat: cpu=3 (3) 26_50_50 65_1_1 59_1_1 - - - - - - - ---irq stat: cpu=0 (2) 26_50_50 0_257_257 - - - ---irq stat: cpu=1 (2) 26_50_50 0_280_280 - - - ---irq stat: cpu=2 (2) 26_50_50 0_280_280 .................................... end of the first excerpt - - - - - - - ---irq stat: cpu=0 (1) 26_50_50 - - - ---irq stat: cpu=1 (1) 26_50_50 - - - ---irq stat: cpu=2 (1) 26_50_50 - - - ---irq stat: cpu=3 (1) 26_50_50 - - - - - - - .................................... end of the another excerpt - - - - - - - ---irq stat: cpu=0 (1) 26_50_51 - - - ---irq stat: cpu=1 (2) 26_50_50 0_4382_4382 - - - ---irq stat: cpu=2 (2) 26_50_49 0_4382_4382 - - - ---irq stat: cpu=3 (2) 26_50_50 0_4382_4382 - - - - - - - .................................... end of the another excerpt - - - - - - - ---irq stat: cpu=0 (2) 26_50_51 1_317_317 - - - ---irq stat: cpu=1 (4) 26_50_50 25_34_34 1_132_132 27_16_16 - - - ---irq stat: cpu=2 (4) 26_50_50 27_17_17 25_32_32 1_130_130 - - - ---irq stat: cpu=3 (2) 26_50_49 1_28_28 - - - - - - - .................................... end of the another excerpt -----Original Message----- From: Shen, Yanyan (Data61, Kensington NSW) <Yanyan.Shen@data61.csiro.au> Sent: Thursday, October 3, 2019 2:44 AM To: Leonid Meyerovich <lmeyerovich@i-a-i.com>; devel@sel4.systems Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running Hi Leonid, Could you provide a bit more about your software configuration? For instance, do you have multiple VMs running on dedicated hardware cores? How are the VM and processes configured? Also, you mean there were no interrupts at all on all the four cores? Regards, Yanyan On Wed, 2019-10-02 at 16:01 +0000, Leonid Meyerovich wrote:
Hello,
We are running seL4 microkernel on 4 cores Zynq UltraScale+ (zcu102 board). The implementation includes multiple processes, hypervisor and virtual machine running on dedicated core. After several hours running (it could be 2 or even 8 hours) the whole microkernel locks up. After some investigation I have found that no interrupts generated anymore - at least there is no interrupts coming to ISR. Inside ISR I have monitored PL2 Physical Timer Control register, which feeds a scheduler and didn't find any problems - it stays enabled and not masked.
I will appreciate any idea/direction for approaching this problem.
Thank you,
Leonid
This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc. _______________________________________________ Devel mailing list Devel@sel4.systems https://sel4.systems/lists/listinfo/devel
________________________________ This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc. _______________________________________________ Devel mailing list Devel@sel4.systems https://sel4.systems/lists/listinfo/devel ________________________________ This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc. _______________________________________________ Devel mailing list Devel@sel4.systems https://sel4.systems/lists/listinfo/devel ________________________________ This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc.