Thank you, Chris.
Now I understand. I was confused, because in kernel/include/plat/zynqmp/plat/machine.h these interrupts are commented out
Leonid
-----Original Message-----
From: Chris Guikema
Sent: Thursday, October 3, 2019 3:44 PM
To: Leonid Meyerovich ; Shen, Yanyan (Data61, Kensington NSW) ; devel@sel4.systems
Subject: RE: [seL4] Zynq UltraScale+ locks up after hours running
Hello Leonid,
I'm assuming you're pointing to code from src/arch/arm/kernel/boot.c? If that’s the case, then you're right, every interrupt in the system is turned off initially. The kernel then turns on some PPIs for timer (and if in hypervisor mode, vgic maintenance). If you're running an SMP system, the irq_remote_call_ipi and irq_reschedule_ipi (interrupts 0 and 1) are turned back on such that cores can communicate with each other. See the links below.
https://github.com/seL4/seL4/blob/8234026c1fb827abee75731b2575ddb741687eff/i...
https://github.com/seL4/seL4/blob/7798b4767dcb7bcd3699f191f685435cfa645518/s...
Chris
-----Original Message-----
From: Devel On Behalf Of Leonid Meyerovich
Sent: Thursday, October 3, 2019 2:51 PM
To: Shen, Yanyan (Data61, Kensington NSW) ; devel@sel4.systems
Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I forgot to mention: my board is zcu102, 4 cores A53
-----Original Message-----
From: Devel On Behalf Of Leonid Meyerovich
Sent: Thursday, October 3, 2019 2:40 PM
To: Shen, Yanyan (Data61, Kensington NSW) ; devel@sel4.systems
Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running
Hi Everybody,
I am not sure this is related to my previous question/e-mail, but I have collected some more data about an interrupts in my system.
I observe a lot of interrupts with ID=0, and later with ID=1 (see below) Both IDs 0 and 1 are initialized as inactive in while booting all cores
init_irqs(cap_t root_cnode_cap)
{
irq_t i;
for (i = 0; i <= maxIRQ; i++) {
setIRQState(IRQInactive, i);
}
COULD SOMEBODY EXPLAIN WHAT THESE INTERRUPTS MEANS AND WHY I AM GETTING THEM.
Thank you,
Leonid
I have printed interrupt statistic from inside of handleInterrupt(irq_t irq) function.
It has a following structure:
void
handleInterrupt(irq_t irq)
{
switch (intStateIRQTable[irq]) {
case IRQSignal: {
...
}
case IRQTimer:
timerTick();
resetTimer();
if ((++tickCount % 200) == 0)
{
printInt(0);
printf("- - - \n");
printInt(1);
printf("- - - \n");
printInt(2);
printf("- - - \n");
printInt(3);
printf("- - - - - - - \n");
cleanIntStat();
}
break;
#ifdef ENABLE_SMP_SUPPORT
case IRQIPI:
handleIPI(irq, true);
break;
#endif /* ENABLE_SMP_SUPPORT */
case IRQReserved:
#ifdef CONFIG_IRQ_REPORTING
//printf("Received reserved IRQ: %d", (int)irq); #endif
handleReservedIRQ(irq);
break;
case IRQInactive:
/*
* This case shouldn't happen anyway unless the hardware or
* platform code is broken. Hopefully masking it again should make
* the interrupt go away.
*/
maskInterrupt(true, irq);
#ifdef CONFIG_IRQ_REPORTING
printf("Received disabled IRQ: %d\n", (int)irq); #endif
break;
default:
/* No corresponding haskell error */
fail("Invalid IRQ state");
}
ackInterrupt(irq);
}
printInt function prints all collected interrupts for one core in the following format:
---irq stat: cpu=0 (2) 26_50_50 0_258_258 the number inside parentheses is the number of interrupt for particular time interval - 1 sec, ignore it
Every interrupt statistic: ID_(enter counter)_(exit_counter) each counter is for 1 sec interval
---irq stat: cpu=0 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=1 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=2 (2) 26_50_50 0_258_258
- - -
---irq stat: cpu=3 (3) 26_50_50 65_1_1 59_1_1
- - - - - - -
---irq stat: cpu=0 (2) 26_50_50 0_257_257
- - -
---irq stat: cpu=1 (2) 26_50_50 0_280_280
- - -
---irq stat: cpu=2 (2) 26_50_50 0_280_280
.................................... end of the first excerpt
- - - - - - -
---irq stat: cpu=0 (1) 26_50_50
- - -
---irq stat: cpu=1 (1) 26_50_50
- - -
---irq stat: cpu=2 (1) 26_50_50
- - -
---irq stat: cpu=3 (1) 26_50_50
- - - - - - -
.................................... end of the another excerpt
- - - - - - -
---irq stat: cpu=0 (1) 26_50_51
- - -
---irq stat: cpu=1 (2) 26_50_50 0_4382_4382
- - -
---irq stat: cpu=2 (2) 26_50_49 0_4382_4382
- - -
---irq stat: cpu=3 (2) 26_50_50 0_4382_4382
- - - - - - -
.................................... end of the another excerpt
- - - - - - -
---irq stat: cpu=0 (2) 26_50_51 1_317_317
- - -
---irq stat: cpu=1 (4) 26_50_50 25_34_34 1_132_132 27_16_16
- - -
---irq stat: cpu=2 (4) 26_50_50 27_17_17 25_32_32 1_130_130
- - -
---irq stat: cpu=3 (2) 26_50_49 1_28_28
- - - - - - -
.................................... end of the another excerpt
-----Original Message-----
From: Shen, Yanyan (Data61, Kensington NSW)
Sent: Thursday, October 3, 2019 2:44 AM
To: Leonid Meyerovich ; devel@sel4.systems
Subject: Re: [seL4] Zynq UltraScale+ locks up after hours running
Hi Leonid,
Could you provide a bit more about your software configuration? For instance, do you have multiple VMs running on dedicated hardware cores?
How are the VM and processes configured?
Also, you mean there were no interrupts at all on all the four cores?
Regards,
Yanyan
On Wed, 2019-10-02 at 16:01 +0000, Leonid Meyerovich wrote:
Hello,
We are running seL4 microkernel on 4 cores Zynq UltraScale+ (zcu102
board). The implementation includes multiple processes, hypervisor and
virtual machine running on dedicated core. After several hours running
(it could be 2 or even 8 hours) the whole microkernel locks up. After
some investigation I have found that no interrupts generated anymore -
at least there is no interrupts coming to ISR.
Inside ISR I have monitored PL2 Physical Timer Control register, which
feeds a scheduler and didn't find any problems - it stays enabled and
not masked.
I will appreciate any idea/direction for approaching this problem.
Thank you,
Leonid
This message and all attachments are PRIVATE, and contain information
that is PROPRIETARY to Intelligent Automation, Inc. You are not
authorized to transmit or otherwise disclose this message or any
attachments to any third party whatsoever without the express written
consent of Intelligent Automation, Inc. If you received this message
in error or you are not willing to view this message or any
attachments on a confidential basis, please immediately delete this
email and any attachments and notify Intelligent Automation, Inc.
_______________________________________________
Devel mailing list
Devel@sel4.systems
https://sel4.systems/lists/listinfo/devel
________________________________
This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc.
_______________________________________________
Devel mailing list
Devel@sel4.systems
https://sel4.systems/lists/listinfo/devel
________________________________
This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc.
_______________________________________________
Devel mailing list
Devel@sel4.systems
https://sel4.systems/lists/listinfo/devel
________________________________
This message and all attachments are PRIVATE, and contain information that is PROPRIETARY to Intelligent Automation, Inc. You are not authorized to transmit or otherwise disclose this message or any attachments to any third party whatsoever without the express written consent of Intelligent Automation, Inc. If you received this message in error or you are not willing to view this message or any attachments on a confidential basis, please immediately delete this email and any attachments and notify Intelligent Automation, Inc.
