Hi Anna, While analyzing some time related APIs, I've noticed that the x86 rt kernel assumes that the tsc frequency is constant and well represented by the measurement made at init by comparing against the PIT. It also exposes only microseconds and uses the internal calibration to convert that to/from ticks. I'm concerned about this behavior. The tsc can have up to 200ppm error if uncalibrated and not temperature controlled, from what I'm able to determine. I'd rather have all the public APIs take ticks and allow user space to determine how their timing requirements best correspond to ticks. This came up when wanting to measure how much running time a process/thread has used. Ideally, it's kept as a running count of ticks. In the current API, one can only observe microseconds consumed, converted by a potentially lossy ticksToUs (lossy when the rate isn't exactly known or tick->us conversion isn't an exact integer multiplication). It'd be more useful for me to have tick count directly. I'm also somewhat concerned with using the PIT to estimate the TSC rate, as I can't find anything about expected error of the PIT, and any slop there will influence all future kernel behaviors until reboot. One potential alternative is the ACPI PM timer or the HPET. Those run at higher frequencies, which would help reduce the error bounds on the rate estimation. Let me know what you think! If it's useful at all, I've written up some stuff about the work that inspired these thoughts: https://gitlab.com/robigalia/meta/blob/master/blog/_drafts/2016-12-23-what-t... Merry Christmas, -- cmr http://octayn.net/
Hi Corey, I will attempt to provide you with what answers I can, although I'm not the expert in this so Anna may say that all of this is wrong later One important thing to consider for Real Time is not just the accuracy of the timer, but the Worst Case Execution Time of the kernel and any padding that needs to be given to timing estimates. The reason that the TSC deadline timer is used in the RT kernel is because it is both extremely cheap to read and, of all the timers, the cheapest to reprogram. This means that the kernel takes less time to perform timer ticks, and allows for a less pessimistic WCET analysis. Any potential error in timing, provided it is bounded, can also be added ass pessimism to the WCET. I have not done the maths to work out what gives less pessimism, the error of the TSC or the cost of a more accurate timer, but that is the pertinent question. This is why when we read the TSC in the kernel we just do 'rdtsc' and make no effort to serialize, the 'error' in doing so is much less than the cost of serializing and so results in less pessimism. It is for similar reasons that the RT kernel goes away from tick based timing to tickless scheduling, although now precision does come into play. Clearly you now have a trade off between tick rate and deadline precision. Taking 'unnecessary' timer interrupts when operating at higher degrees of precision will again increase the pessimism of any scheduling analysis over what it would be with tickless scheduling. Nothing stops you from setting up a timer and counting ticks at user level though. I mentioned on your github PR but will repeat here for the mailing list. The PIT is only used to measure the frequency of the TSC if we cannot pull the frequency directly out of the platform info MSR. Adrian On Sun 25-Dec-2016 12:21 PM, Corey Richardson wrote: Hi Anna, While analyzing some time related APIs, I've noticed that the x86 rt kernel assumes that the tsc frequency is constant and well represented by the measurement made at init by comparing against the PIT. It also exposes only microseconds and uses the internal calibration to convert that to/from ticks. I'm concerned about this behavior. The tsc can have up to 200ppm error if uncalibrated and not temperature controlled, from what I'm able to determine. I'd rather have all the public APIs take ticks and allow user space to determine how their timing requirements best correspond to ticks. This came up when wanting to measure how much running time a process/thread has used. Ideally, it's kept as a running count of ticks. In the current API, one can only observe microseconds consumed, converted by a potentially lossy ticksToUs (lossy when the rate isn't exactly known or tick->us conversion isn't an exact integer multiplication). It'd be more useful for me to have tick count directly. I'm also somewhat concerned with using the PIT to estimate the TSC rate, as I can't find anything about expected error of the PIT, and any slop there will influence all future kernel behaviors until reboot. One potential alternative is the ACPI PM timer or the HPET. Those run at higher frequencies, which would help reduce the error bounds on the rate estimation. Let me know what you think! If it's useful at all, I've written up some stuff about the work that inspired these thoughts: https://gitlab.com/robigalia/meta/blob/master/blog/_drafts/2016-12-23-what-t... Merry Christmas, -- cmr http://octayn.net/ _______________________________________________ Devel mailing list Devel@sel4.systemsmailto:Devel@sel4.systems https://sel4.systems/lists/listinfo/devel
On 01/02/2017 06:30 PM, Adrian.Danis@data61.csiro.au wrote:
Hi Corey,
I will attempt to provide you with what answers I can, although I'm not the expert in this so Anna may say that all of this is wrong later
One important thing to consider for Real Time is not just the accuracy of the timer, but the Worst Case Execution Time of the kernel and any padding that needs to be given to timing estimates. The reason that the TSC deadline timer is used in the RT kernel is because it is both extremely cheap to read and, of all the timers, the cheapest to reprogram. This means that the kernel takes less time to perform timer ticks, and allows for a less pessimistic WCET analysis. Any potential error in timing, provided it is bounded, can also be added ass pessimism to the WCET. I have not done the maths to work out what gives less pessimism, the error of the TSC or the cost of a more accurate timer, but that is the pertinent question. This is why when we read the TSC in the kernel we just do 'rdtsc' and make no effort to serialize, the 'error' in doing so is much less than the cost of serializing and so results in less pessimism.
It is for similar reasons that the RT kernel goes away from tick based timing to tickless scheduling, although now precision does come into play. Clearly you now have a trade off between tick rate and deadline precision. Taking 'unnecessary' timer interrupts when operating at higher degrees of precision will again increase the pessimism of any scheduling analysis over what it would be with tickless scheduling. Nothing stops you from setting up a timer and counting ticks at user level though.
Sure, tickless is great, +1 to tickless. The biggest thing I'm concerned with is accumulated error in the tsc value. I guess that's not really relevant at the timescales that these measurements are used for.
I mentioned on your github PR but will repeat here for the mailing list. The PIT is only used to measure the frequency of the TSC if we cannot pull the frequency directly out of the platform info MSR.
Adrian
On Sun 25-Dec-2016 12:21 PM, Corey Richardson wrote:
Hi Anna,
While analyzing some time related APIs, I've noticed that the x86 rt kernel assumes that the tsc frequency is constant and well represented by the measurement made at init by comparing against the PIT. It also exposes only microseconds and uses the internal calibration to convert that to/from ticks. I'm concerned about this behavior. The tsc can have up to 200ppm error if uncalibrated and not temperature controlled, from what I'm able to determine. I'd rather have all the public APIs take ticks and allow user space to determine how their timing requirements best correspond to ticks. This came up when wanting to measure how much running time a process/thread has used. Ideally, it's kept as a running count of ticks. In the current API, one can only observe microseconds consumed, converted by a potentially lossy ticksToUs (lossy when the rate isn't exactly known or tick->us conversion isn't an exact integer multiplication). It'd be more useful for me to have tick count directly.
I'm also somewhat concerned with using the PIT to estimate the TSC rate, as I can't find anything about expected error of the PIT, and any slop there will influence all future kernel behaviors until reboot. One potential alternative is the ACPI PM timer or the HPET. Those run at higher frequencies, which would help reduce the error bounds on the rate estimation.
Let me know what you think! If it's useful at all, I've written up some stuff about the work that inspired these thoughts: https://gitlab.com/robigalia/meta/blob/master/blog/_drafts/2016-12-23-what-t...
Merry Christmas, -- cmr http://octayn.net/
_______________________________________________ Devel mailing list Devel@sel4.systems https://sel4.systems/lists/listinfo/devel
-- cmr http://octayn.net/ +16038524272
On Mon, 02 Jan 2017 21:40:35 -0500, Corey Richardson wrote:
On 01/02/2017 06:30 PM, Adrian.Danis@data61.csiro.au wrote:
<snip>
It is for similar reasons that the RT kernel goes away from tick based timing to tickless scheduling, although now precision does come into play. Clearly you now have a trade off between tick rate and deadline precision. Taking 'unnecessary' timer interrupts when operating at higher degrees of precision will again increase the pessimism of any scheduling analysis over what it would be with tickless scheduling. Nothing stops you from setting up a timer and counting ticks at user level though.
Sure, tickless is great, +1 to tickless. The biggest thing I'm concerned with is accumulated error in the tsc value. I guess that's not really relevant at the timescales that these measurements are used for.
One issue is that I think the word "ticks" is being aliased here. Corey's concern was that TSC-increment-units ("ticks") are being converted (lossily) into microseconds before being exposed in the API. Meanwhile, your response is discussing whether rescheduling-interrupt- intervals ("ticks") should be exposed in the API. Userspace on Robigalia will _also_ be using TSC-increment-units as its fundamental concept of time, and will have to find its own conversion factors to wall-clock units (nanoseconds, for the APIs we're considering). As a result, there's a (high) risk of _skew_ in the conversions, making it very difficult to use the RT APIs in a way that allows confidence in the result. Since the kernel and userspace will independently maintain distinct conversion factors from TSC increments to seconds, they cannot reliably communicate when both _measure_ the TSC but _interact_ using seconds. <snip>
Hi Corey,
I'm back from leave, but Adrian mostly answered the question. I will reiterate the most important points:
* Currently the PIT is only used to calibrate the tsc if the processor isn't known to support the platform info MSR to read the frequency from. This code is currently only written to work on our local x86 machines, we have an issue for cleaning this code up.
* The TSC freq we read is passed in bootinfo to the initial task, so user space can use this for its calculations.
* I tried using the HPET, but found on our platforms it took hundreds of cycles to reprogram, whereas using tsc deadline mode is far faster. I have not tried the PM timer and am not aware of the perf cost.
Whether the API should be in ticks or microseconds is an interesting question. I initially wrote the API to use ticks and converted it to microseconds based on feedback. However on some arm platforms it's the initial task that calibrates the clock, not seL4, so it may actually make sense to use ticks. It would also remove some calculations from the kernel and make the verification effort (which is starting this year) a bit easier. I will raise a discussion internally, I can't recall if there were any other strong reasons for avoiding ticks in the interface.
Cheers and feel free to hit me up with more questions! I'm to see the RT kernel used out in the wild.
Anna.
-----Original Message-----
From: Corey Richardson [mailto:corey@octayn.net]
Sent: Tuesday, 3 January 2017 1:41 PM
To: Danis, Adrian (Data61, Kensington NSW)
Hi Corey,
I will attempt to provide you with what answers I can, although I'm not the expert in this so Anna may say that all of this is wrong later
One important thing to consider for Real Time is not just the accuracy of the timer, but the Worst Case Execution Time of the kernel and any padding that needs to be given to timing estimates. The reason that the TSC deadline timer is used in the RT kernel is because it is both extremely cheap to read and, of all the timers, the cheapest to reprogram. This means that the kernel takes less time to perform timer ticks, and allows for a less pessimistic WCET analysis. Any potential error in timing, provided it is bounded, can also be added ass pessimism to the WCET. I have not done the maths to work out what gives less pessimism, the error of the TSC or the cost of a more accurate timer, but that is the pertinent question. This is why when we read the TSC in the kernel we just do 'rdtsc' and make no effort to serialize, the 'error' in doing so is much less than the cost of serializing and so results in less pessimism.
It is for similar reasons that the RT kernel goes away from tick based timing to tickless scheduling, although now precision does come into play. Clearly you now have a trade off between tick rate and deadline precision. Taking 'unnecessary' timer interrupts when operating at higher degrees of precision will again increase the pessimism of any scheduling analysis over what it would be with tickless scheduling. Nothing stops you from setting up a timer and counting ticks at user level though.
Sure, tickless is great, +1 to tickless. The biggest thing I'm concerned with is accumulated error in the tsc value. I guess that's not really relevant at the timescales that these measurements are used for.
I mentioned on your github PR but will repeat here for the mailing list. The PIT is only used to measure the frequency of the TSC if we cannot pull the frequency directly out of the platform info MSR.
Adrian
On Sun 25-Dec-2016 12:21 PM, Corey Richardson wrote:
Hi Anna,
While analyzing some time related APIs, I've noticed that the x86 rt kernel assumes that the tsc frequency is constant and well represented by the measurement made at init by comparing against the PIT. It also exposes only microseconds and uses the internal calibration to convert that to/from ticks. I'm concerned about this behavior. The tsc can have up to 200ppm error if uncalibrated and not temperature controlled, from what I'm able to determine. I'd rather have all the public APIs take ticks and allow user space to determine how their timing requirements best correspond to ticks. This came up when wanting to measure how much running time a process/thread has used. Ideally, it's kept as a running count of ticks. In the current API, one can only observe microseconds consumed, converted by a potentially lossy ticksToUs (lossy when the rate isn't exactly known or tick->us conversion isn't an exact integer multiplication). It'd be more useful for me to have tick count directly.
I'm also somewhat concerned with using the PIT to estimate the TSC rate, as I can't find anything about expected error of the PIT, and any slop there will influence all future kernel behaviors until reboot. One potential alternative is the ACPI PM timer or the HPET. Those run at higher frequencies, which would help reduce the error bounds on the rate estimation.
Let me know what you think! If it's useful at all, I've written up some stuff about the work that inspired these thoughts: https://gitlab.com/robigalia/meta/blob/master/blog/_drafts/2016-12-23 -what-time-is-it.adoc
Merry Christmas, -- cmr http://octayn.net/
_______________________________________________ Devel mailing list Devel@sel4.systems https://sel4.systems/lists/listinfo/devel
-- cmr http://octayn.net/ +16038524272
participants (4)
-
Adrian.Danis＠data61.csiro.au -
Alex Elsayed -
Anna.Lyons＠data61.csiro.au -
Corey Richardson