ichthyo/wegen
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

++ FrOSCon25 ++ Vortrag erfolgreich gehalten ++

Browse source 
- Notizen zum Publikum
- Abschließende Dokumentation
- Resourcen in die Lumiera-Website ingegriert
This commit is contained in:
Fischlurch 2026-08-25 05:13:00 +02:00
parent 4a10e7bfbe
commit fe195d101b
4 changed files with 816 additions and 377 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

662
FrOSCon26/FrOSCon.26.mm
View file

File diff suppressed because it is too large Load diff

109
FrOSCon26/research/DemoSpec.txt
View file

@ -6,21 +6,33 @@ Specification for the Demo
:toclevels: 3
This specification is the result of our research and provides a short summary of the relevant
aspects of io_uring. Furthermore, the specific design of the demo code is explained and motivated.
aspects of io_uring. A simple video player was developed as demo code and explained in a
https://programm.froscon.org/froscon2026/talk/d1635d41-1fac-4a43-909b-b8f07672c541/[Talk at FrOSCon 2026].
The following text gives a summary and motivates the specific design of the demo code.
IO_URING kernel interface
-------------------------
`io_uring` is a Linux kernel interface for asynchronous I/O (available since kernel 5.1). It is
built around two ring buffers, shared and memory-mapped between user space and kernel: a
_Submission Queue_ (SQ) and a _Completion Queue_ (CQ). The application fills a _Submission Queue
Entry_ (SQE) describing an operation — read, write, timeout, accept, … — and notifies the kernel;
once the operation has run to completion, the kernel deposits a _Completion Queue Entry_ (CQE)
`io_uring` is a Linux kernel interface for asynchronous I/O (available since kernel 5.1).
It is built around two ring buffers, shared and memory-mapped between user space and kernel:
* a _Submission Queue_ (SQ)
* a _Completion Queue_ (CQ)
Instead of invoking the Kernel directly to instruct I/O operation, the application rather
fills one or several _Submission Queue Entries_ (SQE) that describe I/O operations --
read, write, timeout, accept,... -- and then notifies the kernel; once the operation has run
to completion, the kernel deposits a _Completion Queue Entry_ (CQE) into the other queue,
carrying the result. Submission and completion are thus decoupled: many operations can be in
flight concurrently, and — unlike the classic `read(2)`/`write(2)` model — no thread has to block
while an operation is pending, and no syscall is required per individual operation (a single
`io_uring_submit` hands over a whole batch, and with a polled queue even that can be elided).
flight concurrently, and -- unlike the classic `read(2)`/`write(2)` model -- no thread or process
has to _block_ while an operation is pending, and no syscall is required per individual operation.
One single `io_uring_submit` hands over a whole batch.footnote:[
As an extension, the kernel can even _poll_ on the submission queue, so that any syscall can elided.
This setup lends itself well for high-performance computing with lots of tiny I/O operation, but
has the obvious downside that now the _kernel_ has to run a dedicated polling thread -- which
in the typical use case means to dedicate one core entirely for the dispatch of operations.
]
For a media application this is attractive: playback and editing require moving large volumes of
data with predictable latency, ideally without dedicating a blocked thread to each stream. The
@ -45,23 +57,26 @@ and binary symbol for each function; obviously using these explicit calls adds a
Reading of raw video as example of using IO_URING
-------------------------------------------------
Playing a raw video file is a minimal yet realistic streaming workload — a steady, latency-bound
sequence of large block reads that has to keep pace with a fixed frame beat — and therefore serves
well to demonstrate the decoupling of submission and completion.
Playing a raw video file is a minimal yet realistic streaming workload -- a steady, latency-bound
sequence of large block reads that has to keep pace with a fixed frame beat -- and therefore serves
well to demonstrate the decoupling of submission and completion. Another aspect demonstrated in here
is the need for _flow control,_ so that the consumer (in this case, the GUI display) is not flooded
with the data being read.
The »Proactor« pattern
~~~~~~~~~~~~~~~~~~~~~~
It is instructive to contrast two patterns for event-driven I/O. In the *Reactor* pattern the
program waits for a resource to become _ready_ and then performs the operation itself, synchronously,
from within the event handler. In the *Proactor* pattern the operation is _initiated_ up front, the
operating system carries it out, and a handler is invoked on _completion_ — with the result already
available. io_uring is inherently proactor-shaped: we submit a read, the kernel performs it, and we
later receive a completion that carries the outcome.
operating system carries it out, and a handler is invoked on _completion_ -- with the result already
available. While io_uring in itself is an abstraction and hides the actual I/O handling in the kernel,
the form of the interface fits well into the Proactor pattern: a read operation is submitted, the kernel
performs it, and a completion can later be received to follow-up with processing of the retrieved data.
The demo realises this with a single dedicated *»Proactor« thread* (`UringVideoPlayer::playVideo()`).
The demo realises this pattern with a single dedicated *»Proactor« thread* (`UringVideoPlayer::playVideo()`).
Its whole life is one loop around `io_uring_wait_cqe()`: block until the next completion arrives,
dispatch it to a handler based on a tag carried in the CQE `user_data`, then re-arm further work and
submit. Two kinds of completion are multiplexed onto the same ring and told apart by that tag:
dispatch it to a handler based on a tag carried in the CQE `user_data`, then re-supply further work items
and submit. Two kinds of completion are multiplexed onto the same ring and told apart by that tag:
* a periodic *frame-beat timer* (a multishot timeout, tagged with the sentinel `TICK_TAG`)
* *read completions* for individual video frames (tagged with the running frame number)
@ -74,23 +89,26 @@ The code is deliberately split so that the io_uring logic stands apart from the
The subject of this write-up. Owns the ring, the buffers and their state, and the Proactor thread.
`EglDisplay` (egl-display.hpp)::
A self-contained OpenGL/EGL display component, best read as a _black box_. It is reached only
through the small `PixelFeed` interface — `retrieveFrame()` to obtain the current frame's pixel
data and `completedFrame()` to signal that the data is no longer needed.
A self-contained OpenGL/EGL display component, best read as a _black box_.footnote:[
In fact we re-used the demo code from last year's talk at FrOSCon(2025)
]
, where various technologies were explored to display raw video frames, using the GPU for acceleration.
]
It is reached only through the small `PixelFeed` interface -- `retrieveFrame()` to obtain the current
frame's pixel data and `completedFrame()` to signal that the data was handed over to the GPU (using OpenGL)
and the buffer can thus be repurposed for the next frame.
`GtkUringApp` (gtk-uring-app.hpp)::
A minimal GTK-3 host application providing the window, the start button and the shutdown hook.
Two threads are involved: the *GTK/GUI main thread* and the *Proactor thread*. The player is
constructed on the GUI thread and — following a RAII spirit — immediately launches the Proactor
constructed on the GUI thread and -- following a RAII pattern -- immediately launches the Proactor
thread from its constructor. Note that `proactorThread_` is declared _last_ among the members, so
by C++'s declaration-order construction rule all buffers, state and the display connection are fully
initialised before the thread can observe them. Data flows in one direction: the Proactor reads file
blocks into buffers and publishes a ready frame; the GUI thread uploads that frame into a GPU texture
and draws it.
by C++'s declaration-order construction rule all the buffers, state and the display connection machinery
are fully initialised before the thread can observe them. Data flows in one direction: the Proactor reads
blocks of raw video data from the file into buffers and publishes a ready frame; the GUI thread uploads
that frame into a GPU texture and draws it.
.Overall data flow across the two threads, with the atomic baton `toDisplay_` between them
image::diagrams/thread-handoff.svg[Cross-thread data flow and visibility handoff,align="center"]
The problem of data visibility synchronisation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -101,8 +119,8 @@ The two threads share the buffer array, which raises two questions of _cross-thr
. the Proactor must know when the GUI is _done_ with a buffer, so the slot may be reused safely.
Rather than making the entire state array atomic, or sharing cursor indices across the boundary, the
design funnels the whole cross-thread contract through a *single* atomic — `std::atomic<Buffer*>
toDisplay_` — which _is_ the payload rather than a key to look it up:
design funnels the whole cross-thread contract through a *single* atomic -- `std::atomic<Buffer*>
toDisplay_` -- which _is_ the payload rather than a key to look it up:
[cols="1,3"]
|===
@ -115,27 +133,29 @@ and thereby is guaranteed to see the buffer contents written before the store. W
GUI writes `nullptr` with a *release*-store, which the Proactor observes with an *acquire*-load
before reclaiming the slot. This is the classic single-slot, single-producer/single-consumer
_mailbox_ handoff: the two release/acquire pairs establish the _happens-before_ relations that carry
buffer-content visibility in both directions. Everything else — the `BuffState` array and the frame
cursors — is confined to the Proactor thread alone and therefore needs no synchronisation at all.
buffer-content visibility in both directions. Everything else -- the `BuffState` array and the frame
cursors -- is confined to the Proactor thread alone and therefore needs no synchronisation at all.
Buffer data structure
~~~~~~~~~~~~~~~~~~~~~
A fixed set of `RING_SIZ` frame-sized buffers is held in a `unique_ptr<Buffer[]>` and used as a
_ring_: the slot for a frame is simply `frameNr % RING_SIZ`. Frame numbers themselves are monotonic,
unbounded counters — `nextRead_` for the production side and `nextShow_` for the consumption side —
_ring_: the slot for a frame can be accessed by taking `frameNr % RING_SIZ`. Frame numbers themselves are
monotonic, unbounded counters -- `nextRead_` for the production side and `nextShow_` for the consumption side --
so the modulo maps an ever-growing frame index onto the small cyclic set of buffers. The same running
frame number does triple duty: as the SQE `user_data` tag that later identifies the completed read, as
the key of the per-slot IO timer, and (via modulo) as the slot index. The io_uring queues are sized
`RING_SIZ+1`, the one extra entry accommodating the always-present timer event.
frame number does double duty: as the SQE `user_data` tag that later identifies the completed read,
and (via modulo) as the slot index. A similar addressing scheme is also employed to access a _state flag_,
that is associated with each buffer slot. Notably the io_uring kernel queues are sized as `RING_SIZ+1`,
the one extra entry accommodating the always-present timer event.
.Snapshot of the ring: the two cursors chase each other, the gap holds the in-flight frames
image::diagrams/ring-buffer.svg[Ring-buffer snapshot with production and consumption cursors,align="center"]
image::demo/ring-buffer.svg[Ring-buffer snapshot with production and co]nsumption cursors,align="center"]
Tracking state of a buffer slot
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Each slot carries a `BuffState` describing where in its life-cycle it currently is:
FREE -> READ -> FEED -> SHOW -> FREE (with STOP as terminal state)
.Buffer-slot state engine (transitions driven by the Proactor thread)
image::demo/state-engine.svg[State-engine for buffer slots,align="center"]
[cols="1,4"]
|===
@ -148,10 +168,9 @@ Each slot carries a `BuffState` describing where in its life-cycle it currently
Because this array is written and read _only_ from the Proactor thread, it is a plain, non-atomic
array. The `SHOW` marker records _which_ slot is presently on loan to the GUI, complementing the
`toDisplay_` baton which records _whether_ it has come back yet.
`toDisplay_` baton which marks _whether_ the GUI has finished transferring the pixel data into
the GPU for display.
.The life-cycle of a single buffer slot (all transitions driven by the Proactor thread)
image::diagrams/state-engine.svg[Buffer-slot state engine,align="center"]
Control flow and backpressure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -162,7 +181,7 @@ a completed frame is _reclaimed_, then the next frame is _published_ if its data
The essential back-pressure is that the single show-cursor does not advance while a frame is still out
with the GUI (`toDisplay_` non-null). No new frame is published until the previous one has been
displayed and released, and — since reads only ever refill `FREE` slots — a lagging consumer naturally
displayed and released, and -- since reads only ever refill `FREE` slots -- a lagging consumer naturally
throttles production. A frame whose read has not completed in time, or whose predecessor the GUI has
not yet released, simply _slips_ to the next tick.
@ -196,7 +215,7 @@ UringVideoPlayer::maybeDisplayFrame()
}
--------------------------------------------------------------------------
<1> reclaim only the slot lent to the GUI, and only once it cleared the baton
<2> advance the single show-cursor — this is where the back-pressure lives
<2> advance the single show-cursor -- this is where the back-pressure lives
<3> the frame data has arrived (`FEED`): hand it over
<4> the release-store publishes the buffer and makes its contents visible to the GUI
<5> `READ` (IO still pending) or `SHOW` (GUI not yet caught up) ⇒ the frame slips to the next tick
<5> `READ` (IO still pending) or `SHOW` (GUI not yet caught up) -> the frame slips to the next tick

28
FrOSCon26/research/Resources.txt
View file

@ -6,11 +6,27 @@ Topic: Kernel and IO
--------------------
Asynchronous IO
~~~~~~~~~~~~~~~
- TODO
Background: Asynchronous IO
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- https://www.kegel.com/c10k.html[The C10k problem]: a very frequently quoted (and actually
quite old page (2003-2011) that analyses the problem of serving more than 10k requests from a server,
the bottlenecks and possible solution strategies.
- https://dreamsongs.com/WorseIsBetter.html[The crux with »worse is better«]. This is a summary
by Richard P. Gabriel himself, looking back, and he still has not made up his mind if _worse is better_.
The tension between the »*Berkely*« (or »New Jersy«) style and the »*MIT* style« helps to understand
design choices taken in the original design of Unix -- and with the advent of fast SSDs and finally
an interface like `io_uring` in the Linux kernel, all these unresolved questions are coming back
to haunt us.
* https://dreamsongs.com/Files/WorseIsBetterPositionPaper.pdf[Back to the Future: Is Worse (Still) Better?]
* https://dreamsongs.com/Files/ProWorseIsBetterPosition.pdf[Back to the Future: Worse (Still) is Better!]
History of io_uring
~~~~~~~~~~~~~~~~~~~
- https://web.archive.org/web/20251010072644/https://kernel.dk/io_uring.pdf[Whitepaper(2019): Efficient IO with io_uring]
- https://lwn.net/Articles/810414/[LWN.net: The rapid growth of io_uring]
- https://web.archive.org/web/20200421163135/https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/[2020: io_uring and eBPF will revolutionize programming in linux]
IO_URING
@ -18,10 +34,13 @@ IO_URING
Introduction
~~~~~~~~~~~~
* https://unixism.net/loti/[»Lord of the uring«]
* https://towardsdev.com/io-uring-vs-epoll-syscall-overhead-linux-performance-a3d7fcc6c9b9?gi=e429808ce081[The Linux I/O Interface That Finally Makes Sense]
* https://lwn.net/Articles/776703/[LWN.2019: Ringing in a new asynchronous I/O API] (explains the low-level API)
Reference
~~~~~~~~~
* https://man7.org/linux/man-pages/man7/io_uring.7.html[Manpage (low-level API)]
* https://unixism.net/loti/ref-liburing/index.html[Overview (liburing)]
** https://unixism.net/loti/ref-liburing/submission.html[Submission and Submission helpers]
@ -52,6 +71,3 @@ Reference
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_epoll_ctl[io_uring_prep_epoll_ctl]
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_add[io_uring_prep_poll_add]
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_remove[io_uring_prep_poll_remove]
[purple]#maybe more resources regarding the low-level API...?#

366
FrOSCon26/research/TimeScales.txt
View file

@ -10,18 +10,23 @@ cycle up to the point where a human gives up waiting. Numbers on that scale
resist intuition, and so we tend to reason about ``fast'' and ``slow'' without
noticing that the two words can be separated by a factor of a million.
The following table collects the figures that matter in this context. It is
meant as a rough orientation, not as a benchmark result.
CAUTION: Knowledge-Technology (»AI«) was used to structure and investigate this topic,
starting from my own practical observations. Additional corroborating sources
were identified and findings integrated into a coherent time axis view.
NOTE: All values are order-of-magnitude figures for contemporary desktop and
laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on
microarchitecture, memory configuration, kernel version and load. The
_ratios_ are the point, not the digits.
The last column rescales everything by a single constant factor, chosen so
that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes one second
of human time. This is the most effective way we know to convey what a cache
miss or a disk seek actually costs from the machine's point of view.
The following table collects the figures that matter in this context. It is
meant as a rough orientation, not as a benchmark result.
The last column in the table rescales everything by a single constant factor,
chosen so that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes
one second of human time. Presenting this kind of _human scale proportions_
is an effective way to convey what a cache miss or a disk seek actually costs,
and what this means, assuming a _machine's point of view_.
Overview Table
@ -38,40 +43,39 @@ Overview Table
| ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds
| ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds
| ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` footnoteref:[lkml,LKML thread "[PATCH 3/4] x86,asm: Re-work smp_store_mb()", Nov 2015. Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds reports that on Haswell "lock addq" costs pretty much exactly half of MFENCE, and explains that locked operations were the ones CPU designers had reason to optimise -- https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html] | ≈{nbsp}17 seconds
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` <<lkml>> | ≈{nbsp}17 seconds
| ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW footnoteref:[downs,Travis Downs, "A Concurrency Cost Hierarchy", 6 July 2020, see the discussion of "vanilla" versus "non-vanilla" instructions: on x86 plain accesses and non-atomic read-modify-writes are cheap while LOCK-prefixed RMWs are an order of magnitude slower, with MFENCE comparable to a locked instruction -- https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html] | ≈{nbsp}33 seconds
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW <<downs>> | ≈{nbsp}33 seconds
| ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row footnoteref:[cxl,Measured values in Table III of: Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototype, arXiv:2503.22017. Local DDR5 median latency 122.9 ns under Intel MLC pointer chasing versus 13.1 ns for a plain load; methodology (rdtsc, 10 000 repetitions, median) described in section III-C -- https://arxiv.org/pdf/2503.22017] | ≈{nbsp}43 seconds
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 footnote:[Measurement-oriented analysis of DDR5 sub-channel architecture; the relevant passage is in section "Latency Implications", noting that both DDR4 and DDR5 sit at approximately 14 ns first-access latency at their respective standard speeds. arXiv:2605.08725 -- https://arxiv.org/pdf/2605.08725]
| ≈{nbsp}47 seconds
| 40--100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 footnoteref:[cxl] | *≈{nbsp}5 minutes*
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row <<cxl>> | ≈{nbsp}43 seconds
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 <<ddr5b>> | ≈{nbsp}47 seconds
| 40...100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 <<cxl>> | *≈{nbsp}5 minutes*
4+^s| Nanoseconds to microseconds -- entering the kernel
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call footnoteref:[arkanis,"Measurements of system call performance and overhead", Arkanis Development, 5 Jan 2017 -- compares a plain function call against getpid() issued via the syscall instruction and via the vDSO -- https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/] | ≈{nbsp}1.5 minutes
| 70--100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host footnoteref:[gms,Georg Sauthoff, "On the Costs of Syscalls", 30 Aug 2021. Median of 100 repetitions across a heterogeneous set of hosts; the conclusion is that user-kernel mode switches cost on the order of a few hundred nanoseconds on all hosts, with the fastest host switching modes in under 100 ns, and that the higher figures are explained by CPU-bug mitigations being enabled by default -- https://gms.tf/on-the-costs-of-syscalls.html] | 4--6 minutes
| 200--600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled footnoteref:[gms] | 11--33 minutes
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call <<arkanis>> | ≈{nbsp}1.5 minutes
| 70...100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host <<gms>> | 4--6 minutes
| 200...600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled <<gms>> | 11--33 minutes
| ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour*
| ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours
4+^s| Microseconds -- the operating system
| 10--30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU footnoteref:[lkmlwake,Michal Schmidt, reply in the thread "nanosleep() accuracy", linux-kernel mailing list, 17 Aug 2007: the point being that a process always carries non-zero wake-up latency covering interrupt processing, waking and scheduling, and that 10-30 us is not unreasonable -- https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html] | 9{nbsp}hours -- 1{nbsp}day
| ~31--57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked footnoteref:[rhel,Sample cyclictest output and interpretation in: Finding Realtime Linux Kernel Latencies, Clark Williams (Red Hat), section "Using cyclictest" -- https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt] | 1--2 days
| ~59--90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts footnoteref:[rhel] | 2--3.5 days
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups footnoteref:[slack,Primary documentation: `man 7 time`, section "High-resolution timers", and `man 2 prctl`, description of PR_SET_TIMERSLACK. The 50 us default for non-realtime threads is stated explicitly in: Metronome -- adaptive and precise intermittent packet retrieval in DPDK, section II (discussion of nanosleep limits), arXiv:2103.13263 -- https://arxiv.org/pdf/2103.13263 ; a practical encounter with the same default, including the /proc interface, is documented in OpenJDK issue JDK-8307766 -- https://bugs.openjdk.org/browse/JDK-8307766] | *≈{nbsp}2 days*
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` footnoteref:[slack] | ≈{nbsp}3 seconds
| ~9{nbsp}µs | SSD read, best case | Optane / Z-NAND class at QD1 footnoteref:[anand,AnandTech, review of the Samsung 983 ZET (Z-NAND), page 3, "QD1 Random Read Performance" -- explains why queue depth 1 exposes best-case access latency, and reports an average of just under 9 us for an Optane drive against 16 us for the Z-NAND based 983 ZET -- https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3] | ≈{nbsp}8 hours
| 20--70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive footnoteref:[simply,Summary of the storage latency hierarchy: a modern NVMe SSD serves a random 4K read in roughly 20-70 us, a SATA SSD in 100-200 us, and a spinning disk in 5-10 ms. simplyblock glossary entry "NVMe Latency" -- https://simplyblock.io/glossary/nvme-latency/] | 1--3 days
| 50--100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
| 100--200{nbsp}µs | SATA SSD 4K random read | the interface costs, not just the medium footnoteref:[simply] | 4--8 days
| 100{nbsp}µs -- 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
| 10...30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU <<lkmlwake>> | 9{nbsp}hours -- 1{nbsp}day
| ~31...57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked <<rhel>> | 1--2 days
| ~59...90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts <<rhel>> | 2--3.5 days
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups <<slack>> | *≈{nbsp}2 days*
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` <<slack>> | ≈{nbsp}3 seconds
| ~10{nbsp}µs | SSD read, best case | Optane-class (3D XPoint) at QD1 <<intel>> | ≈{nbsp}8 hours
| 20...70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive <<simply>> | 1--3 days
| 50...100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
| 100...200{nbsp}µs| SATA SSD 4K random read | the interface costs, not just the medium <<simply>> | 4--8 days
| 100{nbsp}µs ... 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
4+^s| Milliseconds -- where machine and human meet
| 5--10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades footnoteref:[simply] | *6{nbsp}months -- 1{nbsp}year*
| 5...10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades <<simply>> | *6{nbsp}months -- 1{nbsp}year*
| ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year
| 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years
| 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years*
@ -112,24 +116,24 @@ become visible to another core at the moment the instruction retires: it first
lands in that core's store buffer. Cache coherence guarantees that the cores
eventually agree on the content of a cache line, but it says nothing about
_when_, nor about the order in which several independent writes become visible.
This is what memory fences are for -- they constrain the order, and thereby
allow one core to reason about what an adjacent core has already done. The cost
shows up in the table twice: an uncontended fence is roughly the price of an
L3 hit, whereas a cache line actually bouncing between two cores costs about as
much as going to DRAM. It is worth internalising that the expensive part is
usually not the synchronisation primitive itself, but the coherence traffic it
provokes.
This is what memory fences (memory barriers) are for -- they constrain the order,
and thereby allow one core to reason about what an adjacent core has already done.
The cost shows up in the table twice: an uncontended fence is roughly the price of
an L3 hit, whereas a cache line actually bouncing between two cores costs about as
much as going to DRAM. It should be noted that the expensive part is usually _not_
the synchronisation primitive itself, but the cache coherence traffic it provokes.
x86 implements a comparatively strong model, _Total Store Order_ (TSO): of the
four possible reorderings, only StoreLoad is permitted. Acquire and release
semantics are therefore free on x86 -- they compile to plain `MOV` -- and only
sequential consistency requires an actual instruction. ARM and POWER are
weakly ordered and consequently offer a more differentiated set of barriers:
ARM distinguishes `DMB` from the heavier `DSB`, and provides graded load
acquire flavours such as `LDAR` and `LDAPR`; POWER separates the heavyweight
`sync` from the cheaper `lwsync`. The trap for the unwary is that C++
`memory_order` distinctions which cost nothing on x86 have a real price on
ARM -- and, worse, that under-synchronised code can appear perfectly correct
And as far as memory synchronisation is concerned, there is another twist that
has already led quite some programmers astray. Intel's x86 implements a comparatively
strong coherency model, _Total Store Order_ (TSO): of the four possible reorderings,
only StoreLoad is permitted. Acquire and release semantics are therefore free on x86 --
they compile to plain `MOV` -- and only sequential consistency requires an actual
instruction. Quite to the contrary, ARM and POWER are weakly ordered and consequently
offer a more differentiated set of barriers: ARM distinguishes `DMB` from the heavier
`DSB`, and provides graded load acquire flavours such as `LDAR` and `LDAPR`; POWER
separates the heavyweight `sync` from the cheaper `lwsync`. The trap for the unwary
is that C++ `memory_order` distinctions which cost nothing on x86 have a real price
on ARM -- and, worse, that under-synchronised code can appear perfectly correct
on x86 and fail on ARM.
@ -145,17 +149,19 @@ attendant TLB consequences) and one or more branch-prediction barriers --
retpolines, IBPB or IBRS depending on hardware and configuration. On return,
the whole dance runs in reverse.
The direct cost is measurable: a minimal syscall such as `getpid()` amounts to
little more than two mode switches, and lands in the region of a hundred
The direct cost is measurable: a minimal syscall such as `getpid()` amounts
to little more than two mode switches, and lands in the region of a hundred
nanoseconds on a fast host without mitigations, rising to several hundred once
mitigations are enabled.footnoteref:[gms] The indirect cost is harder to see
mitigations are enabled <<gms>>. The indirect cost is harder to see
and frequently larger. Kernel entry code and the syscall handler evict
user-space data from L1 and L2, and disturb the branch predictors; the
application resumes with a cold working set and pays for it in misses that no
syscall benchmark will attribute to the syscall. This is precisely why the
interesting question is rarely ``how expensive is one syscall'' but rather
``how many syscalls does this design require per unit of work'' -- which is
the question `io_uring` was built to answer.
syscall benchmark will attribute to the syscall. A simple `read()` that happens
to hit data already in the kernel cache can incur costs of around 1{nbsp}µs.
The interesting question thus is is rarely ``how expensive is one syscall'' but
rather ``how many syscalls does this design require per unit of work'' --
which is the question `io_uring` was built to answer.
Operating System Slack
@ -165,100 +171,244 @@ A thread that asks to sleep for a specific duration will not, in general, wake
at that instant. Part of this is unavoidable: an interrupt must be handled, the
sleeping task must be moved to the run queue, and the scheduler must actually
place it on a CPU, which together account for something in the range of
10--30{nbsp}µs.footnoteref:[lkmlwake]
10--30{nbsp}µs <<lkmlwake>>.
The larger part, however, is deliberate. Linux maintains a per-thread _timer
slack_ value, and the kernel is free to delay a timed wake-up by up to that
amount in order to coalesce several wake-ups into one and thereby let the CPU
stay in a low-power state longer. For every thread not running under a
real-time scheduling policy, the default is 50{nbsp}µs.footnoteref:[slack]
real-time scheduling policy, the default is 50{nbsp}µs <<slack>>.
This is not imprecision; it is a power-versus-precision trade that the kernel
makes on the application's behalf, and it can be adjusted -- down to
1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process
`/proc/PID/timerslack_ns` interface.
The distinction worth carrying away is between _resolution_ and _accuracy_.
The key distinction is therefore that between _resolution_ and _accuracy_.
Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond
resolution, and `clock_getres()` will happily confirm it. The accuracy with
which a `nanosleep()` actually returns is a different quantity altogether, two
which a `nanosleep()` actually returns is a different story altogether, two
to three orders of magnitude coarser, and dominated by slack, wake-up latency
and -- if the core had gone idle -- C-state exit time. Any design that assumes
it can steer a worker thread with microsecond-scale sleeps should be checked
against these figures first.
// ----------------------------------------------------------------------
// Footnote definitions are attached at first use above via footnoteref.
// Repeated references reuse the id; see AsciiDoc footnoteref macro.
// ----------------------------------------------------------------------
Sources
-------
The footnotes above carry the precise locations. Collected here as a reading
list, roughly in order of usefulness for further work.
The in-text markers such as <<downs>> link down to the full citation here.
Collected as a reading list, roughly in order of usefulness for further work.
- Travis Downs, *A Concurrency Cost Hierarchy* --
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html +
The best single entry point on why synchronisation costs what it costs.
[[downs]]
--
downs::
Travis Downs (6 July 2020) --
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html[
A Concurrency Cost Hierarchy
]
//Archive https://web.archive.org/web/20260801071948/https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html
+
The best single entry point on why synchronisation costs what it costs.
+
See the discussion of "vanilla" versus "non-vanilla" instructions: on x86
plain accesses and non-atomic read-modify-writes are cheap while
LOCK-prefixed RMWs are an order of magnitude slower, with `MFENCE`
comparable to a locked instruction.
--
- LKML thread, *[PATCH 3/4] x86,asm: Re-work smp_store_mb()* (Nov 2015) --
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html +
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`.
[[lkml]]
--
lkml::
LKML thread (Nov 2015) --
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html[
&#91;PATCH 3/4&#93; x86,asm: Re-work smp_store_mb()
]
//Archive https://web.archive.org/web/20240818210644/https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html
+
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`:
Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds
reports that on Haswell `lock addq` costs pretty much exactly half of
MFENCE, and explains that locked operations were the ones CPU designers had
reason to optimise.
--
- Georg Sauthoff, *On the Costs of Syscalls* (Aug 2021) --
https://gms.tf/on-the-costs-of-syscalls.html +
Careful cross-host microbenchmark of the mode-switch cost.
[[gms]]
--
gms::
Georg Sauthoff (30 Aug 2021) --
https://gms.tf/on-the-costs-of-syscalls.html[
On the Costs of Syscalls
]
//Archive https://web.archive.org/web/20260725162647/https://gms.tf/on-the-costs-of-syscalls.html
+
Careful cross-host microbenchmark of the mode-switch cost.
+
Median of 100 repetitions across a heterogeneous set of hosts; the
conclusion is that user-kernel mode switches cost on the order of a few
hundred nanoseconds on all hosts, with the fastest host switching modes in
under 100 ns, and that the higher figures are explained by CPU-bug
mitigations being enabled by default.
--
- *Measurements of system call performance and overhead* (Jan 2017) --
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ +
Function call versus `syscall` instruction versus vDSO.
[[arkanis]]
--
arkanis::
Arkanis Development (5 Jan 2017) --
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/[
Measurements of system call performance and overhead
]
//Archive https://web.archive.org/web/20260311003705/https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/
+
Compares a plain function call against `getpid()` issued via the syscall
instruction and via the vDSO.
--
- Brendan Gregg, *KPTI/KAISER Meltdown Initial Performance Regressions* --
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html +
Relates mitigation overhead to syscall rate, context switch rate and working
set size -- the systemic view rather than the per-call view.
[[gregg]]
--
gregg::
Brendan Gregg (9 Feb 2018) --
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html[
KPTI/KAISER Meltdown Initial Performance Regressions
]
//Archive https://web.archive.org/web/20260603211334/https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html
+
Relates mitigation overhead to syscall rate, context switch rate and working
set size -- the systemic view rather than the per-call view.
--
- Clark Williams (Red Hat), *Finding Realtime Linux Kernel Latencies* --
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt +
Walks through a `cyclictest` run and how to read its output.
[[lkmlwake]]
--
lkmlwake::
Michal Schmidt (17 Aug 2007) --
https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html[
Reply in the LKML thread "nanosleep() accuracy"
]
//Archive https://web.archive.org/web/20260824155617/https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html
+
The point being that a process always carries non-zero wake-up latency
covering interrupt processing, waking and scheduling, and that
10--30{nbsp}µs is not unreasonable.
--
- `man 7 time` and `man 2 prctl` (PR_SET_TIMERSLACK) +
Primary documentation for timer slack. The 50{nbsp}µs default is stated in
https://arxiv.org/pdf/2103.13263 (Metronome, DPDK packet retrieval) and
encountered in practice in https://bugs.openjdk.org/browse/JDK-8307766
[[rhel]]
--
rhel::
Clark Williams (Red Hat) --
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt[
Finding Realtime Linux Kernel Latencies
]
//Archive https://web.archive.org/web/20250620155700/https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt
+
Walks through a `cyclictest` run and how to read its output: sample
`cyclictest` output and interpretation, section "Using cyclictest".
--
- *Performance Characterizations ... Samsung CXL Memory Module Hybrid
Prototype*, arXiv:2503.22017 -- https://arxiv.org/pdf/2503.22017 +
Table III contains a well-documented measurement of plain local DDR5.
[[slack]]
--
slack::
Primary documentation --
`man 7 time`, section "High-resolution timers", and `man 2 prctl`,
description of `PR_SET_TIMERSLACK`
+
The 50{nbsp}µs default for non-realtime threads is stated explicitly in
https://arxiv.org/pdf/2103.13263[
Metronome -- adaptive and precise intermittent packet retrieval in DPDK
]
//Archive https://web.archive.org/web/20260725162634/https://arxiv.org/pdf/2103.13263
(section II, discussion of nanosleep limits), and a practical encounter
with the same default, including the `/proc` interface, is documented in
https://bugs.openjdk.org/browse/JDK-8307766[
OpenJDK issue JDK-8307766
].
//Archive https://web.archive.org/web/20251130011408/https://bugs.openjdk.org/browse/JDK-8307766
--
- *Single 32-bit Sub-Channel DDR5 DIMMs*, arXiv:2605.08725 --
https://arxiv.org/pdf/2605.08725 +
Useful for the point that DDR5 did not improve absolute latency over DDR4.
[[cxl]]
--
cxl::
arXiv:2503.22017 --
https://arxiv.org/pdf/2503.22017[
Performance Characterizations and Usage Guidelines of Samsung CXL Memory
Module Hybrid Prototype
]
//Archive https://web.archive.org/web/20260811083139/https://arxiv.org/pdf/2503.22017
+
Table III contains a well-documented measurement of plain local DDR5:
median latency 122.9{nbsp}ns under Intel MLC pointer chasing versus
13.1{nbsp}ns for a plain load; methodology (rdtsc, 10 000 repetitions,
median) described in section III-C.
--
- simplyblock glossary, *NVMe Latency* --
https://simplyblock.io/glossary/nvme-latency/ +
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy.
[[ddr5b]]
--
ddr5b::
arXiv:2605.08725 --
https://arxiv.org/pdf/2605.08725[
Single 32-bit Sub-Channel DDR5 DIMMs
]
//Archive https://web.archive.org/web/20260607161507/https://arxiv.org/pdf/2605.08725
+
Useful for the point that DDR5 did not improve absolute latency over DDR4.
+
The relevant passage is in section "Latency Implications", noting that both
DDR4 and DDR5 sit at approximately 14{nbsp}ns first-access latency at their
respective standard speeds.
--
- AnandTech, *Samsung 983 ZET (Z-NAND) review*, page 3 --
https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3 +
Explains why queue depth 1 exposes best-case access latency.
[[simply]]
--
simply::
simplyblock glossary --
https://simplyblock.io/glossary/nvme-latency/[
NVMe Latency
]
//Archive https://web.archive.org/web/20260418070043/https://simplyblock.io/glossary/nvme-latency/
+
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy: a modern
NVMe SSD serves a random 4K read in roughly 20--70{nbsp}µs, a SATA SSD in
100--200{nbsp}µs, and a spinning disk in 5--10{nbsp}ms.
--
[[intel]]
--
intel::
Intel --
https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf[
Optane SSD DC P4800X Evaluation Guide
]
//Archive https://web.archive.org/web/20260824160928/https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf
+
In Chapter 1.1, Page 5: ``at queue depth of one, 99 out of 100 read operations ...
complete in single digit microseconds.''
+
Corroborated independently by
https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html[
Tom's Hardware's 2017 preview,
]
//Archive https://web.archive.org/web/20260313082242/https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html
which reports 10 µs latency for the same drive.
+
Note: Optane was discontinued by Intel in 2022; the figure is retained as the
best-documented physical floor for this latency class.
--
A Note on Microbenchmarks
-------------------------
One caveat deserves to be stated separately, because it explains why published
figures for synchronisation primitives vary so wildly. A hot microbenchmark
keeps every thread runnable and every core out of idle by construction. Under
those conditions an uncontended atomic operation or a mutex acquisition costs
single-digit to low-three-digit nanoseconds, and a benchmark will faithfully
report that. What such a benchmark structurally cannot observe is the cost of
_blocking_ -- of a thread actually parking and later being woken -- because
pressing the benchmark is precisely what prevents that from happening.
One caveat deserves a separate mention, as it explains why the published
figures on synchronisation primitives differ so drastically from one another.
A hot microbenchmark keeps every thread runnable and every core out of idle by
its very construction. Under those conditions an uncontended atomic operation or
a Mutex acquisition costs single-digit to low-three-digit nanoseconds, and a
benchmark will faithfully report that. What such a benchmark _structurally_
cannot observe is the cost of _blocking_ -- of a thread actually parking and
later being woken -- because pressing the benchmark is precisely what
prevents that from happening.
The instruction-level cost of a synchronisation primitive is what a hot
microbenchmark measures. Its systemic cost -- coherence traffic under
contention, and blocking and wake-up latency when it is not contended -- is
what production tail latency measures. The two can differ by two to four orders
of magnitude, and the table above spans both regimes for exactly that reason.
contention, and blocking and wake-up latency when it is not contended --
is what production tail latency measures. The two values can differ by
two to four orders of magnitude, and it is for this very reason that
the table above covers both regimes.