ichthyo/wegen
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: ichthyo:2947344eda1f5fe11009515112ac14294e29107a
Branches Tags
ichthyo:master
ichthyo:compat/hiv
...
compare: ichthyo:3a48f3ea12c8093fab4afa475e04ff161d5f0512
Branches Tags
ichthyo:master
ichthyo:compat/hiv

2 commits
2947344eda ... 3a48f3ea12

Author SHA1 Message Date
Ichthyostega
3a48f3ea12 Compat: Check for successful GLEW initialisation fails on my PC / Trixie 
I observed same strange behaviour already last year on my »Video Output« talk.
The return value indicates an unkown error, but the subsequent usage seemingly works...

On my Laptop, I don not see this error; it might be due to the very old graphics card on my PC...
 2026-08-25 01:05:02 +02:00
Ichthyostega
fe195d101b ++ FrOSCon25 ++ Vortrag erfolgreich gehalten ++ 
- Notizen zum Publikum
- Abschließende Dokumentation
- Resourcen in die Lumiera-Website ingegriert
 2026-08-25 05:13:00 +02:00
5 changed files with 820 additions and 380 deletions
Show stats Expand all files Collapse all files

660
FrOSCon26/FrOSCon.26.mm
View file

File diff suppressed because it is too large Load diff

7
FrOSCon26/examples/egl-display.hpp
View file

@ -183,9 +183,10 @@ EglDisplay<PIX,WIDTH,HEIGHT>::openDisplay (Gtk::Window& appWindow)
// use lib-GLEW to manage OpenGL extensions (notably GL Shader Language) // use lib-GLEW to manage OpenGL extensions (notably GL Shader Language)
GLenum glewErr = glewInit(); GLenum glewErr = glewInit();
if (GLEW_OK != glewErr) ////////////////////////////////////////////////////////////////////////////////////TODO for some reason I get glewErr == 4 ("unknown error"), but it works non the less
__FAIL ("could not bind OpenGL extensions through lib-GLEW:\n" //if (GLEW_OK != glewErr)
+string{reinterpret_cast<const char*> (glewGetErrorString(glewErr))}); // __FAIL ("could not bind OpenGL extensions through lib-GLEW:\n"
// +string{reinterpret_cast<const char*> (glewGetErrorString(glewErr))});
auto compileShader = [](GLenum kind, string code) auto compileShader = [](GLenum kind, string code)

109
FrOSCon26/research/DemoSpec.txt
View file

@ -6,21 +6,33 @@ Specification for the Demo
:toclevels: 3 :toclevels: 3
This specification is the result of our research and provides a short summary of the relevant This specification is the result of our research and provides a short summary of the relevant
aspects of io_uring. Furthermore, the specific design of the demo code is explained and motivated. aspects of io_uring. A simple video player was developed as demo code and explained in a
https://programm.froscon.org/froscon2026/talk/d1635d41-1fac-4a43-909b-b8f07672c541/[Talk at FrOSCon 2026].
The following text gives a summary and motivates the specific design of the demo code.
IO_URING kernel interface IO_URING kernel interface
------------------------- -------------------------
`io_uring` is a Linux kernel interface for asynchronous I/O (available since kernel 5.1). It is `io_uring` is a Linux kernel interface for asynchronous I/O (available since kernel 5.1).
built around two ring buffers, shared and memory-mapped between user space and kernel: a It is built around two ring buffers, shared and memory-mapped between user space and kernel:
_Submission Queue_ (SQ) and a _Completion Queue_ (CQ). The application fills a _Submission Queue
Entry_ (SQE) describing an operation — read, write, timeout, accept, … — and notifies the kernel; * a _Submission Queue_ (SQ)
once the operation has run to completion, the kernel deposits a _Completion Queue Entry_ (CQE) * a _Completion Queue_ (CQ)
Instead of invoking the Kernel directly to instruct I/O operation, the application rather
fills one or several _Submission Queue Entries_ (SQE) that describe I/O operations --
read, write, timeout, accept,... -- and then notifies the kernel; once the operation has run
to completion, the kernel deposits a _Completion Queue Entry_ (CQE) into the other queue,
carrying the result. Submission and completion are thus decoupled: many operations can be in carrying the result. Submission and completion are thus decoupled: many operations can be in
flight concurrently, and — unlike the classic `read(2)`/`write(2)` model — no thread has to block flight concurrently, and -- unlike the classic `read(2)`/`write(2)` model -- no thread or process
while an operation is pending, and no syscall is required per individual operation (a single has to _block_ while an operation is pending, and no syscall is required per individual operation.
`io_uring_submit` hands over a whole batch, and with a polled queue even that can be elided). One single `io_uring_submit` hands over a whole batch.footnote:[
As an extension, the kernel can even _poll_ on the submission queue, so that any syscall can elided.
This setup lends itself well for high-performance computing with lots of tiny I/O operation, but
has the obvious downside that now the _kernel_ has to run a dedicated polling thread -- which
in the typical use case means to dedicate one core entirely for the dispatch of operations.
]
For a media application this is attractive: playback and editing require moving large volumes of For a media application this is attractive: playback and editing require moving large volumes of
data with predictable latency, ideally without dedicating a blocked thread to each stream. The data with predictable latency, ideally without dedicating a blocked thread to each stream. The
@ -45,23 +57,26 @@ and binary symbol for each function; obviously using these explicit calls adds a
Reading of raw video as example of using IO_URING Reading of raw video as example of using IO_URING
------------------------------------------------- -------------------------------------------------
Playing a raw video file is a minimal yet realistic streaming workload — a steady, latency-bound Playing a raw video file is a minimal yet realistic streaming workload -- a steady, latency-bound
sequence of large block reads that has to keep pace with a fixed frame beat — and therefore serves sequence of large block reads that has to keep pace with a fixed frame beat -- and therefore serves
well to demonstrate the decoupling of submission and completion. well to demonstrate the decoupling of submission and completion. Another aspect demonstrated in here
is the need for _flow control,_ so that the consumer (in this case, the GUI display) is not flooded
with the data being read.
The »Proactor« pattern The »Proactor« pattern
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
It is instructive to contrast two patterns for event-driven I/O. In the *Reactor* pattern the It is instructive to contrast two patterns for event-driven I/O. In the *Reactor* pattern the
program waits for a resource to become _ready_ and then performs the operation itself, synchronously, program waits for a resource to become _ready_ and then performs the operation itself, synchronously,
from within the event handler. In the *Proactor* pattern the operation is _initiated_ up front, the from within the event handler. In the *Proactor* pattern the operation is _initiated_ up front, the
operating system carries it out, and a handler is invoked on _completion_ — with the result already operating system carries it out, and a handler is invoked on _completion_ -- with the result already
available. io_uring is inherently proactor-shaped: we submit a read, the kernel performs it, and we available. While io_uring in itself is an abstraction and hides the actual I/O handling in the kernel,
later receive a completion that carries the outcome. the form of the interface fits well into the Proactor pattern: a read operation is submitted, the kernel
performs it, and a completion can later be received to follow-up with processing of the retrieved data.
The demo realises this with a single dedicated *»Proactor« thread* (`UringVideoPlayer::playVideo()`). The demo realises this pattern with a single dedicated *»Proactor« thread* (`UringVideoPlayer::playVideo()`).
Its whole life is one loop around `io_uring_wait_cqe()`: block until the next completion arrives, Its whole life is one loop around `io_uring_wait_cqe()`: block until the next completion arrives,
dispatch it to a handler based on a tag carried in the CQE `user_data`, then re-arm further work and dispatch it to a handler based on a tag carried in the CQE `user_data`, then re-supply further work items
submit. Two kinds of completion are multiplexed onto the same ring and told apart by that tag: and submit. Two kinds of completion are multiplexed onto the same ring and told apart by that tag:
* a periodic *frame-beat timer* (a multishot timeout, tagged with the sentinel `TICK_TAG`) * a periodic *frame-beat timer* (a multishot timeout, tagged with the sentinel `TICK_TAG`)
* *read completions* for individual video frames (tagged with the running frame number) * *read completions* for individual video frames (tagged with the running frame number)
@ -74,23 +89,26 @@ The code is deliberately split so that the io_uring logic stands apart from the
The subject of this write-up. Owns the ring, the buffers and their state, and the Proactor thread. The subject of this write-up. Owns the ring, the buffers and their state, and the Proactor thread.
`EglDisplay` (egl-display.hpp):: `EglDisplay` (egl-display.hpp)::
A self-contained OpenGL/EGL display component, best read as a _black box_. It is reached only A self-contained OpenGL/EGL display component, best read as a _black box_.footnote:[
through the small `PixelFeed` interface — `retrieveFrame()` to obtain the current frame's pixel In fact we re-used the demo code from last year's talk at FrOSCon(2025)
data and `completedFrame()` to signal that the data is no longer needed. ]
, where various technologies were explored to display raw video frames, using the GPU for acceleration.
]
It is reached only through the small `PixelFeed` interface -- `retrieveFrame()` to obtain the current
frame's pixel data and `completedFrame()` to signal that the data was handed over to the GPU (using OpenGL)
and the buffer can thus be repurposed for the next frame.
`GtkUringApp` (gtk-uring-app.hpp):: `GtkUringApp` (gtk-uring-app.hpp)::
A minimal GTK-3 host application providing the window, the start button and the shutdown hook. A minimal GTK-3 host application providing the window, the start button and the shutdown hook.
Two threads are involved: the *GTK/GUI main thread* and the *Proactor thread*. The player is Two threads are involved: the *GTK/GUI main thread* and the *Proactor thread*. The player is
constructed on the GUI thread and — following a RAII spirit — immediately launches the Proactor constructed on the GUI thread and -- following a RAII pattern -- immediately launches the Proactor
thread from its constructor. Note that `proactorThread_` is declared _last_ among the members, so thread from its constructor. Note that `proactorThread_` is declared _last_ among the members, so
by C++'s declaration-order construction rule all buffers, state and the display connection are fully by C++'s declaration-order construction rule all the buffers, state and the display connection machinery
initialised before the thread can observe them. Data flows in one direction: the Proactor reads file are fully initialised before the thread can observe them. Data flows in one direction: the Proactor reads
blocks into buffers and publishes a ready frame; the GUI thread uploads that frame into a GPU texture blocks of raw video data from the file into buffers and publishes a ready frame; the GUI thread uploads
and draws it. that frame into a GPU texture and draws it.
.Overall data flow across the two threads, with the atomic baton `toDisplay_` between them
image::diagrams/thread-handoff.svg[Cross-thread data flow and visibility handoff,align="center"]
The problem of data visibility synchronisation The problem of data visibility synchronisation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -101,8 +119,8 @@ The two threads share the buffer array, which raises two questions of _cross-thr
. the Proactor must know when the GUI is _done_ with a buffer, so the slot may be reused safely. . the Proactor must know when the GUI is _done_ with a buffer, so the slot may be reused safely.
Rather than making the entire state array atomic, or sharing cursor indices across the boundary, the Rather than making the entire state array atomic, or sharing cursor indices across the boundary, the
design funnels the whole cross-thread contract through a *single* atomic — `std::atomic<Buffer*> design funnels the whole cross-thread contract through a *single* atomic -- `std::atomic<Buffer*>
toDisplay_` — which _is_ the payload rather than a key to look it up: toDisplay_` -- which _is_ the payload rather than a key to look it up:
[cols="1,3"] [cols="1,3"]
|=== |===
@ -115,27 +133,29 @@ and thereby is guaranteed to see the buffer contents written before the store. W
GUI writes `nullptr` with a *release*-store, which the Proactor observes with an *acquire*-load GUI writes `nullptr` with a *release*-store, which the Proactor observes with an *acquire*-load
before reclaiming the slot. This is the classic single-slot, single-producer/single-consumer before reclaiming the slot. This is the classic single-slot, single-producer/single-consumer
_mailbox_ handoff: the two release/acquire pairs establish the _happens-before_ relations that carry _mailbox_ handoff: the two release/acquire pairs establish the _happens-before_ relations that carry
buffer-content visibility in both directions. Everything else — the `BuffState` array and the frame buffer-content visibility in both directions. Everything else -- the `BuffState` array and the frame
cursors — is confined to the Proactor thread alone and therefore needs no synchronisation at all. cursors -- is confined to the Proactor thread alone and therefore needs no synchronisation at all.
Buffer data structure Buffer data structure
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
A fixed set of `RING_SIZ` frame-sized buffers is held in a `unique_ptr<Buffer[]>` and used as a A fixed set of `RING_SIZ` frame-sized buffers is held in a `unique_ptr<Buffer[]>` and used as a
_ring_: the slot for a frame is simply `frameNr % RING_SIZ`. Frame numbers themselves are monotonic, _ring_: the slot for a frame can be accessed by taking `frameNr % RING_SIZ`. Frame numbers themselves are
unbounded counters — `nextRead_` for the production side and `nextShow_` for the consumption side — monotonic, unbounded counters -- `nextRead_` for the production side and `nextShow_` for the consumption side --
so the modulo maps an ever-growing frame index onto the small cyclic set of buffers. The same running so the modulo maps an ever-growing frame index onto the small cyclic set of buffers. The same running
frame number does triple duty: as the SQE `user_data` tag that later identifies the completed read, as frame number does double duty: as the SQE `user_data` tag that later identifies the completed read,
the key of the per-slot IO timer, and (via modulo) as the slot index. The io_uring queues are sized and (via modulo) as the slot index. A similar addressing scheme is also employed to access a _state flag_,
`RING_SIZ+1`, the one extra entry accommodating the always-present timer event. that is associated with each buffer slot. Notably the io_uring kernel queues are sized as `RING_SIZ+1`,
the one extra entry accommodating the always-present timer event.
.Snapshot of the ring: the two cursors chase each other, the gap holds the in-flight frames .Snapshot of the ring: the two cursors chase each other, the gap holds the in-flight frames
image::diagrams/ring-buffer.svg[Ring-buffer snapshot with production and consumption cursors,align="center"] image::demo/ring-buffer.svg[Ring-buffer snapshot with production and co]nsumption cursors,align="center"]
Tracking state of a buffer slot Tracking state of a buffer slot
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Each slot carries a `BuffState` describing where in its life-cycle it currently is: Each slot carries a `BuffState` describing where in its life-cycle it currently is:
FREE -> READ -> FEED -> SHOW -> FREE (with STOP as terminal state) .Buffer-slot state engine (transitions driven by the Proactor thread)
image::demo/state-engine.svg[State-engine for buffer slots,align="center"]
[cols="1,4"] [cols="1,4"]
|=== |===
@ -148,10 +168,9 @@ Each slot carries a `BuffState` describing where in its life-cycle it currently
Because this array is written and read _only_ from the Proactor thread, it is a plain, non-atomic Because this array is written and read _only_ from the Proactor thread, it is a plain, non-atomic
array. The `SHOW` marker records _which_ slot is presently on loan to the GUI, complementing the array. The `SHOW` marker records _which_ slot is presently on loan to the GUI, complementing the
`toDisplay_` baton which records _whether_ it has come back yet. `toDisplay_` baton which marks _whether_ the GUI has finished transferring the pixel data into
the GPU for display.
.The life-cycle of a single buffer slot (all transitions driven by the Proactor thread)
image::diagrams/state-engine.svg[Buffer-slot state engine,align="center"]
Control flow and backpressure Control flow and backpressure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -162,7 +181,7 @@ a completed frame is _reclaimed_, then the next frame is _published_ if its data
The essential back-pressure is that the single show-cursor does not advance while a frame is still out The essential back-pressure is that the single show-cursor does not advance while a frame is still out
with the GUI (`toDisplay_` non-null). No new frame is published until the previous one has been with the GUI (`toDisplay_` non-null). No new frame is published until the previous one has been
displayed and released, and — since reads only ever refill `FREE` slots — a lagging consumer naturally displayed and released, and -- since reads only ever refill `FREE` slots -- a lagging consumer naturally
throttles production. A frame whose read has not completed in time, or whose predecessor the GUI has throttles production. A frame whose read has not completed in time, or whose predecessor the GUI has
not yet released, simply _slips_ to the next tick. not yet released, simply _slips_ to the next tick.
@ -196,7 +215,7 @@ UringVideoPlayer::maybeDisplayFrame()
} }
-------------------------------------------------------------------------- --------------------------------------------------------------------------
<1> reclaim only the slot lent to the GUI, and only once it cleared the baton <1> reclaim only the slot lent to the GUI, and only once it cleared the baton
<2> advance the single show-cursor — this is where the back-pressure lives <2> advance the single show-cursor -- this is where the back-pressure lives
<3> the frame data has arrived (`FEED`): hand it over <3> the frame data has arrived (`FEED`): hand it over
<4> the release-store publishes the buffer and makes its contents visible to the GUI <4> the release-store publishes the buffer and makes its contents visible to the GUI
<5> `READ` (IO still pending) or `SHOW` (GUI not yet caught up) ⇒ the frame slips to the next tick <5> `READ` (IO still pending) or `SHOW` (GUI not yet caught up) -> the frame slips to the next tick

28
FrOSCon26/research/Resources.txt
View file

@ -6,11 +6,27 @@ Topic: Kernel and IO
-------------------- --------------------
Asynchronous IO Background: Asynchronous IO
~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
- TODO - https://www.kegel.com/c10k.html[The C10k problem]: a very frequently quoted (and actually
quite old page (2003-2011) that analyses the problem of serving more than 10k requests from a server,
the bottlenecks and possible solution strategies.
- https://dreamsongs.com/WorseIsBetter.html[The crux with »worse is better«]. This is a summary
by Richard P. Gabriel himself, looking back, and he still has not made up his mind if _worse is better_.
The tension between the »*Berkely*« (or »New Jersy«) style and the »*MIT* style« helps to understand
design choices taken in the original design of Unix -- and with the advent of fast SSDs and finally
an interface like `io_uring` in the Linux kernel, all these unresolved questions are coming back
to haunt us.
* https://dreamsongs.com/Files/WorseIsBetterPositionPaper.pdf[Back to the Future: Is Worse (Still) Better?]
* https://dreamsongs.com/Files/ProWorseIsBetterPosition.pdf[Back to the Future: Worse (Still) is Better!]
History of io_uring
~~~~~~~~~~~~~~~~~~~
- https://web.archive.org/web/20251010072644/https://kernel.dk/io_uring.pdf[Whitepaper(2019): Efficient IO with io_uring]
- https://lwn.net/Articles/810414/[LWN.net: The rapid growth of io_uring]
- https://web.archive.org/web/20200421163135/https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/[2020: io_uring and eBPF will revolutionize programming in linux]
IO_URING IO_URING
@ -18,10 +34,13 @@ IO_URING
Introduction Introduction
~~~~~~~~~~~~ ~~~~~~~~~~~~
* https://unixism.net/loti/[»Lord of the uring«] * https://unixism.net/loti/[»Lord of the uring«]
* https://towardsdev.com/io-uring-vs-epoll-syscall-overhead-linux-performance-a3d7fcc6c9b9?gi=e429808ce081[The Linux I/O Interface That Finally Makes Sense]
* https://lwn.net/Articles/776703/[LWN.2019: Ringing in a new asynchronous I/O API] (explains the low-level API)
Reference Reference
~~~~~~~~~ ~~~~~~~~~
* https://man7.org/linux/man-pages/man7/io_uring.7.html[Manpage (low-level API)]
* https://unixism.net/loti/ref-liburing/index.html[Overview (liburing)] * https://unixism.net/loti/ref-liburing/index.html[Overview (liburing)]
** https://unixism.net/loti/ref-liburing/submission.html[Submission and Submission helpers] ** https://unixism.net/loti/ref-liburing/submission.html[Submission and Submission helpers]
@ -52,6 +71,3 @@ Reference
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_epoll_ctl[io_uring_prep_epoll_ctl] - https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_epoll_ctl[io_uring_prep_epoll_ctl]
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_add[io_uring_prep_poll_add] - https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_add[io_uring_prep_poll_add]
- https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_remove[io_uring_prep_poll_remove] - https://unixism.net/loti/ref-liburing/submission.html#c.io_uring_prep_poll_remove[io_uring_prep_poll_remove]
[purple]#maybe more resources regarding the low-level API...?#

356
FrOSCon26/research/TimeScales.txt
View file

@ -10,18 +10,23 @@ cycle up to the point where a human gives up waiting. Numbers on that scale
resist intuition, and so we tend to reason about ``fast'' and ``slow'' without resist intuition, and so we tend to reason about ``fast'' and ``slow'' without
noticing that the two words can be separated by a factor of a million. noticing that the two words can be separated by a factor of a million.
The following table collects the figures that matter in this context. It is CAUTION: Knowledge-Technology (»AI«) was used to structure and investigate this topic,
meant as a rough orientation, not as a benchmark result. starting from my own practical observations. Additional corroborating sources
were identified and findings integrated into a coherent time axis view.
NOTE: All values are order-of-magnitude figures for contemporary desktop and NOTE: All values are order-of-magnitude figures for contemporary desktop and
laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on
microarchitecture, memory configuration, kernel version and load. The microarchitecture, memory configuration, kernel version and load. The
_ratios_ are the point, not the digits. _ratios_ are the point, not the digits.
The last column rescales everything by a single constant factor, chosen so The following table collects the figures that matter in this context. It is
that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes one second meant as a rough orientation, not as a benchmark result.
of human time. This is the most effective way we know to convey what a cache
miss or a disk seek actually costs from the machine's point of view. The last column in the table rescales everything by a single constant factor,
chosen so that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes
one second of human time. Presenting this kind of _human scale proportions_
is an effective way to convey what a cache miss or a disk seek actually costs,
and what this means, assuming a _machine's point of view_.
Overview Table Overview Table
@ -38,40 +43,39 @@ Overview Table
| ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds | ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds
| ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds | ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds
| ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds | ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` footnoteref:[lkml,LKML thread "[PATCH 3/4] x86,asm: Re-work smp_store_mb()", Nov 2015. Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds reports that on Haswell "lock addq" costs pretty much exactly half of MFENCE, and explains that locked operations were the ones CPU designers had reason to optimise -- https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html] | ≈{nbsp}17 seconds | ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` <<lkml>> | ≈{nbsp}17 seconds
| ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds | ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW footnoteref:[downs,Travis Downs, "A Concurrency Cost Hierarchy", 6 July 2020, see the discussion of "vanilla" versus "non-vanilla" instructions: on x86 plain accesses and non-atomic read-modify-writes are cheap while LOCK-prefixed RMWs are an order of magnitude slower, with MFENCE comparable to a locked instruction -- https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html] | ≈{nbsp}33 seconds | ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW <<downs>> | ≈{nbsp}33 seconds
| ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds | ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row footnoteref:[cxl,Measured values in Table III of: Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototype, arXiv:2503.22017. Local DDR5 median latency 122.9 ns under Intel MLC pointer chasing versus 13.1 ns for a plain load; methodology (rdtsc, 10 000 repetitions, median) described in section III-C -- https://arxiv.org/pdf/2503.22017] | ≈{nbsp}43 seconds | ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row <<cxl>> | ≈{nbsp}43 seconds
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 footnote:[Measurement-oriented analysis of DDR5 sub-channel architecture; the relevant passage is in section "Latency Implications", noting that both DDR4 and DDR5 sit at approximately 14 ns first-access latency at their respective standard speeds. arXiv:2605.08725 -- https://arxiv.org/pdf/2605.08725] | ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 <<ddr5b>> | ≈{nbsp}47 seconds
| ≈{nbsp}47 seconds | 40...100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
| 40--100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes | ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 <<cxl>> | *≈{nbsp}5 minutes*
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 footnoteref:[cxl] | *≈{nbsp}5 minutes*
4+^s| Nanoseconds to microseconds -- entering the kernel 4+^s| Nanoseconds to microseconds -- entering the kernel
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call footnoteref:[arkanis,"Measurements of system call performance and overhead", Arkanis Development, 5 Jan 2017 -- compares a plain function call against getpid() issued via the syscall instruction and via the vDSO -- https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/] | ≈{nbsp}1.5 minutes | ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call <<arkanis>> | ≈{nbsp}1.5 minutes
| 70--100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host footnoteref:[gms,Georg Sauthoff, "On the Costs of Syscalls", 30 Aug 2021. Median of 100 repetitions across a heterogeneous set of hosts; the conclusion is that user-kernel mode switches cost on the order of a few hundred nanoseconds on all hosts, with the fastest host switching modes in under 100 ns, and that the higher figures are explained by CPU-bug mitigations being enabled by default -- https://gms.tf/on-the-costs-of-syscalls.html] | 4--6 minutes | 70...100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host <<gms>> | 4--6 minutes
| 200--600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled footnoteref:[gms] | 11--33 minutes | 200...600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled <<gms>> | 11--33 minutes
| ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour* | ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour*
| ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours | ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours
4+^s| Microseconds -- the operating system 4+^s| Microseconds -- the operating system
| 10--30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU footnoteref:[lkmlwake,Michal Schmidt, reply in the thread "nanosleep() accuracy", linux-kernel mailing list, 17 Aug 2007: the point being that a process always carries non-zero wake-up latency covering interrupt processing, waking and scheduling, and that 10-30 us is not unreasonable -- https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html] | 9{nbsp}hours -- 1{nbsp}day | 10...30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU <<lkmlwake>> | 9{nbsp}hours -- 1{nbsp}day
| ~31--57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked footnoteref:[rhel,Sample cyclictest output and interpretation in: Finding Realtime Linux Kernel Latencies, Clark Williams (Red Hat), section "Using cyclictest" -- https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt] | 1--2 days | ~31...57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked <<rhel>> | 1--2 days
| ~59--90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts footnoteref:[rhel] | 2--3.5 days | ~59...90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts <<rhel>> | 2--3.5 days
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups footnoteref:[slack,Primary documentation: `man 7 time`, section "High-resolution timers", and `man 2 prctl`, description of PR_SET_TIMERSLACK. The 50 us default for non-realtime threads is stated explicitly in: Metronome -- adaptive and precise intermittent packet retrieval in DPDK, section II (discussion of nanosleep limits), arXiv:2103.13263 -- https://arxiv.org/pdf/2103.13263 ; a practical encounter with the same default, including the /proc interface, is documented in OpenJDK issue JDK-8307766 -- https://bugs.openjdk.org/browse/JDK-8307766] | *≈{nbsp}2 days* | *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups <<slack>> | *≈{nbsp}2 days*
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` footnoteref:[slack] | ≈{nbsp}3 seconds | 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` <<slack>> | ≈{nbsp}3 seconds
| ~9{nbsp}µs | SSD read, best case | Optane / Z-NAND class at QD1 footnoteref:[anand,AnandTech, review of the Samsung 983 ZET (Z-NAND), page 3, "QD1 Random Read Performance" -- explains why queue depth 1 exposes best-case access latency, and reports an average of just under 9 us for an Optane drive against 16 us for the Z-NAND based 983 ZET -- https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3] | ≈{nbsp}8 hours | ~10{nbsp}µs | SSD read, best case | Optane-class (3D XPoint) at QD1 <<intel>> | ≈{nbsp}8 hours
| 20--70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive footnoteref:[simply,Summary of the storage latency hierarchy: a modern NVMe SSD serves a random 4K read in roughly 20-70 us, a SATA SSD in 100-200 us, and a spinning disk in 5-10 ms. simplyblock glossary entry "NVMe Latency" -- https://simplyblock.io/glossary/nvme-latency/] | 1--3 days | 20...70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive <<simply>> | 1--3 days
| 50--100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days* | 50...100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
| 100--200{nbsp}µs | SATA SSD 4K random read | the interface costs, not just the medium footnoteref:[simply] | 4--8 days | 100...200{nbsp}µs| SATA SSD 4K random read | the interface costs, not just the medium <<simply>> | 4--8 days
| 100{nbsp}µs -- 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks | 100{nbsp}µs ... 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
4+^s| Milliseconds -- where machine and human meet 4+^s| Milliseconds -- where machine and human meet
| 5--10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades footnoteref:[simply] | *6{nbsp}months -- 1{nbsp}year* | 5...10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades <<simply>> | *6{nbsp}months -- 1{nbsp}year*
| ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year | ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year
| 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years | 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years
| 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years* | 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years*
@ -112,24 +116,24 @@ become visible to another core at the moment the instruction retires: it first
lands in that core's store buffer. Cache coherence guarantees that the cores lands in that core's store buffer. Cache coherence guarantees that the cores
eventually agree on the content of a cache line, but it says nothing about eventually agree on the content of a cache line, but it says nothing about
_when_, nor about the order in which several independent writes become visible. _when_, nor about the order in which several independent writes become visible.
This is what memory fences are for -- they constrain the order, and thereby This is what memory fences (memory barriers) are for -- they constrain the order,
allow one core to reason about what an adjacent core has already done. The cost and thereby allow one core to reason about what an adjacent core has already done.
shows up in the table twice: an uncontended fence is roughly the price of an The cost shows up in the table twice: an uncontended fence is roughly the price of
L3 hit, whereas a cache line actually bouncing between two cores costs about as an L3 hit, whereas a cache line actually bouncing between two cores costs about as
much as going to DRAM. It is worth internalising that the expensive part is much as going to DRAM. It should be noted that the expensive part is usually _not_
usually not the synchronisation primitive itself, but the coherence traffic it the synchronisation primitive itself, but the cache coherence traffic it provokes.
provokes.
x86 implements a comparatively strong model, _Total Store Order_ (TSO): of the And as far as memory synchronisation is concerned, there is another twist that
four possible reorderings, only StoreLoad is permitted. Acquire and release has already led quite some programmers astray. Intel's x86 implements a comparatively
semantics are therefore free on x86 -- they compile to plain `MOV` -- and only strong coherency model, _Total Store Order_ (TSO): of the four possible reorderings,
sequential consistency requires an actual instruction. ARM and POWER are only StoreLoad is permitted. Acquire and release semantics are therefore free on x86 --
weakly ordered and consequently offer a more differentiated set of barriers: they compile to plain `MOV` -- and only sequential consistency requires an actual
ARM distinguishes `DMB` from the heavier `DSB`, and provides graded load instruction. Quite to the contrary, ARM and POWER are weakly ordered and consequently
acquire flavours such as `LDAR` and `LDAPR`; POWER separates the heavyweight offer a more differentiated set of barriers: ARM distinguishes `DMB` from the heavier
`sync` from the cheaper `lwsync`. The trap for the unwary is that C++ `DSB`, and provides graded load acquire flavours such as `LDAR` and `LDAPR`; POWER
`memory_order` distinctions which cost nothing on x86 have a real price on separates the heavyweight `sync` from the cheaper `lwsync`. The trap for the unwary
ARM -- and, worse, that under-synchronised code can appear perfectly correct is that C++ `memory_order` distinctions which cost nothing on x86 have a real price
on ARM -- and, worse, that under-synchronised code can appear perfectly correct
on x86 and fail on ARM. on x86 and fail on ARM.
@ -145,17 +149,19 @@ attendant TLB consequences) and one or more branch-prediction barriers --
retpolines, IBPB or IBRS depending on hardware and configuration. On return, retpolines, IBPB or IBRS depending on hardware and configuration. On return,
the whole dance runs in reverse. the whole dance runs in reverse.
The direct cost is measurable: a minimal syscall such as `getpid()` amounts to The direct cost is measurable: a minimal syscall such as `getpid()` amounts
little more than two mode switches, and lands in the region of a hundred to little more than two mode switches, and lands in the region of a hundred
nanoseconds on a fast host without mitigations, rising to several hundred once nanoseconds on a fast host without mitigations, rising to several hundred once
mitigations are enabled.footnoteref:[gms] The indirect cost is harder to see mitigations are enabled <<gms>>. The indirect cost is harder to see
and frequently larger. Kernel entry code and the syscall handler evict and frequently larger. Kernel entry code and the syscall handler evict
user-space data from L1 and L2, and disturb the branch predictors; the user-space data from L1 and L2, and disturb the branch predictors; the
application resumes with a cold working set and pays for it in misses that no application resumes with a cold working set and pays for it in misses that no
syscall benchmark will attribute to the syscall. This is precisely why the syscall benchmark will attribute to the syscall. A simple `read()` that happens
interesting question is rarely ``how expensive is one syscall'' but rather to hit data already in the kernel cache can incur costs of around 1{nbsp}µs.
``how many syscalls does this design require per unit of work'' -- which is
the question `io_uring` was built to answer. The interesting question thus is is rarely ``how expensive is one syscall'' but
rather ``how many syscalls does this design require per unit of work'' --
which is the question `io_uring` was built to answer.
Operating System Slack Operating System Slack
@ -165,100 +171,244 @@ A thread that asks to sleep for a specific duration will not, in general, wake
at that instant. Part of this is unavoidable: an interrupt must be handled, the at that instant. Part of this is unavoidable: an interrupt must be handled, the
sleeping task must be moved to the run queue, and the scheduler must actually sleeping task must be moved to the run queue, and the scheduler must actually
place it on a CPU, which together account for something in the range of place it on a CPU, which together account for something in the range of
10--30{nbsp}µs.footnoteref:[lkmlwake] 10--30{nbsp}µs <<lkmlwake>>.
The larger part, however, is deliberate. Linux maintains a per-thread _timer The larger part, however, is deliberate. Linux maintains a per-thread _timer
slack_ value, and the kernel is free to delay a timed wake-up by up to that slack_ value, and the kernel is free to delay a timed wake-up by up to that
amount in order to coalesce several wake-ups into one and thereby let the CPU amount in order to coalesce several wake-ups into one and thereby let the CPU
stay in a low-power state longer. For every thread not running under a stay in a low-power state longer. For every thread not running under a
real-time scheduling policy, the default is 50{nbsp}µs.footnoteref:[slack] real-time scheduling policy, the default is 50{nbsp}µs <<slack>>.
This is not imprecision; it is a power-versus-precision trade that the kernel This is not imprecision; it is a power-versus-precision trade that the kernel
makes on the application's behalf, and it can be adjusted -- down to makes on the application's behalf, and it can be adjusted -- down to
1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process 1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process
`/proc/PID/timerslack_ns` interface. `/proc/PID/timerslack_ns` interface.
The distinction worth carrying away is between _resolution_ and _accuracy_. The key distinction is therefore that between _resolution_ and _accuracy_.
Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond
resolution, and `clock_getres()` will happily confirm it. The accuracy with resolution, and `clock_getres()` will happily confirm it. The accuracy with
which a `nanosleep()` actually returns is a different quantity altogether, two which a `nanosleep()` actually returns is a different story altogether, two
to three orders of magnitude coarser, and dominated by slack, wake-up latency to three orders of magnitude coarser, and dominated by slack, wake-up latency
and -- if the core had gone idle -- C-state exit time. Any design that assumes and -- if the core had gone idle -- C-state exit time. Any design that assumes
it can steer a worker thread with microsecond-scale sleeps should be checked it can steer a worker thread with microsecond-scale sleeps should be checked
against these figures first. against these figures first.
// ----------------------------------------------------------------------
// Footnote definitions are attached at first use above via footnoteref.
// Repeated references reuse the id; see AsciiDoc footnoteref macro.
// ----------------------------------------------------------------------
Sources Sources
------- -------
The footnotes above carry the precise locations. Collected here as a reading The in-text markers such as <<downs>> link down to the full citation here.
list, roughly in order of usefulness for further work. Collected as a reading list, roughly in order of usefulness for further work.
- Travis Downs, *A Concurrency Cost Hierarchy* -- [[downs]]
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html + --
downs::
Travis Downs (6 July 2020) --
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html[
A Concurrency Cost Hierarchy
]
//Archive https://web.archive.org/web/20260801071948/https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html
+
The best single entry point on why synchronisation costs what it costs. The best single entry point on why synchronisation costs what it costs.
+
See the discussion of "vanilla" versus "non-vanilla" instructions: on x86
plain accesses and non-atomic read-modify-writes are cheap while
LOCK-prefixed RMWs are an order of magnitude slower, with `MFENCE`
comparable to a locked instruction.
--
- LKML thread, *[PATCH 3/4] x86,asm: Re-work smp_store_mb()* (Nov 2015) -- [[lkml]]
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html + --
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`. lkml::
LKML thread (Nov 2015) --
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html[
&#91;PATCH 3/4&#93; x86,asm: Re-work smp_store_mb()
]
//Archive https://web.archive.org/web/20240818210644/https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html
+
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`:
Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds
reports that on Haswell `lock addq` costs pretty much exactly half of
MFENCE, and explains that locked operations were the ones CPU designers had
reason to optimise.
--
- Georg Sauthoff, *On the Costs of Syscalls* (Aug 2021) -- [[gms]]
https://gms.tf/on-the-costs-of-syscalls.html + --
gms::
Georg Sauthoff (30 Aug 2021) --
https://gms.tf/on-the-costs-of-syscalls.html[
On the Costs of Syscalls
]
//Archive https://web.archive.org/web/20260725162647/https://gms.tf/on-the-costs-of-syscalls.html
+
Careful cross-host microbenchmark of the mode-switch cost. Careful cross-host microbenchmark of the mode-switch cost.
+
Median of 100 repetitions across a heterogeneous set of hosts; the
conclusion is that user-kernel mode switches cost on the order of a few
hundred nanoseconds on all hosts, with the fastest host switching modes in
under 100 ns, and that the higher figures are explained by CPU-bug
mitigations being enabled by default.
--
- *Measurements of system call performance and overhead* (Jan 2017) -- [[arkanis]]
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ + --
Function call versus `syscall` instruction versus vDSO. arkanis::
Arkanis Development (5 Jan 2017) --
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/[
Measurements of system call performance and overhead
]
//Archive https://web.archive.org/web/20260311003705/https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/
+
Compares a plain function call against `getpid()` issued via the syscall
instruction and via the vDSO.
--
- Brendan Gregg, *KPTI/KAISER Meltdown Initial Performance Regressions* -- [[gregg]]
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html + --
gregg::
Brendan Gregg (9 Feb 2018) --
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html[
KPTI/KAISER Meltdown Initial Performance Regressions
]
//Archive https://web.archive.org/web/20260603211334/https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html
+
Relates mitigation overhead to syscall rate, context switch rate and working Relates mitigation overhead to syscall rate, context switch rate and working
set size -- the systemic view rather than the per-call view. set size -- the systemic view rather than the per-call view.
--
- Clark Williams (Red Hat), *Finding Realtime Linux Kernel Latencies* -- [[lkmlwake]]
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt + --
Walks through a `cyclictest` run and how to read its output. lkmlwake::
Michal Schmidt (17 Aug 2007) --
https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html[
Reply in the LKML thread "nanosleep() accuracy"
]
//Archive https://web.archive.org/web/20260824155617/https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html
+
The point being that a process always carries non-zero wake-up latency
covering interrupt processing, waking and scheduling, and that
10--30{nbsp}µs is not unreasonable.
--
- `man 7 time` and `man 2 prctl` (PR_SET_TIMERSLACK) + [[rhel]]
Primary documentation for timer slack. The 50{nbsp}µs default is stated in --
https://arxiv.org/pdf/2103.13263 (Metronome, DPDK packet retrieval) and rhel::
encountered in practice in https://bugs.openjdk.org/browse/JDK-8307766 Clark Williams (Red Hat) --
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt[
Finding Realtime Linux Kernel Latencies
]
//Archive https://web.archive.org/web/20250620155700/https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt
+
Walks through a `cyclictest` run and how to read its output: sample
`cyclictest` output and interpretation, section "Using cyclictest".
--
- *Performance Characterizations ... Samsung CXL Memory Module Hybrid [[slack]]
Prototype*, arXiv:2503.22017 -- https://arxiv.org/pdf/2503.22017 + --
Table III contains a well-documented measurement of plain local DDR5. slack::
Primary documentation --
`man 7 time`, section "High-resolution timers", and `man 2 prctl`,
description of `PR_SET_TIMERSLACK`
+
The 50{nbsp}µs default for non-realtime threads is stated explicitly in
https://arxiv.org/pdf/2103.13263[
Metronome -- adaptive and precise intermittent packet retrieval in DPDK
]
//Archive https://web.archive.org/web/20260725162634/https://arxiv.org/pdf/2103.13263
(section II, discussion of nanosleep limits), and a practical encounter
with the same default, including the `/proc` interface, is documented in
https://bugs.openjdk.org/browse/JDK-8307766[
OpenJDK issue JDK-8307766
].
//Archive https://web.archive.org/web/20251130011408/https://bugs.openjdk.org/browse/JDK-8307766
--
- *Single 32-bit Sub-Channel DDR5 DIMMs*, arXiv:2605.08725 -- [[cxl]]
https://arxiv.org/pdf/2605.08725 + --
cxl::
arXiv:2503.22017 --
https://arxiv.org/pdf/2503.22017[
Performance Characterizations and Usage Guidelines of Samsung CXL Memory
Module Hybrid Prototype
]
//Archive https://web.archive.org/web/20260811083139/https://arxiv.org/pdf/2503.22017
+
Table III contains a well-documented measurement of plain local DDR5:
median latency 122.9{nbsp}ns under Intel MLC pointer chasing versus
13.1{nbsp}ns for a plain load; methodology (rdtsc, 10 000 repetitions,
median) described in section III-C.
--
[[ddr5b]]
--
ddr5b::
arXiv:2605.08725 --
https://arxiv.org/pdf/2605.08725[
Single 32-bit Sub-Channel DDR5 DIMMs
]
//Archive https://web.archive.org/web/20260607161507/https://arxiv.org/pdf/2605.08725
+
Useful for the point that DDR5 did not improve absolute latency over DDR4. Useful for the point that DDR5 did not improve absolute latency over DDR4.
+
The relevant passage is in section "Latency Implications", noting that both
DDR4 and DDR5 sit at approximately 14{nbsp}ns first-access latency at their
respective standard speeds.
--
- simplyblock glossary, *NVMe Latency* -- [[simply]]
https://simplyblock.io/glossary/nvme-latency/ + --
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy. simply::
simplyblock glossary --
https://simplyblock.io/glossary/nvme-latency/[
NVMe Latency
]
//Archive https://web.archive.org/web/20260418070043/https://simplyblock.io/glossary/nvme-latency/
+
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy: a modern
NVMe SSD serves a random 4K read in roughly 20--70{nbsp}µs, a SATA SSD in
100--200{nbsp}µs, and a spinning disk in 5--10{nbsp}ms.
--
- AnandTech, *Samsung 983 ZET (Z-NAND) review*, page 3 -- [[intel]]
https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3 + --
Explains why queue depth 1 exposes best-case access latency. intel::
Intel --
https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf[
Optane SSD DC P4800X Evaluation Guide
]
//Archive https://web.archive.org/web/20260824160928/https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf
+
In Chapter 1.1, Page 5: ``at queue depth of one, 99 out of 100 read operations ...
complete in single digit microseconds.''
+
Corroborated independently by
https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html[
Tom's Hardware's 2017 preview,
]
//Archive https://web.archive.org/web/20260313082242/https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html
which reports 10 µs latency for the same drive.
+
Note: Optane was discontinued by Intel in 2022; the figure is retained as the
best-documented physical floor for this latency class.
--
A Note on Microbenchmarks A Note on Microbenchmarks
------------------------- -------------------------
One caveat deserves to be stated separately, because it explains why published One caveat deserves a separate mention, as it explains why the published
figures for synchronisation primitives vary so wildly. A hot microbenchmark figures on synchronisation primitives differ so drastically from one another.
keeps every thread runnable and every core out of idle by construction. Under A hot microbenchmark keeps every thread runnable and every core out of idle by
those conditions an uncontended atomic operation or a mutex acquisition costs its very construction. Under those conditions an uncontended atomic operation or
single-digit to low-three-digit nanoseconds, and a benchmark will faithfully a Mutex acquisition costs single-digit to low-three-digit nanoseconds, and a
report that. What such a benchmark structurally cannot observe is the cost of benchmark will faithfully report that. What such a benchmark _structurally_
_blocking_ -- of a thread actually parking and later being woken -- because cannot observe is the cost of _blocking_ -- of a thread actually parking and
pressing the benchmark is precisely what prevents that from happening. later being woken -- because pressing the benchmark is precisely what
prevents that from happening.
The instruction-level cost of a synchronisation primitive is what a hot The instruction-level cost of a synchronisation primitive is what a hot
microbenchmark measures. Its systemic cost -- coherence traffic under microbenchmark measures. Its systemic cost -- coherence traffic under
contention, and blocking and wake-up latency when it is not contended -- is contention, and blocking and wake-up latency when it is not contended --
what production tail latency measures. The two can differ by two to four orders is what production tail latency measures. The two values can differ by
of magnitude, and the table above spans both regimes for exactly that reason. two to four orders of magnitude, and it is for this very reason that
the table above covers both regimes.