Ichthyostega
f46f117eda
Content generated by Knowledge-Technology - collect information with the help of Claude - validate my own measurements with sources from the net - find a suitable form for the presentation - instructed Claude to build a SVG with a logarithmic time scale - instructed Claude to summarise the findings as Asciidoc table
264 lines
19 KiB
Text
264 lines
19 KiB
Text
Time Scales
|
|
===========
|
|
:Author: Lumiera Project
|
|
:toc:
|
|
|
|
Anyone reasoning about I/O, threading or real-time media processing eventually
|
|
needs a feeling for _how long things take_. The trouble is that the relevant
|
|
durations span more than twelve orders of magnitude -- from a single CPU clock
|
|
cycle up to the point where a human gives up waiting. Numbers on that scale
|
|
resist intuition, and so we tend to reason about ``fast'' and ``slow'' without
|
|
noticing that the two words can be separated by a factor of a million.
|
|
|
|
The following table collects the figures that matter in this context. It is
|
|
meant as a rough orientation, not as a benchmark result.
|
|
|
|
NOTE: All values are order-of-magnitude figures for contemporary desktop and
|
|
laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on
|
|
microarchitecture, memory configuration, kernel version and load. The
|
|
_ratios_ are the point, not the digits.
|
|
|
|
The last column rescales everything by a single constant factor, chosen so
|
|
that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes one second
|
|
of human time. This is the most effective way we know to convey what a cache
|
|
miss or a disk seek actually costs from the machine's point of view.
|
|
|
|
|
|
Overview Table
|
|
--------------
|
|
|
|
[width="100%",cols="12%,26%,40%,22%",options="header"]
|
|
|=========================================================================
|
|
| Figure | Item | Remarks | Felt as (1{nbsp}cycle{nbsp}≙{nbsp}1{nbsp}s)
|
|
|
|
4+^s| Nanoseconds -- inside the CPU
|
|
|
|
| 0.3{nbsp}ns | one clock cycle | at 3.3{nbsp}GHz; the machine's own time quantum | *1 second*
|
|
| ~1{nbsp}ns | L1 cache hit | about 4 cycles, fully pipelined | ≈{nbsp}3 seconds
|
|
| ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds
|
|
| ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds
|
|
| ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds
|
|
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` footnoteref:[lkml,LKML thread "[PATCH 3/4] x86,asm: Re-work smp_store_mb()", Nov 2015. Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds reports that on Haswell "lock addq" costs pretty much exactly half of MFENCE, and explains that locked operations were the ones CPU designers had reason to optimise -- https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html] | ≈{nbsp}17 seconds
|
|
| ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds
|
|
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW footnoteref:[downs,Travis Downs, "A Concurrency Cost Hierarchy", 6 July 2020, see the discussion of "vanilla" versus "non-vanilla" instructions: on x86 plain accesses and non-atomic read-modify-writes are cheap while LOCK-prefixed RMWs are an order of magnitude slower, with MFENCE comparable to a locked instruction -- https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html] | ≈{nbsp}33 seconds
|
|
| ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds
|
|
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row footnoteref:[cxl,Measured values in Table III of: Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototype, arXiv:2503.22017. Local DDR5 median latency 122.9 ns under Intel MLC pointer chasing versus 13.1 ns for a plain load; methodology (rdtsc, 10 000 repetitions, median) described in section III-C -- https://arxiv.org/pdf/2503.22017] | ≈{nbsp}43 seconds
|
|
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 footnote:[Measurement-oriented analysis of DDR5 sub-channel architecture; the relevant passage is in section "Latency Implications", noting that both DDR4 and DDR5 sit at approximately 14 ns first-access latency at their respective standard speeds. arXiv:2605.08725 -- https://arxiv.org/pdf/2605.08725]
|
|
| ≈{nbsp}47 seconds
|
|
| 40--100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
|
|
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 footnoteref:[cxl] | *≈{nbsp}5 minutes*
|
|
|
|
4+^s| Nanoseconds to microseconds -- entering the kernel
|
|
|
|
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call footnoteref:[arkanis,"Measurements of system call performance and overhead", Arkanis Development, 5 Jan 2017 -- compares a plain function call against getpid() issued via the syscall instruction and via the vDSO -- https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/] | ≈{nbsp}1.5 minutes
|
|
| 70--100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host footnoteref:[gms,Georg Sauthoff, "On the Costs of Syscalls", 30 Aug 2021. Median of 100 repetitions across a heterogeneous set of hosts; the conclusion is that user-kernel mode switches cost on the order of a few hundred nanoseconds on all hosts, with the fastest host switching modes in under 100 ns, and that the higher figures are explained by CPU-bug mitigations being enabled by default -- https://gms.tf/on-the-costs-of-syscalls.html] | 4--6 minutes
|
|
| 200--600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled footnoteref:[gms] | 11--33 minutes
|
|
| ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour*
|
|
| ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours
|
|
|
|
4+^s| Microseconds -- the operating system
|
|
|
|
| 10--30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU footnoteref:[lkmlwake,Michal Schmidt, reply in the thread "nanosleep() accuracy", linux-kernel mailing list, 17 Aug 2007: the point being that a process always carries non-zero wake-up latency covering interrupt processing, waking and scheduling, and that 10-30 us is not unreasonable -- https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html] | 9{nbsp}hours -- 1{nbsp}day
|
|
| ~31--57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked footnoteref:[rhel,Sample cyclictest output and interpretation in: Finding Realtime Linux Kernel Latencies, Clark Williams (Red Hat), section "Using cyclictest" -- https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt] | 1--2 days
|
|
| ~59--90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts footnoteref:[rhel] | 2--3.5 days
|
|
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups footnoteref:[slack,Primary documentation: `man 7 time`, section "High-resolution timers", and `man 2 prctl`, description of PR_SET_TIMERSLACK. The 50 us default for non-realtime threads is stated explicitly in: Metronome -- adaptive and precise intermittent packet retrieval in DPDK, section II (discussion of nanosleep limits), arXiv:2103.13263 -- https://arxiv.org/pdf/2103.13263 ; a practical encounter with the same default, including the /proc interface, is documented in OpenJDK issue JDK-8307766 -- https://bugs.openjdk.org/browse/JDK-8307766] | *≈{nbsp}2 days*
|
|
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` footnoteref:[slack] | ≈{nbsp}3 seconds
|
|
| ~9{nbsp}µs | SSD read, best case | Optane / Z-NAND class at QD1 footnoteref:[anand,AnandTech, review of the Samsung 983 ZET (Z-NAND), page 3, "QD1 Random Read Performance" -- explains why queue depth 1 exposes best-case access latency, and reports an average of just under 9 us for an Optane drive against 16 us for the Z-NAND based 983 ZET -- https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3] | ≈{nbsp}8 hours
|
|
| 20--70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive footnoteref:[simply,Summary of the storage latency hierarchy: a modern NVMe SSD serves a random 4K read in roughly 20-70 us, a SATA SSD in 100-200 us, and a spinning disk in 5-10 ms. simplyblock glossary entry "NVMe Latency" -- https://simplyblock.io/glossary/nvme-latency/] | 1--3 days
|
|
| 50--100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
|
|
| 100--200{nbsp}µs | SATA SSD 4K random read | the interface costs, not just the medium footnoteref:[simply] | 4--8 days
|
|
| 100{nbsp}µs -- 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
|
|
|
|
4+^s| Milliseconds -- where machine and human meet
|
|
|
|
| 5--10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades footnoteref:[simply] | *6{nbsp}months -- 1{nbsp}year*
|
|
| ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year
|
|
| 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years
|
|
| 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years*
|
|
| ~100{nbsp}ms | UI starts to feel sluggish | | ≈{nbsp}11 years
|
|
|
|
4+^s| Seconds and beyond -- the human
|
|
|
|
| ~1{nbsp}s | ``the software takes time to respond'' | acceptable, but noticed | *≈{nbsp}1 century*
|
|
| ~3{nbsp}min | ``time for another cup of coffee'' | | ≈{nbsp}19 000 years
|
|
| ~15{nbsp}min | ``can't wait, I'll do something else'' | ≙{nbsp}1000{nbsp}s, which conveniently closes the fourth decade block | ≈{nbsp}95 000 years
|
|
|=========================================================================
|
|
|
|
|
|
The Memory Wall
|
|
---------------
|
|
|
|
For roughly two decades, CPU clock rates have hovered around 3{nbsp}GHz while
|
|
DRAM access latency has stayed near 60--100{nbsp}ns. The consequence is
|
|
arithmetic: a cache miss that cost a handful of cycles on early-1990s hardware
|
|
now costs several hundred. The processor did not merely get faster than
|
|
memory -- it got faster than memory _by a growing margin_, and the entire
|
|
cache hierarchy exists to paper over that gap. Note in the table above that
|
|
the raw device-level latency of DDR5 is barely distinguishable from DDR4;
|
|
the generational improvements went into bandwidth, not into latency.
|
|
|
|
Once several cores share one memory subsystem, a single uniform path to memory
|
|
becomes the bottleneck. The answer was to move the memory controller onto the
|
|
CPU die and attach memory per socket -- and, with chiplet designs, effectively
|
|
per die group. This is what NUMA (non-uniform memory access) describes: memory
|
|
that ``belongs'' to the core you are running on is reached quickly, while
|
|
memory belonging to a neighbour costs noticeably more. The practical upshot is
|
|
that thread placement and memory placement are no longer independent concerns,
|
|
and that a scheduler which migrates a worker between cores may silently convert
|
|
local accesses into remote ones.
|
|
|
|
Cores also keep private caches, and a store performed on one core does not
|
|
become visible to another core at the moment the instruction retires: it first
|
|
lands in that core's store buffer. Cache coherence guarantees that the cores
|
|
eventually agree on the content of a cache line, but it says nothing about
|
|
_when_, nor about the order in which several independent writes become visible.
|
|
This is what memory fences are for -- they constrain the order, and thereby
|
|
allow one core to reason about what an adjacent core has already done. The cost
|
|
shows up in the table twice: an uncontended fence is roughly the price of an
|
|
L3 hit, whereas a cache line actually bouncing between two cores costs about as
|
|
much as going to DRAM. It is worth internalising that the expensive part is
|
|
usually not the synchronisation primitive itself, but the coherence traffic it
|
|
provokes.
|
|
|
|
x86 implements a comparatively strong model, _Total Store Order_ (TSO): of the
|
|
four possible reorderings, only StoreLoad is permitted. Acquire and release
|
|
semantics are therefore free on x86 -- they compile to plain `MOV` -- and only
|
|
sequential consistency requires an actual instruction. ARM and POWER are
|
|
weakly ordered and consequently offer a more differentiated set of barriers:
|
|
ARM distinguishes `DMB` from the heavier `DSB`, and provides graded load
|
|
acquire flavours such as `LDAR` and `LDAPR`; POWER separates the heavyweight
|
|
`sync` from the cheaper `lwsync`. The trap for the unwary is that C++
|
|
`memory_order` distinctions which cost nothing on x86 have a real price on
|
|
ARM -- and, worse, that under-synchronised code can appear perfectly correct
|
|
on x86 and fail on ARM.
|
|
|
|
|
|
Crossing into the Kernel
|
|
------------------------
|
|
|
|
A system call is not a function call: it is a controlled trap. The `syscall`
|
|
instruction switches the processor to privilege level 0, and the kernel entry
|
|
path then swaps to the kernel `GS` base, switches stacks, and saves the user
|
|
register state. Since the Meltdown/Spectre family of vulnerabilities, that
|
|
sequence also includes a page-table switch (KPTI writes `CR3`, with the
|
|
attendant TLB consequences) and one or more branch-prediction barriers --
|
|
retpolines, IBPB or IBRS depending on hardware and configuration. On return,
|
|
the whole dance runs in reverse.
|
|
|
|
The direct cost is measurable: a minimal syscall such as `getpid()` amounts to
|
|
little more than two mode switches, and lands in the region of a hundred
|
|
nanoseconds on a fast host without mitigations, rising to several hundred once
|
|
mitigations are enabled.footnoteref:[gms] The indirect cost is harder to see
|
|
and frequently larger. Kernel entry code and the syscall handler evict
|
|
user-space data from L1 and L2, and disturb the branch predictors; the
|
|
application resumes with a cold working set and pays for it in misses that no
|
|
syscall benchmark will attribute to the syscall. This is precisely why the
|
|
interesting question is rarely ``how expensive is one syscall'' but rather
|
|
``how many syscalls does this design require per unit of work'' -- which is
|
|
the question `io_uring` was built to answer.
|
|
|
|
|
|
Operating System Slack
|
|
----------------------
|
|
|
|
A thread that asks to sleep for a specific duration will not, in general, wake
|
|
at that instant. Part of this is unavoidable: an interrupt must be handled, the
|
|
sleeping task must be moved to the run queue, and the scheduler must actually
|
|
place it on a CPU, which together account for something in the range of
|
|
10--30{nbsp}µs.footnoteref:[lkmlwake]
|
|
|
|
The larger part, however, is deliberate. Linux maintains a per-thread _timer
|
|
slack_ value, and the kernel is free to delay a timed wake-up by up to that
|
|
amount in order to coalesce several wake-ups into one and thereby let the CPU
|
|
stay in a low-power state longer. For every thread not running under a
|
|
real-time scheduling policy, the default is 50{nbsp}µs.footnoteref:[slack]
|
|
This is not imprecision; it is a power-versus-precision trade that the kernel
|
|
makes on the application's behalf, and it can be adjusted -- down to
|
|
1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process
|
|
`/proc/PID/timerslack_ns` interface.
|
|
|
|
The distinction worth carrying away is between _resolution_ and _accuracy_.
|
|
Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond
|
|
resolution, and `clock_getres()` will happily confirm it. The accuracy with
|
|
which a `nanosleep()` actually returns is a different quantity altogether, two
|
|
to three orders of magnitude coarser, and dominated by slack, wake-up latency
|
|
and -- if the core had gone idle -- C-state exit time. Any design that assumes
|
|
it can steer a worker thread with microsecond-scale sleeps should be checked
|
|
against these figures first.
|
|
|
|
|
|
// ----------------------------------------------------------------------
|
|
// Footnote definitions are attached at first use above via footnoteref.
|
|
// Repeated references reuse the id; see AsciiDoc footnoteref macro.
|
|
// ----------------------------------------------------------------------
|
|
|
|
Sources
|
|
-------
|
|
|
|
The footnotes above carry the precise locations. Collected here as a reading
|
|
list, roughly in order of usefulness for further work.
|
|
|
|
- Travis Downs, *A Concurrency Cost Hierarchy* --
|
|
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html +
|
|
The best single entry point on why synchronisation costs what it costs.
|
|
|
|
- LKML thread, *[PATCH 3/4] x86,asm: Re-work smp_store_mb()* (Nov 2015) --
|
|
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html +
|
|
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`.
|
|
|
|
- Georg Sauthoff, *On the Costs of Syscalls* (Aug 2021) --
|
|
https://gms.tf/on-the-costs-of-syscalls.html +
|
|
Careful cross-host microbenchmark of the mode-switch cost.
|
|
|
|
- *Measurements of system call performance and overhead* (Jan 2017) --
|
|
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ +
|
|
Function call versus `syscall` instruction versus vDSO.
|
|
|
|
- Brendan Gregg, *KPTI/KAISER Meltdown Initial Performance Regressions* --
|
|
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html +
|
|
Relates mitigation overhead to syscall rate, context switch rate and working
|
|
set size -- the systemic view rather than the per-call view.
|
|
|
|
- Clark Williams (Red Hat), *Finding Realtime Linux Kernel Latencies* --
|
|
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt +
|
|
Walks through a `cyclictest` run and how to read its output.
|
|
|
|
- `man 7 time` and `man 2 prctl` (PR_SET_TIMERSLACK) +
|
|
Primary documentation for timer slack. The 50{nbsp}µs default is stated in
|
|
https://arxiv.org/pdf/2103.13263 (Metronome, DPDK packet retrieval) and
|
|
encountered in practice in https://bugs.openjdk.org/browse/JDK-8307766
|
|
|
|
- *Performance Characterizations ... Samsung CXL Memory Module Hybrid
|
|
Prototype*, arXiv:2503.22017 -- https://arxiv.org/pdf/2503.22017 +
|
|
Table III contains a well-documented measurement of plain local DDR5.
|
|
|
|
- *Single 32-bit Sub-Channel DDR5 DIMMs*, arXiv:2605.08725 --
|
|
https://arxiv.org/pdf/2605.08725 +
|
|
Useful for the point that DDR5 did not improve absolute latency over DDR4.
|
|
|
|
- simplyblock glossary, *NVMe Latency* --
|
|
https://simplyblock.io/glossary/nvme-latency/ +
|
|
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy.
|
|
|
|
- AnandTech, *Samsung 983 ZET (Z-NAND) review*, page 3 --
|
|
https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3 +
|
|
Explains why queue depth 1 exposes best-case access latency.
|
|
|
|
|
|
A Note on Microbenchmarks
|
|
-------------------------
|
|
|
|
One caveat deserves to be stated separately, because it explains why published
|
|
figures for synchronisation primitives vary so wildly. A hot microbenchmark
|
|
keeps every thread runnable and every core out of idle by construction. Under
|
|
those conditions an uncontended atomic operation or a mutex acquisition costs
|
|
single-digit to low-three-digit nanoseconds, and a benchmark will faithfully
|
|
report that. What such a benchmark structurally cannot observe is the cost of
|
|
_blocking_ -- of a thread actually parking and later being woken -- because
|
|
pressing the benchmark is precisely what prevents that from happening.
|
|
|
|
The instruction-level cost of a synchronisation primitive is what a hot
|
|
microbenchmark measures. Its systemic cost -- coherence traffic under
|
|
contention, and blocking and wake-up latency when it is not contended -- is
|
|
what production tail latency measures. The two can differ by two to four orders
|
|
of magnitude, and the table above spans both regimes for exactly that reason.
|