wegen/FrOSCon26/research/TimeScales.txt

Time Scales
===========
:Author: Lumiera Project
:toc:

Anyone reasoning about I/O, threading or real-time media processing eventually
needs a feeling for _how long things take_. The trouble is that the relevant
durations span more than twelve orders of magnitude -- from a single CPU clock
cycle up to the point where a human gives up waiting. Numbers on that scale
resist intuition, and so we tend to reason about ``fast'' and ``slow'' without
noticing that the two words can be separated by a factor of a million.

CAUTION: Knowledge-Technology (»AI«) was used to structure and investigate this topic,
         starting from my own practical observations. Additional corroborating sources
         were identified and findings integrated into a coherent time axis view.

NOTE: All values are order-of-magnitude figures for contemporary desktop and
laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on
microarchitecture, memory configuration, kernel version and load. The
_ratios_ are the point, not the digits.

The following table collects the figures that matter in this context. It is
meant as a rough orientation, not as a benchmark result.

The last column in the table rescales everything by a single constant factor,
chosen so that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes
one second of human time. Presenting this kind of _human scale proportions_
is an effective way to convey what a cache miss or a disk seek actually costs,
and what this means, assuming a _machine's point of view_.


Overview Table
--------------

[width="100%",cols="12%,26%,40%,22%",options="header"]
|=========================================================================
| Figure | Item | Remarks | Felt as (1{nbsp}cycle{nbsp}≙{nbsp}1{nbsp}s)

4+^s| Nanoseconds -- inside the CPU

| 0.3{nbsp}ns  | one clock cycle          | at 3.3{nbsp}GHz; the machine's own time quantum                           | *1 second*
| ~1{nbsp}ns   | L1 cache hit             | about 4 cycles, fully pipelined                                           | ≈{nbsp}3 seconds
| ~1.5{nbsp}ns | `sfence`                 | store fence, ≈{nbsp}5 cycles; rarely needed on x86                        | ≈{nbsp}5 seconds
| ~2.7{nbsp}ns | `lfence`                 | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds
| ~4{nbsp}ns   | L2 cache hit             | ≈{nbsp}12--15 cycles                                                      | ≈{nbsp}13 seconds
| ~5{nbsp}ns   | `lock add $0,(%rsp)`     | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` <<lkml>> | ≈{nbsp}17 seconds
| ~5{nbsp}ns   | branch misprediction     | pipeline refill                                                           | ≈{nbsp}17 seconds
| ~10{nbsp}ns  | `mfence` / full barrier  | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW <<downs>>       | ≈{nbsp}33 seconds
| ~15{nbsp}ns  | L3 cache hit             | shared, hence variable                                                    | ≈{nbsp}50 seconds
| ~13{nbsp}ns  | DRAM, open row           | a plain load that happens to hit an already-activated row <<cxl>>         | ≈{nbsp}43 seconds
| ~14{nbsp}ns  | DRAM, first access       | device-level figure; essentially unchanged from DDR4 to DDR5 <<ddr5b>>    | ≈{nbsp}47 seconds
| 40...100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
| ~100{nbsp}ns | DRAM random access       | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 <<cxl>> | *≈{nbsp}5 minutes*

4+^s| Nanoseconds to microseconds -- entering the kernel

| ~25{nbsp}ns  | syscall via vDSO         | not a real trap; almost as cheap as a function call <<arkanis>> | ≈{nbsp}1.5 minutes
| 70...100{nbsp}ns  | syscall trap, bare  | `getpid()` on a mitigation-free host <<gms>>                    | 4--6 minutes
| 200...600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled <<gms>> | 11--33 minutes
| ~1{nbsp}µs   | syscall doing real work  | the round figure we use for the diagram                         | *≈{nbsp}1 hour*
| ~2{nbsp}µs   | context switch           | scheduler plus cold cache on the new task                       | ≈{nbsp}2 hours

4+^s| Microseconds -- the operating system

| 10...30{nbsp}µs  | thread wake-up          | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU <<lkmlwake>>  | 9{nbsp}hours -- 1{nbsp}day
| ~31...57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked <<rhel>>           | 1--2 days
| ~59...90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts <<rhel>>                             | 2--3.5 days
| *50{nbsp}µs*     | *OS timer slack, default*  | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups <<slack>> | *≈{nbsp}2 days*
| 1{nbsp}ns        | OS timer slack, minimum    | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` <<slack>> | ≈{nbsp}3 seconds
| ~10{nbsp}µs      | SSD read, best case        | Optane-class (3D XPoint) at QD1 <<intel>>                             | ≈{nbsp}8 hours
| 20...70{nbsp}µs  | NVMe SSD 4K random read    | typical contemporary drive <<simply>>                              | 1--3 days
| 50...100{nbsp}µs | NVMe SSD, interrupt path   | what you see without polling                                       | *2--4 days*
| 100...200{nbsp}µs| SATA SSD 4K random read    | the interface costs, not just the medium <<simply>>                | 4--8 days
| 100{nbsp}µs ... 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once         | 4{nbsp}days -- 5{nbsp}weeks

4+^s| Milliseconds -- where machine and human meet

| 5...10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades <<simply>> | *6{nbsp}months -- 1{nbsp}year*
| ~10{nbsp}ms    | barely noticeable latency      | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year
| 20{nbsp}ms     | one frame @{nbsp}50{nbsp}fps   |                     | ≈{nbsp}2 years
| 40{nbsp}ms     | one frame @{nbsp}25{nbsp}fps   | our actual deadline | *≈{nbsp}4 years*
| ~100{nbsp}ms   | UI starts to feel sluggish     |                     | ≈{nbsp}11 years

4+^s| Seconds and beyond -- the human

| ~1{nbsp}s    | ``the software takes time to respond'' | acceptable, but noticed | *≈{nbsp}1 century*
| ~3{nbsp}min  | ``time for another cup of coffee''     |                         | ≈{nbsp}19 000 years
| ~15{nbsp}min | ``can't wait, I'll do something else'' | ≙{nbsp}1000{nbsp}s, which conveniently closes the fourth decade block | ≈{nbsp}95 000 years
|=========================================================================


The Memory Wall
---------------

For roughly two decades, CPU clock rates have hovered around 3{nbsp}GHz while
DRAM access latency has stayed near 60--100{nbsp}ns. The consequence is
arithmetic: a cache miss that cost a handful of cycles on early-1990s hardware
now costs several hundred. The processor did not merely get faster than
memory -- it got faster than memory _by a growing margin_, and the entire
cache hierarchy exists to paper over that gap. Note in the table above that
the raw device-level latency of DDR5 is barely distinguishable from DDR4;
the generational improvements went into bandwidth, not into latency.

Once several cores share one memory subsystem, a single uniform path to memory
becomes the bottleneck. The answer was to move the memory controller onto the
CPU die and attach memory per socket -- and, with chiplet designs, effectively
per die group. This is what NUMA (non-uniform memory access) describes: memory
that ``belongs'' to the core you are running on is reached quickly, while
memory belonging to a neighbour costs noticeably more. The practical upshot is
that thread placement and memory placement are no longer independent concerns,
and that a scheduler which migrates a worker between cores may silently convert
local accesses into remote ones.

Cores also keep private caches, and a store performed on one core does not
become visible to another core at the moment the instruction retires: it first
lands in that core's store buffer. Cache coherence guarantees that the cores
eventually agree on the content of a cache line, but it says nothing about
_when_, nor about the order in which several independent writes become visible.
This is what memory fences (memory barriers) are for -- they constrain the order,
and thereby allow one core to reason about what an adjacent core has already done.
The cost shows up in the table twice: an uncontended fence is roughly the price of
an L3 hit, whereas a cache line actually bouncing between two cores costs about as
much as going to DRAM. It should be noted that the expensive part is usually _not_
the synchronisation primitive itself, but the cache coherence traffic it provokes.

And as far as memory synchronisation is concerned, there is another twist that
has already led quite some programmers astray. Intel's x86 implements a comparatively
strong coherency model, _Total Store Order_ (TSO): of the four possible reorderings,
only StoreLoad is permitted. Acquire and release semantics are therefore free on x86 --
they compile to plain `MOV` -- and only sequential consistency requires an actual
instruction. Quite to the contrary, ARM and POWER are weakly ordered and consequently
offer a more differentiated set of barriers: ARM distinguishes `DMB` from the heavier
`DSB`, and provides graded load acquire flavours such as `LDAR` and `LDAPR`; POWER
separates the heavyweight `sync` from the cheaper `lwsync`. The trap for the unwary
is that C++ `memory_order` distinctions which cost nothing on x86 have a real price
on ARM -- and, worse, that under-synchronised code can appear perfectly correct
on x86 and fail on ARM.


Crossing into the Kernel
------------------------

A system call is not a function call: it is a controlled trap. The `syscall`
instruction switches the processor to privilege level 0, and the kernel entry
path then swaps to the kernel `GS` base, switches stacks, and saves the user
register state. Since the Meltdown/Spectre family of vulnerabilities, that
sequence also includes a page-table switch (KPTI writes `CR3`, with the
attendant TLB consequences) and one or more branch-prediction barriers --
retpolines, IBPB or IBRS depending on hardware and configuration. On return,
the whole dance runs in reverse.

The direct cost is measurable: a minimal syscall such as `getpid()` amounts
to little more than two mode switches, and lands in the region of a hundred
nanoseconds on a fast host without mitigations, rising to several hundred once
mitigations are enabled <<gms>>. The indirect cost is harder to see
and frequently larger. Kernel entry code and the syscall handler evict
user-space data from L1 and L2, and disturb the branch predictors; the
application resumes with a cold working set and pays for it in misses that no
syscall benchmark will attribute to the syscall. A simple `read()` that happens
to hit data already in the kernel cache can incur costs of around 1{nbsp}µs.

The interesting question thus is is rarely ``how expensive is one syscall'' but
rather ``how many syscalls does this design require per unit of work'' --
which is the question `io_uring` was built to answer.


Operating System Slack
----------------------

A thread that asks to sleep for a specific duration will not, in general, wake
at that instant. Part of this is unavoidable: an interrupt must be handled, the
sleeping task must be moved to the run queue, and the scheduler must actually
place it on a CPU, which together account for something in the range of
10--30{nbsp}µs <<lkmlwake>>.

The larger part, however, is deliberate. Linux maintains a per-thread _timer
slack_ value, and the kernel is free to delay a timed wake-up by up to that
amount in order to coalesce several wake-ups into one and thereby let the CPU
stay in a low-power state longer. For every thread not running under a
real-time scheduling policy, the default is 50{nbsp}µs <<slack>>.
This is not imprecision; it is a power-versus-precision trade that the kernel
makes on the application's behalf, and it can be adjusted -- down to
1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process
`/proc/PID/timerslack_ns` interface.

The key distinction is therefore that between _resolution_ and _accuracy_.
Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond
resolution, and `clock_getres()` will happily confirm it. The accuracy with
which a `nanosleep()` actually returns is a different story altogether, two
to three orders of magnitude coarser, and dominated by slack, wake-up latency
and -- if the core had gone idle -- C-state exit time. Any design that assumes
it can steer a worker thread with microsecond-scale sleeps should be checked
against these figures first.


Sources
-------

The in-text markers such as <<downs>> link down to the full citation here.
Collected as a reading list, roughly in order of usefulness for further work.

[[downs]]
--
downs::
  Travis Downs (6 July 2020) --
  https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html[
    A Concurrency Cost Hierarchy
  ]
//Archive https://web.archive.org/web/20260801071948/https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html
+
The best single entry point on why synchronisation costs what it costs.
+
See the discussion of "vanilla" versus "non-vanilla" instructions: on x86
plain accesses and non-atomic read-modify-writes are cheap while
LOCK-prefixed RMWs are an order of magnitude slower, with `MFENCE`
comparable to a locked instruction.
--

[[lkml]]
--
lkml::
  LKML thread (Nov 2015) --
  https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html[
    &#91;PATCH 3/4&#93; x86,asm: Re-work smp_store_mb()
  ]
//Archive https://web.archive.org/web/20240818210644/https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html
+
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`:
Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds
reports that on Haswell `lock addq` costs pretty much exactly half of
MFENCE, and explains that locked operations were the ones CPU designers had
reason to optimise.
--

[[gms]]
--
gms::
  Georg Sauthoff (30 Aug 2021) --
  https://gms.tf/on-the-costs-of-syscalls.html[
    On the Costs of Syscalls
  ]
//Archive https://web.archive.org/web/20260725162647/https://gms.tf/on-the-costs-of-syscalls.html
+
Careful cross-host microbenchmark of the mode-switch cost.
+
Median of 100 repetitions across a heterogeneous set of hosts; the
conclusion is that user-kernel mode switches cost on the order of a few
hundred nanoseconds on all hosts, with the fastest host switching modes in
under 100 ns, and that the higher figures are explained by CPU-bug
mitigations being enabled by default.
--

[[arkanis]]
--
arkanis::
  Arkanis Development (5 Jan 2017) --
  https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/[
    Measurements of system call performance and overhead
  ]
//Archive https://web.archive.org/web/20260311003705/https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/
+
Compares a plain function call against `getpid()` issued via the syscall
instruction and via the vDSO.
--

[[gregg]]
--
gregg::
  Brendan Gregg (9 Feb 2018) --
  https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html[
    KPTI/KAISER Meltdown Initial Performance Regressions
  ]
//Archive https://web.archive.org/web/20260603211334/https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html
+
Relates mitigation overhead to syscall rate, context switch rate and working
set size -- the systemic view rather than the per-call view.
--

[[lkmlwake]]
--
lkmlwake::
  Michal Schmidt (17 Aug 2007) --
  https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html[
    Reply in the LKML thread "nanosleep() accuracy"
  ]
//Archive https://web.archive.org/web/20260824155617/https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html
+
The point being that a process always carries non-zero wake-up latency
covering interrupt processing, waking and scheduling, and that
10--30{nbsp}µs is not unreasonable.
--

[[rhel]]
--
rhel::
  Clark Williams (Red Hat) --
  https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt[
    Finding Realtime Linux Kernel Latencies
  ]
//Archive https://web.archive.org/web/20250620155700/https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt
+
Walks through a `cyclictest` run and how to read its output: sample
`cyclictest` output and interpretation, section "Using cyclictest".
--

[[slack]]
--
slack::
  Primary documentation --
  `man 7 time`, section "High-resolution timers", and `man 2 prctl`,
  description of `PR_SET_TIMERSLACK`
+
The 50{nbsp}µs default for non-realtime threads is stated explicitly in
https://arxiv.org/pdf/2103.13263[
  Metronome -- adaptive and precise intermittent packet retrieval in DPDK
]
//Archive https://web.archive.org/web/20260725162634/https://arxiv.org/pdf/2103.13263
(section II, discussion of nanosleep limits), and a practical encounter
with the same default, including the `/proc` interface, is documented in
https://bugs.openjdk.org/browse/JDK-8307766[
  OpenJDK issue JDK-8307766
].
//Archive https://web.archive.org/web/20251130011408/https://bugs.openjdk.org/browse/JDK-8307766
--

[[cxl]]
--
cxl::
  arXiv:2503.22017 --
  https://arxiv.org/pdf/2503.22017[
    Performance Characterizations and Usage Guidelines of Samsung CXL Memory
    Module Hybrid Prototype
  ]
//Archive https://web.archive.org/web/20260811083139/https://arxiv.org/pdf/2503.22017
+
Table III contains a well-documented measurement of plain local DDR5:
median latency 122.9{nbsp}ns under Intel MLC pointer chasing versus
13.1{nbsp}ns for a plain load; methodology (rdtsc, 10 000 repetitions,
median) described in section III-C.
--

[[ddr5b]]
--
ddr5b::
  arXiv:2605.08725 --
  https://arxiv.org/pdf/2605.08725[
    Single 32-bit Sub-Channel DDR5 DIMMs
  ]
//Archive https://web.archive.org/web/20260607161507/https://arxiv.org/pdf/2605.08725
+
Useful for the point that DDR5 did not improve absolute latency over DDR4.
+
The relevant passage is in section "Latency Implications", noting that both
DDR4 and DDR5 sit at approximately 14{nbsp}ns first-access latency at their
respective standard speeds.
--

[[simply]]
--
simply::
  simplyblock glossary --
  https://simplyblock.io/glossary/nvme-latency/[
    NVMe Latency
  ]
//Archive https://web.archive.org/web/20260418070043/https://simplyblock.io/glossary/nvme-latency/
+
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy: a modern
NVMe SSD serves a random 4K read in roughly 20--70{nbsp}µs, a SATA SSD in
100--200{nbsp}µs, and a spinning disk in 5--10{nbsp}ms.
--

[[intel]]
--
intel::
  Intel --
  https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf[
    Optane SSD DC P4800X Evaluation Guide
  ]
//Archive https://web.archive.org/web/20260824160928/https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf
+
In Chapter 1.1, Page 5: ``at queue depth of one, 99 out of 100 read operations ...
complete in single digit microseconds.''
+
Corroborated independently by
https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html[
  Tom's Hardware's 2017 preview,
]
//Archive https://web.archive.org/web/20260313082242/https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html
 which reports 10 µs latency for the same drive.
+
Note: Optane was discontinued by Intel in 2022; the figure is retained as the
best-documented physical floor for this latency class.
--


A Note on Microbenchmarks
-------------------------

One caveat deserves a separate mention, as it explains why the published
figures on synchronisation primitives differ so drastically from one another.
A hot microbenchmark keeps every thread runnable and every core out of idle by
its very construction. Under those conditions an uncontended atomic operation or
a Mutex acquisition costs single-digit to low-three-digit nanoseconds, and a
benchmark will faithfully report that. What such a benchmark _structurally_
cannot observe is the cost of _blocking_ -- of a thread actually parking and
later being woken -- because pressing the benchmark is precisely what
prevents that from happening.

The instruction-level cost of a synchronisation primitive is what a hot
microbenchmark measures. Its systemic cost -- coherence traffic under
contention, and blocking and wake-up latency when it is not contended --
is what production tail latency measures. The two values can differ by
two to four orders of magnitude, and it is for this very reason that
the table above covers both regimes.