Time Scales =========== :Author: Lumiera Project :toc: Anyone reasoning about I/O, threading or real-time media processing eventually needs a feeling for _how long things take_. The trouble is that the relevant durations span more than twelve orders of magnitude -- from a single CPU clock cycle up to the point where a human gives up waiting. Numbers on that scale resist intuition, and so we tend to reason about ``fast'' and ``slow'' without noticing that the two words can be separated by a factor of a million. CAUTION: Knowledge-Technology (»AI«) was used to structure and investigate this topic, starting from my own practical observations. Additional corroborating sources were identified and findings integrated into a coherent time axis view. NOTE: All values are order-of-magnitude figures for contemporary desktop and laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on microarchitecture, memory configuration, kernel version and load. The _ratios_ are the point, not the digits. The following table collects the figures that matter in this context. It is meant as a rough orientation, not as a benchmark result. The last column in the table rescales everything by a single constant factor, chosen so that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes one second of human time. Presenting this kind of _human scale proportions_ is an effective way to convey what a cache miss or a disk seek actually costs, and what this means, assuming a _machine's point of view_. Overview Table -------------- [width="100%",cols="12%,26%,40%,22%",options="header"] |========================================================================= | Figure | Item | Remarks | Felt as (1{nbsp}cycle{nbsp}≙{nbsp}1{nbsp}s) 4+^s| Nanoseconds -- inside the CPU | 0.3{nbsp}ns | one clock cycle | at 3.3{nbsp}GHz; the machine's own time quantum | *1 second* | ~1{nbsp}ns | L1 cache hit | about 4 cycles, fully pipelined | ≈{nbsp}3 seconds | ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds | ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds | ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds | ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` <> | ≈{nbsp}17 seconds | ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds | ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW <> | ≈{nbsp}33 seconds | ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds | ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row <> | ≈{nbsp}43 seconds | ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 <> | ≈{nbsp}47 seconds | 40...100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes | ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 <> | *≈{nbsp}5 minutes* 4+^s| Nanoseconds to microseconds -- entering the kernel | ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call <> | ≈{nbsp}1.5 minutes | 70...100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host <> | 4--6 minutes | 200...600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled <> | 11--33 minutes | ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour* | ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours 4+^s| Microseconds -- the operating system | 10...30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU <> | 9{nbsp}hours -- 1{nbsp}day | ~31...57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked <> | 1--2 days | ~59...90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts <> | 2--3.5 days | *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups <> | *≈{nbsp}2 days* | 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` <> | ≈{nbsp}3 seconds | ~10{nbsp}µs | SSD read, best case | Optane-class (3D XPoint) at QD1 <> | ≈{nbsp}8 hours | 20...70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive <> | 1--3 days | 50...100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days* | 100...200{nbsp}µs| SATA SSD 4K random read | the interface costs, not just the medium <> | 4--8 days | 100{nbsp}µs ... 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks 4+^s| Milliseconds -- where machine and human meet | 5...10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades <> | *6{nbsp}months -- 1{nbsp}year* | ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year | 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years | 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years* | ~100{nbsp}ms | UI starts to feel sluggish | | ≈{nbsp}11 years 4+^s| Seconds and beyond -- the human | ~1{nbsp}s | ``the software takes time to respond'' | acceptable, but noticed | *≈{nbsp}1 century* | ~3{nbsp}min | ``time for another cup of coffee'' | | ≈{nbsp}19 000 years | ~15{nbsp}min | ``can't wait, I'll do something else'' | ≙{nbsp}1000{nbsp}s, which conveniently closes the fourth decade block | ≈{nbsp}95 000 years |========================================================================= The Memory Wall --------------- For roughly two decades, CPU clock rates have hovered around 3{nbsp}GHz while DRAM access latency has stayed near 60--100{nbsp}ns. The consequence is arithmetic: a cache miss that cost a handful of cycles on early-1990s hardware now costs several hundred. The processor did not merely get faster than memory -- it got faster than memory _by a growing margin_, and the entire cache hierarchy exists to paper over that gap. Note in the table above that the raw device-level latency of DDR5 is barely distinguishable from DDR4; the generational improvements went into bandwidth, not into latency. Once several cores share one memory subsystem, a single uniform path to memory becomes the bottleneck. The answer was to move the memory controller onto the CPU die and attach memory per socket -- and, with chiplet designs, effectively per die group. This is what NUMA (non-uniform memory access) describes: memory that ``belongs'' to the core you are running on is reached quickly, while memory belonging to a neighbour costs noticeably more. The practical upshot is that thread placement and memory placement are no longer independent concerns, and that a scheduler which migrates a worker between cores may silently convert local accesses into remote ones. Cores also keep private caches, and a store performed on one core does not become visible to another core at the moment the instruction retires: it first lands in that core's store buffer. Cache coherence guarantees that the cores eventually agree on the content of a cache line, but it says nothing about _when_, nor about the order in which several independent writes become visible. This is what memory fences (memory barriers) are for -- they constrain the order, and thereby allow one core to reason about what an adjacent core has already done. The cost shows up in the table twice: an uncontended fence is roughly the price of an L3 hit, whereas a cache line actually bouncing between two cores costs about as much as going to DRAM. It should be noted that the expensive part is usually _not_ the synchronisation primitive itself, but the cache coherence traffic it provokes. And as far as memory synchronisation is concerned, there is another twist that has already led quite some programmers astray. Intel's x86 implements a comparatively strong coherency model, _Total Store Order_ (TSO): of the four possible reorderings, only StoreLoad is permitted. Acquire and release semantics are therefore free on x86 -- they compile to plain `MOV` -- and only sequential consistency requires an actual instruction. Quite to the contrary, ARM and POWER are weakly ordered and consequently offer a more differentiated set of barriers: ARM distinguishes `DMB` from the heavier `DSB`, and provides graded load acquire flavours such as `LDAR` and `LDAPR`; POWER separates the heavyweight `sync` from the cheaper `lwsync`. The trap for the unwary is that C++ `memory_order` distinctions which cost nothing on x86 have a real price on ARM -- and, worse, that under-synchronised code can appear perfectly correct on x86 and fail on ARM. Crossing into the Kernel ------------------------ A system call is not a function call: it is a controlled trap. The `syscall` instruction switches the processor to privilege level 0, and the kernel entry path then swaps to the kernel `GS` base, switches stacks, and saves the user register state. Since the Meltdown/Spectre family of vulnerabilities, that sequence also includes a page-table switch (KPTI writes `CR3`, with the attendant TLB consequences) and one or more branch-prediction barriers -- retpolines, IBPB or IBRS depending on hardware and configuration. On return, the whole dance runs in reverse. The direct cost is measurable: a minimal syscall such as `getpid()` amounts to little more than two mode switches, and lands in the region of a hundred nanoseconds on a fast host without mitigations, rising to several hundred once mitigations are enabled <>. The indirect cost is harder to see and frequently larger. Kernel entry code and the syscall handler evict user-space data from L1 and L2, and disturb the branch predictors; the application resumes with a cold working set and pays for it in misses that no syscall benchmark will attribute to the syscall. A simple `read()` that happens to hit data already in the kernel cache can incur costs of around 1{nbsp}µs. The interesting question thus is is rarely ``how expensive is one syscall'' but rather ``how many syscalls does this design require per unit of work'' -- which is the question `io_uring` was built to answer. Operating System Slack ---------------------- A thread that asks to sleep for a specific duration will not, in general, wake at that instant. Part of this is unavoidable: an interrupt must be handled, the sleeping task must be moved to the run queue, and the scheduler must actually place it on a CPU, which together account for something in the range of 10--30{nbsp}µs <>. The larger part, however, is deliberate. Linux maintains a per-thread _timer slack_ value, and the kernel is free to delay a timed wake-up by up to that amount in order to coalesce several wake-ups into one and thereby let the CPU stay in a low-power state longer. For every thread not running under a real-time scheduling policy, the default is 50{nbsp}µs <>. This is not imprecision; it is a power-versus-precision trade that the kernel makes on the application's behalf, and it can be adjusted -- down to 1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process `/proc/PID/timerslack_ns` interface. The key distinction is therefore that between _resolution_ and _accuracy_. Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond resolution, and `clock_getres()` will happily confirm it. The accuracy with which a `nanosleep()` actually returns is a different story altogether, two to three orders of magnitude coarser, and dominated by slack, wake-up latency and -- if the core had gone idle -- C-state exit time. Any design that assumes it can steer a worker thread with microsecond-scale sleeps should be checked against these figures first. Sources ------- The in-text markers such as <> link down to the full citation here. Collected as a reading list, roughly in order of usefulness for further work. [[downs]] -- downs:: Travis Downs (6 July 2020) -- https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html[ A Concurrency Cost Hierarchy ] //Archive https://web.archive.org/web/20260801071948/https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html + The best single entry point on why synchronisation costs what it costs. + See the discussion of "vanilla" versus "non-vanilla" instructions: on x86 plain accesses and non-atomic read-modify-writes are cheap while LOCK-prefixed RMWs are an order of magnitude slower, with `MFENCE` comparable to a locked instruction. -- [[lkml]] -- lkml:: LKML thread (Nov 2015) -- https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html[ [PATCH 3/4] x86,asm: Re-work smp_store_mb() ] //Archive https://web.archive.org/web/20240818210644/https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html + Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`: Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds reports that on Haswell `lock addq` costs pretty much exactly half of MFENCE, and explains that locked operations were the ones CPU designers had reason to optimise. -- [[gms]] -- gms:: Georg Sauthoff (30 Aug 2021) -- https://gms.tf/on-the-costs-of-syscalls.html[ On the Costs of Syscalls ] //Archive https://web.archive.org/web/20260725162647/https://gms.tf/on-the-costs-of-syscalls.html + Careful cross-host microbenchmark of the mode-switch cost. + Median of 100 repetitions across a heterogeneous set of hosts; the conclusion is that user-kernel mode switches cost on the order of a few hundred nanoseconds on all hosts, with the fastest host switching modes in under 100 ns, and that the higher figures are explained by CPU-bug mitigations being enabled by default. -- [[arkanis]] -- arkanis:: Arkanis Development (5 Jan 2017) -- https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/[ Measurements of system call performance and overhead ] //Archive https://web.archive.org/web/20260311003705/https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ + Compares a plain function call against `getpid()` issued via the syscall instruction and via the vDSO. -- [[gregg]] -- gregg:: Brendan Gregg (9 Feb 2018) -- https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html[ KPTI/KAISER Meltdown Initial Performance Regressions ] //Archive https://web.archive.org/web/20260603211334/https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html + Relates mitigation overhead to syscall rate, context switch rate and working set size -- the systemic view rather than the per-call view. -- [[lkmlwake]] -- lkmlwake:: Michal Schmidt (17 Aug 2007) -- https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html[ Reply in the LKML thread "nanosleep() accuracy" ] //Archive https://web.archive.org/web/20260824155617/https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html + The point being that a process always carries non-zero wake-up latency covering interrupt processing, waking and scheduling, and that 10--30{nbsp}µs is not unreasonable. -- [[rhel]] -- rhel:: Clark Williams (Red Hat) -- https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt[ Finding Realtime Linux Kernel Latencies ] //Archive https://web.archive.org/web/20250620155700/https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt + Walks through a `cyclictest` run and how to read its output: sample `cyclictest` output and interpretation, section "Using cyclictest". -- [[slack]] -- slack:: Primary documentation -- `man 7 time`, section "High-resolution timers", and `man 2 prctl`, description of `PR_SET_TIMERSLACK` + The 50{nbsp}µs default for non-realtime threads is stated explicitly in https://arxiv.org/pdf/2103.13263[ Metronome -- adaptive and precise intermittent packet retrieval in DPDK ] //Archive https://web.archive.org/web/20260725162634/https://arxiv.org/pdf/2103.13263 (section II, discussion of nanosleep limits), and a practical encounter with the same default, including the `/proc` interface, is documented in https://bugs.openjdk.org/browse/JDK-8307766[ OpenJDK issue JDK-8307766 ]. //Archive https://web.archive.org/web/20251130011408/https://bugs.openjdk.org/browse/JDK-8307766 -- [[cxl]] -- cxl:: arXiv:2503.22017 -- https://arxiv.org/pdf/2503.22017[ Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototype ] //Archive https://web.archive.org/web/20260811083139/https://arxiv.org/pdf/2503.22017 + Table III contains a well-documented measurement of plain local DDR5: median latency 122.9{nbsp}ns under Intel MLC pointer chasing versus 13.1{nbsp}ns for a plain load; methodology (rdtsc, 10 000 repetitions, median) described in section III-C. -- [[ddr5b]] -- ddr5b:: arXiv:2605.08725 -- https://arxiv.org/pdf/2605.08725[ Single 32-bit Sub-Channel DDR5 DIMMs ] //Archive https://web.archive.org/web/20260607161507/https://arxiv.org/pdf/2605.08725 + Useful for the point that DDR5 did not improve absolute latency over DDR4. + The relevant passage is in section "Latency Implications", noting that both DDR4 and DDR5 sit at approximately 14{nbsp}ns first-access latency at their respective standard speeds. -- [[simply]] -- simply:: simplyblock glossary -- https://simplyblock.io/glossary/nvme-latency/[ NVMe Latency ] //Archive https://web.archive.org/web/20260418070043/https://simplyblock.io/glossary/nvme-latency/ + Compact summary of the NVMe / SATA SSD / rotating disk hierarchy: a modern NVMe SSD serves a random 4K read in roughly 20--70{nbsp}µs, a SATA SSD in 100--200{nbsp}µs, and a spinning disk in 5--10{nbsp}ms. -- [[intel]] -- intel:: Intel -- https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf[ Optane SSD DC P4800X Evaluation Guide ] //Archive https://web.archive.org/web/20260824160928/https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf + In Chapter 1.1, Page 5: ``at queue depth of one, 99 out of 100 read operations ... complete in single digit microseconds.'' + Corroborated independently by https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html[ Tom's Hardware's 2017 preview, ] //Archive https://web.archive.org/web/20260313082242/https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html which reports 10 µs latency for the same drive. + Note: Optane was discontinued by Intel in 2022; the figure is retained as the best-documented physical floor for this latency class. -- A Note on Microbenchmarks ------------------------- One caveat deserves a separate mention, as it explains why the published figures on synchronisation primitives differ so drastically from one another. A hot microbenchmark keeps every thread runnable and every core out of idle by its very construction. Under those conditions an uncontended atomic operation or a Mutex acquisition costs single-digit to low-three-digit nanoseconds, and a benchmark will faithfully report that. What such a benchmark _structurally_ cannot observe is the cost of _blocking_ -- of a thread actually parking and later being woken -- because pressing the benchmark is precisely what prevents that from happening. The instruction-level cost of a synchronisation primitive is what a hot microbenchmark measures. Its systemic cost -- coherence traffic under contention, and blocking and wake-up latency when it is not contended -- is what production tail latency measures. The two values can differ by two to four orders of magnitude, and it is for this very reason that the table above covers both regimes.