|
|
|
|
@ -10,18 +10,23 @@ cycle up to the point where a human gives up waiting. Numbers on that scale
|
|
|
|
|
resist intuition, and so we tend to reason about ``fast'' and ``slow'' without
|
|
|
|
|
noticing that the two words can be separated by a factor of a million.
|
|
|
|
|
|
|
|
|
|
The following table collects the figures that matter in this context. It is
|
|
|
|
|
meant as a rough orientation, not as a benchmark result.
|
|
|
|
|
CAUTION: Knowledge-Technology (»AI«) was used to structure and investigate this topic,
|
|
|
|
|
starting from my own practical observations. Additional corroborating sources
|
|
|
|
|
were identified and findings integrated into a coherent time axis view.
|
|
|
|
|
|
|
|
|
|
NOTE: All values are order-of-magnitude figures for contemporary desktop and
|
|
|
|
|
laptop x86 hardware. Expect a deviation of 2--3{nbsp}x depending on
|
|
|
|
|
microarchitecture, memory configuration, kernel version and load. The
|
|
|
|
|
_ratios_ are the point, not the digits.
|
|
|
|
|
|
|
|
|
|
The last column rescales everything by a single constant factor, chosen so
|
|
|
|
|
that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes one second
|
|
|
|
|
of human time. This is the most effective way we know to convey what a cache
|
|
|
|
|
miss or a disk seek actually costs from the machine's point of view.
|
|
|
|
|
The following table collects the figures that matter in this context. It is
|
|
|
|
|
meant as a rough orientation, not as a benchmark result.
|
|
|
|
|
|
|
|
|
|
The last column in the table rescales everything by a single constant factor,
|
|
|
|
|
chosen so that one CPU clock cycle (≙ 0.3{nbsp}ns at 3.3{nbsp}GHz) becomes
|
|
|
|
|
one second of human time. Presenting this kind of _human scale proportions_
|
|
|
|
|
is an effective way to convey what a cache miss or a disk seek actually costs,
|
|
|
|
|
and what this means, assuming a _machine's point of view_.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Overview Table
|
|
|
|
|
@ -38,40 +43,39 @@ Overview Table
|
|
|
|
|
| ~1.5{nbsp}ns | `sfence` | store fence, ≈{nbsp}5 cycles; rarely needed on x86 | ≈{nbsp}5 seconds
|
|
|
|
|
| ~2.7{nbsp}ns | `lfence` | ≈{nbsp}9 cycles; no ordering semantics on x86 beyond blocking speculation | ≈{nbsp}9 seconds
|
|
|
|
|
| ~4{nbsp}ns | L2 cache hit | ≈{nbsp}12--15 cycles | ≈{nbsp}13 seconds
|
|
|
|
|
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` footnoteref:[lkml,LKML thread "[PATCH 3/4] x86,asm: Re-work smp_store_mb()", Nov 2015. Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds reports that on Haswell "lock addq" costs pretty much exactly half of MFENCE, and explains that locked operations were the ones CPU designers had reason to optimise -- https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html] | ≈{nbsp}17 seconds
|
|
|
|
|
| ~5{nbsp}ns | `lock add $0,(%rsp)` | the actual `smp_mb()` on Linux; ≈{nbsp}half the cost of `mfence` <<lkml>> | ≈{nbsp}17 seconds
|
|
|
|
|
| ~5{nbsp}ns | branch misprediction | pipeline refill | ≈{nbsp}17 seconds
|
|
|
|
|
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW footnoteref:[downs,Travis Downs, "A Concurrency Cost Hierarchy", 6 July 2020, see the discussion of "vanilla" versus "non-vanilla" instructions: on x86 plain accesses and non-atomic read-modify-writes are cheap while LOCK-prefixed RMWs are an order of magnitude slower, with MFENCE comparable to a locked instruction -- https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html] | ≈{nbsp}33 seconds
|
|
|
|
|
| ~10{nbsp}ns | `mfence` / full barrier | ≈{nbsp}23--33 cycles; comparable to a `lock`-prefixed RMW <<downs>> | ≈{nbsp}33 seconds
|
|
|
|
|
| ~15{nbsp}ns | L3 cache hit | shared, hence variable | ≈{nbsp}50 seconds
|
|
|
|
|
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row footnoteref:[cxl,Measured values in Table III of: Performance Characterizations and Usage Guidelines of Samsung CXL Memory Module Hybrid Prototype, arXiv:2503.22017. Local DDR5 median latency 122.9 ns under Intel MLC pointer chasing versus 13.1 ns for a plain load; methodology (rdtsc, 10 000 repetitions, median) described in section III-C -- https://arxiv.org/pdf/2503.22017] | ≈{nbsp}43 seconds
|
|
|
|
|
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 footnote:[Measurement-oriented analysis of DDR5 sub-channel architecture; the relevant passage is in section "Latency Implications", noting that both DDR4 and DDR5 sit at approximately 14 ns first-access latency at their respective standard speeds. arXiv:2605.08725 -- https://arxiv.org/pdf/2605.08725]
|
|
|
|
|
| ≈{nbsp}47 seconds
|
|
|
|
|
| 40--100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
|
|
|
|
|
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 footnoteref:[cxl] | *≈{nbsp}5 minutes*
|
|
|
|
|
| ~13{nbsp}ns | DRAM, open row | a plain load that happens to hit an already-activated row <<cxl>> | ≈{nbsp}43 seconds
|
|
|
|
|
| ~14{nbsp}ns | DRAM, first access | device-level figure; essentially unchanged from DDR4 to DDR5 <<ddr5b>> | ≈{nbsp}47 seconds
|
|
|
|
|
| 40...100{nbsp}ns | contended cache line | core-to-core transfer; strongly topology dependent (same CCX / cross CCX / cross socket) | 2--5 minutes
|
|
|
|
|
| ~100{nbsp}ns | DRAM random access | full miss through the whole hierarchy; 122.9{nbsp}ns measured by pointer chasing on DDR5 <<cxl>> | *≈{nbsp}5 minutes*
|
|
|
|
|
|
|
|
|
|
4+^s| Nanoseconds to microseconds -- entering the kernel
|
|
|
|
|
|
|
|
|
|
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call footnoteref:[arkanis,"Measurements of system call performance and overhead", Arkanis Development, 5 Jan 2017 -- compares a plain function call against getpid() issued via the syscall instruction and via the vDSO -- https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/] | ≈{nbsp}1.5 minutes
|
|
|
|
|
| 70--100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host footnoteref:[gms,Georg Sauthoff, "On the Costs of Syscalls", 30 Aug 2021. Median of 100 repetitions across a heterogeneous set of hosts; the conclusion is that user-kernel mode switches cost on the order of a few hundred nanoseconds on all hosts, with the fastest host switching modes in under 100 ns, and that the higher figures are explained by CPU-bug mitigations being enabled by default -- https://gms.tf/on-the-costs-of-syscalls.html] | 4--6 minutes
|
|
|
|
|
| 200--600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled footnoteref:[gms] | 11--33 minutes
|
|
|
|
|
| ~25{nbsp}ns | syscall via vDSO | not a real trap; almost as cheap as a function call <<arkanis>> | ≈{nbsp}1.5 minutes
|
|
|
|
|
| 70...100{nbsp}ns | syscall trap, bare | `getpid()` on a mitigation-free host <<gms>> | 4--6 minutes
|
|
|
|
|
| 200...600{nbsp}ns | syscall trap, mitigated | ``a few hundred nanoseconds'' on typical hosts with CPU-bug mitigations enabled <<gms>> | 11--33 minutes
|
|
|
|
|
| ~1{nbsp}µs | syscall doing real work | the round figure we use for the diagram | *≈{nbsp}1 hour*
|
|
|
|
|
| ~2{nbsp}µs | context switch | scheduler plus cold cache on the new task | ≈{nbsp}2 hours
|
|
|
|
|
|
|
|
|
|
4+^s| Microseconds -- the operating system
|
|
|
|
|
|
|
|
|
|
| 10--30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU footnoteref:[lkmlwake,Michal Schmidt, reply in the thread "nanosleep() accuracy", linux-kernel mailing list, 17 Aug 2007: the point being that a process always carries non-zero wake-up latency covering interrupt processing, waking and scheduling, and that 10-30 us is not unreasonable -- https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html] | 9{nbsp}hours -- 1{nbsp}day
|
|
|
|
|
| ~31--57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked footnoteref:[rhel,Sample cyclictest output and interpretation in: Finding Realtime Linux Kernel Latencies, Clark Williams (Red Hat), section "Using cyclictest" -- https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt] | 1--2 days
|
|
|
|
|
| ~59--90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts footnoteref:[rhel] | 2--3.5 days
|
|
|
|
|
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups footnoteref:[slack,Primary documentation: `man 7 time`, section "High-resolution timers", and `man 2 prctl`, description of PR_SET_TIMERSLACK. The 50 us default for non-realtime threads is stated explicitly in: Metronome -- adaptive and precise intermittent packet retrieval in DPDK, section II (discussion of nanosleep limits), arXiv:2103.13263 -- https://arxiv.org/pdf/2103.13263 ; a practical encounter with the same default, including the /proc interface, is documented in OpenJDK issue JDK-8307766 -- https://bugs.openjdk.org/browse/JDK-8307766] | *≈{nbsp}2 days*
|
|
|
|
|
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` footnoteref:[slack] | ≈{nbsp}3 seconds
|
|
|
|
|
| ~9{nbsp}µs | SSD read, best case | Optane / Z-NAND class at QD1 footnoteref:[anand,AnandTech, review of the Samsung 983 ZET (Z-NAND), page 3, "QD1 Random Read Performance" -- explains why queue depth 1 exposes best-case access latency, and reports an average of just under 9 us for an Optane drive against 16 us for the Z-NAND based 983 ZET -- https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3] | ≈{nbsp}8 hours
|
|
|
|
|
| 20--70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive footnoteref:[simply,Summary of the storage latency hierarchy: a modern NVMe SSD serves a random 4K read in roughly 20-70 us, a SATA SSD in 100-200 us, and a spinning disk in 5-10 ms. simplyblock glossary entry "NVMe Latency" -- https://simplyblock.io/glossary/nvme-latency/] | 1--3 days
|
|
|
|
|
| 50--100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
|
|
|
|
|
| 100--200{nbsp}µs | SATA SSD 4K random read | the interface costs, not just the medium footnoteref:[simply] | 4--8 days
|
|
|
|
|
| 100{nbsp}µs -- 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
|
|
|
|
|
| 10...30{nbsp}µs | thread wake-up | IRQ handling, `try_to_wake_up()`, scheduling onto a CPU <<lkmlwake>> | 9{nbsp}hours -- 1{nbsp}day
|
|
|
|
|
| ~31...57{nbsp}µs | scheduling latency, avg | `cyclictest` at `SCHED_FIFO` 95 with memory locked <<rhel>> | 1--2 days
|
|
|
|
|
| ~59...90{nbsp}µs | scheduling latency, max | same run; the tail is what hurts <<rhel>> | 2--3.5 days
|
|
|
|
|
| *50{nbsp}µs* | *OS timer slack, default* | the kernel may delay a timed wake-up by this much, deliberately, to coalesce wakeups <<slack>> | *≈{nbsp}2 days*
|
|
|
|
|
| 1{nbsp}ns | OS timer slack, minimum | via `prctl(PR_SET_TIMERSLACK)` or `/proc/PID/timerslack_ns` <<slack>> | ≈{nbsp}3 seconds
|
|
|
|
|
| ~10{nbsp}µs | SSD read, best case | Optane-class (3D XPoint) at QD1 <<intel>> | ≈{nbsp}8 hours
|
|
|
|
|
| 20...70{nbsp}µs | NVMe SSD 4K random read | typical contemporary drive <<simply>> | 1--3 days
|
|
|
|
|
| 50...100{nbsp}µs | NVMe SSD, interrupt path | what you see without polling | *2--4 days*
|
|
|
|
|
| 100...200{nbsp}µs| SATA SSD 4K random read | the interface costs, not just the medium <<simply>> | 4--8 days
|
|
|
|
|
| 100{nbsp}µs ... 1{nbsp}ms | deep C-state exit | wake from C6 and below; worse when many cores wake at once | 4{nbsp}days -- 5{nbsp}weeks
|
|
|
|
|
|
|
|
|
|
4+^s| Milliseconds -- where machine and human meet
|
|
|
|
|
|
|
|
|
|
| 5--10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades footnoteref:[simply] | *6{nbsp}months -- 1{nbsp}year*
|
|
|
|
|
| 5...10{nbsp}ms | rotating disk, seek + rotation | 7200{nbsp}rpm; the number that has not improved in decades <<simply>> | *6{nbsp}months -- 1{nbsp}year*
|
|
|
|
|
| ~10{nbsp}ms | barely noticeable latency | the threshold below which a human stops perceiving delay | ≈{nbsp}1 year
|
|
|
|
|
| 20{nbsp}ms | one frame @{nbsp}50{nbsp}fps | | ≈{nbsp}2 years
|
|
|
|
|
| 40{nbsp}ms | one frame @{nbsp}25{nbsp}fps | our actual deadline | *≈{nbsp}4 years*
|
|
|
|
|
@ -112,24 +116,24 @@ become visible to another core at the moment the instruction retires: it first
|
|
|
|
|
lands in that core's store buffer. Cache coherence guarantees that the cores
|
|
|
|
|
eventually agree on the content of a cache line, but it says nothing about
|
|
|
|
|
_when_, nor about the order in which several independent writes become visible.
|
|
|
|
|
This is what memory fences are for -- they constrain the order, and thereby
|
|
|
|
|
allow one core to reason about what an adjacent core has already done. The cost
|
|
|
|
|
shows up in the table twice: an uncontended fence is roughly the price of an
|
|
|
|
|
L3 hit, whereas a cache line actually bouncing between two cores costs about as
|
|
|
|
|
much as going to DRAM. It is worth internalising that the expensive part is
|
|
|
|
|
usually not the synchronisation primitive itself, but the coherence traffic it
|
|
|
|
|
provokes.
|
|
|
|
|
This is what memory fences (memory barriers) are for -- they constrain the order,
|
|
|
|
|
and thereby allow one core to reason about what an adjacent core has already done.
|
|
|
|
|
The cost shows up in the table twice: an uncontended fence is roughly the price of
|
|
|
|
|
an L3 hit, whereas a cache line actually bouncing between two cores costs about as
|
|
|
|
|
much as going to DRAM. It should be noted that the expensive part is usually _not_
|
|
|
|
|
the synchronisation primitive itself, but the cache coherence traffic it provokes.
|
|
|
|
|
|
|
|
|
|
x86 implements a comparatively strong model, _Total Store Order_ (TSO): of the
|
|
|
|
|
four possible reorderings, only StoreLoad is permitted. Acquire and release
|
|
|
|
|
semantics are therefore free on x86 -- they compile to plain `MOV` -- and only
|
|
|
|
|
sequential consistency requires an actual instruction. ARM and POWER are
|
|
|
|
|
weakly ordered and consequently offer a more differentiated set of barriers:
|
|
|
|
|
ARM distinguishes `DMB` from the heavier `DSB`, and provides graded load
|
|
|
|
|
acquire flavours such as `LDAR` and `LDAPR`; POWER separates the heavyweight
|
|
|
|
|
`sync` from the cheaper `lwsync`. The trap for the unwary is that C++
|
|
|
|
|
`memory_order` distinctions which cost nothing on x86 have a real price on
|
|
|
|
|
ARM -- and, worse, that under-synchronised code can appear perfectly correct
|
|
|
|
|
And as far as memory synchronisation is concerned, there is another twist that
|
|
|
|
|
has already led quite some programmers astray. Intel's x86 implements a comparatively
|
|
|
|
|
strong coherency model, _Total Store Order_ (TSO): of the four possible reorderings,
|
|
|
|
|
only StoreLoad is permitted. Acquire and release semantics are therefore free on x86 --
|
|
|
|
|
they compile to plain `MOV` -- and only sequential consistency requires an actual
|
|
|
|
|
instruction. Quite to the contrary, ARM and POWER are weakly ordered and consequently
|
|
|
|
|
offer a more differentiated set of barriers: ARM distinguishes `DMB` from the heavier
|
|
|
|
|
`DSB`, and provides graded load acquire flavours such as `LDAR` and `LDAPR`; POWER
|
|
|
|
|
separates the heavyweight `sync` from the cheaper `lwsync`. The trap for the unwary
|
|
|
|
|
is that C++ `memory_order` distinctions which cost nothing on x86 have a real price
|
|
|
|
|
on ARM -- and, worse, that under-synchronised code can appear perfectly correct
|
|
|
|
|
on x86 and fail on ARM.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@ -145,17 +149,19 @@ attendant TLB consequences) and one or more branch-prediction barriers --
|
|
|
|
|
retpolines, IBPB or IBRS depending on hardware and configuration. On return,
|
|
|
|
|
the whole dance runs in reverse.
|
|
|
|
|
|
|
|
|
|
The direct cost is measurable: a minimal syscall such as `getpid()` amounts to
|
|
|
|
|
little more than two mode switches, and lands in the region of a hundred
|
|
|
|
|
The direct cost is measurable: a minimal syscall such as `getpid()` amounts
|
|
|
|
|
to little more than two mode switches, and lands in the region of a hundred
|
|
|
|
|
nanoseconds on a fast host without mitigations, rising to several hundred once
|
|
|
|
|
mitigations are enabled.footnoteref:[gms] The indirect cost is harder to see
|
|
|
|
|
mitigations are enabled <<gms>>. The indirect cost is harder to see
|
|
|
|
|
and frequently larger. Kernel entry code and the syscall handler evict
|
|
|
|
|
user-space data from L1 and L2, and disturb the branch predictors; the
|
|
|
|
|
application resumes with a cold working set and pays for it in misses that no
|
|
|
|
|
syscall benchmark will attribute to the syscall. This is precisely why the
|
|
|
|
|
interesting question is rarely ``how expensive is one syscall'' but rather
|
|
|
|
|
``how many syscalls does this design require per unit of work'' -- which is
|
|
|
|
|
the question `io_uring` was built to answer.
|
|
|
|
|
syscall benchmark will attribute to the syscall. A simple `read()` that happens
|
|
|
|
|
to hit data already in the kernel cache can incur costs of around 1{nbsp}µs.
|
|
|
|
|
|
|
|
|
|
The interesting question thus is is rarely ``how expensive is one syscall'' but
|
|
|
|
|
rather ``how many syscalls does this design require per unit of work'' --
|
|
|
|
|
which is the question `io_uring` was built to answer.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Operating System Slack
|
|
|
|
|
@ -165,100 +171,244 @@ A thread that asks to sleep for a specific duration will not, in general, wake
|
|
|
|
|
at that instant. Part of this is unavoidable: an interrupt must be handled, the
|
|
|
|
|
sleeping task must be moved to the run queue, and the scheduler must actually
|
|
|
|
|
place it on a CPU, which together account for something in the range of
|
|
|
|
|
10--30{nbsp}µs.footnoteref:[lkmlwake]
|
|
|
|
|
10--30{nbsp}µs <<lkmlwake>>.
|
|
|
|
|
|
|
|
|
|
The larger part, however, is deliberate. Linux maintains a per-thread _timer
|
|
|
|
|
slack_ value, and the kernel is free to delay a timed wake-up by up to that
|
|
|
|
|
amount in order to coalesce several wake-ups into one and thereby let the CPU
|
|
|
|
|
stay in a low-power state longer. For every thread not running under a
|
|
|
|
|
real-time scheduling policy, the default is 50{nbsp}µs.footnoteref:[slack]
|
|
|
|
|
real-time scheduling policy, the default is 50{nbsp}µs <<slack>>.
|
|
|
|
|
This is not imprecision; it is a power-versus-precision trade that the kernel
|
|
|
|
|
makes on the application's behalf, and it can be adjusted -- down to
|
|
|
|
|
1{nbsp}ns -- via `prctl(PR_SET_TIMERSLACK)` or the per-process
|
|
|
|
|
`/proc/PID/timerslack_ns` interface.
|
|
|
|
|
|
|
|
|
|
The distinction worth carrying away is between _resolution_ and _accuracy_.
|
|
|
|
|
The key distinction is therefore that between _resolution_ and _accuracy_.
|
|
|
|
|
Since Linux 2.6.21 the high-resolution timer subsystem reports nanosecond
|
|
|
|
|
resolution, and `clock_getres()` will happily confirm it. The accuracy with
|
|
|
|
|
which a `nanosleep()` actually returns is a different quantity altogether, two
|
|
|
|
|
which a `nanosleep()` actually returns is a different story altogether, two
|
|
|
|
|
to three orders of magnitude coarser, and dominated by slack, wake-up latency
|
|
|
|
|
and -- if the core had gone idle -- C-state exit time. Any design that assumes
|
|
|
|
|
it can steer a worker thread with microsecond-scale sleeps should be checked
|
|
|
|
|
against these figures first.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
// ----------------------------------------------------------------------
|
|
|
|
|
// Footnote definitions are attached at first use above via footnoteref.
|
|
|
|
|
// Repeated references reuse the id; see AsciiDoc footnoteref macro.
|
|
|
|
|
// ----------------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
Sources
|
|
|
|
|
-------
|
|
|
|
|
|
|
|
|
|
The footnotes above carry the precise locations. Collected here as a reading
|
|
|
|
|
list, roughly in order of usefulness for further work.
|
|
|
|
|
The in-text markers such as <<downs>> link down to the full citation here.
|
|
|
|
|
Collected as a reading list, roughly in order of usefulness for further work.
|
|
|
|
|
|
|
|
|
|
- Travis Downs, *A Concurrency Cost Hierarchy* --
|
|
|
|
|
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html +
|
|
|
|
|
[[downs]]
|
|
|
|
|
--
|
|
|
|
|
downs::
|
|
|
|
|
Travis Downs (6 July 2020) --
|
|
|
|
|
https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html[
|
|
|
|
|
A Concurrency Cost Hierarchy
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260801071948/https://travisdowns.github.io/blog/2020/07/06/concurrency-costs.html
|
|
|
|
|
+
|
|
|
|
|
The best single entry point on why synchronisation costs what it costs.
|
|
|
|
|
+
|
|
|
|
|
See the discussion of "vanilla" versus "non-vanilla" instructions: on x86
|
|
|
|
|
plain accesses and non-atomic read-modify-writes are cheap while
|
|
|
|
|
LOCK-prefixed RMWs are an order of magnitude slower, with `MFENCE`
|
|
|
|
|
comparable to a locked instruction.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- LKML thread, *[PATCH 3/4] x86,asm: Re-work smp_store_mb()* (Nov 2015) --
|
|
|
|
|
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html +
|
|
|
|
|
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`.
|
|
|
|
|
[[lkml]]
|
|
|
|
|
--
|
|
|
|
|
lkml::
|
|
|
|
|
LKML thread (Nov 2015) --
|
|
|
|
|
https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html[
|
|
|
|
|
[PATCH 3/4] x86,asm: Re-work smp_store_mb()
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20240818210644/https://lkml.iu.edu/hypermail/linux/kernel/1511.0/01052.html
|
|
|
|
|
+
|
|
|
|
|
Kernel developers measuring `MFENCE` against `XCHG` and `lock addq`:
|
|
|
|
|
Davidlohr Bueso measures XCHG against MFENCE on Ivy Bridge; Linus Torvalds
|
|
|
|
|
reports that on Haswell `lock addq` costs pretty much exactly half of
|
|
|
|
|
MFENCE, and explains that locked operations were the ones CPU designers had
|
|
|
|
|
reason to optimise.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- Georg Sauthoff, *On the Costs of Syscalls* (Aug 2021) --
|
|
|
|
|
https://gms.tf/on-the-costs-of-syscalls.html +
|
|
|
|
|
[[gms]]
|
|
|
|
|
--
|
|
|
|
|
gms::
|
|
|
|
|
Georg Sauthoff (30 Aug 2021) --
|
|
|
|
|
https://gms.tf/on-the-costs-of-syscalls.html[
|
|
|
|
|
On the Costs of Syscalls
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260725162647/https://gms.tf/on-the-costs-of-syscalls.html
|
|
|
|
|
+
|
|
|
|
|
Careful cross-host microbenchmark of the mode-switch cost.
|
|
|
|
|
+
|
|
|
|
|
Median of 100 repetitions across a heterogeneous set of hosts; the
|
|
|
|
|
conclusion is that user-kernel mode switches cost on the order of a few
|
|
|
|
|
hundred nanoseconds on all hosts, with the fastest host switching modes in
|
|
|
|
|
under 100 ns, and that the higher figures are explained by CPU-bug
|
|
|
|
|
mitigations being enabled by default.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- *Measurements of system call performance and overhead* (Jan 2017) --
|
|
|
|
|
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/ +
|
|
|
|
|
Function call versus `syscall` instruction versus vDSO.
|
|
|
|
|
[[arkanis]]
|
|
|
|
|
--
|
|
|
|
|
arkanis::
|
|
|
|
|
Arkanis Development (5 Jan 2017) --
|
|
|
|
|
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/[
|
|
|
|
|
Measurements of system call performance and overhead
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260311003705/https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/
|
|
|
|
|
+
|
|
|
|
|
Compares a plain function call against `getpid()` issued via the syscall
|
|
|
|
|
instruction and via the vDSO.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- Brendan Gregg, *KPTI/KAISER Meltdown Initial Performance Regressions* --
|
|
|
|
|
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html +
|
|
|
|
|
[[gregg]]
|
|
|
|
|
--
|
|
|
|
|
gregg::
|
|
|
|
|
Brendan Gregg (9 Feb 2018) --
|
|
|
|
|
https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html[
|
|
|
|
|
KPTI/KAISER Meltdown Initial Performance Regressions
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260603211334/https://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html
|
|
|
|
|
+
|
|
|
|
|
Relates mitigation overhead to syscall rate, context switch rate and working
|
|
|
|
|
set size -- the systemic view rather than the per-call view.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- Clark Williams (Red Hat), *Finding Realtime Linux Kernel Latencies* --
|
|
|
|
|
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt +
|
|
|
|
|
Walks through a `cyclictest` run and how to read its output.
|
|
|
|
|
[[lkmlwake]]
|
|
|
|
|
--
|
|
|
|
|
lkmlwake::
|
|
|
|
|
Michal Schmidt (17 Aug 2007) --
|
|
|
|
|
https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html[
|
|
|
|
|
Reply in the LKML thread "nanosleep() accuracy"
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260824155617/https://lkml.iu.edu/hypermail/linux/kernel/0708.2/0423.html
|
|
|
|
|
+
|
|
|
|
|
The point being that a process always carries non-zero wake-up latency
|
|
|
|
|
covering interrupt processing, waking and scheduling, and that
|
|
|
|
|
10--30{nbsp}µs is not unreasonable.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- `man 7 time` and `man 2 prctl` (PR_SET_TIMERSLACK) +
|
|
|
|
|
Primary documentation for timer slack. The 50{nbsp}µs default is stated in
|
|
|
|
|
https://arxiv.org/pdf/2103.13263 (Metronome, DPDK packet retrieval) and
|
|
|
|
|
encountered in practice in https://bugs.openjdk.org/browse/JDK-8307766
|
|
|
|
|
[[rhel]]
|
|
|
|
|
--
|
|
|
|
|
rhel::
|
|
|
|
|
Clark Williams (Red Hat) --
|
|
|
|
|
https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt[
|
|
|
|
|
Finding Realtime Linux Kernel Latencies
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20250620155700/https://people.redhat.com/williams/latency-howto/rt-latency-howto.txt
|
|
|
|
|
+
|
|
|
|
|
Walks through a `cyclictest` run and how to read its output: sample
|
|
|
|
|
`cyclictest` output and interpretation, section "Using cyclictest".
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- *Performance Characterizations ... Samsung CXL Memory Module Hybrid
|
|
|
|
|
Prototype*, arXiv:2503.22017 -- https://arxiv.org/pdf/2503.22017 +
|
|
|
|
|
Table III contains a well-documented measurement of plain local DDR5.
|
|
|
|
|
[[slack]]
|
|
|
|
|
--
|
|
|
|
|
slack::
|
|
|
|
|
Primary documentation --
|
|
|
|
|
`man 7 time`, section "High-resolution timers", and `man 2 prctl`,
|
|
|
|
|
description of `PR_SET_TIMERSLACK`
|
|
|
|
|
+
|
|
|
|
|
The 50{nbsp}µs default for non-realtime threads is stated explicitly in
|
|
|
|
|
https://arxiv.org/pdf/2103.13263[
|
|
|
|
|
Metronome -- adaptive and precise intermittent packet retrieval in DPDK
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260725162634/https://arxiv.org/pdf/2103.13263
|
|
|
|
|
(section II, discussion of nanosleep limits), and a practical encounter
|
|
|
|
|
with the same default, including the `/proc` interface, is documented in
|
|
|
|
|
https://bugs.openjdk.org/browse/JDK-8307766[
|
|
|
|
|
OpenJDK issue JDK-8307766
|
|
|
|
|
].
|
|
|
|
|
//Archive https://web.archive.org/web/20251130011408/https://bugs.openjdk.org/browse/JDK-8307766
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- *Single 32-bit Sub-Channel DDR5 DIMMs*, arXiv:2605.08725 --
|
|
|
|
|
https://arxiv.org/pdf/2605.08725 +
|
|
|
|
|
[[cxl]]
|
|
|
|
|
--
|
|
|
|
|
cxl::
|
|
|
|
|
arXiv:2503.22017 --
|
|
|
|
|
https://arxiv.org/pdf/2503.22017[
|
|
|
|
|
Performance Characterizations and Usage Guidelines of Samsung CXL Memory
|
|
|
|
|
Module Hybrid Prototype
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260811083139/https://arxiv.org/pdf/2503.22017
|
|
|
|
|
+
|
|
|
|
|
Table III contains a well-documented measurement of plain local DDR5:
|
|
|
|
|
median latency 122.9{nbsp}ns under Intel MLC pointer chasing versus
|
|
|
|
|
13.1{nbsp}ns for a plain load; methodology (rdtsc, 10 000 repetitions,
|
|
|
|
|
median) described in section III-C.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
[[ddr5b]]
|
|
|
|
|
--
|
|
|
|
|
ddr5b::
|
|
|
|
|
arXiv:2605.08725 --
|
|
|
|
|
https://arxiv.org/pdf/2605.08725[
|
|
|
|
|
Single 32-bit Sub-Channel DDR5 DIMMs
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260607161507/https://arxiv.org/pdf/2605.08725
|
|
|
|
|
+
|
|
|
|
|
Useful for the point that DDR5 did not improve absolute latency over DDR4.
|
|
|
|
|
+
|
|
|
|
|
The relevant passage is in section "Latency Implications", noting that both
|
|
|
|
|
DDR4 and DDR5 sit at approximately 14{nbsp}ns first-access latency at their
|
|
|
|
|
respective standard speeds.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- simplyblock glossary, *NVMe Latency* --
|
|
|
|
|
https://simplyblock.io/glossary/nvme-latency/ +
|
|
|
|
|
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy.
|
|
|
|
|
[[simply]]
|
|
|
|
|
--
|
|
|
|
|
simply::
|
|
|
|
|
simplyblock glossary --
|
|
|
|
|
https://simplyblock.io/glossary/nvme-latency/[
|
|
|
|
|
NVMe Latency
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260418070043/https://simplyblock.io/glossary/nvme-latency/
|
|
|
|
|
+
|
|
|
|
|
Compact summary of the NVMe / SATA SSD / rotating disk hierarchy: a modern
|
|
|
|
|
NVMe SSD serves a random 4K read in roughly 20--70{nbsp}µs, a SATA SSD in
|
|
|
|
|
100--200{nbsp}µs, and a spinning disk in 5--10{nbsp}ms.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
- AnandTech, *Samsung 983 ZET (Z-NAND) review*, page 3 --
|
|
|
|
|
https://at-web1.www.anandtech.com/show/13951/the-samsung-983-zet-znand-ssd-review/3 +
|
|
|
|
|
Explains why queue depth 1 exposes best-case access latency.
|
|
|
|
|
[[intel]]
|
|
|
|
|
--
|
|
|
|
|
intel::
|
|
|
|
|
Intel --
|
|
|
|
|
https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf[
|
|
|
|
|
Optane SSD DC P4800X Evaluation Guide
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260824160928/https://cdrdv2-public.intel.com/840898/Windows_Intel_Optane_SSD_DC_P4800X_Eval-Guide_336586.pdf
|
|
|
|
|
+
|
|
|
|
|
In Chapter 1.1, Page 5: ``at queue depth of one, 99 out of 100 read operations ...
|
|
|
|
|
complete in single digit microseconds.''
|
|
|
|
|
+
|
|
|
|
|
Corroborated independently by
|
|
|
|
|
https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html[
|
|
|
|
|
Tom's Hardware's 2017 preview,
|
|
|
|
|
]
|
|
|
|
|
//Archive https://web.archive.org/web/20260313082242/https://www.tomshardware.com/reviews/intel-optane-3d-xpoint-p4800x,5030.html
|
|
|
|
|
which reports 10 µs latency for the same drive.
|
|
|
|
|
+
|
|
|
|
|
Note: Optane was discontinued by Intel in 2022; the figure is retained as the
|
|
|
|
|
best-documented physical floor for this latency class.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A Note on Microbenchmarks
|
|
|
|
|
-------------------------
|
|
|
|
|
|
|
|
|
|
One caveat deserves to be stated separately, because it explains why published
|
|
|
|
|
figures for synchronisation primitives vary so wildly. A hot microbenchmark
|
|
|
|
|
keeps every thread runnable and every core out of idle by construction. Under
|
|
|
|
|
those conditions an uncontended atomic operation or a mutex acquisition costs
|
|
|
|
|
single-digit to low-three-digit nanoseconds, and a benchmark will faithfully
|
|
|
|
|
report that. What such a benchmark structurally cannot observe is the cost of
|
|
|
|
|
_blocking_ -- of a thread actually parking and later being woken -- because
|
|
|
|
|
pressing the benchmark is precisely what prevents that from happening.
|
|
|
|
|
One caveat deserves a separate mention, as it explains why the published
|
|
|
|
|
figures on synchronisation primitives differ so drastically from one another.
|
|
|
|
|
A hot microbenchmark keeps every thread runnable and every core out of idle by
|
|
|
|
|
its very construction. Under those conditions an uncontended atomic operation or
|
|
|
|
|
a Mutex acquisition costs single-digit to low-three-digit nanoseconds, and a
|
|
|
|
|
benchmark will faithfully report that. What such a benchmark _structurally_
|
|
|
|
|
cannot observe is the cost of _blocking_ -- of a thread actually parking and
|
|
|
|
|
later being woken -- because pressing the benchmark is precisely what
|
|
|
|
|
prevents that from happening.
|
|
|
|
|
|
|
|
|
|
The instruction-level cost of a synchronisation primitive is what a hot
|
|
|
|
|
microbenchmark measures. Its systemic cost -- coherence traffic under
|
|
|
|
|
contention, and blocking and wake-up latency when it is not contended -- is
|
|
|
|
|
what production tail latency measures. The two can differ by two to four orders
|
|
|
|
|
of magnitude, and the table above spans both regimes for exactly that reason.
|
|
|
|
|
contention, and blocking and wake-up latency when it is not contended --
|
|
|
|
|
is what production tail latency measures. The two values can differ by
|
|
|
|
|
two to four orders of magnitude, and it is for this very reason that
|
|
|
|
|
the table above covers both regimes.
|
|
|
|
|
|
