Textentwurf für den Call-of-Papers
Mit dem etablierten Schema war das nicht schwer... Fehlen nur noch ein paar technische Details
This commit is contained in:
parent
0af1b5c355
commit
df04350e33
2 changed files with 199 additions and 20 deletions
|
|
@ -4,28 +4,55 @@ Antrag für einen Vortrag auf der FrOSCon 26
|
|||
:date: 23.5.2026
|
||||
|
||||
|
||||
TODO -- Title TODO
|
||||
------------------
|
||||
.Untertitel
|
||||
In diesem Vortrag geht es nicht um abgefahrene C++ Features, sondern um eine Darstellung praktischer Herausforderungen, wenn eine große und komplexe Codebasis die Möglichkeiten der Sprache nutzt und sich gleichzeitig mit der Sprache weiterentwickelt. Ich werde dazu drei konkrete Fälle herausgreifen, jeweils mit leicht abstrahierten vorher / nachher Code-Beispielen. Die C++-Features werden soweit erläutert, wie es zum Verständnis des Themas notwendig ist.
|
||||
Reading Video data with IO_URING
|
||||
--------------------------------
|
||||
.putting a new kernel framework into practical use
|
||||
In diesem Vortrag geht es um eine technische Herausforderung, die wir für unser
|
||||
OpenSource-Projekt lösen mussten: eine große Menge Video-Daten effizient einlesen.
|
||||
Vor diesem Hintergrund möchte ich das neue IO_URING Kernel-Interface erläutern
|
||||
und mit vereinfachten Code-Beispielen in C++ zeigen, wie es in eine Applikation
|
||||
eingebunden werden kann. Rudimentäre C++ Kenntnisse sollten genügen, um dem
|
||||
Vortrag zu folgen.
|
||||
|
||||
Für die eigentliche Darstellung rechne ich mit 40 Minuten.
|
||||
|
||||
|
||||
Kurzfassung
|
||||
~~~~~~~~~~~
|
||||
IO_URING is a fairly new IO interface in the Linux kernel.
|
||||
It marks a paradigm shift in how the kernel is interacted with.
|
||||
Instead of passing control thorough an interface function to enter into kernel mode,
|
||||
messages are passed asynchronously.
|
||||
|
||||
<i>Blah</i>.
|
||||
In this talk, patterns for dealing with asynchronous IO are discussed and a setup to handle
|
||||
high volume file reading using this framework is demonstrated with code examples in C++.
|
||||
|
||||
<i>Lumiera</i> is a video editing application under development in C++ -- and is used here to
|
||||
illustrate that asynchronous I/O can play a key role in achieving high throughput for video processing.
|
||||
|
||||
|
||||
Langfassung
|
||||
~~~~~~~~~~~
|
||||
The usual simple way of reading data from a file is to just invoke a `read()` function, directly
|
||||
getting the data in response. This function is accessible as part of the programming language's
|
||||
standard library, yet in the end it maps down onto a function of the OS kernel. And while this
|
||||
form of reading data from a file is easy to handle, in terms of program logic, it has the downside
|
||||
that the current thread is blocked, for an unknown period of time, until the data is ready.
|
||||
|
||||
<i>Blubb</i>.
|
||||
Blocking is particularly problematic in high-performance computing, as it means losing control
|
||||
over the timings and leaving resources unused. Various interfaces and frameworks have been
|
||||
developed over time to improve this, both for network communication and for file operations.
|
||||
The Linux kernel system call interface IO_URING, adopted in kernel version 5.1 (2019) and
|
||||
gradually expanded since then, takes these efforts one step further and provides an uniform
|
||||
framework for various kinds of IO operations: The client code instructs the kernel to perform
|
||||
some IO related task by sending it a message, and the kernel returns the completed work items
|
||||
via a different queue.
|
||||
|
||||
Beispiele hierfür sind:
|
||||
<ul><li>Muh</li>
|
||||
<li>Meh</li>
|
||||
</ul>
|
||||
Jaleck.
|
||||
In theory, it is possible to achieve an extreme throughput or very low latency; yet in practice,
|
||||
it proves quite challenging to structure the client code accordingly to achieve these goals.
|
||||
In our use case, related to rendering video frames based on the arrangement made by the film
|
||||
editor, expensive computations need to be chained-up behind the IO operations, all involving
|
||||
a massive amount of data. The talk will discuss some common solution patterns, and the approach
|
||||
taken by the Lumiera project to cope with this situation. A simplified demo level setup will
|
||||
then be explained with code examples in C++
|
||||
|
||||
|
|
|
|||
|
|
@ -43,7 +43,6 @@
|
|||
<edge COLOR="#7b8885"/>
|
||||
<font NAME="SansSerif" SIZE="16"/>
|
||||
<node CREATED="1519424889703" ID="ID_1477284495" MODIFIED="1722708303049" TEXT="Grundsätzlich">
|
||||
<linktarget COLOR="#1738b9" DESTINATION="ID_309609059" ENDARROW="Default" ENDINCLINATION="-420;71;" ID="Arrow_ID_858985657" SOURCE="ID_1394954330" STARTARROW="None" STARTINCLINATION="-388;31;"/>
|
||||
<icon BUILTIN="yes"/>
|
||||
<node CREATED="1519424938729" ID="ID_688989453" MODIFIED="1715874616706" TEXT="»Talk« als Vehikel"/>
|
||||
<node CREATED="1519425448059" ID="ID_344656292" MODIFIED="1519425453814" TEXT="Praxisbezug">
|
||||
|
|
@ -202,8 +201,7 @@
|
|||
neues <b>Git-Repo</b> nur für den Demo-Code anlegen
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</html></richcontent>
|
||||
<icon BUILTIN="hourglass"/>
|
||||
</node>
|
||||
</node>
|
||||
|
|
@ -263,7 +261,6 @@
|
|||
</node>
|
||||
</node>
|
||||
<node CREATED="1520540782924" ID="ID_1394954330" MODIFIED="1779388492717" TEXT="was wir wollen und wo wir stehen">
|
||||
<arrowlink COLOR="#1738b9" DESTINATION="ID_309609059" ENDARROW="Default" ENDINCLINATION="-420;71;" ID="Arrow_ID_858985657" STARTARROW="None" STARTINCLINATION="-388;31;"/>
|
||||
<icon BUILTIN="yes"/>
|
||||
<node CREATED="1753133245990" ID="ID_1737109820" MODIFIED="1753133263062" TEXT="einen plausiblen Anlaß, um einen Vortrag mit Lumiera-Bezug zu halten"/>
|
||||
<node CREATED="1753133263959" ID="ID_1208154525" MODIFIED="1753133281172" TEXT="Kompetenz in »hard-core«-Themen demonstrieren"/>
|
||||
|
|
@ -305,12 +302,162 @@
|
|||
</node>
|
||||
<node BACKGROUND_COLOR="#fdfdcf" COLOR="#ff0000" CREATED="1753885887109" ID="ID_1991141897" MODIFIED="1779390904717" TEXT="Recherche zu Themenbereichen">
|
||||
<icon BUILTIN="flag-pink"/>
|
||||
<node CREATED="1753885928239" ID="ID_1844778340" MODIFIED="1753885930457" TEXT="Pixelformate"/>
|
||||
<node CREATED="1753885928239" ID="ID_1844778340" MODIFIED="1779396286307" TEXT="io_uring">
|
||||
<node CREATED="1779396294447" ID="ID_1848792339" MODIFIED="1779396305267" TEXT="Quellen">
|
||||
<node CREATED="1779396306814" ID="ID_1325281102" LINK="https://en.wikipedia.org/wiki/Io_uring" MODIFIED="1779396315914" TEXT="Wikipedia">
|
||||
<node CREATED="1779397158957" ID="ID_635788695" LINK="https://web.archive.org/web/20251010072644/https://kernel.dk/io_uring.pdf" MODIFIED="1779397176550" TEXT="verlinkt auf Einführungs-Artikel: »Efficient IO with io_uring«"/>
|
||||
<node CREATED="1779397215245" ID="ID_533486044" LINK="https://lwn.net/Articles/810414/" MODIFIED="1779397234502" TEXT="LWN: rapid growth of io_uring (2020)"/>
|
||||
</node>
|
||||
<node CREATED="1779399048372" ID="ID_1376049416" LINK="https://unixism.net/loti/" MODIFIED="1779399068596" TEXT="»Lord of the uring«"/>
|
||||
</node>
|
||||
<node CREATED="1779396319748" ID="ID_652484743" MODIFIED="1779396323865" TEXT="Basisinfos">
|
||||
<node CREATED="1779396332330" ID="ID_1277198826" MODIFIED="1779396361916" TEXT="zwei Ringbuffer ("queue rings" ⟶ daher der Name)"/>
|
||||
<node CREATED="1779396785527" ID="ID_1581052613" MODIFIED="1779396800974" TEXT="eingeführt 2019 mit Linux 5.1"/>
|
||||
<node CREATED="1779396508382" ID="ID_1790185181" MODIFIED="1779396544764" TEXT="soll den Rückstand von Linux für async-IO beheben">
|
||||
<node CREATED="1779398726939" ID="ID_64708046" MODIFIED="1779398814014" TEXT="die de-facto-Lösung ist EPOLL">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
man setzt einen Socket auf "asynchron". Durch cleveres Polling bekommt man mit, wann die IO-Operation »ready« wird. <b>Aber</b> das ist readyness-semantics, not completion-semantics. Und es ist sehr limitiert, und setzt eine komplexe State-Machine im Userland vorraus
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</node>
|
||||
<node CREATED="1779396620988" ID="ID_769607449" MODIFIED="1779396634370" TEXT="es gab seit Linux 2.5 bereits async_io"/>
|
||||
<node CREATED="1779396635421" ID="ID_1823781286" MODIFIED="1779396769771" TEXT="das ist aber auf Spezialfälle limitiert und schwierig zu nutzen">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
Arbeitet in einem speziellen Modus, IO_DIRECT
|
||||
</p>
|
||||
<p>
|
||||
Kann nur bereits allozierte/ geöffnete Files bearbeiten
|
||||
</p>
|
||||
<p>
|
||||
kann nicht mit generischen Sockets arbeiten ⟶ auch kein Networking
|
||||
</p>
|
||||
<p>
|
||||
geht am Page Cache vorbei
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</node>
|
||||
<node CREATED="1779396658091" ID="ID_1042616589" MODIFIED="1779396675648" TEXT="Windows kann da seit den Nuller-Jahren wesentlich mehr">
|
||||
<icon BUILTIN="smily_bad"/>
|
||||
</node>
|
||||
<node CREATED="1779398012338" ID="ID_157830257" MODIFIED="1779398079260" TEXT="Hardware-Entwicklung hat eine Rolle gespielt">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
Aus einem Kommentar auf LWN
|
||||
</p>
|
||||
<p>
|
||||
https://lwn.net/Articles/810898/
|
||||
</p>
|
||||
<p http-equiv="content-type" content="text/html; charset=utf-8">
|
||||
What it says about the state of hardware is easy.<br />
|
||||
</p>
|
||||
<p>
|
||||
Meltdown mitigations made context switching more expensive, and SSDs made the actual I/O cheaper. At some point, you're spending almost as much time context switching as you are doing actual work, and there's nothing Linux can do about it because it's all tied up in the architecture of the MMU and the processor's cache. Thus, it now makes sense to design the interface around reducing the number of context switches at all cost, instead of just assuming that the cost of doing the actual I/O will dominate.
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</node>
|
||||
</node>
|
||||
<node CREATED="1779396548708" ID="ID_1822858626" MODIFIED="1779396599007" TEXT="kann mit dBPF zusammenarbeiten">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
extended Berkeley Package Filter: ein Framework, das Code als Reaktion auf Events direkt im Kernel-Kontext ausführt
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</node>
|
||||
<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1779396822977" ID="ID_211122289" MODIFIED="1779397158118" TEXT="bedingt durch die komplett neue Struktur ⟹ massive Sicherheitsprobleme">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
Es wurde alsbald erheblich als Einfallstor für Exploits genutzt, da das bestehende Security-Tooling diese neuen Mechanismen überhaupt nicht »auf dem Schirm« hat. Im Jahr 2023 hat das Google-Security-Team 60% aller Linux-Exploits dem IO_URING subsystem zugeordnet. Daher ist es auf Android und ChromeOS und unter Docker in der Standard-Konfiguration deaktiviert (und hat einen schlechten Ruf unter den »Oldtimern«)
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
<icon BUILTIN="messagebox_warning"/>
|
||||
</node>
|
||||
<node CREATED="1779397402228" ID="ID_1111670287" LINK="https://github.com/axboe/liburing" MODIFIED="1779397445634" TEXT="die liburing erleichtert den Umgang mit dem anspruchsvollen API">
|
||||
<icon BUILTIN="idea"/>
|
||||
</node>
|
||||
<node BACKGROUND_COLOR="#accdc3" COLOR="#3f4015" CREATED="1779398825661" ID="ID_1355173640" MODIFIED="1779398859668" STYLE="fork" TEXT="Chat-GPT meint: das wird die zukünftige default-Lösung">
|
||||
<edge COLOR="#808080" STYLE="bezier" WIDTH="thin"/>
|
||||
<icon BUILTIN="wizard"/>
|
||||
<node CREATED="1779398860929" ID="ID_676326792" MODIFIED="1779398870836" TEXT="aber Vorsicht, mein Promt war subtil biassed"/>
|
||||
<node CREATED="1779398873080" ID="ID_1322619305" MODIFIED="1779398910066" TEXT="jedenfalls meint Chat-GPT...">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<ul>
|
||||
<li>
|
||||
adoption is rising
|
||||
</li>
|
||||
<li>
|
||||
io_uring is not going away anywhere soon
|
||||
</li>
|
||||
</ul>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</node>
|
||||
<node CREATED="1779398911786" ID="ID_200652684" MODIFIED="1779398922097" TEXT="nennt nur AIO und EPOLL als relevante Alternativen"/>
|
||||
<node COLOR="#338800" CREATED="1779398922776" ID="ID_614335681" MODIFIED="1779398955570" TEXT="das genügt mir">
|
||||
<richcontent TYPE="NOTE"><html>
|
||||
<head>
|
||||
|
||||
</head>
|
||||
<body>
|
||||
<p>
|
||||
...denn das bedeutet, da ist nichts »mainstreamiges« unterwegs, von dem ich noch nichts wüßte.
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
<icon BUILTIN="button_ok"/>
|
||||
</node>
|
||||
</node>
|
||||
</node>
|
||||
</node>
|
||||
<node CREATED="1753885953712" ID="ID_1705360472" MODIFIED="1779390897547" TEXT="Architektur: Linux"/>
|
||||
<node CREATED="1753885985959" ID="ID_1416320830" MODIFIED="1753886099702" TEXT="VBlank-Sync"/>
|
||||
</node>
|
||||
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1519001955679" ID="ID_65625223" MODIFIED="1779386731937" TEXT="Demo-Inhalte">
|
||||
<icon BUILTIN="flag-yellow"/>
|
||||
<node CREATED="1779504843301" ID="ID_1215238444" MODIFIED="1779504961470" TEXT="Idee">
|
||||
<icon BUILTIN="idea"/>
|
||||
<node CREATED="1779504852591" ID="ID_628756969" MODIFIED="1779504956035" TEXT="ein Raw-Video-File framewise einlesen">
|
||||
<linktarget COLOR="#d60714" DESTINATION="ID_628756969" ENDARROW="Default" ENDINCLINATION="497;-40;" ID="Arrow_ID_347382746" SOURCE="ID_649621958" STARTARROW="None" STARTINCLINATION="347;-14;"/>
|
||||
</node>
|
||||
<node CREATED="1779504863066" ID="ID_986008832" MODIFIED="1779504921775" TEXT="jeden Frame in den Output-Code aus dem Video-output-Vortrag schicken"/>
|
||||
</node>
|
||||
</node>
|
||||
</node>
|
||||
<node COLOR="#435e98" CREATED="1519000413497" HGAP="-43" ID="ID_1743786269" MODIFIED="1715873905102" POSITION="left" VSHIFT="108">
|
||||
|
|
@ -385,14 +532,19 @@
|
|||
-----END PGP MESSAGE-----
|
||||
</p>
|
||||
</body>
|
||||
</html>
|
||||
</richcontent>
|
||||
</html></richcontent>
|
||||
<icon BUILTIN="ksmiletris"/>
|
||||
</node>
|
||||
</node>
|
||||
</node>
|
||||
<node BACKGROUND_COLOR="#eee5c3" COLOR="#990000" CREATED="1519329836328" ID="ID_214079495" MODIFIED="1779386001215" TEXT="formulieren">
|
||||
<icon BUILTIN="flag-yellow"/>
|
||||
<node BACKGROUND_COLOR="#eef0c5" COLOR="#990000" CREATED="1519329836328" ID="ID_214079495" MODIFIED="1779504752558" TEXT="formulieren">
|
||||
<icon BUILTIN="pencil"/>
|
||||
<node CREATED="1779504756681" ID="ID_466048202" MODIFIED="1779504767437" TEXT="halte mich an das Schema aus den letzten Jahren"/>
|
||||
<node CREATED="1779504768195" ID="ID_531662550" MODIFIED="1779504785771" TEXT="Kurzform und Langform ergeben auch zusammen einen Text"/>
|
||||
<node BACKGROUND_COLOR="#f0d5c5" COLOR="#990033" CREATED="1779504789243" ID="ID_649621958" MODIFIED="1779504956035" TEXT="Klären: kann man mit IO_URING gezielt bestimmte Bereiche eines Files lesen?">
|
||||
<arrowlink COLOR="#d60714" DESTINATION="ID_628756969" ENDARROW="Default" ENDINCLINATION="497;-40;" ID="Arrow_ID_347382746" STARTARROW="None" STARTINCLINATION="347;-14;"/>
|
||||
<icon BUILTIN="help"/>
|
||||
</node>
|
||||
</node>
|
||||
</node>
|
||||
<node CREATED="1519001760129" ID="ID_1268047369" MODIFIED="1519001762637" TEXT="Konferenz">
|
||||
|
|
|
|||
Loading…
Reference in a new issue