ichthyo/LUMIERA.clone
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
LUMIERA.clone/doc/devel/rfc/DataBackend.txt

117 lines
4.3 KiB
Text
Raw Normal View History

Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
Design Process : Data Backend
=============================
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
[options="autowidth"]
|====================================
clean-up: re-classify essential RfCs The RfC documents were written to complement discussions of the Lumiera developers; yet since the time where ''Ichthyo'' is working basically alone on the project, this kind of discussions have ceased. During the following years, some ideas promoted by the existing RfC documents became rather detached from the actual state of development in the code base. Many of the existing RfC documents require some commentary to place them into context, and some of the decisions taken in the early stage of the project should be **re-assessed**. This includes the decision to reject some proposals, which initially might have seemed desirable, yet could not be reconciled with the understanding of the matter and topic in question, as was gained through the ongoing analysis and development.
2025-10-05 02:02:22 +02:00
|*State* | _Dropped_
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
|*Date* | _2007-06-04_
|*Proposed by* | ct
|====================================
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
DataBackend
-----------
Describe the DataBackend functionality.
Description
~~~~~~~~~~~
This just starts as braindump, I will refine it soon:
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
- handle all files lumiera uses at runtime (media, edl, temp data)
- manage filehandles, lumiera might use more more files than available filehandles
- manage temporary data
- do caching
- IO will be blocked where the backend tells the core where it can expect the data (not `read()`/`write()` like)
- kindof garbage collector
- do prefetching
- no/low latency for the core the prefetcher and other things ensure that data is available in time
- translate any input into a format which the lumiera core understands (demux, decode)
- same for encoding to output formats
- offer a plugin API for encoders/decoders
- maybe network backend for serving data to distributed rendernodes
- can do some load control or management (trigger adaptive rendering if system is idle etc)
- pull based architecture
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
.Notes:
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
* ichthyo wrote also some ideas regarding
link:http://localhost:8888/project/background/history/PossibilitiesAtHand.html[things possible in the middle layer]...
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
clean-up: re-classify essential RfCs The RfC documents were written to complement discussions of the Lumiera developers; yet since the time where ''Ichthyo'' is working basically alone on the project, this kind of discussions have ceased. During the following years, some ideas promoted by the existing RfC documents became rather detached from the actual state of development in the code base. Many of the existing RfC documents require some commentary to place them into context, and some of the decisions taken in the early stage of the project should be **re-assessed**. This includes the decision to reject some proposals, which initially might have seemed desirable, yet could not be reconciled with the understanding of the matter and topic in question, as was gained through the ongoing analysis and development.
2025-10-05 02:02:22 +02:00
Conclusion: Dropped
-------------------
DOC: update technical docs to reflect recent development At various places, concepts and drafts from the early stage of the Lumiera Project are still reflected in the online documentation pages. During the last months, development focussed on the Render Engine, causing a shift in some parts of the design, and obsoleting other parts altogether (notably we consider to use IO_URING for async IO)
2023-10-25 00:02:08 +02:00
The underlying principles remain valid, yet development took another
direction during the last years. Special frameworks for high-performance
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
asynchronous IO will be used for dedicated use cases. A garbage collector
is not necessary, since Lumiera relies on deterministic memory management.
Furthermore, decoding, transcoding and encoding is handled as part of the
pipeline and not through a separate service.
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
DOC: update technical docs to reflect recent development At various places, concepts and drafts from the early stage of the Lumiera Project are still reflected in the online documentation pages. During the last months, development focussed on the Render Engine, causing a shift in some parts of the design, and obsoleting other parts altogether (notably we consider to use IO_URING for async IO)
2023-10-25 00:02:08 +02:00
Ichthyostega:: '2023-10-24' ~<prg@ichthyostega.de>~
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
Comments
--------
Sounds fairly complete to me
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
Ichthyostega:: '2007-06-16T23:19:44Z'
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
Developement takes place in the repo now
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
ct:: '2007-06-27T16:14:56Z'
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
clean-up: re-classify essential RfCs The RfC documents were written to complement discussions of the Lumiera developers; yet since the time where ''Ichthyo'' is working basically alone on the project, this kind of discussions have ceased. During the following years, some ideas promoted by the existing RfC documents became rather detached from the actual state of development in the code base. Many of the existing RfC documents require some commentary to place them into context, and some of the decisions taken in the early stage of the project should be **re-assessed**. This includes the decision to reject some proposals, which initially might have seemed desirable, yet could not be reconciled with the understanding of the matter and topic in question, as was gained through the ongoing analysis and development.
2025-10-05 02:02:22 +02:00
.State -> Parked
DOC: update technical docs to reflect recent development At various places, concepts and drafts from the early stage of the Lumiera Project are still reflected in the online documentation pages. During the last months, development focussed on the Render Engine, causing a shift in some parts of the design, and obsoleting other parts altogether (notably we consider to use IO_URING for async IO)
2023-10-25 00:02:08 +02:00
Development took another direction over the last years;
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
the former »Backend« layer is called »Vault« now and structured differently.
Ichthyostega:: '2023-10-24T22:45:55Z'
DOC: update technical docs to reflect recent development At various places, concepts and drafts from the early stage of the Lumiera Project are still reflected in the online documentation pages. During the last months, development focussed on the Render Engine, causing a shift in some parts of the design, and obsoleting other parts altogether (notably we consider to use IO_URING for async IO)
2023-10-25 00:02:08 +02:00
clean-up: re-classify essential RfCs The RfC documents were written to complement discussions of the Lumiera developers; yet since the time where ''Ichthyo'' is working basically alone on the project, this kind of discussions have ceased. During the following years, some ideas promoted by the existing RfC documents became rather detached from the actual state of development in the code base. Many of the existing RfC documents require some commentary to place them into context, and some of the decisions taken in the early stage of the project should be **re-assessed**. This includes the decision to reject some proposals, which initially might have seemed desirable, yet could not be reconciled with the understanding of the matter and topic in question, as was gained through the ongoing analysis and development.
2025-10-05 02:02:22 +02:00
.State -> Dropped
The approach to most of the topics listed in this RfC has shifted, and thus
it is time to draw the conclusion that overall Lumiera will not use a »Backend«
as an _adaptation layer,_ in the way envisioned here. Notably the idea of
centralising _any access to resources_ to apply some kind of instrumentation
and regulation is rejected. Rather, the »Vault« (as it is called now),
provides specialised low-level services with deep vertical integration;
this implies rather to establish dedicated control circuits for a small
number of special resources deemed performance critical.
Most topics touched by this RfC however remain relevant,
yet will be approached differently...
- while non-blocking, asynchronous I/O is still considered essential,
it is acknowledged that getting a grip at the resource usage of
external libraries and frameworks, integrated via plug-in, is a
non-trivial task, and must be approached at a case-by-case base.
- the Scheduler provides a much more high-level and specialised
execution service then intended initially; the majority of
resource usage regulation will be achieved by adjusting the
job generation, which is connected to high-level structures,
while relying on generic resource observation facilities
in the Vault.
- the Cache service will require a deep integration with the Builder
and the media / asset management; _cache invalidation_ will be
based on a _naming and identification scheme,_ which requires
explicit support by the high-level metadata management.
- the Vault has no connection to media processing whatsoever,
it does not decode / encode data or mark and map parts of
media streams; this topic is coordinated at the level of
the builder and delegated to external facilities, while
the Vault provides some dedicated resource management
(notably handling GPU access and shader code)
- file handles and generic file access are considered a non-issue,
as is the occasional usage of multithreading -- as long as any
substantial parallelism is organised through the scheduler.
Ichthyostega:: '2025-09-20'
Import old DesignProcess into rfc_pending
2010-07-23 18:24:30 +02:00
''''
clean-up: comb through the historical pages to fix markup errors Some sections of the Lumiera website document meeting minutes, discussion protocols and design proposals from the early days of the project; these pages were initially authored in the »Moin Moin Wiki« operated by Cehteh on pipapo.org at that time; this wiki backed the first publications of the »Cinelerra-3« initiative, which turned into the Lumiera project eventually. Some years later, those pages were transliterated into Asciidoc semi-automatically, resulting in a lot of broken markup and links. This is a long standing maintenance problem problem plaguing the Lumiera website, since those breakages cause a lot of warnings and flood the logs of any linkchecker run.
2025-09-08 04:04:39 +02:00
Back to link:/x/DesignProcess.html[Lumiera Design Process overview]
Reference in a new issue Copy permalink