MERGE: prepare for upgrade and release
With the ability to invoke a Render Node graph, the development on branch `play` reached some kind of milestone regarding the »Playback Vertical Slice«. This is a good opportunity to update the reference platform and upgrade the preview releases and packaging setup accordingly. This will include adjustments to compile on recent compilers and upgrade the build system to support Python-3.
This commit is contained in:
commit
cc9a1e410a
26 changed files with 1492 additions and 901 deletions
|
|
@ -278,6 +278,11 @@ function edit()
|
|||
EDITOR="${EDITOR:-$(git config --get core.editor)}"
|
||||
EDITOR="${EDITOR:-$VISUAL}"
|
||||
|
||||
if [ -z "$EDITOR" ]; then
|
||||
echo -e "\nFATAL\n\$EDITOR undefined\n\n"
|
||||
exit -1
|
||||
fi
|
||||
|
||||
local file="$1"
|
||||
local line=0
|
||||
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ title = Lumiera
|
|||
version = 0.pre.03
|
||||
website = http://www.lumiera.org
|
||||
authors = Joel Holdsworth|Christian Thäter|Hermann Voßeler|Stefan Kangas|Michael Fisher|Michael Ploujnikov|and others...
|
||||
copyright = 2007 - 2015
|
||||
copyright = 2007 - 2025
|
||||
|
||||
[Gui]
|
||||
#stylesheet = lumiera.css
|
||||
|
|
|
|||
|
|
@ -1,25 +0,0 @@
|
|||
The Scheduler
|
||||
-------------
|
||||
:Author: CehTeh
|
||||
:Date: 6/2007
|
||||
|
||||
//MENU: label Scheduler
|
||||
|
||||
Scheduling is done with two priority queues, one for high priority jobs and one for low priority jobs.
|
||||
These priority queues are ordered by absolute time values plus some job specific identified.
|
||||
|
||||
There are following (non exhaustive) kinds of jobs:
|
||||
|
||||
* started job
|
||||
* job to be canceled
|
||||
* unscheduled job
|
||||
* dependency providing jobs
|
||||
|
||||
Jobs implement a kind of future. We try hard to avoid any blocking waits.
|
||||
The Job scheduler runs singlethreaded. Its only task is to schedule and delegate jobs to worker threads,
|
||||
by itself it will never do any extensive processing.
|
||||
|
||||
Each job has an pre configured behaviour for the case of failure or deadline miss.
|
||||
Any canceling and expireing jobs gets noted in *Statistics* to adjust performance and timings
|
||||
for optimal performance and I/O throughput.
|
||||
|
||||
|
|
@ -9,106 +9,8 @@ data access. Within Lumiera, there are two main kinds of data handling:
|
|||
* The Session and the object models manipulated through the GUI are kept in memory.
|
||||
They are backed by a _storage backend,_ which provides database-like storage and
|
||||
especially logging, replaying and ``Undo'' of all ongoing modifications..
|
||||
* Media data is handled _frame wise_ -- as described below.
|
||||
|
||||
The vault layer (``backend'') uses *memory mapping* to make data available to the program.
|
||||
This is somewhat different to the more common open/read/write/close file access,
|
||||
while giving superior performance and much better memory utilization.
|
||||
The Vault-Layer must be able to handle more data than will fit into the memory
|
||||
or even address space on 32 bit architectures. Moreover, a project might access more files
|
||||
than the OS can keep open simultaneously, thus the for _Files used by the Vault,_ it needs a
|
||||
*FilehandleCache* to manage file handle dynamically.
|
||||
|
||||
Which parts of a file are actually mapped to physical RAM is managed by the kernel;
|
||||
it keeps a *FileMapCache* to manage the *FileMaps* we've set up.
|
||||
In the End, the application itself only requests *Data Frames* from the Vault.
|
||||
|
||||
To minimize latency and optimize CPU utilization we have a *Prefetch thread* which operates
|
||||
a *Scheduler* to render and cache frames which are _expected to be consumed soon_. The intention
|
||||
is to manage the rendering _just in time_.
|
||||
|
||||
The prefetcher keeps *Statistics* for optimizing performance.
|
||||
|
||||
|
||||
Accessing Files
|
||||
---------------
|
||||
|
||||
+FileDescriptor+ is the superclass of all possible filetypes, it has a weak reference to a
|
||||
+FileHandle+ which is managed in within the +FilehandleCache+. On creation, only the existence
|
||||
(when reading) or access for write for new files are checked. The +FileDescriptor+ stores some
|
||||
generic metadata about the underlying file and intended use. But the actual opening is done on demand.
|
||||
|
||||
The _content of files is memory mapped_ into the process address space.
|
||||
This is managed by +FileMap+ entries and a +FileMapCache+.
|
||||
|
||||
File Handles
|
||||
~~~~~~~~~~~~
|
||||
A +FilehandleCache+ serves to store a finite maximum number of +FileHandles+ as a MRU list.
|
||||
FileHandles are managed by the +FilehandleCache+; basically they are just storing the underlying OS file
|
||||
handles and managed in a lazy/weak way, (re)opened when needed and aging in the cache when not needed,
|
||||
since the amount of open file handles is limited aged ones will be closed and reused when the system
|
||||
needs to open another file.
|
||||
|
||||
File Mapping
|
||||
~~~~~~~~~~~~
|
||||
The +FileMapCache+ keeps a list of +FileMaps+, which are currently not in use and subject of aging.
|
||||
Each +FileMap+ object contains many +Frames+. The actual layout depends on the type of the File.
|
||||
Mappings need to be _page aligned_ while Frames can be anywhere within a file and dynamically sized.
|
||||
|
||||
All established ++FileMap++s are managed together in a central +FileMapCache+.
|
||||
Actually, +FileMap+ objects are transparent to the application. The upper layers will just
|
||||
request Frames by position and size. Thus, the +File+ entities associate a filename with the underlying
|
||||
low level File Descriptor and access
|
||||
|
||||
Frames
|
||||
~~~~~~
|
||||
+Frames+ are the smallest datablocks handled by the Vault. The application tells the Vault Layer to make
|
||||
Files available and from then on just requests Frames. Actually, those Frames are (references to) blocks
|
||||
of continuous memory. They can be anything depending on the usage of the File (Video frames, encoder frames,
|
||||
blocks of sound samples). Frames are referenced by a smart-pointer like object which manages the lifetime
|
||||
and caching behavior.
|
||||
|
||||
Each frame referece can be in one out of three states:
|
||||
|
||||
readonly::
|
||||
the backing +FileMap+ is checked out from the aging list, frames can be read
|
||||
|
||||
readwrite::
|
||||
the backing +FileMap+ is checked out from the aging list, frames can be read and written
|
||||
|
||||
weak::
|
||||
the +FileMap+ object is checked back into the aging list, the frame can't be accessed but we can
|
||||
try to transform a weak reference into a readonly or readwrite reference
|
||||
|
||||
|
||||
Frames can be addressed uniquely whenever a frame is not available. The vault can't serve a cached
|
||||
version of the frame, a (probably recursive) rendering request will be issued.
|
||||
|
||||
Prefetching
|
||||
~~~~~~~~~~~
|
||||
There are 2 important points when we want to access data with low latency:
|
||||
|
||||
. Since we handle much more data than it will fit into most computers RAM.
|
||||
The data which is backed in files has to be paged in and available when needed.
|
||||
The +Prefetch+ Thread manages page hinting to the kernel (posix_madvise()..)
|
||||
. Intermediate Frames must eventually be rendered to the cache.
|
||||
The Vault Layer will send +Renderjobs+ to the +Scheduler+.
|
||||
|
||||
Whenever something queries a +Frame+ from the vault it provides hints about what it is doing.
|
||||
These hints contain:
|
||||
|
||||
* Timing constraints
|
||||
- When will the +Frame+ be needed
|
||||
- could we drop the request if it won't be available (rendered) in-time
|
||||
* Priority of this job (as soon as possible, or just in time?)
|
||||
* action (Playing forward, playing backward, tweaking, playback speed, recursive rendering of dependent frames)
|
||||
|
||||
.Notes
|
||||
* The Vault Layer will try to render related frames in groups.
|
||||
* This means that following frames are scheduled with lower priority.
|
||||
* Whenever the program really requests them the priority will be adjusted.
|
||||
|
||||
|
||||
-> more about link:Scheduler.html[the Scheduling of calculation jobs]
|
||||
|
||||
* Media data is handled _frame wise_ -- prefetching data asynchronously.
|
||||
|
||||
The goal is to optimize CPU utilization; large scale data IO will be performed
|
||||
asynchronously. Data retrieval and processing of prerequisites will
|
||||
be *scheduled* such as to manage the rendering _just in time_.
|
||||
|
|
|
|||
1
doc/devel/draw/DIR_INFO
Normal file
1
doc/devel/draw/DIR_INFO
Normal file
|
|
@ -0,0 +1 @@
|
|||
SVG Vectorgraphics - directly edited with Inkscape
|
||||
2
doc/devel/draw/build_svg/DIR_INFO
Normal file
2
doc/devel/draw/build_svg/DIR_INFO
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
Scripts to produce graphics
|
||||
Currently only using LaTeX/TikZ to produce graphics
|
||||
4
doc/devel/draw/build_svg/TikZ/DIR_INFO
Normal file
4
doc/devel/draw/build_svg/TikZ/DIR_INFO
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
LaTeX/TikZ source to build graphics in svg formnat
|
||||
To run these, you'll require LaTeX and TikZ
|
||||
LateX is available on Debian via the texlive package
|
||||
TikZ is available froom the texlive-pictures package
|
||||
76
doc/devel/draw/build_svg/TikZ/doc_map.tex
Normal file
76
doc/devel/draw/build_svg/TikZ/doc_map.tex
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
% doc_map.tex: Generate map of how Lumiera developer documentation is organised
|
||||
% glossary.sh - Lumiera RfC:
|
||||
%
|
||||
% Copyright (C) Lumiera.org
|
||||
% 2023, Benny Lyons <benny.lyons@gmx.net>
|
||||
%
|
||||
% This program is free software; you can redistribute it and/or
|
||||
% modify it under the terms of the GNU General Public License as
|
||||
% published by the Free Software Foundation; either version 2 of
|
||||
% the License, or (at your option) any later version.
|
||||
%
|
||||
% This program is distributed in the hope that it will be useful,
|
||||
% but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
% GNU General Public License for more details.
|
||||
%
|
||||
% You should have received a copy of the GNU General Public License
|
||||
% along with this program; if not, write to the Free Software
|
||||
% Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
|
||||
|
||||
%
|
||||
% To compile the graphics, you will require TikZ.
|
||||
% On Debian, this is to be found in the texlive-pictures package.
|
||||
%
|
||||
\documentclass[dvisgvm]{article}
|
||||
\usepackage{tikz}
|
||||
|
||||
|
||||
\begin{document}
|
||||
|
||||
\pagestyle{empty}
|
||||
|
||||
\begin{tikzpicture}
|
||||
[font=\footnotesize,
|
||||
grow=right,
|
||||
level 1/.style={sibling distance=8em},
|
||||
level 2/.style={sibling distance=0.8em},level distance=3.5cm]
|
||||
|
||||
\node{\hbox{\vbox{
|
||||
\hbox{Lumiera (as seen)}
|
||||
\hbox{from Outer Space}
|
||||
}
|
||||
}
|
||||
} % root
|
||||
child {node {\hbox{\vbox{
|
||||
\hbox{Technical Docs}
|
||||
\hbox{The Inner Core}
|
||||
}
|
||||
}
|
||||
}
|
||||
child {node {Developer Howtos} }
|
||||
child {node {Infrastructure} }
|
||||
child {node {Code Base} }
|
||||
child {node {Support Lib} }
|
||||
child {node {Vault} child { node {API doc}}}
|
||||
child {node {Steam} child { node {API doc}}}
|
||||
child {node {Stage} child { node {API doc}}}
|
||||
child {node {Build System} }
|
||||
}
|
||||
child {node {Design Docs}
|
||||
child {node {Governance} }
|
||||
child {node {Workflow} }
|
||||
child {node {Engine} }
|
||||
child {node {Low-level Model} }
|
||||
child {node {Application} }
|
||||
child {node {Plugins} }
|
||||
child {node {GUI} }
|
||||
child {node {Model} }
|
||||
child {node {Architecture} }
|
||||
};
|
||||
|
||||
\end{tikzpicture}
|
||||
|
||||
|
||||
|
||||
\end{document}
|
||||
47
doc/devel/draw/build_svg/TikZ/makefile
Normal file
47
doc/devel/draw/build_svg/TikZ/makefile
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# makefile - build graphic map illustrating the organisation of the Lumiera developer doc
|
||||
#
|
||||
# Copyright (C) Lumiera.org
|
||||
# 2023, Benny Lyons <benny.lyons@gmx.net>
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License as
|
||||
# published by the Free Software Foundation; either version 2 of
|
||||
# the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
|
||||
#
|
||||
#
|
||||
# Do compile the graphics, you will require TikZ.
|
||||
# On Debian, this is to be found in the texlive-pictures package.
|
||||
|
||||
#
|
||||
# Requirements
|
||||
# tex, e.g., texlive
|
||||
# TikZ (on Debian in the package tetxlive-pictures
|
||||
# convert contained in Imagemagic
|
||||
# It is possible the png file using other tools
|
||||
|
||||
BUILDME=doc_map
|
||||
$(BUILDME): $(BUILDME).pdf $(BUILDME).svg $(BUILDME).png
|
||||
|
||||
$(BUILDME).pdf: $(BUILDME).tex
|
||||
pdflatex $< $@
|
||||
|
||||
$(BUILDME).svg: $(BUILDME).dvi
|
||||
dvisvgm $<
|
||||
$(BUILDME).dvi: $(BUILDME).tex
|
||||
latex $<
|
||||
$(BUILDME).png: $(BUILDME).svg
|
||||
convert -background none -size 1024x1024 $< $@
|
||||
|
||||
|
||||
:PHONEY: clean
|
||||
clean:
|
||||
@rm -f $(BUILDME).pdf $(BUILDME).svg $(BUILDME).dvi $(BUILDME).png
|
||||
BIN
doc/devel/images/doc_map.png
Normal file
BIN
doc/devel/images/doc_map.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
124
doc/devel/meeting_summary/2023-09-13.txt
Normal file
124
doc/devel/meeting_summary/2023-09-13.txt
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
2023-09-13 Lumiera Developers Meeting
|
||||
=====================================
|
||||
:Author: Ichthyo, Benny
|
||||
:Date: 2023-09-14
|
||||
|
||||
Sep 13, 2023 on #lumiera 20:00 - 22:30 UTC +
|
||||
|
||||
__Participants__
|
||||
|
||||
* ichthyo
|
||||
* benny
|
||||
* cehteh
|
||||
|
||||
_Summary written by Ichthyo and reviewed by Benny_
|
||||
|
||||
''''
|
||||
|
||||
Focus Topics
|
||||
------------
|
||||
_Ichthyo_ started to define a collection of *FocusTopics*.
|
||||
These are topics towards which we focus our attention for further development *after* the
|
||||
current Engine integration is completed. Some tickets currently of interest were enhanced
|
||||
and extended, and new tickets created, which were all marked with the `FocusTopic` Tag.
|
||||
|
||||
-> See https://issues.lumiera.org/report/17[Report-17] in the Lumiera Issue Tracker
|
||||
|
||||
_Ichthyo_ identified some difficult issues concerning session modifications and change logging,
|
||||
especially in connection with rules based configuration.
|
||||
|
||||
We discussed in some detail event sourcing, session modification, session configuration
|
||||
and representation of a configuration possibly as a context in the session. Of particular
|
||||
interest is the portability of a session from one machine to another and, consequently,
|
||||
the problems that can arise when a change in configuration occurs: these should be
|
||||
recorded as an event in the Event-Log. However, this requires the representation of the
|
||||
entire configuration in a consolidated form as part of The Model. A change in the
|
||||
configuration could then be detected and recorded as an event.
|
||||
|
||||
Another task of concern is how the actual Diff Messages will be generated. We could first
|
||||
manipulate the data model and then _detect_ a diff. Yet the favourable solution would be
|
||||
to interpret the commands and directly derive the events associated with the commands,
|
||||
with an attached diff, which can then be applied against the data model as a _projection._
|
||||
|
||||
A discussion followed to sort out which topics should be addressed first.
|
||||
|
||||
It seems the topics regarding Event Log and Session persistence
|
||||
https://issues.lumiera.org/ticket/1234[#1234]
|
||||
https://issues.lumiera.org/ticket/540[#540] and
|
||||
https://issues.lumiera.org/ticket/569[#569]
|
||||
should be addressed first, since these may lead to decisions relevant to many other parts
|
||||
of the system. Possibly also
|
||||
https://issues.lumiera.org/ticket/332[#332] regarding a full definition of placement capabilities.
|
||||
|
||||
|
||||
The topic of Placement interpretation will be addressed in the *next* vertical slice,
|
||||
where we intend to add a clip to the timeline. To do this, we'll require at least a
|
||||
_simple_ resolution capability of Placements, because we'll attempt do the groundwork for
|
||||
The Builder.
|
||||
|
||||
_Benny_ proposes to add »interface« and »mock« tickets for Event-Log and Placements,
|
||||
blocking the full implementation follow-up tickets. Then these preliminary parts can be
|
||||
addressed first and integrated as part of the »Add Clip« Vertical Slice. For Context:
|
||||
currently the »Playback« Vertical Slice is being worked on, aiming towards a first
|
||||
integration of the Render Engine.
|
||||
|
||||
|
||||
Playback Vertical Slice
|
||||
-----------------------
|
||||
_Ichthyo_ reports on the progress of the »Playback Vertical Slice« effort.
|
||||
|
||||
- Memory Manager is implemented and tested
|
||||
- Activity Language is also completed and tested
|
||||
- a passive worker pool has been implemented, based on the C++17 concurrency framework
|
||||
|
||||
Next steps:
|
||||
|
||||
- retrofit our existing thread wrapper (which was based on POSIX) to use C\+\+17 thread support;
|
||||
this should be a drop-in replacement, since our own old design almost completely matches the
|
||||
design and feature set standardised with C++17
|
||||
- complete implementation of the Scheduler Layer-2, integrating Activity-Language and worker pool
|
||||
- integration test of The Scheduler
|
||||
|
||||
Documentation Index and Glossary
|
||||
--------------------------------
|
||||
_Benny_ reworked and extended the documentation overview in the Lumiera website.
|
||||
This can be published now.
|
||||
Moreover, _Benny_ investigated Glossary and Index generation. It turns out that
|
||||
ASCIIDOC already provides an index generation feature, which could be configured and used
|
||||
to build an index. But there is no support to generate a glossary which we'll build soon.
|
||||
The next goal would be to build a prototype for testing and to see how this can be
|
||||
integrated into the website infrastructure.
|
||||
|
||||
Library solutions for Video Display
|
||||
-----------------------------------
|
||||
_Benny_ conducted some research regarding library solutions for *video display*, notably
|
||||
ffmpeg, GStreamer and libVLC. It was rather easy to code up an "hello world" example based on
|
||||
GStreamer, while it was difficult to find suitable documentation regarding the ffmpeg based
|
||||
solution. Moreover, GStreamer seems to have a much more active and helpful community.
|
||||
|
||||
Next steps would be to look in more depth into libVLC, and especially to compare the
|
||||
feature sets of these solutions. _Cehteh_ points out that MPV (formerly MPlayer) would also
|
||||
be an interesting point of reference. _Benny_ mentions that GTK4 offers an integrated
|
||||
video display capability. We'll investigate these frameworks further.
|
||||
|
||||
Extended discussion ensued regarding video frameworks and the difficulties regarding the plethora
|
||||
of interfaces, especially when it comes to hardware acceleration support. Ideally, the Lumiera
|
||||
project should follow the solutions provided by a sufficiently active community, capable of following
|
||||
the constant evolution of interfaces and frameworks.
|
||||
|
||||
It was agreed upon that the »Playback Vertical Slice« just needs a very simple solution to output
|
||||
raw video (RGB888 or YUV), and offers a good opportunity for experimentation to find a suitable
|
||||
support framework.
|
||||
|
||||
|
||||
''''
|
||||
|
||||
Next meeting
|
||||
------------
|
||||
|
||||
The next meeting will be Wednesday Oct 11, 20:00 UTC.
|
||||
You are welcome to join.
|
||||
|
||||
''''
|
||||
|
||||
|
||||
71
doc/devel/meeting_summary/2023-10-11.txt
Normal file
71
doc/devel/meeting_summary/2023-10-11.txt
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
2023-10-11 Lumiera Developers Meeting
|
||||
=====================================
|
||||
:Author: Ichthyo
|
||||
:Date: 2023-10-11
|
||||
|
||||
Oct 11, 2023 on #lumiera 20:00 - 22:00 UTC +
|
||||
|
||||
__Participants__
|
||||
|
||||
* ichthyo
|
||||
* benny
|
||||
|
||||
_Summary written by Ichthyo_
|
||||
|
||||
''''
|
||||
|
||||
Development Summary
|
||||
-------------------
|
||||
_Ichthyo_ summarises the progress regarding the »Playback Vertical Slice«.
|
||||
|
||||
- the design now favours _active_ workers and a _passive_ pool
|
||||
- the Scheduler now features _two layers_ and provides also some support for dependency management
|
||||
- there is a direct collaboration between workers and the Scheduler Layer-2
|
||||
- thus the Scheduler takes on some aspects of work management, in collaboration with the workers
|
||||
|
||||
As a follow-up, the state of the C\++ concurrency framework was
|
||||
https://issues.lumiera.org/ticket/1279#comment:2[investigated] and compared with the features
|
||||
provided by some of our own ``homegrown'' support library. A lot of modern concurrency support
|
||||
facilities are meanwhile provided by the standard library of the language, and so we can abandon
|
||||
some of our old solutions to favour a modern approach. This investigation led to a complete rework
|
||||
of our thread-wrapper. Previously we had to implement a lot of our threading support ourselves,
|
||||
based on POSIX threading. The same feature set can now be achieved by delegating to `std::thread`
|
||||
from the C++14 language standard. Design analysis showed however that the application can still
|
||||
benefit from using a custom adaptation layer on top, especially for uniform error handling and
|
||||
to support lifecycle management of components executed concurrently. This allows especially to
|
||||
simplify some load tests performed as part of the Lumiera test-suite.
|
||||
|
||||
|
||||
Extended discussion ensued regarding the modern concurrency primitives (Atomics, Future+Promise,
|
||||
Coroutines), and their relation to the established long-standing solutions (Mutex, Condition Var).
|
||||
There is no single ``best'' solution, and each of these techniques to manage concurrency presents
|
||||
the developer with an unique set of benefits and trade-offs.
|
||||
|
||||
For Lumiera it is important to assess the right order of magnitude regarding the precision of timings.
|
||||
The duration of one video frame at 25fps is 40ms, which implies that the scheduler must be reliable
|
||||
with its timings in the order of magnitude of 1ms. Thus it is desirable that the internal overhead
|
||||
is kept below 100µs. This goal is not difficult to achieve; it suffices to pay some attention to
|
||||
aspects of performance. To place this into context, single instructions in a modern CPU operate
|
||||
at the nanoseconds level.
|
||||
|
||||
|
||||
Compiler and Platform upgrades
|
||||
------------------------------
|
||||
Most of everyday development is done currently with GCC-8, which is rather dated already.
|
||||
Compilation with GCC-12 has been attempted recently, and did not turn up any serious problems.
|
||||
Two minor bugs could be fixed. With the planned system upgrade on _Ichthyo's_ developer PC
|
||||
most notably the build system *Scons* now uses Python-3. The necessary adjustments are mostly
|
||||
repetitive, and were committed by _Benny_ to a Git branch. Previously, the Lumiera build system
|
||||
used a ready-made plug-in to start *Doxygen*. This code does not work any more with Python-3,
|
||||
but a replacement solution was identified, which still needs to be integrated.
|
||||
|
||||
|
||||
''''
|
||||
|
||||
Next meeting
|
||||
------------
|
||||
|
||||
The next meeting will be Wednesday Nov 8, 20:00 UTC.
|
||||
You are welcome to join.
|
||||
|
||||
''''
|
||||
99
doc/devel/meeting_summary/2023-11-08.txt
Normal file
99
doc/devel/meeting_summary/2023-11-08.txt
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
2023-11-08 Lumiera Developers Meeting
|
||||
=====================================
|
||||
:Author: Ichthyo
|
||||
:Date: 2023-11-09
|
||||
|
||||
Nov 8, 2023 on #lumiera 22:30 - 23:30 UTC +
|
||||
|
||||
__Participants__
|
||||
|
||||
* ichthyo
|
||||
* benny
|
||||
* cehteh
|
||||
|
||||
_Summary written by Ichthyo_
|
||||
|
||||
''''
|
||||
|
||||
Wikipedia
|
||||
---------
|
||||
https://en.wikipedia.org/wiki/User:IgorVladimirsky[Igor Vladimirsky]
|
||||
contacted _Ichthyo_ last week regarding a mention of Lumiera in the context of Cinelerra.
|
||||
He was involved with the
|
||||
https://cinelerra-cv.wikidot.com[Cinelerra-CV project]
|
||||
and already contributed several improvements to the Wikipedia entry for Cinelerra,
|
||||
which includes a mention of »the Lumiera rewrite« since
|
||||
https://en.wikipedia.org/w/index.php?title=Cinelerra&diff=prev&oldid=207875710[April 2008].
|
||||
_Igor_ rightfully pointed out that Lumiera is not a fork of Cinelerra and the description of the project's
|
||||
origin might be too much biased towards the developer's perspective at that time. This critique extends also
|
||||
to the descriptions at the introductory pages of the Lumiera website itself; these characterisations were
|
||||
written during the same founding phase and not much reviewed ever since. After some exchange of proposals
|
||||
and ideas, an update and reformulation of the introductory pages was accomplished. The update to the
|
||||
content at Wikipedia, which _Igor_ thankfully completed, was a good opportunity to add some further
|
||||
sources, notably an archive.org link to an article at
|
||||
https://web.archive.org/web/20090129083001/https://www.linux.com/feature/126441[Linux.com]
|
||||
and remove a ^[citation needed]^ tag.
|
||||
|
||||
-> see the reworked
|
||||
{l}/project/index.html[``About''] and {l}/project/faq.html[``FAQ''] pages +
|
||||
-> section in the
|
||||
https://en.wikipedia.org/wiki/Cinelerra#Lumiera[Wikipedia page for Cinelerra].
|
||||
|
||||
|
||||
Development Summary
|
||||
-------------------
|
||||
As usual, _Ichthyo_ summarises the progress of coding work under the umbrella of the »Playback Vertical Slice«.
|
||||
Activities were focussed on integration of developed components and parts forming the *Scheduler Service*
|
||||
at the core of the Render Engine. Some further aspects were addressed, notably the proper use of locking
|
||||
and a new scheme to distribute available computing capacity in accordance to the planned schedule and
|
||||
an integrated load indicator. The Scheduler implementation is now largely complete and passes the
|
||||
component integration tests. The level of performance observed using short bursts of load seems adequate
|
||||
and scheduling delay is mostly dominated by the leeway of the OS thread scheduling and caching effects.
|
||||
|
||||
The next step would be to setup some meaningful load- and stress testing, using a synthetic load,
|
||||
which can be adjusted to reflect typical patterns of the expected load in actual usage. Care has
|
||||
to be taken however to distil an indicator actually describing the scheduling overhead and to avoid
|
||||
capturing just the overall system performance.
|
||||
|
||||
|
||||
Investigation: Video Display
|
||||
----------------------------
|
||||
_Benny_ continued his research regarding solutions for video display in the GUI. We agreed previously
|
||||
that the current ``vertical slice'' affords the opportunity to try out available mainstream solutions
|
||||
to find out more about their strength and weaknesses. _Benny_ decided to look at *GStreamer* first,
|
||||
since this framework is in very widespread use, with tutorials and documentation readily available.
|
||||
He conducted some experiments and was able to put together a demo which decodes an AVI file and
|
||||
displays the video content into a given GTK window. The following investigations will focus on
|
||||
the details how to inject arbitrary content into such a display, without directly relying on the
|
||||
complete decoding and rendering pipeline of GStreamer, since we intend to use the Lumiera Render
|
||||
Engine to coordinate all processing and manage all resources.
|
||||
|
||||
Another point worth mentioning is the fact that the GTK project intends to discontinue support
|
||||
of the Xlib / X.org output backend, focussing entirely on Wayland. GStreamer fully supports
|
||||
Wayland already.
|
||||
|
||||
_Ichthyo_ adds that he intends to use directly generated video content for this ``vertical slice'',
|
||||
so to avoid involving any file-IO. It seems quite feasible to use libCairo (which is already a project
|
||||
dependency and well known) do draw some simple animation. This code could than be packaged as a generator
|
||||
node and used in the more or less synthetic node graph, which will be provided by the setup for this
|
||||
integration effort.
|
||||
|
||||
Coding News
|
||||
-----------
|
||||
During all the past years, it was customary to publish a ``coding news'' report at fall time.
|
||||
There was much progress in the Lumiera project lately, since _Ichthyo_ was able to quit his regular
|
||||
work job this spring to focus entirely on his personal programming and artwork projects. The
|
||||
``vertical slice'' however will not be completed this year, and thus we agreed to author the usual
|
||||
news update now, while providing an additional report once the work will be completed and we are
|
||||
able to demonstrate running video display in the GUI.
|
||||
|
||||
|
||||
''''
|
||||
|
||||
Next meeting
|
||||
------------
|
||||
|
||||
The next meeting will be Wednesday Dec 13, 20:00 UTC.
|
||||
You are welcome to join.
|
||||
|
||||
''''
|
||||
87
doc/devel/meeting_summary/2024-01-10.txt
Normal file
87
doc/devel/meeting_summary/2024-01-10.txt
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
2024-01-10 Lumiera Developers Meeting
|
||||
=====================================
|
||||
:Author: Benny
|
||||
:Date: 2024-01-20
|
||||
|
||||
Jan 10, 2024
|
||||
on {l}/project/contact.html#_irc[`#lumiera`]
|
||||
20:00 - 23:00 UTC +
|
||||
|
||||
__Participants__
|
||||
|
||||
* ichthyo
|
||||
* benn
|
||||
|
||||
''''
|
||||
Two main points of discussion:
|
||||
|
||||
- Scheduler tests to examine its suitability
|
||||
- Using Gstreamer to view video for our vertical slice
|
||||
|
||||
|
||||
Test Scheduler Suitability
|
||||
--------------------------
|
||||
|
||||
Once the scheduler had been implemented it was necessary to subject it
|
||||
to various load tests to establish its stability, and stress tests were necessary to
|
||||
provide information on performance limitations so that we would be able to establish
|
||||
whether these limitations met the demands required by Lumiera.
|
||||
Results were encouraging.
|
||||
|
||||
_Ichthyo_ reported on stress tests on the new scheduler. After resolving a series
|
||||
of initial bugs, processing runs stable now, even when entering over 500 test jobs
|
||||
simultaneously to the queue. Stress tests revealed typical limitations of the scheduler,
|
||||
which exhibited behaviour that is well within the requirements of Lumiera. Load tests
|
||||
showed that the scheduler remained stable when subjected to short heavy load bursts
|
||||
within expected bounds.
|
||||
|
||||
Indeed performance testing is always a delicate and complex area to
|
||||
investigate, as it is necessary to establish reasonable performance boundaries
|
||||
of the system: we can always overload any scheduler with fanatically huge numbers
|
||||
of jobs; while the scheduler is still able to process all enqueued jobs eventually,
|
||||
congestion builds up and jobs start missing their deadlines. Of interest, however,
|
||||
is to establish some reasonable information on the processing rate of the scheduler
|
||||
which _Ichthyo_ was able to estimate statistically by carefully observing the
|
||||
scheduler under a collection of pre-calibrated loads.
|
||||
|
||||
Some goals of the tests:
|
||||
|
||||
- establish the scheduler breaking point or performance limitations
|
||||
- tests under pressure with parallelism and complex dependencies between jobs
|
||||
- demonstrate that the scheduler is stable under typical load for several seconds
|
||||
|
||||
Tests on a wider array of hardware constellations are not expected to reveal much
|
||||
relevant information in this stage of the project and will thus be conducted later.
|
||||
This kind of analytical, isolated component tests will not ultimately resolve the
|
||||
issue of whether the scheduler can perform as desired, but subjecting the scheduler
|
||||
to carefully selected tests can help to reduce the probability of missing serious
|
||||
performance limitations inherent to the chosen design.
|
||||
|
||||
|
||||
GStreamer Demo to Play Video
|
||||
----------------------------
|
||||
|
||||
_Benn_ has written a simple GStreamer demo so that a video in raw format can be
|
||||
displayed on the GUI for the current vertical slice. The next step in this
|
||||
investigation is to identify a low-level entrance point to the functionality
|
||||
provided by the GStreamer library, since for use in this vertical slice the
|
||||
render engine produces a buffer filled with raw video data for each frame,
|
||||
which will then be displayed in a video widget as an ongoing process.
|
||||
|
||||
This is a general question requiring further investigation: how to make use of
|
||||
established functionality from a library to simplify our implementation task,
|
||||
while retaining the full control and coordination of the rendering and playback
|
||||
process and resource management within the Lumiera render engine itself.
|
||||
|
||||
''''
|
||||
|
||||
Next meeting
|
||||
------------
|
||||
|
||||
The next Lumiera meeting is scheduled for Wednesday, Feb 14 at 20:00 UTC.
|
||||
Should this be inconvenient for you, please speak up on, for example,
|
||||
link:http://lists.lumiera.org/cgi-bin/mailman/listinfo/lumiera[lumiera@lists.lumiera.org]
|
||||
and let us know of when you would prefer to attend.
|
||||
|
||||
''''
|
||||
|
||||
|
|
@ -11,8 +11,113 @@ spending time on the task of writing a summary.
|
|||
Anyone interested in Lumiera development is also encouraged to read mailing list
|
||||
archives and other documentation.
|
||||
|
||||
8 May / 12 June 2024
|
||||
--------------------
|
||||
Discussion regarding the approach taken towards the next core topic, which
|
||||
is how to structure the render nodes in the low-level model and how to perform
|
||||
an invocation, leading to the calculation of media data. It is considered very
|
||||
important for Lumiera to remain library / framework agnostic, even while we
|
||||
use some frameworks and libraries for prototyping. The structure drafted over
|
||||
the course of the last years and about to be integrated now must be flexible
|
||||
enough to support adapt to the requirements of various libraries, _without
|
||||
attempting to build a super structure_ to ``rule them all''.
|
||||
|
||||
12 Sep 2012
|
||||
|
||||
13 Mar / 10 Apr 2024
|
||||
--------------------
|
||||
Status update and informal exchange regarding current development activities
|
||||
|
||||
* after factoring out further external influences, a _breaking point_ for
|
||||
processing complex dependent schedules could be established, which closely
|
||||
matches theoretical expectations
|
||||
* a further measurement series (linear regression) was necessary to establish
|
||||
performance characteristics of the scheduler under constant high load
|
||||
* as expected, a serious weakness pertaining contention could be confirmed +
|
||||
(yet contention is not expected to be a problem in practice)
|
||||
* as mitigation, concurrency is scaled down on very short (100µs) jobs
|
||||
* overall we are confident that the new scheduler is capable to handle the
|
||||
expected load pattern and does not suffer from major (unknown) weaknesses.
|
||||
|
||||
Moreover, the problems related to integration and use of external frameworks
|
||||
for video display were discussed further.
|
||||
|
||||
|
||||
14 Feb 2024
|
||||
-----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* Discussion about GTK\+, Python bindings vs. C++ bindings vs. the C-API,
|
||||
challenges due to the dynamic nature of layout and inconsistent CSS settings,
|
||||
and further difficulties integrating video display.
|
||||
* Update regarding the ongoing effort to establish performance limits of the
|
||||
Scheduler and discussion about methods of measurement.
|
||||
|
||||
10 Jan 2024
|
||||
-----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* Scheduler performance testing
|
||||
* using GStreamer for Video output to the GUI
|
||||
|
||||
Summary
|
||||
^^^^^^^
|
||||
* link:2024-01-10.html[Summary (by benn)]
|
||||
|
||||
|
||||
13 Dec 2023
|
||||
-----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* Extended technical discussion about Scheduler load testing
|
||||
and how to find out about inherent limitations of the approach taken
|
||||
|
||||
8 Nov 2023
|
||||
----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* Wikipedia
|
||||
* Scheduler integration and testing
|
||||
* Video output to the GUI
|
||||
|
||||
Summary
|
||||
^^^^^^^
|
||||
* link:2023-11-08.html[Summary (by ichthyo)]
|
||||
|
||||
|
||||
11 Oct 2023
|
||||
-----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* Scheduler and worker pool design
|
||||
* Rework of multithreading support
|
||||
* Compiler and platform upgrades
|
||||
|
||||
Summary
|
||||
^^^^^^^
|
||||
* link:2023-10-11.html[Summary (by ichthyo)]
|
||||
|
||||
|
||||
13 Sep 2023
|
||||
-----------
|
||||
|
||||
Topics
|
||||
~~~~~~
|
||||
* »Focus Topics«
|
||||
* Progress on the »Playback Vertical Slice«
|
||||
* Website Index and Glossary generator
|
||||
* Library solutions for video display in the GUI
|
||||
|
||||
Summary
|
||||
^^^^^^^
|
||||
* link:2023-09-13.html[Summary (by ichthyo)]
|
||||
|
||||
|
||||
12 Sep 2013
|
||||
-----------
|
||||
|
||||
Topics
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ Design Process : All Plugin Interfaces Are C
|
|||
|
||||
[grid="all"]
|
||||
`------------`-----------------------
|
||||
*State* _Final_
|
||||
*State* _Dropped_
|
||||
*Date* _2007-06-29_
|
||||
*Proposed by* link:ct[]
|
||||
-------------------------------------
|
||||
|
|
@ -159,17 +159,50 @@ After a talk on IRC ichthyo and me agreed on making lumiera a multi language
|
|||
project where each part can be written in the language which will fit it best.
|
||||
Language purists might disagree on such a mix, but I believe the benefits
|
||||
outweigh the drawbacks.
|
||||
-- link:ct[] [[DateTime(2007-07-03T05:51:06Z)]]
|
||||
|
||||
ct:: '2007-07-03 05:51:06'
|
||||
|
||||
C is the only viable choice here. Perhaps some sort of "test bench" could be
|
||||
designed to rigorously test plugins for any problems which may cause Lumiera to
|
||||
become unstable (memory leaks etc).
|
||||
-- link:Deltafire[] [[DateTime(2007-07-03T12:17:09Z)]]
|
||||
|
||||
Deltafire:: '2007-07-03 12:17:09'
|
||||
|
||||
after a talk on irc, we decided to do it this way, further work will be
|
||||
documented in the repository (tiddlywiki/source)
|
||||
-- link:ct[] [[DateTime(2007-07-11T13:10:07Z)]]
|
||||
|
||||
ct:: '2007-07-11 13:10:07'
|
||||
|
||||
|
||||
Development took another direction over course of the years;
|
||||
Lumiera is not based on a _generic plug-in Architecture_ and the topic
|
||||
of interfaces for _dedicated plugins_ still needs to be worked out
|
||||
|
||||
Ichthyostega:: '2023-10-24 22:55:23'
|
||||
|
||||
|
||||
|
||||
Conclusion
|
||||
----------
|
||||
Initially there was agreement over the general direction set out by this proposal.
|
||||
However, _Ichthyo_ was always sceptical regarding the benefits of a generic plug-in
|
||||
Architecture. Experience with high-profile projects based on such a concept seem
|
||||
to show both tremendous possibilities, especially regarding user involvement, but
|
||||
at the same time also indicate serious problems with long-term sustainability.
|
||||
|
||||
The practical development -- mostly driven ahead by _Ichthyo_ -- thus never really
|
||||
embraced that idea; rather structuring by internal interfaces and contracts was
|
||||
preferred. The basic system for loading of Plug-ins, as indicated by this proposal,
|
||||
is still used though to load some dedicated plug-ins, most notably the GUI.
|
||||
|
||||
To draw an conclusion: this proposal is now considered as *rejected*.
|
||||
Instead, Ticket https://issues.lumiera.org/ticket/1212[#1212 »Extension Concept«]
|
||||
was added to the list of relevant
|
||||
https://issues.lumiera.org/report/17[»Focus Topics«] for further development.
|
||||
|
||||
.........
|
||||
|
||||
.........
|
||||
|
||||
''''
|
||||
Back to link:/documentation/devel/rfc.html[Lumiera Design Process overview]
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -3,7 +3,7 @@ Design Process : Data Backend
|
|||
|
||||
[grid="all"]
|
||||
`------------`-----------------------
|
||||
*State* _Final_
|
||||
*State* _Parked_
|
||||
*Date* _2007-06-04_
|
||||
*Proposed by* link:ct[]
|
||||
-------------------------------------
|
||||
|
|
@ -42,24 +42,14 @@ This just starts as braindump, I will refine it soon:
|
|||
http://www.pipapo.org/pipawiki/Cinelerra/Developers/ichthyo/Cinelerra3/Architecture[Architecture] and a sketch/draft about http://www.pipapo.org/pipawiki/Cinelerra/Developers/ichthyo/Possibilities_at_hand[things possible in the middle layer]
|
||||
|
||||
|
||||
Tasks
|
||||
^^^^^
|
||||
Parked
|
||||
~~~~~~
|
||||
The underlying principles remain valid, yet development took another
|
||||
direction during the last years. Special frameworks for high-performance
|
||||
asynchronous IO will be used for dedicated use cases.
|
||||
|
||||
Ichthyostega:: '2023-10-24' ~<prg@ichthyostega.de>~
|
||||
|
||||
Pros
|
||||
^^^^
|
||||
|
||||
|
||||
Cons
|
||||
^^^^
|
||||
|
||||
|
||||
Alternatives
|
||||
^^^^^^^^^^^^
|
||||
|
||||
|
||||
Rationale
|
||||
~~~~~~~~~
|
||||
|
||||
|
||||
Comments
|
||||
|
|
@ -71,5 +61,10 @@ Sounds fairly complete to me
|
|||
Developement takes place in the repo now
|
||||
-- link:ct[] [[DateTime(2007-06-27T16:14:56Z)]]
|
||||
|
||||
Development took another direction over the last years;
|
||||
the former »Backend« layer is restructured
|
||||
-- link:Ichthyostega[] [[DateTime(2023-10-24T22:45:55Z)]]
|
||||
|
||||
|
||||
''''
|
||||
Back to link:/documentation/devel/rfc.html[Lumiera Design Process overview]
|
||||
|
|
|
|||
|
|
@ -12,8 +12,8 @@ WebsiteSupportMarkup
|
|||
|
||||
********************************************************************************
|
||||
.Abstract
|
||||
The Lumiera Website and Documentation uses a lightweight, plaintext based
|
||||
Infrastructure built on top of Asciidoc and Git. To help with everyday
|
||||
The Lumiera Website and Documentation uses a lightweight, plain text based
|
||||
infrastructure built on top of Asciidoc and Git. To help with everyday
|
||||
authoring and editorial tasks, a set of support facilities is proposed,
|
||||
to be integrated seamlessly into the existing infrastructure. The front-end
|
||||
to these tools is specific markup, allowing for *cross-linking*, *tag*-based
|
||||
|
|
@ -34,38 +34,44 @@ Use Cases
|
|||
~~~~~~~~~
|
||||
|
||||
Authoring::
|
||||
For now, the _authoring of new content_ is mostly a responsibility of the core developers.
|
||||
It needs to be done close to the actual coding activities, typically once a new facility
|
||||
is roughly finished. It is crucial for this activity not to impose a _context switch_
|
||||
on the writer. It must be ``just writing down the obvious'' -- otherwise developers tend
|
||||
to defer this activity ``for later''. This situation creates some unique problems...
|
||||
For now, the _authoring of new content_ is mostly the responsibility of the core developers.
|
||||
It needs to be done close to the related coding activities, typically once a new facility
|
||||
is finished or in the final stages of completion. It is crucial for this
|
||||
activity not to impose a _context switch_ on the writer. It must be ``just
|
||||
writing down the obvious'' -- otherwise developers tend to defer this activity
|
||||
``for later''. This situation creates some unique problems...
|
||||
|
||||
- during the write-up, the coder-author realises several external presuppositions, which,
|
||||
during the act of coding, seemed to be ``obvious'' and ``self evident''.
|
||||
- since expanding on all of those secondary topics is out of question (the author will rather
|
||||
- since expanding on all of those secondary topics is out of scope (the author would rather
|
||||
abandon the task of documentation altogether), the only solution is to allow for *cross linking*,
|
||||
even if the created links are dangling for the moment.
|
||||
- the author can't effort to locate other documents and determine URLs; he won't even be willing
|
||||
to consult a markup syntax reference. Because, what he's about to write is essentially hard to
|
||||
put into words and thus requires his full attention.
|
||||
even if the new links are dangling; well at least for the moment.
|
||||
- the author can't invest the effort to locate additional relevant documents,
|
||||
nor hunt for URLs; the author is disinclined to consult a markup syntax
|
||||
reference. This arises from the nature of the required text which is
|
||||
essentially difficult to grasp in the first place and even more difficult
|
||||
too put into words. A task requiring full mental concentration.
|
||||
|
||||
|
||||
Integrating Content::
|
||||
This task is often prompted by some kind of external cause: it might be someone asking for
|
||||
explanations and while trying to respond, it was determined that ``this should be in the documentation''.
|
||||
Thus, some existing markup is extracted from an external system and pasted into some location,
|
||||
``just for the moment''. Of course, this content will be forgotten and rest there for
|
||||
years to come. To deal with this situation...
|
||||
years to come. To counter act such a situation, we propose the following:
|
||||
|
||||
- adapting the structural cross-references of the integrated markup needs to be an easy task.
|
||||
- we need a way to hook the new content somehow into our existing categorisation
|
||||
- structural cross-references of the integrated markup needs to be easy to do.
|
||||
- a way in which we can somehow hook the new content into our current scheme
|
||||
of categories
|
||||
- the person integrating the content wont't be willing to visit a lot of other locations,
|
||||
or to read a syntax reference for some kind of advanced markup.
|
||||
|
||||
Editorial work::
|
||||
The _editor_ is reviewing existing content. He'll try to emulate an assumed user's point of view
|
||||
The _editor_ reviews current content. He'll try to emulate an assumed user's point of view
|
||||
to judge the adequacy of the presentation. This leads to rearranging and re-locating pages and whole
|
||||
trees of pages, even splitting some of them, and adding introductory paragraphs and pages here and
|
||||
there -- all without the ability to read and understand any of the reviewed material in-depth.
|
||||
there -- all without the ability to read or understand any of the reviewed
|
||||
material to any degree of depth.
|
||||
|
||||
- this work will be performed on the primary content using its ``natural'' representation: that is,
|
||||
the editor will copy and move files in the file system and edit text. There is no room for using
|
||||
|
|
@ -76,7 +82,7 @@ Editorial work::
|
|||
|
||||
Tools for the task at hand
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
This RfC proposes a set of tools to cope with those problems. More specifically, this proposal details
|
||||
This RfC proposes a set of tools to improve these problems. More specifically, this proposal details
|
||||
a possible _front end_ for these tools, in a way which blends well with the existing Git / Asciidoc
|
||||
website infrastructure.
|
||||
|
||||
|
|
@ -96,18 +102,18 @@ Variations and extensions
|
|||
- we might consider adding some domain prefixes, e.g. +link:rfc:SomeTopic+ or +link:ticket:SomeTopic+
|
||||
or +link:code:SomeTopic+
|
||||
|
||||
Obviously, these cross-links needs to be resilient towards content reorganisation after-the-fact.
|
||||
Effectively this mandates introducing some kind of indirection, since we can't effort to regenerate the
|
||||
Obviously, these cross-links need to be resilient against content reorganisation after-the-fact.
|
||||
Effectively this mandates introducing some kind of indirection, since we can't afford to regenerate the
|
||||
whole website's HTML pages after each localised change. A possible solution could be to make the rendered
|
||||
cross link invoke a JavaScript function, which in turn consults some kind of index table. Another solution
|
||||
would be to let the cross link point to a server sided script.
|
||||
|
||||
Tag extractor and Index
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Define suitable ways for attaching *tags* to various kinds of content. The syntax will be tailored to the
|
||||
kind of content in question, typically placing the tags in some kind of comment or specific header. For
|
||||
larger documents, it would be desirable to attach tags in a more fine-grained manner, e.g. tag only one
|
||||
paragraph or sub-section (but this is a nice-to-have, since it is anything but trivial to implement).
|
||||
Define suitable ways to attach *tags* to various kinds of content. The syntax will be tailored to the
|
||||
kind of content in question, typically placing a tag in, for example, a comment or possibly in a special
|
||||
header. For larger documents, it would be desirable to attach tags in a more fine-grained manner, e.g. tag
|
||||
only one paragraph or sub-section (but this is a nice-to-have, since it is anything but trivial to implement).
|
||||
|
||||
Based on these tags, there should be a mechanism to integrate a list of links into some Asciidoc page.
|
||||
Obviously this list needs to be dynamic, e.g. by using JavaScript or by including pre-fabricated HTML
|
||||
|
|
@ -123,10 +129,10 @@ to take precedence.
|
|||
Definition entries
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
Define a suitable format to promote an existing piece of information into a *definition*. While not
|
||||
interfering with the presentation at the textual location of this definition, this mechanism should
|
||||
allow to extract this definition and present it within a glossary of terms. It would be nice if
|
||||
such a generated glossary could provide an automatic back-link to the location where the definition
|
||||
was picked up initially.
|
||||
interfering with the presentation at the textual location of this definition, the mechanism should support the
|
||||
possibility of providing a definition in the source which is then extracted from the text and presented
|
||||
in a new file as, for example, a glossary of terms. It would be nice if such a generated glossary could
|
||||
provide an automatic back-link to the location where the definition was picked up initially.
|
||||
|
||||
Of course, defining such a markup includes some reasoning about the suitable format of a glossary
|
||||
and definition-of-terms. (Scope? Just a sentence? Just a paragraph? How to utilise existing headings?)
|
||||
|
|
@ -144,7 +150,8 @@ interested in using them. He will not recall any fancy syntax, and he won't stic
|
|||
So, anything not outright obvious is out of question.
|
||||
|
||||
- since we don't want fully dynamic page generation and we can't afford regenerating the whole website
|
||||
for each small update, all of these facilities need some way to adapt after-the-fact.
|
||||
for each small update, all of these facilities need some way to be integrated into the present
|
||||
infrastructure with the least amount of change as pssible.
|
||||
- we need to build leeway into the system at various places. E.g. the cross-link facility needs a
|
||||
strategy to generate and match the IDs and order possible matches in a sensible way. What initially
|
||||
links to some doxygen comment might later on point to a glossary if applicable.
|
||||
|
|
|
|||
1
doc/devel/rfc_parked/ConfigLoader.txt
Symbolic link
1
doc/devel/rfc_parked/ConfigLoader.txt
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
../rfc/ConfigLoader.txt
|
||||
107
doc/index.txt
107
doc/index.txt
|
|
@ -1,7 +1,6 @@
|
|||
Documentation
|
||||
=============
|
||||
|
||||
|
||||
This documentation section contains documentation for both users and developers.
|
||||
|
||||
== User ==
|
||||
|
|
@ -14,30 +13,108 @@ A user manual containing the full reference for using Lumiera is planned:
|
|||
* link:user/manual.html[User Manual] _(planned)_
|
||||
|
||||
== Developer ==
|
||||
Information about the structure and concepts Lumiera is based upon.
|
||||
These documents give an overview on the main components of the application:
|
||||
|
||||
* link:design/index.html[Design Documents]
|
||||
This section is a guide on how the developer documentation is arranged, organised and grouped. We present here
|
||||
a map of the developer documentation to assist you in navigating through the documents.
|
||||
|
||||
Programming-oriented details about the functional layers of Lumiera as well as a Doxygen documentation :
|
||||
=== Developer Documentation is Arranged into Layers
|
||||
|
||||
* link:technical/index.html[Technical Documents]
|
||||
* link:technical/code/codingGuidelines.html[Style and Coding Guidelines]
|
||||
image:devel/images/doc_map.png[Map]
|
||||
|
||||
=== the TiddlyWiki ===
|
||||
The various pieces of documentation are not scattered about randomly. These are arranged and organised along
|
||||
established simple yet flexible guidelines to enable easy and intuitive access. It is not rigid and conformity
|
||||
is not required. There may be very good reasons to adapt the rules with some degree of flexibility at times;
|
||||
and sometimes it might be difficult to select a unique section in which to place new documents.
|
||||
|
||||
The various documents are organised in an hierarchical structure of layers. The uppermost layer, or root, is
|
||||
the point of entry into the Lumiera developer documentation universe. The lowest layers are populated by the
|
||||
Lumiera API documentation which is currently produced from the source code using Doxygen.
|
||||
|
||||
The map is depicted above in the illustration. Below we provide links to the top-level documents to all
|
||||
elements in the map froom which you can launch deeper into the lower layers of the documentation. We present
|
||||
these in tabular form to ease navigation.
|
||||
--
|
||||
|
||||
--
|
||||
|
||||
The *Documentation Layer Structure* appears at various places in the project...
|
||||
|
||||
Lumiera homepage ::
|
||||
https://Lumiera.org[Lumiera homepage] by the menus on the left hand-side of the
|
||||
page. This menu is also organised into layers that reflects the organisation of
|
||||
the documentation to help you navigate through the documentation.
|
||||
|
||||
Source code doc directory structure ::
|
||||
The layout of the sub-directories below the *doc* directory is structured around
|
||||
the hierarchy,
|
||||
|
||||
==== The Uppermost Layer
|
||||
|
||||
At the highest, uppermost layer is the document
|
||||
https://lumiera.org/documentation/user/intro/intro.html[Lumiera (as seen) from Outer Space]. This is the node
|
||||
on the left in the illustration above. All new to Lumiera should read this document. You can always return
|
||||
to this document later if you are entirely lost and wish to find your bearings before navigating into the
|
||||
deeper documentation dungeons.
|
||||
|
||||
==== The Second Layer
|
||||
|
||||
At the second layer, the reader must make the first major decision on which
|
||||
router to pursue: documentation with a heavy code based leaning; or more descriptive
|
||||
documentation on the design of Lumiera which avoids a code narrative. This
|
||||
section has two main sections:
|
||||
|
||||
* link:design/index.html[Lumiera Developer Design Documents]
|
||||
* link:technical/index.html[Lumiera Developer Technical Documents]
|
||||
|
||||
--
|
||||
|
||||
--
|
||||
|
||||
|
||||
===== Design Documentation
|
||||
|
||||
[width="90%",frame="topbot",options="header",cols="^,2<"]
|
||||
|================================================================================================
|
||||
|Sub-System|Remarks
|
||||
|https://lumiera.org/documentation/design/architecture/index.html[Architecture]|Structure of Lumiera
|
||||
|https://lumiera.org/documentation/design/gui/index.html[GUI]|The Lumiera ⟷ user interaction layer
|
||||
|https://lumiera.org/documentation/design/model/index.html[Model]|Low-level and high-level models
|
||||
|https://lumiera.org/documentation/design/engine/index.html[Engine]|Renderer, output generation, ...
|
||||
|https://lumiera.org/documentation/design/lowlevel/index.html[Low-level]|Low-level and system services
|
||||
|https://lumiera.org/documentation/design/application/index.html[Application]|Application framework
|
||||
|https://lumiera.org/documentation/design/plugins/index.html[Plugins]|Dynamically loaded Extensions
|
||||
|https://lumiera.org/documentation/design/workflow/index.html[Workflow]|Workflow and Interaction Design
|
||||
|https://lumiera.org/documentation/design/governance/index.html[Governance]|Meta project concerns
|
||||
|================================================================================================
|
||||
|
||||
|
||||
|
||||
===== Technical Documentation
|
||||
|
||||
[width="90%",frame="topbot",options="header",cols="^,2<"]
|
||||
|================================================================================================
|
||||
|Sub-System|Remarks
|
||||
|https://lumiera.org/documentation/technical/build/index.html[Build System]|Build, Packaging and CI
|
||||
|https://lumiera.org/documentation/technical/stage/index.html[Stage]|GUI (technical aspects)
|
||||
|https://lumiera.org/documentation/technical/steam/index.html[Steam]|Session and processing coordination
|
||||
|https://lumiera.org/documentation/technical/vault/index.html[Vault]|Low-level operations
|
||||
|https://lumiera.org/documentation/technical/library/index.html[Support lib]|Interface and support libraries
|
||||
|https://lumiera.org/documentation/technical/code/index.html[Code base]|Code management, organisation, ...
|
||||
|https://lumiera.org/documentation/technical/infra/index.html[Infrastructure]|Website and developer tooling ...
|
||||
|https://lumiera.org/documentation/technical/howto/index.html[Developer HowTos]|Instructions and guides for developers
|
||||
|link:/doxy/index.html[API Doc]|Using link:http://doxygen.org[Doxygen] to generate doc from code
|
||||
|================================================================================================
|
||||
|
||||
|
||||
The TiddlyWiki::
|
||||
Currently, Lumiera is still in the design- and evolution phase.
|
||||
There is an embedded JavaScript wiki (TiddlyWiki) within the source tree, mostly
|
||||
used as design notebook, featuring day-to-day design sketches, notes but also
|
||||
There is an embedded JavaScript wiki (https://classic.tiddlywiki.com/[TiddlyWiki]) within the source tree,
|
||||
mostly used as design notebook, featuring day-to-day design sketches, notes but also
|
||||
quite some more persistent planning. Finished documentation text is constantly
|
||||
moved over to the documentation section(s) of the Lumiera website.
|
||||
|
||||
-> access the Development link:{l}/wiki/renderengine.html[TiddlyWiki online here]
|
||||
|
||||
=== API Documentation ===
|
||||
We use the link:http://doxygen.org[Doxygen] tool to extract
|
||||
code comments and generate code level API and reference documentation.
|
||||
|
||||
-> link:/doxy/index.html[Doxygen generated documentation]
|
||||
|
||||
== Media and Presentations ==
|
||||
|
||||
|
|
|
|||
|
|
@ -1,121 +0,0 @@
|
|||
The incomplete Guide to Lumiera Configuration
|
||||
=============================================
|
||||
:author: ct
|
||||
:date: 8/2008
|
||||
|
||||
WARNING: this is a draft from the early days of Lumiera +
|
||||
IMHO, the whole topic ``Configuration'' requires further
|
||||
discussion and design.
|
||||
+
|
||||
-- -- Ichthyo
|
||||
|
||||
'''''''''''''
|
||||
'''''''''''''
|
||||
|
||||
Order is roughly alphabetically, depending on the mood of the writer.
|
||||
Defaults are noted if present. Not all are implemented yet.
|
||||
|
||||
General Introduction
|
||||
--------------------
|
||||
|
||||
Lumiera uses plaintext files with a INI file like syntax for
|
||||
configuration. This Syntax is strictly line based. There are only a
|
||||
few syntactic elements.
|
||||
|
||||
TODO:describe config syntax here
|
||||
|
||||
Config Subsystem
|
||||
----------------
|
||||
|
||||
The path where Lumiera searches its configuration. Single components are
|
||||
separated by colons as in PATH and other such environment variables.
|
||||
Here it might be handy that any Lumiera configuration can be
|
||||
overridden by a environment variable:
|
||||
'LUMIERA_CONFIG_PATH=somewhere:else lumiera ...'
|
||||
A default are initialized at installation time, this is important to
|
||||
bootstrap the whole configuration system.
|
||||
|
||||
config.path
|
||||
|
||||
|
||||
The config system check for a preferred format when writing config
|
||||
entries. For each key 'foo.bar', these can be overridden with a key
|
||||
'config.format.foo.bar' linking to the desired format.
|
||||
|
||||
config.formatkey ='config.format.%s'
|
||||
|
||||
|
||||
The following are links to the default formatting when no explicit
|
||||
format is set for a key. Changing these to a wrong type will break the
|
||||
system!
|
||||
|
||||
config.formatdef.link < config.formatstr.link
|
||||
config.formatdef.number < config.formatstr.number.dec
|
||||
config.formatdef.real < config.formatstr.real
|
||||
config.formatdef.string < config.formatstr.string
|
||||
config.formatdef.word < config.formatstr.word
|
||||
config.formatdef.bool < config.formatstr.bool
|
||||
|
||||
|
||||
This are the low level formating specifications for the buildin
|
||||
types, DONT TOUCH THESE!
|
||||
|
||||
config.formatstr.link = '< %s'
|
||||
config.formatstr.number.dec = '= %lld'
|
||||
config.formatstr.number.hex = '= 0x%llX'
|
||||
config.formatstr.number.oct = '= 0%llo'
|
||||
config.formatstr.real = '= %Lg'
|
||||
config.formatstr.real.dec = '= %Lf'
|
||||
config.formatstr.real.sci = '= %Le'
|
||||
config.formatstr.string = '= %s'
|
||||
config.formatstr.string.dquoted = '= \"%s\"'
|
||||
config.formatstr.string.quoted = '= ''%s'''
|
||||
config.formatstr.word = '= %s'
|
||||
config.formatstr.bool = '= %d'
|
||||
|
||||
|
||||
Plugin System
|
||||
-------------
|
||||
|
||||
The path where Lumiera searches its plugins. Single components are
|
||||
separated by colons as in PATH and other such environment variables.
|
||||
Here it might be handy that any Lumiera configuration can be
|
||||
overridden by a environment variable:
|
||||
'LUMIERA_PLUGIN_PATH=somewhere:else lumiera ...'
|
||||
Sensible defaults are initialized at installation time.
|
||||
|
||||
plugin.path
|
||||
|
||||
|
||||
I/O Backend
|
||||
-----------
|
||||
|
||||
File handling
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
How many filehandles the backend shall use [approx 2/3 of all available]
|
||||
|
||||
vault.file.max_handles
|
||||
|
||||
|
||||
|
||||
Memory mapped Files
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Address space limit (memory mapping)
|
||||
Defaults:
|
||||
3GiB on 32 bit arch
|
||||
192TiB on 64 bit arch
|
||||
|
||||
vault.mmap.as_limit
|
||||
|
||||
Default start size for mmaping windows.
|
||||
128MB on 32 bit arch
|
||||
2GB on 64 bit arch
|
||||
|
||||
vault.mmap.window_size
|
||||
|
||||
How many memory mappings shall be established at most
|
||||
Default 60000
|
||||
|
||||
vault.mmap.max_maps
|
||||
|
|
@ -5,5 +5,4 @@ Here we collect bits of technical documentation for the Vault-Layer.
|
|||
|
||||
For now, we have:
|
||||
|
||||
* link:ConfigLoader.html[Config Loader brainstorming from 2008]
|
||||
* link:scheduler.html[Scheduler and Jobs]
|
||||
|
|
|
|||
|
|
@ -4,37 +4,16 @@ Scheduler and Job handling
|
|||
The purpose of the _Scheduler_ is to run small self contained _Jobs_
|
||||
ordered by priority and observing specific timing constraints.
|
||||
|
||||
Scheduler implementation ideas
|
||||
------------------------------
|
||||
NOTE: Subject to [yellow-background]#active design and implementation# work as of 10/2023
|
||||
|
||||
Use multiple priority queues
|
||||
Work-in-progress documentation can be found in the
|
||||
link:{l}/wiki/renderengine.html#PlaybackVerticalSlice%20RenderEngine%20Scheduler%20SchedulerWorker%20SchedulerMemory%20RenderActivity%20Player%20FrameDispatcher%20JobPlanningPipeline%20PlayProcess%20Rendering%20ProcNode%20NodeOperationProtocol[Tiddly Wiki]
|
||||
|
||||
- background work
|
||||
- foreground high-priority
|
||||
- soft-realtime actions
|
||||
|
||||
About Jobs
|
||||
----------
|
||||
A job is a closure to run a small and limited action or operation, which
|
||||
in itself _should not block_. Job may depend on each other and on resources
|
||||
to be provided. A job may be conained in multiple queues and may be marked
|
||||
as _canceled_ -- in which case the job function will never run and the job
|
||||
will be discarded on occasion.
|
||||
|
||||
Job States
|
||||
~~~~~~~~~~
|
||||
|
||||
[source,C]
|
||||
--------------
|
||||
enum job_state
|
||||
{
|
||||
done, // already done, nothing to do
|
||||
running, // job is running
|
||||
waiting, // waits for some resource (annother job)
|
||||
rejected, // sorry, cant do that dave, time will run out
|
||||
expired, // time expired
|
||||
aborted // got aborted
|
||||
}
|
||||
--------------
|
||||
|
||||
to be provided. A job may be rescheduled prior to invocation, but it is
|
||||
activated at most once by an _Scheduler Action._
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue