diff --git a/admin/rfc.sh b/admin/rfc.sh index bbec05e0c..17a78b456 100755 --- a/admin/rfc.sh +++ b/admin/rfc.sh @@ -278,6 +278,11 @@ function edit() EDITOR="${EDITOR:-$(git config --get core.editor)}" EDITOR="${EDITOR:-$VISUAL}" + if [ -z "$EDITOR" ]; then + echo -e "\nFATAL\n\$EDITOR undefined\n\n" + exit -1 + fi + local file="$1" local line=0 diff --git a/data/config/setup.ini b/data/config/setup.ini index 868a0333a..01f9dd73d 100644 --- a/data/config/setup.ini +++ b/data/config/setup.ini @@ -15,7 +15,7 @@ title = Lumiera version = 0.pre.03 website = http://www.lumiera.org authors = Joel Holdsworth|Christian Thäter|Hermann Voßeler|Stefan Kangas|Michael Fisher|Michael Ploujnikov|and others... -copyright = 2007 - 2015 +copyright = 2007 - 2025 [Gui] #stylesheet = lumiera.css diff --git a/doc/design/lowlevel/Scheduler.txt b/doc/design/lowlevel/Scheduler.txt deleted file mode 100644 index 49793e06f..000000000 --- a/doc/design/lowlevel/Scheduler.txt +++ /dev/null @@ -1,25 +0,0 @@ -The Scheduler -------------- -:Author: CehTeh -:Date: 6/2007 - -//MENU: label Scheduler - -Scheduling is done with two priority queues, one for high priority jobs and one for low priority jobs. -These priority queues are ordered by absolute time values plus some job specific identified. - -There are following (non exhaustive) kinds of jobs: - -* started job -* job to be canceled -* unscheduled job -* dependency providing jobs - -Jobs implement a kind of future. We try hard to avoid any blocking waits. -The Job scheduler runs singlethreaded. Its only task is to schedule and delegate jobs to worker threads, -by itself it will never do any extensive processing. - -Each job has an pre configured behaviour for the case of failure or deadline miss. -Any canceling and expireing jobs gets noted in *Statistics* to adjust performance and timings -for optimal performance and I/O throughput. - diff --git a/doc/design/lowlevel/index.txt b/doc/design/lowlevel/index.txt index e27f8cfa0..e4bf2de20 100644 --- a/doc/design/lowlevel/index.txt +++ b/doc/design/lowlevel/index.txt @@ -9,106 +9,8 @@ data access. Within Lumiera, there are two main kinds of data handling: * The Session and the object models manipulated through the GUI are kept in memory. They are backed by a _storage backend,_ which provides database-like storage and especially logging, replaying and ``Undo'' of all ongoing modifications.. -* Media data is handled _frame wise_ -- as described below. - -The vault layer (``backend'') uses *memory mapping* to make data available to the program. -This is somewhat different to the more common open/read/write/close file access, -while giving superior performance and much better memory utilization. -The Vault-Layer must be able to handle more data than will fit into the memory -or even address space on 32 bit architectures. Moreover, a project might access more files -than the OS can keep open simultaneously, thus the for _Files used by the Vault,_ it needs a -*FilehandleCache* to manage file handle dynamically. - -Which parts of a file are actually mapped to physical RAM is managed by the kernel; -it keeps a *FileMapCache* to manage the *FileMaps* we've set up. -In the End, the application itself only requests *Data Frames* from the Vault. - -To minimize latency and optimize CPU utilization we have a *Prefetch thread* which operates -a *Scheduler* to render and cache frames which are _expected to be consumed soon_. The intention -is to manage the rendering _just in time_. - -The prefetcher keeps *Statistics* for optimizing performance. - - -Accessing Files ---------------- - -+FileDescriptor+ is the superclass of all possible filetypes, it has a weak reference to a -+FileHandle+ which is managed in within the +FilehandleCache+. On creation, only the existence -(when reading) or access for write for new files are checked. The +FileDescriptor+ stores some -generic metadata about the underlying file and intended use. But the actual opening is done on demand. - -The _content of files is memory mapped_ into the process address space. -This is managed by +FileMap+ entries and a +FileMapCache+. - -File Handles -~~~~~~~~~~~~ -A +FilehandleCache+ serves to store a finite maximum number of +FileHandles+ as a MRU list. -FileHandles are managed by the +FilehandleCache+; basically they are just storing the underlying OS file -handles and managed in a lazy/weak way, (re)opened when needed and aging in the cache when not needed, -since the amount of open file handles is limited aged ones will be closed and reused when the system -needs to open another file. - -File Mapping -~~~~~~~~~~~~ -The +FileMapCache+ keeps a list of +FileMaps+, which are currently not in use and subject of aging. -Each +FileMap+ object contains many +Frames+. The actual layout depends on the type of the File. -Mappings need to be _page aligned_ while Frames can be anywhere within a file and dynamically sized. - -All established ++FileMap++s are managed together in a central +FileMapCache+. -Actually, +FileMap+ objects are transparent to the application. The upper layers will just -request Frames by position and size. Thus, the +File+ entities associate a filename with the underlying -low level File Descriptor and access - -Frames -~~~~~~ -+Frames+ are the smallest datablocks handled by the Vault. The application tells the Vault Layer to make -Files available and from then on just requests Frames. Actually, those Frames are (references to) blocks -of continuous memory. They can be anything depending on the usage of the File (Video frames, encoder frames, -blocks of sound samples). Frames are referenced by a smart-pointer like object which manages the lifetime -and caching behavior. - -Each frame referece can be in one out of three states: - -readonly:: - the backing +FileMap+ is checked out from the aging list, frames can be read - -readwrite:: - the backing +FileMap+ is checked out from the aging list, frames can be read and written - -weak:: - the +FileMap+ object is checked back into the aging list, the frame can't be accessed but we can - try to transform a weak reference into a readonly or readwrite reference - - -Frames can be addressed uniquely whenever a frame is not available. The vault can't serve a cached -version of the frame, a (probably recursive) rendering request will be issued. - -Prefetching -~~~~~~~~~~~ -There are 2 important points when we want to access data with low latency: - -. Since we handle much more data than it will fit into most computers RAM. - The data which is backed in files has to be paged in and available when needed. - The +Prefetch+ Thread manages page hinting to the kernel (posix_madvise()..) -. Intermediate Frames must eventually be rendered to the cache. - The Vault Layer will send +Renderjobs+ to the +Scheduler+. - -Whenever something queries a +Frame+ from the vault it provides hints about what it is doing. -These hints contain: - -* Timing constraints - - When will the +Frame+ be needed - - could we drop the request if it won't be available (rendered) in-time -* Priority of this job (as soon as possible, or just in time?) -* action (Playing forward, playing backward, tweaking, playback speed, recursive rendering of dependent frames) - -.Notes -* The Vault Layer will try to render related frames in groups. -* This means that following frames are scheduled with lower priority. -* Whenever the program really requests them the priority will be adjusted. - - --> more about link:Scheduler.html[the Scheduling of calculation jobs] - +* Media data is handled _frame wise_ -- prefetching data asynchronously. +The goal is to optimize CPU utilization; large scale data IO will be performed +asynchronously. Data retrieval and processing of prerequisites will +be *scheduled* such as to manage the rendering _just in time_. diff --git a/doc/devel/draw/DIR_INFO b/doc/devel/draw/DIR_INFO new file mode 100644 index 000000000..ebee2b3f7 --- /dev/null +++ b/doc/devel/draw/DIR_INFO @@ -0,0 +1 @@ +SVG Vectorgraphics - directly edited with Inkscape diff --git a/doc/devel/draw/build_svg/DIR_INFO b/doc/devel/draw/build_svg/DIR_INFO new file mode 100644 index 000000000..450db99db --- /dev/null +++ b/doc/devel/draw/build_svg/DIR_INFO @@ -0,0 +1,2 @@ +Scripts to produce graphics +Currently only using LaTeX/TikZ to produce graphics \ No newline at end of file diff --git a/doc/devel/draw/build_svg/TikZ/DIR_INFO b/doc/devel/draw/build_svg/TikZ/DIR_INFO new file mode 100644 index 000000000..e48810eef --- /dev/null +++ b/doc/devel/draw/build_svg/TikZ/DIR_INFO @@ -0,0 +1,4 @@ +LaTeX/TikZ source to build graphics in svg formnat +To run these, you'll require LaTeX and TikZ +LateX is available on Debian via the texlive package +TikZ is available froom the texlive-pictures package \ No newline at end of file diff --git a/doc/devel/draw/build_svg/TikZ/doc_map.tex b/doc/devel/draw/build_svg/TikZ/doc_map.tex new file mode 100644 index 000000000..7ce0d83be --- /dev/null +++ b/doc/devel/draw/build_svg/TikZ/doc_map.tex @@ -0,0 +1,76 @@ +% doc_map.tex: Generate map of how Lumiera developer documentation is organised +% glossary.sh - Lumiera RfC: +% +% Copyright (C) Lumiera.org +% 2023, Benny Lyons +% +% This program is free software; you can redistribute it and/or +% modify it under the terms of the GNU General Public License as +% published by the Free Software Foundation; either version 2 of +% the License, or (at your option) any later version. +% +% This program is distributed in the hope that it will be useful, +% but WITHOUT ANY WARRANTY; without even the implied warranty of +% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +% GNU General Public License for more details. +% +% You should have received a copy of the GNU General Public License +% along with this program; if not, write to the Free Software +% Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + +% +% To compile the graphics, you will require TikZ. +% On Debian, this is to be found in the texlive-pictures package. +% +\documentclass[dvisgvm]{article} +\usepackage{tikz} + + +\begin{document} + +\pagestyle{empty} + +\begin{tikzpicture} + [font=\footnotesize, + grow=right, + level 1/.style={sibling distance=8em}, + level 2/.style={sibling distance=0.8em},level distance=3.5cm] + + \node{\hbox{\vbox{ + \hbox{Lumiera (as seen)} + \hbox{from Outer Space} + } + } + } % root + child {node {\hbox{\vbox{ + \hbox{Technical Docs} + \hbox{The Inner Core} + } + } + } + child {node {Developer Howtos} } + child {node {Infrastructure} } + child {node {Code Base} } + child {node {Support Lib} } + child {node {Vault} child { node {API doc}}} + child {node {Steam} child { node {API doc}}} + child {node {Stage} child { node {API doc}}} + child {node {Build System} } + } + child {node {Design Docs} + child {node {Governance} } + child {node {Workflow} } + child {node {Engine} } + child {node {Low-level Model} } + child {node {Application} } + child {node {Plugins} } + child {node {GUI} } + child {node {Model} } + child {node {Architecture} } + }; + +\end{tikzpicture} + + + +\end{document} diff --git a/doc/devel/draw/build_svg/TikZ/makefile b/doc/devel/draw/build_svg/TikZ/makefile new file mode 100644 index 000000000..23fe0653b --- /dev/null +++ b/doc/devel/draw/build_svg/TikZ/makefile @@ -0,0 +1,47 @@ +# makefile - build graphic map illustrating the organisation of the Lumiera developer doc +# +# Copyright (C) Lumiera.org +# 2023, Benny Lyons +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of +# the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +# +# +# Do compile the graphics, you will require TikZ. +# On Debian, this is to be found in the texlive-pictures package. + +# +# Requirements +# tex, e.g., texlive +# TikZ (on Debian in the package tetxlive-pictures +# convert contained in Imagemagic +# It is possible the png file using other tools + +BUILDME=doc_map +$(BUILDME): $(BUILDME).pdf $(BUILDME).svg $(BUILDME).png + +$(BUILDME).pdf: $(BUILDME).tex + pdflatex $< $@ + +$(BUILDME).svg: $(BUILDME).dvi + dvisvgm $< +$(BUILDME).dvi: $(BUILDME).tex + latex $< +$(BUILDME).png: $(BUILDME).svg + convert -background none -size 1024x1024 $< $@ + + +:PHONEY: clean +clean: + @rm -f $(BUILDME).pdf $(BUILDME).svg $(BUILDME).dvi $(BUILDME).png diff --git a/doc/devel/images/doc_map.png b/doc/devel/images/doc_map.png new file mode 100644 index 000000000..1259e4aa6 Binary files /dev/null and b/doc/devel/images/doc_map.png differ diff --git a/doc/devel/meeting_summary/2023-09-13.txt b/doc/devel/meeting_summary/2023-09-13.txt new file mode 100644 index 000000000..b5e8e7eb8 --- /dev/null +++ b/doc/devel/meeting_summary/2023-09-13.txt @@ -0,0 +1,124 @@ +2023-09-13 Lumiera Developers Meeting +===================================== +:Author: Ichthyo, Benny +:Date: 2023-09-14 + +Sep 13, 2023 on #lumiera 20:00 - 22:30 UTC + + +__Participants__ + + * ichthyo + * benny + * cehteh + +_Summary written by Ichthyo and reviewed by Benny_ + +'''' + +Focus Topics +------------ +_Ichthyo_ started to define a collection of *FocusTopics*. +These are topics towards which we focus our attention for further development *after* the +current Engine integration is completed. Some tickets currently of interest were enhanced +and extended, and new tickets created, which were all marked with the `FocusTopic` Tag. + +-> See https://issues.lumiera.org/report/17[Report-17] in the Lumiera Issue Tracker + +_Ichthyo_ identified some difficult issues concerning session modifications and change logging, +especially in connection with rules based configuration. + +We discussed in some detail event sourcing, session modification, session configuration +and representation of a configuration possibly as a context in the session. Of particular +interest is the portability of a session from one machine to another and, consequently, +the problems that can arise when a change in configuration occurs: these should be +recorded as an event in the Event-Log. However, this requires the representation of the +entire configuration in a consolidated form as part of The Model. A change in the +configuration could then be detected and recorded as an event. + +Another task of concern is how the actual Diff Messages will be generated. We could first +manipulate the data model and then _detect_ a diff. Yet the favourable solution would be +to interpret the commands and directly derive the events associated with the commands, +with an attached diff, which can then be applied against the data model as a _projection._ + +A discussion followed to sort out which topics should be addressed first. + +It seems the topics regarding Event Log and Session persistence +https://issues.lumiera.org/ticket/1234[#1234] +https://issues.lumiera.org/ticket/540[#540] and +https://issues.lumiera.org/ticket/569[#569] +should be addressed first, since these may lead to decisions relevant to many other parts +of the system. Possibly also +https://issues.lumiera.org/ticket/332[#332] regarding a full definition of placement capabilities. + + +The topic of Placement interpretation will be addressed in the *next* vertical slice, +where we intend to add a clip to the timeline. To do this, we'll require at least a +_simple_ resolution capability of Placements, because we'll attempt do the groundwork for +The Builder. + +_Benny_ proposes to add »interface« and »mock« tickets for Event-Log and Placements, +blocking the full implementation follow-up tickets. Then these preliminary parts can be +addressed first and integrated as part of the »Add Clip« Vertical Slice. For Context: +currently the »Playback« Vertical Slice is being worked on, aiming towards a first +integration of the Render Engine. + + +Playback Vertical Slice +----------------------- +_Ichthyo_ reports on the progress of the »Playback Vertical Slice« effort. + +- Memory Manager is implemented and tested +- Activity Language is also completed and tested +- a passive worker pool has been implemented, based on the C++17 concurrency framework + +Next steps: + +- retrofit our existing thread wrapper (which was based on POSIX) to use C\+\+17 thread support; + this should be a drop-in replacement, since our own old design almost completely matches the + design and feature set standardised with C++17 +- complete implementation of the Scheduler Layer-2, integrating Activity-Language and worker pool +- integration test of The Scheduler + +Documentation Index and Glossary +-------------------------------- +_Benny_ reworked and extended the documentation overview in the Lumiera website. +This can be published now. +Moreover, _Benny_ investigated Glossary and Index generation. It turns out that +ASCIIDOC already provides an index generation feature, which could be configured and used +to build an index. But there is no support to generate a glossary which we'll build soon. +The next goal would be to build a prototype for testing and to see how this can be +integrated into the website infrastructure. + +Library solutions for Video Display +----------------------------------- +_Benny_ conducted some research regarding library solutions for *video display*, notably +ffmpeg, GStreamer and libVLC. It was rather easy to code up an "hello world" example based on +GStreamer, while it was difficult to find suitable documentation regarding the ffmpeg based +solution. Moreover, GStreamer seems to have a much more active and helpful community. + +Next steps would be to look in more depth into libVLC, and especially to compare the +feature sets of these solutions. _Cehteh_ points out that MPV (formerly MPlayer) would also +be an interesting point of reference. _Benny_ mentions that GTK4 offers an integrated +video display capability. We'll investigate these frameworks further. + +Extended discussion ensued regarding video frameworks and the difficulties regarding the plethora +of interfaces, especially when it comes to hardware acceleration support. Ideally, the Lumiera +project should follow the solutions provided by a sufficiently active community, capable of following +the constant evolution of interfaces and frameworks. + +It was agreed upon that the »Playback Vertical Slice« just needs a very simple solution to output +raw video (RGB888 or YUV), and offers a good opportunity for experimentation to find a suitable +support framework. + + +'''' + +Next meeting +------------ + +The next meeting will be Wednesday Oct 11, 20:00 UTC. +You are welcome to join. + +'''' + + diff --git a/doc/devel/meeting_summary/2023-10-11.txt b/doc/devel/meeting_summary/2023-10-11.txt new file mode 100644 index 000000000..e0c2fd4ec --- /dev/null +++ b/doc/devel/meeting_summary/2023-10-11.txt @@ -0,0 +1,71 @@ +2023-10-11 Lumiera Developers Meeting +===================================== +:Author: Ichthyo +:Date: 2023-10-11 + +Oct 11, 2023 on #lumiera 20:00 - 22:00 UTC + + +__Participants__ + + * ichthyo + * benny + +_Summary written by Ichthyo_ + +'''' + +Development Summary +------------------- +_Ichthyo_ summarises the progress regarding the »Playback Vertical Slice«. + +- the design now favours _active_ workers and a _passive_ pool +- the Scheduler now features _two layers_ and provides also some support for dependency management +- there is a direct collaboration between workers and the Scheduler Layer-2 +- thus the Scheduler takes on some aspects of work management, in collaboration with the workers + +As a follow-up, the state of the C\++ concurrency framework was +https://issues.lumiera.org/ticket/1279#comment:2[investigated] and compared with the features +provided by some of our own ``homegrown'' support library. A lot of modern concurrency support +facilities are meanwhile provided by the standard library of the language, and so we can abandon +some of our old solutions to favour a modern approach. This investigation led to a complete rework +of our thread-wrapper. Previously we had to implement a lot of our threading support ourselves, +based on POSIX threading. The same feature set can now be achieved by delegating to `std::thread` +from the C++14 language standard. Design analysis showed however that the application can still +benefit from using a custom adaptation layer on top, especially for uniform error handling and +to support lifecycle management of components executed concurrently. This allows especially to +simplify some load tests performed as part of the Lumiera test-suite. + + +Extended discussion ensued regarding the modern concurrency primitives (Atomics, Future+Promise, +Coroutines), and their relation to the established long-standing solutions (Mutex, Condition Var). +There is no single ``best'' solution, and each of these techniques to manage concurrency presents +the developer with an unique set of benefits and trade-offs. + +For Lumiera it is important to assess the right order of magnitude regarding the precision of timings. +The duration of one video frame at 25fps is 40ms, which implies that the scheduler must be reliable +with its timings in the order of magnitude of 1ms. Thus it is desirable that the internal overhead +is kept below 100µs. This goal is not difficult to achieve; it suffices to pay some attention to +aspects of performance. To place this into context, single instructions in a modern CPU operate +at the nanoseconds level. + + +Compiler and Platform upgrades +------------------------------ +Most of everyday development is done currently with GCC-8, which is rather dated already. +Compilation with GCC-12 has been attempted recently, and did not turn up any serious problems. +Two minor bugs could be fixed. With the planned system upgrade on _Ichthyo's_ developer PC +most notably the build system *Scons* now uses Python-3. The necessary adjustments are mostly +repetitive, and were committed by _Benny_ to a Git branch. Previously, the Lumiera build system +used a ready-made plug-in to start *Doxygen*. This code does not work any more with Python-3, +but a replacement solution was identified, which still needs to be integrated. + + +'''' + +Next meeting +------------ + +The next meeting will be Wednesday Nov 8, 20:00 UTC. +You are welcome to join. + +'''' diff --git a/doc/devel/meeting_summary/2023-11-08.txt b/doc/devel/meeting_summary/2023-11-08.txt new file mode 100644 index 000000000..d72c55748 --- /dev/null +++ b/doc/devel/meeting_summary/2023-11-08.txt @@ -0,0 +1,99 @@ +2023-11-08 Lumiera Developers Meeting +===================================== +:Author: Ichthyo +:Date: 2023-11-09 + +Nov 8, 2023 on #lumiera 22:30 - 23:30 UTC + + +__Participants__ + + * ichthyo + * benny + * cehteh + +_Summary written by Ichthyo_ + +'''' + +Wikipedia +--------- +https://en.wikipedia.org/wiki/User:IgorVladimirsky[Igor Vladimirsky] +contacted _Ichthyo_ last week regarding a mention of Lumiera in the context of Cinelerra. +He was involved with the +https://cinelerra-cv.wikidot.com[Cinelerra-CV project] +and already contributed several improvements to the Wikipedia entry for Cinelerra, +which includes a mention of »the Lumiera rewrite« since +https://en.wikipedia.org/w/index.php?title=Cinelerra&diff=prev&oldid=207875710[April 2008]. +_Igor_ rightfully pointed out that Lumiera is not a fork of Cinelerra and the description of the project's +origin might be too much biased towards the developer's perspective at that time. This critique extends also +to the descriptions at the introductory pages of the Lumiera website itself; these characterisations were +written during the same founding phase and not much reviewed ever since. After some exchange of proposals +and ideas, an update and reformulation of the introductory pages was accomplished. The update to the +content at Wikipedia, which _Igor_ thankfully completed, was a good opportunity to add some further +sources, notably an archive.org link to an article at +https://web.archive.org/web/20090129083001/https://www.linux.com/feature/126441[Linux.com] +and remove a ^[citation needed]^ tag. + +-> see the reworked +{l}/project/index.html[``About''] and {l}/project/faq.html[``FAQ''] pages + +-> section in the +https://en.wikipedia.org/wiki/Cinelerra#Lumiera[Wikipedia page for Cinelerra]. + + +Development Summary +------------------- +As usual, _Ichthyo_ summarises the progress of coding work under the umbrella of the »Playback Vertical Slice«. +Activities were focussed on integration of developed components and parts forming the *Scheduler Service* +at the core of the Render Engine. Some further aspects were addressed, notably the proper use of locking +and a new scheme to distribute available computing capacity in accordance to the planned schedule and +an integrated load indicator. The Scheduler implementation is now largely complete and passes the +component integration tests. The level of performance observed using short bursts of load seems adequate +and scheduling delay is mostly dominated by the leeway of the OS thread scheduling and caching effects. + +The next step would be to setup some meaningful load- and stress testing, using a synthetic load, +which can be adjusted to reflect typical patterns of the expected load in actual usage. Care has +to be taken however to distil an indicator actually describing the scheduling overhead and to avoid +capturing just the overall system performance. + + +Investigation: Video Display +---------------------------- +_Benny_ continued his research regarding solutions for video display in the GUI. We agreed previously +that the current ``vertical slice'' affords the opportunity to try out available mainstream solutions +to find out more about their strength and weaknesses. _Benny_ decided to look at *GStreamer* first, +since this framework is in very widespread use, with tutorials and documentation readily available. +He conducted some experiments and was able to put together a demo which decodes an AVI file and +displays the video content into a given GTK window. The following investigations will focus on +the details how to inject arbitrary content into such a display, without directly relying on the +complete decoding and rendering pipeline of GStreamer, since we intend to use the Lumiera Render +Engine to coordinate all processing and manage all resources. + +Another point worth mentioning is the fact that the GTK project intends to discontinue support +of the Xlib / X.org output backend, focussing entirely on Wayland. GStreamer fully supports +Wayland already. + +_Ichthyo_ adds that he intends to use directly generated video content for this ``vertical slice'', +so to avoid involving any file-IO. It seems quite feasible to use libCairo (which is already a project +dependency and well known) do draw some simple animation. This code could than be packaged as a generator +node and used in the more or less synthetic node graph, which will be provided by the setup for this +integration effort. + +Coding News +----------- +During all the past years, it was customary to publish a ``coding news'' report at fall time. +There was much progress in the Lumiera project lately, since _Ichthyo_ was able to quit his regular +work job this spring to focus entirely on his personal programming and artwork projects. The +``vertical slice'' however will not be completed this year, and thus we agreed to author the usual +news update now, while providing an additional report once the work will be completed and we are +able to demonstrate running video display in the GUI. + + +'''' + +Next meeting +------------ + +The next meeting will be Wednesday Dec 13, 20:00 UTC. +You are welcome to join. + +'''' diff --git a/doc/devel/meeting_summary/2024-01-10.txt b/doc/devel/meeting_summary/2024-01-10.txt new file mode 100644 index 000000000..847e911b1 --- /dev/null +++ b/doc/devel/meeting_summary/2024-01-10.txt @@ -0,0 +1,87 @@ +2024-01-10 Lumiera Developers Meeting +===================================== +:Author: Benny +:Date: 2024-01-20 + +Jan 10, 2024 +on {l}/project/contact.html#_irc[`#lumiera`] +20:00 - 23:00 UTC + + +__Participants__ + + * ichthyo + * benn + +'''' +Two main points of discussion: + + - Scheduler tests to examine its suitability + - Using Gstreamer to view video for our vertical slice + + +Test Scheduler Suitability +-------------------------- + +Once the scheduler had been implemented it was necessary to subject it +to various load tests to establish its stability, and stress tests were necessary to +provide information on performance limitations so that we would be able to establish +whether these limitations met the demands required by Lumiera. +Results were encouraging. + +_Ichthyo_ reported on stress tests on the new scheduler. After resolving a series +of initial bugs, processing runs stable now, even when entering over 500 test jobs +simultaneously to the queue. Stress tests revealed typical limitations of the scheduler, +which exhibited behaviour that is well within the requirements of Lumiera. Load tests +showed that the scheduler remained stable when subjected to short heavy load bursts +within expected bounds. + +Indeed performance testing is always a delicate and complex area to +investigate, as it is necessary to establish reasonable performance boundaries +of the system: we can always overload any scheduler with fanatically huge numbers +of jobs; while the scheduler is still able to process all enqueued jobs eventually, +congestion builds up and jobs start missing their deadlines. Of interest, however, +is to establish some reasonable information on the processing rate of the scheduler +which _Ichthyo_ was able to estimate statistically by carefully observing the +scheduler under a collection of pre-calibrated loads. + +Some goals of the tests: + + - establish the scheduler breaking point or performance limitations + - tests under pressure with parallelism and complex dependencies between jobs + - demonstrate that the scheduler is stable under typical load for several seconds + +Tests on a wider array of hardware constellations are not expected to reveal much +relevant information in this stage of the project and will thus be conducted later. +This kind of analytical, isolated component tests will not ultimately resolve the +issue of whether the scheduler can perform as desired, but subjecting the scheduler +to carefully selected tests can help to reduce the probability of missing serious +performance limitations inherent to the chosen design. + + +GStreamer Demo to Play Video +---------------------------- + +_Benn_ has written a simple GStreamer demo so that a video in raw format can be +displayed on the GUI for the current vertical slice. The next step in this +investigation is to identify a low-level entrance point to the functionality +provided by the GStreamer library, since for use in this vertical slice the +render engine produces a buffer filled with raw video data for each frame, +which will then be displayed in a video widget as an ongoing process. + +This is a general question requiring further investigation: how to make use of +established functionality from a library to simplify our implementation task, +while retaining the full control and coordination of the rendering and playback +process and resource management within the Lumiera render engine itself. + +'''' + +Next meeting +------------ + +The next Lumiera meeting is scheduled for Wednesday, Feb 14 at 20:00 UTC. +Should this be inconvenient for you, please speak up on, for example, +link:http://lists.lumiera.org/cgi-bin/mailman/listinfo/lumiera[lumiera@lists.lumiera.org] +and let us know of when you would prefer to attend. + +'''' + diff --git a/doc/devel/meeting_summary/index.txt b/doc/devel/meeting_summary/index.txt index 2ca5b0411..2c693dcbd 100644 --- a/doc/devel/meeting_summary/index.txt +++ b/doc/devel/meeting_summary/index.txt @@ -11,8 +11,113 @@ spending time on the task of writing a summary. Anyone interested in Lumiera development is also encouraged to read mailing list archives and other documentation. +8 May / 12 June 2024 +-------------------- +Discussion regarding the approach taken towards the next core topic, which +is how to structure the render nodes in the low-level model and how to perform +an invocation, leading to the calculation of media data. It is considered very +important for Lumiera to remain library / framework agnostic, even while we +use some frameworks and libraries for prototyping. The structure drafted over +the course of the last years and about to be integrated now must be flexible +enough to support adapt to the requirements of various libraries, _without +attempting to build a super structure_ to ``rule them all''. -12 Sep 2012 + +13 Mar / 10 Apr 2024 +-------------------- +Status update and informal exchange regarding current development activities + + * after factoring out further external influences, a _breaking point_ for + processing complex dependent schedules could be established, which closely + matches theoretical expectations + * a further measurement series (linear regression) was necessary to establish + performance characteristics of the scheduler under constant high load + * as expected, a serious weakness pertaining contention could be confirmed + + (yet contention is not expected to be a problem in practice) + * as mitigation, concurrency is scaled down on very short (100µs) jobs + * overall we are confident that the new scheduler is capable to handle the + expected load pattern and does not suffer from major (unknown) weaknesses. + +Moreover, the problems related to integration and use of external frameworks +for video display were discussed further. + + +14 Feb 2024 +----------- + +Topics +~~~~~~ + * Discussion about GTK\+, Python bindings vs. C++ bindings vs. the C-API, + challenges due to the dynamic nature of layout and inconsistent CSS settings, + and further difficulties integrating video display. + * Update regarding the ongoing effort to establish performance limits of the + Scheduler and discussion about methods of measurement. + +10 Jan 2024 +----------- + +Topics +~~~~~~ + * Scheduler performance testing + * using GStreamer for Video output to the GUI + +Summary +^^^^^^^ + * link:2024-01-10.html[Summary (by benn)] + + +13 Dec 2023 +----------- + +Topics +~~~~~~ + * Extended technical discussion about Scheduler load testing + and how to find out about inherent limitations of the approach taken + +8 Nov 2023 +---------- + +Topics +~~~~~~ + * Wikipedia + * Scheduler integration and testing + * Video output to the GUI + +Summary +^^^^^^^ + * link:2023-11-08.html[Summary (by ichthyo)] + + +11 Oct 2023 +----------- + +Topics +~~~~~~ + * Scheduler and worker pool design + * Rework of multithreading support + * Compiler and platform upgrades + +Summary +^^^^^^^ + * link:2023-10-11.html[Summary (by ichthyo)] + + +13 Sep 2023 +----------- + +Topics +~~~~~~ + * »Focus Topics« + * Progress on the »Playback Vertical Slice« + * Website Index and Glossary generator + * Library solutions for video display in the GUI + +Summary +^^^^^^^ + * link:2023-09-13.html[Summary (by ichthyo)] + + +12 Sep 2013 ----------- Topics diff --git a/doc/devel/rfc/AllPluginInterfacesAreC.txt b/doc/devel/rfc/AllPluginInterfacesAreC.txt index 5408d2f23..68c1bf7cb 100644 --- a/doc/devel/rfc/AllPluginInterfacesAreC.txt +++ b/doc/devel/rfc/AllPluginInterfacesAreC.txt @@ -3,7 +3,7 @@ Design Process : All Plugin Interfaces Are C [grid="all"] `------------`----------------------- -*State* _Final_ +*State* _Dropped_ *Date* _2007-06-29_ *Proposed by* link:ct[] ------------------------------------- @@ -159,17 +159,50 @@ After a talk on IRC ichthyo and me agreed on making lumiera a multi language project where each part can be written in the language which will fit it best. Language purists might disagree on such a mix, but I believe the benefits outweigh the drawbacks. - -- link:ct[] [[DateTime(2007-07-03T05:51:06Z)]] + +ct:: '2007-07-03 05:51:06' C is the only viable choice here. Perhaps some sort of "test bench" could be designed to rigorously test plugins for any problems which may cause Lumiera to become unstable (memory leaks etc). - -- link:Deltafire[] [[DateTime(2007-07-03T12:17:09Z)]] + +Deltafire:: '2007-07-03 12:17:09' after a talk on irc, we decided to do it this way, further work will be documented in the repository (tiddlywiki/source) - -- link:ct[] [[DateTime(2007-07-11T13:10:07Z)]] + +ct:: '2007-07-11 13:10:07' +Development took another direction over course of the years; +Lumiera is not based on a _generic plug-in Architecture_ and the topic +of interfaces for _dedicated plugins_ still needs to be worked out + +Ichthyostega:: '2023-10-24 22:55:23' + + + +Conclusion +---------- +Initially there was agreement over the general direction set out by this proposal. +However, _Ichthyo_ was always sceptical regarding the benefits of a generic plug-in +Architecture. Experience with high-profile projects based on such a concept seem +to show both tremendous possibilities, especially regarding user involvement, but +at the same time also indicate serious problems with long-term sustainability. + +The practical development -- mostly driven ahead by _Ichthyo_ -- thus never really +embraced that idea; rather structuring by internal interfaces and contracts was +preferred. The basic system for loading of Plug-ins, as indicated by this proposal, +is still used though to load some dedicated plug-ins, most notably the GUI. + +To draw an conclusion: this proposal is now considered as *rejected*. +Instead, Ticket https://issues.lumiera.org/ticket/1212[#1212 »Extension Concept«] +was added to the list of relevant +https://issues.lumiera.org/report/17[»Focus Topics«] for further development. + +......... + +......... + '''' Back to link:/documentation/devel/rfc.html[Lumiera Design Process overview] diff --git a/doc/technical/vault/ConfigLoader.txt b/doc/devel/rfc/ConfigLoader.txt similarity index 86% rename from doc/technical/vault/ConfigLoader.txt rename to doc/devel/rfc/ConfigLoader.txt index 9f60e3932..27ef21b42 100644 --- a/doc/technical/vault/ConfigLoader.txt +++ b/doc/devel/rfc/ConfigLoader.txt @@ -1,116 +1,237 @@ -ConfigLoader -============ -:Author: ct -:Date: 2008-08-07 +Config Loader +============= +:toc: -link:Lumiera/ConfigLoader[] is a brainstorming about our configuration file loader. +// please don't remove the //word: comments -Table of Contents ------------------ +[grid="all"] +`------------`----------------------- +*State* _Parked_ +*Date* _2008-08-07_ +*Proposed by* **ct** +------------------------------------- -. xref:anchor_api[API] - .. xref:anchor_config[lumiera_configfile] - .. xref:anchor_types[provided TYPES] - .. xref:anchor_syntax[Syntax/Semantics] - * xref:anchor_proposal[Syntax Proposal by cehteh] - .. xref:anchor_links[links] - .. xref:anchor_internals[internals/implementation sketch] -. xref:anchor_discussion[Discussion on IRC] -. xref:anchor_brainstorm[Brainstorming] - -anchor:anchor_api[]API ----------------------- - -* Namespace is lumiera_config_* -* `struct lumiera_config_struct` for the config system, only one object needed -* `lumiera_config_init (...)` - - does only initialize the variables, so that they get valid values, but does not allocate them as they will be allocated before as they are singleton. - - lumiera_config_init (const char* searchpath) searchpath is a buildin-default, can be changed via configure and can be appended and overridden by using a flag, e.g. `--config-path-append=""` or `--config-path=""` - -* `lumiera_config_destroy(...)` - - frees all space allocated by the link:ConfigLoader[]. - -* `lumiera_config_load(...)` - - reads *one* single configuration file that will include all settings from other files. - - does not read itself but give delegates reading. The actual reading and parsing will be done in configfile object. s.later. - -* `lumiera_config_save(...)` - - saves all the changed settings to user's configuration files, but recognizes where settings came from and will write them to an appropriate named file. Example: *changed* values from `'/usr/local/share/lumiera/plugins/blur.conf'` will be saved into `'~/.lumiera/plugins/blur.conf'` - - finds out which files are dirty and which settings have to be written to user's config files. - - does initiate the actual saving procedure by delegating the save to the actual configfile objects, see below. - - empty user configuration files in RAM will be deleted from disk on write. - - checks whether the file has changed since last read, and will print out an error if necessary instead of overriding it without notification. +******************************************************************************** +.Abstract +Load application configuration from plaintext files with INI syntax. +Build an elaborate system on top to combine several configuration files. +******************************************************************************** + +Description +----------- +//description: add a detailed description: +After some initial discussion, it soon became clear that Lumiera needs to load +configuration in some way. A lot of things should be configurable, but the +syntax and implementation should be kept simple to make it easy to use + + +Tasks +~~~~~ +// List what needs to be done to implement this Proposal: + * initial implementation draft ([green]#✔ done#) +// * second step [yellow-background]#WIP# + * extended requirement analysis [red yellow-background]#TBD# + + +Discussion +~~~~~~~~~~ +NOTE: (by _Ichthyostega,_ 2023) + + Indeed this never became an actual RfC, and was rather treated as + implementation prototype. With ongoing development, this draft + became sidelined, and for the time being, it rests as unused code + in the code base. Some accompanying documentation pages + are hereby attached for later referral. + + + +Parked +~~~~~~ +While the general idea seams adequate, the topic of Configuration +requires a much more in-depth Requirement Analysis. Moreover, considering +the state of tooling today (2023), I consider it questionable still to rely +on plain-C to implement configuration handling, given that the C++ Language +offers a real `string` data type and is complemented by an extensive, safe +and performant string handling library, including even regular expressions. + +In a recent +link:{ldoc}/devel/meeting_summary/2023-09-13.html[Developer Meeting (IRC)] +we decided to include this topic into a list of +https://issues.lumiera.org/report/17[»Focus Topics«], +to be addressed _after achieving the first integration_ of the Engine. + +-> see https://issues.lumiera.org/ticket/1224[#1224 Layered Configuration] + +Ichthyostega:: '2023-10-24' ~~ + + + +//Conclusion +//---------- +//conclusion: When approbate (this proposal becomes a Final) +// write some conclusions about its process: + + + +'''''''''''''''' + +[big]#🙞 🙜# + + + +The incomplete Guide to Lumiera Configuration +--------------------------------------------- + +Lumiera uses plaintext files with a INI file like syntax for +configuration. This Syntax is strictly line based. There are only a +few syntactic elements. + +TODO:describe config syntax here + +Config Subsystem +~~~~~~~~~~~~~~~~ + +The path where Lumiera searches its configuration. Single components are +separated by colons as in PATH and other such environment variables. +Here it might be handy that any Lumiera configuration can be +overridden by a environment variable: +'LUMIERA_CONFIG_PATH=somewhere:else lumiera ...' +A default are initialized at installation time, this is important to +bootstrap the whole configuration system. + + config.path + + +The config system check for a preferred format when writing config +entries. For each key 'foo.bar', these can be overridden with a key +'config.format.foo.bar' linking to the desired format. + + config.formatkey ='config.format.%s' + + +The following are links to the default formatting when no explicit +format is set for a key. Changing these to a wrong type will break the +system! + + config.formatdef.link < config.formatstr.link + config.formatdef.number < config.formatstr.number.dec + config.formatdef.real < config.formatstr.real + config.formatdef.string < config.formatstr.string + config.formatdef.word < config.formatstr.word + config.formatdef.bool < config.formatstr.bool + + +This are the low level formating specifications for the buildin +types, DONT TOUCH THESE! + + config.formatstr.link = '< %s' + config.formatstr.number.dec = '= %lld' + config.formatstr.number.hex = '= 0x%llX' + config.formatstr.number.oct = '= 0%llo' + config.formatstr.real = '= %Lg' + config.formatstr.real.dec = '= %Lf' + config.formatstr.real.sci = '= %Le' + config.formatstr.string = '= %s' + config.formatstr.string.dquoted = '= \"%s\"' + config.formatstr.string.quoted = '= ''%s''' + config.formatstr.word = '= %s' + config.formatstr.bool = '= %d' + + +API +~~~ + +* Namespace is lumiera_config_* +* `struct lumiera_config_struct` for the config system, only one object needed +* `lumiera_config_init (...)` + - does only initialize the variables, so that they get valid values, but does not allocate them as they will be allocated before as they are singleton. + - lumiera_config_init (const char* searchpath) searchpath is a buildin-default, can be changed via configure and can be appended and overridden by using a flag, e.g. `--config-path-append=""` or `--config-path=""` + +* `lumiera_config_destroy(...)` + - frees all space allocated by the link:ConfigLoader[]. + +* `lumiera_config_load(...)` + - reads *one* single configuration file that will include all settings from other files. + - does not read itself but give delegates reading. The actual reading and parsing will be done in configfile object. s.later. + +* `lumiera_config_save(...)` + - saves all the changed settings to user's configuration files, but recognizes where settings came from and will write them to an appropriate named file. Example: *changed* values from `'/usr/local/share/lumiera/plugins/blur.conf'` will be saved into `'~/.lumiera/plugins/blur.conf'` + - finds out which files are dirty and which settings have to be written to user's config files. + - does initiate the actual saving procedure by delegating the save to the actual configfile objects, see below. + - empty user configuration files in RAM will be deleted from disk on write. + - checks whether the file has changed since last read, and will print out an error if necessary instead of overriding it without notification. + -`lumiera_config_save () { LLIST_FOREACH(config_singleton.files, f) { LumieraFile file = (LumieraFile) f; if(lumiera_configfile_isdirty (file)) lumiera_configfile_save(file); } }` - -* `lumiera_config_purge(const char* filename)` removes all configs loaded from filename * `lumiera_config_get(...)` - - get a value by key - - handles internally everything as string:string key:value pair. - - lowlevel function - - lumiera_config_integer_get (const char* key, int *value) will return integers instead of strings and return 0 if succeeded and -1 if it failed. - - - 'int lumiera_config_TYPE_get(const char* key, TYPE* value, const char* default)' - High level config interface for different types. - if default is given (!NULL) then value is set to default in case key was not found or any other error occured. - error code is still set and -1 (fail) is returned in case of an error, but it might be cleared with no ill effects. - NOTE: errors are persistent in our error handler, they must still be cleared, even when ignored. - if default is given then `KEY_NOT_FOUND` is not a error here, if default is NULL then it is - NOTE2: default values are given as strings, the config loader remembers a given default value and checks if it got changed - when it is _set(). Thus a default value can be suppressed when set/written - -* `lumiera_config_set(...)` - - set a value by key - - handles internally everything as string:string key:value pair. - - lowlevel function - - tag file as dirty - - set will create a new user configuration file if it does not exist yet or will append a line to the existing one in RAM. These files, tagged as \'dirty', will be only written if save() is called. - - `lumiera_config_TYPE_set (const char* key, TYPE*value, const char* fmt)` - Highlevel interface for different types, fmt is a printf format specifier for the desired format, when NULL, defaults apply. - -* `lumiera_config_reset(...)` - - reset a value by key to the system default values, thus removes a user's configuration line. - -* `lumiera_config_info (const char* key, const char** filename, unsigned* line)` - - Find exact place of a setting. +`lumiera_config_save () { LLIST_FOREACH(config_singleton.files, f) { LumieraFile file = (LumieraFile) f; if(lumiera_configfile_isdirty (file)) lumiera_configfile_save(file); } }` + +* `lumiera_config_purge(const char* filename)` removes all configs loaded from filename + +* `lumiera_config_get(...)` + - get a value by key + - handles internally everything as string:string key:value pair. + - lowlevel function + - lumiera_config_integer_get (const char* key, int *value) will return integers instead of strings and return 0 if succeeded and -1 if it failed. + + - 'int lumiera_config_TYPE_get(const char* key, TYPE* value, const char* default)' + High level config interface for different types. + if default is given (!NULL) then value is set to default in case key was not found or any other error occured. + error code is still set and -1 (fail) is returned in case of an error, but it might be cleared with no ill effects. + NOTE: errors are persistent in our error handler, they must still be cleared, even when ignored. + if default is given then `KEY_NOT_FOUND` is not a error here, if default is NULL then it is + NOTE2: default values are given as strings, the config loader remembers a given default value and checks if it got changed + when it is _set(). Thus a default value can be suppressed when set/written + +* `lumiera_config_set(...)` + - set a value by key + - handles internally everything as string:string key:value pair. + - lowlevel function + - tag file as dirty + - set will create a new user configuration file if it does not exist yet or will append a line to the existing one in RAM. These files, tagged as \'dirty', will be only written if save() is called. + - `lumiera_config_TYPE_set (const char* key, TYPE*value, const char* fmt)` + Highlevel interface for different types, fmt is a printf format specifier for the desired format, when NULL, defaults apply. + +* `lumiera_config_reset(...)` + - reset a value by key to the system default values, thus removes a user's configuration line. + +* `lumiera_config_info (const char* key, const char** filename, unsigned* line)` + - Find exact place of a setting. [NOTE] -======================================= -* multiple config items with the same key stack (to allow undo/removing custom configs) -* same keys in the same file stack also, the _get api returns the last value the _set api sets the last value -* we might need to implement some cursor api to retrieve all values with the same key, this should allow to filter keys from single files! -* if one queries a setting with a default, this default should be stored in the configuration system without an backing file on disk. Rationale is, that when one later _set's a value and this value is found to be equivalent to the default, this _set option is a no-op -======================================= - +======================================= +* multiple config items with the same key stack (to allow undo/removing custom configs) +* same keys in the same file stack also, the _get api returns the last value the _set api sets the last value +* we might need to implement some cursor api to retrieve all values with the same key, this should allow to filter keys from single files! +* if one queries a setting with a default, this default should be stored in the configuration system without an backing file on disk. Rationale is, that when one later _set's a value and this value is found to be equivalent to the default, this _set option is a no-op +======================================= + .foo.bar -======================================= -`integer_set("foo.bar", 123)` -looks up "foo.bar" -this returns as well file and line -it will be decided whether it's a system file or a user's config file. -if systemfile: loop for user's config file, if necessary create it in our RAM but not yet on disk -set the value in the line. -======================================= +======================================= +`integer_set("foo.bar", 123)` +looks up "foo.bar" +this returns as well file and line +it will be decided whether it's a system file or a user's config file. +if systemfile: loop for user's config file, if necessary create it in our RAM but not yet on disk +set the value in the line. +======================================= anchor:anchor_config[]lumiera_configfile ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -* `lumiera_configfile_struct` - - contains a `llist` with all lines stored as `lumiera_configline_struct` - - contains an integer that indicates how \'dirty' the file is, i.e. how many settings were changed. (one increment per _set() call) - - contains a stat struct for comparison while saving if the file was altered since last read - -* `lumiera_configline_struct` - - stores the line exactly as read from file, but holds as well pointers to the key and the value. - -* lumiera_configfile_load -* lumiera_configfile_save -* lumiera_configfile_new -* lumiera_configfile_delete -* lumiera_configfile_init -* lumiera_configfile_destroy - - + +* `lumiera_configfile_struct` + - contains a `llist` with all lines stored as `lumiera_configline_struct` + - contains an integer that indicates how \'dirty' the file is, i.e. how many settings were changed. (one increment per _set() call) + - contains a stat struct for comparison while saving if the file was altered since last read + +* `lumiera_configline_struct` + - stores the line exactly as read from file, but holds as well pointers to the key and the value. + +* lumiera_configfile_load +* lumiera_configfile_save +* lumiera_configfile_new +* lumiera_configfile_delete +* lumiera_configfile_init +* lumiera_configfile_destroy + + anchor:anchor_types[]provided TYPES ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -125,479 +246,481 @@ string const char* optionaly quoted string (foo.bar="this string" #this not) word const char* first optionaly quoted word of the config (foo.bar= first this # not) bool int common representations for bools (0,1,true,false,yes,no,on,off,set,clear) ------------------------------------------------------------------------------------------------------------------- - -... add more types on demand (ichthyo mentioned a tuple type for color triples etc.) - -config files/strings are utf-8! - + +... add more types on demand (ichthyo mentioned a tuple type for color triples etc.) + +config files/strings are utf-8! + anchor:anchor_syntax[]Syntax/Semantics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -...to be defined... - + +...to be defined... + in short: - -* configfiles allow comments, -* some kind of include directive -* includes happen only \'once', its impossible to create include recursions -* config settings are key:value pairs where the key is a hierachical string as in "foo.bar.baz" - + +* configfiles allow comments, +* some kind of include directive +* includes happen only \'once', its impossible to create include recursions +* config settings are key:value pairs where the key is a hierachical string as in "foo.bar.baz" + anchor:anchor_proposal[]Syntax Proposal by cehteh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -simple example: ------------------------------------------------------------- -@directive options -key = value -[prefix] -[prefix suffix] ------------------------------------------------------------- - -ebnf: +simple example: + +------------------------------------------------------------ +@directive options +key = value +[prefix] +[prefix suffix] +------------------------------------------------------------ + +ebnf: + +------------------------------------------------------------ +configline ::= [ whitespace ] , { comment | directive | configentry | section } , ? end of line ? ; + +whitespace ::= ? one to many tab or blank ? ; + +comment ::= "#" , ? arbitary text ? ; + +directive ::= "@", ? commandword ? , whitespace, ? arguments ? ; + +configentry ::= key, { whitespace } , assignop, value ; + +key ::= ? lowercase characters and _ ? , [ "." , key ] ; + +assignop ::= " " | ":" | "=" ; + +value ::= ? arbitary text ? ; + +section ::= "link:, [ whitespace ] , prefix, [ whitespace, suffix ] , [ whitespace ] []" | emptysection ; + +emptysection ::= "[]" ; +prefix ::= key ; +suffix ::= key ; +------------------------------------------------------------ ------------------------------------------------------------- -configline ::= [ whitespace ] , { comment | directive | configentry | section } , ? end of line ? ; - -whitespace ::= ? one to many tab or blank ? ; - -comment ::= "#" , ? arbitary text ? ; - -directive ::= "@", ? commandword ? , whitespace, ? arguments ? ; - -configentry ::= key, { whitespace } , assignop, value ; - -key ::= ? lowercase characters and _ ? , [ "." , key ] ; - -assignop ::= " " | ":" | "=" ; - -value ::= ? arbitary text ? ; - -section ::= "link:, [ whitespace ] , prefix, [ whitespace, suffix ] , [ whitespace ] []" | emptysection ; - -emptysection ::= "[]" ; -prefix ::= key ; -suffix ::= key ; ------------------------------------------------------------- - semantics notes: - -* The format is strictly line-based -* directives inside non empty sections are not yet defined -* sections can't nest so far (idea: thinking about `[ .extend.with.leading.dot ]`) -* section semantics: a key inside a section `[foo bar]` as in `key=value` is treated as `foo.key.bar=value` - + +* The format is strictly line-based +* directives inside non empty sections are not yet defined +* sections can't nest so far (idea: thinking about `[ .extend.with.leading.dot ]`) +* section semantics: a key inside a section `[foo bar]` as in `key=value` is treated as `foo.key.bar=value` + anchor:anchor_links[]links ~~~~~~~~~~~~~~~~~~~~~~~~~~ - -http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html[] - + +http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html[] + anchor:anchor_internals[]internals/implementation sketch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -* `config` *singleton* - - has a cuckoo hash for lookups - - holds many `configfile` in a linked list - -* `configfile` is a `configitem` - - holds `lines` as `section` or `directive` starting with the empty section - - there is one special unnamed (non-disk) file for _defaults_ - -* `configitem` - - just an abstraction, we prolly dont need an implementation - -* `section` is a `line` - - parses prefix/suffix - - holds list of childs of `configline` or `other` lines - -* `line` is a `configitem` - - holds the allocated string (char*) - - backpointer to parent item (section or file) - - 2 pointers+length to parsed data, key/value for `configline`, prefix/suffix for `section` - -* `configline` is a `line` - - parses key/value as raw strings - - maintains lookup hash in ctor/dtor - -* `other` is a `line` - - comment or empty line, just preserved nothing special - -* `directive` is a `line` - - must appear in a empty section, else this is an error - -anchor:anchor_discussion[]discussion on irc -------------------------------------------- +* `config` *singleton* + - has a cuckoo hash for lookups + - holds many `configfile` in a linked list + +* `configfile` is a `configitem` + - holds `lines` as `section` or `directive` starting with the empty section + - there is one special unnamed (non-disk) file for _defaults_ + +* `configitem` + - just an abstraction, we prolly dont need an implementation + +* `section` is a `line` + - parses prefix/suffix + - holds list of childs of `configline` or `other` lines + +* `line` is a `configitem` + - holds the allocated string (char*) + - backpointer to parent item (section or file) + - 2 pointers+length to parsed data, key/value for `configline`, prefix/suffix for `section` + +* `configline` is a `line` + - parses key/value as raw strings + - maintains lookup hash in ctor/dtor + +* `other` is a `line` + - comment or empty line, just preserved nothing special + +* `directive` is a `line` + - must appear in a empty section, else this is an error + + +Brainstorming +------------- + + * support for successive dereferencing foo.bar > baz.barf > blah.boo = final value + * support for alternatives, query a set of keys, return the first existing + * provide a @readonly directive which marks a file to be handled readonly (even if writing would be permitted) + + +Discussion on irc +~~~~~~~~~~~~~~~~~ + +------------------------------------------------------------ +[01:21] joelholdsworth_: you seen the upcoming config interface i planned with simav? +[01:21] what about using ardours icons to start with? +[01:21] *least +[01:22] cehteh: not yeat, can I see +[01:22] mno: ardour icons arn't that useful +[01:22] there aren't that many +[01:22] http://www.pipapo.org/pipawiki/Lumiera/ConfigLoader[] +[01:22] question for you: do you need any more 'types' +[01:22] and those that are there are probably among the gtk stock set anywhere +[01:22] ichthyo suggested color triples +[01:23] anythinfg you have in mind, just suggest it +[01:23] so is the this structured like an ini file +[01:23] sections and keys? +[01:23] or is it treelike? +[01:24] little unordered, you basically just need to care for lumiera_config_TYPE_get ... and *_set +[01:24] like ini, not nested +[01:24] flat is good +[01:24] simple +[01:24] the namespaces are nested +[01:24] but the sectioning is not +[01:24] you can have +[01:24] [gui] +[01:24] <__nasa__> Have you all see Agave (for colorscheme generation)? +[01:24] foo = bar +[01:25] [gui.baz] +[01:25] barf= 123 +[01:25] its not exactly ini style but close +[01:25] cehteh: ooh - does mean we can levarge an ini parser? +[01:25] that would save some work and maintenance effore +[01:25] hey simav want to write the parser himself :P +[01:25] __nasa__: agave: interesting +[01:26] cehteh: would he though +[01:26] well i think thats fun to do by ourself :) +[01:26] it is just like ini +[01:26] yeah +[01:26] and i plan some special things +[01:26] but reduction of volume is good +[01:26] like for example setting values from the gui preserves the rest of the data +[01:26] when you have +[01:26] <__nasa__> cehteh: suffixes? +[01:27] [gui.foo bar] +[01:27] dada = 1234 #comment +[01:27] <__nasa__> I just saw the bottom... +[01:27] that is gui.foo.dada.bar = 1234 # comment +[01:27] see suffix +[01:28] more in a moment +[01:28] still an ini parser would be happy with that +[01:28] what i want to tell to joel +[01:28] of course you have to add some extra code to the end to make the tree +[01:28] but that would make our lives simpler +[01:29] when you now have a gui which does gui_config_number_set ("gui.foo.dada.bar", 6666) +[01:29] then +[01:29] [gui.foo bar] +[01:29] dada = 6666 #comment +[01:29] appears in the configfile +[01:29] did you mean... +[01:29] meaning preserving everything manually set up +[01:29] [gui.foo.dada] +[01:29] <__nasa__> cehteh: How does it know not to do +[01:29] bar = 6666 #comment +[01:29] ? +[01:30] <__nasa__> [gui dada.bar] +[01:30] <__nasa__> foo = 6666 #comment +[01:30] <__nasa__> joelholdsworth_: that works as well :) +[01:31] it just selects best fit sections which already exist +[01:31] but isn't it supposed to be this way... +[01:31] best fit .. first prefix, then suffix +[01:31] so [foo.bar] is different from [bar.foo] +[01:31] <__nasa__> Ok. +[01:31] if nothing fits then it places it at a non sectioned part in verbose way +[01:31] joelholdsworth_: yes +[01:31] <__nasa__> (I just wanted to make sure we had a decision on that, as I agree that is the best way) +[01:31] so rationale about sections: +[01:32] rather rarely used but you can do things like +[01:32] [plugin radius] +[01:32] blur = 1 +[01:32] sharpen = 2 +[01:33] or same for key shortcuts what simav initiall intended +[01:33] wouldn't it be simpler to have simpler systaxx +[01:34] and have it always with dots or with spaces +[01:34] well the prefix stuff is completely optinal +[01:34] that way you can't have 1 section in twice +[01:34] eh? +[01:34] well is [plugin radius] the same as [plugin.radius] ? +[01:34] no +[01:35] good +[01:35] ok +[01:35] [plugin radius] would be suffix prefix +[01:35] seems overkill to me +[01:35] err opposite +[01:35] I'd just stick with one or the other +[01:35] [plugin radius] would be prefix suffix +[01:36] [plugin.radius] is just prefix +[01:36] i'm trying to understand all this hehe +[01:37] the idea is that with just prefix you have to settle with a quite rigid hierachy where you know that things will always be ordered by the leaves +[01:37] that doesnt fit for everyone +[01:38] "ordered by the leaves" ? +[01:38] one might want to configure his plugins one affter each other +[01:38] <__nasa__> Here is what I think is cool about the setup: it means that we can treat builtins and plugins the same. +[01:38] imagine we have blur and sharben +[01:38] sharpen +[01:38] <__nasa__> [plugin radius] +[01:39] <__nasa__> cinelerra.sharpen = 3 +[01:39] both have a radius and a algorihtm. lets say horizontal or vertical +[01:39] <__nasa__> rotate = 2 +[01:39] so then you can either first configure blur +[01:39] [plugin.blur] +[01:39] radius = 1 +[01:39] algo = horiz +[01:40] oooh! - have you thought of simply using libconfig? +[01:40] # and then sharpen +[01:40] [plugin.sharpen] +[01:40] radius = 2 +[01:40] algo = horiz +[01:40] .. +[01:40] <__nasa__> joelholdsworth_: that format looks a little painful, but it could work. +[01:41] it's like JSON I think +[01:41] hey so far thats just like ini +[01:41] well now about suffix +[01:41] ok +[01:41] imagine you want to configure things by 'topics' .. +[01:42] # first define radius for our plugins +[01:42] [plugin radius] +[01:42] blur = 1 +[01:42] sharpen = 2 +[01:42] # then algo +[01:43] [plugin algo] +[01:43] yes that's quite clever +[01:43] sharpen = horiz +[01:43] blur = horiz +[01:43] but still it seems overkill +[01:43] <__nasa__> joelholdsworth_: It does look similar to JSON, but more C-like. For example, there are semi-colons instead of commas at the ends of single values. +[01:43] well its completely optional and we can leave it out +[01:43] these files are not primarily for humans +[01:44] yeah +[01:44] and also that prefix thing while clever is quite non obvious when you first see it in the file +[01:44] <__nasa__> Yes, but I am a big proponent of trying to make things human readable +[01:44] git works the same way +[01:44] sure it should be human readable +[01:44] .git/config +[01:44] <__nasa__> So if anything does go wrong, the user can fix it. +[01:45] well I'm not sure the prefixes thing is obvious enough +[01:45] well and we will have plenty of values which dont need a gui +[01:45] see about:config for firefox +[01:45] <__nasa__> I think that INI is much easier to read than libconfig, even though I have been programming in C for years and I have never edited a windows INI file. +[01:46] I think I agree +[01:46] they have all defaults and only when altered they are recorded to a config file but they dont need any special gui +[01:46] <__nasa__> about:config is so nasty. +[01:46] well its mighty +[01:46] but yes all of about:config could be stored in ini format +[01:46] [accessibility.typeaheadfind] +[01:46] and anything which is commonly changed should have a preferences gui +[01:46] linksonly=false +[01:47] btw our configs are not enumerateable +[01:47] i dont want the rest of the system to register there, that will be pita +[01:47] you can freely introduce configs (within your namespace) +[01:48] <-- benG has quit (Remote closed the connection) +[01:48] i'd rather make a grep script which goes through the sources and searched for lumiera_config_.*_get (".*", .*) +[01:49] yeah that's ok +[01:49] to have some idea what configs are actually used +[01:49] well maybe we can use libini +[01:49] so a about:config like ui wont be possible +[01:49] <__nasa__> which libini? There are quite a few. +[01:49] ahh +[01:49] that's a shame +[01:49] he forget about some libs .. thats so trivial to write by ourself :P +[01:49] <__nasa__> I assume the SF one is the best, as it is GPL. +[01:50] and simav wants to do something :P +[01:50] yeah +[01:50] anyway... time for bed for me +[01:50] well and what i rate high is that one can retrieve typed values and that one can add/set values +[01:51] and this setting should not destroy the existing file structure +[01:51] yeah +[01:51] just read and write values is no magic .. but preserving human structured files is +[01:52] that is hard, yes +[01:52] anyway... got to go +[01:52] <__nasa__> Bye! +[01:52] n8 :) +[01:52] <-- joelholdsworth_ has quit ("Ex-Chat") +[01:52] <__nasa__> It looks like libini uses [plugin/blur] instead, but that is an easy fix. +[01:52] url? +[01:52] <__nasa__> Or we just let simav do the whole thing +[01:53] well i dont know how much time he has +[01:53] <__nasa__> http://sourceforge.net/project/showfiles.php?group_id=25464[] +[01:54] last change 3 year ago :P +[01:56] * cehteh looks at the source +[01:57] fails standard C++ conformance :P +[01:58] and no @include like directive +[01:58] * __nasa__ thinks we should look elsewhere... +[01:58] * cehteh things we should just do ourself :P +[01:58] * __nasa__ agrees. +[01:59] the syntax i typed there is not very special and can be replaced +[01:59] but some other features are important +[01:59] * __nasa__ thinks it should PRE. That way we don't have to totally agree. +[01:59] pre? +[01:59] <__nasa__> (On syntax) +[01:59] no suffix? +[02:00] well as saied its just an idea and completely optional, we can leave it out +[02:00] <__nasa__> sorry -- PCRE. +[02:00] ah +[02:00] i dont see how PCRE helps for a syntax +[02:00] <__nasa__> I always think of it as "Perl Regular Expressions" and forget the "Compatible" +[02:01] anyways: features matter +[02:01] <__nasa__> I was thinking in terms of plugins/blend vs. plugins.blend. +[02:01] that is @include which follows some LUMIERA_CONFIG_PATH +[02:02] <__nasa__> We should have some way to only include certain sections. +[02:02] <__nasa__> Like have the default config be +[02:02] yes +[02:02] <__nasa__> [lumiera.core] +[02:02] <__nasa__> ... +[02:02] <__nasa__> then +[02:02] but thats setup by the administrator/user +[02:02] <__nasa__> @include ~/.lumi/plugins/ +[02:02] if the user changes lumiera.core.somevalue +[02:03] it will find the file where it is already defined +[02:03] <__nasa__> and it adds all preferences for plugins from +[02:03] then check if thats a file in the users config space (~/.lumiera/*) +[02:03] and edit it there +[02:03] <__nasa__> Ok. +[02:04] if it is a system wide file it will place a correspondending file in the users config space +[02:04] if the value is not configured already it will search for [lumiera.core] and then [lumiera] and then [] +[02:04] best fit the section where it would apply +[02:05] and then do the same write or create into correspondending user config +[02:05] thats why i saied 'features matter' +[02:06] <__nasa__> There should be a callback for a plugin to add a config file for that plugin. +[02:06] <__nasa__> Ooh +[02:06] <__nasa__> cool idea +[02:06] this is not really complicated to do but would give a big usage experience link:JustWorks[][TM] +[02:06] <__nasa__> ~/.lumi/config is the main configuration directory +[02:06] <__nasa__> inside, there is a ~/.lumi/config/config.ini which is main configuration +[02:07] no callbacks .. the idea is that the config system is independent and nothing else need to register there, that makes it much easier +[02:07] <__nasa__> Then, for example accessing plugin.shade. +[02:07] if you install a plugin this plugin should come with a system config file +[02:07] that can be a empty one +[02:07] <__nasa__> either find it in ~/.lumi/config/config.ini or ~/.lumi/config/plugin/config.ini. +[02:08] /usr/local/share/lumiera/plugin/shade +[02:08] containing just +[02:08] [plugin.shade] +[02:08] when the user edits some value +[02:09] plugin.shade.transparency = 209 +[02:09] then this will automatically create a correspondending ~/.lumiera/plugin/shade in the user config space +[02:09] <__nasa__> yeah, so what I am saying is that instead of [plugin.shade] only [] is needed +[02:10] using the system one as template +[02:10] well if its [] +[02:10] <__nasa__> since the file is already in config/PLUGIN/SHADE +[02:10] or that +[02:10] this is not yet decided +[02:10] we can do that too +[02:10] <__nasa__> so automatically add [plugin.shade] to the prefix of files in config/plugin/shade, etc +[02:11] but i would be careful to tie filenames and internal namespaces together +[02:11] yes doable +[02:11] well should still have [plugin.shade] +[02:12] else we have a nested section mess and files loose their identity +[02:12] <__nasa__> Yeah, I just realized it would be a pain to have default configs +[02:12] <__nasa__> Since where does the file go? you have to look inside to see if it had [plugin.shade], tec +[02:12] <__nasa__> etc* +[02:12] anyways nothing is etched in stone yet, we are just brainstorming +[02:13] most important thing is the interface the applciation can use +[02:13] then we can make a mockup for that which only returns defaults +[02:13] then we can use it already even when the config loader isnt finished +[02:14] <__nasa__> We could have a warning if a file in e.g. config/plugin/blur had a section that was [plugin.shade] or something +[02:14] actually the inital plan was to bootstrap the config system by one single site config file +[02:14] <__nasa__> to prevent hackery +[02:14] but plugins are a good point .. they might have their own configs +[02:15] which is not @included by the site config +[02:15] <__nasa__> lumiera_config_t *plugin_shade.request_default_config() or something +[02:15] but the plugin itself loads it when the plugin is used +[02:15] <__nasa__> Yeah +[02:16] nah +[02:16] lumiera_config is just singleton +[02:16] after you loaded plugin.shade it becomes visible there +[02:16] thats way easier +[02:16] <__nasa__> that is lumiera_config_struct +[02:16] <__nasa__> ok +[02:16] not even the struct +[02:17] thats an implementation detail +[02:17] <__nasa__> yeah -- to deal with later +[02:17] the programmers use only lumiera_coonfig_TYPE_get or _set +[02:17] where TYPE is to be defined .. see my notes +[02:18] number real string word ... and whatever else we need +[02:19] so you get typed data back .. when you provide a default then you can be even sure that it is always valid +[02:19] <__nasa__> Yeah, I do like that structure. +[02:19] <__nasa__> It also makes mass configuration setup very easy. +[02:21] btw file must have identity ... +[02:21] <__nasa__> meaning? +[02:21] <__nasa__> like they are not all config.ini? +[02:22] i mean a config file has to define for what kinds of values it applies +[02:22] [plugin.shade] for the plugin/shade file +[02:22] <__nasa__> Ok, I see. +[02:22] then changing a values there will create the correspondending user file +[02:23] *thinking* +[02:23] the user could write [plugin.shade] +[02:23] into some other config file which then becomes target for that +[02:23] (best fit) +[02:24] just wondering if the user just writes [pluin] +[02:24] [plugin] +[02:24] well i am sure we solve that +[02:25] <__nasa__> [plugin] +[02:26] <__nasa__> shade.radius = 3 +[02:26] <__nasa__> in like the plugin/blur folder? +[02:28] no in the users config somewher +[02:28] <__nasa__> oh +[02:28] ~/.lumiera/myconfig +[02:29] shall it pick and mirror /usr/share/lumiera/plugin/blur to the user config space +[02:29] or write it to the [plugin] section in myconfig +[02:30] well maybe we add a [plugin*] syntax which hints the best fit +[02:30] <__nasa__> I would think it makes more sense to add a plugin/blur folder +[02:30] <__nasa__> or else lumiera/myconfig would be huge after configuring one project ;-) +[02:30] yes .. thats what you think .. i just wondering how to give the user the control to fit it to his needs +[02:31] ~/.lumiera/all_my_pluginmyconfig +[02:31] [plugin*] +[02:31] hints the best fit algo +[02:32] well just thinking about optional things +[02:32] <__nasa__> I think I am confused -- how is [plugin*] different from [plugin] +[02:33] when you change plugin.blur.radius then the best fit algo will search the section which covers it best +[02:33] that would be [plugin.blur] in /usr/share/lumiera/plugin/blur +[02:33] <__nasa__> Oh, unless you have [plugin*] +[02:34] so it will create a correspondending ~/.lumiera/plugin/blur file for you +[02:34] yes exactly +[02:34] <__nasa__> Ok, that makes sense. Good idea! +[02:34] [plugin*] will take precedence +[02:34] when you have that then you can hint it not to create many tiny files +[02:35] <__nasa__> so that way the config is in one file +[02:35] <__nasa__> yeah +[02:35] hehe the suffux thing comes in mind here +[02:35] [plugin* radius] in ~/.lumiera/plugin_radius_conf +[02:35] <__nasa__> I was thinking suffixed are only allowed at the e.g. plugin/ level if they effect multiple plugins. +[02:35] well or as trivial as +[02:36] <__nasa__> Ok. +[02:36] [plugin* enable] in ~/.lumiera/plugins_enabled +[02:36] <__nasa__> Fun! +[02:36] blur = on +[02:36] sharpen = off +[02:36] ah .. we need a bool TYPE +[02:36] *blink* +[02:37] <__nasa__> on/off or true/false? +[02:37] <__nasa__> Well, either way. +[02:38] bool +[02:38] +[02:38] int +[02:38] +[02:38] common representations for bools (0,1,yes,no,on,off,set,clear) +[02:38] we can add more +[02:38] <__nasa__> It should be reasonably good at guessing +[02:40] yeah .. i would like to make this little intelligent +[02:40] for example when you want to set something it might look at the existing thing before doing so and then flavor the set with that +[02:41] when a string was quoted, the new one will be quoted to +[02:41] <__nasa__> "Weaken the blur effect" +[02:41] if there was a bool with 'yes' then it will choose 'no' .. +[02:41] and so on +[02:42] this are generally just a few extra lines of code, nothing dramatic .. but that makes it much nicer in detail +[02:43] <__nasa__> Yeah, I think that it will improve the user experience +[02:43] ok i add this irclog to the wiki +[02:43] <__nasa__> for most people, editing a config file is scary +[02:43] <__nasa__> so making it as easy and friendly as possible is good +[02:44] <__nasa__> All right +------------------------------------------------------------ + ------------------------------------------------------------- -[01:21] joelholdsworth_: you seen the upcoming config interface i planned with simav? -[01:21] what about using ardours icons to start with? -[01:21] *least -[01:22] cehteh: not yeat, can I see -[01:22] mno: ardour icons arn't that useful -[01:22] there aren't that many -[01:22] http://www.pipapo.org/pipawiki/Lumiera/ConfigLoader[] -[01:22] question for you: do you need any more 'types' -[01:22] and those that are there are probably among the gtk stock set anywhere -[01:22] ichthyo suggested color triples -[01:23] anythinfg you have in mind, just suggest it -[01:23] so is the this structured like an ini file -[01:23] sections and keys? -[01:23] or is it treelike? -[01:24] little unordered, you basically just need to care for lumiera_config_TYPE_get ... and *_set -[01:24] like ini, not nested -[01:24] flat is good -[01:24] simple -[01:24] the namespaces are nested -[01:24] but the sectioning is not -[01:24] you can have -[01:24] [gui] -[01:24] <__nasa__> Have you all see Agave (for colorscheme generation)? -[01:24] foo = bar -[01:25] [gui.baz] -[01:25] barf= 123 -[01:25] its not exactly ini style but close -[01:25] cehteh: ooh - does mean we can levarge an ini parser? -[01:25] that would save some work and maintenance effore -[01:25] hey simav want to write the parser himself :P -[01:25] __nasa__: agave: interesting -[01:26] cehteh: would he though -[01:26] well i think thats fun to do by ourself :) -[01:26] it is just like ini -[01:26] yeah -[01:26] and i plan some special things -[01:26] but reduction of volume is good -[01:26] like for example setting values from the gui preserves the rest of the data -[01:26] when you have -[01:26] <__nasa__> cehteh: suffixes? -[01:27] [gui.foo bar] -[01:27] dada = 1234 #comment -[01:27] <__nasa__> I just saw the bottom... -[01:27] that is gui.foo.dada.bar = 1234 # comment -[01:27] see suffix -[01:28] more in a moment -[01:28] still an ini parser would be happy with that -[01:28] what i want to tell to joel -[01:28] of course you have to add some extra code to the end to make the tree -[01:28] but that would make our lives simpler -[01:29] when you now have a gui which does gui_config_number_set ("gui.foo.dada.bar", 6666) -[01:29] then -[01:29] [gui.foo bar] -[01:29] dada = 6666 #comment -[01:29] appears in the configfile -[01:29] did you mean... -[01:29] meaning preserving everything manually set up -[01:29] [gui.foo.dada] -[01:29] <__nasa__> cehteh: How does it know not to do -[01:29] bar = 6666 #comment -[01:29] ? -[01:30] <__nasa__> [gui dada.bar] -[01:30] <__nasa__> foo = 6666 #comment -[01:30] <__nasa__> joelholdsworth_: that works as well :) -[01:31] it just selects best fit sections which already exist -[01:31] but isn't it supposed to be this way... -[01:31] best fit .. first prefix, then suffix -[01:31] so [foo.bar] is different from [bar.foo] -[01:31] <__nasa__> Ok. -[01:31] if nothing fits then it places it at a non sectioned part in verbose way -[01:31] joelholdsworth_: yes -[01:31] <__nasa__> (I just wanted to make sure we had a decision on that, as I agree that is the best way) -[01:31] so rationale about sections: -[01:32] rather rarely used but you can do things like -[01:32] [plugin radius] -[01:32] blur = 1 -[01:32] sharpen = 2 -[01:33] or same for key shortcuts what simav initiall intended -[01:33] wouldn't it be simpler to have simpler systaxx -[01:34] and have it always with dots or with spaces -[01:34] well the prefix stuff is completely optinal -[01:34] that way you can't have 1 section in twice -[01:34] eh? -[01:34] well is [plugin radius] the same as [plugin.radius] ? -[01:34] no -[01:35] good -[01:35] ok -[01:35] [plugin radius] would be suffix prefix -[01:35] seems overkill to me -[01:35] err opposite -[01:35] I'd just stick with one or the other -[01:35] [plugin radius] would be prefix suffix -[01:36] [plugin.radius] is just prefix -[01:36] i'm trying to understand all this hehe -[01:37] the idea is that with just prefix you have to settle with a quite rigid hierachy where you know that things will always be ordered by the leaves -[01:37] that doesnt fit for everyone -[01:38] "ordered by the leaves" ? -[01:38] one might want to configure his plugins one affter each other -[01:38] <__nasa__> Here is what I think is cool about the setup: it means that we can treat builtins and plugins the same. -[01:38] imagine we have blur and sharben -[01:38] sharpen -[01:38] <__nasa__> [plugin radius] -[01:39] <__nasa__> cinelerra.sharpen = 3 -[01:39] both have a radius and a algorihtm. lets say horizontal or vertical -[01:39] <__nasa__> rotate = 2 -[01:39] so then you can either first configure blur -[01:39] [plugin.blur] -[01:39] radius = 1 -[01:39] algo = horiz -[01:40] oooh! - have you thought of simply using libconfig? -[01:40] # and then sharpen -[01:40] [plugin.sharpen] -[01:40] radius = 2 -[01:40] algo = horiz -[01:40] .. -[01:40] <__nasa__> joelholdsworth_: that format looks a little painful, but it could work. -[01:41] it's like JSON I think -[01:41] hey so far thats just like ini -[01:41] well now about suffix -[01:41] ok -[01:41] imagine you want to configure things by 'topics' .. -[01:42] # first define radius for our plugins -[01:42] [plugin radius] -[01:42] blur = 1 -[01:42] sharpen = 2 -[01:42] # then algo -[01:43] [plugin algo] -[01:43] yes that's quite clever -[01:43] sharpen = horiz -[01:43] blur = horiz -[01:43] but still it seems overkill -[01:43] <__nasa__> joelholdsworth_: It does look similar to JSON, but more C-like. For example, there are semi-colons instead of commas at the ends of single values. -[01:43] well its completely optional and we can leave it out -[01:43] these files are not primarily for humans -[01:44] yeah -[01:44] and also that prefix thing while clever is quite non obvious when you first see it in the file -[01:44] <__nasa__> Yes, but I am a big proponent of trying to make things human readable -[01:44] git works the same way -[01:44] sure it should be human readable -[01:44] .git/config -[01:44] <__nasa__> So if anything does go wrong, the user can fix it. -[01:45] well I'm not sure the prefixes thing is obvious enough -[01:45] well and we will have plenty of values which dont need a gui -[01:45] see about:config for firefox -[01:45] <__nasa__> I think that INI is much easier to read than libconfig, even though I have been programming in C for years and I have never edited a windows INI file. -[01:46] I think I agree -[01:46] they have all defaults and only when altered they are recorded to a config file but they dont need any special gui -[01:46] <__nasa__> about:config is so nasty. -[01:46] well its mighty -[01:46] but yes all of about:config could be stored in ini format -[01:46] [accessibility.typeaheadfind] -[01:46] and anything which is commonly changed should have a preferences gui -[01:46] linksonly=false -[01:47] btw our configs are not enumerateable -[01:47] i dont want the rest of the system to register there, that will be pita -[01:47] you can freely introduce configs (within your namespace) -[01:48] <-- benG has quit (Remote closed the connection) -[01:48] i'd rather make a grep script which goes through the sources and searched for lumiera_config_.*_get (".*", .*) -[01:49] yeah that's ok -[01:49] to have some idea what configs are actually used -[01:49] well maybe we can use libini -[01:49] so a about:config like ui wont be possible -[01:49] <__nasa__> which libini? There are quite a few. -[01:49] ahh -[01:49] that's a shame -[01:49] he forget about some libs .. thats so trivial to write by ourself :P -[01:49] <__nasa__> I assume the SF one is the best, as it is GPL. -[01:50] and simav wants to do something :P -[01:50] yeah -[01:50] anyway... time for bed for me -[01:50] well and what i rate high is that one can retrieve typed values and that one can add/set values -[01:51] and this setting should not destroy the existing file structure -[01:51] yeah -[01:51] just read and write values is no magic .. but preserving human structured files is -[01:52] that is hard, yes -[01:52] anyway... got to go -[01:52] <__nasa__> Bye! -[01:52] n8 :) -[01:52] <-- joelholdsworth_ has quit ("Ex-Chat") -[01:52] <__nasa__> It looks like libini uses [plugin/blur] instead, but that is an easy fix. -[01:52] url? -[01:52] <__nasa__> Or we just let simav do the whole thing -[01:53] well i dont know how much time he has -[01:53] <__nasa__> http://sourceforge.net/project/showfiles.php?group_id=25464[] -[01:54] last change 3 year ago :P -[01:56] * cehteh looks at the source -[01:57] fails standard C++ conformance :P -[01:58] and no @include like directive -[01:58] * __nasa__ thinks we should look elsewhere... -[01:58] * cehteh things we should just do ourself :P -[01:58] * __nasa__ agrees. -[01:59] the syntax i typed there is not very special and can be replaced -[01:59] but some other features are important -[01:59] * __nasa__ thinks it should PRE. That way we don't have to totally agree. -[01:59] pre? -[01:59] <__nasa__> (On syntax) -[01:59] no suffix? -[02:00] well as saied its just an idea and completely optional, we can leave it out -[02:00] <__nasa__> sorry -- PCRE. -[02:00] ah -[02:00] i dont see how PCRE helps for a syntax -[02:00] <__nasa__> I always think of it as "Perl Regular Expressions" and forget the "Compatible" -[02:01] anyways: features matter -[02:01] <__nasa__> I was thinking in terms of plugins/blend vs. plugins.blend. -[02:01] that is @include which follows some LUMIERA_CONFIG_PATH -[02:02] <__nasa__> We should have some way to only include certain sections. -[02:02] <__nasa__> Like have the default config be -[02:02] yes -[02:02] <__nasa__> [lumiera.core] -[02:02] <__nasa__> ... -[02:02] <__nasa__> then -[02:02] but thats setup by the administrator/user -[02:02] <__nasa__> @include ~/.lumi/plugins/ -[02:02] if the user changes lumiera.core.somevalue -[02:03] it will find the file where it is already defined -[02:03] <__nasa__> and it adds all preferences for plugins from -[02:03] then check if thats a file in the users config space (~/.lumiera/*) -[02:03] and edit it there -[02:03] <__nasa__> Ok. -[02:04] if it is a system wide file it will place a correspondending file in the users config space -[02:04] if the value is not configured already it will search for [lumiera.core] and then [lumiera] and then [] -[02:04] best fit the section where it would apply -[02:05] and then do the same write or create into correspondending user config -[02:05] thats why i saied 'features matter' -[02:06] <__nasa__> There should be a callback for a plugin to add a config file for that plugin. -[02:06] <__nasa__> Ooh -[02:06] <__nasa__> cool idea -[02:06] this is not really complicated to do but would give a big usage experience link:JustWorks[][TM] -[02:06] <__nasa__> ~/.lumi/config is the main configuration directory -[02:06] <__nasa__> inside, there is a ~/.lumi/config/config.ini which is main configuration -[02:07] no callbacks .. the idea is that the config system is independent and nothing else need to register there, that makes it much easier -[02:07] <__nasa__> Then, for example accessing plugin.shade. -[02:07] if you install a plugin this plugin should come with a system config file -[02:07] that can be a empty one -[02:07] <__nasa__> either find it in ~/.lumi/config/config.ini or ~/.lumi/config/plugin/config.ini. -[02:08] /usr/local/share/lumiera/plugin/shade -[02:08] containing just -[02:08] [plugin.shade] -[02:08] when the user edits some value -[02:09] plugin.shade.transparency = 209 -[02:09] then this will automatically create a correspondending ~/.lumiera/plugin/shade in the user config space -[02:09] <__nasa__> yeah, so what I am saying is that instead of [plugin.shade] only [] is needed -[02:10] using the system one as template -[02:10] well if its [] -[02:10] <__nasa__> since the file is already in config/PLUGIN/SHADE -[02:10] or that -[02:10] this is not yet decided -[02:10] we can do that too -[02:10] <__nasa__> so automatically add [plugin.shade] to the prefix of files in config/plugin/shade, etc -[02:11] but i would be careful to tie filenames and internal namespaces together -[02:11] yes doable -[02:11] well should still have [plugin.shade] -[02:12] else we have a nested section mess and files loose their identity -[02:12] <__nasa__> Yeah, I just realized it would be a pain to have default configs -[02:12] <__nasa__> Since where does the file go? you have to look inside to see if it had [plugin.shade], tec -[02:12] <__nasa__> etc* -[02:12] anyways nothing is etched in stone yet, we are just brainstorming -[02:13] most important thing is the interface the applciation can use -[02:13] then we can make a mockup for that which only returns defaults -[02:13] then we can use it already even when the config loader isnt finished -[02:14] <__nasa__> We could have a warning if a file in e.g. config/plugin/blur had a section that was [plugin.shade] or something -[02:14] actually the inital plan was to bootstrap the config system by one single site config file -[02:14] <__nasa__> to prevent hackery -[02:14] but plugins are a good point .. they might have their own configs -[02:15] which is not @included by the site config -[02:15] <__nasa__> lumiera_config_t *plugin_shade.request_default_config() or something -[02:15] but the plugin itself loads it when the plugin is used -[02:15] <__nasa__> Yeah -[02:16] nah -[02:16] lumiera_config is just singleton -[02:16] after you loaded plugin.shade it becomes visible there -[02:16] thats way easier -[02:16] <__nasa__> that is lumiera_config_struct -[02:16] <__nasa__> ok -[02:16] not even the struct -[02:17] thats an implementation detail -[02:17] <__nasa__> yeah -- to deal with later -[02:17] the programmers use only lumiera_coonfig_TYPE_get or _set -[02:17] where TYPE is to be defined .. see my notes -[02:18] number real string word ... and whatever else we need -[02:19] so you get typed data back .. when you provide a default then you can be even sure that it is always valid -[02:19] <__nasa__> Yeah, I do like that structure. -[02:19] <__nasa__> It also makes mass configuration setup very easy. -[02:21] btw file must have identity ... -[02:21] <__nasa__> meaning? -[02:21] <__nasa__> like they are not all config.ini? -[02:22] i mean a config file has to define for what kinds of values it applies -[02:22] [plugin.shade] for the plugin/shade file -[02:22] <__nasa__> Ok, I see. -[02:22] then changing a values there will create the correspondending user file -[02:23] *thinking* -[02:23] the user could write [plugin.shade] -[02:23] into some other config file which then becomes target for that -[02:23] (best fit) -[02:24] just wondering if the user just writes [pluin] -[02:24] [plugin] -[02:24] well i am sure we solve that -[02:25] <__nasa__> [plugin] -[02:26] <__nasa__> shade.radius = 3 -[02:26] <__nasa__> in like the plugin/blur folder? -[02:28] no in the users config somewher -[02:28] <__nasa__> oh -[02:28] ~/.lumiera/myconfig -[02:29] shall it pick and mirror /usr/share/lumiera/plugin/blur to the user config space -[02:29] or write it to the [plugin] section in myconfig -[02:30] well maybe we add a [plugin*] syntax which hints the best fit -[02:30] <__nasa__> I would think it makes more sense to add a plugin/blur folder -[02:30] <__nasa__> or else lumiera/myconfig would be huge after configuring one project ;-) -[02:30] yes .. thats what you think .. i just wondering how to give the user the control to fit it to his needs -[02:31] ~/.lumiera/all_my_pluginmyconfig -[02:31] [plugin*] -[02:31] hints the best fit algo -[02:32] well just thinking about optional things -[02:32] <__nasa__> I think I am confused -- how is [plugin*] different from [plugin] -[02:33] when you change plugin.blur.radius then the best fit algo will search the section which covers it best -[02:33] that would be [plugin.blur] in /usr/share/lumiera/plugin/blur -[02:33] <__nasa__> Oh, unless you have [plugin*] -[02:34] so it will create a correspondending ~/.lumiera/plugin/blur file for you -[02:34] yes exactly -[02:34] <__nasa__> Ok, that makes sense. Good idea! -[02:34] [plugin*] will take precedence -[02:34] when you have that then you can hint it not to create many tiny files -[02:35] <__nasa__> so that way the config is in one file -[02:35] <__nasa__> yeah -[02:35] hehe the suffux thing comes in mind here -[02:35] [plugin* radius] in ~/.lumiera/plugin_radius_conf -[02:35] <__nasa__> I was thinking suffixed are only allowed at the e.g. plugin/ level if they effect multiple plugins. -[02:35] well or as trivial as -[02:36] <__nasa__> Ok. -[02:36] [plugin* enable] in ~/.lumiera/plugins_enabled -[02:36] <__nasa__> Fun! -[02:36] blur = on -[02:36] sharpen = off -[02:36] ah .. we need a bool TYPE -[02:36] *blink* -[02:37] <__nasa__> on/off or true/false? -[02:37] <__nasa__> Well, either way. -[02:38] bool -[02:38] -[02:38] int -[02:38] -[02:38] common representations for bools (0,1,yes,no,on,off,set,clear) -[02:38] we can add more -[02:38] <__nasa__> It should be reasonably good at guessing -[02:40] yeah .. i would like to make this little intelligent -[02:40] for example when you want to set something it might look at the existing thing before doing so and then flavor the set with that -[02:41] when a string was quoted, the new one will be quoted to -[02:41] <__nasa__> "Weaken the blur effect" -[02:41] if there was a bool with 'yes' then it will choose 'no' .. -[02:41] and so on -[02:42] this are generally just a few extra lines of code, nothing dramatic .. but that makes it much nicer in detail -[02:43] <__nasa__> Yeah, I think that it will improve the user experience -[02:43] ok i add this irclog to the wiki -[02:43] <__nasa__> for most people, editing a config file is scary -[02:43] <__nasa__> so making it as easy and friendly as possible is good -[02:44] <__nasa__> All right ------------------------------------------------------------- - - -anchor:anchor_brainstorm[]Brainstorming ---------------------------------------- - - * support for successive dereferencing foo.bar > baz.barf > blah.boo = final value - * support for alternatives, query a set of keys, return the first existing - * provide a @readonly directive which marks a file to be handled readonly (even if writing would be permitted) diff --git a/doc/devel/rfc/DataBackend.txt b/doc/devel/rfc/DataBackend.txt index b416a9404..a4710224f 100644 --- a/doc/devel/rfc/DataBackend.txt +++ b/doc/devel/rfc/DataBackend.txt @@ -3,7 +3,7 @@ Design Process : Data Backend [grid="all"] `------------`----------------------- -*State* _Final_ +*State* _Parked_ *Date* _2007-06-04_ *Proposed by* link:ct[] ------------------------------------- @@ -42,24 +42,14 @@ This just starts as braindump, I will refine it soon: http://www.pipapo.org/pipawiki/Cinelerra/Developers/ichthyo/Cinelerra3/Architecture[Architecture] and a sketch/draft about http://www.pipapo.org/pipawiki/Cinelerra/Developers/ichthyo/Possibilities_at_hand[things possible in the middle layer] -Tasks -^^^^^ +Parked +~~~~~~ +The underlying principles remain valid, yet development took another +direction during the last years. Special frameworks for high-performance +asynchronous IO will be used for dedicated use cases. +Ichthyostega:: '2023-10-24' ~~ -Pros -^^^^ - - -Cons -^^^^ - - -Alternatives -^^^^^^^^^^^^ - - -Rationale -~~~~~~~~~ Comments @@ -71,5 +61,10 @@ Sounds fairly complete to me Developement takes place in the repo now -- link:ct[] [[DateTime(2007-06-27T16:14:56Z)]] +Development took another direction over the last years; +the former »Backend« layer is restructured + -- link:Ichthyostega[] [[DateTime(2023-10-24T22:45:55Z)]] + + '''' Back to link:/documentation/devel/rfc.html[Lumiera Design Process overview] diff --git a/doc/devel/rfc/WebsiteSupportMarkup.txt b/doc/devel/rfc/WebsiteSupportMarkup.txt index 2b4c6ad28..b1c5f49fd 100644 --- a/doc/devel/rfc/WebsiteSupportMarkup.txt +++ b/doc/devel/rfc/WebsiteSupportMarkup.txt @@ -12,8 +12,8 @@ WebsiteSupportMarkup ******************************************************************************** .Abstract -The Lumiera Website and Documentation uses a lightweight, plaintext based -Infrastructure built on top of Asciidoc and Git. To help with everyday +The Lumiera Website and Documentation uses a lightweight, plain text based +infrastructure built on top of Asciidoc and Git. To help with everyday authoring and editorial tasks, a set of support facilities is proposed, to be integrated seamlessly into the existing infrastructure. The front-end to these tools is specific markup, allowing for *cross-linking*, *tag*-based @@ -34,38 +34,44 @@ Use Cases ~~~~~~~~~ Authoring:: - For now, the _authoring of new content_ is mostly a responsibility of the core developers. - It needs to be done close to the actual coding activities, typically once a new facility - is roughly finished. It is crucial for this activity not to impose a _context switch_ - on the writer. It must be ``just writing down the obvious'' -- otherwise developers tend - to defer this activity ``for later''. This situation creates some unique problems... + For now, the _authoring of new content_ is mostly the responsibility of the core developers. + It needs to be done close to the related coding activities, typically once a new facility + is finished or in the final stages of completion. It is crucial for this + activity not to impose a _context switch_ on the writer. It must be ``just + writing down the obvious'' -- otherwise developers tend to defer this activity + ``for later''. This situation creates some unique problems... - during the write-up, the coder-author realises several external presuppositions, which, during the act of coding, seemed to be ``obvious'' and ``self evident''. - - since expanding on all of those secondary topics is out of question (the author will rather + - since expanding on all of those secondary topics is out of scope (the author would rather abandon the task of documentation altogether), the only solution is to allow for *cross linking*, - even if the created links are dangling for the moment. - - the author can't effort to locate other documents and determine URLs; he won't even be willing - to consult a markup syntax reference. Because, what he's about to write is essentially hard to - put into words and thus requires his full attention. + even if the new links are dangling; well at least for the moment. + - the author can't invest the effort to locate additional relevant documents, + nor hunt for URLs; the author is disinclined to consult a markup syntax + reference. This arises from the nature of the required text which is + essentially difficult to grasp in the first place and even more difficult + too put into words. A task requiring full mental concentration. + Integrating Content:: This task is often prompted by some kind of external cause: it might be someone asking for explanations and while trying to respond, it was determined that ``this should be in the documentation''. Thus, some existing markup is extracted from an external system and pasted into some location, ``just for the moment''. Of course, this content will be forgotten and rest there for - years to come. To deal with this situation... + years to come. To counter act such a situation, we propose the following: - - adapting the structural cross-references of the integrated markup needs to be an easy task. - - we need a way to hook the new content somehow into our existing categorisation + - structural cross-references of the integrated markup needs to be easy to do. + - a way in which we can somehow hook the new content into our current scheme + of categories - the person integrating the content wont't be willing to visit a lot of other locations, or to read a syntax reference for some kind of advanced markup. Editorial work:: - The _editor_ is reviewing existing content. He'll try to emulate an assumed user's point of view + The _editor_ reviews current content. He'll try to emulate an assumed user's point of view to judge the adequacy of the presentation. This leads to rearranging and re-locating pages and whole trees of pages, even splitting some of them, and adding introductory paragraphs and pages here and - there -- all without the ability to read and understand any of the reviewed material in-depth. + there -- all without the ability to read or understand any of the reviewed + material to any degree of depth. - this work will be performed on the primary content using its ``natural'' representation: that is, the editor will copy and move files in the file system and edit text. There is no room for using @@ -76,7 +82,7 @@ Editorial work:: Tools for the task at hand ~~~~~~~~~~~~~~~~~~~~~~~~~~ -This RfC proposes a set of tools to cope with those problems. More specifically, this proposal details +This RfC proposes a set of tools to improve these problems. More specifically, this proposal details a possible _front end_ for these tools, in a way which blends well with the existing Git / Asciidoc website infrastructure. @@ -96,18 +102,18 @@ Variations and extensions - we might consider adding some domain prefixes, e.g. +link:rfc:SomeTopic+ or +link:ticket:SomeTopic+ or +link:code:SomeTopic+ -Obviously, these cross-links needs to be resilient towards content reorganisation after-the-fact. -Effectively this mandates introducing some kind of indirection, since we can't effort to regenerate the +Obviously, these cross-links need to be resilient against content reorganisation after-the-fact. +Effectively this mandates introducing some kind of indirection, since we can't afford to regenerate the whole website's HTML pages after each localised change. A possible solution could be to make the rendered cross link invoke a JavaScript function, which in turn consults some kind of index table. Another solution would be to let the cross link point to a server sided script. Tag extractor and Index ^^^^^^^^^^^^^^^^^^^^^^^ -Define suitable ways for attaching *tags* to various kinds of content. The syntax will be tailored to the -kind of content in question, typically placing the tags in some kind of comment or specific header. For -larger documents, it would be desirable to attach tags in a more fine-grained manner, e.g. tag only one -paragraph or sub-section (but this is a nice-to-have, since it is anything but trivial to implement). +Define suitable ways to attach *tags* to various kinds of content. The syntax will be tailored to the +kind of content in question, typically placing a tag in, for example, a comment or possibly in a special +header. For larger documents, it would be desirable to attach tags in a more fine-grained manner, e.g. tag +only one paragraph or sub-section (but this is a nice-to-have, since it is anything but trivial to implement). Based on these tags, there should be a mechanism to integrate a list of links into some Asciidoc page. Obviously this list needs to be dynamic, e.g. by using JavaScript or by including pre-fabricated HTML @@ -123,10 +129,10 @@ to take precedence. Definition entries ^^^^^^^^^^^^^^^^^^ Define a suitable format to promote an existing piece of information into a *definition*. While not -interfering with the presentation at the textual location of this definition, this mechanism should -allow to extract this definition and present it within a glossary of terms. It would be nice if -such a generated glossary could provide an automatic back-link to the location where the definition -was picked up initially. +interfering with the presentation at the textual location of this definition, the mechanism should support the +possibility of providing a definition in the source which is then extracted from the text and presented +in a new file as, for example, a glossary of terms. It would be nice if such a generated glossary could +provide an automatic back-link to the location where the definition was picked up initially. Of course, defining such a markup includes some reasoning about the suitable format of a glossary and definition-of-terms. (Scope? Just a sentence? Just a paragraph? How to utilise existing headings?) @@ -144,7 +150,8 @@ interested in using them. He will not recall any fancy syntax, and he won't stic So, anything not outright obvious is out of question. - since we don't want fully dynamic page generation and we can't afford regenerating the whole website - for each small update, all of these facilities need some way to adapt after-the-fact. + for each small update, all of these facilities need some way to be integrated into the present + infrastructure with the least amount of change as pssible. - we need to build leeway into the system at various places. E.g. the cross-link facility needs a strategy to generate and match the IDs and order possible matches in a sensible way. What initially links to some doxygen comment might later on point to a glossary if applicable. diff --git a/doc/devel/rfc_final/AllPluginInterfacesAreC.txt b/doc/devel/rfc_dropped/AllPluginInterfacesAreC.txt similarity index 100% rename from doc/devel/rfc_final/AllPluginInterfacesAreC.txt rename to doc/devel/rfc_dropped/AllPluginInterfacesAreC.txt diff --git a/doc/devel/rfc_parked/ConfigLoader.txt b/doc/devel/rfc_parked/ConfigLoader.txt new file mode 120000 index 000000000..d13f2692b --- /dev/null +++ b/doc/devel/rfc_parked/ConfigLoader.txt @@ -0,0 +1 @@ +../rfc/ConfigLoader.txt \ No newline at end of file diff --git a/doc/devel/rfc_final/DataBackend.txt b/doc/devel/rfc_parked/DataBackend.txt similarity index 100% rename from doc/devel/rfc_final/DataBackend.txt rename to doc/devel/rfc_parked/DataBackend.txt diff --git a/doc/index.txt b/doc/index.txt index 53421ae24..a15dd2f25 100644 --- a/doc/index.txt +++ b/doc/index.txt @@ -1,7 +1,6 @@ Documentation ============= - This documentation section contains documentation for both users and developers. == User == @@ -14,30 +13,108 @@ A user manual containing the full reference for using Lumiera is planned: * link:user/manual.html[User Manual] _(planned)_ == Developer == -Information about the structure and concepts Lumiera is based upon. -These documents give an overview on the main components of the application: - * link:design/index.html[Design Documents] +This section is a guide on how the developer documentation is arranged, organised and grouped. We present here +a map of the developer documentation to assist you in navigating through the documents. -Programming-oriented details about the functional layers of Lumiera as well as a Doxygen documentation : +=== Developer Documentation is Arranged into Layers - * link:technical/index.html[Technical Documents] - * link:technical/code/codingGuidelines.html[Style and Coding Guidelines] +image:devel/images/doc_map.png[Map] -=== the TiddlyWiki === +The various pieces of documentation are not scattered about randomly. These are arranged and organised along +established simple yet flexible guidelines to enable easy and intuitive access. It is not rigid and conformity +is not required. There may be very good reasons to adapt the rules with some degree of flexibility at times; +and sometimes it might be difficult to select a unique section in which to place new documents. + +The various documents are organised in an hierarchical structure of layers. The uppermost layer, or root, is +the point of entry into the Lumiera developer documentation universe. The lowest layers are populated by the +Lumiera API documentation which is currently produced from the source code using Doxygen. + +The map is depicted above in the illustration. Below we provide links to the top-level documents to all +elements in the map froom which you can launch deeper into the lower layers of the documentation. We present +these in tabular form to ease navigation. +-- +  +-- + +The *Documentation Layer Structure* appears at various places in the project... + +Lumiera homepage :: +https://Lumiera.org[Lumiera homepage] by the menus on the left hand-side of the +page. This menu is also organised into layers that reflects the organisation of +the documentation to help you navigate through the documentation. + +Source code doc directory structure :: +The layout of the sub-directories below the *doc* directory is structured around +the hierarchy, + +==== The Uppermost Layer + +At the highest, uppermost layer is the document +https://lumiera.org/documentation/user/intro/intro.html[Lumiera (as seen) from Outer Space]. This is the node +on the left in the illustration above. All new to Lumiera should read this document. You can always return +to this document later if you are entirely lost and wish to find your bearings before navigating into the +deeper documentation dungeons. + +==== The Second Layer + +At the second layer, the reader must make the first major decision on which +router to pursue: documentation with a heavy code based leaning; or more descriptive +documentation on the design of Lumiera which avoids a code narrative. This +section has two main sections: + + * link:design/index.html[Lumiera Developer Design Documents] + * link:technical/index.html[Lumiera Developer Technical Documents] + +-- +  +-- + + +===== Design Documentation + +[width="90%",frame="topbot",options="header",cols="^,2<"] +|================================================================================================ +|Sub-System|Remarks +|https://lumiera.org/documentation/design/architecture/index.html[Architecture]|Structure of Lumiera +|https://lumiera.org/documentation/design/gui/index.html[GUI]|The Lumiera ⟷ user interaction layer +|https://lumiera.org/documentation/design/model/index.html[Model]|Low-level and high-level models +|https://lumiera.org/documentation/design/engine/index.html[Engine]|Renderer, output generation, ... +|https://lumiera.org/documentation/design/lowlevel/index.html[Low-level]|Low-level and system services +|https://lumiera.org/documentation/design/application/index.html[Application]|Application framework +|https://lumiera.org/documentation/design/plugins/index.html[Plugins]|Dynamically loaded Extensions +|https://lumiera.org/documentation/design/workflow/index.html[Workflow]|Workflow and Interaction Design +|https://lumiera.org/documentation/design/governance/index.html[Governance]|Meta project concerns +|================================================================================================ + + + +===== Technical Documentation + +[width="90%",frame="topbot",options="header",cols="^,2<"] +|================================================================================================ +|Sub-System|Remarks +|https://lumiera.org/documentation/technical/build/index.html[Build System]|Build, Packaging and CI +|https://lumiera.org/documentation/technical/stage/index.html[Stage]|GUI (technical aspects) +|https://lumiera.org/documentation/technical/steam/index.html[Steam]|Session and processing coordination +|https://lumiera.org/documentation/technical/vault/index.html[Vault]|Low-level operations +|https://lumiera.org/documentation/technical/library/index.html[Support lib]|Interface and support libraries +|https://lumiera.org/documentation/technical/code/index.html[Code base]|Code management, organisation, ... +|https://lumiera.org/documentation/technical/infra/index.html[Infrastructure]|Website and developer tooling ... +|https://lumiera.org/documentation/technical/howto/index.html[Developer HowTos]|Instructions and guides for developers +|link:/doxy/index.html[API Doc]|Using link:http://doxygen.org[Doxygen] to generate doc from code +|================================================================================================ + + +The TiddlyWiki:: Currently, Lumiera is still in the design- and evolution phase. -There is an embedded JavaScript wiki (TiddlyWiki) within the source tree, mostly -used as design notebook, featuring day-to-day design sketches, notes but also +There is an embedded JavaScript wiki (https://classic.tiddlywiki.com/[TiddlyWiki]) within the source tree, +mostly used as design notebook, featuring day-to-day design sketches, notes but also quite some more persistent planning. Finished documentation text is constantly moved over to the documentation section(s) of the Lumiera website. -> access the Development link:{l}/wiki/renderengine.html[TiddlyWiki online here] -=== API Documentation === -We use the link:http://doxygen.org[Doxygen] tool to extract -code comments and generate code level API and reference documentation. - --> link:/doxy/index.html[Doxygen generated documentation] == Media and Presentations == diff --git a/doc/technical/vault/ConfigGuide.txt b/doc/technical/vault/ConfigGuide.txt deleted file mode 100644 index 3e66ac9b6..000000000 --- a/doc/technical/vault/ConfigGuide.txt +++ /dev/null @@ -1,121 +0,0 @@ -The incomplete Guide to Lumiera Configuration -============================================= -:author: ct -:date: 8/2008 - -WARNING: this is a draft from the early days of Lumiera + - IMHO, the whole topic ``Configuration'' requires further - discussion and design. -+ --- -- Ichthyo - -''''''''''''' -''''''''''''' - -Order is roughly alphabetically, depending on the mood of the writer. -Defaults are noted if present. Not all are implemented yet. - -General Introduction --------------------- - -Lumiera uses plaintext files with a INI file like syntax for -configuration. This Syntax is strictly line based. There are only a -few syntactic elements. - -TODO:describe config syntax here - -Config Subsystem ----------------- - -The path where Lumiera searches its configuration. Single components are -separated by colons as in PATH and other such environment variables. -Here it might be handy that any Lumiera configuration can be -overridden by a environment variable: -'LUMIERA_CONFIG_PATH=somewhere:else lumiera ...' -A default are initialized at installation time, this is important to -bootstrap the whole configuration system. - - config.path - - -The config system check for a preferred format when writing config -entries. For each key 'foo.bar', these can be overridden with a key -'config.format.foo.bar' linking to the desired format. - - config.formatkey ='config.format.%s' - - -The following are links to the default formatting when no explicit -format is set for a key. Changing these to a wrong type will break the -system! - - config.formatdef.link < config.formatstr.link - config.formatdef.number < config.formatstr.number.dec - config.formatdef.real < config.formatstr.real - config.formatdef.string < config.formatstr.string - config.formatdef.word < config.formatstr.word - config.formatdef.bool < config.formatstr.bool - - -This are the low level formating specifications for the buildin -types, DONT TOUCH THESE! - - config.formatstr.link = '< %s' - config.formatstr.number.dec = '= %lld' - config.formatstr.number.hex = '= 0x%llX' - config.formatstr.number.oct = '= 0%llo' - config.formatstr.real = '= %Lg' - config.formatstr.real.dec = '= %Lf' - config.formatstr.real.sci = '= %Le' - config.formatstr.string = '= %s' - config.formatstr.string.dquoted = '= \"%s\"' - config.formatstr.string.quoted = '= ''%s''' - config.formatstr.word = '= %s' - config.formatstr.bool = '= %d' - - -Plugin System -------------- - -The path where Lumiera searches its plugins. Single components are -separated by colons as in PATH and other such environment variables. -Here it might be handy that any Lumiera configuration can be -overridden by a environment variable: -'LUMIERA_PLUGIN_PATH=somewhere:else lumiera ...' -Sensible defaults are initialized at installation time. - - plugin.path - - -I/O Backend ------------ - -File handling -~~~~~~~~~~~~~ - -How many filehandles the backend shall use [approx 2/3 of all available] - - vault.file.max_handles - - - -Memory mapped Files -~~~~~~~~~~~~~~~~~~~ - -Address space limit (memory mapping) -Defaults: -3GiB on 32 bit arch -192TiB on 64 bit arch - - vault.mmap.as_limit - -Default start size for mmaping windows. -128MB on 32 bit arch -2GB on 64 bit arch - - vault.mmap.window_size - -How many memory mappings shall be established at most -Default 60000 - - vault.mmap.max_maps diff --git a/doc/technical/vault/index.txt b/doc/technical/vault/index.txt index c5b1241ee..75d4eb62f 100644 --- a/doc/technical/vault/index.txt +++ b/doc/technical/vault/index.txt @@ -5,5 +5,4 @@ Here we collect bits of technical documentation for the Vault-Layer. For now, we have: - * link:ConfigLoader.html[Config Loader brainstorming from 2008] * link:scheduler.html[Scheduler and Jobs] diff --git a/doc/technical/vault/scheduler.txt b/doc/technical/vault/scheduler.txt index f62da23dd..508cce15e 100644 --- a/doc/technical/vault/scheduler.txt +++ b/doc/technical/vault/scheduler.txt @@ -4,37 +4,16 @@ Scheduler and Job handling The purpose of the _Scheduler_ is to run small self contained _Jobs_ ordered by priority and observing specific timing constraints. -Scheduler implementation ideas ------------------------------- +NOTE: Subject to [yellow-background]#active design and implementation# work as of 10/2023 -Use multiple priority queues +Work-in-progress documentation can be found in the +link:{l}/wiki/renderengine.html#PlaybackVerticalSlice%20RenderEngine%20Scheduler%20SchedulerWorker%20SchedulerMemory%20RenderActivity%20Player%20FrameDispatcher%20JobPlanningPipeline%20PlayProcess%20Rendering%20ProcNode%20NodeOperationProtocol[Tiddly Wiki] -- background work -- foreground high-priority -- soft-realtime actions About Jobs ---------- A job is a closure to run a small and limited action or operation, which in itself _should not block_. Job may depend on each other and on resources -to be provided. A job may be conained in multiple queues and may be marked -as _canceled_ -- in which case the job function will never run and the job -will be discarded on occasion. - -Job States -~~~~~~~~~~ - -[source,C] --------------- -enum job_state -{ - done, // already done, nothing to do - running, // job is running - waiting, // waits for some resource (annother job) - rejected, // sorry, cant do that dave, time will run out - expired, // time expired - aborted // got aborted -} --------------- - +to be provided. A job may be rescheduled prior to invocation, but it is +activated at most once by an _Scheduler Action._