From 5d214bbf5814cadffc234b43eac273622d1a8bbb Mon Sep 17 00:00:00 2001
From: Christian Thaeter <ct@pipapo.org>
Date: Wed, 20 Jun 2007 02:42:28 +0200
Subject: [PATCH 1/2] written a Overview about the data backend

---
 wiki/backend.html | 36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/wiki/backend.html b/wiki/backend.html
index 039878cb9..e5d61dd05 100644
--- a/wiki/backend.html
+++ b/wiki/backend.html
@@ -44,7 +44,7 @@ DAMAGE.
 
 <style type="text/css">#contentWrapper {display:none;}</style><div id="SplashScreen" style="border: 3px solid #ccc; display: block; text-align: center; width: 320px; margin: 100px auto; padding: 50px; color:#000; font-size: 28px; font-family:Tahoma; background-color:#eee;"><b>My TiddlyWiki</b> is loading<blink> ...</blink><br><br><span style="font-size: 14px; color:red;">Requires Javascript.</span></div>
 <!--PRE-HEAD-END-->
-<title> Cinelerra3 Data Backend - design draft </title>
+<title> Data Backend - design draft </title>
 <style type="text/css">
 #saveTest {display:none;}
 #messageArea {display:none;}
@@ -764,6 +764,25 @@ TertiaryMid: #999
 TertiaryDark: #666
 Error: #f88</pre>
 </div>
+<div title="DataBackend" modifier="CehTeh" modified="200706192335" created="200706192251" changecount="3">
+<pre>This just starts as braindump, I will refine it soon:
+* handle all files cinelerra uses at runtime (media, edl, temp data)
+* manage filehandles, cinelerra might use more more files than available filehandles
+* manage temporary data
+* do caching
+* io will be blocked where the backend tells the core where it can expect the data (not read()/write() like)
+* kindof garbage collector
+* do prefetching
+* no/low latency for the core the prefetcher and other things ensure that data is available in time
+* translate any input into a format which the cinelerra core understands (demux, decode)
+* same for encoding to output formats
+* offer a plugin API for encoders/decoders
+* maybe network backend for serving data to distributed rendernodes
+* can do some load control or management (trigger adaptive rendering if system is idle etc)
+* pull based arch
+
+Look at [[Overview]] for the current design proposal</pre>
+</div>
 <div title="DefaultTiddlers" modifier="CehTeh" created="200706192002" changecount="1">
 <pre>DataBackend
 </pre>
@@ -1164,6 +1183,17 @@ DataBackend
 
 &lt;style type=&quot;text/css&quot;&gt;#contentWrapper {display:none;}&lt;/style&gt;&lt;div id=&quot;SplashScreen&quot; style=&quot;border: 3px solid #ccc; display: block; text-align: center; width: 320px; margin: 100px auto; padding: 50px; color:#000; font-size: 28px; font-family:Tahoma; background-color:#eee;&quot;&gt;&lt;b&gt;My TiddlyWiki&lt;/b&gt; is loading&lt;blink&gt; ...&lt;/blink&gt;&lt;br&gt;&lt;br&gt;&lt;span style=&quot;font-size: 14px; color:red;&quot;&gt;Requires Javascript.&lt;/span&gt;&lt;/div&gt;</pre>
 </div>
+<div title="Overview" modifier="CehTeh" created="200706200040" changecount="1">
+<pre>Whenever cinelerra needs to access data this is done through the DataBackend described here. The backend uses memory mapping to make data available to the program. This is little different to more common open/read/write/close file access while giving superior performance and much better memory utilization.
+
+The data backend must be capable to handle more data than will fit into the memory or even address space on 32 bit archs.  Moreover a project may access more files than the OS can handle at a time, thus the Backend needs a FileHandleCache to manage filehandles dynamically.
+
+Which parts of a File are actually mapped to physical ram is managed by the kernel, but we keep a FileCache to manage the mappings we set up.
+
+To minimize latency and optimize cpu utilization we have a [[Prefetch]] thread which operates a [[Scheduler]] to render and cache frames which are expected to be consumed soon. This prefetcher keeps [[Statistics]] for optimizing performance.
+
+The core accesses data through smart pointers which are described in [[DataLocking]].</pre>
+</div>
 <div title="PageTemplate" modifier="CehTeh" modified="200706110330" created="200701131624" tags="MPTWTheme" server.type="file" server.host="file:///home/ct/.homepage/home.html" server.page.revision="200706110330">
 <pre>&lt;!--{{{--&gt;
 &lt;div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'&gt;
@@ -1969,8 +1999,8 @@ see : &lt;&lt;tag RSSFeed&gt;&gt; for the full list.
 <pre>design draft
 </pre>
 </div>
-<div title="SiteTitle" modifier="CehTeh" created="200706192003" changecount="1">
-<pre>Cinelerra3 Data Backend</pre>
+<div title="SiteTitle" modifier="CehTeh" modified="200706192245" created="200706192003" changecount="2">
+<pre>Data Backend</pre>
 </div>
 <div title="SplashScreenPlugin" modifier="Saq" modified="200607202048" created="200607191631" tags="lewcidExtension systemConfig" server.type="file" server.host="file:///home/ct/.homepage/home.html" server.page.revision="200607202048">
 <pre>/***

From 5b22add38ecb03ec27142a54c7526cc10d2b2b05 Mon Sep 17 00:00:00 2001
From: Christian Thaeter <ct@pipapo.org>
Date: Wed, 20 Jun 2007 02:50:10 +0200
Subject: [PATCH 2/2] we don't need the 'journal' option in the draft wikis

---
 wiki/backend.html       | 3 +++
 wiki/ichthyo-draft.html | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/wiki/backend.html b/wiki/backend.html
index e5d61dd05..827bcc931 100644
--- a/wiki/backend.html
+++ b/wiki/backend.html
@@ -1995,6 +1995,9 @@ see : &lt;&lt;tag RSSFeed&gt;&gt; for the full list.
 
 </pre>
 </div>
+<div title="SideBarOptions" modifier="CehTeh" created="200706200047" changecount="1">
+<pre>&lt;&lt;search&gt;&gt;&lt;&lt;closeAll&gt;&gt;&lt;&lt;permaview&gt;&gt;&lt;&lt;newTiddler&gt;&gt;&lt;&lt;saveChanges&gt;&gt;&lt;&lt;slider chkSliderOptionsPanel OptionsPanel &quot;options »&quot; &quot;Change TiddlyWiki advanced options&quot;&gt;&gt;</pre>
+</div>
 <div title="SiteSubtitle" modifier="CehTeh" modified="200706192003" created="200706192002" changecount="2">
 <pre>design draft
 </pre>
diff --git a/wiki/ichthyo-draft.html b/wiki/ichthyo-draft.html
index 982644400..7b44edaa1 100644
--- a/wiki/ichthyo-draft.html
+++ b/wiki/ichthyo-draft.html
@@ -2024,6 +2024,9 @@ The design of Cinelerra 2 basically follows this design, but __fails because of
 
  * see also the [[Entities invoved in Rendering|RenderEntities]]</pre>
 </div>
+<div title="SideBarOptions" modifier="CehTeh" created="200706200048" changecount="1">
+<pre>&lt;&lt;search&gt;&gt;&lt;&lt;closeAll&gt;&gt;&lt;&lt;permaview&gt;&gt;&lt;&lt;newTiddler&gt;&gt;&lt;&lt;saveChanges&gt;&gt;&lt;&lt;slider chkSliderOptionsPanel OptionsPanel &quot;options »&quot; &quot;Change TiddlyWiki advanced options&quot;&gt;&gt;</pre>
+</div>
 <div title="SiteSubtitle" modifier="Ichthyostega" created="200706190044" changecount="1">
 <pre>some aspects of Cinelerra-3 design</pre>
 </div>