From a374d0a5b545eed7d7e9e7088c15afb8acedfa45 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 21 Jun 2007 03:15:52 +0200 Subject: [PATCH 1/5] make some DIR_INFO's nicer --- DIR_INFO | 2 +- admin/DIR_INFO | 2 +- admin/git_hooks/DIR_INFO | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/DIR_INFO b/DIR_INFO index 1544e301d..39fd1850f 100644 --- a/DIR_INFO +++ b/DIR_INFO @@ -1 +1 @@ -This is the root dir for the cinelerra3 project +The root dir for the cinelerra3 project diff --git a/admin/DIR_INFO b/admin/DIR_INFO index bc76e0079..b3e55c8ad 100644 --- a/admin/DIR_INFO +++ b/admin/DIR_INFO @@ -1 +1 @@ -administrative scripts, tools, git hook scripts, etc.. +administrative scripts diff --git a/admin/git_hooks/DIR_INFO b/admin/git_hooks/DIR_INFO index 3c413e34c..7f834c917 100755 --- a/admin/git_hooks/DIR_INFO +++ b/admin/git_hooks/DIR_INFO @@ -1,2 +1,3 @@ -here are some git hook scripts taken out of .git/hooks since .git is not under revision control itself +git hook scripts +taken out of .git/hooks since .git is not under revision control itself if you want to use these, just symlink them From 9ac23fc7e485d261e4e1bc114b77c71e4566cfa3 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 21 Jun 2007 03:16:27 +0200 Subject: [PATCH 2/5] small script generating a description of the source tree --- admin/treeinfo.sh | 2 ++ 1 file changed, 2 insertions(+) create mode 100755 admin/treeinfo.sh diff --git a/admin/treeinfo.sh b/admin/treeinfo.sh new file mode 100755 index 000000000..f57a6cfee --- /dev/null +++ b/admin/treeinfo.sh @@ -0,0 +1,2 @@ +#!/bin/sh +find -name DIR_INFO -printf '%-40h: ' -exec head -1 {} \; | sort | cut -c 1-92 From fe96d470b34bcab0e93565e62cd4f77bee26fabc Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 22 Jun 2007 03:12:30 +0200 Subject: [PATCH 3/5] Some more notes about the backend, Prefetch and Scheduler --- wiki/backend.html | 40 ++++++++++++++++++++++++++++++++++------ 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/wiki/backend.html b/wiki/backend.html index f3e61df74..1601ad6bd 100644 --- a/wiki/backend.html +++ b/wiki/backend.html @@ -801,8 +801,8 @@ Files content is memory mapped into the process address space, this is managed b 
'FileHandleCache' storing a finite maximum number of [[FileHandle]]s as a list. As long the configured maximum of open files is not reached new file handles are stored at the begin of the list. Whenever a filehandle is accessed it is moved to the begin of the list too. Unused filehandles propagate towards the end of the list. When the maximum of open filehandles is reached, aged filehandles are closed and taken from the end.
-
-
Each 'FileMap' object contains many [[Frame]]s as well as metadata. The actual layout depends on the type of the [[File]]. Mappings need to be page aligned while [[Frame]]s can can be anywhere within a file and dynamic sized.
+

+
Each 'FileMap' object contains many [[Frame]]s. The actual layout depends on the type of the [[File]]. Mappings need to be page aligned while [[Frame]]s can can be anywhere within a file and dynamic sized.
 
 All established [[FileMap]]s are managed in a FileMapCache. This is similar to the FileHandleCache but mappings which are in use are checked out of the aging list and thus become locked from aging/purging.
 
@@ -1212,9 +1212,10 @@ config.formatters.push( {
 } )
 //}}}

 

-

+

 
''[[Cinelerra3|index.html]]''
 DataBackend
+[[Overview]]
 <<fullscreen>>
 

 

@@ -1242,16 +1243,17 @@ background rendering
 
 renderfarm
-
+
Whenever cinelerra needs to access data this is done through the DataBackend described here. The backend uses memory mapping to make data available to the program. This is little different to more common open/read/write/close file access while giving superior performance and much better memory utilization.
 
 The data backend must be capable to handle more data than will fit into the memory or even address space on 32 bit archs.  Moreover a project may access more files than the OS can handle at a time, thus the for [[File]]s used by the Backend it needs a FileHandleCache to manage filehandles dynamically.
 
-Which parts of a File are actually mapped to physical ram is managed by the kernel, to keep a FileMapCache to manage the [[FileMap]]s we've set up.
+Which parts of a File are actually mapped to physical RAM is managed by the kernel, it keeps a FileMapCache to manage the [[FileMap]]s we've set up.
+
+The application itself only requests [[Frame]]s from the backend.
 
 To minimize latency and optimize cpu utilization we have a [[Prefetch]] thread which operates a [[Scheduler]] to render and cache frames which are expected to be consumed soon. This prefetcher keeps [[Statistics]] for optimizing performance.
 
-The core accesses data through smart pointers which are described in [[DataLocking]].
@@ -1779,6 +1781,27 @@ DAMAGE. <html><sub><a href="javascript:;" onclick="scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html> ***/
+
+
There are 2 important points when we want to access data with low latency:
+# Since we handle much more data than it will fit into most computers RAM. The data which is backed in files has to be paged in and available when needed. The [[Prefetch]] Thread manages page hinting to the kernel (posix_madvise()..)
+# Intermediate [[Frame]]s must eventually be rendered to the cache. The Backend will send Renderjobs to the Controler.
+
+Both of these actions are managed by a [[Scheduler]].
+
+Whenever something queries a [[Frame]] from the backend it provides hints about what it is doing.
+These hints contain:
+
+* Timing constraints
+** When will the [[Frame]] be needed
+** could we drop the request when it won't be renderable in-time
+* Priority of this job (as soon as possible, or just in time?)
+* action (Playing forward, playing backward, tweeking, playback speed, recursive rendering of dependent frames)
+
+Notes:
+* the Backend will try to render related frames as as groups
+** this means that following frames are scheduled with lower priority, whenever the program requests them really the priority will be adjusted
+
+
/***
 |''Name:''|RSSReaderPlugin|
@@ -2055,6 +2078,11 @@ see : <<tag RSSFeed>> for the full list.
+
+
Scheduling is done with two priority queues, one for high priority jobs and one for low priority jobs. These priority queues are ordered by absolute time values (and a job identifier, details will be worked out at implementation time).
+
+There is also a variable defining how to handle too-late schedulings (proceed, abort) for each scheduled job.
+
<<search>><<closeAll>><<permaview>><<newTiddler>><<saveChanges>><<slider chkSliderOptionsPanel OptionsPanel "options ยป" "Change TiddlyWiki advanced options">>
From 9c4054b20c495edb5bba2b73dcd90d16437666f7 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 22 Jun 2007 04:14:14 +0200 Subject: [PATCH 4/5] few more DIR_INFO's --- doc/DIR_INFO | 2 ++ doc/devel/uml/DIR_INFO | 1 + uml/DIR_INFO | 2 +- uml/cinelerra3/DIR_INFO | 1 + 4 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 doc/DIR_INFO create mode 100644 doc/devel/uml/DIR_INFO create mode 100644 uml/cinelerra3/DIR_INFO diff --git a/doc/DIR_INFO b/doc/DIR_INFO new file mode 100644 index 000000000..9d0722d69 --- /dev/null +++ b/doc/DIR_INFO @@ -0,0 +1,2 @@ +documentation +source for documentation and on some cases generated documentation in its own subdirs diff --git a/doc/devel/uml/DIR_INFO b/doc/devel/uml/DIR_INFO new file mode 100644 index 000000000..f8fec95f1 --- /dev/null +++ b/doc/devel/uml/DIR_INFO @@ -0,0 +1 @@ +autogenerated UML documentation diff --git a/uml/DIR_INFO b/uml/DIR_INFO index b98ecbf05..896a0163e 100644 --- a/uml/DIR_INFO +++ b/uml/DIR_INFO @@ -1 +1 @@ -uml model, created with bouml +uml models, created with bouml diff --git a/uml/cinelerra3/DIR_INFO b/uml/cinelerra3/DIR_INFO new file mode 100644 index 000000000..c9bfb0e73 --- /dev/null +++ b/uml/cinelerra3/DIR_INFO @@ -0,0 +1 @@ +cinelerra3 UML model From a37d29c4eefbc31ec216fffb43f7ea4b21c268cb Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 22 Jun 2007 04:25:28 +0200 Subject: [PATCH 5/5] add a git-mrproper script which will *delete* all unversioned files --- admin/git-mrproper.sh | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 admin/git-mrproper.sh diff --git a/admin/git-mrproper.sh b/admin/git-mrproper.sh new file mode 100755 index 000000000..46184161a --- /dev/null +++ b/admin/git-mrproper.sh @@ -0,0 +1,11 @@ +#!/bin/sh +if test "$1" = "yes"; then + git ls-files -o | xargs rm -f +else + ( + echo "'git-mrproper.sh yes'" + echo "would delete following files:" + echo + git ls-files -o | while read file; do echo " $file"; done + ) | less +fi