From 8b299e74937960b57603a12c22a00cc6e765d281 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 13 May 2008 20:00:34 +0200 Subject: [PATCH 01/59] add a compatibility/dependecy/style wiki Conflicts: wiki/compatibility.html wiki/index.html --- wiki/index.html | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/wiki/index.html b/wiki/index.html index 1ed553afd..d1bf1bbea 100755 --- a/wiki/index.html +++ b/wiki/index.html @@ -1807,17 +1807,16 @@ Wiki works. It is simple to use and just flexible enough to handle the task. I d Please __end your tiddlers in a newline__, this makes merging in git easier since the /pre tag used in tiddlywiki will become on a single line. ---- -!Architecture and Subsystems +!Design Draft to get started, we create design drafts emphasizing different aspects and regions of Lumiera -* Cehteh works on the data backend, see [[this page|backend.html]] -* Ichthyo focuses mainly on Edit operations and Builder, [[see this separate page|renderengine.html]] -* Gmerlin is in charge of [[GAVL|http://gmerlin.sourceforge.net/gavl.html]] for processing of video data -* as of 4/08 a GuiWorkingGroup is emerging... +* Ichthyo focuses mainly on the Render Engine and its interconnection to the EDL, [[see this separate page|renderengine.html]] +* Cehteh works on the data backend draft, see [[this page|backend.html]] * Some tools which don't fit somewhere else and are used everywhere are put into a [[Support Library|support_library.html]] +* [[Description of the Test System|TestSh]] -!Coding — Building — Testing -how to organize several aspects of the practical coding... +!Coding Structures +next we should //start thinking// on how to organize several aspects of the practical coding... * what to do in BOUML? → [[more|whatInBOUML]] * how to organize packages, files, includes? → [[more|SrcTreeStructure]] * how to organize the executable to be built? From 4b07cd6f856f15d2d1d6c70e10959c3f55fe6cb9 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 22:40:40 +0100 Subject: [PATCH 02/59] Some more notes for the compatibility wiki --- wiki/compatibility.html | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/wiki/compatibility.html b/wiki/compatibility.html index 979f2301c..6eec84d85 100644 --- a/wiki/compatibility.html +++ b/wiki/compatibility.html @@ -2104,13 +2104,26 @@ if (oldText.indexOf("SplashScreen")==-1) } //}}} -
+
! Source code formatting
-http://www.gnu.org/prep/standards/html_node/Formatting.html#Formatting
+We decided to use the 'gnu-style' for indenting (using spaces and never tabs!):
+ http://www.gnu.org/prep/standards/html_node/Formatting.html#Formatting
+
+It is reasonable to be relaxed about soem formatting rules:
+ * line length might be longer when required
+ * inter expession spacing can be changed to the actual needed ex: (2*x + 2)
+
+Things we are pedantic about:
+ * use never ever tab characters in C/C++ sources
+ * be consistent
+ * source files should end with a newline
+ * no trailing or bogus whitespaces
+
+! Coding Practices
+
+! Writing Tests
 
 ! Contibuting
-
-! Tests
From 8d59d601a8c2b78f5f1a9eaca281eb35588b0eae Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 13 May 2008 20:33:44 +0200 Subject: [PATCH 03/59] Add protocol of the 2nd developer meeting to the wiki Conflicts: wiki/index.html --- wiki/index.html | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/wiki/index.html b/wiki/index.html index d1bf1bbea..c0c2e2a5f 100755 --- a/wiki/index.html +++ b/wiki/index.html @@ -930,11 +930,10 @@ git push git://git.pipapo.org/lumiera/mob lumiera/mob is an anonymous account at pipapo.org where everyone can commit changes.
-
+
We keep a protocol or short summary of each important discussion. The summaries of the monthly developer meetings are posted to the Mailinglist and can be found on pipapo.org too
 
-* [[04-08 developer meeting 3.Apr.2008|IRC_2008-04-03]]
-* [[03-08 developer meeting 6.Mar.2008|IRC_2008-03-06]]
+* [[2.official developer meeting 6.March.2008|IRC_2008-03-06]]
 * [[1.official developer meeting 1.Feb.2008|IRC_2008-02-01]]
 * [[informal developer meeting 10.Aug.2007|IRC_2007-08-10]]
From 19ee8afa087b0e0ade66c4e04e29d8e2d4bdd47e Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 16:44:55 +0100 Subject: [PATCH 04/59] safeclib wraps some common c library functions and adds some tools * lumiera_malloc which succeeds or dies * some safe string functions * Thread local round robin buffers for temporary data --- src/lib/Makefile.am | 7 +- src/lib/safeclib.c | 181 ++++++++++++++++++++++++++++++++++++++++++++ src/lib/safeclib.h | 97 ++++++++++++++++++++++++ 3 files changed, 283 insertions(+), 2 deletions(-) create mode 100644 src/lib/safeclib.c create mode 100644 src/lib/safeclib.h diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 77fc26d90..6ecd00e97 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -28,7 +28,9 @@ liblumi_a_SOURCES = \ $(liblumi_a_srcdir)/mutex.c \ $(liblumi_a_srcdir)/rwlock.c \ $(liblumi_a_srcdir)/condition.c \ - $(liblumi_a_srcdir)/references.c + $(liblumi_a_srcdir)/references.c \ + $(liblumi_a_srcdir)/safeclib.c + noinst_HEADERS += \ @@ -39,5 +41,6 @@ noinst_HEADERS += \ $(liblumi_a_srcdir)/mutex.h \ $(liblumi_a_srcdir)/rwlock.h \ $(liblumi_a_srcdir)/condition.h \ - $(liblumi_a_srcdir)/references.h + $(liblumi_a_srcdir)/references.h \ + $(liblumi_a_srcdir)/safeclib.h diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c new file mode 100644 index 000000000..58116ae90 --- /dev/null +++ b/src/lib/safeclib.c @@ -0,0 +1,181 @@ +/* + safe_clib.c - Portable and safe wrapers around some clib functions and some tools + + Copyright (C) CinelerraCV + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#define _GNU_SOURCE + +#include "error.h" + +#include +#include +#include +#include + +/** + * @file Portable and safe wrapers around some clib functions and some tools + */ + +LUMIERA_ERROR_DEFINE (NO_MEMORY, "Out of Memory!"); + +/** + * Allocate memory. + * always succeeds or dies + * @param size memory to be allocated + * @return pointer to the allocated memory + */ +void* +lumiera_malloc (size_t sz) +{ + void* o = malloc (sz); + if (!o) + LUMIERA_DIE (NO_MEMORY); + return o; +} + + +/** + * Duplicate a C string. + * always succeeds or dies + * @param str string to be copied + * @param len maximal length to be copied + * @return pointer to the new string, "" if NULL was passed as str + */ +char* +lumiera_strndup (const char* str, size_t len) +{ + char* o; + if (str) + o = strndup (str, len); + else + o = strdup (""); + + if (!o) + LUMIERA_DIE (NO_MEMORY); + return o; +} + + +/** + * Compare two C strings. + * Handles NULL pointers as "", shortcut for same addresses + * @param a first string for comparsion + * @param b second string for comparsion + * @param len maximal length for the comparsion + * @return 0 if the strings are identical, -1 if smaller 1 if bigger. + */ +int +lumiera_strncmp (const char* a, const char* b, size_t len) +{ + return a == b ? 0 : strncmp (a?a:"", b?b:"", len); +} + + +/** + * check 2 strings for identity. + * @param a first string for comparsion + * @param b second string for comparsion + * @return 1 when the strings are the the same, 0 if not. + */ +int +lumiera_streq (const char* a, const char* b) +{ + return !lumiera_strncmp (a, b, SIZE_MAX); +} + + +/** + * Round robin temporary buffers. + * This provides 64 buffers per thread which are recycled with each use. + * The idea here is to have fast buffers for temporal data without need for explicit heap management. + */ +struct lumiera_buffer_struct +{ + void* buffers[64]; + size_t sizes[64]; + unsigned idx; +}; + +static pthread_once_t lumiera_buffer_tls_once = PTHREAD_ONCE_INIT; +static pthread_key_t lumiera_buffer_tls_key; + +void +lumiera_buffer_freeall (void); + +static void +lumiera_buffer_destroy (void* buf) +{ + lumiera_buffer_freeall (); + free (buf); +} + +static void +lumiera_buffer_init (void) +{ + pthread_key_create (&lumiera_buffer_tls_key, lumiera_buffer_destroy); +} + + +/** + * free all buffers associated with this thread. + * This function is called automatically, usually one doesnt need to call it. + */ +void +lumiera_buffer_freeall (void) +{ + pthread_once (&lumiera_buffer_tls_once, lumiera_buffer_init); + struct lumiera_buffer_struct* buf = pthread_getspecific (lumiera_buffer_tls_key); + if (!buf) + pthread_setspecific (lumiera_buffer_tls_key, + buf = lumiera_malloc (sizeof (struct lumiera_buffer_struct))); + + for (buf->idx = 0; buf->idx < 64; ++buf->idx) + { + free (buf->buffers[buf->idx]); + buf->buffers[buf->idx] = NULL; + buf->sizes[buf->idx] = 0; + } + buf->idx = 0; +} + + +/** + * Query a thread local buffer. + * @param size minimal needed size for the buffer + * @return the buffer + */ +void* +lumiera_buffer_provide (size_t size) +{ + pthread_once (&lumiera_buffer_tls_once, lumiera_buffer_init); + struct lumiera_buffer_struct* buf = pthread_getspecific (lumiera_buffer_tls_key); + if (!buf) + pthread_setspecific (lumiera_buffer_tls_key, + buf = lumiera_malloc (sizeof (struct lumiera_buffer_struct))); + + buf->idx = (buf->idx + 1) & 0x3f; + + if (buf->sizes[buf->idx] < size || buf->sizes[buf->idx] > 8*size) + { + free (buf->buffers[buf->idx]); + buf->sizes[buf->idx] = (size+4*sizeof(long)) & ~(4*sizeof(long)-1); + buf->buffers[buf->idx] = lumiera_malloc (buf->sizes[buf->idx]); + } + return buf->buffers[buf->idx]; +} + diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h new file mode 100644 index 000000000..e4e803acc --- /dev/null +++ b/src/lib/safeclib.h @@ -0,0 +1,97 @@ +/* + safe_clib.h - Portable and safe wrapers around some clib functions and some tools + + Copyright (C) CinelerraCV + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "error.h" + +//#include +//#include +//#include + +/** + * @file Portable and safe wrapers around some clib functions and some tools + */ +LUMIERA_ERROR_DECLARE(NO_MEMORY); + +/** + * Allocate memory. + * always succeeds or dies + * @param size memory to be allocated + * @return pointer to the allocated memory + */ +void* +lumiera_malloc (size_t sz); + + +/** + * Duplicate a C string. + * always succeeds or dies + * @param str string to be copied + * @param len maximal length to be copied + * @return pointer to the new string, "" if NULL was passed as str + */ +char* +lumiera_strndup (const char* str, size_t len); + + +/** + * Compare two C strings. + * Handles NULL pointers as "", shortcut for same addresses + * @param a first string for comparsion + * @param b second string for comparsion + * @param len maximal length for the comparsion + * @return 0 if the strings are identical, -1 if smaller 1 if bigger. + */ +int +lumiera_strncmp (const char* a, const char* b, size_t len); + + +/** + * check 2 strings for identity. + * @param a first string for comparsion + * @param b second string for comparsion + * @return 1 when the strings are the the same, 0 if not. + */ +int +lumiera_streq (const char* a, const char* b); + + +/** + * Round robin temporary buffers. + * This provides 64 buffers per thread which are recycled with each use. + * The idea here is to have fast buffers for temporal data without need for explicit heap management. + */ + +/** + * free all buffers associated with this thread. + * This function is called automatically, usually one doesnt need to call it. + */ +void +lumiera_buffer_freeall (void); + + +/** + * Query a thread local buffer. + * @param size minimal needed size for the buffer + * @return the buffer + */ +void* +lumiera_buffer_provide (size_t size); + From 8b162cada426ac46229e442c7ba8767033e27686 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 16:48:44 +0100 Subject: [PATCH 05/59] llist update --- src/lib/llist.h | 96 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 76 insertions(+), 20 deletions(-) diff --git a/src/lib/llist.h b/src/lib/llist.h index 4f77b20dc..efb3f5202 100644 --- a/src/lib/llist.h +++ b/src/lib/llist.h @@ -88,6 +88,7 @@ typedef llist * LList; typedef const llist * const_LList; typedef llist ** LList_ref; + /** * Macro to instantiate a local llist. * @param name of the llist node @@ -95,6 +96,14 @@ typedef llist ** LList_ref; #define LLIST_AUTO(name) llist name = {&name,&name} +/* + some macros for convenience +*/ +#define llist_insert_head(list, element) llist_insert_next (list, element) +#define llist_insert_tail(list, element) llist_insert_prev (list, element) +#define llist_head llist_next +#define llist_tail llist_prev + /** * cast back from a member of a structure to a pointer of the structure */ @@ -129,6 +138,30 @@ typedef llist ** LList_ref; ! llist_is_end (node, list); \ llist_backward (&node)) + +/** + * Iterate forward over a range. + * @param start first node to be interated + * @param end node after the last node be iterated + * @param node pointer to the iterated node + */ +#define LLIST_FORRANGE(start, end, node) \ + for (LList node = start; \ + node != end; \ + llist_forward (&node)) + +/** + * Iterate backward over a range. + * @param rstart first node to be interated + * @param rend node before the last node be iterated + * @param node pointer to the iterated node + */ +#define LLIST_FORRANGE_REV(rstart, rend, node) \ + for (LList node = rstart; \ + node != rend; \ + llist_backward (&node)) + + /** * Consume a list from head. * The body of this statement should remove the head from the list, else it would be a infinite loop @@ -290,7 +323,7 @@ LLIST_FUNC (LList llist_insert_next (LList self, LList next), /** * Insert a node before another. * @param self node before which we want to insert - * @param prev node which shall be inserted nefore self. Could already linked to a list from where it will be removed. + * @param prev node which shall be inserted before self. Could already linked to a list from where it will be removed. * @return self */ LLIST_FUNC (LList llist_insert_prev (LList self, LList prev), @@ -314,10 +347,9 @@ LLIST_FUNC (LList llist_insertlist_next (LList self, LList next), if (!llist_is_empty (next)) { self->next->prev = next->prev; - self->next = next->next; - - next->next->prev = self; next->prev->next = self->next; + self->next = next->next; + next->next->prev = self; next->prev = next->next = next; } @@ -335,10 +367,9 @@ LLIST_FUNC (LList llist_insertlist_prev (LList self, LList prev), if (!llist_is_empty (prev)) { self->prev->next = prev->next; - self->prev = prev->prev; - - prev->prev->next = self; prev->next->prev = self->prev; + self->prev = prev->prev; + prev->prev->next = self; prev->prev = prev->next = prev; } @@ -494,20 +525,45 @@ LLIST_FUNC (LList llist_get_nth_stop (LList self, int n, const_LList stop), return self; ); -/* - some macros for convenience -*/ -#define llist_insert_head(list, element) llist_insert_next (list, element) -#define llist_insert_tail(list, element) llist_insert_prev (list, element) -#define llist_head llist_next -#define llist_tail llist_prev + +/** + * Sort a list. + * @param self list to be sorted + * @param cmp function takeing 2 LLists + * simple recursive mergesort + */ +typedef int (*llist_cmpfn)(LList a, LList b); + +LLIST_FUNC (LList llist_sort (LList self, llist_cmpfn cmp), + llist left; + llist right; + llist_init (&left); + llist_init (&right); + unsigned n = 0; + if (!llist_is_single (self)) + { + LLIST_WHILE_HEAD (self, head) + llist_insert_prev (++n & 1 ? &left : &right, head); + + llist_sort (&left, cmp); + llist_sort (&right, cmp); + + while (!llist_is_empty (&left) && !llist_is_empty (&right)) + llist_insert_prev (self, cmp (left.next, right.next) < 0 ? left.next : right.next); + + if (!llist_is_empty (&left)) + llist_insertlist_prev (self, &left); + if (!llist_is_empty (&right)) + llist_insertlist_prev (self, &right); + } + return self; +) #endif /* LLIST_H */ /* -// Local Variables: -// mode: C -// c-file-style: "gnu" -// End: -// arch-tag: e8fe4a59-fd55-4c45-b860-5cd1e0771213 -// end_of_file +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: */ From 27ca8a73620399f94981a4a87b242c2e0103388c Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 18:09:56 +0100 Subject: [PATCH 06/59] let LUMIERA_DIE take an error identifier as parameter --- src/lib/condition.c | 6 ++++-- src/lib/condition.h | 15 ++++++++------- src/lib/error.h | 6 ++++-- src/lib/locking.h | 5 +++++ src/lib/mutex.c | 6 +++++- src/lib/mutex.h | 9 ++++----- src/lib/plugin.c | 15 ++++++++------- src/lib/references.c | 4 ++-- src/lib/rwlock.c | 14 +++++++++----- src/lib/rwlock.h | 6 +++++- tests/plugin/plugin_main.c | 9 +++++---- 11 files changed, 59 insertions(+), 36 deletions(-) diff --git a/src/lib/condition.c b/src/lib/condition.c index 7427b431d..1bfa2ef96 100644 --- a/src/lib/condition.c +++ b/src/lib/condition.c @@ -25,6 +25,8 @@ * @file Condition variables */ +LUMIERA_ERROR_DEFINE (CONDITION_DESTROY, "condition destroy failed"); + /** * Initialize a condition variable * @param self is a pointer to the condition variable to be initialized @@ -53,9 +55,9 @@ lumiera_condition_destroy (LumieraCondition self) if (self) { if (pthread_mutex_destroy (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_DESTROY); else if (pthread_cond_destroy (&self->cond)) - LUMIERA_DIE; + LUMIERA_DIE (CONDITION_DESTROY); } return self; } diff --git a/src/lib/condition.h b/src/lib/condition.h index b2fd76c63..066c6b267 100644 --- a/src/lib/condition.h +++ b/src/lib/condition.h @@ -28,6 +28,7 @@ * @file Condition variables, header */ +LUMIERA_ERROR_DECLARE (CONDITION_DESTROY); /** * Condition variables. @@ -59,10 +60,10 @@ lumiera_condition_signal (LumieraCondition self) { REQUIRE (self); if (pthread_mutex_lock (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); pthread_cond_signal (&self->cond); if (pthread_mutex_unlock (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_UNLOCK); } /** @@ -74,10 +75,10 @@ lumiera_condition_broadcast (LumieraCondition self) { REQUIRE (self); if (pthread_mutex_lock (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); pthread_cond_broadcast (&self->cond); if (pthread_mutex_unlock (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_UNLOCK); } @@ -124,7 +125,7 @@ lumiera_conditionacquirer_init (LumieraConditionacquirer self, LumieraCondition self->state = state; if (state == LUMIERA_LOCKED) if (pthread_mutex_lock (&cond->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); return self; } @@ -141,7 +142,7 @@ lumiera_conditionacquirer_lock (LumieraConditionacquirer self) REQUIRE (self->state == LUMIERA_UNLOCKED, "mutex already locked"); if (pthread_mutex_lock (&self->cond->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); self->state = LUMIERA_LOCKED; } @@ -172,7 +173,7 @@ lumiera_conditionacquirer_unlock (LumieraConditionacquirer self) REQUIRE (self); REQUIRE (self->state == LUMIERA_LOCKED, "mutex was not locked"); if (pthread_mutex_unlock (&self->cond->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_UNLOCK); self->state = LUMIERA_UNLOCKED; } diff --git a/src/lib/error.h b/src/lib/error.h index 24c73f0a5..e71c0cdfa 100644 --- a/src/lib/error.h +++ b/src/lib/error.h @@ -39,7 +39,8 @@ extern "C" { * Abort unconditionally with a 'Fatal Error!' message. * This macro is used whenever the program end up in a invalid state from which no runtime recovery is possible */ -#define LUMIERA_DIE do { NOBUG_ERROR(NOBUG_ON, "Fatal Error!"); abort(); } while(0) +#define LUMIERA_DIE(err) \ + do { NOBUG_ERROR(NOBUG_ON, "Fatal Error: %s ", strchr(LUMIERA_ERROR_##err, ':')); abort(); } while(0) /** * Forward declare an error constant. @@ -58,7 +59,8 @@ extern const char* LUMIERA_ERROR_##err #define LUMIERA_ERROR_DEFINE(err, msg) \ const char* LUMIERA_ERROR_##err = "LUMIERA_ERROR_" #err ":" msg -/** Helper macro to raise an error for the current thread. +/** + * Helper macro to raise an error for the current thread. * This macro eases setting an error. It adds NoBug logging support to the low level error handling. * @param flag NoBug flag describing the subsystem where the error was raised * @param err name of the error without the 'LUMIERA_ERROR_' prefix (example: NO_MEMORY) diff --git a/src/lib/locking.h b/src/lib/locking.h index 01ad8bb29..f62dd6c29 100644 --- a/src/lib/locking.h +++ b/src/lib/locking.h @@ -28,6 +28,11 @@ #include "lib/error.h" + +LUMIERA_ERROR_DECLARE (MUTEX_LOCK); +LUMIERA_ERROR_DECLARE (MUTEX_UNLOCK); +LUMIERA_ERROR_DECLARE (MUTEX_DESTROY); + /** * @file Shared declarations for all locking primitives. */ diff --git a/src/lib/mutex.c b/src/lib/mutex.c index a08d89e51..009502e5a 100644 --- a/src/lib/mutex.c +++ b/src/lib/mutex.c @@ -25,6 +25,10 @@ * @file Mutual exclusion locking. */ +LUMIERA_ERROR_DEFINE (MUTEX_LOCK, "Mutex locking failed"); +LUMIERA_ERROR_DEFINE (MUTEX_UNLOCK, "Mutex unlocking failed"); +LUMIERA_ERROR_DEFINE (MUTEX_DESTROY, "Mutex destroy failed"); + /** * Initialize a mutex variable @@ -52,7 +56,7 @@ lumiera_mutex_destroy (LumieraMutex self) if (self) { if (pthread_mutex_destroy (&self->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_DESTROY); } return self; } diff --git a/src/lib/mutex.h b/src/lib/mutex.h index ced7118e2..ecd99529d 100644 --- a/src/lib/mutex.h +++ b/src/lib/mutex.h @@ -28,7 +28,6 @@ * @file Mutual exclusion locking, header. */ - /** * Mutex. * @@ -106,7 +105,7 @@ lumiera_mutexacquirer_init_mutex (LumieraMutexacquirer self, LumieraMutex mutex, self->state = state; if (state == LUMIERA_LOCKED) if (pthread_mutex_lock (&mutex->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); return self; } @@ -124,7 +123,7 @@ lumiera_mutexacquirer_lock (LumieraMutexacquirer self) REQUIRE (self->state == LUMIERA_UNLOCKED, "mutex already locked"); if (pthread_mutex_lock (&self->mutex->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); self->state = LUMIERA_LOCKED; } @@ -164,7 +163,7 @@ lumiera_mutexacquirer_try_mutex (LumieraMutexacquirer self, LumieraMutex mutex) case EBUSY: return LUMIERA_UNLOCKED; default: - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_LOCK); } } @@ -180,7 +179,7 @@ lumiera_mutexacquirer_unlock (LumieraMutexacquirer self) REQUIRE (self); REQUIRE (self->state == LUMIERA_LOCKED, "mutex was not locked"); if (pthread_mutex_unlock (&self->mutex->mutex)) - LUMIERA_DIE; + LUMIERA_DIE (MUTEX_UNLOCK); self->state = LUMIERA_UNLOCKED; } diff --git a/src/lib/plugin.c b/src/lib/plugin.c index 87939533d..8345d9887 100644 --- a/src/lib/plugin.c +++ b/src/lib/plugin.c @@ -25,8 +25,10 @@ #include #include #include +#include #include "plugin.h" +#include "safeclib.h" /** * @file Plugin loader. @@ -159,8 +161,8 @@ lumiera_plugin_lookup (struct lumiera_plugin* self, const char* path) { /* got it */ TRACE (lumiera_plugin, "found %s", pathname); - self->pathname = strdup (pathname); - if (!self->pathname) LUMIERA_DIE; + self->pathname = lumiera_strndup (pathname, PATH_MAX); + self->type = lumiera_plugin_extensions[i].type; return 0; } @@ -195,21 +197,20 @@ lumiera_interface_open (const char* name, const char* interface, size_t min_revi plugin.name = name; /* for searching */ found = tsearch (&plugin, &lumiera_plugin_registry, lumiera_plugin_name_cmp); - if (!found) LUMIERA_DIE; + if (!found) + LUMIERA_DIE (NO_MEMORY); if (*found == &plugin) { NOTICE (lumiera_plugin, "new plugin"); /* now really create new item */ - *found = malloc (sizeof (struct lumiera_plugin)); - if (!*found) LUMIERA_DIE; + *found = lumiera_malloc (sizeof (struct lumiera_plugin)); if (name) /* NULL is main app, no lookup needed */ { /*lookup for $LUMIERA_PLUGIN_PATH*/ - (*found)->name = strdup (name); - if (!(*found)->name) LUMIERA_DIE; + (*found)->name = lumiera_strndup (name, PATH_MAX); if (!!lumiera_plugin_lookup (*found, getenv("LUMIERA_PLUGIN_PATH")) #ifdef LUMIERA_PLUGIN_PATH diff --git a/src/lib/references.c b/src/lib/references.c index ad4f9562f..3e9d5f260 100644 --- a/src/lib/references.c +++ b/src/lib/references.c @@ -19,6 +19,7 @@ Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ #include "lib/references.h" +#include "lib/safeclib.h" /** * @file Strong and Weak references @@ -43,8 +44,7 @@ LumieraReference lumiera_reference_strong_init_once (LumieraReference self, void* obj, void (*dtor)(void*)) { - LumieraReftarget target = malloc (sizeof(lumiera_reftarget)); - if (!target) LUMIERA_DIE; + LumieraReftarget target = lumiera_malloc (sizeof(lumiera_reftarget)); target->object = obj; target->dtor = dtor; diff --git a/src/lib/rwlock.c b/src/lib/rwlock.c index c7ee3a37d..0862aa808 100644 --- a/src/lib/rwlock.c +++ b/src/lib/rwlock.c @@ -25,6 +25,10 @@ LUMIERA_ERROR_DEFINE(RWLOCK_AGAIN, "maximum number of readlocks exceed"); LUMIERA_ERROR_DEFINE(RWLOCK_DEADLOCK, "deadlock detected"); +LUMIERA_ERROR_DEFINE(RWLOCK_DESTROY, "destroy rwlock"); +LUMIERA_ERROR_DEFINE(RWLOCK_UNLOCK, "unlock"); +LUMIERA_ERROR_DEFINE(RWLOCK_RLOCK, "rlock"); +LUMIERA_ERROR_DEFINE(RWLOCK_WLOCK, "wlock"); /** * @file Read/write locks. @@ -57,7 +61,7 @@ lumiera_rwlock_destroy (LumieraRWLock self) if (self) { if (pthread_rwlock_destroy (&self->rwlock)) - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_DESTROY); } return self; } @@ -95,7 +99,7 @@ lumiera_rwlockacquirer_init (LumieraRWLockacquirer self, LumieraRWLock rwlock, e lumiera_error_set (LUMIERA_ERROR_RWLOCK_DEADLOCK); return NULL; default: - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_RLOCK); } case LUMIERA_WRLOCKED: switch (pthread_rwlock_wrlock (&rwlock->rwlock)) @@ -106,7 +110,7 @@ lumiera_rwlockacquirer_init (LumieraRWLockacquirer self, LumieraRWLock rwlock, e lumiera_error_set (LUMIERA_ERROR_RWLOCK_DEADLOCK); return NULL; default: - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_WLOCK); } default: break; @@ -138,7 +142,7 @@ lumiera_rwlockacquirer_rdlock (LumieraRWLockacquirer self) lumiera_error_set (LUMIERA_ERROR_RWLOCK_DEADLOCK); return NULL; default: - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_RLOCK); } self->state = LUMIERA_RDLOCKED; @@ -166,7 +170,7 @@ lumiera_rwlockacquirer_wrlock (LumieraRWLockacquirer self) lumiera_error_set (LUMIERA_ERROR_RWLOCK_DEADLOCK); return NULL; default: - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_WLOCK); } self->state = LUMIERA_WRLOCKED; diff --git a/src/lib/rwlock.h b/src/lib/rwlock.h index 922a05e7b..5e121fbcc 100644 --- a/src/lib/rwlock.h +++ b/src/lib/rwlock.h @@ -33,6 +33,10 @@ LUMIERA_ERROR_DECLARE(RWLOCK_AGAIN); LUMIERA_ERROR_DECLARE(RWLOCK_DEADLOCK); +LUMIERA_ERROR_DECLARE(RWLOCK_DESTROY); +LUMIERA_ERROR_DECLARE(RWLOCK_UNLOCK); +LUMIERA_ERROR_DECLARE(RWLOCK_RLOCK); +LUMIERA_ERROR_DECLARE(RWLOCK_WLOCK); /** * @file Read/write locks, header. @@ -106,7 +110,7 @@ lumiera_rwlockacquirer_unlock (LumieraRWLockacquirer self) REQUIRE (self); REQUIRE (self->state != LUMIERA_UNLOCKED, "rwlock was not locked"); if (pthread_rwlock_unlock (&self->rwlock->rwlock)) - LUMIERA_DIE; + LUMIERA_DIE (RWLOCK_UNLOCK); self->state = LUMIERA_UNLOCKED; } diff --git a/tests/plugin/plugin_main.c b/tests/plugin/plugin_main.c index 2efe32471..4b95ea60c 100644 --- a/tests/plugin/plugin_main.c +++ b/tests/plugin/plugin_main.c @@ -3,6 +3,7 @@ #include "lib/plugin.h" #include "hello_interface.h" +LUMIERA_ERROR_DEFINE(FAILURE, "test failure"); int main(int argc, char** argv) @@ -25,7 +26,7 @@ main(int argc, char** argv) LUMIERA_INTERFACE_TYPE(hello, 1)* hello_de = (LUMIERA_INTERFACE_TYPE(hello, 1)*) lumiera_interface_open ("example_plugin", "german_1", sizeof(LUMIERA_INTERFACE_TYPE(hello, 1))); - if (!hello_de) LUMIERA_DIE; + if (!hello_de) LUMIERA_DIE (FAILURE); hello_de->hello(); hello_de->goodbye(argv[1]); @@ -33,7 +34,7 @@ main(int argc, char** argv) LUMIERA_INTERFACE_TYPE(hello, 1)* hello_en = (LUMIERA_INTERFACE_TYPE(hello, 1)*) lumiera_interface_open ("example_plugin", "english_1", sizeof(LUMIERA_INTERFACE_TYPE(hello, 1))); - if (!hello_en) LUMIERA_DIE; + if (!hello_en) LUMIERA_DIE (FAILURE); hello_en->hello(); hello_en->goodbye(argv[1]); @@ -48,7 +49,7 @@ main(int argc, char** argv) LUMIERA_INTERFACE_TYPE(hello, 1)* hello_de = (LUMIERA_INTERFACE_TYPE(hello, 1)*) lumiera_interface_open ("example_plugin_cpp", "german_1", sizeof(LUMIERA_INTERFACE_TYPE(hello, 1))); - if (!hello_de) LUMIERA_DIE; + if (!hello_de) LUMIERA_DIE (FAILURE); hello_de->hello(); hello_de->goodbye(argv[1]); @@ -56,7 +57,7 @@ main(int argc, char** argv) LUMIERA_INTERFACE_TYPE(hello, 1)* hello_en = (LUMIERA_INTERFACE_TYPE(hello, 1)*) lumiera_interface_open ("example_plugin_cpp", "english_1", sizeof(LUMIERA_INTERFACE_TYPE(hello, 1))); - if (!hello_en) LUMIERA_DIE; + if (!hello_en) LUMIERA_DIE (FAILURE); hello_en->hello(); hello_en->goodbye(argv[1]); From 0b8b5bf5078ddf98955d56dc1216ab36fb6290a0 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 18:11:03 +0100 Subject: [PATCH 07/59] uuid functions --- src/lib/Makefile.am | 2 + src/lib/uuid.c | 94 +++++++++++++++++++++++++++++++++++++++ src/lib/uuid.h | 66 +++++++++++++++++++++++++++ tests/15uuid.tests | 15 +++++++ tests/Makefile.am | 5 +++ tests/library/test-uuid.c | 71 +++++++++++++++++++++++++++++ 6 files changed, 253 insertions(+) create mode 100644 src/lib/uuid.c create mode 100644 src/lib/uuid.h create mode 100644 tests/15uuid.tests create mode 100644 tests/library/test-uuid.c diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 6ecd00e97..844f817a6 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -29,6 +29,7 @@ liblumi_a_SOURCES = \ $(liblumi_a_srcdir)/rwlock.c \ $(liblumi_a_srcdir)/condition.c \ $(liblumi_a_srcdir)/references.c \ + $(liblumi_a_srcdir)/uuid.c \ $(liblumi_a_srcdir)/safeclib.c @@ -42,5 +43,6 @@ noinst_HEADERS += \ $(liblumi_a_srcdir)/rwlock.h \ $(liblumi_a_srcdir)/condition.h \ $(liblumi_a_srcdir)/references.h \ + $(liblumi_a_srcdir)/uuid.h \ $(liblumi_a_srcdir)/safeclib.h diff --git a/src/lib/uuid.c b/src/lib/uuid.c new file mode 100644 index 000000000..e313df992 --- /dev/null +++ b/src/lib/uuid.c @@ -0,0 +1,94 @@ +/* + uuid - Universal unique identifiers + + Copyright (C) CinelerraCV + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "uuid.h" + +#include +#include +#include +#include +#include + +void +lumiera_uuid_set_ptr (lumiera_uuid* uuid, void* ptr) +{ + memset (uuid, 0, 16); + *(void**)uuid = ptr; +} + + +void* +lumiera_uuid_ptr_get (lumiera_uuid* uuid) +{ + return *(void**)uuid; +} + + +void +lumiera_uuid_gen (lumiera_uuid* uuid) +{ + static int fd = -2; + if (!uuid) + return; + + if (fd == -2) + { + fd = open ("/dev/urandom", O_RDONLY); + if (fd == -1) + fd = open ("/dev/random", O_RDONLY); + if (fd >= 0) + fcntl (fd, F_SETFD, FD_CLOEXEC); + else + srand (getpid () + time (NULL)); + } + if (fd < 0) + { + for (int i = 0; i < 16; ++i) + ((unsigned char*)uuid)[i] = (unsigned char)(rand()>>7); + } + else + { + if (read (fd, uuid, 16) < 0) + abort (); + } +} + + +void +lumiera_uuid_copy (lumiera_uuid* dest, lumiera_uuid* src) +{ + memcpy (dest, src, 16); +} + + +int +lumiera_uuid_eq (lumiera_uuid* uuida, lumiera_uuid* uuidb) +{ + return !memcmp (uuida, uuidb, 16); +} + +size_t +lumiera_uuid_hash (lumiera_uuid* uuid) +{ + return *(size_t*)uuid; +} + + diff --git a/src/lib/uuid.h b/src/lib/uuid.h new file mode 100644 index 000000000..a1ab6fa6a --- /dev/null +++ b/src/lib/uuid.h @@ -0,0 +1,66 @@ +/* + uuid - Universal unique identifiers + + Copyright (C) CinelerraCV + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#ifndef LUMIERA_UUID_H +#define LUMIERA_UUID_H +#include "lumiera.h" + +typedef unsigned char lumiera_uuid[16]; + +/** + * Retrieve a generic pointer stored in a uuid + */ +void* +lumiera_uuid_ptr_get (lumiera_uuid* uuid); + +/** + * Generate a new uuid + */ +void +lumiera_uuid_gen (lumiera_uuid* uuid); + +/** + * Store a generic pointer in a uuid + */ +void +lumiera_uuid_set_ptr (lumiera_uuid* uuid, void* ptr); + + +/** + * Copy an uuid + */ +void +lumiera_uuid_copy (lumiera_uuid* dest, lumiera_uuid* src); + + +/** + * Test 2 uuid's for equality + */ +int +lumiera_uuid_eq (lumiera_uuid* uuida, lumiera_uuid* uuidb); + + +/** + * Generate a hashsum over an uuid + */ +size_t +lumiera_uuid_hash (lumiera_uuid* uuid); + +#endif diff --git a/tests/15uuid.tests b/tests/15uuid.tests new file mode 100644 index 000000000..072b2545e --- /dev/null +++ b/tests/15uuid.tests @@ -0,0 +1,15 @@ +TESTING "UUID's" ./test-uuid + +TEST "uuid generate" uuidgen_2 < + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +//#include "cinelerra-config.h" + +//#include +//#include + +#include "lib/uuid.h" + +//CINELERRA_ERROR_DEFINE(TEST, "test error"); + +int +main (int argc, char** argv) +{ + NOBUG_INIT; + + if (argc == 1) + return 0; + + if (!strcmp(argv[1], "uuidgen_2")) + { + lumiera_uuid uuid1; + lumiera_uuid uuid2; + + lumiera_uuid_gen (&uuid1); + lumiera_uuid_gen (&uuid2); + + printf ("%d\n", lumiera_uuid_eq (&uuid2, &uuid1)); + } + else if (!strcmp(argv[1], "uuidgen_copy")) + { + lumiera_uuid uuid1; + lumiera_uuid uuid2; + + lumiera_uuid_gen (&uuid1); + + lumiera_uuid_copy (&uuid2, &uuid1); + + printf ("%d\n", lumiera_uuid_eq (&uuid2, &uuid1)); + } + else if (!strcmp(argv[1], "ptrs")) + { + lumiera_uuid uuid; + + lumiera_uuid_set_ptr (&uuid, &uuid); + + printf ("%d\n", lumiera_uuid_ptr_get (&uuid) == &uuid); + } + else + return 1; + + return 0; +} From 646d6fa0d3370e17d79028401dba7d79cdacbc15 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 19:26:51 +0100 Subject: [PATCH 08/59] Test simplification, plaything * test.h provides some macros to aid test writing * applied that to some tests --- tests/library/test-llist.c | 321 +++++++++++++++++------------------ tests/locking/test-locking.c | 34 ++-- tests/test.h | 52 ++++++ 3 files changed, 220 insertions(+), 187 deletions(-) create mode 100644 tests/test.h diff --git a/tests/library/test-llist.c b/tests/library/test-llist.c index 12592aa4a..66bfa614d 100644 --- a/tests/library/test-llist.c +++ b/tests/library/test-llist.c @@ -23,173 +23,166 @@ //#include #include "lib/llist.h" -#include "lib/error.h" +#include "tests/test.h" +TESTS_BEGIN -LUMIERA_ERROR_DEFINE(TEST, "test error"); - -int -main (int argc, char** argv) +TEST ("basic") { - NOBUG_INIT; + LLIST_AUTO (node1); - if (argc == 1) - return 0; + llist node2; + llist_init (&node2); - if (!strcmp(argv[1], "basic")) - { - LLIST_AUTO (node1); - - llist node2; - llist_init (&node2); - - printf ("%d\n", llist_is_empty (&node1)); - printf ("%d\n", llist_is_empty (&node2)); - } - else if (!strcmp(argv[1], "nodeinsert")) - { - LLIST_AUTO (list); - LLIST_AUTO (node1); - LLIST_AUTO (node2); - LLIST_AUTO (node3); - - llist_insert_next (&list, &node1); - printf ("%d\n", llist_is_empty (&list)); - printf ("%d\n", llist_is_empty (&node1)); - printf ("%d\n", llist_is_single (&node1)); - llist_insert_next (&node1, &node2); - printf ("%d\n", llist_is_single (&node1)); - llist_insert_prev (&node1, &node3); - printf ("%d\n", llist_next (&list) == &node3); - printf ("%d\n", llist_next (&node3) == &node1); - printf ("%d\n", llist_next (&node1) == &node2); - printf ("%d\n", llist_prev (&list) == &node2); - printf ("%d\n", llist_count (&list)); - } - else if (!strcmp(argv[1], "predicates")) - { - LLIST_AUTO (list); - LLIST_AUTO (node1); - LLIST_AUTO (node2); - LLIST_AUTO (node3); - LLIST_AUTO (node4); - LLIST_AUTO (nil); - - llist_insert_tail (&list, &node2); - llist_insert_tail (&list, &node3); - llist_insert_tail (&list, &node4); - llist_insert_head (&list, &node1); - - printf ("%d\n", llist_is_head (&list, &node1)); - printf ("%d\n", llist_is_tail (&list, &node4)); - printf ("%d\n", llist_is_head (&list, &node4)); - printf ("%d\n", llist_is_tail (&list, &node1)); - printf ("%d\n", llist_is_end (&list, &list)); - printf ("%d\n", llist_is_member (&list, &node3)); - printf ("%d\n", llist_is_member (&list, &nil)); - - printf ("%d\n", llist_is_before_after (&list, &node1, &node3)); - printf ("%d\n", llist_is_before_after (&list, &node3, &node1)); - printf ("%d\n", llist_is_before_after (&list, &node1, &nil)); - } - else if (!strcmp(argv[1], "unlink")) - { - LLIST_AUTO (list); - LLIST_AUTO (node1); - LLIST_AUTO (node2); - LLIST_AUTO (node3); - LLIST_AUTO (node4); - LLIST_AUTO (nil); - - llist_insert_tail (&list, &node2); - llist_insert_tail (&list, &node3); - llist_insert_tail (&list, &node4); - llist_insert_head (&list, &node1); - - LLIST_FOREACH_REV (&list, itr) - { - if(itr == &node1) printf ("node1 "); - else if(itr == &node2) printf ("node2 "); - else if(itr == &node3) printf ("node3 "); - else if(itr == &node4) printf ("node4 "); - else printf ("unknown "); - } - printf (".\n"); - - llist_unlink (&nil); - llist_unlink (&node2); - llist_unlink (&node3); - - LLIST_FOREACH (&list, itr) - { - if(itr == &node1) printf ("node1 "); - else if(itr == &node2) printf ("node2 "); - else if(itr == &node3) printf ("node3 "); - else if(itr == &node4) printf ("node4 "); - else printf ("unknown "); - } - printf (".\n"); - printf ("%d\n", llist_is_empty (&node2)); - printf ("%d\n", llist_is_empty (&node3)); - printf ("%d\n", llist_is_empty (&nil)); - } - else if (!strcmp(argv[1], "whiles")) - { - LLIST_AUTO (list); - LLIST_AUTO (node1); - LLIST_AUTO (node2); - LLIST_AUTO (node3); - LLIST_AUTO (node4); - LLIST_AUTO (nil); - - llist_insert_tail (&list, &node2); - llist_insert_tail (&list, &node3); - llist_insert_tail (&list, &node4); - llist_insert_head (&list, &node1); - - LLIST_FOREACH_REV (&list, itr) - { - if(itr == &node1) printf ("node1 "); - else if(itr == &node2) printf ("node2 "); - else if(itr == &node3) printf ("node3 "); - else if(itr == &node4) printf ("node4 "); - else printf ("unknown "); - } - printf (".\n"); - - LLIST_WHILE_HEAD (&list, head) - llist_unlink (head); - - LLIST_FOREACH (&list, itr) - { - if(itr == &node1) printf ("node1 "); - else if(itr == &node2) printf ("node2 "); - else if(itr == &node3) printf ("node3 "); - else if(itr == &node4) printf ("node4 "); - else printf ("unknown "); - } - printf (".\n"); - - llist_insert_tail (&list, &node2); - llist_insert_tail (&list, &node3); - llist_insert_tail (&list, &node4); - llist_insert_head (&list, &node1); - - LLIST_WHILE_TAIL (&list, tail) - llist_unlink (tail); - - LLIST_FOREACH (&list, itr) - { - if(itr == &node1) printf ("node1 "); - else if(itr == &node2) printf ("node2 "); - else if(itr == &node3) printf ("node3 "); - else if(itr == &node4) printf ("node4 "); - else printf ("unknown "); - } - printf (".\n"); - } - else - return 1; - - return 0; + printf ("%d\n", llist_is_empty (&node1)); + printf ("%d\n", llist_is_empty (&node2)); } + +TEST ("nodeinsert") +{ + LLIST_AUTO (list); + LLIST_AUTO (node1); + LLIST_AUTO (node2); + LLIST_AUTO (node3); + + llist_insert_next (&list, &node1); + printf ("%d\n", llist_is_empty (&list)); + printf ("%d\n", llist_is_empty (&node1)); + printf ("%d\n", llist_is_single (&node1)); + llist_insert_next (&node1, &node2); + printf ("%d\n", llist_is_single (&node1)); + llist_insert_prev (&node1, &node3); + printf ("%d\n", llist_next (&list) == &node3); + printf ("%d\n", llist_next (&node3) == &node1); + printf ("%d\n", llist_next (&node1) == &node2); + printf ("%d\n", llist_prev (&list) == &node2); + printf ("%d\n", llist_count (&list)); +} + +TEST ("predicates") +{ + LLIST_AUTO (list); + LLIST_AUTO (node1); + LLIST_AUTO (node2); + LLIST_AUTO (node3); + LLIST_AUTO (node4); + LLIST_AUTO (nil); + + llist_insert_tail (&list, &node2); + llist_insert_tail (&list, &node3); + llist_insert_tail (&list, &node4); + llist_insert_head (&list, &node1); + + printf ("%d\n", llist_is_head (&list, &node1)); + printf ("%d\n", llist_is_tail (&list, &node4)); + printf ("%d\n", llist_is_head (&list, &node4)); + printf ("%d\n", llist_is_tail (&list, &node1)); + printf ("%d\n", llist_is_end (&list, &list)); + printf ("%d\n", llist_is_member (&list, &node3)); + printf ("%d\n", llist_is_member (&list, &nil)); + + printf ("%d\n", llist_is_before_after (&list, &node1, &node3)); + printf ("%d\n", llist_is_before_after (&list, &node3, &node1)); + printf ("%d\n", llist_is_before_after (&list, &node1, &nil)); +} + +TEST ("unlink") +{ + LLIST_AUTO (list); + LLIST_AUTO (node1); + LLIST_AUTO (node2); + LLIST_AUTO (node3); + LLIST_AUTO (node4); + LLIST_AUTO (nil); + + llist_insert_tail (&list, &node2); + llist_insert_tail (&list, &node3); + llist_insert_tail (&list, &node4); + llist_insert_head (&list, &node1); + + LLIST_FOREACH_REV (&list, itr) + { + if(itr == &node1) printf ("node1 "); + else if(itr == &node2) printf ("node2 "); + else if(itr == &node3) printf ("node3 "); + else if(itr == &node4) printf ("node4 "); + else printf ("unknown "); + } + printf (".\n"); + + llist_unlink (&nil); + llist_unlink (&node2); + llist_unlink (&node3); + + LLIST_FOREACH (&list, itr) + { + if(itr == &node1) printf ("node1 "); + else if(itr == &node2) printf ("node2 "); + else if(itr == &node3) printf ("node3 "); + else if(itr == &node4) printf ("node4 "); + else printf ("unknown "); + } + printf (".\n"); + printf ("%d\n", llist_is_empty (&node2)); + printf ("%d\n", llist_is_empty (&node3)); + printf ("%d\n", llist_is_empty (&nil)); +} + +TEST ("whiles") +{ + LLIST_AUTO (list); + LLIST_AUTO (node1); + LLIST_AUTO (node2); + LLIST_AUTO (node3); + LLIST_AUTO (node4); + LLIST_AUTO (nil); + + llist_insert_tail (&list, &node2); + llist_insert_tail (&list, &node3); + llist_insert_tail (&list, &node4); + llist_insert_head (&list, &node1); + + LLIST_FOREACH_REV (&list, itr) + { + if(itr == &node1) printf ("node1 "); + else if(itr == &node2) printf ("node2 "); + else if(itr == &node3) printf ("node3 "); + else if(itr == &node4) printf ("node4 "); + else printf ("unknown "); + } + printf (".\n"); + + LLIST_WHILE_HEAD (&list, head) + llist_unlink (head); + + LLIST_FOREACH (&list, itr) + { + if(itr == &node1) printf ("node1 "); + else if(itr == &node2) printf ("node2 "); + else if(itr == &node3) printf ("node3 "); + else if(itr == &node4) printf ("node4 "); + else printf ("unknown "); + } + printf (".\n"); + + llist_insert_tail (&list, &node2); + llist_insert_tail (&list, &node3); + llist_insert_tail (&list, &node4); + llist_insert_head (&list, &node1); + + LLIST_WHILE_TAIL (&list, tail) + llist_unlink (tail); + + LLIST_FOREACH (&list, itr) + { + if(itr == &node1) printf ("node1 "); + else if(itr == &node2) printf ("node2 "); + else if(itr == &node3) printf ("node3 "); + else if(itr == &node4) printf ("node4 "); + else printf ("unknown "); + } + printf (".\n"); +} + + +TESTS_END diff --git a/tests/locking/test-locking.c b/tests/locking/test-locking.c index 1b54d152e..b37c2c07f 100644 --- a/tests/locking/test-locking.c +++ b/tests/locking/test-locking.c @@ -21,33 +21,21 @@ #include #include - -#include "lib/error.h" - -LUMIERA_ERROR_DEFINE(TEST, "test error"); +#include "tests/test.h" int conditionforgotunlock (); int mutexforgotunlock (); +TESTS_BEGIN -int -main (int argc, char** argv) +TEST ("conditionforgotunlock") { - NOBUG_INIT; - - if (argc == 1) - return 0; - - if (!strcmp(argv[1], "conditionforgotunlock")) - { - return conditionforgotunlock (); - } - if (!strcmp(argv[1], "mutexforgotunlock")) - { - return mutexforgotunlock (); - } - else - return 1; - - return 0; + return conditionforgotunlock (); } + +TEST ("mutexforgotunlock") +{ + return mutexforgotunlock (); +} + +TESTS_END diff --git a/tests/test.h b/tests/test.h new file mode 100644 index 000000000..d25fa0a59 --- /dev/null +++ b/tests/test.h @@ -0,0 +1,52 @@ +/* + test.h - macros for running tests + + Copyright (C) CinelerraCV + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef TEST_H +#define TEST_H + +#include + +#include "lib/error.h" + +LUMIERA_ERROR_DEFINE (TEST, "test error"); + +#define TESTS_BEGIN \ +int \ +main (int argc, char** argv) \ +{ \ + NOBUG_INIT; \ + \ + if (argc == 1) \ + return 1; + +#define TEST(name) \ + else if (!strcmp(argv[1], name)) + + +#define TESTS_END \ + else \ + return 1; \ + \ + return 0; \ +} + + +#endif From 7686a2ec564e02c88ec263bde0b28d93c72816c3 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 20:11:28 +0100 Subject: [PATCH 09/59] Renamed the temporary 'buffer' functions to 'tmpbuf' --- src/lib/safeclib.c | 38 +++++++++++++++++++------------------- src/lib/safeclib.h | 10 +++++----- 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 58116ae90..2a4bc50b4 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -42,7 +42,7 @@ LUMIERA_ERROR_DEFINE (NO_MEMORY, "Out of Memory!"); void* lumiera_malloc (size_t sz) { - void* o = malloc (sz); + void* o = sz? malloc (sz) : NULL; if (!o) LUMIERA_DIE (NO_MEMORY); return o; @@ -104,30 +104,30 @@ lumiera_streq (const char* a, const char* b) * This provides 64 buffers per thread which are recycled with each use. * The idea here is to have fast buffers for temporal data without need for explicit heap management. */ -struct lumiera_buffer_struct +struct lumiera_tmpbuf_struct { void* buffers[64]; size_t sizes[64]; unsigned idx; }; -static pthread_once_t lumiera_buffer_tls_once = PTHREAD_ONCE_INIT; -static pthread_key_t lumiera_buffer_tls_key; +static pthread_once_t lumiera_tmpbuf_tls_once = PTHREAD_ONCE_INIT; +static pthread_key_t lumiera_tmpbuf_tls_key; void -lumiera_buffer_freeall (void); +lumiera_tmpbuf_freeall (void); static void -lumiera_buffer_destroy (void* buf) +lumiera_tmpbuf_destroy (void* buf) { - lumiera_buffer_freeall (); + lumiera_tmpbuf_freeall (); free (buf); } static void -lumiera_buffer_init (void) +lumiera_tmpbuf_init (void) { - pthread_key_create (&lumiera_buffer_tls_key, lumiera_buffer_destroy); + pthread_key_create (&lumiera_tmpbuf_tls_key, lumiera_tmpbuf_destroy); } @@ -136,13 +136,13 @@ lumiera_buffer_init (void) * This function is called automatically, usually one doesnt need to call it. */ void -lumiera_buffer_freeall (void) +lumiera_tmpbuf_freeall (void) { - pthread_once (&lumiera_buffer_tls_once, lumiera_buffer_init); - struct lumiera_buffer_struct* buf = pthread_getspecific (lumiera_buffer_tls_key); + pthread_once (&lumiera_tmpbuf_tls_once, lumiera_tmpbuf_init); + struct lumiera_tmpbuf_struct* buf = pthread_getspecific (lumiera_tmpbuf_tls_key); if (!buf) - pthread_setspecific (lumiera_buffer_tls_key, - buf = lumiera_malloc (sizeof (struct lumiera_buffer_struct))); + pthread_setspecific (lumiera_tmpbuf_tls_key, + buf = lumiera_malloc (sizeof (struct lumiera_tmpbuf_struct))); for (buf->idx = 0; buf->idx < 64; ++buf->idx) { @@ -160,13 +160,13 @@ lumiera_buffer_freeall (void) * @return the buffer */ void* -lumiera_buffer_provide (size_t size) +lumiera_tmpbuf_provide (size_t size) { - pthread_once (&lumiera_buffer_tls_once, lumiera_buffer_init); - struct lumiera_buffer_struct* buf = pthread_getspecific (lumiera_buffer_tls_key); + pthread_once (&lumiera_tmpbuf_tls_once, lumiera_tmpbuf_init); + struct lumiera_tmpbuf_struct* buf = pthread_getspecific (lumiera_tmpbuf_tls_key); if (!buf) - pthread_setspecific (lumiera_buffer_tls_key, - buf = lumiera_malloc (sizeof (struct lumiera_buffer_struct))); + pthread_setspecific (lumiera_tmpbuf_tls_key, + buf = lumiera_malloc (sizeof (struct lumiera_tmpbuf_struct))); buf->idx = (buf->idx + 1) & 0x3f; diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index e4e803acc..90333abd0 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -84,14 +84,14 @@ lumiera_streq (const char* a, const char* b); * This function is called automatically, usually one doesnt need to call it. */ void -lumiera_buffer_freeall (void); +lumiera_tmpbuf_freeall (void); /** - * Query a thread local buffer. - * @param size minimal needed size for the buffer - * @return the buffer + * Query a thread local tmpbuf. + * @param size minimal needed size for the tmpbuf + * @return the tmpbuf */ void* -lumiera_buffer_provide (size_t size); +lumiera_tmpbuf_provide (size_t size); From ecd6ba66b829f6a4b028257c7bfc991363529d03 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 20:11:42 +0100 Subject: [PATCH 10/59] safeclib testsuite --- tests/15safeclib.tests | 28 ++++++++++++ tests/Makefile.am | 5 +++ tests/library/test-safeclib.c | 85 +++++++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 tests/15safeclib.tests create mode 100644 tests/library/test-safeclib.c diff --git a/tests/15safeclib.tests b/tests/15safeclib.tests new file mode 100644 index 000000000..7e4039ead --- /dev/null +++ b/tests/15safeclib.tests @@ -0,0 +1,28 @@ + +TESTING "Safe clib and tools" ./test-safeclib + +TEST "Allocating 0 bytes" allocation0 < + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#include "lib/safeclib.h" +#include "tests/test.h" + +#include +#include + +TESTS_BEGIN + +TEST ("allocation0") +{ + lumiera_malloc (0); + NOTREACHED; +} + +TEST ("allocation1024") +{ + void* data[1024]; + for (int i = 0; i < 1024; ++i) + { + data[i] = lumiera_malloc (1024); + ENSURE (data); + } + for (int i = 0; i < 1024; ++i) + { + free (data[i]); + } +} + +TEST ("allocationtoobig") +{ + struct rlimit rl; + rl.rlim_cur = 100*1024*1024; + rl.rlim_max = 100*1024*1024; + setrlimit (RLIMIT_AS, &rl); + lumiera_malloc (200*1024*1024); + NOTREACHED; +} + +TEST ("streq") +{ + if (!lumiera_streq ("foo", "foo")) + LUMIERA_DIE (TEST); + if (!lumiera_streq (NULL, NULL)) + LUMIERA_DIE (TEST); + if (!!lumiera_streq (NULL, "foo")) + LUMIERA_DIE (TEST); + if (!!lumiera_streq ("foo", NULL)) + LUMIERA_DIE (TEST); + if (!!lumiera_streq ("foo", "bar")) + LUMIERA_DIE (TEST); +} + +TEST ("tmpbuf") +{ + for (int i = 0; i < 256; ++i) + { + char* buf = lumiera_tmpbuf_provide (1024); + + for (int j = 0; j < 1024; ++j) + { + buf[j] = i; + } + } +} + + +TESTS_END From 328127980a43d11c1e1550fd79a764aa7f7a443d Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 26 Mar 2008 22:32:01 +0100 Subject: [PATCH 11/59] a small most-recent-used cache implementation --- src/lib/Makefile.am | 8 +-- src/lib/mrucache.c | 65 ++++++++++++++++++++++++ src/lib/mrucache.h | 119 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 188 insertions(+), 4 deletions(-) create mode 100644 src/lib/mrucache.c create mode 100644 src/lib/mrucache.h diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 844f817a6..2c90304fb 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -30,8 +30,8 @@ liblumi_a_SOURCES = \ $(liblumi_a_srcdir)/condition.c \ $(liblumi_a_srcdir)/references.c \ $(liblumi_a_srcdir)/uuid.c \ - $(liblumi_a_srcdir)/safeclib.c - + $(liblumi_a_srcdir)/safeclib.c \ + $(liblumi_a_srcdir)/mrucache.c noinst_HEADERS += \ @@ -44,5 +44,5 @@ noinst_HEADERS += \ $(liblumi_a_srcdir)/condition.h \ $(liblumi_a_srcdir)/references.h \ $(liblumi_a_srcdir)/uuid.h \ - $(liblumi_a_srcdir)/safeclib.h - + $(liblumi_a_srcdir)/safeclib.h \ + $(liblumi_a_srcdir)/mrucache.h diff --git a/src/lib/mrucache.c b/src/lib/mrucache.c new file mode 100644 index 000000000..15b38d1d1 --- /dev/null +++ b/src/lib/mrucache.c @@ -0,0 +1,65 @@ +/* + mrucache.h - most recent used cache + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "lib/mrucache.h" + +/** + * @file Most recent used cache + * Elements (addressed by a LList node) are either in the cache and thereby subject of aging + * or checked out under control of the user. + */ + + +LumieraMruCache +lumiera_mrucache_init (LumieraMruCache self, lumiera_cache_destructor_fn destructor_cb) +{ + REQUIRE (self); + llist_init (&self->cache_list); + self->cached = 0; + self->destructor_cb = destructor_cb; + return self; +} + +LumieraMruCache +lumiera_mrucache_destroy (LumieraMruCache self) +{ + LLIST_WHILE_TAIL (&self->cache_list, node) + { + if (self->destructor_cb) + free (self->destructor_cb (node)); + else + free (node); + } + self->cached = 0; + return self; +} + + +/** + * destroy and free the nelem oldest elements + */ +void +lumiera_mrucache_age (LumieraMruCache self, size_t nelem) +{ + REQUIRE (self); + while (self->cached && nelem--) + free (lumiera_mrucache_pop (self)); +} diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h new file mode 100644 index 000000000..4802ed6a1 --- /dev/null +++ b/src/lib/mrucache.h @@ -0,0 +1,119 @@ +/* + mrucache.h - most recent used cache + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_CACHE_H +#define LUMIERA_CACHE_H + +#include "lib/llist.h" + +#include + +/** + * @file Most recent used cache + * Elements (addressed by a LList node) are either in the cache and thereby subject of aging + * or checked out under control of the user. + */ + +/** + * Callback function used to destruct/cleanup aged elements. + * shall clean the element sufficiently up to be ready for being freed or reused + * @param node the llist node used to link cache elements (will be empty at call) + * @return pointer to the begin of the element + */ +typedef void* (*lumiera_cache_destructor_fn)(LList node); + +struct lumiera_mrucache_struct +{ + llist cache_list; + size_t cached; + lumiera_cache_destructor_fn destructor_cb; +}; +typedef struct lumiera_mrucache_struct lumiera_mrucache; +typedef lumiera_mrucache* LumieraMruCache; + +/** + * Initialize a mrucache + */ +LumieraMruCache +lumiera_mrucache_init (LumieraMruCache self, lumiera_cache_destructor_fn destructor_cb); + + +/** + * Destroy a mrucache + */ +LumieraMruCache +lumiera_mrucache_destroy (LumieraMruCache self); + + +/** + * add an element to a mrucache + */ +static inline void +lumiera_mrucache_add (LumieraMruCache self, LList node) +{ + REQUIRE (self); + REQUIRE (node && llist_is_empty (node)); + llist_insert_head (&self->cache_list, node); + ++self->cached; +} + + +/** + * Remove an element from a mrucache. + */ +static inline void +lumiera_mrucache_remove (LumieraMruCache self, LList node) +{ + REQUIRE (self); + REQUIRE (node && llist_is_member (&self->cache_list, node)); /* speedup loop warning :P, this check is costly */ + llist_unlink (node); + --self->cached; +} + +/** + * destroy the oldest element and return a pointer to it for reuse. + */ +static inline void* +lumiera_mrucache_pop (LumieraMruCache self) +{ + REQUIRE (self); + if (llist_is_empty (&self->cache_list)) + return NULL; + + LList node = llist_tail (&self->cache_list); + llist_unlink (node); + --self->cached; + + if (self->destructor_cb) + return self->destructor_cb (node); + else + return node; +} + + +/** + * destroy and free the nelem oldest elements + */ +void +lumiera_mrucache_age (LumieraMruCache self, size_t nelem); + + +#endif From eab6b64f6d8f25cc8a096b2473175488d9ffe44e Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 27 Mar 2008 11:58:38 +0100 Subject: [PATCH 12/59] let mrucache age return how much elements it couldn't purge --- src/lib/mrucache.c | 6 ++++-- src/lib/mrucache.h | 6 +++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/src/lib/mrucache.c b/src/lib/mrucache.c index 15b38d1d1..ebe9ef71b 100644 --- a/src/lib/mrucache.c +++ b/src/lib/mrucache.c @@ -56,10 +56,12 @@ lumiera_mrucache_destroy (LumieraMruCache self) /** * destroy and free the nelem oldest elements */ -void -lumiera_mrucache_age (LumieraMruCache self, size_t nelem) +int +lumiera_mrucache_age (LumieraMruCache self, size_t int) { REQUIRE (self); while (self->cached && nelem--) free (lumiera_mrucache_pop (self)); + + return nelem; } diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h index 4802ed6a1..8af7434d9 100644 --- a/src/lib/mrucache.h +++ b/src/lib/mrucache.h @@ -36,7 +36,7 @@ * Callback function used to destruct/cleanup aged elements. * shall clean the element sufficiently up to be ready for being freed or reused * @param node the llist node used to link cache elements (will be empty at call) - * @return pointer to the begin of the element + * @return pointer to the begin of the element. */ typedef void* (*lumiera_cache_destructor_fn)(LList node); @@ -112,8 +112,8 @@ lumiera_mrucache_pop (LumieraMruCache self) /** * destroy and free the nelem oldest elements */ -void -lumiera_mrucache_age (LumieraMruCache self, size_t nelem); +int +lumiera_mrucache_age (LumieraMruCache self, int nelem); #endif From 69ba3b389c829fd832b3827a8ac8469740615233 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 27 Mar 2008 14:30:08 +0100 Subject: [PATCH 13/59] LUMIERA_MUTEX_SECTION(mutex) macro for sections of mutex protected code --- src/lib/mutex.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/lib/mutex.h b/src/lib/mutex.h index ecd99529d..9381c3ca1 100644 --- a/src/lib/mutex.h +++ b/src/lib/mutex.h @@ -28,6 +28,13 @@ * @file Mutual exclusion locking, header. */ +#define LUMIERA_MUTEX_SECTION(mutex) \ +lumiera_mutexacquirer lock_##__LINE__##_; \ +for (lumiera_mutexacquirer_init_mutex (&lock_##__LINE__##_, mutex, LUMIERA_LOCKED); \ + lock_##__LINE__##_.state == LUMIERA_LOCKED; \ + lumiera_mutexacquirer_unlock (&lock_##__LINE__##_)) + + /** * Mutex. * From d32c74361be4d1e6ffd6749f016be8655f375167 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 28 Mar 2008 16:53:35 +0100 Subject: [PATCH 14/59] Fix: typo in mrucache_age() --- src/lib/mrucache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/mrucache.c b/src/lib/mrucache.c index ebe9ef71b..d98976184 100644 --- a/src/lib/mrucache.c +++ b/src/lib/mrucache.c @@ -57,7 +57,7 @@ lumiera_mrucache_destroy (LumieraMruCache self) * destroy and free the nelem oldest elements */ int -lumiera_mrucache_age (LumieraMruCache self, size_t int) +lumiera_mrucache_age (LumieraMruCache self, int nelem) { REQUIRE (self); while (self->cached && nelem--) From 3a8b3feb96d30c744d607209dc0073fbe5b0cc28 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 28 Mar 2008 16:56:36 +0100 Subject: [PATCH 15/59] add string creating functions to tmpbuf * lumiera_tmpbuf_strndup() duplicates a string * lumiera_tmpbuf_sprintf() creates a formatted string --- src/lib/safeclib.c | 33 +++++++++++++++++++++++++++++++++ src/lib/safeclib.h | 23 ++++++++++++++++++++--- 2 files changed, 53 insertions(+), 3 deletions(-) diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 2a4bc50b4..71e347ab7 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -26,6 +26,7 @@ #include #include #include +#include /** * @file Portable and safe wrapers around some clib functions and some tools @@ -179,3 +180,35 @@ lumiera_tmpbuf_provide (size_t size) return buf->buffers[buf->idx]; } + +char* +lumiera_tmpbuf_strndup (const char* src, size_t size) +{ + size_t len = strlen (src); + len = len > size ? size : len; + + char* buf = lumiera_tmpbuf_provide (len + 1); + strncpy (buf, src, len); + buf[len] = '\0'; + + return buf; +} + + +char* +lumiera_tmpbuf_sprintf (size_t size, const char* fmt, ...) +{ + va_list args; + + va_start (args, fmt); + size_t len = vsnprintf (NULL, 0, fmt, args); + va_end (args); + + len = len > size ? size : len; + char* buf = lumiera_tmpbuf_provide (len); + va_start (args, fmt); + vsnprintf (buf, len, fmt, args); + va_end (args); + + return buf; +} diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index 90333abd0..02d931cb7 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -21,9 +21,7 @@ #include "error.h" -//#include -//#include -//#include +#include /** * @file Portable and safe wrapers around some clib functions and some tools @@ -95,3 +93,22 @@ lumiera_tmpbuf_freeall (void); void* lumiera_tmpbuf_provide (size_t size); +/** + * Duplicate string to a tmpbuf. + * @param src string to be duplicated + * @param size maximal length to be copied + * @return temporary buffer containing a copy of the string + */ +char* +lumiera_tmpbuf_strndup (const char* src, size_t size); + +/** + * Construct a string in a tmpbuf. + * @param size maximal length for the string + * @param fmt printf like formatstring + * @param ... parameters + * @return temporary buffer containing the constructed of the string + */ +char* +lumiera_tmpbuf_sprintf (size_t size, const char* fmt, ...); + From b41115d1b2dd4ef5165302c8f5479f0fad938b06 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 28 Mar 2008 16:59:11 +0100 Subject: [PATCH 16/59] Add cuckoo hashing to the library This implementation of cuckoo hashing gives guaranteed O(1) lookup complexity and amortized O(1) insert and remove complexity. Hash tables by default grow and shrink automatically. It is posible to preallocate entries and turn automatic shrinking off, taking out the memory management factors for insert and remove operations. --- src/lib/Makefile.am | 2 + src/lib/cuckoo.c | 497 ++++++++++++++++++++++++++++++++++++++++++++ src/lib/cuckoo.h | 167 +++++++++++++++ 3 files changed, 666 insertions(+) create mode 100644 src/lib/cuckoo.c create mode 100644 src/lib/cuckoo.h diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 2c90304fb..89a418b03 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -31,6 +31,7 @@ liblumi_a_SOURCES = \ $(liblumi_a_srcdir)/references.c \ $(liblumi_a_srcdir)/uuid.c \ $(liblumi_a_srcdir)/safeclib.c \ + $(liblumi_a_srcdir)/cuckoo.c \ $(liblumi_a_srcdir)/mrucache.c @@ -45,4 +46,5 @@ noinst_HEADERS += \ $(liblumi_a_srcdir)/references.h \ $(liblumi_a_srcdir)/uuid.h \ $(liblumi_a_srcdir)/safeclib.h \ + $(liblumi_a_srcdir)/cuckoo.h \ $(liblumi_a_srcdir)/mrucache.h diff --git a/src/lib/cuckoo.c b/src/lib/cuckoo.c new file mode 100644 index 000000000..0d0b4efef --- /dev/null +++ b/src/lib/cuckoo.c @@ -0,0 +1,497 @@ +/* + A cuckoo hash implementation + + Copyright (C) + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "cuckoo.h" + +#include + +enum compact_state + { + COMPACTING_OFF, + COMPACTING_AUTO, + COMPACTED + }; + +struct cuckoo_struct +{ + size_t size; /* t1 = 4*size; t2 = 2*size; t3 = size */ + size_t itemsize; + + cuckoo_hashfunc h1; /* hash function */ + uint32_t r1; /* random, reset for each rehash */ + cuckoo_hashfunc h2; + uint32_t r2; + cuckoo_hashfunc h3; + uint32_t r3; + + cuckoo_cmpfunc cmp; + + void* t1; + void* t2; + void* t3; + + unsigned maxloops; /* sqrt (4 * size) */ + + enum compact_state autocompact; + size_t elements; +}; + + +static inline uint32_t cuckoo_fast_prng () +{ + static uint32_t rnd = 0xbabeface; + return rnd = rnd<<1 ^ ((rnd>>30) & 1) ^ ((rnd>>2) & 1); +} + +Cuckoo +cuckoo_init (Cuckoo self, + cuckoo_hashfunc h1, + cuckoo_hashfunc h2, + cuckoo_hashfunc h3, + cuckoo_cmpfunc cmp, + size_t itemsize, + unsigned startsize) +{ + if (!self) + return NULL; + + self->size = 1<itemsize = itemsize; + self->h1 = h1; + self->r1 = cuckoo_fast_prng (); + self->h2 = h2; + self->r2 = cuckoo_fast_prng (); + self->h3 = h3; + self->r3 = cuckoo_fast_prng (); + + self->cmp = cmp; + + self->t1 = calloc (self->size * 4, itemsize); + self->t2 = calloc (self->size * 2, itemsize); + self->t3 = calloc (self->size, itemsize); + if (!self->t1 || !self->t2 || !self->t3) + { + free (self->t1); + free (self->t2); + free (self->t3); + return NULL; + } + + self->maxloops = 1; + while (self->maxloops * self->maxloops < self->size * 4) + ++self->maxloops; + + self->autocompact = COMPACTING_AUTO; + self->elements = 0; + return self; +} + +Cuckoo +cuckoo_new (cuckoo_hashfunc h1, + cuckoo_hashfunc h2, + cuckoo_hashfunc h3, + cuckoo_cmpfunc cmp, + size_t itemsize, + unsigned startsize) +{ + Cuckoo self = malloc (sizeof (struct cuckoo_struct)); + if (!cuckoo_init (self, h1, h2, h3, cmp, itemsize, startsize)) + { + free (self); + return NULL; + } + return self; +} + +Cuckoo +cuckoo_destroy (Cuckoo self) +{ + if (self) + { + free (self->t1); + free (self->t2); + free (self->t3); + } + return self; +} + + +void +cuckoo_free (Cuckoo self) +{ + free (cuckoo_destroy (self)); +} + + +static inline int +iszero (void* mem, size_t size) +{ + while (size && !*(int*)mem) + { + size -= sizeof (int); + mem += sizeof (int); + } + return !size; +} + +static inline void +xmemcpy (void* dst, void* src, size_t size) +{ + while (size) + { + size -= sizeof (int); + *(int*)(dst + size) = *(int*)(src + size); + } +} + + +static int +cuckoo_insert_internal_ (Cuckoo self, void* item) +{ + void* pos; + char tmp[self->itemsize]; + + for (unsigned n = 0; n < self->maxloops; ++n) + { + /* find nest */ + pos = self->t1 + self->itemsize * (self->h1 (item, self->r1) % (4*self->size)); + /* kick old egg out */ + xmemcpy (tmp, pos, self->itemsize); + /* lay egg */ + xmemcpy (pos, item, self->itemsize); + + if (iszero (tmp, self->itemsize)) + return 1; + + /* find nest */ + pos = self->t2 + self->itemsize * (self->h2 (tmp, self->r2) % (2*self->size)); + /* kick old egg out */ + xmemcpy (item, pos, self->itemsize); + /* lay egg */ + xmemcpy (pos, tmp, self->itemsize); + + if (iszero (item, self->itemsize)) + return 1; + + /* find nest */ + pos = self->t3 + self->itemsize * (self->h3 (item, self->r3) % self->size); + /* kick old egg out */ + xmemcpy (tmp, pos, self->itemsize); + /* lay egg */ + xmemcpy (pos, item, self->itemsize); + + if (iszero (tmp, self->itemsize)) + return 1; + + /* copy tmp to item, which will be reinserted on next interation / after rehashing */ + xmemcpy (item, tmp, self->itemsize); + } + return 0; +} + + +static void +cuckoo_rehash (Cuckoo self) +{ + retry1: + + self->r1 = cuckoo_fast_prng (); + + for (size_t i = 0; i < 4*self->size; ++i) + { + unsigned n; + void* pos = self->t1 + self->itemsize * i; + if (!iszero (pos, self->itemsize)) + { + for (n = 0; n < self->maxloops; ++n) + { + unsigned hash = self->h1 (pos, self->r1) % (4*self->size); + if (hash != i) + { + char t[self->itemsize]; + void* hpos = self->t1 + self->itemsize * hash; + xmemcpy (t, hpos, self->itemsize); + xmemcpy (hpos, pos, self->itemsize); + xmemcpy (pos, t, self->itemsize); + if (iszero (t, self->itemsize)) + break; + } + else + break; + } + if (n == self->maxloops) + goto retry1; + } + } + + retry2: + self->r2 = cuckoo_fast_prng (); + + for (size_t i = 0; i < 2*self->size; ++i) + { + unsigned n; + void* pos = self->t2 + self->itemsize * i; + if (!iszero (pos, self->itemsize)) + { + for (n = 0; n < self->maxloops; ++n) + { + unsigned hash = self->h2 (pos, self->r2) % (2*self->size); + if (hash != i) + { + char t[self->itemsize]; + void* hpos = self->t2 + self->itemsize * hash; + xmemcpy (t, hpos, self->itemsize); + xmemcpy (hpos, pos, self->itemsize); + xmemcpy (pos, t, self->itemsize); + if (iszero (t, self->itemsize)) + break; + } + else + break; + } + if (n == self->maxloops) + goto retry2; + } + } + + retry3: + self->r3 = cuckoo_fast_prng (); + + for (size_t i = 0; i < self->size; ++i) + { + unsigned n; + void* pos = self->t3 + self->itemsize * i; + if (!iszero (pos, self->itemsize)) + { + for (n = 0; n < self->maxloops; ++n) + { + unsigned hash = self->h3 (pos, self->r3) % self->size; + if (hash != i) + { + char t[self->itemsize]; + void* hpos = self->t3 + self->itemsize * hash; + xmemcpy (t, hpos, self->itemsize); + xmemcpy (hpos, pos, self->itemsize); + xmemcpy (pos, t, self->itemsize); + if (iszero (t, self->itemsize)) + break; + } + else + break; + } + if (n == self->maxloops) + goto retry3; + } + } +} + +static int +cuckoo_grow (Cuckoo self) +{ + /* rotate hashfuncs, tables, randoms */ + cuckoo_hashfunc th = self->h3; + self->h3 = self->h2; + self->h2 = self->h1; + self->h1 = th; + + uint32_t tr = self->r3; + self->r3 = self->r2; + self->r2 = self->r1; + self->r1 = tr; + + void* tt = self->t3; + self->t3 = self->t2; + self->t2 = self->t1; + + /* double new base size */ + self->size *= 2; + while (self->maxloops * self->maxloops < self->size * 4) + ++self->maxloops; + + /* alloc new t1 */ + self->t1 = calloc (self->size * 4, self->itemsize); + if (!self->t1) + { + self->t1 = tt; + return 0; + } + + /* reinsert tt */ + size_t ttsize = self->size / 2; + for (size_t i = 0; i < ttsize; ++i) + { + void* pos = tt + i * self->itemsize; + if (!iszero (pos, self->itemsize)) + { + while (!cuckoo_insert_internal_ (self, pos)) + cuckoo_rehash (self); + } + } + free (tt); + + self->autocompact = COMPACTING_AUTO; + return 1; +} + + +int +cuckoo_reserve (Cuckoo self, size_t more) +{ + int ret = 1; + if (more) + while (self->elements+self->maxloops+more >= 6*self->size) + ret = cuckoo_grow (self); + + self->autocompact = COMPACTING_OFF; + return ret; +} + + +int +cuckoo_compact (Cuckoo self) +{ + if (self->autocompact == COMPACTED) + return 1; + + if (self->size > 2 && self->elements < self->size * 3) + { + cuckoo_hashfunc th = self->h1; + self->h1 = self->h2; + self->h2 = self->h3; + self->h3 = th; + + uint32_t tr = self->r1; + self->r1 = self->r2; + self->r2 = self->r3; + self->r3 = tr; + + void* tt = self->t1; + self->t1 = self->t2; + self->t2 = self->t3; + + /* halve base size */ + self->size /= 2; + while (self->maxloops * self->maxloops >= self->size * 4) + --self->maxloops; + + /* alloc new t3 */ + self->t3 = calloc (self->size, self->itemsize); + if (!self->t3) + { + self->t3 = tt; + return 0; + } + + /* reinsert tt */ + size_t ttsize = self->size * 8; + for (size_t i = 0; i < ttsize; ++i) + { + void* pos = tt + i * self->itemsize; + if (!iszero (pos, self->itemsize)) + { + --self->elements; + cuckoo_insert (self, pos); + } + } + free (tt); + self->autocompact = COMPACTED; + } + return 1; +} + + +int +cuckoo_insert (Cuckoo self, void* item) +{ + char tmp[self->itemsize]; + + void* found; + if ((found = cuckoo_find (self, item))) + { + xmemcpy (found, item, self->itemsize); + return 1; + } + + xmemcpy (tmp, item, self->itemsize); + + for (unsigned n = 6; n; --n) /* rehash/grow loop */ + { + if (cuckoo_insert_internal_ (self, tmp)) + { + ++self->elements; + return 1; + } + + if (self->elements > n*self->size) + { + n = 6; + if (!cuckoo_grow (self)) + return 0; + } + else + cuckoo_rehash (self); + } + return 0; +} + + +void* +cuckoo_find (Cuckoo self, void* item) +{ + void* pos; + + pos = self->t1 + self->itemsize * (self->h1 (item, self->r1) % (4*self->size)); + if (!iszero (pos, self->itemsize) && self->cmp (item, pos)) + return pos; + + pos = self->t2 + self->itemsize * (self->h2 (item, self->r2) % (2*self->size)); + if (!iszero (pos, self->itemsize) && self->cmp (item, pos)) + return pos; + + pos = self->t3 + self->itemsize * (self->h3 (item, self->r3) % self->size); + if (!iszero (pos, self->itemsize) && self->cmp (item, pos)) + return pos; + + return NULL; +} + + +void +cuckoo_remove (Cuckoo self, void* item) +{ + if (item) + { + memset (item, 0, self->itemsize); + --self->elements; + + if (self->autocompact == COMPACTING_AUTO && self->size > 2 && self->elements <= self->size*2) + cuckoo_compact (self); + } +} + + +/* +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: +*/ diff --git a/src/lib/cuckoo.h b/src/lib/cuckoo.h new file mode 100644 index 000000000..db1d33da3 --- /dev/null +++ b/src/lib/cuckoo.h @@ -0,0 +1,167 @@ +/* + A cuckoo hash implementation + + Copyright (C) + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#ifndef CUCKOO_H +#define CUCKOO_H + +/** + * @file Cuckoo hashing + * This hashing gives guaranteed O(1) lookup complexity and amortized O(1) insert and remove complexity. + * Hash tables by default grow and shrink automatically. It is posible to preallocate entries and turn + * automatic shrinking off taking out the memory management factors for insert and remove operations. + * This implementation uses 3 Tables which exponential growing sizes. + */ + +#include +#include + +struct cuckoo_struct; +typedef struct cuckoo_struct * Cuckoo; + +/** + * Hash function. + * User supplied universal hash function + * @param item address of the item to be hashed + * @param r pseudo random number, 31 significant bits, regenerated on each rehash + * @return hash value + */ +typedef size_t (*cuckoo_hashfunc)(const void* item, const uint32_t r); + +/** + * Compare function. + * User supplied compare function + * @param item1 address of the first item + * @param item2 address of the second item + * @return 1 when the items are identical, 0 otherwise + */ +typedef int (*cuckoo_cmpfunc)(const void* item1, const void* item2); + +/** + * Initialize a cuckoo hash. + * @param self pointer to a uninitialized cuckoo datastructure + * @param h1 hash function for the first table + * @param h2 hash function for the second table + * @param h3 hash function for the third table + * @param cmp function which compares two keys + * @param startsize initial size of table t3, as 2's exponent + * @return The initialized hashtable or NULL at allocation failure + */ +Cuckoo +cuckoo_init (Cuckoo self, + cuckoo_hashfunc h1, + cuckoo_hashfunc h2, + cuckoo_hashfunc h3, + cuckoo_cmpfunc cmp, + size_t itemsize, + unsigned startsize); + +/** + * Allocate a new cuckoo hash. + * @param h1 hash function for the first table + * @param h2 hash function for the second table + * @param h3 hash function for the third table + * @param cmp function which compares two keys + * @param startsize initial size of table t3, as 2's exponent + * @return The initialized hashtable or NULL at allocation failure + */ +Cuckoo +cuckoo_new (cuckoo_hashfunc h1, + cuckoo_hashfunc h2, + cuckoo_hashfunc h3, + cuckoo_cmpfunc cmp, + size_t itemsize, + unsigned startsize); + +/** + * Destroy a cuckoo hash. + * Frees internal used resources. + * @param self cuckoo hash to destroy + * @return The now uninitialized hashtable + */ +Cuckoo +cuckoo_destroy (Cuckoo self); + +/** + * Deallocate a cuckoo hash. + * @param self handle of the hash table to be freed + */ +void +cuckoo_free (Cuckoo self); + +/** + * Insert an element into a hash. + * amortized O(1) complexity because it may rarely rehash the tables or even grow them. + * see cuckoo_reserve() about how to preallocate entries to prevent the growing costs. + * @param self handle to the hash table + * @param item pointer to a item to be inserted + * @return 1 on successful insert, 0 on allocation failure + * Note: at failure there is one arbitary hash cell lost! + */ +int +cuckoo_insert (Cuckoo self, void* item); + +/** + * Find a value by key in a hash. + * @param self handle to the hash table + * @param item pointer to an item to be looked up + * @return found object or NULL if not present + */ +void* +cuckoo_find (Cuckoo self, void* item); + +/** + * Remove a item from a hash. + * amortized O(1) complexity when it automatically shrink its tables. + * guaranteed O(1) complexity when automatic shrinking is turned off. + * see cuckoo_reserve() about how turn automatic shrinking off. + * @param self handle to the hash table + * @param item pointer to the item to be removed + */ +void +cuckoo_remove (Cuckoo self, void* item); + +/** + * Shrink the hash sizes when possible and turn autocompacting on. + * only useful when autmatic shrinking is turned off. + * see cuckoo_reserve() about how turn shrinking off. + * @param self handle to the hash table + */ +int +cuckoo_compact (Cuckoo self); + +/** + * Prereserve hash entries and turn autocompacting off. + * @param self handle to the hash table + * @param more number of entries to be additionally reserved + * In rare circumstances inserting into a hash which has reserved some entries + * it may still need to rehash (or even rarer) to grow the table. + * While autocompacting is turned off, removing is completely O(1). + */ +int +cuckoo_reserve (Cuckoo self, size_t more); + +#endif +/* +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: +*/ From 380c1d2c47a90abbe69c5695b17ff55530e319e6 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 06:48:49 +0200 Subject: [PATCH 17/59] dropped the pre-gavl framerate sources out of lib --- src/lib/Makefile.am | 2 - src/lib/framerate.h | 107 -------------------------------------------- 2 files changed, 109 deletions(-) delete mode 100644 src/lib/framerate.h diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 89a418b03..0e0c2f22e 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -24,7 +24,6 @@ liblumi_a_CPPFLAGS = -I$(top_srcdir)/src/ liblumi_a_SOURCES = \ $(liblumi_a_srcdir)/plugin.c \ $(liblumi_a_srcdir)/error.c \ - $(liblumi_a_srcdir)/framerate.c \ $(liblumi_a_srcdir)/mutex.c \ $(liblumi_a_srcdir)/rwlock.c \ $(liblumi_a_srcdir)/condition.c \ @@ -38,7 +37,6 @@ liblumi_a_SOURCES = \ noinst_HEADERS += \ $(liblumi_a_srcdir)/plugin.h \ $(liblumi_a_srcdir)/error.h \ - $(liblumi_a_srcdir)/framerate.h \ $(liblumi_a_srcdir)/locking.h \ $(liblumi_a_srcdir)/mutex.h \ $(liblumi_a_srcdir)/rwlock.h \ diff --git a/src/lib/framerate.h b/src/lib/framerate.h deleted file mode 100644 index 581d7fa78..000000000 --- a/src/lib/framerate.h +++ /dev/null @@ -1,107 +0,0 @@ -/* - framerate.h - framerate calculations - - Copyright (C) Lumiera.org - 2008, Christian Thaeter - - This program is free software; you can redistribute it and/or - modify it under the terms of the GNU General Public License as - published by the Free Software Foundation; either version 2 of the - License, or (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program; if not, write to the Free Software - Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. -*/ - -#ifndef LUMIERA_FRAMERATE_H -#define LUMIERA_FRAMERATE_H - -#include - -#include "lib/error.h" - -/** - * @file Framerate calculations, header. - */ - - -/** - * framerates are defined as a rational number - * for example NTSC with 30000/1001fps - */ -struct lumiera_framerate_struct -{ - unsigned n; //numerator - unsigned d; //denominator -}; - -typedef struct lumiera_framerate_struct lumiera_framerate; -typedef lumiera_framerate* LumieraFramerate; - -typedef signed long lumiera_framepos; - -LUMIERA_ERROR_DECLARE(FRAMERATE_ILLEGAL_TIME); -LUMIERA_ERROR_DECLARE(FRAMERATE_ILLEGAL_FRAME); - -#define LUMIERA_FRAMEPOS_ERROR LONG_MIN - -/** - * Get the frame number of a given time at a given frame rate. - * frame indexing starts with 1 - * @param framerate is a pointer to the framerate used, defined as rational number. Must be given. - * @param time is a pointer to a lumiera_time which shall be converted. - * @return frame at the given time or LUMIERA_FRAMEPOS_ERROR on error. - */ -static inline lumiera_framepos -lumiera_framerate_frame_get_time (const LumieraFramerate framerate, LumieraTime time) -{ - REQUIRE (framerate); - if (!time || time->tv_sec == (time_t)-1) - { - lumiera_error_set(LUMIERA_ERROR_FRAMERATE_ILLEGAL_TIME); - return LUMIERA_FRAMEPOS_ERROR; - } - - /* we add a magic microsecond for rounding, because of integer truncation frames may be calculated at most 1us earlier, - the idea is to compensate odd framerates which fall out off microsecond precision. - */ - return ((time->tv_sec * 1000000ULL + time->tv_usec + /*magic*/1) * framerate->n)/(framerate->d * 1000000ULL) + 1; -} - -/** - * Get the start time for a frame. - * frame indexing starts with 1 - * @param framerate is a pointer to the framerate used, defined as rational number. Must be given. - * @param time is a pointer to a lumiera_time which shall take the result. - * @param frame frame number to be converted to time. This frame number must be greater than 0. - * @return the pointer given in time or NULL on error (or when it was given as time). - */ -static inline LumieraTime -lumiera_framerate_time_get_time_frame (const LumieraFramerate framerate, - LumieraTime time, - lumiera_framepos frame) -{ - REQUIRE (framerate); - if (time) - { - if (frame < 1) - { - lumiera_error_set(LUMIERA_ERROR_FRAMERATE_ILLEGAL_FRAME); - return NULL; - } - - unsigned long long usec = ((frame-1) * framerate->d * 1000000ULL - (/*magic*/frame>1?1:0)) / framerate->n; - - time->tv_sec = usec / 1000000; - time->tv_usec = usec % 1000000; - } - return time; -} - -#endif From 23e32d31fca47b82d8cd932451da1c678e7d6de3 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 13 May 2008 21:42:17 +0200 Subject: [PATCH 18/59] don't build the backend things for now --- Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile.am b/Makefile.am index bf7ebec19..69cc34288 100644 --- a/Makefile.am +++ b/Makefile.am @@ -34,7 +34,7 @@ include $(top_srcdir)/admin/Makefile.am # core include $(top_srcdir)/src/lib/Makefile.am -include $(top_srcdir)/src/backend/Makefile.am +#include $(top_srcdir)/src/backend/Makefile.am # plugins #include $(top_srcdir)/src... From a9cf2c719df9cf7a79d523ea52c98ac78716d86d Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 13 May 2008 21:42:51 +0200 Subject: [PATCH 19/59] Fix: uuid includes --- src/lib/uuid.c | 3 ++- src/lib/uuid.h | 3 ++- tests/library/test-uuid.c | 4 +++- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/src/lib/uuid.c b/src/lib/uuid.c index e313df992..d4eee69be 100644 --- a/src/lib/uuid.c +++ b/src/lib/uuid.c @@ -19,13 +19,14 @@ Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#include "uuid.h" +#include "lib/uuid.h" #include #include #include #include #include +#include void lumiera_uuid_set_ptr (lumiera_uuid* uuid, void* ptr) diff --git a/src/lib/uuid.h b/src/lib/uuid.h index a1ab6fa6a..d4cf8ce9d 100644 --- a/src/lib/uuid.h +++ b/src/lib/uuid.h @@ -20,7 +20,8 @@ */ #ifndef LUMIERA_UUID_H #define LUMIERA_UUID_H -#include "lumiera.h" + +#include typedef unsigned char lumiera_uuid[16]; diff --git a/tests/library/test-uuid.c b/tests/library/test-uuid.c index b5b3b6746..4a6050163 100644 --- a/tests/library/test-uuid.c +++ b/tests/library/test-uuid.c @@ -21,10 +21,12 @@ //#include "cinelerra-config.h" //#include -//#include + #include "lib/uuid.h" +#include + //CINELERRA_ERROR_DEFINE(TEST, "test error"); int From 2a1d6c6232d2f84924bfd2d03d9fbc04ee601977 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 13 May 2008 21:43:08 +0200 Subject: [PATCH 20/59] Fix: safeclib-test --- tests/library/test-safeclib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/library/test-safeclib.c b/tests/library/test-safeclib.c index 52a99405f..837e3d5fd 100644 --- a/tests/library/test-safeclib.c +++ b/tests/library/test-safeclib.c @@ -36,7 +36,7 @@ TEST ("allocation1024") for (int i = 0; i < 1024; ++i) { data[i] = lumiera_malloc (1024); - ENSURE (data); + ENSURE (data[i]); } for (int i = 0; i < 1024; ++i) { From 2d11d5186082aea7b6c818d28700ccafa1268834 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 16 May 2008 20:17:43 +0200 Subject: [PATCH 21/59] Checks for NoBug (and other) headers have to follow the particular configurations. If not the generated conftest would bail out NoBug complaining that no build level is selected --- configure.ac | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 78d76c136..72c616a17 100644 --- a/configure.ac +++ b/configure.ac @@ -37,9 +37,6 @@ AC_PROG_LIBTOOL # test for headers # AC_STDC_HEADERS -AC_CHECK_HEADER([nobug.h], AC_DEFINE(HAVE_NOBUG_H)) -AC_CHECK_HEADER([execinfo.h], AC_DEFINE(HAVE_EXECINFO_H)) -# there is a warning in nobug, disabled til fixed AC_CHECK_HEADER([valgrind/valgrind.h], AC_DEFINE(HAVE_VALGRIND_VALGRIND_H)) ACX_PTHREAD ##AM_CONDITIONAL([HAVE_THREADING], [test x"$acx_pthread_ok" = xyes]) @@ -71,6 +68,12 @@ AC_ARG_ENABLE(release, AC_HELP_STRING([--enable-release], [select NoBug RELEASE nobug_level=alpha AC_DEFINE(EBUG_ALPHA) )])]) + +AC_CHECK_HEADER([nobug.h], AC_DEFINE(HAVE_NOBUG_H)) +AC_CHECK_HEADER([execinfo.h], AC_DEFINE(HAVE_EXECINFO_H)) +# there is a warning in nobug, disabled til fixed AC_CHECK_HEADER([valgrind/valgrind.h], AC_DEFINE(HAVE_VALGRIND_VALGRIND_H)) + + AC_MSG_RESULT([NoBug build level: $nobug_level]) # END NoBug From 19fd89d13ddd392e6c244c2a3889f361896317bf Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Sat, 17 May 2008 20:00:05 +0200 Subject: [PATCH 22/59] cleanup and add more configure checks * GNU_SOURCE by default now * improved use of ACX_PTHREAD, replaces $CC and others * all checks for mandatory things instantly exit for now in future this might produce an report at the end * ported checks from scons over here --- configure.ac | 59 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/configure.ac b/configure.ac index 72c616a17..b72b7dfee 100644 --- a/configure.ac +++ b/configure.ac @@ -23,6 +23,8 @@ AC_COPYRIGHT([ Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. ]) +AC_GNU_SOURCE + # # Required programs # @@ -39,6 +41,10 @@ AC_PROG_LIBTOOL AC_STDC_HEADERS ACX_PTHREAD +LIBS="$PTHREAD_LIBS $LIBS" +CFLAGS="$CFLAGS $PTHREAD_CFLAGS" +CC="$PTHREAD_CC" + ##AM_CONDITIONAL([HAVE_THREADING], [test x"$acx_pthread_ok" = xyes]) @@ -68,15 +74,62 @@ AC_ARG_ENABLE(release, AC_HELP_STRING([--enable-release], [select NoBug RELEASE nobug_level=alpha AC_DEFINE(EBUG_ALPHA) )])]) +# END NoBug + +# C headers and libraries +AC_LANG_PUSH([C]) + +AC_CHECK_HEADER([nobug.h], + AC_DEFINE(HAVE_NOBUG_H), + AC_MSG_ERROR([NoBug header missing (http://www.pipapo.org/pipawiki/NoBug)]) + ) +AC_CHECK_LIB([nobugmt], nobug_thread_id_get, + , + AC_MSG_ERROR([NoBug multithreading library missing (http://www.pipapo.org/pipawiki/NoBug)]) + ) -AC_CHECK_HEADER([nobug.h], AC_DEFINE(HAVE_NOBUG_H)) AC_CHECK_HEADER([execinfo.h], AC_DEFINE(HAVE_EXECINFO_H)) # there is a warning in nobug, disabled til fixed AC_CHECK_HEADER([valgrind/valgrind.h], AC_DEFINE(HAVE_VALGRIND_VALGRIND_H)) +AC_LANG_POP([C]) -AC_MSG_RESULT([NoBug build level: $nobug_level]) -# END NoBug +# C++ headers and libraries +AC_LANG_PUSH([C++]) + +AC_CHECK_HEADER([tr1/memory], + , + AC_MSG_ERROR([std::tr1 proposed standard extension for shared_ptr missing]) + ) + +AC_CHECK_HEADER([boost/shared_ptr.hpp], + , + AC_MSG_ERROR([boost::shared_ptr missing (http://www.boost.org/)]) + ) + +AC_CHECK_HEADER([boost/program_options.hpp], + , + AC_MSG_ERROR([boost::program_options missing (http://www.boost.org/)]) + ) +# checking for library left out, name contains compiler version here (libboost_program_options-gcc42-mt-1_34_1.so.1.34.1) any ides how to use that? + +AC_CHECK_HEADER([boost/regex.hpp], + , + AC_MSG_ERROR([boost::regex missing (http://www.boost.org/)]) + ) + +AC_LANG_POP([C++]) + + +# Print a summary +AC_MSG_RESULT([ +Configuration Summary:]) +AC_MSG_RESULT([ NoBug build level: $nobug_level]) +# Add more summary results here + +AC_MSG_RESULT([ +Configuration complete, you can now build Lumiera with 'make' +]) AC_CONFIG_FILES(Makefile) AC_OUTPUT From 2cbfeda4df2ab5de06d8094124a52fccd53f10cc Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 10 Apr 2008 06:20:51 +0200 Subject: [PATCH 23/59] correct return types in condition.h --- src/lib/condition.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/condition.h b/src/lib/condition.h index 066c6b267..ffc00523c 100644 --- a/src/lib/condition.h +++ b/src/lib/condition.h @@ -167,7 +167,7 @@ lumiera_conditionacquirer_wait (LumieraConditionacquirer self) * a conditionacquirer must be unlocked before leaving scope * @param self conditionacquirer associated with a condition variable */ -static inline int +static inline void lumiera_conditionacquirer_unlock (LumieraConditionacquirer self) { REQUIRE (self); @@ -195,7 +195,7 @@ lumiera_conditionacquirer_signal (LumieraConditionacquirer self) * signal all waiting threads * @param self conditionacquirer associated with the condition variable to be signaled */ -static inline int +static inline void lumiera_conditionacquirer_broadcast (LumieraConditionacquirer self) { REQUIRE (self); From 249c6d5eda5c8ccbf4bc9f575c80b048812e6954 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Sat, 17 May 2008 23:33:21 +0200 Subject: [PATCH 24/59] Remove _GNU_SOURCE from certain files GNU_SOURCE is globally activated in configure.ac --- admin/vgsuppression.c | 1 - src/lib/rwlock.c | 2 -- src/lib/safeclib.c | 2 -- 3 files changed, 5 deletions(-) diff --git a/admin/vgsuppression.c b/admin/vgsuppression.c index 0e1f41fca..cad026e52 100644 --- a/admin/vgsuppression.c +++ b/admin/vgsuppression.c @@ -18,7 +18,6 @@ along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#define _GNU_SOURCE /* just place any problematic calls where valgrind whines about in main (with comments please) diff --git a/src/lib/rwlock.c b/src/lib/rwlock.c index 0862aa808..092c7b54b 100644 --- a/src/lib/rwlock.c +++ b/src/lib/rwlock.c @@ -18,8 +18,6 @@ along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#define _GNU_SOURCE - #include #include "lib/rwlock.h" diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 71e347ab7..a0b562643 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -18,8 +18,6 @@ along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#define _GNU_SOURCE - #include "error.h" #include From e8164062a29d8e68951678f58ae8935637471c59 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Sun, 18 May 2008 00:01:51 +0200 Subject: [PATCH 25/59] integrate proc into automake * all files in src/proc are build into libs * TODO: -Werror still missing * TODO: Tests are not yet compiled --- Makefile.am | 1 + src/proc/Makefile.am | 263 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+) create mode 100644 src/proc/Makefile.am diff --git a/Makefile.am b/Makefile.am index 69cc34288..eb67e0a23 100644 --- a/Makefile.am +++ b/Makefile.am @@ -33,6 +33,7 @@ SUBDIRS = include $(top_srcdir)/admin/Makefile.am # core +include $(top_srcdir)/src/proc/Makefile.am include $(top_srcdir)/src/lib/Makefile.am #include $(top_srcdir)/src/backend/Makefile.am diff --git a/src/proc/Makefile.am b/src/proc/Makefile.am new file mode 100644 index 000000000..612ffc2d6 --- /dev/null +++ b/src/proc/Makefile.am @@ -0,0 +1,263 @@ +# Copyright (C) Lumiera.org +# 2008, Christian Thaeter +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# Copyright (C) Lumiera.org +# 2008, Christian Thaeter +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + + +# TODO use -Wextra -Werror + +liblumiproc_a_srcdir = $(top_srcdir)/src/proc +noinst_LIBRARIES += liblumiproc.a + +liblumiproc_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiproc_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiproc_a_SOURCES = \ + $(liblumiproc_a_srcdir)/controllerfacade.cpp \ + $(liblumiproc_a_srcdir)/frame.cpp \ + $(liblumiproc_a_srcdir)/stateproxy.cpp \ + $(liblumiproc_a_srcdir)/asset.cpp \ + $(liblumiproc_a_srcdir)/assetmanager.cpp \ + $(liblumiproc_a_srcdir)/nobugcfg.cpp + + + +liblumiprocasset_a_srcdir = $(top_srcdir)/src/proc/asset +noinst_LIBRARIES += liblumiprocasset.a + +liblumiprocasset_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocasset_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocasset_a_SOURCES = \ + $(liblumiprocasset_a_srcdir)/codec.cpp \ + $(liblumiprocasset_a_srcdir)/compoundmedia.cpp \ + $(liblumiprocasset_a_srcdir)/dataset.cpp \ + $(liblumiprocasset_a_srcdir)/effect.cpp \ + $(liblumiprocasset_a_srcdir)/preview.cpp \ + $(liblumiprocasset_a_srcdir)/unknown.cpp \ + $(liblumiprocasset_a_srcdir)/clip.cpp \ + $(liblumiprocasset_a_srcdir)/meta.cpp \ + $(liblumiprocasset_a_srcdir)/category.cpp \ + $(liblumiprocasset_a_srcdir)/media.cpp \ + $(liblumiprocasset_a_srcdir)/pipe.cpp \ + $(liblumiprocasset_a_srcdir)/proc.cpp \ + $(liblumiprocasset_a_srcdir)/procpatt.cpp \ + $(liblumiprocasset_a_srcdir)/struct.cpp \ + $(liblumiprocasset_a_srcdir)/track.cpp + + + +liblumiprocengine_a_srcdir = $(top_srcdir)/src/proc/engine +noinst_LIBRARIES += liblumiprocengine.a + +liblumiprocengine_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocengine_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocengine_a_SOURCES = \ + $(liblumiprocengine_a_srcdir)/aframe.cpp \ + $(liblumiprocengine_a_srcdir)/arender.cpp \ + $(liblumiprocengine_a_srcdir)/codecadapter.cpp \ + $(liblumiprocengine_a_srcdir)/exitnode.cpp \ + $(liblumiprocengine_a_srcdir)/glbuf.cpp \ + $(liblumiprocengine_a_srcdir)/glpipe.cpp \ + $(liblumiprocengine_a_srcdir)/glrender.cpp \ + $(liblumiprocengine_a_srcdir)/hub.cpp \ + $(liblumiprocengine_a_srcdir)/link.cpp \ + $(liblumiprocengine_a_srcdir)/mask.cpp \ + $(liblumiprocengine_a_srcdir)/pluginadapter.cpp \ + $(liblumiprocengine_a_srcdir)/processor.cpp \ + $(liblumiprocengine_a_srcdir)/procnode.cpp \ + $(liblumiprocengine_a_srcdir)/projector.cpp \ + $(liblumiprocengine_a_srcdir)/renderengine.cpp \ + $(liblumiprocengine_a_srcdir)/source.cpp \ + $(liblumiprocengine_a_srcdir)/trafo.cpp \ + $(liblumiprocengine_a_srcdir)/vframe.cpp \ + $(liblumiprocengine_a_srcdir)/vrender.cpp + + + +liblumiprocmobject_a_srcdir = $(top_srcdir)/src/proc/mobject +noinst_LIBRARIES += liblumiprocmobject.a + +liblumiprocmobject_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocmobject_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocmobject_a_SOURCES = \ + $(liblumiprocmobject_a_srcdir)/builderfacade.cpp \ + $(liblumiprocmobject_a_srcdir)/interpolator.cpp \ + $(liblumiprocmobject_a_srcdir)/mobject.cpp \ + $(liblumiprocmobject_a_srcdir)/parameter.cpp \ + $(liblumiprocmobject_a_srcdir)/paramprovider.cpp \ + $(liblumiprocmobject_a_srcdir)/placement.cpp + + + +liblumiprocmobjectbuilder_a_srcdir = $(top_srcdir)/src/proc/mobject/builder +noinst_LIBRARIES += liblumiprocmobjectbuilder.a + +liblumiprocmobjectbuilder_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocmobjectbuilder_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocmobjectbuilder_a_SOURCES = \ + $(liblumiprocmobjectbuilder_a_srcdir)/assembler.cpp \ + $(liblumiprocmobjectbuilder_a_srcdir)/conmanager.cpp \ + $(liblumiprocmobjectbuilder_a_srcdir)/nodecreatertool.cpp \ + $(liblumiprocmobjectbuilder_a_srcdir)/segmentationtool.cpp \ + $(liblumiprocmobjectbuilder_a_srcdir)/toolfactory.cpp + + +liblumiprocmobjectcontroller_a_srcdir = $(top_srcdir)/src/proc/mobject/controller +noinst_LIBRARIES += liblumiprocmobjectcontroller.a + +liblumiprocmobjectcontroller_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocmobjectcontroller_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocmobjectcontroller_a_SOURCES = \ + $(liblumiprocmobjectcontroller_a_srcdir)/pathmanager.cpp \ + $(liblumiprocmobjectcontroller_a_srcdir)/renderstate.cpp + + +liblumiprocmobjectsession_a_srcdir = $(top_srcdir)/src/proc/mobject/session +noinst_LIBRARIES += liblumiprocmobjectsession.a + +liblumiprocmobjectsession_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumiprocmobjectsession_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumiprocmobjectsession_a_SOURCES = \ + $(liblumiprocmobjectsession_a_srcdir)/abstractmo.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/allocation.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/auto.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/compoundclip.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/effect.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/fixedlocation.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/label.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/meta.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/relativelocation.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/segment.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/simpleclip.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/plug.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/sessionimpl.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/sessmanagerimpl.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/edl.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/session.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/track.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/clip.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/constraint.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/defsmanager.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/fixture.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/locatingpin.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/mobjectfactory.cpp \ + $(liblumiprocmobjectsession_a_srcdir)/wish.cpp + + + + +noinst_HEADERS += \ + $(liblumiproc_a_srcdir)/asset/category.hpp \ + $(liblumiproc_a_srcdir)/asset/clip.hpp \ + $(liblumiproc_a_srcdir)/asset/codec.hpp \ + $(liblumiproc_a_srcdir)/asset/compoundmedia.hpp \ + $(liblumiproc_a_srcdir)/asset/dataset.hpp \ + $(liblumiproc_a_srcdir)/asset/effect.hpp \ + $(liblumiproc_a_srcdir)/asset/preview.hpp \ + $(liblumiproc_a_srcdir)/asset/db.hpp \ + $(liblumiproc_a_srcdir)/asset/buildinstruct.hpp \ + $(liblumiproc_a_srcdir)/asset/media.hpp \ + $(liblumiproc_a_srcdir)/asset/meta.hpp \ + $(liblumiproc_a_srcdir)/asset/pipe.hpp \ + $(liblumiproc_a_srcdir)/asset/proc.hpp \ + $(liblumiproc_a_srcdir)/asset/procpatt.hpp \ + $(liblumiproc_a_srcdir)/asset/struct.hpp \ + $(liblumiproc_a_srcdir)/asset/structfactoryimpl.hpp \ + $(liblumiproc_a_srcdir)/asset/track.hpp \ + $(liblumiproc_a_srcdir)/asset/unknown.hpp \ + $(liblumiproc_a_srcdir)/controllerfacade.hpp \ + $(liblumiproc_a_srcdir)/engine/aframe.hpp \ + $(liblumiproc_a_srcdir)/engine/arender.hpp \ + $(liblumiproc_a_srcdir)/engine/codecadapter.hpp \ + $(liblumiproc_a_srcdir)/engine/exitnode.hpp \ + $(liblumiproc_a_srcdir)/engine/glbuf.hpp \ + $(liblumiproc_a_srcdir)/engine/glpipe.hpp \ + $(liblumiproc_a_srcdir)/engine/glrender.hpp \ + $(liblumiproc_a_srcdir)/engine/hub.hpp \ + $(liblumiproc_a_srcdir)/engine/link.hpp \ + $(liblumiproc_a_srcdir)/engine/mask.hpp \ + $(liblumiproc_a_srcdir)/engine/pluginadapter.hpp \ + $(liblumiproc_a_srcdir)/engine/procnode.hpp \ + $(liblumiproc_a_srcdir)/engine/projector.hpp \ + $(liblumiproc_a_srcdir)/engine/renderengine.hpp \ + $(liblumiproc_a_srcdir)/engine/source.hpp \ + $(liblumiproc_a_srcdir)/engine/trafo.hpp \ + $(liblumiproc_a_srcdir)/engine/vframe.hpp \ + $(liblumiproc_a_srcdir)/engine/vrender.hpp \ + $(liblumiproc_a_srcdir)/engine/processor.hpp \ + $(liblumiproc_a_srcdir)/frame.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/assembler.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/buildertool.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/conmanager.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/nodecreatertool.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/segmentationtool.hpp \ + $(liblumiproc_a_srcdir)/mobject/builder/toolfactory.hpp \ + $(liblumiproc_a_srcdir)/mobject/builderfacade.hpp \ + $(liblumiproc_a_srcdir)/mobject/controller/pathmanager.hpp \ + $(liblumiproc_a_srcdir)/mobject/controller/renderstate.hpp \ + $(liblumiproc_a_srcdir)/mobject/interpolator.hpp \ + $(liblumiproc_a_srcdir)/mobject/parameter.hpp \ + $(liblumiproc_a_srcdir)/mobject/paramprovider.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/abstractmo.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/allocation.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/auto.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/compoundclip.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/effect.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/fixedlocation.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/label.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/meta.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/relativelocation.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/simpleclip.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/mobjectfactory.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/segment.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/track.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/wish.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/clip.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/constraint.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/defsmanager.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/edl.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/fixture.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/locatingpin.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/plug.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/sessionimpl.hpp \ + $(liblumiproc_a_srcdir)/mobject/session/defsregistry.hpp \ + $(liblumiproc_a_srcdir)/mobject/explicitplacement.hpp \ + $(liblumiproc_a_srcdir)/mobject/mobject.hpp \ + $(liblumiproc_a_srcdir)/mobject/placement.hpp \ + $(liblumiproc_a_srcdir)/mobject/session.hpp \ + $(liblumiproc_a_srcdir)/stateproxy.hpp \ + $(liblumiproc_a_srcdir)/asset.hpp \ + $(liblumiproc_a_srcdir)/assetmanager.hpp \ + $(liblumiproc_a_srcdir)/lumiera.hpp \ + $(liblumiproc_a_srcdir)/nobugcfg.hpp + From b6bd181ca8bd05d736ed215f2945562c3a39dd14 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 01:48:02 +0200 Subject: [PATCH 26/59] correct pasting glitch and copyright header date --- src/proc/Makefile.am | 12 ------------ tests/Makefile.am | 2 +- 2 files changed, 1 insertion(+), 13 deletions(-) diff --git a/src/proc/Makefile.am b/src/proc/Makefile.am index 612ffc2d6..603f4d825 100644 --- a/src/proc/Makefile.am +++ b/src/proc/Makefile.am @@ -10,18 +10,6 @@ # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. -# Copyright (C) Lumiera.org -# 2008, Christian Thaeter -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License as -# published by the Free Software Foundation; either version 2 of the -# License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software diff --git a/tests/Makefile.am b/tests/Makefile.am index a6ccae010..1dce33db8 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -1,5 +1,5 @@ # Copyright (C) Lumiera.org -# 2007, Christian Thaeter +# 2007-2008, Christian Thaeter # Hermann Vosseler # # This program is free software; you can redistribute it and/or From 3b6713b23c2612f8ecfe19ff647e8395ced23ee5 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 01:50:53 +0200 Subject: [PATCH 27/59] build common and (re-)enable backend build (C++ part) - not (yet) merged backend C parts are commented out for now --- Makefile.am | 3 +- src/backend/Makefile.am | 25 +++++++++------- src/common/Makefile.am | 65 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 11 deletions(-) create mode 100644 src/common/Makefile.am diff --git a/Makefile.am b/Makefile.am index eb67e0a23..2ea3aba54 100644 --- a/Makefile.am +++ b/Makefile.am @@ -34,8 +34,9 @@ include $(top_srcdir)/admin/Makefile.am # core include $(top_srcdir)/src/proc/Makefile.am +include $(top_srcdir)/src/common/Makefile.am include $(top_srcdir)/src/lib/Makefile.am -#include $(top_srcdir)/src/backend/Makefile.am +include $(top_srcdir)/src/backend/Makefile.am # plugins #include $(top_srcdir)/src... diff --git a/src/backend/Makefile.am b/src/backend/Makefile.am index 5f693f8bf..3d70a09d5 100644 --- a/src/backend/Makefile.am +++ b/src/backend/Makefile.am @@ -22,16 +22,21 @@ liblumibackend_a_CFLAGS = $(CFLAGS) -std=gnu99 -Wall -Werror liblumibackend_a_CPPFLAGS = -I$(top_srcdir)/src/ liblumibackend_a_SOURCES = \ - $(liblumibackend_a_srcdir)/backend.c \ - $(liblumibackend_a_srcdir)/file.c \ - $(liblumibackend_a_srcdir)/filehandle.c \ - $(liblumibackend_a_srcdir)/filedescriptor.c \ - $(liblumibackend_a_srcdir)/filehandlecache.c + $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp + +# $(liblumibackend_a_srcdir)/backend.c \ +# $(liblumibackend_a_srcdir)/file.c \ +# $(liblumibackend_a_srcdir)/filehandle.c \ +# $(liblumibackend_a_srcdir)/filedescriptor.c \ +# $(liblumibackend_a_srcdir)/filehandlecache.c + noinst_HEADERS += \ - $(liblumibackend_a_srcdir)/backend.h \ - $(liblumibackend_a_srcdir)/file.h \ - $(liblumibackend_a_srcdir)/filehandle.h \ - $(liblumibackend_a_srcdir)/filedescriptor.h \ - $(liblumibackend_a_srcdir)/filehandlecache.h + $(liblumibackend_a_srcdir)mediaaccessfacade.cpp + +# $(liblumibackend_a_srcdir)/backend.h \ +# $(liblumibackend_a_srcdir)/file.h \ +# $(liblumibackend_a_srcdir)/filehandle.h \ +# $(liblumibackend_a_srcdir)/filedescriptor.h \ +# $(liblumibackend_a_srcdir)/filehandlecache.h diff --git a/src/common/Makefile.am b/src/common/Makefile.am new file mode 100644 index 000000000..06082da31 --- /dev/null +++ b/src/common/Makefile.am @@ -0,0 +1,65 @@ +# Copyright (C) Lumiera.org +# 2008, Christian Thaeter +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + + +# TODO use -Wextra -Werror + +liblumicommon_a_srcdir = $(top_srcdir)/src/common +noinst_LIBRARIES += liblumicommon.a + +liblumicommon_a_CXXFLAGS = $(CXXFLAGS) -Wall +liblumicommon_a_CPPFLAGS = -I$(top_srcdir)/src/ + +liblumicommon_a_SOURCES = \ + $(liblumicommon_a_srcdir)/time.cpp \ + $(liblumicommon_a_srcdir)/util.cpp \ + $(liblumicommon_a_srcdir)/visitor.cpp \ + $(liblumicommon_a_srcdir)/cmdline.cpp \ + $(liblumicommon_a_srcdir)/configrules.cpp \ + $(liblumicommon_a_srcdir)/error.cpp \ + $(liblumicommon_a_srcdir)/query.cpp \ + $(liblumicommon_a_srcdir)/query/mockconfigrules.cpp \ + $(liblumicommon_a_srcdir)/test/suite.cpp \ + $(liblumicommon_a_srcdir)/test/testoption.cpp + + +noinst_HEADERS += \ + $(liblumicommon_a_srcdir)/cmdline.hpp \ + $(liblumicommon_a_srcdir)/factory.hpp \ + $(liblumicommon_a_srcdir)/singleton.hpp \ + $(liblumicommon_a_srcdir)/singletonpolicies.hpp \ + $(liblumicommon_a_srcdir)/singletonpreconfigure.hpp \ + $(liblumicommon_a_srcdir)/time.hpp \ + $(liblumicommon_a_srcdir)/typelist.hpp \ + $(liblumicommon_a_srcdir)/visitor.hpp \ + $(liblumicommon_a_srcdir)/visitordispatcher.hpp \ + $(liblumicommon_a_srcdir)/visitorpolicies.hpp \ + $(liblumicommon_a_srcdir)/configrules.hpp \ + $(liblumicommon_a_srcdir)/error.hpp \ + $(liblumicommon_a_srcdir)/multithread.hpp \ + $(liblumicommon_a_srcdir)/p.hpp \ + $(liblumicommon_a_srcdir)/query.hpp \ + $(liblumicommon_a_srcdir)/query/mockconfigrules.hpp \ + $(liblumicommon_a_srcdir)/singletonfactory.hpp \ + $(liblumicommon_a_srcdir)/singletonsubclass.hpp \ + $(liblumicommon_a_srcdir)/typelistutil.hpp \ + $(liblumicommon_a_srcdir)/util.hpp \ + $(liblumicommon_a_srcdir)/test/mockinjector.hpp \ + $(liblumicommon_a_srcdir)/test/suite.hpp \ + $(liblumicommon_a_srcdir)/test/testoption.hpp \ + $(liblumicommon_a_srcdir)/test/run.hpp + From 5ae909e2a0f7b9e445a95329d7cc15b09cfef912 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 01:51:34 +0200 Subject: [PATCH 28/59] add C++ parts to lib build --- src/lib/Makefile.am | 50 +++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 0e0c2f22e..ffdb9a176 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -21,28 +21,30 @@ noinst_LIBRARIES += liblumi.a liblumi_a_CFLAGS = $(CFLAGS) -std=gnu99 -Wall -Werror liblumi_a_CPPFLAGS = -I$(top_srcdir)/src/ -liblumi_a_SOURCES = \ - $(liblumi_a_srcdir)/plugin.c \ - $(liblumi_a_srcdir)/error.c \ - $(liblumi_a_srcdir)/mutex.c \ - $(liblumi_a_srcdir)/rwlock.c \ - $(liblumi_a_srcdir)/condition.c \ - $(liblumi_a_srcdir)/references.c \ - $(liblumi_a_srcdir)/uuid.c \ - $(liblumi_a_srcdir)/safeclib.c \ - $(liblumi_a_srcdir)/cuckoo.c \ - $(liblumi_a_srcdir)/mrucache.c +liblumi_a_SOURCES = \ + $(liblumi_a_srcdir)/plugin.c \ + $(liblumi_a_srcdir)/error.c \ + $(liblumi_a_srcdir)/mutex.c \ + $(liblumi_a_srcdir)/rwlock.c \ + $(liblumi_a_srcdir)/condition.c \ + $(liblumi_a_srcdir)/references.c \ + $(liblumi_a_srcdir)/uuid.c \ + $(liblumi_a_srcdir)/safeclib.c \ + $(liblumi_a_srcdir)/cuckoo.c \ + $(liblumi_a_srcdir)/mrucache.c \ + $(liblumi_a_srcdir)/appconfig.cpp - -noinst_HEADERS += \ - $(liblumi_a_srcdir)/plugin.h \ - $(liblumi_a_srcdir)/error.h \ - $(liblumi_a_srcdir)/locking.h \ - $(liblumi_a_srcdir)/mutex.h \ - $(liblumi_a_srcdir)/rwlock.h \ - $(liblumi_a_srcdir)/condition.h \ - $(liblumi_a_srcdir)/references.h \ - $(liblumi_a_srcdir)/uuid.h \ - $(liblumi_a_srcdir)/safeclib.h \ - $(liblumi_a_srcdir)/cuckoo.h \ - $(liblumi_a_srcdir)/mrucache.h +noinst_HEADERS += \ + $(liblumi_a_srcdir)/plugin.h \ + $(liblumi_a_srcdir)/error.h \ + $(liblumi_a_srcdir)/locking.h \ + $(liblumi_a_srcdir)/mutex.h \ + $(liblumi_a_srcdir)/rwlock.h \ + $(liblumi_a_srcdir)/condition.h \ + $(liblumi_a_srcdir)/references.h \ + $(liblumi_a_srcdir)/uuid.h \ + $(liblumi_a_srcdir)/safeclib.h \ + $(liblumi_a_srcdir)/cuckoo.h \ + $(liblumi_a_srcdir)/mrucache.h \ + $(liblumi_a_srcdir)/appconfig.hpp \ + $(liblumi_a_srcdir)/lifecycleregistry.hpp From 8ba3df3b31f34580e199bb306f0726a21e22f69c Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 01:54:40 +0200 Subject: [PATCH 29/59] missing slash --- src/backend/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/Makefile.am b/src/backend/Makefile.am index 3d70a09d5..eb5853d5f 100644 --- a/src/backend/Makefile.am +++ b/src/backend/Makefile.am @@ -32,7 +32,7 @@ liblumibackend_a_SOURCES = \ noinst_HEADERS += \ - $(liblumibackend_a_srcdir)mediaaccessfacade.cpp + $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp # $(liblumibackend_a_srcdir)/backend.h \ # $(liblumibackend_a_srcdir)/file.h \ From d6d6a1040499e59274fe027fdf5c536a3562395f Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 01:55:05 +0200 Subject: [PATCH 30/59] add components test build --- Makefile.am | 1 + tests/components/Makefile.am | 72 ++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 tests/components/Makefile.am diff --git a/Makefile.am b/Makefile.am index 2ea3aba54..79d054de2 100644 --- a/Makefile.am +++ b/Makefile.am @@ -42,6 +42,7 @@ include $(top_srcdir)/src/backend/Makefile.am #include $(top_srcdir)/src... # tests +include $(top_srcdir)/tests/components/Makefile.am include $(top_srcdir)/tests/Makefile.am include $(top_srcdir)/tests/plugin/Makefile.am diff --git a/tests/components/Makefile.am b/tests/components/Makefile.am new file mode 100644 index 000000000..5e8d1e070 --- /dev/null +++ b/tests/components/Makefile.am @@ -0,0 +1,72 @@ +# Copyright (C) Lumiera.org +# 2007, Christian Thaeter +# Hermann Vosseler +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + +testcomponents_srcdir = $(top_srcdir)/tests/components + +check_PROGRAMS += test-components +test_components_CPPFLAGS = $(AM_CPPFLAGS) -Wall -I$(top_srcdir)/src -I$(testcomponents_srcdir) +test_components_LDADD = \ + liblumiproc.a \ + liblumiprocengine.a \ + liblumiprocmobjectbuilder.a \ + liblumiprocmobjectsession.a \ + liblumiprocasset.a \ + liblumiprocmobject.a \ + liblumiprocmobjectcontroller.a \ + liblumi.a \ + liblumicommon.a \ + liblumibackend.a \ + -lnobugmt -lpthread -ldl -lboost_program_options-mt -lboost_regex-mt + +test_components_SOURCES = \ + $(testcomponents_srcdir)/backend/mediaaccessmock.cpp \ + $(testcomponents_srcdir)/backend/mediaaccessmocktest.cpp \ + $(testcomponents_srcdir)/proc/asset/assetcategorytest.cpp \ + $(testcomponents_srcdir)/proc/asset/compoundmediatest.cpp \ + $(testcomponents_srcdir)/proc/asset/deleteassettest.cpp \ + $(testcomponents_srcdir)/proc/asset/identityofassetstest.cpp \ + $(testcomponents_srcdir)/proc/asset/mediastructurequerytest.cpp \ + $(testcomponents_srcdir)/proc/asset/basicpipetest.cpp \ + $(testcomponents_srcdir)/proc/asset/createassettest.cpp \ + $(testcomponents_srcdir)/proc/asset/dependantassetstest.cpp \ + $(testcomponents_srcdir)/proc/asset/makecliptest.cpp \ + $(testcomponents_srcdir)/proc/asset/orderingofassetstest.cpp \ + $(testcomponents_srcdir)/proc/asset/testasset.cpp \ + $(testcomponents_srcdir)/proc/engine/sourcenodetest.cpp \ + $(testcomponents_srcdir)/proc/mobject/builder/buildertooltest.cpp \ + $(testcomponents_srcdir)/proc/mobject/builder/buildsegmenttest.cpp \ + $(testcomponents_srcdir)/proc/mobject/controller/rendersegmenttest.cpp \ + $(testcomponents_srcdir)/proc/mobject/placementbasictest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/addcliptest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/deletecliptest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/rebuildfixturetest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/sessionmanagertest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/testclip.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/defsmanagerimpltest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/defsmanagertest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/defsregistryimpltest.cpp \ + $(testcomponents_srcdir)/proc/mobject/session/sessionstructuretest.cpp \ + $(testcomponents_srcdir)/mainsuite.cpp + +noinst_HEADERS += \ + $(testcomponents_srcdir)/backend/mediaaccessmock.hpp \ + $(testcomponents_srcdir)/proc/asset/testclipasset.hpp \ + $(testcomponents_srcdir)/proc/asset/assetdiagnostics.hpp \ + $(testcomponents_srcdir)/proc/asset/testasset.hpp \ + $(testcomponents_srcdir)/proc/mobject/session/testclip.hpp \ + $(testcomponents_srcdir)/proc/mobject/session/testsession1.hpp From e1b8849d0c23896d5bd18fceeb1fda1c0057e643 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 13:30:20 +0200 Subject: [PATCH 31/59] build 'common' tests --- Makefile.am | 1 + tests/common/Makefile.am | 61 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 tests/common/Makefile.am diff --git a/Makefile.am b/Makefile.am index 79d054de2..e6f29bf97 100644 --- a/Makefile.am +++ b/Makefile.am @@ -42,6 +42,7 @@ include $(top_srcdir)/src/backend/Makefile.am #include $(top_srcdir)/src... # tests +include $(top_srcdir)/tests/common/Makefile.am include $(top_srcdir)/tests/components/Makefile.am include $(top_srcdir)/tests/Makefile.am include $(top_srcdir)/tests/plugin/Makefile.am diff --git a/tests/common/Makefile.am b/tests/common/Makefile.am new file mode 100644 index 000000000..cc027e477 --- /dev/null +++ b/tests/common/Makefile.am @@ -0,0 +1,61 @@ +# Copyright (C) Lumiera.org +# 2008, Christian Thaeter +# Hermann Vosseler +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation; either version 2 of the +# License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + +testcommon_srcdir = $(top_srcdir)/tests/common + +check_PROGRAMS += test-common +test_common_CPPFLAGS = $(AM_CPPFLAGS) -Wall -I$(top_srcdir)/src -I$(testcommon_srcdir) +test_common_LDADD = \ + liblumiproc.a \ + liblumiprocengine.a \ + liblumiprocmobjectbuilder.a \ + liblumiprocmobjectsession.a \ + liblumiprocasset.a \ + liblumiprocmobject.a \ + liblumiprocmobjectcontroller.a \ + liblumi.a \ + liblumicommon.a \ + liblumibackend.a \ + -lnobugmt -lpthread -ldl -lboost_program_options-mt -lboost_regex-mt + +test_common_SOURCES = \ + $(testcommon_srcdir)/appconfigtest.cpp \ + $(testcommon_srcdir)/customsharedptrtest.cpp \ + $(testcommon_srcdir)/exceptionerrortest.cpp \ + $(testcommon_srcdir)/factoryspecialtest.cpp \ + $(testcommon_srcdir)/factorytest.cpp \ + $(testcommon_srcdir)/helloworldtest.cpp \ + $(testcommon_srcdir)/lifecycletest.cpp \ + $(testcommon_srcdir)/mainsuite.cpp \ + $(testcommon_srcdir)/query/queryutilstest.cpp \ + $(testcommon_srcdir)/removefromsettest.cpp \ + $(testcommon_srcdir)/sanitizedidentifiertest.cpp \ + $(testcommon_srcdir)/singletonsubclasstest.cpp \ + $(testcommon_srcdir)/singletontest.cpp \ + $(testcommon_srcdir)/singletontestmocktest.cpp \ + $(testcommon_srcdir)/test/cmdlinewrappertest.cpp \ + $(testcommon_srcdir)/test/testoptiontest.cpp \ + $(testcommon_srcdir)/typelisttest.cpp \ + $(testcommon_srcdir)/typelistutiltest.cpp \ + $(testcommon_srcdir)/visitingtoolconcept.cpp \ + $(testcommon_srcdir)/visitingtoolextendedtest.cpp \ + $(testcommon_srcdir)/visitingtooltest.cpp + +noinst_HEADERS += \ + $(testcommon_srcdir)/query/querydiagnostics.hpp \ + $(testcommon_srcdir)/testtargetobj.hpp From bc436d1d6a0cf75c66050f317020b366dba69b44 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 13:30:55 +0200 Subject: [PATCH 32/59] fix typo in component test --- tests/50components.tests | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/50components.tests b/tests/50components.tests index 2bac409cb..18f04ee4b 100644 --- a/tests/50components.tests +++ b/tests/50components.tests @@ -62,7 +62,7 @@ out: caught error::Logic: LUMIERA_ERROR_FATAL:floundered (test-2). out: caught error::Invalid: LUMIERA_ERROR_INVALID:invalid input or parameters (test-3). out: caught lumiera::Error: LUMIERA_ERROR_EXTERNAL:failure in external service (test-4). out: caught std::runtime_error: test-5 -out: caught std::exception: St9exception +out: caught std::exception: std::exception out: intermediate handler caught: LUMIERA_ERROR_EXTERNAL:failure in external service (test-7).....will rethrow as error::State out: caught lumiera::Error: LUMIERA_ERROR_STATE:unforseen state -- caused by: LUMIERA_ERROR_EXTERNAL:failure in external service (test-7). out: intermediate handler caught: LUMIERA_ERROR_EXTERNAL:failure in external service (test-8).....will rethrow as error::State From 5659219e55c3348b7ea1aab8e7c3b823c172aaf4 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 13:31:18 +0200 Subject: [PATCH 33/59] correct date in copyright header --- tests/components/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/components/Makefile.am b/tests/components/Makefile.am index 5e8d1e070..088dbe668 100644 --- a/tests/components/Makefile.am +++ b/tests/components/Makefile.am @@ -1,5 +1,5 @@ # Copyright (C) Lumiera.org -# 2007, Christian Thaeter +# 2008, Christian Thaeter # Hermann Vosseler # # This program is free software; you can redistribute it and/or From b4183ed44456c830e61a848e09b3a34fd6c70b9f Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 1 Apr 2008 07:55:20 +0200 Subject: [PATCH 34/59] safeclib improvements * add lumiera_calloc() * improve cleanup functions for tmpbuf --- src/lib/safeclib.c | 40 ++++++++++++++++++++++++++-------------- src/lib/safeclib.h | 1 - 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index a0b562643..934c3ff5b 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -1,7 +1,7 @@ /* safe_clib.c - Portable and safe wrapers around some clib functions and some tools - Copyright (C) CinelerraCV + Copyright (C) Lumiera.org 2008, Christian Thaeter This program is free software; you can redistribute it and/or @@ -39,9 +39,26 @@ LUMIERA_ERROR_DEFINE (NO_MEMORY, "Out of Memory!"); * @return pointer to the allocated memory */ void* -lumiera_malloc (size_t sz) +lumiera_malloc (size_t size) { - void* o = sz? malloc (sz) : NULL; + void* o = size ? malloc (size) : NULL; + if (!o) + LUMIERA_DIE (NO_MEMORY); + return o; +} + + +/** + * Allocate cleared memory for an array. + * always succeeds or dies + * @param n number of elements + * @param size memory to be allocated + * @return pointer to the allocated memory + */ +void* +lumiera_calloc (size_t n, size_t size) +{ + void* o = (n&&size)? calloc (n, size) : NULL; if (!o) LUMIERA_DIE (NO_MEMORY); return o; @@ -120,7 +137,6 @@ static void lumiera_tmpbuf_destroy (void* buf) { lumiera_tmpbuf_freeall (); - free (buf); } static void @@ -139,17 +155,13 @@ lumiera_tmpbuf_freeall (void) { pthread_once (&lumiera_tmpbuf_tls_once, lumiera_tmpbuf_init); struct lumiera_tmpbuf_struct* buf = pthread_getspecific (lumiera_tmpbuf_tls_key); - if (!buf) - pthread_setspecific (lumiera_tmpbuf_tls_key, - buf = lumiera_malloc (sizeof (struct lumiera_tmpbuf_struct))); - - for (buf->idx = 0; buf->idx < 64; ++buf->idx) + if (buf) { - free (buf->buffers[buf->idx]); - buf->buffers[buf->idx] = NULL; - buf->sizes[buf->idx] = 0; + pthread_setspecific (lumiera_tmpbuf_tls_key, NULL); + for (int idx = 0; idx < 64; ++idx) + free (buf->buffers[idx]); + free (buf); } - buf->idx = 0; } @@ -165,7 +177,7 @@ lumiera_tmpbuf_provide (size_t size) struct lumiera_tmpbuf_struct* buf = pthread_getspecific (lumiera_tmpbuf_tls_key); if (!buf) pthread_setspecific (lumiera_tmpbuf_tls_key, - buf = lumiera_malloc (sizeof (struct lumiera_tmpbuf_struct))); + buf = lumiera_calloc (1, sizeof (struct lumiera_tmpbuf_struct))); buf->idx = (buf->idx + 1) & 0x3f; diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index 02d931cb7..1aa806fd1 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -84,7 +84,6 @@ lumiera_streq (const char* a, const char* b); void lumiera_tmpbuf_freeall (void); - /** * Query a thread local tmpbuf. * @param size minimal needed size for the tmpbuf From 293b872b5d89e7a32dbfbd8b998c7476b5cd3a3e Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 4 Apr 2008 07:58:11 +0200 Subject: [PATCH 35/59] extend the MUTEX_SECTION macro with NoBug resource tracking LUMIERA_MUTEX_SECTION takes now a nobug flag as first argument and a nobug resource-handle as second argument. This change works forward for the NoBug resource tracker and deadlock detector. --- src/lib/mutex.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/lib/mutex.h b/src/lib/mutex.h index 9381c3ca1..ca48ea592 100644 --- a/src/lib/mutex.h +++ b/src/lib/mutex.h @@ -27,12 +27,15 @@ /** * @file Mutual exclusion locking, header. */ - -#define LUMIERA_MUTEX_SECTION(mutex) \ +#define LUMIERA_MUTEX_SECTION(flag, handle, mutex) \ +RESOURCE_HANDLE (rh_##__LINE__##_); \ lumiera_mutexacquirer lock_##__LINE__##_; \ +RESOURCE_ENTER (flag, handle, "acquire mutex", &lock_##__LINE__##_, \ + NOBUG_RESOURCE_EXCLUSIVE, rh_##__LINE__##_); \ for (lumiera_mutexacquirer_init_mutex (&lock_##__LINE__##_, mutex, LUMIERA_LOCKED); \ lock_##__LINE__##_.state == LUMIERA_LOCKED; \ - lumiera_mutexacquirer_unlock (&lock_##__LINE__##_)) + lumiera_mutexacquirer_unlock (&lock_##__LINE__##_), \ + ({RESOURCE_LEAVE(flag, rh_##__LINE__##_);})) /** From b909c2247d3ef8c545c96e534db6d025038ec445 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 05:43:40 +0200 Subject: [PATCH 36/59] add a cuckoo_nelements function Query the number of elements stored in a hash, useful for debugging. --- src/lib/cuckoo.c | 7 +++++++ src/lib/cuckoo.h | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/src/lib/cuckoo.c b/src/lib/cuckoo.c index 0d0b4efef..bbcb0e71a 100644 --- a/src/lib/cuckoo.c +++ b/src/lib/cuckoo.c @@ -208,6 +208,13 @@ cuckoo_insert_internal_ (Cuckoo self, void* item) } +size_t +cuckoo_nelements (Cuckoo self) +{ + return self->elements; +} + + static void cuckoo_rehash (Cuckoo self) { diff --git a/src/lib/cuckoo.h b/src/lib/cuckoo.h index db1d33da3..63b6d260d 100644 --- a/src/lib/cuckoo.h +++ b/src/lib/cuckoo.h @@ -105,6 +105,13 @@ cuckoo_destroy (Cuckoo self); void cuckoo_free (Cuckoo self); +/** + * Get the number of elements stored in a hash. + * @return number of elements, 0 when empty + */ +size_t +cuckoo_nelements (Cuckoo self); + /** * Insert an element into a hash. * amortized O(1) complexity because it may rarely rehash the tables or even grow them. From 0996c386bbab0250fda9fcde0061ecc8c8f26156 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 06:38:05 +0200 Subject: [PATCH 37/59] Mrucache update * Fix: unlinking at destroy * rename add/remove to checkin/checkout * add a drop function which moves a node to the end for fast reuse --- src/lib/mrucache.c | 1 + src/lib/mrucache.h | 30 +++++++++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/src/lib/mrucache.c b/src/lib/mrucache.c index d98976184..b05169db8 100644 --- a/src/lib/mrucache.c +++ b/src/lib/mrucache.c @@ -43,6 +43,7 @@ lumiera_mrucache_destroy (LumieraMruCache self) { LLIST_WHILE_TAIL (&self->cache_list, node) { + llist_unlink (node); if (self->destructor_cb) free (self->destructor_cb (node)); else diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h index 8af7434d9..4644abe8d 100644 --- a/src/lib/mrucache.h +++ b/src/lib/mrucache.h @@ -67,7 +67,7 @@ lumiera_mrucache_destroy (LumieraMruCache self); * add an element to a mrucache */ static inline void -lumiera_mrucache_add (LumieraMruCache self, LList node) +lumiera_mrucache_checkin (LumieraMruCache self, LList node) { REQUIRE (self); REQUIRE (node && llist_is_empty (node)); @@ -76,11 +76,34 @@ lumiera_mrucache_add (LumieraMruCache self, LList node) } +/** + * place an element at the tail of a mrucache, to be reused as first + */ +static inline void +lumiera_mrucache_drop (LumieraMruCache self, LList node) +{ + REQUIRE (self); + REQUIRE (node); + + if (llist_is_empty (node)) + { + /* was not in list, we need to count it */ + ++self->cached; + } + else + { + /* speedup loop warning :P, this check is costly */ + REQUIRE (llist_is_member (&self->cache_list, node), "node must be empty or member of cache"); + } + llist_insert_tail (&self->cache_list, node); +} + + /** * Remove an element from a mrucache. */ static inline void -lumiera_mrucache_remove (LumieraMruCache self, LList node) +lumiera_mrucache_checkout (LumieraMruCache self, LList node) { REQUIRE (self); REQUIRE (node && llist_is_member (&self->cache_list, node)); /* speedup loop warning :P, this check is costly */ @@ -88,6 +111,7 @@ lumiera_mrucache_remove (LumieraMruCache self, LList node) --self->cached; } + /** * destroy the oldest element and return a pointer to it for reuse. */ @@ -98,7 +122,7 @@ lumiera_mrucache_pop (LumieraMruCache self) if (llist_is_empty (&self->cache_list)) return NULL; - LList node = llist_tail (&self->cache_list); + LList node = llist_tail (&self->cache_list); llist_unlink (node); --self->cached; From ff6ce603a4b683210449efc92dc14257f70fdfb4 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 19:21:52 +0200 Subject: [PATCH 38/59] tmpbuf_sprintf renamed to tmpbuf_snprintf --- src/lib/safeclib.c | 2 +- src/lib/safeclib.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 934c3ff5b..1240cb673 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -206,7 +206,7 @@ lumiera_tmpbuf_strndup (const char* src, size_t size) char* -lumiera_tmpbuf_sprintf (size_t size, const char* fmt, ...) +lumiera_tmpbuf_snprintf (size_t size, const char* fmt, ...) { va_list args; diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index 1aa806fd1..2c835c8c2 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -109,5 +109,5 @@ lumiera_tmpbuf_strndup (const char* src, size_t size); * @return temporary buffer containing the constructed of the string */ char* -lumiera_tmpbuf_sprintf (size_t size, const char* fmt, ...); +lumiera_tmpbuf_snprintf (size_t size, const char* fmt, ...); From dc0cbf0ad8ef18b329ef4ae163b07712a363e762 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 20:36:24 +0200 Subject: [PATCH 39/59] Fix: off by one allocation error in tmpbuf_snprintf() --- src/lib/safeclib.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 1240cb673..0e82e2dc0 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -215,9 +215,9 @@ lumiera_tmpbuf_snprintf (size_t size, const char* fmt, ...) va_end (args); len = len > size ? size : len; - char* buf = lumiera_tmpbuf_provide (len); + char* buf = lumiera_tmpbuf_provide (len+1); va_start (args, fmt); - vsnprintf (buf, len, fmt, args); + vsnprintf (buf, len+1, fmt, args); va_end (args); return buf; From 63c07d12749dafa3dac7a18a3d8580a949729612 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 9 Apr 2008 07:57:03 +0200 Subject: [PATCH 40/59] doxydocing mrucache --- src/lib/mrucache.c | 10 ---------- src/lib/mrucache.h | 50 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 40 insertions(+), 20 deletions(-) diff --git a/src/lib/mrucache.c b/src/lib/mrucache.c index b05169db8..0f9bb86df 100644 --- a/src/lib/mrucache.c +++ b/src/lib/mrucache.c @@ -21,12 +21,6 @@ #include "lib/mrucache.h" -/** - * @file Most recent used cache - * Elements (addressed by a LList node) are either in the cache and thereby subject of aging - * or checked out under control of the user. - */ - LumieraMruCache lumiera_mrucache_init (LumieraMruCache self, lumiera_cache_destructor_fn destructor_cb) @@ -53,10 +47,6 @@ lumiera_mrucache_destroy (LumieraMruCache self) return self; } - -/** - * destroy and free the nelem oldest elements - */ int lumiera_mrucache_age (LumieraMruCache self, int nelem) { diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h index 4644abe8d..3edb680f0 100644 --- a/src/lib/mrucache.h +++ b/src/lib/mrucache.h @@ -28,8 +28,9 @@ /** * @file Most recent used cache - * Elements (addressed by a LList node) are either in the cache and thereby subject of aging - * or checked out under control of the user. + * Elements (addressed by a LList node) are either checked in the cache and thereby subject of aging + * or checked out under control of the user. Most operations require that the chache is locked. This locking + * must be done from elsewhere. */ /** @@ -50,21 +51,31 @@ typedef struct lumiera_mrucache_struct lumiera_mrucache; typedef lumiera_mrucache* LumieraMruCache; /** - * Initialize a mrucache + * Initialize a cache. + * @param self cache to be initialized + * @param destructor_cb function for destructing a cache node, preparing for reuse. + * @return self */ LumieraMruCache lumiera_mrucache_init (LumieraMruCache self, lumiera_cache_destructor_fn destructor_cb); - /** - * Destroy a mrucache + * Destroy a cache. + * frees all checked in items. + * @param self cache to be destroyed + * @return self */ LumieraMruCache lumiera_mrucache_destroy (LumieraMruCache self); /** - * add an element to a mrucache + * Add an element to a mrucache. + * When added the element is subject of aging and must not be accessed anymore. + * To acces elements they have to be checked out again. + * Checkin and checkout operations must be protected by a lock over the cache. + * @param self cache where to checkin + * @param node a llist member in the cached data which is used as queue in the cache */ static inline void lumiera_mrucache_checkin (LumieraMruCache self, LList node) @@ -77,7 +88,13 @@ lumiera_mrucache_checkin (LumieraMruCache self, LList node) /** - * place an element at the tail of a mrucache, to be reused as first + * Schedule an element for fast aging. + * When an element is not longer needed it can be placed at the end of the chache + * aging queue and thus become the first one to be reused when a new element is queried. + * This can be done on a checked out element as well as on an element which is in the cache. + * The cache must be locked for this operation. + * @param self cache + * @param node a llist member in the cached data which is used as queue in the cache */ static inline void lumiera_mrucache_drop (LumieraMruCache self, LList node) @@ -100,7 +117,12 @@ lumiera_mrucache_drop (LumieraMruCache self, LList node) /** - * Remove an element from a mrucache. + * Checkout an element from a cache. + * A checked out element is not under cache control anymore until is gets checked in again. + * The code which checked the element out takes ownership of the element. + * The cache must be locked for this operation. + * @param self cache + * @param node a llist member in the cached data which is used as queue in the cache */ static inline void lumiera_mrucache_checkout (LumieraMruCache self, LList node) @@ -113,7 +135,11 @@ lumiera_mrucache_checkout (LumieraMruCache self, LList node) /** - * destroy the oldest element and return a pointer to it for reuse. + * Destroy the oldest element from the cache and return it. + * This function is used to get a new element by deleting the oldest least used one from the cache. + * The cache must be locked for this operation. + * @param self cache + * @return pointer to the uninitialized memory ready for being reused */ static inline void* lumiera_mrucache_pop (LumieraMruCache self) @@ -134,7 +160,11 @@ lumiera_mrucache_pop (LumieraMruCache self) /** - * destroy and free the nelem oldest elements + * Destroy and free the nelem oldest elements. + * Used to free up resources and memory. + * @param self cache where to free elements. + * @param nelem number of elements wished to be freed + * @return nelem-(numer of elements which got freed), that is 0 if all requested elements got freed */ int lumiera_mrucache_age (LumieraMruCache self, int nelem); From 03ed87051d8c8377a22f8767e61414660efa4349 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 2 May 2008 23:58:12 +0200 Subject: [PATCH 41/59] fix missing declaration of lumiera_calloc() in safeclib.h --- src/lib/safeclib.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index 2c835c8c2..f486ed056 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -38,6 +38,17 @@ void* lumiera_malloc (size_t sz); +/** + * Allocate cleared memory for an array. + * always succeeds or dies + * @param n number of elements + * @param size memory to be allocated + * @return pointer to the allocated memory + */ +void* +lumiera_calloc (size_t n, size_t size); + + /** * Duplicate a C string. * always succeeds or dies From 94e075d2c2ea02042b809eb48c8635203d9c8f75 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 2 May 2008 23:59:26 +0200 Subject: [PATCH 42/59] fix: mrucache_drop() must call the element destructor --- src/lib/mrucache.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h index 3edb680f0..c81bd7599 100644 --- a/src/lib/mrucache.h +++ b/src/lib/mrucache.h @@ -35,7 +35,8 @@ /** * Callback function used to destruct/cleanup aged elements. - * shall clean the element sufficiently up to be ready for being freed or reused + * shall clean the element sufficiently up to be ready for being freed or reused, + * this callback function must be reentrant and prepared to be called twice. * @param node the llist node used to link cache elements (will be empty at call) * @return pointer to the begin of the element. */ @@ -113,6 +114,9 @@ lumiera_mrucache_drop (LumieraMruCache self, LList node) REQUIRE (llist_is_member (&self->cache_list, node), "node must be empty or member of cache"); } llist_insert_tail (&self->cache_list, node); + + if (self->destructor_cb) + self->destructor_cb (node); } From 791f42c10474cd26f096c070a0feba51e4c72706 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Sun, 30 Mar 2008 04:31:32 +0200 Subject: [PATCH 43/59] WIP: Low level file management We have 'File' which associates a name with a file and will be used to offer an interface for the application. Posix 'Filehandles' which are managed in a 'Filehandlecache' to utilize a limited number of fd's at optimum. Internally we use 'Filedescriptor's which manage files, mapping and all kinds of metadata. Filedescriptors weak referenced by a registry and freed when the last user vanishes. --- src/backend/Makefile.am | 15 ++- src/backend/file.c | 64 +++++++++++ src/backend/file.h | 90 +++++++++++++++ src/backend/filedescriptor.c | 199 ++++++++++++++++++++++++++++++++++ src/backend/filedescriptor.h | 58 ++++++++++ src/backend/filehandle.h | 51 +++++++++ src/backend/filehandlecache.c | 83 ++++++++++++++ src/backend/filehandlecache.h | 97 +++++++++++++++++ 8 files changed, 649 insertions(+), 8 deletions(-) create mode 100644 src/backend/file.c create mode 100644 src/backend/file.h create mode 100644 src/backend/filedescriptor.c create mode 100644 src/backend/filedescriptor.h create mode 100644 src/backend/filehandle.h create mode 100644 src/backend/filehandlecache.c create mode 100644 src/backend/filehandlecache.h diff --git a/src/backend/Makefile.am b/src/backend/Makefile.am index eb5853d5f..45ce555a8 100644 --- a/src/backend/Makefile.am +++ b/src/backend/Makefile.am @@ -31,12 +31,11 @@ liblumibackend_a_SOURCES = \ # $(liblumibackend_a_srcdir)/filehandlecache.c -noinst_HEADERS += \ - $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp - -# $(liblumibackend_a_srcdir)/backend.h \ -# $(liblumibackend_a_srcdir)/file.h \ -# $(liblumibackend_a_srcdir)/filehandle.h \ -# $(liblumibackend_a_srcdir)/filedescriptor.h \ -# $(liblumibackend_a_srcdir)/filehandlecache.h +noinst_HEADERS += \ + $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp \ + $(liblumibackend_a_srcdir)/backend.h \ + $(liblumibackend_a_srcdir)/file.h \ + $(liblumibackend_a_srcdir)/filehandle.h \ + $(liblumibackend_a_srcdir)/filedescriptor.h \ + $(liblumibackend_a_srcdir)/filehandlecache.h diff --git a/src/backend/file.c b/src/backend/file.c new file mode 100644 index 000000000..7a4aa5050 --- /dev/null +++ b/src/backend/file.c @@ -0,0 +1,64 @@ +/* + file.c - file handling + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "lib/mutex.h" +#include "lib/safeclib.h" + +#include "backend/file.h" +#include "backend/filedescriptor.h" + +#include + + +NOBUG_DEFINE_FLAG (lumiera_file); + +LumieraFile +lumiera_file_init (LumieraFile self, const char* name, int flags) +{ + if (!lumiera_filedescriptor_acquire (&self->descriptor, name, flags)) + return NULL; + self->name = lumiera_strndup (name, PATH_MAX); + + return self; +} + +LumieraFile +lumiera_file_destroy (LumieraFile self) +{ + lumiera_reference_destroy (&self->descriptor); + free ((void*)self->name); + return self; +} + + +LumieraFile +lumiera_file_new (const char* name, int flags) +{ + LumieraFile self = lumiera_malloc (sizeof (lumiera_file)); + return lumiera_file_init (self, name, flags); +} + +void +lumiera_file_delete (LumieraFile self) +{ + free (lumiera_file_destroy (self)); +} + diff --git a/src/backend/file.h b/src/backend/file.h new file mode 100644 index 000000000..5018d321a --- /dev/null +++ b/src/backend/file.h @@ -0,0 +1,90 @@ +/* + file.h - interface to files by filename + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_FILE_H +#define LUMIERA_FILE_H + + +#include "lib/mutex.h" +#include "lib/llist.h" +#include "lib/error.h" +#include "lib/references.h" + +/** + * @file File management + * Handling Files is splitted into different classes: + * 1. The 'lumiera_file' class which acts as interface to the outside for managing files. + * 'lumiera_file' is addressed by the name of the file. Since files can have more than one name (hardlinks) + * many 'lumiera_file' can point to a single 'lumiera_filedescriptor' + * 2. The 'lumiera_filedescriptor' class which does the real work managing the file in the back. + * 3. Since OS-filehandles are a limited resource we access the lazily as 'lumiera_filehandle' which are + * managed in a 'lumiera_filehandlecache' + */ + +typedef struct lumiera_file_struct lumiera_file; +typedef lumiera_file* LumieraFile; + + +#include "backend/filehandle.h" +#include "backend/filedescriptor.h" + + + +NOBUG_DECLARE_FLAG (lumiera_file); + +#define LUMIERA_FILE_READONLY (O_RDONLY | O_LARGEFILE | O_NOATIME) +#define LUMIERA_FILE_READWRITE (O_RDWR | O_LARGEFILE | O_NOATIME) +#define LUMIERA_FILE_CREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_EXCL) +#define LUMIERA_FILE_RECREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_TRUNC) + +struct lumiera_file_struct +{ + const char* name; + struct lumiera_reference_struct descriptor; +}; + +/** + * + */ +LumieraFile +lumiera_file_init (LumieraFile self, const char* name, int flags); + +/** + * + */ +LumieraFile +lumiera_file_destroy (LumieraFile self); + +/** + * + */ +LumieraFile +lumiera_file_new (const char* name, int flags); + +/** + * + */ +void +lumiera_file_delete (LumieraFile self); + + +#endif + diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c new file mode 100644 index 000000000..fdb22ffb3 --- /dev/null +++ b/src/backend/filedescriptor.c @@ -0,0 +1,199 @@ +/* + filedescriptor.c - file handling + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "lib/mutex.h" +#include "lib/safeclib.h" +#include "lib/cuckoo.h" + +#include "backend/filedescriptor.h" + +#include +#include +#include +#include + + +NOBUG_DEFINE_FLAG (filedescriptor); + +static Cuckoo registry = NULL; +static lumiera_mutex registry_mutex = {PTHREAD_MUTEX_INITIALIZER}; + + +static size_t +h1 (const void* item, const uint32_t r) +{ + const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>7)^r; +} + +static size_t +h2 (const void* item, const uint32_t r) +{ + const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>5)^r; +} + +static size_t +h3 (const void* item, const uint32_t r) +{ + const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>3)^r; +} + + +static int +cmp (const void* keya, const void* keyb) +{ + const LumieraFiledescriptor a = (const LumieraFiledescriptor) keya; + const LumieraFiledescriptor b = (const LumieraFiledescriptor) keyb; + return a->stat.st_dev == b->stat.st_dev && a->stat.st_ino == b->stat.st_ino && a->flags == b->flags; +} + + +static Cuckoo +lumiera_filedescriptor_registry_new (void) +{ + NOBUG_INIT_FLAG (filedescriptor); + + Cuckoo registry = cuckoo_new (h1, h2, h3, cmp, + sizeof (struct lumiera_reference_struct), + 3); + if (!registry) + LUMIERA_DIE (NO_MEMORY); + + return registry; +} + +/** + * Find existing filedescriptor or create one + * @param descriptor strong reference to be initialized with the filedescriptor, + * a filedescriptor will be deleted when its last string reference gets deleted. + * @param name name of the file + * @param flags opening flags for the filedescriptor + * @return descriptor on success or NULL on error + */ +LumieraReference +lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, int flags) +{ + UNCHECKED; + lumiera_mutexacquirer registry_lock; + lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); + + if (!registry) + registry = lumiera_filedescriptor_registry_new (); + + struct lumiera_filedescriptor_struct findme; + findme.flags = flags; + + for (int retry = 0; !retry; ++retry) + { + if (stat (name, &findme.stat) == 0) + break; + else + { + if (errno == ENOENT && flags&O_CREAT) + { + TRACE (filedescriptor, "try creating file: %s", name); + char* slash = lumiera_tmpbuf_strndup (name, PATH_MAX); + + while ((slash = strchr (slash+1, '/'))) + { + *slash = '\0'; + if (mkdir (name, 0777) == -1) + { + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; + } + *slash = '/'; + } + + int fd; + fd = creat (name, 0777); + if (fd == -1) + { + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; + } + close (fd); + /* will now retry the fstat once */ + } + else + { + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; + } + } + } + + /* lookup/create descriptor */ + LumieraReference elem = cuckoo_find (registry, &findme); + + if (!elem) + { + TRACE (filedescriptor, "Descriptor not found"); + + lumiera_reference_strong_init_once (descriptor, + lumiera_filedescriptor_new (&findme), + (void(*)(void*))lumiera_filedescriptor_delete); + + if (!lumiera_reference_get (descriptor)) + goto ecreate; + + lumiera_reference weak_descriptor; + lumiera_reference_weak_init (&weak_descriptor, descriptor); + cuckoo_insert (registry, &weak_descriptor); + } + else + { + lumiera_reference_strong_init (descriptor, elem); + } + lumiera_mutexacquirer_unlock (®istry_lock); + + return descriptor; + + efile: + ecreate: + lumiera_mutexacquirer_unlock (®istry_lock); + return NULL; +} + + +/** + * Allocate a new filedescriptor cloned from a template + * @param template the source filedescriptor + * @return the constrccted filedescriptor + */ +LumieraFiledescriptor +lumiera_filedescriptor_new (LumieraFiledescriptor template) +{ + UNIMPLEMENTED(""); + return NULL; +} + +/** + * delete a filedescriptor and free its resources + * @param descriptor filedescriptor to be freed + */ +void +lumiera_filedescriptor_delete (LumieraFiledescriptor descriptor) +{ + UNIMPLEMENTED(""); +} diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h new file mode 100644 index 000000000..060839d0a --- /dev/null +++ b/src/backend/filedescriptor.h @@ -0,0 +1,58 @@ +/* + filedescriptor.h - file handling + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_FILEDESCRIPTOR_H +#define LUMIERA_FILEDESCRIPTOR_H + +#include "lib/mutex.h" + +#include + +typedef struct lumiera_filedescriptor_struct lumiera_filedescriptor; +typedef lumiera_filedescriptor* LumieraFiledescriptor; + + +#include "backend/filehandle.h" +#include "backend/file.h" + + +struct lumiera_filedescriptor_struct +{ + struct stat stat; /* create after first open, maintained metadata, MUST BE FIRST! */ + int flags; /* open flags, must be masked for reopen */ + lumiera_mutex lock; /* locks operations on this file descriptor */ + unsigned reopened; /* count for reopens 0=not yet opened, 1=first, 2..=reopened*/ + //LumieraFileMap mappings; + //LumieraWriteBuffer writebuffer; +}; + + +LumieraReference +lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, int flags); + +LumieraFiledescriptor +lumiera_filedescriptor_new (LumieraFiledescriptor template); + +void +lumiera_filedescriptor_delete (LumieraFiledescriptor self); + + +#endif diff --git a/src/backend/filehandle.h b/src/backend/filehandle.h new file mode 100644 index 000000000..00b4737c9 --- /dev/null +++ b/src/backend/filehandle.h @@ -0,0 +1,51 @@ +/* + filehandle - filehandle management and caching + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#ifndef LUMIERA_FILEHANDLE_H +#define LUMIERA_FILEHANDLE_H + +#include "lib/error.h" +#include "lib/llist.h" + +/** + * @file Filehandles + */ + + +/** + * File handles + */ +struct lumiera_filehandle_struct +{ + llist cachenode; + int handle; + //LumieraFile file; + //lumiera_mutex lock; +}; +typedef struct lumiera_filehandle_struct lumiera_filehandle; +typedef lumiera_filehandle* LumieraFilehandle; + +LumieraFilehandle +lumiera_filehandle_new (); + +void* +lumiera_filehandle_destroy_node (LList node); + +#endif diff --git a/src/backend/filehandlecache.c b/src/backend/filehandlecache.c new file mode 100644 index 000000000..98ebdddc9 --- /dev/null +++ b/src/backend/filehandlecache.c @@ -0,0 +1,83 @@ +/* + filehandlecache - filehandle management and caching + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "backend/filehandlecache.h" + +NOBUG_DEFINE_FLAG (lumiera_filehandlecache); + +/* errors */ + +LUMIERA_ERROR_DEFINE (FILEHANDLECACHE_NOHANDLE, "No filehandle available"); + + +/**/ + +LumieraFilehandlecache +lumiera_filehandlecache_init (LumieraFilehandlecache self, int max_entries) +{ + lumiera_mrucache_init (&self->cache, lumiera_filehandle_destroy_node); + self->available =max_entries; + lumiera_mutex_init (&self->lock); + return self; +} + +LumieraFilehandlecache +lumiera_filehandlecache_destroy (LumieraFilehandlecache self) +{ + self->available += self->cache.cached; + lumiera_mrucache_destroy (&self->cache); + lumiera_mutex_destroy (&self->lock); + return self; +} + + +LumieraFilehandle +lumiera_filehandlecache_filehandle_acquire (LumieraFilehandlecache self) +{ + LumieraFilehandle ret; + LUMIERA_MUTEX_SECTION (&self->lock) + { + if (self->available <= 0 && self->cache.cached) + { + /* pop a filehandle from cache */ + ret = lumiera_mrucache_pop (&self->cache); + if (self->available < 0) + /* try to free overallocated filehandles */ + self->available -= self->available + lumiera_mrucache_age (&self->cache, -self->available); + } + else + { + /* allocate new filehandle if we are below the limit or no cached handles where available (overallocating) */ + ret = lumiera_filehandle_new (); + if (!ret) + LUMIERA_ERROR_SET (lumiera_filehandlecache, FILEHANDLECACHE_NOHANDLE); + else + --self->available; + } + } + return ret; +} + +void +lumiera_filehandlecache_add (LumieraFilehandlecache self, LumieraFilehandle handle) +{ + UNIMPLEMENTED (""); +} diff --git a/src/backend/filehandlecache.h b/src/backend/filehandlecache.h new file mode 100644 index 000000000..40386defe --- /dev/null +++ b/src/backend/filehandlecache.h @@ -0,0 +1,97 @@ +/* + filehandlecache - filehandle management and caching + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#ifndef LUMIERA_FILEHANDLECACHE_H +#define LUMIERA_FILEHANDLECACHE_H + +#include "lib/error.h" +#include "lib/mrucache.h" +#include "lib/mutex.h" + +#include "backend/filehandle.h" + +/** + * @file Filehandle management and caching + * The number of filehandles a program can held open is usually limited, since we want to support + * using a less limited number of files and closing/opening for each operation is expensive, we + * provide a cache to keep the most frequent used files open and gracefully close/recycle unused filehandles. + */ + + + +/* + File handle cache manages file handles + */ +struct lumiera_filehandlecache_struct +{ + lumiera_mrucache cache; + int available; + lumiera_mutex lock; +}; +typedef struct lumiera_filehandlecache_struct lumiera_filehandlecache; +typedef lumiera_filehandlecache* LumieraFilehandlecache; + + +/** + * Initializes a filehandle cache. + * @param self pointer to the cache to be initialized + * @param max_entries number how much filehandles shall be managed + * @return self as given + * The number of elements the cache can hold is static and should be + * determined by sysconf (_SC_OPEN_MAX) minus some safety margin. + */ +LumieraFilehandlecache +lumiera_filehandlecache_init (LumieraFilehandlecache self, int max_entries); + +/** + * Destroy a filehandle cache. + * @param self the cache to be destroyed + * @return self as given to the now uninitialized! cache + * No filehandles in the cache must be locked, this would be a fatal error. + * The handles are closed automatically. + */ +LumieraFilehandlecache +lumiera_filehandlecache_destroy (LumieraFilehandlecache self); + +/** + * Get a fresh filehandle. + * @param self pointer to the cache + * @return the new filehandle + * one must call lumiera_mutexacquirer_unlock (&lock) to free the lock on the handle when done + */ +LumieraFilehandle +lumiera_filehandlecache_filehandle_acquire (LumieraFilehandlecache self); + +/** + * Add filehande back to cache. + * @param self pointer to the cache + * @param handle filehandle to be put back + */ +void +lumiera_filehandlecache_add (LumieraFilehandlecache self, LumieraFilehandle handle); + +/** + * + */ +LumieraFilehandle +lumiera_filehandlecache_remove (LumieraFilehandlecache self, int fd); + + +#endif From 7a266f22ba1dc41fd0519f8a47b86570cb891582 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 31 Mar 2008 02:28:25 +0200 Subject: [PATCH 44/59] integrate filedescriptor tests --- tests/20filedescriptor.tests | 20 ++++++++ tests/Makefile.am | 5 ++ tests/backend/test-filedescriptors.c | 70 ++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+) create mode 100644 tests/20filedescriptor.tests create mode 100644 tests/backend/test-filedescriptors.c diff --git a/tests/20filedescriptor.tests b/tests/20filedescriptor.tests new file mode 100644 index 000000000..703a3351e --- /dev/null +++ b/tests/20filedescriptor.tests @@ -0,0 +1,20 @@ + +TESTING "Filedescriptor management" ./test-filedescriptors + +echo testdata > ,tmp_testfile + +TEST "acquire existing file" acquire_existing < + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#define _GNU_SOURCE + +#include "lib/references.h" +#include "lib/safeclib.h" + +#include "backend/filedescriptor.h" + +#include "tests/test.h" + +TESTS_BEGIN + + +TEST ("acquire_existing") +{ + lumiera_reference descriptor; + if (lumiera_filedescriptor_acquire (&descriptor, ",tmp_testfile", LUMIERA_FILE_READONLY)) + { + lumiera_reference_destroy (&descriptor); + return 0; + } + else + return 1; +} + +TEST ("acquire_create") +{ + lumiera_reference descriptor; + if (lumiera_filedescriptor_acquire (&descriptor, ",tmp_testfile", LUMIERA_FILE_CREATE)) + { + lumiera_reference_destroy (&descriptor); + return 0; + } + else + return 1; +} + +TEST ("acquire_create_dir") +{ + lumiera_reference descriptor; + if (lumiera_filedescriptor_acquire (&descriptor, ",tmp_testdir/,tmp_testfile", LUMIERA_FILE_CREATE)) + { + lumiera_reference_destroy (&descriptor); + return 0; + } + else + return 1; +} + + +TESTS_END From e2e94884badc0e58606f0841dfd91d3368b051f6 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 1 Apr 2008 19:20:17 +0200 Subject: [PATCH 45/59] some refinements to the todo wiki --- wiki/todo.html | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/wiki/todo.html b/wiki/todo.html index d6caf66be..e66ec929f 100644 --- a/wiki/todo.html +++ b/wiki/todo.html @@ -3344,7 +3344,11 @@ Someone might pickup a Doxygen-Maintainer. This is not writing actual documentat 
Use the backend, gmerlin/avdecoder and a Player widget to playback videos. This should be a small application which can open and playback many videos from one instance in parallel. Later on scrubbing for this videos might be supported.
 
-The idea here is to show and measure backend and scheduler performance and beeing a proof of concept for the design. It will run independent of the Proc layer but needs quite some work on the backend to be done.
+The idea here is to show and measure backend and scheduler performance + and beeing a proof of concept for the design. It will run + independent of the Proc layer but needs quite some work on the + backend to be done. +
From 52868388866c6c038cbeba9286092ee65d1f3ed6 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 1 Apr 2008 22:32:35 +0200 Subject: [PATCH 46/59] filedescriptor handling basics working, including some tests It is now possible to acquire/release filedescriptors for existing files and files to be created. Filedescriptors are managed in a registry by refcounts, they get automatically destructed when the last user releases them. --- src/backend/filedescriptor.c | 123 ++++++++++++++++++--------- src/backend/filedescriptor.h | 14 ++- tests/20filedescriptor.tests | 8 +- tests/backend/test-filedescriptors.c | 35 +++++--- 4 files changed, 125 insertions(+), 55 deletions(-) diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index fdb22ffb3..6db2882c9 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -33,28 +33,35 @@ NOBUG_DEFINE_FLAG (filedescriptor); +/* + Filedescriptor registry + + This registry stores all acquired filedescriptors for lookup, they will be freed when not referenced anymore. + */ static Cuckoo registry = NULL; static lumiera_mutex registry_mutex = {PTHREAD_MUTEX_INITIALIZER}; - +/* + * setup hashing and compare functions for cuckoo hashing + */ static size_t h1 (const void* item, const uint32_t r) { - const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>7)^r; } static size_t h2 (const void* item, const uint32_t r) { - const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>5)^r; } static size_t h3 (const void* item, const uint32_t r) { - const LumieraFiledescriptor i = (const LumieraFiledescriptor) item; + const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>3)^r; } @@ -62,19 +69,18 @@ h3 (const void* item, const uint32_t r) static int cmp (const void* keya, const void* keyb) { - const LumieraFiledescriptor a = (const LumieraFiledescriptor) keya; - const LumieraFiledescriptor b = (const LumieraFiledescriptor) keyb; + const LumieraFiledescriptor a = *(const LumieraFiledescriptor*)keya; + const LumieraFiledescriptor b = *(const LumieraFiledescriptor*)keyb; return a->stat.st_dev == b->stat.st_dev && a->stat.st_ino == b->stat.st_ino && a->flags == b->flags; } - static Cuckoo lumiera_filedescriptor_registry_new (void) { NOBUG_INIT_FLAG (filedescriptor); Cuckoo registry = cuckoo_new (h1, h2, h3, cmp, - sizeof (struct lumiera_reference_struct), + sizeof (LumieraFiledescriptor), 3); if (!registry) LUMIERA_DIE (NO_MEMORY); @@ -82,6 +88,12 @@ lumiera_filedescriptor_registry_new (void) return registry; } +static void +lumiera_filedescriptor_registry_delete (void) +{ + cuckoo_free (registry); +} + /** * Find existing filedescriptor or create one * @param descriptor strong reference to be initialized with the filedescriptor, @@ -90,34 +102,40 @@ lumiera_filedescriptor_registry_new (void) * @param flags opening flags for the filedescriptor * @return descriptor on success or NULL on error */ -LumieraReference -lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, int flags) +LumieraFiledescriptor +lumiera_filedescriptor_acquire (const char* name, int flags) { + TRACE (filedescriptor); UNCHECKED; lumiera_mutexacquirer registry_lock; lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); if (!registry) - registry = lumiera_filedescriptor_registry_new (); + { + registry = lumiera_filedescriptor_registry_new (); + atexit (lumiera_filedescriptor_registry_delete); + } - struct lumiera_filedescriptor_struct findme; - findme.flags = flags; + lumiera_filedescriptor fdesc; + fdesc.flags = flags; for (int retry = 0; !retry; ++retry) { - if (stat (name, &findme.stat) == 0) + if (stat (name, &fdesc.stat) == 0) break; else { if (errno == ENOENT && flags&O_CREAT) { - TRACE (filedescriptor, "try creating file: %s", name); - char* slash = lumiera_tmpbuf_strndup (name, PATH_MAX); + INFO (filedescriptor, "try creating file: %s %d", name, PATH_MAX); + char* dir = lumiera_tmpbuf_strndup (name, PATH_MAX); + char* slash = dir; while ((slash = strchr (slash+1, '/'))) { *slash = '\0'; - if (mkdir (name, 0777) == -1) + INFO (filedescriptor, "try creating dir: %s", dir); + if (mkdir (dir, 0777) == -1) { LUMIERA_ERROR_SET (filedescriptor, ERRNO); goto efile; @@ -133,41 +151,37 @@ lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, i goto efile; } close (fd); - /* will now retry the fstat once */ - } - else - { - LUMIERA_ERROR_SET (filedescriptor, ERRNO); - goto efile; + continue; /* will now retry the fstat once */ } } + /* finally, no luck */ + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; } /* lookup/create descriptor */ - LumieraReference elem = cuckoo_find (registry, &findme); + LumieraFiledescriptor dest = &fdesc; + LumieraFiledescriptor* found = cuckoo_find (registry, &dest); - if (!elem) + if (!found) { TRACE (filedescriptor, "Descriptor not found"); - lumiera_reference_strong_init_once (descriptor, - lumiera_filedescriptor_new (&findme), - (void(*)(void*))lumiera_filedescriptor_delete); - - if (!lumiera_reference_get (descriptor)) + dest = lumiera_filedescriptor_new (&fdesc); + if (!dest) goto ecreate; - lumiera_reference weak_descriptor; - lumiera_reference_weak_init (&weak_descriptor, descriptor); - cuckoo_insert (registry, &weak_descriptor); + cuckoo_insert (registry, &dest); } else { - lumiera_reference_strong_init (descriptor, elem); + TRACE (filedescriptor, "Descriptor already existing"); + dest = *found; + ++dest->refcount; } - lumiera_mutexacquirer_unlock (®istry_lock); - return descriptor; + lumiera_mutexacquirer_unlock (®istry_lock); + return dest; efile: ecreate: @@ -175,6 +189,13 @@ lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, i return NULL; } +void +lumiera_filedescriptor_release (LumieraFiledescriptor self) +{ + TRACE (filedescriptor); + if (!--self->refcount) + lumiera_filedescriptor_delete (self); +} /** * Allocate a new filedescriptor cloned from a template @@ -184,8 +205,17 @@ lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, i LumieraFiledescriptor lumiera_filedescriptor_new (LumieraFiledescriptor template) { - UNIMPLEMENTED(""); - return NULL; + LumieraFiledescriptor self = lumiera_malloc (sizeof (lumiera_filedescriptor)); + TRACE (filedescriptor, "at %p", self); + + self->stat = template->stat; + + self->flags = template->flags; + lumiera_mutex_init (&self->lock); + self->reopened = 0; + self->refcount = 1; + + return self; } /** @@ -193,7 +223,20 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template) * @param descriptor filedescriptor to be freed */ void -lumiera_filedescriptor_delete (LumieraFiledescriptor descriptor) +lumiera_filedescriptor_delete (LumieraFiledescriptor self) { - UNIMPLEMENTED(""); + TRACE (filedescriptor, "%p", self); + lumiera_mutexacquirer registry_lock; + lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); + + REQUIRE (self->refcount == 0); + + cuckoo_remove (registry, cuckoo_find (registry, &self)); + + TODO ("destruct other members (WIP)"); + + lumiera_mutex_destroy (&self->lock); + free (self); + + lumiera_mutexacquirer_unlock (®istry_lock); } diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 060839d0a..59c4eacef 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -24,7 +24,10 @@ #include "lib/mutex.h" + +#include #include +#include typedef struct lumiera_filedescriptor_struct lumiera_filedescriptor; typedef lumiera_filedescriptor* LumieraFiledescriptor; @@ -39,14 +42,19 @@ struct lumiera_filedescriptor_struct struct stat stat; /* create after first open, maintained metadata, MUST BE FIRST! */ int flags; /* open flags, must be masked for reopen */ lumiera_mutex lock; /* locks operations on this file descriptor */ - unsigned reopened; /* count for reopens 0=not yet opened, 1=first, 2..=reopened*/ + unsigned reopened; /* count for reopens 0=not yet opened, 1=first, 2..=reopened */ + unsigned refcount; /* reference counter, all users sans registry */ + //LumieraFileMap mappings; //LumieraWriteBuffer writebuffer; }; -LumieraReference -lumiera_filedescriptor_acquire (LumieraReference descriptor, const char* name, int flags); +LumieraFiledescriptor +lumiera_filedescriptor_acquire (const char* name, int flags); + +void +lumiera_filedescriptor_release (LumieraFiledescriptor self); LumieraFiledescriptor lumiera_filedescriptor_new (LumieraFiledescriptor template); diff --git a/tests/20filedescriptor.tests b/tests/20filedescriptor.tests index 703a3351e..78639e7b8 100644 --- a/tests/20filedescriptor.tests +++ b/tests/20filedescriptor.tests @@ -4,17 +4,23 @@ TESTING "Filedescriptor management" ./test-filedescriptors echo testdata > ,tmp_testfile TEST "acquire existing file" acquire_existing < Date: Fri, 4 Apr 2008 08:01:59 +0200 Subject: [PATCH 47/59] add nobug resource tracking to filedescriptors --- src/backend/filedescriptor.c | 8 ++++++++ src/backend/filedescriptor.h | 3 +++ 2 files changed, 11 insertions(+) diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index 6db2882c9..51b536e56 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -214,6 +214,9 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template) lumiera_mutex_init (&self->lock); self->reopened = 0; self->refcount = 1; + self->handle = 0; + + RESOURCE_ANNOUNCE (filedescriptor, "mutex", "filedescriptor", self, self->rh); return self; } @@ -231,10 +234,15 @@ lumiera_filedescriptor_delete (LumieraFiledescriptor self) REQUIRE (self->refcount == 0); + RESOURCE_FORGET (filedescriptor, self->rh); + cuckoo_remove (registry, cuckoo_find (registry, &self)); TODO ("destruct other members (WIP)"); + + TODO ("release filehandle"); + lumiera_mutex_destroy (&self->lock); free (self); diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 59c4eacef..48d0cd21c 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -45,8 +45,11 @@ struct lumiera_filedescriptor_struct unsigned reopened; /* count for reopens 0=not yet opened, 1=first, 2..=reopened */ unsigned refcount; /* reference counter, all users sans registry */ + LumieraFilehandle handle; //LumieraFileMap mappings; //LumieraWriteBuffer writebuffer; + + RESOURCE_HANDLE (rh); }; From b11ff40fbe33846b3f12f38bac35bf00029a85f8 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 05:51:32 +0200 Subject: [PATCH 48/59] Filedescriptor updates * make registry initialization singleton, no parameters. * remove registry init from all functions, has to be called explicitly. * some nobug improvements --- src/backend/filedescriptor.c | 36 ++++++++++++++++++------------------ src/backend/filedescriptor.h | 6 +++++- 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index 51b536e56..0a3d6e048 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -31,7 +31,7 @@ #include -NOBUG_DEFINE_FLAG (filedescriptor); +NOBUG_DEFINE_FLAG_PARENT (filedescriptor, file_all); /* Filedescriptor registry @@ -74,26 +74,28 @@ cmp (const void* keya, const void* keyb) return a->stat.st_dev == b->stat.st_dev && a->stat.st_ino == b->stat.st_ino && a->flags == b->flags; } -static Cuckoo -lumiera_filedescriptor_registry_new (void) +void +lumiera_filedescriptor_registry_init (void) { NOBUG_INIT_FLAG (filedescriptor); + TRACE (filedescriptor); - Cuckoo registry = cuckoo_new (h1, h2, h3, cmp, - sizeof (LumieraFiledescriptor), - 3); + registry = cuckoo_new (h1, h2, h3, cmp, + sizeof (LumieraFiledescriptor), + 3); if (!registry) LUMIERA_DIE (NO_MEMORY); - - return registry; } -static void -lumiera_filedescriptor_registry_delete (void) +void +lumiera_filedescriptor_registry_destroy (void) { + TRACE (filedescriptor); + REQUIRE (!cuckoo_nelements (registry)); cuckoo_free (registry); } + /** * Find existing filedescriptor or create one * @param descriptor strong reference to be initialized with the filedescriptor, @@ -106,18 +108,13 @@ LumieraFiledescriptor lumiera_filedescriptor_acquire (const char* name, int flags) { TRACE (filedescriptor); + REQUIRE (registry, "not initialized"); UNCHECKED; lumiera_mutexacquirer registry_lock; lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); - if (!registry) - { - registry = lumiera_filedescriptor_registry_new (); - atexit (lumiera_filedescriptor_registry_delete); - } - lumiera_filedescriptor fdesc; - fdesc.flags = flags; + fdesc.flags = flags&LUMIERA_FILE_MASK; for (int retry = 0; !retry; ++retry) { @@ -216,7 +213,10 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template) self->refcount = 1; self->handle = 0; - RESOURCE_ANNOUNCE (filedescriptor, "mutex", "filedescriptor", self, self->rh); + const char* type = "mutex"; + const char* name = "filedescriptor"; + + RESOURCE_ANNOUNCE (filedescriptor, type, name, self, self->rh); return self; } diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 48d0cd21c..817e8bc13 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -52,6 +52,11 @@ struct lumiera_filedescriptor_struct RESOURCE_HANDLE (rh); }; +void +lumiera_filedescriptor_registry_init (void); + +void +lumiera_filedescriptor_registry_destroy (void); LumieraFiledescriptor lumiera_filedescriptor_acquire (const char* name, int flags); @@ -65,5 +70,4 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template); void lumiera_filedescriptor_delete (LumieraFiledescriptor self); - #endif From 232afe77e9aefb10cf9832aa678aa2e4b11f620a Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 7 Apr 2008 06:47:57 +0200 Subject: [PATCH 49/59] Filehandling * add 'backend' for backend global initialization/destruction and common functions. * WIP: 'file' acts as interface to named files, provides posix filehandles with and acquire/release function pair. * 'filehandle' manages posix filehandles internally, refcounted. * WIP: 'filehandlecache' keeps unused filehandles in a MRU cache * WIP: test for filehandle management * improved filedescriptor test --- src/backend/backend.c | 53 ++++++++++++++ src/backend/backend.h | 36 ++++++++++ src/backend/file.c | 53 ++++++++++++-- src/backend/file.h | 17 +++-- src/backend/filedescriptor.c | 4 ++ src/backend/filehandle.c | 70 ++++++++++++++++++ src/backend/filehandle.h | 16 +++-- src/backend/filehandlecache.c | 102 +++++++++++++++++++++------ src/backend/filehandlecache.h | 41 ++++++----- tests/20filehandle.tests | 7 ++ tests/Makefile.am | 8 +-- tests/backend/test-filedescriptors.c | 10 ++- tests/backend/test-filehandles.c | 58 +++++++++++++++ 13 files changed, 415 insertions(+), 60 deletions(-) create mode 100644 src/backend/backend.c create mode 100644 src/backend/backend.h create mode 100644 src/backend/filehandle.c create mode 100644 tests/20filehandle.tests create mode 100644 tests/backend/test-filehandles.c diff --git a/src/backend/backend.c b/src/backend/backend.c new file mode 100644 index 000000000..024f7e6f8 --- /dev/null +++ b/src/backend/backend.c @@ -0,0 +1,53 @@ +/* + backend - common lumiera backend things + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "backend/backend.h" +#include "backend/filehandlecache.h" +#include "backend/filedescriptor.h" + +//NOBUG_DEFINE_FLAG_PARENT (backend, lumiera); TODO +NOBUG_DEFINE_FLAG (backend); +NOBUG_DEFINE_FLAG_PARENT (file_all, backend); + +int +lumiera_backend_init (void) +{ + NOBUG_INIT_FLAG (backend); + NOBUG_INIT_FLAG (file_all); + NOBUG_INIT_FLAG (file); + TRACE (backend); + lumiera_filedescriptor_registry_init (); + + int max_entries = 10; TODO("determine by sysconf (_SC_OPEN_MAX) minus some (big) safety margin " + "add some override to run tests with few filehandles"); + + lumiera_filehandlecache_new (max_entries); + + return 0; +} + +void +lumiera_backend_destroy (void) +{ + TRACE (backend); + lumiera_filedescriptor_registry_destroy (); + lumiera_filehandlecache_delete (); +} diff --git a/src/backend/backend.h b/src/backend/backend.h new file mode 100644 index 000000000..bd4c86815 --- /dev/null +++ b/src/backend/backend.h @@ -0,0 +1,36 @@ +/* + backend - common lumiera backend things + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#ifndef LUMIERA_BACKEND_H +#define LUMIERA_BACKEND_H + +#include + +NOBUG_DECLARE_FLAG (backend); +NOBUG_DECLARE_FLAG (file_all); +NOBUG_DECLARE_FLAG (file); + +int +lumiera_backend_init (void); + +void +lumiera_backend_destroy (void); + +#endif diff --git a/src/backend/file.c b/src/backend/file.c index 7a4aa5050..e9cd208dc 100644 --- a/src/backend/file.c +++ b/src/backend/file.c @@ -23,17 +23,18 @@ #include "lib/safeclib.h" #include "backend/file.h" +#include "backend/filehandlecache.h" #include "backend/filedescriptor.h" #include - -NOBUG_DEFINE_FLAG (lumiera_file); +NOBUG_DEFINE_FLAG_PARENT (file, file_all); LumieraFile lumiera_file_init (LumieraFile self, const char* name, int flags) { - if (!lumiera_filedescriptor_acquire (&self->descriptor, name, flags)) + TRACE (file); + if (!(self->descriptor = lumiera_filedescriptor_acquire (name, flags))) return NULL; self->name = lumiera_strndup (name, PATH_MAX); @@ -43,7 +44,8 @@ lumiera_file_init (LumieraFile self, const char* name, int flags) LumieraFile lumiera_file_destroy (LumieraFile self) { - lumiera_reference_destroy (&self->descriptor); + TRACE (file); + lumiera_filedescriptor_release (self->descriptor); free ((void*)self->name); return self; } @@ -52,6 +54,7 @@ lumiera_file_destroy (LumieraFile self) LumieraFile lumiera_file_new (const char* name, int flags) { + TRACE (file); LumieraFile self = lumiera_malloc (sizeof (lumiera_file)); return lumiera_file_init (self, name, flags); } @@ -59,6 +62,48 @@ lumiera_file_new (const char* name, int flags) void lumiera_file_delete (LumieraFile self) { + TRACE (file); free (lumiera_file_destroy (self)); } + +int +lumiera_file_handle_acquire (LumieraFile self) +{ + TRACE (file); + REQUIRE (self); + REQUIRE (self->descriptor); + REQUIRE (fhcache); + + LUMIERA_MUTEX_SECTION (file, self->descriptor->rh, &self->descriptor->lock) + { + if (!self->descriptor->handle) + /* no handle yet, get a new one */ + lumiera_filehandlecache_handle_acquire (fhcache, self->descriptor); + else + lumiera_filehandlecache_checkout (fhcache, self->descriptor->handle); + + if (self->descriptor->handle->fd == -1) + { + TODO ("stat handling/update"); + self->descriptor->handle->fd = open (self->name, self->descriptor->flags); + if (self->descriptor->handle->fd == -1) + { + LUMIERA_ERROR_SET (file, ERRNO); + } + } + } + + return self->descriptor->handle->fd; +} + +void +lumiera_file_handle_release (LumieraFile self) +{ + TRACE (file); + + LUMIERA_MUTEX_SECTION (file, self->descriptor->rh, &self->descriptor->lock) + { + lumiera_filehandlecache_checkin (fhcache, self->descriptor->handle); + } +} diff --git a/src/backend/file.h b/src/backend/file.h index 5018d321a..60c35447c 100644 --- a/src/backend/file.h +++ b/src/backend/file.h @@ -26,7 +26,8 @@ #include "lib/mutex.h" #include "lib/llist.h" #include "lib/error.h" -#include "lib/references.h" + +NOBUG_DECLARE_FLAG (file); /** * @file File management @@ -47,18 +48,18 @@ typedef lumiera_file* LumieraFile; #include "backend/filedescriptor.h" - -NOBUG_DECLARE_FLAG (lumiera_file); - #define LUMIERA_FILE_READONLY (O_RDONLY | O_LARGEFILE | O_NOATIME) #define LUMIERA_FILE_READWRITE (O_RDWR | O_LARGEFILE | O_NOATIME) #define LUMIERA_FILE_CREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_EXCL) #define LUMIERA_FILE_RECREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_TRUNC) +/* creat and excl flags will be masked out for descriptor lookup */ +#define LUMIERA_FILE_MASK ~(O_CREAT | O_EXCL) + struct lumiera_file_struct { const char* name; - struct lumiera_reference_struct descriptor; + LumieraFiledescriptor descriptor; }; /** @@ -86,5 +87,11 @@ void lumiera_file_delete (LumieraFile self); +int +lumiera_file_handle_acquire (LumieraFile self); + +void +lumiera_file_handle_release (LumieraFile self); + #endif diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index 0a3d6e048..be166d9c7 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -23,7 +23,10 @@ #include "lib/safeclib.h" #include "lib/cuckoo.h" +#include "backend/file.h" #include "backend/filedescriptor.h" +#include "backend/filehandle.h" +#include "backend/filehandlecache.h" #include #include @@ -141,6 +144,7 @@ lumiera_filedescriptor_acquire (const char* name, int flags) } int fd; + INFO (filedescriptor, "try creating file: %s", name); fd = creat (name, 0777); if (fd == -1) { diff --git a/src/backend/filehandle.c b/src/backend/filehandle.c new file mode 100644 index 000000000..60d54cf82 --- /dev/null +++ b/src/backend/filehandle.c @@ -0,0 +1,70 @@ +/* + filehandle - filehandle management and caching + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#include "lib/llist.h" +#include "lib/safeclib.h" + +#include "backend/file.h" +#include "backend/filehandle.h" +#include "backend/filedescriptor.h" + +#include + +NOBUG_DEFINE_FLAG_PARENT (filehandle, file_all); + +LumieraFilehandle +lumiera_filehandle_new () +{ + LumieraFilehandle self = lumiera_malloc (sizeof (lumiera_filehandle)); + TRACE (filehandle, "%p", self); + + llist_init (&self->cachenode); + self->fd = -1; + self->use_cnt = 1; + self->descriptor = NULL; + + return self; +} + + +void* +lumiera_filehandle_destroy_node (LList node) +{ + TRACE (filehandle); + REQUIRE (llist_is_empty (node)); + LumieraFilehandle self = (LumieraFilehandle)node; + REQUIRE (self->use_cnt == 0); + + if (self->fd >= 0) + close (self->fd); + self->fd = -1; + self->descriptor = NULL; + return self; +} + + +int +lumiera_filehandle_get (LumieraFilehandle self) +{ + REQUIRE (self->descriptor); + return -1; +} + diff --git a/src/backend/filehandle.h b/src/backend/filehandle.h index 00b4737c9..54e4d6936 100644 --- a/src/backend/filehandle.h +++ b/src/backend/filehandle.h @@ -24,6 +24,11 @@ #include "lib/error.h" #include "lib/llist.h" +typedef struct lumiera_filehandle_struct lumiera_filehandle; +typedef lumiera_filehandle* LumieraFilehandle; + +#include "backend/filedescriptor.h" + /** * @file Filehandles */ @@ -35,16 +40,17 @@ struct lumiera_filehandle_struct { llist cachenode; - int handle; - //LumieraFile file; - //lumiera_mutex lock; + int fd; + unsigned use_cnt; + LumieraFiledescriptor descriptor; }; -typedef struct lumiera_filehandle_struct lumiera_filehandle; -typedef lumiera_filehandle* LumieraFilehandle; LumieraFilehandle lumiera_filehandle_new (); +int +lumiera_filehandle_get (LumieraFilehandle self); + void* lumiera_filehandle_destroy_node (LList node); diff --git a/src/backend/filehandlecache.c b/src/backend/filehandlecache.c index 98ebdddc9..c629f765b 100644 --- a/src/backend/filehandlecache.c +++ b/src/backend/filehandlecache.c @@ -19,41 +19,60 @@ Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ +#include "lib/safeclib.h" + +#include "backend/file.h" #include "backend/filehandlecache.h" -NOBUG_DEFINE_FLAG (lumiera_filehandlecache); +NOBUG_DEFINE_FLAG_PARENT (filehandlecache, file_all); /* errors */ LUMIERA_ERROR_DEFINE (FILEHANDLECACHE_NOHANDLE, "No filehandle available"); -/**/ +LumieraFilehandlecache fhcache = NULL; -LumieraFilehandlecache -lumiera_filehandlecache_init (LumieraFilehandlecache self, int max_entries) + +void +lumiera_filehandlecache_new (int max_entries) { - lumiera_mrucache_init (&self->cache, lumiera_filehandle_destroy_node); - self->available =max_entries; - lumiera_mutex_init (&self->lock); - return self; + if (!fhcache) + { + NOBUG_INIT_FLAG (filehandlecache); + fhcache = lumiera_malloc (sizeof (lumiera_filehandlecache)); + lumiera_mrucache_init (&fhcache->cache, lumiera_filehandle_destroy_node); + fhcache->available = max_entries; + fhcache->checked_out = 0; + lumiera_mutex_init (&fhcache->lock); + RESOURCE_ANNOUNCE (filehandlecache, "mutex", "filehandlecache", fhcache, fhcache->rh); + } + else + NOTREACHED; } -LumieraFilehandlecache -lumiera_filehandlecache_destroy (LumieraFilehandlecache self) + +void +lumiera_filehandlecache_delete (void) { - self->available += self->cache.cached; - lumiera_mrucache_destroy (&self->cache); - lumiera_mutex_destroy (&self->lock); - return self; + if (fhcache) + { + REQUIRE (!fhcache->checked_out, "Filehandles in use at shutdown"); + RESOURCE_FORGET (filehandlecache, fhcache->rh); + lumiera_mrucache_destroy (&fhcache->cache); + lumiera_mutex_destroy (&fhcache->lock); + free (fhcache); + fhcache = NULL; + } } LumieraFilehandle -lumiera_filehandlecache_filehandle_acquire (LumieraFilehandlecache self) +lumiera_filehandlecache_handle_acquire (LumieraFilehandlecache self, LumieraFiledescriptor desc) { - LumieraFilehandle ret; - LUMIERA_MUTEX_SECTION (&self->lock) + TRACE (filehandlecache); + LumieraFilehandle ret = NULL; + LUMIERA_MUTEX_SECTION (filehandlecache, self->rh, &self->lock) { if (self->available <= 0 && self->cache.cached) { @@ -65,19 +84,58 @@ lumiera_filehandlecache_filehandle_acquire (LumieraFilehandlecache self) } else { - /* allocate new filehandle if we are below the limit or no cached handles where available (overallocating) */ + /* allocate new filehandle if we are below the limit or no cached handles are available (overallocating) */ ret = lumiera_filehandle_new (); if (!ret) - LUMIERA_ERROR_SET (lumiera_filehandlecache, FILEHANDLECACHE_NOHANDLE); + LUMIERA_ERROR_SET (filehandlecache, FILEHANDLECACHE_NOHANDLE); else --self->available; } + ret->use_cnt = 1; + ret->descriptor = desc; + desc->handle = ret; + ++self->checked_out; } return ret; } -void -lumiera_filehandlecache_add (LumieraFilehandlecache self, LumieraFilehandle handle) + +LumieraFilehandle +lumiera_filehandlecache_checkout (LumieraFilehandlecache self, LumieraFilehandle handle) { - UNIMPLEMENTED (""); + REQUIRE (self); + REQUIRE (handle); + /* This function is called with the associated descriptor locked, nothing can modify 'handle' */ + if (!handle->use_cnt) + { + /* lock cache and checkout */ + LUMIERA_MUTEX_SECTION (filehandlecache, self->rh, &self->lock) + { + lumiera_mrucache_checkout (&self->cache, &handle->cachenode); + } + } + ++handle->use_cnt; + ++self->checked_out; + + return handle; } + +void +lumiera_filehandlecache_checkin (LumieraFilehandlecache self, LumieraFilehandle handle) +{ + REQUIRE (self); + REQUIRE (handle); + REQUIRE (handle->use_cnt); + + /* This function is called with the associated descriptor locked, nothing can modify 'self' */ + if (!--handle->use_cnt) + { + /* lock cache and checin */ + LUMIERA_MUTEX_SECTION (filehandlecache, self->rh, &self->lock) + { + --self->checked_out; + lumiera_mrucache_checkin (&self->cache, &handle->cachenode); + } + } +} + diff --git a/src/backend/filehandlecache.h b/src/backend/filehandlecache.h index 40386defe..8d8de44cb 100644 --- a/src/backend/filehandlecache.h +++ b/src/backend/filehandlecache.h @@ -25,6 +25,9 @@ #include "lib/mrucache.h" #include "lib/mutex.h" +typedef struct lumiera_filehandlecache_struct lumiera_filehandlecache; +typedef lumiera_filehandlecache* LumieraFilehandlecache; + #include "backend/filehandle.h" /** @@ -43,55 +46,55 @@ struct lumiera_filehandlecache_struct { lumiera_mrucache cache; int available; + int checked_out; lumiera_mutex lock; + RESOURCE_HANDLE (rh); }; -typedef struct lumiera_filehandlecache_struct lumiera_filehandlecache; -typedef lumiera_filehandlecache* LumieraFilehandlecache; +extern LumieraFilehandlecache fhcache; /** - * Initializes a filehandle cache. - * @param self pointer to the cache to be initialized + * Initializes the filehandle cache. * @param max_entries number how much filehandles shall be managed - * @return self as given * The number of elements the cache can hold is static and should be - * determined by sysconf (_SC_OPEN_MAX) minus some safety margin. + * determined by sysconf (_SC_OPEN_MAX) minus some (big) safety margin. */ -LumieraFilehandlecache -lumiera_filehandlecache_init (LumieraFilehandlecache self, int max_entries); +void +lumiera_filehandlecache_new (int max_entries); /** - * Destroy a filehandle cache. - * @param self the cache to be destroyed - * @return self as given to the now uninitialized! cache + * Delete the filehandle cache. * No filehandles in the cache must be locked, this would be a fatal error. * The handles are closed automatically. */ -LumieraFilehandlecache -lumiera_filehandlecache_destroy (LumieraFilehandlecache self); +void +lumiera_filehandlecache_delete (void); /** * Get a fresh filehandle. * @param self pointer to the cache * @return the new filehandle - * one must call lumiera_mutexacquirer_unlock (&lock) to free the lock on the handle when done */ LumieraFilehandle -lumiera_filehandlecache_filehandle_acquire (LumieraFilehandlecache self); +lumiera_filehandlecache_handle_acquire (LumieraFilehandlecache self, LumieraFiledescriptor desc); /** - * Add filehande back to cache. + * Add filehande back to cache, the filehandle becomes subject of aging. * @param self pointer to the cache * @param handle filehandle to be put back */ void -lumiera_filehandlecache_add (LumieraFilehandlecache self, LumieraFilehandle handle); +lumiera_filehandlecache_add_filehandle (LumieraFilehandlecache self, LumieraFilehandle handle); /** - * + * Remove a filehandle from cache aging + * Filehandles which are subject of cache aging must be checked out before they can be used + * */ LumieraFilehandle -lumiera_filehandlecache_remove (LumieraFilehandlecache self, int fd); +lumiera_filehandlecache_checkout (LumieraFilehandlecache self, LumieraFilehandle handle); +void +lumiera_filehandlecache_checkin (LumieraFilehandlecache self, LumieraFilehandle handle); #endif diff --git a/tests/20filehandle.tests b/tests/20filehandle.tests new file mode 100644 index 000000000..068304f54 --- /dev/null +++ b/tests/20filehandle.tests @@ -0,0 +1,7 @@ + +TESTING "Filehandle management" ./test-filehandles + + +TEST "basic filehandle usage" basic < + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ +#define _GNU_SOURCE + +#include "lib/llist.h" + +#include "backend/backend.h" +#include "backend/filehandlecache.h" + +#include "tests/test.h" + +TESTS_BEGIN + +TEST ("basic") +{ + lumiera_backend_init (); + LumieraFile file = lumiera_file_new (",tmp_testfile", LUMIERA_FILE_CREATE); + + /* get the filehandle */ + int fd = lumiera_file_handle_acquire (file); + + /* we now 'own'it and can use it */ + ENSURE (fd > -1); + printf ("got filehandle #%d\n", fd); + + /* put it into aging, can't use anymore */ + lumiera_file_handle_release (file); + + lumiera_file_delete (file); + lumiera_backend_destroy (); +} + + +TEST ("more") +{ + lumiera_backend_init (); + lumiera_backend_destroy (); +} + +TESTS_END From dd63a0e1902a8a49f3b730b3729046e42097abdd Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Tue, 8 Apr 2008 07:46:25 +0200 Subject: [PATCH 50/59] filedescriptor cleanup and fixes basically working now, flag masking is not complete yet. --- src/backend/filedescriptor.c | 58 ++++++++++++++------------------ tests/20filehandle.tests | 6 ++++ tests/backend/test-filehandles.c | 31 +++++++++++++++++ 3 files changed, 63 insertions(+), 32 deletions(-) diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index be166d9c7..0df739670 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -110,54 +110,48 @@ lumiera_filedescriptor_registry_destroy (void) LumieraFiledescriptor lumiera_filedescriptor_acquire (const char* name, int flags) { - TRACE (filedescriptor); + TRACE (filedescriptor, "%s", name); REQUIRE (registry, "not initialized"); - UNCHECKED; + lumiera_mutexacquirer registry_lock; lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); lumiera_filedescriptor fdesc; fdesc.flags = flags&LUMIERA_FILE_MASK; - for (int retry = 0; !retry; ++retry) + if (stat (name, &fdesc.stat) != 0) { - if (stat (name, &fdesc.stat) == 0) - break; - else + if (errno == ENOENT && flags&O_CREAT) { - if (errno == ENOENT && flags&O_CREAT) + char* dir = lumiera_tmpbuf_strndup (name, PATH_MAX); + char* slash = dir; + while ((slash = strchr (slash+1, '/'))) { - INFO (filedescriptor, "try creating file: %s %d", name, PATH_MAX); - char* dir = lumiera_tmpbuf_strndup (name, PATH_MAX); - char* slash = dir; - - while ((slash = strchr (slash+1, '/'))) - { - *slash = '\0'; - INFO (filedescriptor, "try creating dir: %s", dir); - if (mkdir (dir, 0777) == -1) - { - LUMIERA_ERROR_SET (filedescriptor, ERRNO); - goto efile; - } - *slash = '/'; - } - - int fd; - INFO (filedescriptor, "try creating file: %s", name); - fd = creat (name, 0777); - if (fd == -1) + *slash = '\0'; + INFO (filedescriptor, "try creating dir: %s", dir); + if (mkdir (dir, 0777) == -1 && errno != EEXIST) { LUMIERA_ERROR_SET (filedescriptor, ERRNO); goto efile; } - close (fd); - continue; /* will now retry the fstat once */ + *slash = '/'; + } + int fd; + INFO (filedescriptor, "try creating file: %s", name); + fd = creat (name, 0777); + if (fd == -1) + { + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; + } + close (fd); + if (stat (name, &fdesc.stat) != 0) + { + /* finally, no luck */ + LUMIERA_ERROR_SET (filedescriptor, ERRNO); + goto efile; } } - /* finally, no luck */ - LUMIERA_ERROR_SET (filedescriptor, ERRNO); - goto efile; } /* lookup/create descriptor */ diff --git a/tests/20filehandle.tests b/tests/20filehandle.tests index 068304f54..0365bbffc 100644 --- a/tests/20filehandle.tests +++ b/tests/20filehandle.tests @@ -5,3 +5,9 @@ TESTING "Filehandle management" ./test-filehandles TEST "basic filehandle usage" basic < -1); + printf ("got filehandle #%d\n", fds[i]); + } + + /* put them into aging, can't use anymore */ + for (int i=0; i<100; ++i) + { + lumiera_file_handle_release (files[i]); + } + + /* cleanup */ + for (int i=0; i<100; ++i) + { + lumiera_file_delete (files[i]); + } + lumiera_backend_destroy (); } From 8311316a9210ff719c1c061d5899c07cfff76447 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 9 Apr 2008 07:56:39 +0200 Subject: [PATCH 51/59] Doxydocing for the filehandling stuff --- src/backend/file.h | 34 +++++++++++++++++++++++++----- src/backend/filedescriptor.c | 25 +++++++--------------- src/backend/filedescriptor.h | 39 +++++++++++++++++++++++++++++++++++ src/backend/filehandle.h | 15 ++++++++++++-- src/backend/filehandlecache.h | 13 +++++++++--- 5 files changed, 98 insertions(+), 28 deletions(-) diff --git a/src/backend/file.h b/src/backend/file.h index 60c35447c..9e774b94d 100644 --- a/src/backend/file.h +++ b/src/backend/file.h @@ -63,33 +63,57 @@ struct lumiera_file_struct }; /** - * + * Initialize a file structure. + * @param self pointer to the file structure + * @param name filename + * @param flags open flags + * @return self */ LumieraFile lumiera_file_init (LumieraFile self, const char* name, int flags); /** - * + * Destroy a file structure. + * frees all associated resources, releases the filedescriptor etc. + * @param self file structure to be destroyed + * @return self */ LumieraFile lumiera_file_destroy (LumieraFile self); /** - * + * Allocate a new file structure. + * @param name filename + * @param flags open flags + * @return new file structure */ LumieraFile lumiera_file_new (const char* name, int flags); /** - * + * Frees a file structure. + * @param self file structure to be freed */ void lumiera_file_delete (LumieraFile self); - +/** + * Get a POSIX filehandle for a file. + * Filehandles are opened on demand and must be acquired for use. + * Using filehandles is refcounted and might be nested. + * After using them they must be released which puts them back into filehandlecache aging. + * @param self file structure + * @return POSIX filehandle or -1 on error, check lumiera_error() to retrieve the errorcode + * Currently only LUMIERA_ERROR_ERRNO will be raised but this might change in future. + * Opening files can fail for many reasons and at any time! + */ int lumiera_file_handle_acquire (LumieraFile self); +/** + * Put filehandle back into cache aging. + * @param self file which handle to be released + */ void lumiera_file_handle_release (LumieraFile self); diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index 0df739670..5a513a7b4 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -82,6 +82,7 @@ lumiera_filedescriptor_registry_init (void) { NOBUG_INIT_FLAG (filedescriptor); TRACE (filedescriptor); + REQUIRE (!registry); registry = cuckoo_new (h1, h2, h3, cmp, sizeof (LumieraFiledescriptor), @@ -95,18 +96,12 @@ lumiera_filedescriptor_registry_destroy (void) { TRACE (filedescriptor); REQUIRE (!cuckoo_nelements (registry)); - cuckoo_free (registry); + if (registry) + cuckoo_free (registry); + registry = NULL; } -/** - * Find existing filedescriptor or create one - * @param descriptor strong reference to be initialized with the filedescriptor, - * a filedescriptor will be deleted when its last string reference gets deleted. - * @param name name of the file - * @param flags opening flags for the filedescriptor - * @return descriptor on success or NULL on error - */ LumieraFiledescriptor lumiera_filedescriptor_acquire (const char* name, int flags) { @@ -184,6 +179,7 @@ lumiera_filedescriptor_acquire (const char* name, int flags) return NULL; } + void lumiera_filedescriptor_release (LumieraFiledescriptor self) { @@ -192,11 +188,7 @@ lumiera_filedescriptor_release (LumieraFiledescriptor self) lumiera_filedescriptor_delete (self); } -/** - * Allocate a new filedescriptor cloned from a template - * @param template the source filedescriptor - * @return the constrccted filedescriptor - */ + LumieraFiledescriptor lumiera_filedescriptor_new (LumieraFiledescriptor template) { @@ -219,10 +211,7 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template) return self; } -/** - * delete a filedescriptor and free its resources - * @param descriptor filedescriptor to be freed - */ + void lumiera_filedescriptor_delete (LumieraFiledescriptor self) { diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 817e8bc13..8ae19a04d 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -36,6 +36,12 @@ typedef lumiera_filedescriptor* LumieraFiledescriptor; #include "backend/filehandle.h" #include "backend/file.h" +/** + * @file Filedescriptors + * Filedescriptors are the underlying working horse in accessing files. + */ + + struct lumiera_filedescriptor_struct { @@ -52,21 +58,54 @@ struct lumiera_filedescriptor_struct RESOURCE_HANDLE (rh); }; +/** + * Initialize the global filedescriptor registry. + * All filedescriptors are looked up by dev/ino/open-flags in a hashtable. Opening hardlinked + * files will be targeted to the same filedescriptor. + * This function never fails but dies on error. + */ void lumiera_filedescriptor_registry_init (void); +/** + * Destroy and free the global filedescriptor registry. + * Never fails. + */ void lumiera_filedescriptor_registry_destroy (void); + +/** + * Find existing filedescriptor or create one + * @param name name of the file + * @param flags opening flags for the filedescriptor + * @return descriptor on success or NULL on error + */ LumieraFiledescriptor lumiera_filedescriptor_acquire (const char* name, int flags); + +/** + * Release a filedescriptor. + * @param self filedescriptor to be released + */ void lumiera_filedescriptor_release (LumieraFiledescriptor self); + +/** + * Allocate a new filedescriptor cloned from a template + * @param template the source filedescriptor + * @return the constrccted filedescriptor + */ LumieraFiledescriptor lumiera_filedescriptor_new (LumieraFiledescriptor template); +/** + * Delete a filedescriptor + * Called whenever its reference count drops to zero. + * @param self the filedescriptor to be deleted + */ void lumiera_filedescriptor_delete (LumieraFiledescriptor self); diff --git a/src/backend/filehandle.h b/src/backend/filehandle.h index 54e4d6936..47fd0f3a7 100644 --- a/src/backend/filehandle.h +++ b/src/backend/filehandle.h @@ -31,6 +31,9 @@ typedef lumiera_filehandle* LumieraFilehandle; /** * @file Filehandles + * Filehandles manage the underlying POSIX filehandle for a filedescriptor. + * Since we want to support handling of more files than POSIX filehandles are available on a common system + * the filehandles are opened, cached and closed on demand, see 'filehandlecache'. */ @@ -45,12 +48,20 @@ struct lumiera_filehandle_struct LumieraFiledescriptor descriptor; }; +/** + * Allocate a new filehandle structure. + * @return new filehandle structure + */ LumieraFilehandle lumiera_filehandle_new (); -int -lumiera_filehandle_get (LumieraFilehandle self); +/** + * destroy the resources associated eith a filehandle structure. + * This function is used by the filehandlecache to recycle filehandle structs. + * @param node pointer to the cachenode member of a struct filehandle + * @return pointer to the start of the memory of the destroyed filehandle + */ void* lumiera_filehandle_destroy_node (LList node); diff --git a/src/backend/filehandlecache.h b/src/backend/filehandlecache.h index 8d8de44cb..fd6ddfd13 100644 --- a/src/backend/filehandlecache.h +++ b/src/backend/filehandlecache.h @@ -35,10 +35,10 @@ typedef lumiera_filehandlecache* LumieraFilehandlecache; * The number of filehandles a program can held open is usually limited, since we want to support * using a less limited number of files and closing/opening for each operation is expensive, we * provide a cache to keep the most frequent used files open and gracefully close/recycle unused filehandles. + * The filehandlecache defined here protects all operations on the cache with a mutex. */ - /* File handle cache manages file handles */ @@ -88,12 +88,19 @@ lumiera_filehandlecache_add_filehandle (LumieraFilehandlecache self, LumieraFile /** * Remove a filehandle from cache aging - * Filehandles which are subject of cache aging must be checked out before they can be used - * + * Filehandles which are subject of cache aging must be checked out before they can be used. + * @param self the filehandlecache + * @param handle the filehandle to be checked out */ LumieraFilehandle lumiera_filehandlecache_checkout (LumieraFilehandlecache self, LumieraFilehandle handle); +/** + * Put a filehandle into the cache + * Filehandles which are checked in are subject of cache aging and might get destroyed and reused. + * @param self the filehandlecache + * @param handle the filehandle to be checked in + */ void lumiera_filehandlecache_checkin (LumieraFilehandlecache self, LumieraFilehandle handle); From 4ed2fe9e7cf4503b4f94f5b9b2368c07b52acbee Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Wed, 9 Apr 2008 07:58:29 +0200 Subject: [PATCH 52/59] few more todos in the wiki, plan for a player in roadmap --- wiki/todo.html | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/wiki/todo.html b/wiki/todo.html index e66ec929f..763c8a80a 100644 --- a/wiki/todo.html +++ b/wiki/todo.html @@ -3350,6 +3350,11 @@ The idea here is to show and measure backend and scheduler performance backend to be done.
+
+
Use the backend, gmerlin/avdecoder and a Player widget to playback videos. This should be a small application which can open and playback many videos from on instance in parallel. Later on scrubbing for this videos might be supported.
+
+The idea here is to show and measure backend and scheduler performance and beeing a proof of concept for the design. It will run independent of the Proc layer but needs quite some work on the backend to be done.
+
From 00b9dff106d018c03a704e3b5db1e92467c25f3b Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 10 Apr 2008 02:40:34 +0200 Subject: [PATCH 53/59] Doxygen config update * set EXPORT_ALL=NO to leave out undcoumented entities. * Fix a lot @file directives --- src/backend/file.h | 3 ++- src/backend/filedescriptor.h | 4 +++- src/backend/filehandle.h | 3 ++- src/backend/filehandlecache.h | 3 ++- src/lib/condition.c | 3 ++- src/lib/condition.h | 3 ++- src/lib/cuckoo.h | 3 ++- src/lib/error.c | 3 ++- src/lib/error.h | 3 ++- src/lib/llist.h | 3 ++- src/lib/locking.h | 3 ++- src/lib/mrucache.h | 3 ++- src/lib/mutex.c | 3 ++- src/lib/mutex.h | 3 ++- src/lib/plugin.c | 3 ++- src/lib/plugin.h | 3 ++- src/lib/references.c | 3 ++- src/lib/references.h | 3 ++- src/lib/rwlock.c | 3 ++- src/lib/rwlock.h | 3 ++- src/lib/safeclib.c | 3 ++- src/lib/safeclib.h | 3 ++- 22 files changed, 45 insertions(+), 22 deletions(-) diff --git a/src/backend/file.h b/src/backend/file.h index 9e774b94d..175b93e9d 100644 --- a/src/backend/file.h +++ b/src/backend/file.h @@ -30,7 +30,8 @@ NOBUG_DECLARE_FLAG (file); /** - * @file File management + * @file + * File management * Handling Files is splitted into different classes: * 1. The 'lumiera_file' class which acts as interface to the outside for managing files. * 'lumiera_file' is addressed by the name of the file. Since files can have more than one name (hardlinks) diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 8ae19a04d..4789f9ce2 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -37,8 +37,10 @@ typedef lumiera_filedescriptor* LumieraFiledescriptor; #include "backend/file.h" /** - * @file Filedescriptors + * @file + * Filedescriptors. * Filedescriptors are the underlying working horse in accessing files. + * All information associated with managing a file is kept here. */ diff --git a/src/backend/filehandle.h b/src/backend/filehandle.h index 47fd0f3a7..e1b02b2e4 100644 --- a/src/backend/filehandle.h +++ b/src/backend/filehandle.h @@ -30,7 +30,8 @@ typedef lumiera_filehandle* LumieraFilehandle; #include "backend/filedescriptor.h" /** - * @file Filehandles + * @file + * Filehandles. * Filehandles manage the underlying POSIX filehandle for a filedescriptor. * Since we want to support handling of more files than POSIX filehandles are available on a common system * the filehandles are opened, cached and closed on demand, see 'filehandlecache'. diff --git a/src/backend/filehandlecache.h b/src/backend/filehandlecache.h index fd6ddfd13..25964f986 100644 --- a/src/backend/filehandlecache.h +++ b/src/backend/filehandlecache.h @@ -31,7 +31,8 @@ typedef lumiera_filehandlecache* LumieraFilehandlecache; #include "backend/filehandle.h" /** - * @file Filehandle management and caching + * @file + * Filehandle management and caching * The number of filehandles a program can held open is usually limited, since we want to support * using a less limited number of files and closing/opening for each operation is expensive, we * provide a cache to keep the most frequent used files open and gracefully close/recycle unused filehandles. diff --git a/src/lib/condition.c b/src/lib/condition.c index 1bfa2ef96..60e8b6f10 100644 --- a/src/lib/condition.c +++ b/src/lib/condition.c @@ -22,7 +22,8 @@ #include "lib/condition.h" /** - * @file Condition variables + * @file + * Condition variables */ LUMIERA_ERROR_DEFINE (CONDITION_DESTROY, "condition destroy failed"); diff --git a/src/lib/condition.h b/src/lib/condition.h index ffc00523c..dd92f326b 100644 --- a/src/lib/condition.h +++ b/src/lib/condition.h @@ -25,7 +25,8 @@ #include "lib/locking.h" /** - * @file Condition variables, header + * @file + * Condition variables, header */ LUMIERA_ERROR_DECLARE (CONDITION_DESTROY); diff --git a/src/lib/cuckoo.h b/src/lib/cuckoo.h index 63b6d260d..323b5cc75 100644 --- a/src/lib/cuckoo.h +++ b/src/lib/cuckoo.h @@ -22,7 +22,8 @@ #define CUCKOO_H /** - * @file Cuckoo hashing + * @file + * Cuckoo hashing * This hashing gives guaranteed O(1) lookup complexity and amortized O(1) insert and remove complexity. * Hash tables by default grow and shrink automatically. It is posible to preallocate entries and turn * automatic shrinking off taking out the memory management factors for insert and remove operations. diff --git a/src/lib/error.c b/src/lib/error.c index 81e24bad8..b2bc390b6 100644 --- a/src/lib/error.c +++ b/src/lib/error.c @@ -24,7 +24,8 @@ #include "lib/error.h" /** - * @file C Error handling in Lumiera. + * @file + * C Error handling in Lumiera. */ diff --git a/src/lib/error.h b/src/lib/error.h index e71c0cdfa..1eeb42dda 100644 --- a/src/lib/error.h +++ b/src/lib/error.h @@ -31,7 +31,8 @@ extern "C" { #include /** - * @file C Error handling in Lumiera, header. + * @file + * C Error handling in Lumiera, header. */ diff --git a/src/lib/llist.h b/src/lib/llist.h index efb3f5202..128a3e826 100644 --- a/src/lib/llist.h +++ b/src/lib/llist.h @@ -27,7 +27,8 @@ #include /** - * @file Intrusive cyclic double linked list + * @file + * Intrusive cyclic double linked list * There is only one node type which contains a forward and a backward pointer. In a empty initialized node, * this pointers point to the node itself. Note that these pointers can never ever become NULL. * This lists are used by using one node as 'root' node where its both pointers are the head/tail pointer to the actual list. diff --git a/src/lib/locking.h b/src/lib/locking.h index f62dd6c29..3c4d497d8 100644 --- a/src/lib/locking.h +++ b/src/lib/locking.h @@ -34,7 +34,8 @@ LUMIERA_ERROR_DECLARE (MUTEX_UNLOCK); LUMIERA_ERROR_DECLARE (MUTEX_DESTROY); /** - * @file Shared declarations for all locking primitives. + * @file + * Shared declarations for all locking primitives. */ /** diff --git a/src/lib/mrucache.h b/src/lib/mrucache.h index c81bd7599..08654754f 100644 --- a/src/lib/mrucache.h +++ b/src/lib/mrucache.h @@ -27,7 +27,8 @@ #include /** - * @file Most recent used cache + * @file + * Most recent used cache * Elements (addressed by a LList node) are either checked in the cache and thereby subject of aging * or checked out under control of the user. Most operations require that the chache is locked. This locking * must be done from elsewhere. diff --git a/src/lib/mutex.c b/src/lib/mutex.c index 009502e5a..b7c5f1844 100644 --- a/src/lib/mutex.c +++ b/src/lib/mutex.c @@ -22,7 +22,8 @@ #include "lib/mutex.h" /** - * @file Mutual exclusion locking. + * @file + * Mutual exclusion locking. */ LUMIERA_ERROR_DEFINE (MUTEX_LOCK, "Mutex locking failed"); diff --git a/src/lib/mutex.h b/src/lib/mutex.h index ca48ea592..f19058e7a 100644 --- a/src/lib/mutex.h +++ b/src/lib/mutex.h @@ -25,7 +25,8 @@ #include "lib/locking.h" /** - * @file Mutual exclusion locking, header. + * @file + * Mutual exclusion locking, header. */ #define LUMIERA_MUTEX_SECTION(flag, handle, mutex) \ RESOURCE_HANDLE (rh_##__LINE__##_); \ diff --git a/src/lib/plugin.c b/src/lib/plugin.c index 8345d9887..1f5e8576e 100644 --- a/src/lib/plugin.c +++ b/src/lib/plugin.c @@ -31,7 +31,8 @@ #include "safeclib.h" /** - * @file Plugin loader. + * @file + * Plugin loader. */ diff --git a/src/lib/plugin.h b/src/lib/plugin.h index f8916ae96..54dad6372 100644 --- a/src/lib/plugin.h +++ b/src/lib/plugin.h @@ -33,7 +33,8 @@ extern "C" { #include "error.h" /** - * @file Plugin loader, header. + * @file + * Plugin loader, header. */ diff --git a/src/lib/references.c b/src/lib/references.c index 3e9d5f260..925258377 100644 --- a/src/lib/references.c +++ b/src/lib/references.c @@ -22,7 +22,8 @@ #include "lib/safeclib.h" /** - * @file Strong and Weak references + * @file + * Strong and Weak references * Strong references keep some object alive while they existing * Weak references become invalidated when the referenced object gets destroyed * diff --git a/src/lib/references.h b/src/lib/references.h index 3e00457e7..6ad65d62f 100644 --- a/src/lib/references.h +++ b/src/lib/references.h @@ -25,7 +25,8 @@ #include /** - * @file Strong and Weak references, header. + * @file + * Strong and Weak references, header. */ diff --git a/src/lib/rwlock.c b/src/lib/rwlock.c index 092c7b54b..077ff9766 100644 --- a/src/lib/rwlock.c +++ b/src/lib/rwlock.c @@ -29,7 +29,8 @@ LUMIERA_ERROR_DEFINE(RWLOCK_RLOCK, "rlock"); LUMIERA_ERROR_DEFINE(RWLOCK_WLOCK, "wlock"); /** - * @file Read/write locks. + * @file + * Read/write locks. */ diff --git a/src/lib/rwlock.h b/src/lib/rwlock.h index 5e121fbcc..d58627dda 100644 --- a/src/lib/rwlock.h +++ b/src/lib/rwlock.h @@ -39,7 +39,8 @@ LUMIERA_ERROR_DECLARE(RWLOCK_RLOCK); LUMIERA_ERROR_DECLARE(RWLOCK_WLOCK); /** - * @file Read/write locks, header. + * @file + * Read/write locks, header. */ diff --git a/src/lib/safeclib.c b/src/lib/safeclib.c index 0e82e2dc0..f500ea71c 100644 --- a/src/lib/safeclib.c +++ b/src/lib/safeclib.c @@ -27,7 +27,8 @@ #include /** - * @file Portable and safe wrapers around some clib functions and some tools + * @file + * Portable and safe wrapers around some clib functions and some tools */ LUMIERA_ERROR_DEFINE (NO_MEMORY, "Out of Memory!"); diff --git a/src/lib/safeclib.h b/src/lib/safeclib.h index f486ed056..6fc532239 100644 --- a/src/lib/safeclib.h +++ b/src/lib/safeclib.h @@ -24,7 +24,8 @@ #include /** - * @file Portable and safe wrapers around some clib functions and some tools + * @file + * Portable and safe wrapers around some clib functions and some tools */ LUMIERA_ERROR_DECLARE(NO_MEMORY); From c4816c39fdc802a4db00e65dfab12077e18970c3 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 10 Apr 2008 06:03:21 +0200 Subject: [PATCH 54/59] Proper flag and stat handling for files Filedescriptors create files at construction, each later access becomes a reopen on the now existing file. file_handle_acquire() reopens the file to gain a handle when required and checks that the file is really the one we expected (ino and dev num) --- src/backend/file.c | 29 ++++++++++++++++++++++++++--- src/backend/file.h | 3 ++- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/src/backend/file.c b/src/backend/file.c index e9cd208dc..861f3ce21 100644 --- a/src/backend/file.c +++ b/src/backend/file.c @@ -27,9 +27,13 @@ #include "backend/filedescriptor.h" #include +#include NOBUG_DEFINE_FLAG_PARENT (file, file_all); +LUMIERA_ERROR_DEFINE(FILE_CHANGED, "File changed unexpected"); + + LumieraFile lumiera_file_init (LumieraFile self, const char* name, int flags) { @@ -85,12 +89,31 @@ lumiera_file_handle_acquire (LumieraFile self) if (self->descriptor->handle->fd == -1) { - TODO ("stat handling/update"); - self->descriptor->handle->fd = open (self->name, self->descriptor->flags); - if (self->descriptor->handle->fd == -1) + int fd; + fd = open (self->name, self->descriptor->flags & ~(O_EXCL|O_TRUNC|O_CREAT)); + if (fd == -1) { LUMIERA_ERROR_SET (file, ERRNO); + return -1; } + else + { + struct stat st; + if (fstat (fd, &st) == -1) + { + close (fd); + LUMIERA_ERROR_SET (file, ERRNO); + return -1; + } + else if (self->descriptor->stat.st_dev != st.st_dev || self->descriptor->stat.st_ino != st.st_ino) + { + close (fd); + /* Woops this is not the file we expected to use */ + LUMIERA_ERROR_SET (file, FILE_CHANGED); + return -1; + } + } + self->descriptor->handle->fd = fd; } } diff --git a/src/backend/file.h b/src/backend/file.h index 175b93e9d..71ca15751 100644 --- a/src/backend/file.h +++ b/src/backend/file.h @@ -29,6 +29,8 @@ NOBUG_DECLARE_FLAG (file); +LUMIERA_ERROR_DECLARE(FILE_CHANGED); + /** * @file * File management @@ -48,7 +50,6 @@ typedef lumiera_file* LumieraFile; #include "backend/filehandle.h" #include "backend/filedescriptor.h" - #define LUMIERA_FILE_READONLY (O_RDONLY | O_LARGEFILE | O_NOATIME) #define LUMIERA_FILE_READWRITE (O_RDWR | O_LARGEFILE | O_NOATIME) #define LUMIERA_FILE_CREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_EXCL) From 444b142d854ed51c727db86dd2fd280189e107df Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Thu, 10 Apr 2008 06:07:04 +0200 Subject: [PATCH 55/59] Correct File flag masking The hash lookup needs to ignore O_EXCL|O_TRUNC|O_CREAT as well as file reopening must not use this flags. --- src/backend/file.c | 2 +- src/backend/file.h | 2 +- src/backend/filedescriptor.c | 15 +++++++++------ src/backend/filedescriptor.h | 1 - 4 files changed, 11 insertions(+), 9 deletions(-) diff --git a/src/backend/file.c b/src/backend/file.c index 861f3ce21..e315537fc 100644 --- a/src/backend/file.c +++ b/src/backend/file.c @@ -90,7 +90,7 @@ lumiera_file_handle_acquire (LumieraFile self) if (self->descriptor->handle->fd == -1) { int fd; - fd = open (self->name, self->descriptor->flags & ~(O_EXCL|O_TRUNC|O_CREAT)); + fd = open (self->name, self->descriptor->flags & LUMIERA_FILE_MASK); if (fd == -1) { LUMIERA_ERROR_SET (file, ERRNO); diff --git a/src/backend/file.h b/src/backend/file.h index 71ca15751..bba12a167 100644 --- a/src/backend/file.h +++ b/src/backend/file.h @@ -56,7 +56,7 @@ typedef lumiera_file* LumieraFile; #define LUMIERA_FILE_RECREATE (O_RDWR | O_LARGEFILE | O_NOATIME | O_CREAT | O_TRUNC) /* creat and excl flags will be masked out for descriptor lookup */ -#define LUMIERA_FILE_MASK ~(O_CREAT | O_EXCL) +#define LUMIERA_FILE_MASK ~(O_CREAT | O_EXCL | O_TRUNC) struct lumiera_file_struct { diff --git a/src/backend/filedescriptor.c b/src/backend/filedescriptor.c index 5a513a7b4..b9608984f 100644 --- a/src/backend/filedescriptor.c +++ b/src/backend/filedescriptor.c @@ -51,21 +51,24 @@ static size_t h1 (const void* item, const uint32_t r) { const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; - return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>7)^r; + return i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK) + ^((i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK))>>7)^r; } static size_t h2 (const void* item, const uint32_t r) { const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; - return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>5)^r; + return i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK) + ^((i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK))>>5)^r; } static size_t h3 (const void* item, const uint32_t r) { const LumieraFiledescriptor i = *(const LumieraFiledescriptor*)item; - return i->stat.st_dev^i->stat.st_ino^i->flags^((i->stat.st_dev^i->stat.st_ino^i->flags)>>3)^r; + return i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK) + ^((i->stat.st_dev^i->stat.st_ino^(i->flags&LUMIERA_FILE_MASK))>>3)^r; } @@ -74,7 +77,8 @@ cmp (const void* keya, const void* keyb) { const LumieraFiledescriptor a = *(const LumieraFiledescriptor*)keya; const LumieraFiledescriptor b = *(const LumieraFiledescriptor*)keyb; - return a->stat.st_dev == b->stat.st_dev && a->stat.st_ino == b->stat.st_ino && a->flags == b->flags; + return a->stat.st_dev == b->stat.st_dev && a->stat.st_ino == b->stat.st_ino + && (a->flags&LUMIERA_FILE_MASK) == (b->flags&LUMIERA_FILE_MASK); } void @@ -112,7 +116,7 @@ lumiera_filedescriptor_acquire (const char* name, int flags) lumiera_mutexacquirer_init_mutex (®istry_lock, ®istry_mutex, LUMIERA_LOCKED); lumiera_filedescriptor fdesc; - fdesc.flags = flags&LUMIERA_FILE_MASK; + fdesc.flags = flags; if (stat (name, &fdesc.stat) != 0) { @@ -199,7 +203,6 @@ lumiera_filedescriptor_new (LumieraFiledescriptor template) self->flags = template->flags; lumiera_mutex_init (&self->lock); - self->reopened = 0; self->refcount = 1; self->handle = 0; diff --git a/src/backend/filedescriptor.h b/src/backend/filedescriptor.h index 4789f9ce2..528be41e8 100644 --- a/src/backend/filedescriptor.h +++ b/src/backend/filedescriptor.h @@ -50,7 +50,6 @@ struct lumiera_filedescriptor_struct struct stat stat; /* create after first open, maintained metadata, MUST BE FIRST! */ int flags; /* open flags, must be masked for reopen */ lumiera_mutex lock; /* locks operations on this file descriptor */ - unsigned reopened; /* count for reopens 0=not yet opened, 1=first, 2..=reopened */ unsigned refcount; /* reference counter, all users sans registry */ LumieraFilehandle handle; From 241327bcd423fb27a925a0a25b4195b11e1cdae9 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 2 May 2008 23:55:14 +0200 Subject: [PATCH 56/59] rename 'fhcache' global var to 'lumiera_fhcache' --- src/backend/file.c | 9 +++++---- src/backend/filehandlecache.c | 37 ++++++++++++++++------------------- src/backend/filehandlecache.h | 2 +- 3 files changed, 23 insertions(+), 25 deletions(-) diff --git a/src/backend/file.c b/src/backend/file.c index e315537fc..4a309ed3a 100644 --- a/src/backend/file.c +++ b/src/backend/file.c @@ -77,15 +77,15 @@ lumiera_file_handle_acquire (LumieraFile self) TRACE (file); REQUIRE (self); REQUIRE (self->descriptor); - REQUIRE (fhcache); + REQUIRE (lumiera_fhcache); LUMIERA_MUTEX_SECTION (file, self->descriptor->rh, &self->descriptor->lock) { if (!self->descriptor->handle) /* no handle yet, get a new one */ - lumiera_filehandlecache_handle_acquire (fhcache, self->descriptor); + lumiera_filehandlecache_handle_acquire (lumiera_fhcache, self->descriptor); else - lumiera_filehandlecache_checkout (fhcache, self->descriptor->handle); + lumiera_filehandlecache_checkout (lumiera_fhcache, self->descriptor->handle); if (self->descriptor->handle->fd == -1) { @@ -120,6 +120,7 @@ lumiera_file_handle_acquire (LumieraFile self) return self->descriptor->handle->fd; } + void lumiera_file_handle_release (LumieraFile self) { @@ -127,6 +128,6 @@ lumiera_file_handle_release (LumieraFile self) LUMIERA_MUTEX_SECTION (file, self->descriptor->rh, &self->descriptor->lock) { - lumiera_filehandlecache_checkin (fhcache, self->descriptor->handle); + lumiera_filehandlecache_checkin (lumiera_fhcache, self->descriptor->handle); } } diff --git a/src/backend/filehandlecache.c b/src/backend/filehandlecache.c index c629f765b..181807661 100644 --- a/src/backend/filehandlecache.c +++ b/src/backend/filehandlecache.c @@ -31,38 +31,35 @@ NOBUG_DEFINE_FLAG_PARENT (filehandlecache, file_all); LUMIERA_ERROR_DEFINE (FILEHANDLECACHE_NOHANDLE, "No filehandle available"); -LumieraFilehandlecache fhcache = NULL; +LumieraFilehandlecache lumiera_fhcache = NULL; void lumiera_filehandlecache_new (int max_entries) { - if (!fhcache) - { - NOBUG_INIT_FLAG (filehandlecache); - fhcache = lumiera_malloc (sizeof (lumiera_filehandlecache)); - lumiera_mrucache_init (&fhcache->cache, lumiera_filehandle_destroy_node); - fhcache->available = max_entries; - fhcache->checked_out = 0; - lumiera_mutex_init (&fhcache->lock); - RESOURCE_ANNOUNCE (filehandlecache, "mutex", "filehandlecache", fhcache, fhcache->rh); - } - else - NOTREACHED; + REQUIRE (!lumiera_fhcache, "Filehandlecache already initialized"); + + NOBUG_INIT_FLAG (filehandlecache); + lumiera_fhcache = lumiera_malloc (sizeof (lumiera_filehandlecache)); + lumiera_mrucache_init (&lumiera_fhcache->cache, lumiera_filehandle_destroy_node); + lumiera_fhcache->available = max_entries; + lumiera_fhcache->checked_out = 0; + lumiera_mutex_init (&lumiera_fhcache->lock); + RESOURCE_ANNOUNCE (filehandlecache, "mutex", "filehandlecache", lumiera_fhcache, lumiera_fhcache->rh); } void lumiera_filehandlecache_delete (void) { - if (fhcache) + if (lumiera_fhcache) { - REQUIRE (!fhcache->checked_out, "Filehandles in use at shutdown"); - RESOURCE_FORGET (filehandlecache, fhcache->rh); - lumiera_mrucache_destroy (&fhcache->cache); - lumiera_mutex_destroy (&fhcache->lock); - free (fhcache); - fhcache = NULL; + REQUIRE (!lumiera_fhcache->checked_out, "Filehandles in use at shutdown"); + RESOURCE_FORGET (filehandlecache, lumiera_fhcache->rh); + lumiera_mrucache_destroy (&lumiera_fhcache->cache); + lumiera_mutex_destroy (&lumiera_fhcache->lock); + free (lumiera_fhcache); + lumiera_fhcache = NULL; } } diff --git a/src/backend/filehandlecache.h b/src/backend/filehandlecache.h index 25964f986..37c2a416b 100644 --- a/src/backend/filehandlecache.h +++ b/src/backend/filehandlecache.h @@ -52,7 +52,7 @@ struct lumiera_filehandlecache_struct RESOURCE_HANDLE (rh); }; -extern LumieraFilehandlecache fhcache; +extern LumieraFilehandlecache lumiera_fhcache; /** * Initializes the filehandle cache. From 9ad62d0edc0e7f2bd9ba6f480ec1ea9c18d0f73f Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Fri, 2 May 2008 23:56:49 +0200 Subject: [PATCH 57/59] Fix: one must not return from a mutex section --- src/backend/file.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/backend/file.c b/src/backend/file.c index 4a309ed3a..9af608567 100644 --- a/src/backend/file.c +++ b/src/backend/file.c @@ -94,7 +94,6 @@ lumiera_file_handle_acquire (LumieraFile self) if (fd == -1) { LUMIERA_ERROR_SET (file, ERRNO); - return -1; } else { @@ -103,14 +102,12 @@ lumiera_file_handle_acquire (LumieraFile self) { close (fd); LUMIERA_ERROR_SET (file, ERRNO); - return -1; } else if (self->descriptor->stat.st_dev != st.st_dev || self->descriptor->stat.st_ino != st.st_ino) { close (fd); /* Woops this is not the file we expected to use */ LUMIERA_ERROR_SET (file, FILE_CHANGED); - return -1; } } self->descriptor->handle->fd = fd; From 64e549dc8125f923aab3d9e02d941f1ddca28dee Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Sat, 3 May 2008 16:32:27 +0200 Subject: [PATCH 58/59] headers for the memory mapping backend (WIP) --- src/backend/mmap.h | 88 ++++++++++++++++++++++++++ src/backend/mmapcache.h | 136 ++++++++++++++++++++++++++++++++++++++++ src/backend/mmapings.h | 118 ++++++++++++++++++++++++++++++++++ 3 files changed, 342 insertions(+) create mode 100644 src/backend/mmap.h create mode 100644 src/backend/mmapcache.h create mode 100644 src/backend/mmapings.h diff --git a/src/backend/mmap.h b/src/backend/mmap.h new file mode 100644 index 000000000..608e40133 --- /dev/null +++ b/src/backend/mmap.h @@ -0,0 +1,88 @@ +/* + mmap.h - memory mapped acces to files + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_MMAP_H +#define LUMIERA_MMAP_H + +#include "lib/llist.h" + + +typedef struct lumiera_mmap_struct lumiera_mmap; +typedef lumiera_mmap* LumieraMMap; + + +#include "backend/filedescriptor.h" + + +#include +#include + +NOBUG_DECLARE_FLAG (mmap); + +/** + * @file + * MMap objects cover a memory maped range in a file + */ + + + +/** + * mmaped area + * + */ +struct lumiera_mmap_struct +{ + /** used for the mrucache when checked in the cache OR for attaching owners when checked out **/ + llist cachenode; + + /** all mmaps of a file are chained in this list, used to find ranges **/ + llist searchnode; + + off_t start; + size_t size; + void* address; + + /** array with refcounters per page **/ + unsigned short* refmap; + + /** 0 when this mmap is in cache, else the count of attached owners **/ + unsigned use_cnt; + //RESOURCE_HANDLE (rh); +}; + + +LumieraMMap +lumiera_mmap_init (LumieraMMap self, LumieraFile file, LList acquirer, off_t start, size_t size, size_t chunksize); + +LumieraMMap +lumiera_mmap_new (LumieraFile file, LList acquirer, off_t start, size_t size, size_t chunksize); + +void* +lumiera_mmap_destroy_node (LList node); + +#endif +/* +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: +*/ diff --git a/src/backend/mmapcache.h b/src/backend/mmapcache.h new file mode 100644 index 000000000..ed2efcb43 --- /dev/null +++ b/src/backend/mmapcache.h @@ -0,0 +1,136 @@ +/* + mmapcache.h - handle aging of mmap objects + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_MMAPCACHE_H +#define LUMIERA_MMAPCACHE_H + +#include "lib/error.h" +#include "lib/mrucache.h" +#include "lib/mutex.h" + + +typedef struct lumiera_mmapcache_struct lumiera_mmapcache; +typedef lumiera_mmapcache* LumieraMMapcache; + +#include "backend/mmap.h" + +#include + +/** + * @file + * Mmapcache stores a MRU cache of all established mmaped memory regions which are currently not in use. + * The mmapcache also manages the upper limit about how much memory can be mmaped. + */ + +struct lumiera_mmapcache_struct +{ + lumiera_mrucache cache; + size_t limit; + size_t total; + size_t cached; + lumiera_mutex lock; + RESOURCE_HANDLE (rh); +}; + +extern LumieraMMapcache lumiera_mcache; + + +/** + * Initializes the mmapcache. + * @param limit the mmapcache will drop elements when the sum of all mmapings gives over limit + */ +void +lumiera_mmapcache_new (size_t limit); + +/** + * Delete the mmap cache. + * No mmaps in the cache must be locked, this would be a fatal error. + * The handles are closed automatically. + */ +void +lumiera_mmapcache_delete (void); + +/** + * Get a fresh mmap object. + * when mmaped_limit is reached, the oldest mmap object gets dropped else a new allocated object + * is returned + * @param self pointer to the cache + * @return the new mmap + */ +LumieraMMap +lumiera_mmapcache_mmap_acquire (LumieraMMapcache self); + + +/** + * Announce a new mmap object to the cache quotas. + * Update the statistics kept in the cache, + * the map object is still considered to be checked out + * @param self pointer to the cache + * @param map object to be announced + */ +void +lumiera_mmapcache_announce (LumieraMMapcache self, LumieraMMap map); + +/** + * Remove a mmap object from the cache quotas. + * Update the statistics kept in the cache, + * called by mmap's destructor only + * @param self pointer to the cache + * @param map object to be removed + */ +void +lumiera_mmapcache_forget (LumieraMMapcache self, LumieraMMap map); + +/** + * Destroy and free the nelem oldest elements. + * Used to free up resources and memory. + * @param self cache where to free elements. + * @return nelem-(numer of elements which got freed), that is 0 if all requested elements got freed + */ +int +lumiera_mmapcache_age (LumieraMMapcache self); + +/** + * Remove a mmap from cache aging + * Mmaps which are subject of cache aging must be checked out before they can be used. + * @param self the mmapcache + * @param handle the mmap to be checked out + */ +LumieraMMap +lumiera_mmapcache_checkout (LumieraMMapcache self, LumieraMMap handle); + +/** + * Put a mmap into the cache + * Mmaps which are checked in are subject of cache aging and might get destroyed and reused. + * @param self the mmapcache + * @param handle the mmap to be checked in + */ +void +lumiera_mmapcache_checkin (LumieraMMapcache self, LumieraMMap handle); + +#endif +/* +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: +*/ diff --git a/src/backend/mmapings.h b/src/backend/mmapings.h new file mode 100644 index 000000000..234ab280b --- /dev/null +++ b/src/backend/mmapings.h @@ -0,0 +1,118 @@ +/* + mmapings.h - manage ranges of mmaped areas on a filedescriptor + + Copyright (C) Lumiera.org + 2008, Christian Thaeter + + This program is free software; you can redistribute it and/or + modify it under the terms of the GNU General Public License as + published by the Free Software Foundation; either version 2 of the + License, or (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. +*/ + +#ifndef LUMIERA_MMAPINGS_H +#define LUMIERA_MMAPINGS_H + +#include "lib/mutex.h" +#include "lib/llist.h" + +typedef struct lumiera_mmapings_struct lumiera_mmapings; +typedef lumiera_mmapings* LumieraMMapings; + +#include "backend/filedescriptor.h" +#include "backend/mmap.h" + +#include + + +/** + * @file + * Keeps all mmaped areas + * + */ + +struct lumiera_mmapings_struct +{ + /** mmaped ranges are kept in an list sorted by the size of the mmaping, might be improved to a tree someday **/ + llist mmaps; + + /** sum of all mmaped areas, areas might be overlapping **/ + // size_t vsz; + + /** + * chunkssize is the smallest granularity which is used for mmapping files, it + * should reflect the intended file usage, that is 'pagesize' for small or non growing + * files and some MB for media files. Must be a 2's exponent of pagesize. + **/ + size_t chunksize; + + LumieraFiledescriptor descriptor; + lumiera_mutex lock; + RESOURCE_HANDLE (rh); +}; + +/** + * initialize mmapings container + * + */ +LumieraMMapings +lumiera_mmapings_init (LumieraMMapings self, LumieraFile file, size_t chunksize); + +/** + * destroy mmapings container and free all resources. + * + */ +LumieraMMapings +lumiera_mmapings_destroy (LumieraMMapings self); + +/** + * allocate and initialize new mmapings container + * + */ +LumieraMMapings +lumiera_mmapings_new (LumieraFile file, size_t chunksize); + +/** + * destroy and free mmapings container and all its resources + * + */ +void +lumiera_mmapings_delete (LumieraMMapings self); + +/** + * acquire a mmap which covers the given range + * @param self mmapings where to search + * @param acquirer list node of the new owner which will registered in the mmap + * @param start begin of the required range + * @param size requested size + * @return MMap object covering the requested range or NULL on error + */ +LumieraMMap +lumiera_mmapings_mmap_acquire (LumieraMMapings self, LumieraFile file, LList acquirer, off_t start, size_t size); + +/** + * release a previously acquired MMap object + * @param self mmapings to which the map belongs + * @param acquirer holding node + * @param map object to be released + */ +void +lumiera_mmapings_release_mmap (LumieraMMapings self, LList acquirer, LumieraMMap map); + +#endif +/* +// Local Variables: +// mode: C +// c-file-style: "gnu" +// indent-tabs-mode: nil +// End: +*/ From f58be32a0563d6a5849fcd528b6aaec416a74b21 Mon Sep 17 00:00:00 2001 From: Christian Thaeter Date: Mon, 19 May 2008 16:37:50 +0200 Subject: [PATCH 59/59] Fixes after rebase * remove _GNU_SOURCE from tests * re-enable C source building in backend --- src/backend/Makefile.am | 15 +++++++-------- tests/backend/test-filedescriptors.c | 2 -- tests/backend/test-filehandles.c | 2 -- 3 files changed, 7 insertions(+), 12 deletions(-) diff --git a/src/backend/Makefile.am b/src/backend/Makefile.am index 45ce555a8..8b33df06a 100644 --- a/src/backend/Makefile.am +++ b/src/backend/Makefile.am @@ -21,14 +21,13 @@ noinst_LIBRARIES += liblumibackend.a liblumibackend_a_CFLAGS = $(CFLAGS) -std=gnu99 -Wall -Werror liblumibackend_a_CPPFLAGS = -I$(top_srcdir)/src/ -liblumibackend_a_SOURCES = \ - $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp - -# $(liblumibackend_a_srcdir)/backend.c \ -# $(liblumibackend_a_srcdir)/file.c \ -# $(liblumibackend_a_srcdir)/filehandle.c \ -# $(liblumibackend_a_srcdir)/filedescriptor.c \ -# $(liblumibackend_a_srcdir)/filehandlecache.c +liblumibackend_a_SOURCES = \ + $(liblumibackend_a_srcdir)/mediaaccessfacade.cpp \ + $(liblumibackend_a_srcdir)/backend.c \ + $(liblumibackend_a_srcdir)/file.c \ + $(liblumibackend_a_srcdir)/filehandle.c \ + $(liblumibackend_a_srcdir)/filedescriptor.c \ + $(liblumibackend_a_srcdir)/filehandlecache.c noinst_HEADERS += \ diff --git a/tests/backend/test-filedescriptors.c b/tests/backend/test-filedescriptors.c index 40f516048..3e14ead1d 100644 --- a/tests/backend/test-filedescriptors.c +++ b/tests/backend/test-filedescriptors.c @@ -18,8 +18,6 @@ along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#define _GNU_SOURCE - #include "lib/safeclib.h" #include "backend/backend.h" diff --git a/tests/backend/test-filehandles.c b/tests/backend/test-filehandles.c index 71056144e..17c237d4f 100644 --- a/tests/backend/test-filehandles.c +++ b/tests/backend/test-filehandles.c @@ -18,8 +18,6 @@ along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. */ -#define _GNU_SOURCE - #include "lib/llist.h" #include "lib/safeclib.h"