diff --git a/doc/technical/build/LumieraDebianPackage.txt b/doc/technical/build/LumieraDebianPackage.txt new file mode 100644 index 000000000..9e97f3162 --- /dev/null +++ b/doc/technical/build/LumieraDebianPackage.txt @@ -0,0 +1,89 @@ +The Lumiera Debian Package +========================== +:Author: Hermann Voßeler deb@ichthyostega.de +:Date: 11/2011 + +//Menu: label Debian Package + +Since several of the Lumiera core developers run a Debian flavour as their primary development platform, +it seems natural to care for the debian packaging of Lumiera ourselves. Moreover, we've declared +Debian/Stable to be our *reference platform* -- we'll provide any additionally required, more recent +packages through our own link:http://Lumiera.org/debian[Debian Apt-Repository] (Debian depot). + +TIP: readers new to debian packages may want to have a look into our + link:{ldoc}/user/tutorials/DebianBuilding.html[Debian build tutorial], + describing the commands for building and the structure of a debian + (source) package in general + + +Package build process +--------------------- +As our whole infrastructure relies heavily on the Git version management tool, it's only natural +also to organise the (debian) packaging with the help of Git. Fortunatlely, there is a nifty tool +called link:https://honk.sigxcpu.org/piki/projects/git-buildpackage/[git-buildpackage], written +by Guido Günther exactly for this purpuse: It treats the _debianisation_ as a branch in the +Git repository, forking off the mainline at the release point. + +For Lumiera, this _debianisation branch_ is called 'deb' and can be found in the +link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=summary[git:/git.lumiera.org/lumiera/debian] repository. + +Installing Lumiera +~~~~~~~~~~~~~~~~~~ +The link:SCons.html[SCons build] generates a relocatable distribution directory structure, +holding the main executable, additional tools and utilities, plus the core libraries and resources. +Here _relocatable_ means that this subtree can be moved and placed anywhere on the system. +As long as the relative directory layout remains intact, the executables will be able to find +and load the accompanying libraries and resources. By invoking the +scons install+ target, +this directory structure is placed into the installation target directory. + +Actually, this lookup process at application startup is performed in two phases + +- in the first phase the main application locates the directly linked shared libraries. + Especially those libraries belonging first class to the Lumiera application are built + with a relative searche path ('rpath' with $ORIGIN token). These libraries are placed + into the +modules/+ subfolder + +- after successfully launching the executable, the second phase performes a lookup programatically, + starting from the path location of the executable, trying to find a +setup.ini+ which defines + additional plug-ins and resources to load. Notably, the GUI to launch is loaded as a + plug-in through this mechanism. Moreover, this bootstrap configuration defines the + additional platform and user configuration to load for further outfitting of the system. + +LSB Installation Layout +~~~~~~~~~~~~~~~~~~~~~~~ +This organisation is used as foundation for packaging and installing. The primariy application +distribution structure will be located into a subfolder below +/usr/lib/+. Only the main application +executable will be symlinked into +/usr/bin/+. The LSB explicitly allows for such a layout, which is +typically used by large application bundles (OpenOffice, Gimp, Eclipse). Since the application +startup encompasses two phases, loading the extended configuration programmatically after +launching the application, such extended resources can easily be relocated into a separate +folder below +/usr/share/+, as required by LSB. + + +Releases, Branches and Conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Official releases are marked by a tag on the master branch. Usually, we'll also fork a +*release branch* at that point, featuring bugfixes only. From here we'll merge to the +*debian branch* + + +Package build commands +~~~~~~~~~~~~~~~~~~~~~~ +To (re)build the debian package + +. +git clone git:/git.lumiera.org/lumiera/debian+ +. +mkdir pack.deb+ +. +cd debian+ +. +git-buildpackage --git-upstream-branch=+ _RELEASE-TAG_ + +-- here _RELEASE-TAG_ denotes the point in the Git history, which should become +the reference source and be packaged into the *.orig.tar.gz. Usually, it's just +sufficient to use 'master' for that purpose. + + +Debian-Depot for installation via Apt +------------------------------------- +In addition to the packaging, we maintain a dedicated Apt-Repository for automated +installation and upgrades. We try to build the package for several Debian derived +distributions (like Ubuntu). -> link:../infra/debianDepot.html[more on the repository organisation] + diff --git a/doc/technical/build/index.txt b/doc/technical/build/index.txt index a9b3261da..83805ba4e 100644 --- a/doc/technical/build/index.txt +++ b/doc/technical/build/index.txt @@ -3,13 +3,15 @@ Lumiera build system As work progresses, we will add more information on the Lumiera build system. +//Menu: label Build System + build -- continuous integration -- packaging * SCons * Autotools * Dependencies * link:BuildDroneDraft.html[»Builddrone« concept from 2008] -* Packaging: Debian RPM -* Lumiera debian depot +* Packaging: link:LumieraDebianPackage.html[Debian] RPM +* Lumiera link:../infra/debianDepot.html/[debian depot] diff --git a/doc/technical/code/darkCorners.txt b/doc/technical/code/darkCorners.txt new file mode 100644 index 000000000..732c2c9ee --- /dev/null +++ b/doc/technical/code/darkCorners.txt @@ -0,0 +1,35 @@ +Dark Corners +============ + +_this page accounts for some problematic areas, sketchy solutions, +nonportable hacks, terrorism and other misdemeanour_ + +Library +------- + +Equality of Functors +~~~~~~~~~~~~~~~~~~~~ +One of the more important recent additions to the C++ language are function objects. +In addition to the features actually provided by the boost implementation, the tr1 report +also requires function instances to implement an equality operator. Unfortunately the +implementation approach choosen by boost makes a 100% correct implementation of +comparision very dificult, if not impossible. Thus, the boost developers refused +to implement this feature. + +The bad news is that really using the power of opaque function objects quickly drove +us (Lumiera) into a situation where such an equalty test and a hash calculation on +function objects would be necessary. The whole point of using function objects is +the ability to ``erase'' specific details, which has the downside that the resulting +generic objects are opaque and often dificult to manage, when it comes to storing +and retrieving objects building on such functors. + +Thus I built an hack, based on the implementation details of boost::function. +In +functor-util.hpp+ we define a +class HijackedFunction+, which has the same +data layout as the original boost::function. After forcibly casting such an function +(reference or pointer) into a +HijackedFunction+, we're able to inspect and evaluate +the implementation pointers for equality comparison and hash value calculation. +This approach works and actually detects copied functions to be _equal_, but is +unable to pinpoint _equivalence_, e.g. functors bound to the same function with +the same arguments through separate but otherwise identical invocations of +bind+. +Besides, should boost or the standard library implementors eventually change the +implementation, this workaround will break. diff --git a/doc/technical/code/index.txt b/doc/technical/code/index.txt new file mode 100644 index 000000000..f0f1553f8 --- /dev/null +++ b/doc/technical/code/index.txt @@ -0,0 +1,49 @@ +Code Base Organisation +====================== + +//Menu: label Code Base + + +This section of Lumiera's technical documentation deals with the code base as such, +not so much with the internals of the Lumiera application. It is targeted towards +people participating in Lumiera development. + +The Lumiera source base is huge and expected to grow for some time to come. Just +judging by the _lines of code_, Lumiera can be considered as a *medium sized system*, +which gives us some hints towards the adequate practices, conventions and the +degree of formalism to apply. + +Generally speaking, a *small system* doesn't exceed the mental capacity of a single +knowledgeable human. A single person can know everything and understand everything +in such a code base. Thus, any formalism and organisational overhead is detrimental; +we're best off doing away with ``Methods'' and just rely on taste and craftsmanship. + +But there is a hard limit for this approach, kind of a _sonic wall_ -- usually somewhere +around a size of 60000 LoC. At that point, the adequate value system flips around as a +whole. Since beyond that point, no single person can understand and know everything about +such a system, _communication and understanding become the primary concerns_ of the +development process. Suddenly, we _do need_ some organisation, some rituals and +a methodical approach. Yet still we don't need a high degree of formalism, since +at least we can easily _know and recall everyone involved_. + +To give a delineation to the other side, a constantly growing project might eventually +reach the point where the actual coding of new functionality makes up less than 25% +of the overall work required. More people might be involved than anyone can recall or +overlook reasonably. A code size of 1 Million LoC seems to exceed any kind of sufficiently +precise imagination. From that point on, a rigorosely formalsed organisation is vital +to keep such a *large scale project* from floundering. + +//// +Conventions +----------- +_to be written_ + +Guidelines +---------- +//// + +'''' + * also see the link:{ldoc}/devel/rfc.html[Design Process] for ongoing discussions + * see the link:/devs-vault/[Developers Vault] for frequently used developer's resources + + diff --git a/doc/technical/howto/HashFunctions.txt b/doc/technical/howto/HashFunctions.txt new file mode 100644 index 000000000..b30e23989 --- /dev/null +++ b/doc/technical/howto/HashFunctions.txt @@ -0,0 +1,111 @@ +Hash functions (C++) +==================== + +_This page is for collecting know-how related to hash functions and hash tables._ + +The original STL was lacking proper support for hashtables, hash based associative arrays +and hash calculation in general. To quite some developers, hash tables feel like some kind +of _impure_ data structure -- unfortunately the properties of modern CPUs turned the balance +significantly in favour of hash tables due to memory locality. Pointer based datastructures +can't be considered especially _performant_ as they were in the good old times. + +The tr1 extension and the new C++11 standard amended the problem by defining a framework +for hash functions and hash tables. When sticking to some rules, custom written hash functions +can be picked up automatically by the standard library and -containers. + +Standard Hash Definitions +------------------------- + +Hash values:: + hash values are unsigned integral numbers of type 'size_t' ++ +Basically this means that the range of hash values roughly matches the memory address space. +But it also means that this range is _platform dependant_ (32 or 64bit) and -- given the usual +hash calculation based on modulus (wrap around) -- that generated hash values are nonportable. + +Hash function:: + a hash function calculates a hash value for objects of its argument type. Thus, for every + supported type, there is a dedicated hash function. Quite some hash functions are generated + from function templates though. + +Hash functor:: + a function object able to calculate hash values when invoked. The standard library and the + corresponding boost libraries accept functors of type 'hash' to calculate hash values + for objects or values of type 'TY' + +Hash based containers:: + While the standard Set and Map types (including the Multiset and Multimap) are based on + balanced binary trees, the new C\+\+11 standard includes hash based variants (with name + prefix +unordered_+). These hashtable based containers require a +hash+ functor + to be able to derive the hash value of any encountered key value. Hash functors may + be provided as additional type parameter to the container; if omitted, the compiler + tries to find a (maybe custom defined) hash functor by *ADL* (see below) + + +C++11 versus Boost +~~~~~~~~~~~~~~~~~~ +The Boost library *functional-hash* provided the foundation for the definition now accepted +into the new C++ standard. Yet the boost library provides some additional facilities not +part of the standard. Thus we're bound to choose + +* either including ++ and +using std::tr1::hash+ +* or including ++ and +using boost::hash+ + +The boost version additionally provides pre defined hash functors for STL containers holding +custom types -- and it provides an easy to use extension mechanism for writing hash functions +for custom types. Effectively this means that, assuming usage of the boost-include, the actual +implementation and the way it is picked up is _different but compatible_ to the standard way. + +Boost: hashing custom types +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The extension mechanism used by the boost version is best explained by looking +at the code + +.boost/functional/hash/extensions.hpp +[source,C] +---- + template struct hash + : std::unary_function + { + std::size_t operator()(T const& val) const + { + return hash_value(val); + } + } +---- +So this templated standard implementation just _invokes an unqualified function_ +with the name +hash_value(val)+ -- when instantiating this template for your custom +class or type, the compiler will search this function not only in the current scope, +but also in the namespace defining your custom type +T+ (this mechanism is known as +``**A**rgument **D**ependant **L**ookup''). Meaning that all we'd need to do is to define a +free function or friend function named +hash_value+ alongside with our custom data types (classes). + +To further facilitate providing custom hash functions, boost defines a function ++boost::hash_combine(size_t seed, size_t hashValue)+, allowing to _chain up_ the +calculated hash values of the parts forming a composite data structure. + +- see Lumiera's link:http://git.lumiera.org/gitweb?p=LUMIERA;a=blob;f=src/proc/asset/category.hpp;h=b7c8df2f2ce69b0ccf89439954de8346fe8d9276;hb=master#l104[asset::Category] + for a simple usage example +- our link:http://git.lumiera.org/gitweb?p=LUMIERA;a=blob;f=src/lib/symbol-impl.cpp;h=9e09b4254ac57baefeb0a0c06ccd423318e923c1;hb=master#l67[lib::Symbol datatype] + uses the standard implementation of a string hash function combining the individual + character's hashes. + + + +LUID values +----------- +Lumiera's uniform identifier values shouldn't be confused with regular hash values. +The purpose of LUID values is to use just plain random numbers as ID values. But, because +of using such a incredibly large number space (128bit), we can just assume any collision +between such random LUID to be so unlikely as to reasonably ignore this possibility +altogether. Let's say, the collision of random LUID values won't ever happen, same as +the meltdown of an atomic power plant, which, as we all know, won't ever happen either. + +Relation to hash values +~~~~~~~~~~~~~~~~~~~~~~~ +When objects incorporate sich an unique LUID, this provides for a prime candidate to +derive hash values as a side-effect of that design: Since incorporating an LUID typically +means that this object has an _distinguishable identity_, all objects with the same LUID +should be considered _equivalent_ and thus hash to the same value. Consequently we can just +use a +size_t+ prefix of the LUID bitstring as hash value, without any further calculations. + diff --git a/doc/technical/howto/index.txt b/doc/technical/howto/index.txt index 20450cd49..49970fc99 100644 --- a/doc/technical/howto/index.txt +++ b/doc/technical/howto/index.txt @@ -12,3 +12,5 @@ similar usefull pieces of information targeted at Lumiera developers. See also == Notepad - link:DebugGdbPretty.html[Python pretty printers for GDB] +- link:HashFunctions.html[Notes regarding standard hash functions] + diff --git a/doc/technical/index.txt b/doc/technical/index.txt index 147751f86..b56ad484d 100644 --- a/doc/technical/index.txt +++ b/doc/technical/index.txt @@ -29,7 +29,8 @@ components. == Tools .Development -* link:http://www.lumiera.org/doxy/[*Doxygen generated documentation*] : Complete reference for the code of Lumiera. +* link:http://www.lumiera.org/doxy/[*Doxygen generated documentation*] : API documentation of the Lumiera code. +* organisation of the link:code/index.html[Code Base] in general .Building * link:build/index.html[*Buildsystem*] : Installation & compilation tools, dependencies and packaging. diff --git a/doc/technical/infra/debianDepot.txt b/doc/technical/infra/debianDepot.txt index f599950b2..8573e3803 100644 --- a/doc/technical/infra/debianDepot.txt +++ b/doc/technical/infra/debianDepot.txt @@ -25,7 +25,7 @@ To (re)build the debian package . +git clone git:/git.lumiera.org/lumiera/debian+ . +mkdir pack.deb+ . +cd debian+ -. +git-buildpackage --git-upstream-branch=+_RELEASE-TAG_ +. +git-buildpackage --git-upstream-branch=+ _RELEASE-TAG_ -- here _RELEASE-TAG_ denotes the point in the Git history, which should become the reference source and be packaged into the *.orig.tar.gz. Usually, it's just @@ -76,15 +76,15 @@ everyday usage ~~~~~~~~~~~~~~ import a package:: - +reprepro -V -C experimental include lenny lumiera_0.pre.01-1+maverick_i386.changes+ + +reprepro -V -C experimental include squeeze lumiera_0.pre.01-1+squeeze_i386.changes+ + this adds the given binary lumiera package, together with all sources and the original -tarball to the 'lenny' repository, into the 'experimental' section +tarball to the 'squeeze' repository, into the 'experimental' section dump out an entire repository:: - +reprepro -V export lenny+ + +reprepro -V export squeeze+ + -this will __re__generate all of the indices, signatures and metadata of the 'lenny' repository +this will __re__generate all of the indices, signatures and metadata of the 'squeeze' repository Configuration @@ -98,12 +98,12 @@ I choose this layout because of the marginal relevance of this depot management; want to create _yet another not so useful Git repository..._) * primary link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=conf/distributions;hb=refs/heads/depot[configuration] - * Logfile of imports: link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=log/lenny.log;hb=refs/heads/depot[for Lenny] + * Logfile of imports: link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=log/squeeze.log;hb=refs/heads/depot[for Debian/Squeeze] [NOTE] .some special details to note in our setup ======================================================================================================================= -- each block in the 'distributions' file defines a repository for a ``distribution'' (e.g. Lenny, Lucid, Maverick). +- each block in the 'distributions' file defines a repository for a ``distribution'' (e.g. Squeeze, Lucid, Maverick). Within such a repo, there are sections named 'Components'. - The _override_ files mentioned in the configuration allow to overwrite / replace arbitrary fields in the metadata of all packages added to that distribution. diff --git a/doc/user/tutorials/DebianBuilding.txt b/doc/user/tutorials/DebianBuilding.txt index 5e33957b6..8c249b667 100644 --- a/doc/user/tutorials/DebianBuilding.txt +++ b/doc/user/tutorials/DebianBuilding.txt @@ -35,7 +35,7 @@ should _not_ be done as root. . add a suitable source line to your *Apt configuration* ('/etc/apt/sources.lst') + ---- -deb-src http://lumiera.org/debian/ lenny experimental +deb-src http://lumiera.org/debian/ squeeze experimental ---- . get all the *build dependencies* +