ichthyo/LUMIERA.clone
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

resync with current documentation on the Website

Browse source
This commit is contained in:
Fischlurch 2011-10-21 02:03:01 +02:00
commit fb48ec70f0
9 changed files with 300 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

89
doc/technical/build/LumieraDebianPackage.txt Normal file
View file

@ -0,0 +1,89 @@
The Lumiera Debian Package
==========================
:Author: Hermann Voßeler deb@ichthyostega.de
:Date: 11/2011
//Menu: label Debian Package
Since several of the Lumiera core developers run a Debian flavour as their primary development platform,
it seems natural to care for the debian packaging of Lumiera ourselves. Moreover, we've declared
Debian/Stable to be our *reference platform* -- we'll provide any additionally required, more recent
packages through our own link:http://Lumiera.org/debian[Debian Apt-Repository] (Debian depot).
TIP: readers new to debian packages may want to have a look into our
link:{ldoc}/user/tutorials/DebianBuilding.html[Debian build tutorial],
describing the commands for building and the structure of a debian
(source) package in general
Package build process
---------------------
As our whole infrastructure relies heavily on the Git version management tool, it's only natural
also to organise the (debian) packaging with the help of Git. Fortunatlely, there is a nifty tool
called link:https://honk.sigxcpu.org/piki/projects/git-buildpackage/[git-buildpackage], written
by Guido Günther exactly for this purpuse: It treats the _debianisation_ as a branch in the
Git repository, forking off the mainline at the release point.
For Lumiera, this _debianisation branch_ is called 'deb' and can be found in the
link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=summary[git:/git.lumiera.org/lumiera/debian] repository.
Installing Lumiera
~~~~~~~~~~~~~~~~~~
The link:SCons.html[SCons build] generates a relocatable distribution directory structure,
holding the main executable, additional tools and utilities, plus the core libraries and resources.
Here _relocatable_ means that this subtree can be moved and placed anywhere on the system.
As long as the relative directory layout remains intact, the executables will be able to find
and load the accompanying libraries and resources. By invoking the +scons install+ target,
this directory structure is placed into the installation target directory.
Actually, this lookup process at application startup is performed in two phases
- in the first phase the main application locates the directly linked shared libraries.
Especially those libraries belonging first class to the Lumiera application are built
with a relative searche path ('rpath' with $ORIGIN token). These libraries are placed
into the +modules/+ subfolder
- after successfully launching the executable, the second phase performes a lookup programatically,
starting from the path location of the executable, trying to find a +setup.ini+ which defines
additional plug-ins and resources to load. Notably, the GUI to launch is loaded as a
plug-in through this mechanism. Moreover, this bootstrap configuration defines the
additional platform and user configuration to load for further outfitting of the system.
LSB Installation Layout
~~~~~~~~~~~~~~~~~~~~~~~
This organisation is used as foundation for packaging and installing. The primariy application
distribution structure will be located into a subfolder below +/usr/lib/+. Only the main application
executable will be symlinked into +/usr/bin/+. The LSB explicitly allows for such a layout, which is
typically used by large application bundles (OpenOffice, Gimp, Eclipse). Since the application
startup encompasses two phases, loading the extended configuration programmatically after
launching the application, such extended resources can easily be relocated into a separate
folder below +/usr/share/+, as required by LSB.
Releases, Branches and Conventions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Official releases are marked by a tag on the master branch. Usually, we'll also fork a
*release branch* at that point, featuring bugfixes only. From here we'll merge to the
*debian branch*
Package build commands
~~~~~~~~~~~~~~~~~~~~~~
To (re)build the debian package
. +git clone git:/git.lumiera.org/lumiera/debian+
. +mkdir pack.deb+
. +cd debian+
. +git-buildpackage --git-upstream-branch=+ _RELEASE-TAG_
-- here _RELEASE-TAG_ denotes the point in the Git history, which should become
the reference source and be packaged into the *.orig.tar.gz. Usually, it's just
sufficient to use 'master' for that purpose.
Debian-Depot for installation via Apt
-------------------------------------
In addition to the packaging, we maintain a dedicated Apt-Repository for automated
installation and upgrades. We try to build the package for several Debian derived
distributions (like Ubuntu). -> link:../infra/debianDepot.html[more on the repository organisation]

6
doc/technical/build/index.txt
View file

@ -3,13 +3,15 @@ Lumiera build system
As work progresses, we will add more information on the Lumiera build system.
//Menu: label Build System
build -- continuous integration -- packaging
* SCons
* Autotools
* Dependencies
* link:BuildDroneDraft.html[»Builddrone« concept from 2008]
* Packaging: Debian RPM
* Lumiera debian depot
* Packaging: link:LumieraDebianPackage.html[Debian] RPM
* Lumiera link:../infra/debianDepot.html/[debian depot]

35
doc/technical/code/darkCorners.txt Normal file
View file

@ -0,0 +1,35 @@
Dark Corners
============
_this page accounts for some problematic areas, sketchy solutions,
nonportable hacks, terrorism and other misdemeanour_
Library
-------
Equality of Functors
~~~~~~~~~~~~~~~~~~~~
One of the more important recent additions to the C++ language are function objects.
In addition to the features actually provided by the boost implementation, the tr1 report
also requires function instances to implement an equality operator. Unfortunately the
implementation approach choosen by boost makes a 100% correct implementation of
comparision very dificult, if not impossible. Thus, the boost developers refused
to implement this feature.
The bad news is that really using the power of opaque function objects quickly drove
us (Lumiera) into a situation where such an equalty test and a hash calculation on
function objects would be necessary. The whole point of using function objects is
the ability to ``erase'' specific details, which has the downside that the resulting
generic objects are opaque and often dificult to manage, when it comes to storing
and retrieving objects building on such functors.
Thus I built an hack, based on the implementation details of boost::function.
In +functor-util.hpp+ we define a +class HijackedFunction+, which has the same
data layout as the original boost::function. After forcibly casting such an function
(reference or pointer) into a +HijackedFunction+, we're able to inspect and evaluate
the implementation pointers for equality comparison and hash value calculation.
This approach works and actually detects copied functions to be _equal_, but is
unable to pinpoint _equivalence_, e.g. functors bound to the same function with
the same arguments through separate but otherwise identical invocations of +bind+.
Besides, should boost or the standard library implementors eventually change the
implementation, this workaround will break.

49
doc/technical/code/index.txt Normal file
View file

@ -0,0 +1,49 @@
Code Base Organisation
======================
//Menu: label Code Base
This section of Lumiera's technical documentation deals with the code base as such,
not so much with the internals of the Lumiera application. It is targeted towards
people participating in Lumiera development.
The Lumiera source base is huge and expected to grow for some time to come. Just
judging by the _lines of code_, Lumiera can be considered as a *medium sized system*,
which gives us some hints towards the adequate practices, conventions and the
degree of formalism to apply.
Generally speaking, a *small system* doesn't exceed the mental capacity of a single
knowledgeable human. A single person can know everything and understand everything
in such a code base. Thus, any formalism and organisational overhead is detrimental;
we're best off doing away with ``Methods'' and just rely on taste and craftsmanship.
But there is a hard limit for this approach, kind of a _sonic wall_ -- usually somewhere
around a size of 60000 LoC. At that point, the adequate value system flips around as a
whole. Since beyond that point, no single person can understand and know everything about
such a system, _communication and understanding become the primary concerns_ of the
development process. Suddenly, we _do need_ some organisation, some rituals and
a methodical approach. Yet still we don't need a high degree of formalism, since
at least we can easily _know and recall everyone involved_.
To give a delineation to the other side, a constantly growing project might eventually
reach the point where the actual coding of new functionality makes up less than 25%
of the overall work required. More people might be involved than anyone can recall or
overlook reasonably. A code size of 1 Million LoC seems to exceed any kind of sufficiently
precise imagination. From that point on, a rigorosely formalsed organisation is vital
to keep such a *large scale project* from floundering.
////
Conventions
-----------
_to be written_
Guidelines
----------
////
''''
* also see the link:{ldoc}/devel/rfc.html[Design Process] for ongoing discussions
* see the link:/devs-vault/[Developers Vault] for frequently used developer's resources

111
doc/technical/howto/HashFunctions.txt Normal file
View file

@ -0,0 +1,111 @@
Hash functions (C++)
====================
_This page is for collecting know-how related to hash functions and hash tables._
The original STL was lacking proper support for hashtables, hash based associative arrays
and hash calculation in general. To quite some developers, hash tables feel like some kind
of _impure_ data structure -- unfortunately the properties of modern CPUs turned the balance
significantly in favour of hash tables due to memory locality. Pointer based datastructures
can't be considered especially _performant_ as they were in the good old times.
The tr1 extension and the new C++11 standard amended the problem by defining a framework
for hash functions and hash tables. When sticking to some rules, custom written hash functions
can be picked up automatically by the standard library and -containers.
Standard Hash Definitions
-------------------------
Hash values::
hash values are unsigned integral numbers of type 'size_t'
+
Basically this means that the range of hash values roughly matches the memory address space.
But it also means that this range is _platform dependant_ (32 or 64bit) and -- given the usual
hash calculation based on modulus (wrap around) -- that generated hash values are nonportable.
Hash function::
a hash function calculates a hash value for objects of its argument type. Thus, for every
supported type, there is a dedicated hash function. Quite some hash functions are generated
from function templates though.
Hash functor::
a function object able to calculate hash values when invoked. The standard library and the
corresponding boost libraries accept functors of type 'hash<TY>' to calculate hash values
for objects or values of type 'TY'
Hash based containers::
While the standard Set and Map types (including the Multiset and Multimap) are based on
balanced binary trees, the new C\+\+11 standard includes hash based variants (with name
prefix +unordered_+). These hashtable based containers require a +hash<KEY>+ functor
to be able to derive the hash value of any encountered key value. Hash functors may
be provided as additional type parameter to the container; if omitted, the compiler
tries to find a (maybe custom defined) hash functor by *ADL* (see below)
C++11 versus Boost
~~~~~~~~~~~~~~~~~~
The Boost library *functional-hash* provided the foundation for the definition now accepted
into the new C++ standard. Yet the boost library provides some additional facilities not
part of the standard. Thus we're bound to choose
* either including +<tr1/functional>+ and +using std::tr1::hash+
* or including +<boost/functional-hash>+ and +using boost::hash+
The boost version additionally provides pre defined hash functors for STL containers holding
custom types -- and it provides an easy to use extension mechanism for writing hash functions
for custom types. Effectively this means that, assuming usage of the boost-include, the actual
implementation and the way it is picked up is _different but compatible_ to the standard way.
Boost: hashing custom types
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The extension mechanism used by the boost version is best explained by looking
at the code
.boost/functional/hash/extensions.hpp
[source,C]
----
template <class T> struct hash
: std::unary_function<T, std::size_t>
{
std::size_t operator()(T const& val) const
{
return hash_value(val);
}
}
----
So this templated standard implementation just _invokes an unqualified function_
with the name +hash_value(val)+ -- when instantiating this template for your custom
class or type, the compiler will search this function not only in the current scope,
but also in the namespace defining your custom type +T+ (this mechanism is known as
``**A**rgument **D**ependant **L**ookup''). Meaning that all we'd need to do is to define a
free function or friend function named +hash_value+ alongside with our custom data types (classes).
To further facilitate providing custom hash functions, boost defines a function
+boost::hash_combine(size_t seed, size_t hashValue)+, allowing to _chain up_ the
calculated hash values of the parts forming a composite data structure.
- see Lumiera's link:http://git.lumiera.org/gitweb?p=LUMIERA;a=blob;f=src/proc/asset/category.hpp;h=b7c8df2f2ce69b0ccf89439954de8346fe8d9276;hb=master#l104[asset::Category]
for a simple usage example
- our link:http://git.lumiera.org/gitweb?p=LUMIERA;a=blob;f=src/lib/symbol-impl.cpp;h=9e09b4254ac57baefeb0a0c06ccd423318e923c1;hb=master#l67[lib::Symbol datatype]
uses the standard implementation of a string hash function combining the individual
character's hashes.
LUID values
-----------
Lumiera's uniform identifier values shouldn't be confused with regular hash values.
The purpose of LUID values is to use just plain random numbers as ID values. But, because
of using such a incredibly large number space (128bit), we can just assume any collision
between such random LUID to be so unlikely as to reasonably ignore this possibility
altogether. Let's say, the collision of random LUID values won't ever happen, same as
the meltdown of an atomic power plant, which, as we all know, won't ever happen either.
Relation to hash values
~~~~~~~~~~~~~~~~~~~~~~~
When objects incorporate sich an unique LUID, this provides for a prime candidate to
derive hash values as a side-effect of that design: Since incorporating an LUID typically
means that this object has an _distinguishable identity_, all objects with the same LUID
should be considered _equivalent_ and thus hash to the same value. Consequently we can just
use a +size_t+ prefix of the LUID bitstring as hash value, without any further calculations.

2
doc/technical/howto/index.txt
View file

@ -12,3 +12,5 @@ similar usefull pieces of information targeted at Lumiera developers. See also
== Notepad
- link:DebugGdbPretty.html[Python pretty printers for GDB]
- link:HashFunctions.html[Notes regarding standard hash functions]

3
doc/technical/index.txt
View file

@ -29,7 +29,8 @@ components.
== Tools
.Development
* link:http://www.lumiera.org/doxy/[*Doxygen generated documentation*] : Complete reference for the code of Lumiera.
* link:http://www.lumiera.org/doxy/[*Doxygen generated documentation*] : API documentation of the Lumiera code.
* organisation of the link:code/index.html[Code Base] in general
.Building
* link:build/index.html[*Buildsystem*] : Installation & compilation tools, dependencies and packaging.

14
doc/technical/infra/debianDepot.txt
View file

@ -25,7 +25,7 @@ To (re)build the debian package
. +git clone git:/git.lumiera.org/lumiera/debian+
. +mkdir pack.deb+
. +cd debian+
. +git-buildpackage --git-upstream-branch=+_RELEASE-TAG_
. +git-buildpackage --git-upstream-branch=+ _RELEASE-TAG_
-- here _RELEASE-TAG_ denotes the point in the Git history, which should become
the reference source and be packaged into the *.orig.tar.gz. Usually, it's just
@ -76,15 +76,15 @@ everyday usage
~~~~~~~~~~~~~~
import a package::
+reprepro -V -C experimental include lenny lumiera_0.pre.01-1+maverick_i386.changes+
+reprepro -V -C experimental include squeeze lumiera_0.pre.01-1+squeeze_i386.changes+
+
this adds the given binary lumiera package, together with all sources and the original
tarball to the 'lenny' repository, into the 'experimental' section
tarball to the 'squeeze' repository, into the 'experimental' section
dump out an entire repository::
+reprepro -V export lenny+
+reprepro -V export squeeze+
+
this will __re__generate all of the indices, signatures and metadata of the 'lenny' repository
this will __re__generate all of the indices, signatures and metadata of the 'squeeze' repository
Configuration
@ -98,12 +98,12 @@ I choose this layout because of the marginal relevance of this depot management;
want to create _yet another not so useful Git repository..._)
* primary link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=conf/distributions;hb=refs/heads/depot[configuration]
* Logfile of imports: link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=log/lenny.log;hb=refs/heads/depot[for Lenny]
* Logfile of imports: link:http://git.lumiera.org/gitweb?p=lumiera/debian;a=blob;f=log/squeeze.log;hb=refs/heads/depot[for Debian/Squeeze]
[NOTE]
.some special details to note in our setup
=======================================================================================================================
- each block in the 'distributions' file defines a repository for a ``distribution'' (e.g. Lenny, Lucid, Maverick).
- each block in the 'distributions' file defines a repository for a ``distribution'' (e.g. Squeeze, Lucid, Maverick).
Within such a repo, there are sections named 'Components'.
- The _override_ files mentioned in the configuration allow to overwrite / replace arbitrary fields in the metadata of
all packages added to that distribution.

2
doc/user/tutorials/DebianBuilding.txt
View file

@ -35,7 +35,7 @@ should _not_ be done as root.
. add a suitable source line to your *Apt configuration* ('/etc/apt/sources.lst')
+
----
deb-src http://lumiera.org/debian/ lenny experimental
deb-src http://lumiera.org/debian/ squeeze experimental
----
. get all the *build dependencies*
+