SCons Build-System ================== :author: Ichthyo :date: 2012 //MENU: label SCons Build Lumiera uses a build system based on link:http://scons.org[SCons] SCons is an open source software construction tool based on Python build definition scripts. Within these build scripts, we define a data structure to describe the parts and dependencies of our software. When executed, SCons evaluates those definitions and the actual files in the source tree to derive a build strategy, which is then performed. SCons core concepts ------------------- ^_this section is based on the introductory pages on the link:http://www.scons.org/wiki/BasicConcepts[SCons Wiki]_^ .SCons Environment When SCons starts building the project, it creates its own environment with dependency trees, helper functions, builders and other stuff. The SCons environment is created in memory and some parts of it are saved to disk to speed up things on the next start. The definition of the build happens within this artificial build environment. This often confuses people who used Makefiles, where environment is actually the System Environment. .System Environment the familiar operating system container with environment variables such as PATH, HOME etc. It is usually accessible via os.environ mapping in Python and therefore in SCons too. SCons doesn't import any settings from System Environment automatically (like flags for compilers, or paths for tools), because it's designed to be a cross platform tool with _predictable behaviour._ That's why, if you rely on any system PATHs or environment variables -- you need to extract those settings explicitly in your build definition. .SConstruct when SCons executes, it performs a build definition python script written by the user. By convention, this main script is called 'SConstruct' and is located in the root of the source tree. It is a full featured Python module executed within a specifically prepared environment. .SConscript these files are also SCons scripts, but they are placed in subdirectories of the project. Typically they are used to organize hierarchical builds and are included from the main SConstruct file from the project root. Often, all of the actual build definitions reside in SConscript files in the sub-trees of the project. .Builder The SCons buildsystem revolves around the metaphor of a _Builder_. This is a SCons object that you explicitly invoke from the scripts to _define_ that there is something to build, transforming a _source_ into a _target_. So the target depends on the sources, and typically those _source nodes_ were created by previous builder invocations. The power of SCons is in the fact that dependencies are tracked automatically. When source files change, the system is able to derive which targets to rebuild. .Scanner when defining a builder, SCons relies on modular scanner components to ``understand'' the source of the build step. They may scan source files to discover additional dependencies referenced inside. Thus, SCons comes with built-in knowledge about the source files and artefacts to be created by a typical build, and further types can be added through plug-ins. .Tool any further, external component that adds Builders, Scanners and other helpers to SCons environments for use within scripts. There are special tools for _configuring the platform_ to detect libraries and further requirements. Relying on those tools. the build environment will be outfitted to reflect the needs of the specific build. Sub-environments with special configuration may be created. .Target any _node_ or ``build step'' encountered through the definition of the build is a _target_. The actual build will be triggered by requesting a target, which typically might be just an executable known to reside at some location in the tree. Special _alias targets_ may be defined, based on other targets, to trigger off standard build situations. Especially, a _default_ target may be defined. '''' Organisation of the Lumiera SCons build --------------------------------------- Within our build system, we build upon the power of the Python programming language to create abstractions tailored to the needs of our project. Located in the `admin/scons` subdirectory, you'll find a collection of Python modules to provide these building blocks. - the *LumieraEnvironment* is created as a subclass of the standard SCons build environment; it is outfitted with pre-configured custom builders for executables, libraries, extension module, Lumiera plug-in and icon resources. - all these *custom builders* implement a set of conventions and directory locations within the tree. These are defined (and can be adjusted) in the *Setup.py* module. This way, each builder automatically places the generated artefacts at standard build and installation locations. - for defining individual targets and builder invocations, we rely on *build helpers* to process whole *source sub trees* rather than individual files. Mostly, just placing a source file into the appropriate sub tree is sufficient to get it compiled, linked and installed in a standard way. Sub-trees ~~~~~~~~~ .the source tree All sourcecode of the core application resides below `src/`. Building these components is controlled by the SConscript residing in this application source root. By convention, this is also the root for header includes -- _all headers should be included relative_ to `src/`. .the three layers Within this application core tree, there are sub-trees for the main layers comprising the application. Each of these sub-trees will be built into a shared library and then linked against the application framework and common services residing in `src/common`. These common services in turn are also built into a shared library `liblumieracommon.so`, as is the collection of helper classes and support facilities, known as our 'support library' `liblumierasupport.so`. Besides, there is a sub-tree for core plug-ins and helper tools. .the GTK Gui one of the sub-trees, residing in `src/stage` forms the _upper layer_ or _user-interaction layer_. Contrary to the lower layers, the Stage Layer (GUI) is _optional_ and the application is fully operational _without GUI._ Thus, the GTK Gui is built and loaded as Lumiera a plug-in. .unit tests Since our development is test-driven, about half of the overall code can be found in unit- and integration tests, residing below `test/`. There is a separate SConscript file, to define the various link:{ldoc}/technical/infra/TestSupport.html[kinds of test artefacts] to be created. - plain-C tests are defined in _test-collections_, grouped thematically into several subdirectories. Here, each translation unit provides a separate +main()+ function and is linked into a stand-alone executable (yet still linked against the appropriate shared libraries of the main application layers) - the tests covering C++ components are organised into test-suites, residing in separate sub-trees. Currently (as of 5/2015), we link each sub-tree into a shared test library. Here individual translation units define individual test case classes. At the end, all these unit tests are linked together with a testrunner `main()` into the `test-suite` executable. .research There is a separate subtree for research and experiments. The rationale being to avoid re-building most of the core application when it comes to experimenting and trying out new technologies. .icons and resources the +data/+ subtree holds resources, configuration files and icons for the GUI. Most of our icons are defined as SVG graphics. The build process creates a helper executable (+rsvg_convert+) to render these vector graphics with the help of lib Cairo into icon collections of various sizes. .documentation Largely, the documentation is written in Asciidoc and provided online at link:{ldoc}[the documentation section] of our website. The plain-text sources of this documentation tree is shipped alongside with the code. Besides, we build *Doxygen* API documentation there, and we create design and technical specs and drawings in SVG and in UML. .the target directory This is where the results of the build process are created. Lumiera is organised into a _self contained folder structure_. As long as the relative locations, as found within +target/+, are kept intact, the Application will be able to start up and find all its resources. Consequently, there is no need to ``install'' Lumiera (and the ``install'' target just copies this folder structure into the standard installation locations of a typical Unix system) Unfortunately SCons is a bit weird regarding the object files created during the build process. So, for the time being, we're just building in-tree. Apologies for that. Invoking the Build ~~~~~~~~~~~~~~~~~~ All of the build process is launched through the `scons` python script, usually installed into `/usr/bin` when installing the SCons package onto the system. Just invoking scons -h prints a summary of all custom options, targets and toggles defined for our build. Targets ^^^^^^^ - *build* is the default target: it creates the shared libs, the application, core plug-ins and the GUI. - *testcode* additionally builds the research and unit test code - *check* builds test code and runs our test-suites - *research* builds just the research tree - *doc* builds documentation (currently just Doxygen) - *all* builds the Application, test-suites and documentation - *install* builds and installs the Lumiera Application By convention, invoking `scons -c` will _clean up_ everything the given target _would_ build. Thus, invoking `scons -c /` is the most global clean operation: it will clean up al build artefacts and will un-install Lumiera (recall: every defined node, or directory is also a target). Configure checks ^^^^^^^^^^^^^^^^ SCons does not support the concept of a separate ``configure'' step. The necessary dependency detection is performed before each build. Currently, we expect _all dependencies to be installed first-class_ into the system. We do not (yet) support custom libraries in strange locations. Please use your package manager. Caching and MD5 sums ^^^^^^^^^^^^^^^^^^^^ SCons stores MD5 sums of all source files, all configure checks and all the command lines used to invoke compilers and external tools. The decision, what needs to be rebuilt is based entirely on these checksums. For one, this means that configure checks are re-run only when necessary. It also means that changes to some compiler switches will automatically cause all affected parts of the application to be re-built. And of course it means, that you only ever compile what is necessary. With SCons, there is no need for the usual ``cleaning paranoia''. Similarly, there is no need for CCache (but using DistCC rocks !). Unfortunately, calculating those MD5 sums requires some time on each build, even if the net result is that nothing will happen at all. Configuration options ^^^^^^^^^^^^^^^^^^^^^ We provide several custom configuration options (run `scons -h` to get a summary). All of these options are *sticky*: once set, the build system will recall them in a file '.optcache' and apply them the same way in subsequent builds. It is fine to edit '.optcache' with a text editor.