From 02eec48c3b242e3ba520ca6f25cc05398f215ef3 Mon Sep 17 00:00:00 2001
From: Ichthyostega <prg@ichthyostega.de>
Date: Mon, 21 Feb 2011 08:24:58 +0100
Subject: [PATCH] Documentation of the new menu generating script for the
 website

Signed-off-by: Ichthyostega <prg@ichthyostega.de>
---
 doc/technical/infra/MenuGen.txt | 265 ++++++++++++++++++++++++++++++--
 1 file changed, 249 insertions(+), 16 deletions(-)

diff --git a/doc/technical/infra/MenuGen.txt b/doc/technical/infra/MenuGen.txt
index f2b3d0b0b..a6f402cd1 100644
--- a/doc/technical/infra/MenuGen.txt
+++ b/doc/technical/infra/MenuGen.txt
@@ -1,43 +1,276 @@
 Website Navigation Generator
 ============================
-:Author: Hermann Voßeler
-:Date: 2/2011
+:author: Hermann Voßeler
+:revdate: 2/2011
+
+
+This page contains documentation and notes regarding the +menugen.py+ --
+written 2/2011 during our attempt to get the new Lumiera website online finally.
+The link::http://git.lumiera.org/gitweb?p=website-staging;a=blob;f=menugen.lua;h=aad2129d7f4ed3f3b35b2fc3ac2a63a9f1bfb62d;hb=menugen[initial draft version] was written by _cehteh_ in Lua
+
+
+**************************************************************************
+The purpose of the +*menugen*+ script is to maintain the navigation menu
+on the Lumiera website semi-automatically. In the usual setup, this script
+is triggered from a _Git push_ -- it walks the web subdirectories and
+discovers menu entries. The generated HTML page contains both visible
+elements and JavaScript snippets to display and highlight the menu
+on the client side appropriately
+**************************************************************************
+
+Overview: how it works
+----------------------
+The menu generation and display is comprised of several parts working together
+
+. the +build_website.sh+ is triggered as a Git post-receive hook, whenever new
+  commits are transfered to the website Git repository. After discovering new
+  Asciidoc source files and generating the corresponding HTML files, the
+  menu generator script is invoked
+. the +menugen+ python script walks the subdirectories to discover possible
+  menu contents. It visits Asciidoc source files (`*.txt`) and picks up
+  
+  - the location / URL
+  - the title
+  - special `//MENU:` directives embedded in Asciidoc comments
+
+. after building a complete menu tree (actually a DAG), this data structure
+  is walked to generate output HTML into a `menu.html` file in website root.
+. the page template (`page.conf`) for generated Asciidoc pages contains an
+  +<IFrame>+ to display this `menu.html`
+. when loading `menu.html`, some JavaScript elements generated into the body
+  alongside with the visible content will execute, causing a lookup table
+  in the client side memory being populated with the menu entries and parent
+  dependencies. Each individual menu entry has an attached unique ID, originally
+  generated by the server side +menugen+ script. The clientside JavaScript always
+  addresses elements directly through these IDs, mostly ignoring the actual DOM
+  structure
+. whenever a new webpage is loaded, the `onload` handler on the +<IFrame>+ (or
+  a similar mechanism) invokes the +markPageInMenu()+ JavaScript function, which
+  addresses the IFrame by its ID `inavi`, and calls into the JavaScript located
+  there. This script in turn finds the menu entry corresponding to the current
+  page with the help of the lookup table mentioned above; this allows to highlight
+  the current page and fold any other branches of the menu to keep the visible
+  part reasonably small to fit on a single page
+. folding and highlighting changes are done by manipulating the style of these
+  elements; the actual presentation is mostly controlled by a `menu.css`
+. any further JavaScript functions used to operate the menu are located in
+  the statically served `menu.js` -- the generated menu contains only the
+  ``moving parts''
+
+Configuring menu generation
+---------------------------
+While, generally speaking, the script was written to remove the need to care
+for the menu most of the time, there are numerous extension points and configuration
+options to deal with special cases. Adjustments can be done on several levels:
+
+* the +menugen+ python script contains in embedded set of _predefined menu entries,_
+  forming the backbone of the generated menu. The use of this feature is optional
+  and can be enabled with the `-p` or `--predefined` switch. These predefined
+  configuration steps are done in a function +addPredefined()+ right at the top;
+  the configuration is written in the style of an _internal DSL_ and should be
+  fairly self explanatory.
+* when discovering Asciidoc page sources, special `//MENU:` directives are
+  processed (`//` marks an Asciidoc comment). The remainder of such a line
+  is always parsed as a single directive; in case of a parsing error a warning
+  is printed and the line will be ignored. The individual directives mostly
+  correspond to similar functions usable in the aforementioned internal DSL;
+  actually both kinds of configuration have the same effect: they attach
+  some modification command to the menu element in question. Note especially
+  that such directives can modify the discovery of further pages -- pages
+  can be attached, excluded, ordered; and the discovery can be redirected
+  to another subdirectory.
+* the actual code generation is mostly based on python template code contained
+  in a separate script +menuformat.py+ -- located alongside the main menu generator
+  script. This code generation is driven by a classical recursive tree visitation
+  over the menu data structure built up thus far; code generation hooks are called
+  on each tree leaf and when entering and leaving inner nodes (submenu nodes).
+* the highlighting is done by the client side JavaScript in +js/menu.js+ -- 
+  mostly just by _adding or removing CSS classes_ dynamically. The actual styling
+  of the menu entries is thus largely independent of the menu generation (but of
+  course the CSS selectors must line up with the basic structure of the generated
+  code). The current version of this CSS stylesheet makes heavy use of _contextual
+  selectors_ and the general cascading mechanism to build up the final style; e.g.
+  the indentation according to the menu level is done by attaching a style based
+  on the number of nested HTML elements.
+
+
+Summary of menu placement directives
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+With the term _placement directives_ we denote all the adjustments and configuration
+possible either through the internal DSL for the predefined menu structure, or through
+the `//Menu:` lines in the individual pages.
+
+addressing menu nodes
+^^^^^^^^^^^^^^^^^^^^^
+
+Each menu entry corresponds to a menu node in the internal data structure. In the most
+general case, this structure is a _Directed Acyclic Graph_, because a node might be
+hooked up below several different parent nodes. In this case, such a node will also
+be visited multiple times for code generation -- one time for each parent it is
+attached below. Amongst these parent nodes, the first parent node attached is called
+the _primary parent_, because this first attachment of a node defines the _logical
+path_ uniquely describing this node. Note, this logical path can be different to
+the actual web paths / URLs generated, and also be different to the file system
+path where the source file resides. It is just defined by the chain of parent
+nodes leading to the root of the menu data structure.
+
+The leaf element of this logical menu path is called the _ID_ of the node. Typically
+this ID corresponds to the filename without the extension. But for the code generation
+and the client sides JavaScripts, the full menu path is used as an HTML id element,
+because -- generally speaking -- only the full menu path denotes an element unambiguously.
+
+When working with nodes, and especially when writing placement directives in the individual
+source files, in most cases it is not necessary to specify the full menu path of a node.
+Actually, nodes can be addressed by any path suffix, and even just by the bare node ID.
+But when there is an ambiguity, just the first node found is picked. Because nodes have
+an unique identity, this can sometimes yield rather wired results. To minimise the
+danger of ambiguities, the _discovery_ of source pages always addresses the menu node
+to be populated with the full menu path.
+
+configuration example
+^^^^^^^^^^^^^^^^^^^^^
+
+[source,python]
+--------------------------------------------------------------------------
+def addPredefined():
+    root = Node(TREE_ROOT, label='Lumiera')                                <1>
+    proj = root.linkChild('project')                                       <2>
+    proj.linkChild('faq')
+    
+    proj.prependChild ('screenshots')                                      <3>
+    proj.putChildLast ('press')
+    proj.putChildAfter('faq', refPoint=Node('screenshots'))                <4>
+    
+    proj.link('http://issues.lumiera.org/roadmap', label="Roadmap (Trac)") <5>
+    Node('rfc').sortChildren()
+--------------------------------------------------------------------------
+<1> the _root node_ by convention uses a special ID token. Additional
+    fields of the node object can be given as named parameters. Here
+    we define the visual menu label to be ``Lumiera''
+<2> a child node `root/project` is attached. Note: this node will
+    later be picked up, when the actual page discovery delves down
+    into the 'project' subdirectory and encounters a 'index.txt'
+    there. Index files are always searced _within_ the directory;
+    they may be called `index.txt` or use the same name as the
+    enclosing directory.
+<3> this placement directive defines that a node `screenshots`
+    shall be prepended at the start of the list. Because such a node
+    doesn't yet exist, a new node `root/project/screenshots` is
+    created as a side-effect.
+<4> this directive places an entry after another entry, which is
+    assumed to exist when this directive gets applied finally.
+    All placement directives get applied in order of definition,
+    just before the output for a given node is generated.
+    Note also the constructor syntax +Node(\'screenshots')+: here
+    the constructor just acts as a general factory function; either
+    it creates a new node, or it fetches an existing node with matching
+    node path from the internal +NodeIndex+
+<5> here we create a submenu entry in the project menu, featuring
+    an external link. The ID of that menu node will be derived from
+    the name in the url (here `roadmap`) -- it can be defined explicitly
+    if necessary (+id=...+)
+
+
+
+
+supported placement directives
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+
+[options="header", width="70%",cols="^m,<m,s<",frame="topbot",grid="all"]
+`------------------------`------------------------------------`---------------------------------------------------
+  internal DSL            Asciidoc source
+  Node(<id>)              -- discover `id.txt` --              create new node or retrieve existing node
+  linkChild(id)                                                basic function for attaching child node
+  linkParent(id)                                               basic function to attach below parent
+  putChildLast(id)        [attach] child <id>                  move child to current end of list
+  appendChild(id)         [append] child <id>
+  putChildFirst(id)                                            move child to current list start
+  prependChild(id)        prepend [child] <id> 
+  putChildAfter(id,ref)   [attach|put] child <id> after <ref>  move child after the given ref entry
+  link(url[,id][,label])  [child <id>] link ::<url>[<label>]    attach an entry, holding an external link
+  Node(<id>,label=<lbl>)  label|title <lbl>                    define the visible text in the menu entry
+  sortChildren()          sort [children]                      sort all children currently in list
+  enable(False)           off|disable|deactivate               make node passive; any children/parents added later are ignored
+  enable([True])          on|active|activate                   make node active again (this is the default)
+  detach()                detach                               cut away any parents and children, disable the node
+  discover(includes=...)  include <token>[,<token>]            instead of current dir, navigate to the _relative path_ 
+  discover(excludes=...)  exclude <token>[,<token>]            after discovering, filter names matching the <token> (without extension)
+--------------------------------------------------------------------------------------------------------------------
+
+
+commandline options
+^^^^^^^^^^^^^^^^^^^
+The behaviour of the +menugen+ script can be influenced by some options:
+predefined:: using the built-in predefined nodes
+scan::    discover nodes 
+debug::   dump data structure after discovery
+text::    generate plaintext version of the menu
+webpage:: actually generate HTML / JavaScript
+
+a positional parameter denotes the start directory for discovery (default is current).
+This directory is assumed also to be the web root; any URLs are generated relative
+
+
+
+Design and Implementation notes
+-------------------------------
+The initial observation was that actually we're parsing and processing some kind of
+_Domain Specific Language_ here. Thus the general advice for such undertakings does
+apply: we should try to handle the actual language just as a thin layer on top of
+some kind of _semantic model_. In our case, this model is the menu tree to be generated,
+while the actual ``syntax tree'' is the real filesytem, holding Asciidoc files with
+embedded comments. Thus, the semantic model was developed first, and separate of the
+syntax of the specifications; it was tested to generate suitable HTML and CSS.
+
+The syntactic elements where then added as a collection of parser or matcher objects,
+each with the ability to recognise and implement one kind of placement specification.
+Each such +Placement+ subclass exposes an +acceptVerb()+ function for handling invocations
+of the internal DSL functions, and an +acceptDSL()+ function to parse and accept a 
+`//Menu:` line from some Asciidoc source file. This approach makes adding further
+configuration options simple.
+
+Another interesting question is to what extent the actual path handling and file discovery
+logic should be configurable. My reasoning is, that any attempts towards larger flexibility
+are mostly moot, because we can't overcome the fact that this *is* logic to be cast into
+program code. Extension points or strategy objects will just have the effect to tear apart
+the actual code thus will make the code harder to read. Thus I confined myself just to
+configure the index file name and file extensions.
 
 
-Notes regarding the +menugen.py+ -- written 2/2011 during our attempt
-to get the new Lumiera website online finally.
 
 Known issues
-------------
+~~~~~~~~~~~~
 
 * for sake of simplicity, there is _one_ generated container HTML element
-  per menu entry. In case this entry is a submenu, the +<ul>+-element is
-  used, _not_ the preceeding headline +<li>+ -- this is due to the fact
+  per menu entry. In case this entry is a submenu, the `<ul>`-element is
+  used, _not_ the preceding headline `<li>` -- this is due to the fact
   that this submenu entry is going to be collapsed eventually, but has
   the side-effect of highlighting _only_ that submenu block, _not_ the
-  preceeding headline.
+  preceding headline.
 * the acceptable DSL syntax needs to be documented manually; there is
   no way to generate this information. Doing so would require to add
   specific information methods into Placement subclasses, and it would
   result in duplicated information between the regular expressions
   and the informations returned by such information methods.
   This was deemed dangerous.
-* the +__repr__+ of the Placement subclasses is not an _representation_
-  but rather a +__str__+ -- but unfortunately the debugger in PyDev
-  invokes +__repr__+
-* the startdir for automatic discovery is an global variable and indeed
-  is used in several un-obvious places and contexts
+* the +\_\_repr\_\_+ of the Placement subclasses is not an _representation_
+  but rather a +\_\_str\_\_+ -- but unfortunately the debugger in PyDev
+  invokes +\_\_repr\_\_+
+* the startdir for automatic discovery is an global variable
 * when through the use of redirection, the same file is encountered
   multiple times during discovery, it is treated repeatedly, each times
   associated with another node, because, on discovery, the node-ID is
-  generated as +parentPath/fileID+, to avoid mixing up similarily named
+  generated as +parentPath/fileID+, to avoid mixing up similarly named
   files in different directories. (The NodeIndex allows to retrieve
   a node just by its bare ID, without path anyway)
 * no escaping: currently any variable text is written to the generated 
-  HTML without any sanitising or escaping. This might be a security issue
+  HTML without any sanitising or escaping. This might be a security issue,
+  especially because Git pushes immediately trigger menu generation.
 * the method Node.matches() is implemented sloppily: it uses just a mutual
   postfix match, while actually it should line up full path components and
   check equality on components, starting from the path end. This cheesy
   implementation can yield surprising side-effects: e.g. an not-yet attached
-  node +'end'+ could match a new menu page +'documentation/backend'+
+  node `\'end'` could match a new menu page `\'documentation/backend'`