Iterators and Pipelines ======================= The http://wiki.c2.com/?IteratorPattern[Iterator Pattern] allows to expose the contents or elements of any kind of collection, set or container for use by client code, without exposing the implementation of the underlying data structure. Thus, iterators are one of the primary API building blocks. Lumiera Forward Iterator ------------------------ While most modern languages provide some kind of _Iterator,_ the actual semantics and the fine points of the implementation vary greatly from language to language. Unfortunately the C++ standard library uses a very elaborate and rather low-level notion of iterators, which does not mix well with the task of building clean interfaces. Thus, within the Lumiera core application, we're using our own Iterator concept, initially defined as link:/x/rfc/LumieraForwardIterator.html[RfC], which places the primary focus on interfaces and decoupling, trading off readability and simplicity for (lesser) performance. .Definition An Iterator is a self-contained token value, representing the promise to pull a sequence of data - rather then deriving from an specific interface, anything which behaves appropriately _is a Lumiera Forward Iterator._ (``duck typing'' or ``a Concept'') - the client finds a typedef at a suitable, nearby location. Objects of this type can be created, copied and compared. - any Lumiera forward iterator can be in _exhausted_ (invalid) state, which can be checked through +bool+ conversion. - notably, any default constructed iterator is fixed to that exhausted state. Non-exhausted iterators may only be obtained by API call. - the exhausted state is final and can't be reset, meaning that any iterator is a disposable one-way-off object. - when an iterator is _not_ in the exhausted state, it may be _dereferenced_ (`*i`), yielding the ``current'' value - moreover, iterators may be incremented (`++i`) until exhaustion. Motivation ~~~~~~~~~~ The Lumiera Forward Iterator concept is a blend of the STL iterators and iterator concepts found in Java, C#, Python and Ruby. The chosen syntax should look fairly familiar to C++ programmers and indeed is compatible to STL containers and ranges. Yet while a STL iterator can be thought of as being just a pointer in disguise, the semantics of Lumiera Forward Iterators is deliberately more high-level, reduced to a single, one-way-off forward iteration. Lumiera Forward Iterators can't be reset, they can not be manipulated by any arithmetic, and the result of assigning to an dereferenced iterator is unspecified, as is the meaning of post-increment and stored copies in general. You _should not think of an iterator as denoting a position_ -- just treat it as an one-way off promise to yield data. Another notable difference to the STL iterators is the default ctor and the +bool+ conversion. The latter allows using iterators painlessly within +for+ and +while+ loops; a default constructed iterator is equivalent to the STL container's +end()+ value -- indeed any _container-like_ object exposing Lumiera Forward Iteration is encouraged to provide such an `end()`-function, which additionally enables iteration by `std::for_each` (or Lumiera's even more convenient `util::for_each()`), and use in ``range for''-loops. Implementation ~~~~~~~~~~~~~~ As pointed out above, within Lumiera the notion of ``Iterator'' is a concept (generic programming) and does not mean a supertype (object orientation). Any object providing a suitable set of operations can be used for iteration. - must be default constructible to _exhausted state_ - must be a copyable value object - must provide a `bool` conversion to detect _exhausted state_ - must provide a pre-increment operator (`++i`) - must allow dereferentiation (`*i`) to yield the current object - must throw on any usage in _exhausted state_. But, typically you wouldn't write all those operations again and again. Rather, there are two basic styles of iterator implementations, each of which is supported by some pre defined templates and a framework of helper functions. Iterator Adapters ^^^^^^^^^^^^^^^^^ Iterators built based on these adaptor templates are lightweight and simple to use for the implementor. But they don't decouple from the actual implementation, and the resulting type of the iterator usually is rather convoluted. So the typical usage scenario is, when defining some kind of custom container, we want to add a `begin()` and `end()` function, allowing to make it behave similar to a STL container. There should be an embedded typedef `iterator` (and maybe `const_iterator`). This style is best used within generic code at the implementation level, but is not well suited for interfaces. -> see 'lib/iter-adapter.hpp' Iteration Sources ^^^^^^^^^^^^^^^^^ Here we define a classical abstract base class to be used at interfaces. The template `lib::IterSource` is an abstract promise to yield elements of type TY. It defines an embedded type `iterator` (which is an iterator adapter, just only depending on this abstract interface). Typically, interfaces declare to return an `IterSource::iterator` as the result of some API call. These iterators will hold an embedded back-reference to ``their'' source, while the exact nature of this source remains opaque. Obviously, the price to pay for this abstraction barrier is calling through virtual functions into the actual implementation of the ``source''. Helpers to define iterators ^^^^^^^^^^^^^^^^^^^^^^^^^^^ For both kinds of iterator implementation, there is a complete set of adaptors based on STL containers. Thus, it's possible to expose the contents of such a container, or the keys, the values or the unique values just with a single line of code. Either as iterator adapter (-> see 'lib/iter-adapter-stl.hpp'), or as iteration source (-> see 'lib/iter-source.hpp') Pipelines --------- The extended use of iterators as an API building block naturally leads to building _filter pipelines_: This technique form functional programming completely abstracts away the actual iteration, focussing solely on the selecting and processing of individual items. For this to work, we need special manipulation functions, which take an iterator and yield a new iterator incorporating the manipulation. (Thus, in the terminology of functional programming, these would be considered to be ``higher order functions'', i.e. functions processing other functions, not values). The most notable building blocks for such pipelines are filtering:: each element yielded by the _source iterator_ is evaluated by a _predicate function,_ i.e. a function taking the element as argument and returning a `bool`, thus answering a ``yes or no'' question. Only elements passing the test by the predicate can pass on and will appear from the result iterator, which thus is a _filtered iterator_ transforming:: each element yielded by the _source iterator_ is passed through a _transformnig function,_ i.e. a function taking an source element and returing a ``transformed'' element, which thus may be of a completely different type than the source. Since these elements can be chained up, such a pipeline may pass several abstraction barriers and APIs, without either the source or the destination being aware of this fact. The actual processing only happens _on demand_, when pulling elements from the end of the pipeline. Oten, this end is either a _collecting step_ (pulling all elements and filling a new container) or again a IterSource to expose the promise to yield elements of the target type. Pipelines work best on _value objects_ -- special care is necessary when objects with _reference semantics_ are involved.