Library: attempt workaround to problem with references

There is an insidious problem when the Transformer takes references to internal state
within upstream iterators or state core. This problem only manifests when
a invariant based filtering or grouping operation is added after the Transformer,
because such an operation (notably Filter) will typically attempt to establish
the invariant from the constructor (to avoid dangling state). Unfortunately
doing so involves pulling data ''before the overall pipeline is moved into final location''

A workaround is to make the Transformer ''disengage'' on copy, so to provoke
a refresh and new pull in the new location after the copy / move / swap.
This only works if the transformer function as such is idempotent.

doc/technical/code/codingGuidelines.txt

@@ -78,6 +78,11 @@ The Lumiera project uses GNU indentation style with slight adaptations.
   `&`,`|`, `^`, sometimes also for `!` -- using those is taken as an indication of
   entering the ``danger zone''...
 
+
+Spelling
+~~~~~~~~
+Lumiera uses _British spelling._ Please set your spell checker accordingly.
+
 Naming conventions
 ~~~~~~~~~~~~~~~~~~
 Naming conventions are used to characterise the kind of element at hand and give a visual
@@ -110,11 +115,6 @@ In case a definition actually denotes an object, there should be
 The object pointer/handle should be passed as 1^st^ argument with the name +self+
 
 
-Spelling
-~~~~~~~~
-Lumiera uses _British spelling._ Please set your spell checker accordingly.
-
-
 General Code Arrangement and Layout
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 - Headers and translation units are named `*.hpp` and `*.cpp` rsp. `*.h` and `*.c` +

doc/technical/howto/crackNuts.txt

@@ -119,7 +119,7 @@ Lumiera iterators::
   construct and in while-loops.
 IterExplorer::
   The function `lib::explore(IT)` builds on top of these features and is meant to basically
-  iterate anything that is iterable -- so use it to abstract away the details.
+  iterate anything that is iterable -- this can be used to level and abstract away the details.
 +
 - can be filtered and transformed
 - can be reduced or collected into a vector

src/lib/iter-explorer.hpp

@@ -79,10 +79,11 @@
  ** 
  ** @warning all builder operations work by _moving_ the existing pipeline built thus far into the parent
  **          of the newly built subclass object. The previously existing pipeline is defunct after that
- **          move; if you captured it into a variable, be sure to capture the _result_ of the new
+ **          move; if you captured it into a variable, be sure to capture the _result_ of the new builder
- **          builder operation as well and don't use the old variable anymore. Moreover, it should
+ **          operation as well and don't use the old variable anymore. An insidious trap can be to store
- **          be ensured that any "state core" used within IterExplorer has an efficient move ctor;
+ **          references to source iterator state _from within_ the pipeline. Moreover, it should be ensured
- **          including RVO, the compiler is typically able to optimise such move calls away altogether.
+ **          that any "state core" used within IterExplorer has an efficient move ctor; including RVO,
+ **          the compiler is typically able to optimise such move calls away altogether.
  ** 
  ** @see IterExplorer_test
  ** @see iter-adapter.hpp
@@ -763,6 +764,10 @@ namespace lib {
      *       `operator->` on any iterator downstream. This is also the reason why the
      *       ItemWrapper is necessary, precisely _because we want to support_ functions
      *       producing a value; it provides a safe location for this value to persist.
+     * @warning handling a transformer function which exposes references can be dangerous.
+     *       For this reason, Transformer attempts to »dis-engage« on each copy / assignment,
+     *       in order to provoke a re-invocation of the transformer function, which hopefully
+     *       picks up references to the copied / moved / swapped location. Be careful though!
      */
     template<class SRC, class RES>
     class Transformer
@@ -782,15 +787,38 @@ namespace lib {
         using pointer    = typename meta::ValueTypeBinding<RES>::pointer;
         
         
-        Transformer() =default;
-        // inherited default copy operations
-        
         template<typename FUN>
         Transformer (SRC&& dataSrc, FUN&& transformFunctor)
           : SRC{move (dataSrc)}                            // NOTE: slicing move to strip IterExplorer (Builder)
           , trafo_{_FunTraits<FUN,SRC>::adaptFunctor (forward<FUN> (transformFunctor))}
           { }
         
+        Transformer() =default;
+        Transformer (Transformer const& o)
+          : SRC{o}
+          , trafo_{o.trafo_}
+          , treated_{/* deliberately empty: force re-engage */}
+          { }
+        Transformer (Transformer && o)
+          : SRC{move (o)}
+          , trafo_{move (o.trafo_)}
+          , treated_{/* deliberately empty: force re-engage */}
+          { }
+        Transformer&
+        operator= (Transformer changed)
+          {
+            swap (*this,changed);
+            return *this;
+          }
+        friend void
+        swap (Transformer& t1, Transformer& t2)
+          {
+            using std::swap;
+            t1.treated_.reset();
+            t2.treated_.reset();
+            swap (t1.trafo_, t2.trafo_);
+          }
+        
         
         /** refresh state when other layers manipulate the source sequence
          * @remark expansion replaces the current element by a sequence of
@@ -981,6 +1009,8 @@ namespace lib {
      *       This limitation was deemed acceptable (adapting a function with several arguments would
      *       require quite some nasty technicalities). The first argument of this `aggFun` refers
      *       to the accumulator by value, and thereby also implicitly defines the aggregate result type.
+     * @warning the Aggregator \a AGG *must not capture references* to upstream internal state, because
+     *       the overall pipeline will be moved into final location _after the initial ctor call._
      */
     template<class SRC, typename AGG, class GRP>
     class GroupAggregator

wiki/thinkPad.ichthyo.mm

@@ -19134,9 +19134,7 @@
 </node>
 <node CREATED="1664834616551" ID="ID_1590264446" MODIFIED="1664834836962" TEXT="in der Endversion wird uns die Hysterese sch&#xfc;tzen">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       ...unter der Annahme, da&#223; das K&#252;rzen ggfs.auch verl&#228;ngern kann, und damit schon relativ nahe am verf&#252;gbaren Platz ist. Dann verhindert die Hysterese, da&#223; nochmal gepr&#252;ft wird
@@ -19590,9 +19588,7 @@
 <node CREATED="1665873235907" ID="ID_438342430" MODIFIED="1665873254132" TEXT="hier soll sich das ElementBoxWidget m&#xf6;glichst passiv verhalten"/>
 <node CREATED="1665873270653" ID="ID_340799218" MODIFIED="1665873303819">
 <richcontent TYPE="NODE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       es gibt <i>anderswo </i>einen dedizierten &quot;Content Controller&quot;
@@ -20045,9 +20041,7 @@
 </node>
 <node CREATED="1666284884372" ID="ID_1055383793" MODIFIED="1666285096886">
 <richcontent TYPE="NODE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       liegt an der <i>inkrementellen Natur</i>&#160;der Layout-Zuteilung
@@ -20593,9 +20587,7 @@
 <icon BUILTIN="messagebox_warning"/>
 <node CREATED="1480724761876" ID="ID_335451126" MODIFIED="1576282358098" TEXT="wegen BusTerm">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       braucht feste Speicher-Addresse
@@ -21256,9 +21248,7 @@
 </node>
 <node CREATED="1575220914655" ID="ID_646969276" MODIFIED="1575220994424" TEXT="nur TrackPresenter oder DisplayFrame k&#xe4;men in Frage">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       weil nach hier etablierter Policy diesen erlaubt w&#228;re, &quot;von oben&quot; in den trackPresenter.displayFrame.trackBody reinzugreifen f&#252;r die startLine_
@@ -23206,9 +23196,7 @@
   </body>
 </html></richcontent>
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       ...hab ich mich je anders entschieden?
@@ -26893,9 +26881,7 @@
 <node CREATED="1563469368703" ID="ID_856512794" MODIFIED="1563469385040" TEXT="jeder weitere Sub-Track h&#xe4;ngt eine Zeile an"/>
 <node CREATED="1563469415896" ID="ID_307382994" MODIFIED="1563469442614">
 <richcontent TYPE="NODE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       aber verschachtelte sub-Tracks werden <i>in</i>&#160;dieser gehandhabt
@@ -37050,9 +37036,7 @@
 <icon BUILTIN="stop-sign"/>
 <node CREATED="1533310564980" ID="ID_1796116534" MODIFIED="1576282358027" TEXT="ist es nicht...">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       nur bei laufender Event-Loop
@@ -40250,9 +40234,7 @@
 </node>
 <node CREATED="1666964041691" ID="ID_190646112" MODIFIED="1666964239733">
 <richcontent TYPE="NODE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p style="text-align: center">
       Menu
@@ -41827,9 +41809,7 @@
 <node CREATED="1670608693763" ID="ID_1403342865" MODIFIED="1670608757928" TEXT="genauere Betrachtung zeigt: hier keine Gefahr"/>
 <node CREATED="1670608703396" ID="ID_1449271152" MODIFIED="1670960623932" TEXT="Duration wird errechnet und ist daher ggfs. sehr gro&#xdf;, aber nicht giftig">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       der Nenner ist limitiert (kleiner Time::SCALE) und nicht toxisch
@@ -42665,9 +42645,7 @@
 <icon BUILTIN="ksmiletris"/>
 <node CREATED="1670618176639" ID="ID_393011013" MODIFIED="1670618207065">
 <richcontent TYPE="NODE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       und die Testsuite ist auf Anhieb <b><font color="#0ba911">GR&#220;N</font></b>
@@ -43340,9 +43318,7 @@
 </node>
 <node COLOR="#338800" CREATED="1670889821294" ID="ID_842321492" MODIFIED="1670889891273" TEXT="aufgekl&#xe4;rt: war t&#xfc;ckische Falle in der Metrik-Optimierung">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       ...und der ganze Ansatz mit Newton-N&#228;herung stellte sich als unsinnig heraus; jetzt rechnen wir die Optimierung mit einem Schu&#223; in float, und reizen dann sogar noch den Headroom aus, wodurch die Werte viel genauer werden
@@ -43693,9 +43669,7 @@
 <icon BUILTIN="button_ok"/>
 <node CREATED="1668353519645" ID="ID_1709187608" MODIFIED="1668474384018" TEXT="so exakt wie m&#xf6;glich">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       da reichen meine Algebra-Kenntnisse nicht aus....
@@ -43933,9 +43907,7 @@
 </node>
 <node CREATED="1669673828418" ID="ID_1875664128" MODIFIED="1669673980107" TEXT="f&#xfc;r solche F&#xe4;lle w&#xfc;rde sogar die &quot;Bereinigung&quot; selber entgleisen">
 <richcontent TYPE="NOTE"><html>
-  <head>
+  <head/>
-    
-  </head>
   <body>
     <p>
       Und zwar, wenn der Nenner viel kleiner ist als der Z&#228;hler, und der Z&#228;hler extrem gro&#223;. Dann w&#252;rde n&#228;mlich die Ganzzahl-Division keine signifikante Verringerung der Dimension bewirken, und die anschlie&#223;ende re-Quantisierung das Ergebnis (bedingt durch die Normierung auf einen gemeinsamen Nenner) sogar noch vergr&#246;&#223;ern
@@ -53155,6 +53127,45 @@
 </node>
 <node BACKGROUND_COLOR="#fdfdcf" COLOR="#ff0000" CREATED="1732335516273" ID="ID_77601" MODIFIED="1732335551076" TEXT="Konsequenz: KEINE Aktivit&#xe4;t in einem Konstruktor darf Referenzen binden">
 <icon BUILTIN="flag-pink"/>
+<node BACKGROUND_COLOR="#f8f1cb" COLOR="#a50125" CREATED="1732336927298" ID="ID_1133143344" MODIFIED="1732336965422" TEXT="das betrifft auch Weitere, z.B. das Grouping">
+<icon BUILTIN="messagebox_warning"/>
+</node>
+<node CREATED="1732336967005" ID="ID_643449650" MODIFIED="1732337008685">
+<richcontent TYPE="NODE"><html>
+  <head/>
+  <body>
+    <p>
+      ich brauche einen <b>engaged</b>-State
+    </p>
+  </body>
+</html></richcontent>
+<node CREATED="1732337022157" ID="ID_544252009" MODIFIED="1732337084587" TEXT="das k&#xf6;nnte dann weiteres Verschieben sperren"/>
+<node CREATED="1732337085316" ID="ID_487485041" MODIFIED="1732337105606" TEXT="zumindest darf Invarianten-Processing erst mit diesem Schirt passieren"/>
+<node CREATED="1732372728476" ID="ID_1531001529" MODIFIED="1732373067818" TEXT="&#xe4;rgerlich: bl&#xe4;ht Storage auf und bricht Invarianten"/>
+</node>
+<node BACKGROUND_COLOR="#c8c0b6" COLOR="#435e98" CREATED="1732377283840" ID="ID_1863518121" MODIFIED="1732378140499" TEXT="versuche Workaround: re-engage">
+<icon BUILTIN="help"/>
+<icon BUILTIN="idea"/>
+<node CREATED="1732377307860" ID="ID_410007394" MODIFIED="1732377339117" TEXT="das Problem besteht einzig im Transformer">
+<icon BUILTIN="idea"/>
+</node>
+<node CREATED="1732377323718" ID="ID_250925762" MODIFIED="1732377333805" TEXT="die Grouper kopieren n&#xe4;mlich per-Value"/>
+<node CREATED="1732377340359" ID="ID_1293483567" MODIFIED="1732377393572" TEXT="der Transformer hat schon impliziten &quot;engaged&quot;-State">
+<richcontent TYPE="NOTE"><html>
+  <head/>
+  <body>
+    <p>
+      ...er l&#228;&#223;t n&#228;mlich den ItemWrapper zun&#228;chst leer, und verwendet das f&#252;r eine Lazy-invocation der Transformation
+    </p>
+  </body>
+</html></richcontent>
+</node>
+<node CREATED="1732377394409" ID="ID_205213439" MODIFIED="1732377416893" TEXT="also w&#xfc;rde es gen&#xfc;gen, da&#xdf; jedes Copy/Move &#x27fc; disengaged"/>
+<node CREATED="1732377419435" ID="ID_343740069" LINK="https://stackoverflow.com/a/3279550/444796" MODIFIED="1732377731172" TEXT="Nebenbei: copy-and-swap-Idiom verwenden"/>
+<node COLOR="#338800" CREATED="1732378142413" ID="ID_1308227786" MODIFIED="1732378149468" TEXT="best&#xe4;tigt: l&#xf6;st das Problem">
+<icon BUILTIN="button_ok"/>
+</node>
+</node>
 </node>
 </node>
 </node>