diff --git a/doc/technical/code/darkCorners.txt b/doc/technical/code/darkCorners.txt index 8e40f0831..705449ebf 100644 --- a/doc/technical/code/darkCorners.txt +++ b/doc/technical/code/darkCorners.txt @@ -43,3 +43,9 @@ When relying on that hack, we should make sure always to place some kind of `static_assert` into the corresponding implementation files to ensure the real facilites actually _do fit_ into the guessed storage dimensions. + +Portability +----------- +- Linux-only solution to discover the path of the executable: `findExePath` in 'lib/searchpath.hpp' +- hard-coded usage of `/dev/urandom` (e.g. in 'lib/random.cpp' as spec for the `std::random_device` +- using the 64bit Murmur-2.64A hash function in '/lib/hash-combine.hpp' \ No newline at end of file diff --git a/doc/technical/howto/HashFunctions.txt b/doc/technical/howto/HashFunctions.txt index d7a848880..07e353195 100644 --- a/doc/technical/howto/HashFunctions.txt +++ b/doc/technical/howto/HashFunctions.txt @@ -90,6 +90,14 @@ calculated hash values of the parts forming a composite data structure. uses the standard implementation of a string hash function combining the individual character's hashes. +Hash-chaining +~~~~~~~~~~~~~ +We use a dedicated function `lib::hash::combine(s,h)` to join several source(component) hashes. +This usage pattern was pioneered by Boost and is based on the +https://github.com/aappleby/smhasher/blob/master/src/MurmurHash2.cpp[Murmur-2.64A] hash algorithm. + +WARNING: as of [yellow-background]#11/2024#, portability of hash values is an unresolved issue; + this code does not work on 32bit systems https://issues.lumiera.org/ticket/722#comment:10[see #722] LUID values @@ -103,9 +111,10 @@ the meltdown of an atomic power plant, which, as we all know, won't ever happen Relation to hash values ~~~~~~~~~~~~~~~~~~~~~~~ -When objects incorporate sich an unique LUID, this provides for a prime candidate to +When objects incorporate such an unique LUID, this provides for a prime candidate to derive hash values as a side-effect of that design: Since incorporating an LUID typically means that this object has an _distinguishable identity_, all objects with the same LUID should be considered _equivalent_ and thus hash to the same value. Consequently we can just use a +size_t+ prefix of the LUID bitstring as hash value, without any further calculations. +This relies on LUID being generated from a reliable _entropy source._ diff --git a/doc/technical/howto/crackNuts.txt b/doc/technical/howto/crackNuts.txt new file mode 100644 index 000000000..96f753f5b --- /dev/null +++ b/doc/technical/howto/crackNuts.txt @@ -0,0 +1,117 @@ +how to crack nut #47 +==================== +:toc: + +.collection of successfully employed technical solutions +Some nasty problems are recurring time and again. Maybe a trick could be found somewhere +in the net, and a library function was created to settle this damn topic once and for all. +Maybe even a nice test and demo is provided. And then the whole story will be forgotten. + +_Sounds familiar?_ => ☹☻☺ ~[red]#then please leave a note here...#~ + + +Methods +------- +Mathematics +~~~~~~~~~~~ +- some basic descriptive statistics computation are defined in 'lib/stat/statistic.hpp' +- the simple case for _linear regression_ is also implemented there +- Gnuplot provides also common statistics functions, which may come in handy when the + goal is anyway to create a visualisation (-> see <>) + + + + +Situations +---------- +Investigation +~~~~~~~~~~~~~ +summary test:: + Reformulate the research and the findings into a test, which can be read top-down like a novel. + Start with documenting the basics, package helpers into a tool class, or package setup into a + workbench-style class, with individual tool modules. Provide a short version of this test with + the basic demo, which should be able to run with the regular test suite. Extended long-running + tests can be started conditionally with commandline-arguments. See 'scheduler-stress-test.cpp' +visualisation:: + Use code generation to visualise data structures or functions and observation statistics. + Typically these generation statements can be packaged into an invocation helper and included + into a relevant test, but commented-out there. ++ +- generate Graphviz diagrams: 'lib/dot-gen.hpp' provides a simple DSL. See 'test-chain-load-test.cpp' +- generate Gnuplot scripts: use the Text-Template engine to fill in data, possibly from a data table + + * 'lib/gnuplot-gen.hpp' provides some pre-canned scripts for statistics plots + * used by the 'stress-test-rig.cpp', in combination with 'data.hpp' to collect measurement results + + + +Common Tasks +------------ +Data handling +~~~~~~~~~~~~~ +persistent data set:: + use the `lib::stat::DataTable` ('data.hpp') with CSV rendering -> see 'data-csv-test.cpp' + + +Formatting +~~~~~~~~~~ +- implement a conversion-to-string operator. +- include the C++ IOStreams via 'lib/format-cout.hpp' -> this magically uses the `util::toString()` +- for testing, temporarily include 'lib/test/diagnostic-output' and use the `SHOW_EXPR` macro. +- use 'util::join' ('lib/format-util.hpp') to join arbitrary elements with `util::toString()` conversion +- use _printf-style formatters_ from Boost-format. We provide a light-weight front-end via 'lib/format-string.hpp' + + * the heavyweight boost-format include is only required once, for 'lib/format-string.cpp'. + * the templated front-end passes-through most basic types and types with string-conversion + * all invocations are strictly error safe (never throw) and can thus be used from catch-handlers + +- use the *Text-Template* engine. See 'text-template-test.cpp'. Can be used with simple map bindings, + but also from a `lib::GenNode` tree, or with a custom defined `DataSource` template + + +Language constructs +------------------- +Templates +~~~~~~~~~ +build-from-anything:: + use a templated constructor, possibly even with varargs ++ +- use a _deduction guide_ to pick the right ctor and arguments -> see + + * `ThreadJoinable` in 'thread.hpp', 698 + * `DataSource` specialisation only when argument can be converted to string, + in 'text-template.hpp', 745 + +- prevent shadowing of _automatically generated copy operations._ + See https://issues.lumiera.org/ticket/963[#963]. Based on the ``disable if'' SFINAE technique. + A ready-made templated typedef `lib::metadisable_if_self` can be found in 'lib/meta/util.hpp' + + +Varargs +~~~~~~~ +pick and manipulate individually:: + The key trick is to define an _index sequence template,_ which can then be matched against + a processing template for a single argument; and the latter can have partial specialisations ++ +- see 'variadic-argument-picker-test.cpp' +- but sometimes it is easier to use the tried and true technique of the Loki-Typelists, which + can be programmed recursively, similar to LISP. The »bridge« is to unpack the variadic argument pack + into the `lib::meta::Types` ([yellow-background]#⚠ still broken in 2024# + see https://issues.lumiera.org/ticket/987[#987], use `lib::meta::TySeq` from 'variadic-helper.hpp' as workaround... ++ +apply functor to each:: + A common trick is to use `std::apply` in combination with a _fold-expression_ ++ +- provided as `lib::meta::forEach` in 'lib/meta/util.hpp +- The design of the `DataTable` with CSV-Formatting is based on this technique, see 'lib/stat/data.hpp' +- 'test-rand-ontology.cpp' uses this in `manipulateFrame()` to accept an arbitrary number of input chains ++ +unpack iterator into tuple:: + Under controlled conditions this is possible (even while it seems like time travel from the runtime into + the compile-time domain). The number of results to extract from the iterator must be known at compile time, + and the possible result types must be limited, so that a visitor can be used for double-dispatch. ++ +- see tuple-record-init-test.cpp +- used in 'command-simple-closure.hpp' to receive parameter values sent via UI-Bus and package them + into a tuple for invocation of a Steam-Layer command. + diff --git a/src/lib/hash-combine.hpp b/src/lib/hash-combine.hpp index cf651a850..90bb34ea5 100644 --- a/src/lib/hash-combine.hpp +++ b/src/lib/hash-combine.hpp @@ -51,10 +51,10 @@ namespace lib { namespace hash{ - /** meld the additional hash value into the given - * base hash value. This is the standard formula - * used by Lib-Boost to combine the hash values - * of parts into a composite. + /** meld the additional hash value into the given base hash value. + * This is the standard formula used by Lib-Boost to combine the hash values + * of parts into a composite, and is based on the [Murmur-2.64A] hash algorithm. + * [Murmur-2.64A]: https://github.com/aappleby/smhasher/blob/master/src/MurmurHash2.cpp */ inline void combine (size_t & combinedHash, size_t additionalHash) diff --git a/tests/core/steam/engine/node-devel-test.cpp b/tests/core/steam/engine/node-devel-test.cpp index 68c83b28c..cc4c24e48 100644 --- a/tests/core/steam/engine/node-devel-test.cpp +++ b/tests/core/steam/engine/node-devel-test.cpp @@ -17,6 +17,7 @@ #include "lib/test/run.hpp" +#include "lib/hash-combine.hpp" #include "steam/engine/test-rand-ontology.hpp" ///////////TODO #include "lib/test/diagnostic-output.hpp"/////////////////TODO #include "lib/random.hpp" @@ -40,9 +41,15 @@ namespace test { alignas(TestFrame) std::byte storage[sizeof(TestFrame)]; - operator TestFrame* () { return std::launder (reinterpret_cast (&storage)); } - TestFrame* operator->() { return std::launder (reinterpret_cast (&storage)); } - TestFrame& operator* () { return * std::launder (reinterpret_cast (&storage)); } + operator TestFrame* () { return std::launder (reinterpret_cast (&storage)); } + TestFrame* operator->() { return std::launder (reinterpret_cast (&storage)); } + TestFrame& operator* () { return * std::launder (reinterpret_cast (&storage)); } + + TestFrame& + buildData (uint seq=0, uint family=0) + { + return * new(&storage) TestFrame{seq,family}; + } }; } @@ -57,9 +64,11 @@ namespace test { run (Arg) { seedRand(); + TestFrame::reseed(); processing_generateFrame(); processing_generateMultichan(); + processing_manipulateFrame(); } @@ -102,6 +111,42 @@ namespace test { CHECK (*(buffs[i]) == TestFrame(frameNr,flavour+i)); } } + + /** @test function to apply a numeric computation to test data frames + */ + void + processing_manipulateFrame() + { + size_t frameNr = defaultGen.u64(); + uint flavour = defaultGen.u64(); + + Buffer iBuff, oBuff; + iBuff.buildData(frameNr,flavour); + oBuff.buildData(frameNr,flavour); + CHECK (iBuff->isPristine()); + CHECK (iBuff->isPristine()); + + uint64_t param = defaultGen.u64(); + manipulateFrame (oBuff, iBuff, param); + CHECK ( oBuff->isValid()); + CHECK (not oBuff->isPristine()); + CHECK ( iBuff->isPristine()); + + for (uint i=0; idata64().size(); ++i) + { + uint64_t feed = param; + uint64_t data = iBuff->data64()[i]; + lib::hash::combine (feed, data); + CHECK (data == iBuff->data64()[i]); + CHECK (feed != iBuff->data64()[i]); + CHECK (feed == oBuff->data64()[i]); + } + // can also process in-place + manipulateFrame (iBuff, iBuff, param); + CHECK (not iBuff->isPristine()); + CHECK ( iBuff->isValid()); + CHECK (*iBuff == *oBuff); // second invocation exactly reproduced data from first invocation + } }; diff --git a/tests/core/steam/engine/test-rand-ontology.cpp b/tests/core/steam/engine/test-rand-ontology.cpp index d08b0c9d4..eea99625e 100644 --- a/tests/core/steam/engine/test-rand-ontology.cpp +++ b/tests/core/steam/engine/test-rand-ontology.cpp @@ -13,11 +13,18 @@ /** @file test-rand-ontology.cpp ** Implementation of fake data processing to verify invocation logic. + ** The emulated »media computations« work on TestFrame data buffers, which can be + ** filled with deterministically generated pseudo-random data, that can be verified afterwards. + ** Computations manipulate or combine individual data points, and mark the result again with a + ** valid checksum. Hash-chaining computations are used in order to ensure that the resulting + ** data values depend on all input- and parameter values, and the _exact order_ of processing. + ** All computations are reproducible, and thus a test can verify a computation carried out + ** within the context of the Render-Engine code. */ #include "steam/engine/test-rand-ontology.hpp" -#include "lib/error.hpp" +#include "lib/hash-combine.hpp" //#include @@ -70,18 +77,19 @@ namespace test { /** * @param out existing allocation to place the generated TestFrame into * @param in allocation holding the input TestFrame data - * @param param parameter to control the data manipulation (to be multiplied into the data) - * @remark this function emulates „media data processing“: each byte of the input data is multiplied - * with the given \a param, wrapping each result into the corresponding output byte. The - * generated result TestFrame is marked with a valid checksum. + * @param param parameter to control or »mark« the data manipulation (hash-combining) + * @remark this function emulates „media data processing“: data is processed in 64-bit words, + * by hash-chaining with \a param. The generated result is marked with a valid checksum. */ void - manipulateFrame (TestFrame* out, TestFrame* in, int param) + manipulateFrame (TestFrame* out, TestFrame const* in, uint64_t param) { REQUIRE (in); REQUIRE (out); - for (size_t i=0; i < in->data().size(); ++i) - out->data()[i] = char(param * in->data()[i]); + auto calculate = [](uint64_t chain, uint64_t val){ lib::hash::combine(chain,val); return chain; }; + for (size_t i=0; i < in->data64().size(); ++i) + out->data64()[i] = calculate(param, in->data64()[i]); + out->markChecksum(); } /** @@ -93,13 +101,14 @@ namespace test { * each result byte is the linear interpolation between the corresponding inputs. */ void - combineFrames (TestFrame* out, TestFrame* srcA, TestFrame* srcB, int mix) + combineFrames (TestFrame* out, TestFrame const* srcA, TestFrame const* srcB, int mix) { REQUIRE (srcA); REQUIRE (srcB); REQUIRE (out); for (size_t i=0; i < srcA->data().size(); ++i) out->data()[i] = char((1-mix) * srcA->data()[i] + mix * srcB->data()[i]); + out->markChecksum(); } diff --git a/tests/core/steam/engine/test-rand-ontology.hpp b/tests/core/steam/engine/test-rand-ontology.hpp index d29b6a967..76b21fe84 100644 --- a/tests/core/steam/engine/test-rand-ontology.hpp +++ b/tests/core/steam/engine/test-rand-ontology.hpp @@ -40,11 +40,11 @@ namespace test { /** produce planar multi channel output of random data frames */ void generateMultichan (uint chanCnt, TestFrame* buffArry, size_t frameNr, uint flavour); - /** »process« random frame date by multiply-wrapping with a parameter */ - void manipulateFrame (TestFrame* out, TestFrame* in, int param); + /** »process« random frame date by hash-chaining with a parameter */ + void manipulateFrame (TestFrame* out, TestFrame const* in, uint64_t param); /** mix two random data frames by a parameter-controlled proportion */ - void combineFrames (TestFrame* out, TestFrame* srcA, TestFrame* srcB, int mix); + void combineFrames (TestFrame* out, TestFrame const* srcA, TestFrame const* srcB, int mix); diff --git a/tests/core/steam/engine/testframe-test.cpp b/tests/core/steam/engine/testframe-test.cpp index d453de936..eba99406c 100644 --- a/tests/core/steam/engine/testframe-test.cpp +++ b/tests/core/steam/engine/testframe-test.cpp @@ -116,6 +116,13 @@ namespace test { for (uint i=0; i; + using _A64 = std::array; struct Meta { @@ -128,8 +129,10 @@ namespace test { friend bool operator!= (TestFrame const& f1, TestFrame const& f2) { return !f1.contentEquals(f2); } /** Array-style direct access to the payload data */ - _Arr& data() { return * std::launder (reinterpret_cast<_Arr* > (&buffer_)); } - _Arr const& data() const { return * std::launder (reinterpret_cast<_Arr const*> (&buffer_)); } + _Arr& data() { return * std::launder (reinterpret_cast<_Arr* > (&buffer_)); } + _Arr const& data() const { return * std::launder (reinterpret_cast<_Arr const*> (&buffer_)); } + _A64& data64() { return * std::launder (reinterpret_cast<_A64* > (&buffer_)); } + _A64 const& data64() const { return * std::launder (reinterpret_cast<_A64 const*> (&buffer_)); } private: void buildData(); diff --git a/wiki/thinkPad.ichthyo.mm b/wiki/thinkPad.ichthyo.mm index 8c023d4df..80e158df1 100644 --- a/wiki/thinkPad.ichthyo.mm +++ b/wiki/thinkPad.ichthyo.mm @@ -18615,9 +18615,7 @@ - - - +

...das heißt... @@ -18956,9 +18954,7 @@ - - - +

Mehrstufige Prüfung mit Hysterese (um Flackern zu vermeiden)... @@ -19342,9 +19338,7 @@ - - - +

Aua! @@ -19794,9 +19788,7 @@ - - - +

Naja, stimmt nicht wirklich, denn Pixel-Angaben müssen angepaßt werden, also besteht faktisch eine Verbindung @@ -20645,9 +20637,7 @@ - - - +

..d.h. der Controller muß wieder auf das Widget zugreifen @@ -22170,9 +22160,7 @@ - - - +

...ich wollte dadurch ausdrücken, daß das übergebene ViewHooked<Widget>& ursprünglich schon einmal geHooked worden war. Tatsächlich hat ja im originalen Design der ViewHook das Hookable sogar erst konstruiert, und niemand sonst konnte das. Da wir aber nun inzwischen immer mit einem ViewHookable mit eingebettetem Widget arbeiten, muß dieses freistehend konstruiert werden, und des gibt keine direkte Möglichkeit mehr, diese "Verdongelung" auszudrücken. Und außerdem sind auch alle weiteren Ideen aufgegeben, welche auf eine engere Verzahnung der Interfaces aufbauen würden (Stichwort "quer-Beweglichkeit"). @@ -25510,9 +25498,7 @@ - - - +

damit die Höhen für die Kind-Tracks bereits gesetzt sind, wenn die Gesamthöhe des Parent-Track betrachtet wird @@ -38913,9 +38899,7 @@ - - - +

  • @@ -41151,9 +41135,7 @@ - - - +

    Der Grenzfall ist ja changedMetric = MAX_ZOOM @@ -42692,9 +42674,7 @@ - - - +

    weil hier... @@ -43736,9 +43716,7 @@ - - - +

    ...in vertretbarem Rahmen... @@ -44239,9 +44217,7 @@ - - - +

    ⟹ also muß man hier technisch runden @@ -45849,9 +45825,7 @@ - - - +

    schon analog zu dem, was wir hier mit dem "timing" machen @@ -46006,9 +45980,7 @@ - - - +

    (GlobalCtx)->WindowLocator->PanelLocator @@ -92231,8 +92203,11 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
    - - + + + + + @@ -92486,7 +92461,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
     - + @@ -92503,10 +92478,11 @@ Date:   Thu Apr 20 18:53:17 2023 +0200

 - - + + + - + @@ -92526,12 +92502,12 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 - +

- ...zwar handelt es sich stets nur um eine Hash-Verknüpfung, aber die Arität (und ggfs. später auch noch ein Array-Wertiger Input) kommen als Steuerparameter hinzu. Diese Parameter können als lesbarer Spec-String ausgegeben werden, und in einen Hash eingerechnet werden, der dann die Funktion markiert und unterscheidbar macht. Wenn ein Seed explizit angegeben wird, dann fließt er zusätzlich mit ein, und markiert außerdem auch die Berechnung an jedem Datenpunkt + ...zwar handelt es sich stets nur um eine Hash-Verknüpfung, aber die Arität (und ggfs. später auch noch ein Array-wertiger Input) kommen als Steuerparameter hinzu. Diese Parameter können als lesbarer Spec-String ausgegeben werden, und in einen Hash eingerechnet werden, der dann die Funktion markiert und unterscheidbar macht. Wenn ein Seed explizit angegeben wird, dann fließt er zusätzlich mit ein, und markiert außerdem auch die Berechnung an jedem Datenpunkt

@@ -92557,7 +92533,13 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 - + + + + + + + @@ -92566,6 +92548,7 @@ Date:   Thu Apr 20 18:53:17 2023 +0200

 + @@ -92797,7 +92780,8 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
- + + @@ -92921,6 +92905,18 @@ Date:   Thu Apr 20 18:53:17 2023 +0200
 + + + + +

+ Im Datenblock liegt lediglich eine Sequenz von 1014 Bytes — man möchte darauf jedoch flexibel aber getypt zugreifen können +

+ + + + +