diff --git a/src/lib/parse.hpp b/src/lib/parse.hpp index 9295c8905..295e0b2a5 100644 --- a/src/lib/parse.hpp +++ b/src/lib/parse.hpp @@ -27,6 +27,111 @@ ** and automatically consuming any leading whitespace. And notably the focus was ** _not placed_ on the challenging aspects of parsing — while still allowing a ** pathway towards definition of arbitrarily recursive grammars, if so desired. + ** + ** ## Functionality + ** This framework supports the construction of a recursive-descent parser without + ** underlying lexer. Rather, _terminal expressions_ are delegated to a regular + ** expression matcher, which allows for some additional leeway, like a negative + ** match, or using lookahead-assertions. The implementation relies on parser + ** _functions_ and the _combinator technique_ to combine building blocks — yet + ** those functions and combinators are wrapped into a syntax-clause builder DSL. + ** Each parse yields a result, depending on the structure of the parse-function. + ** The base building block, a parser to accept a regular expression, will yield + ** the C++ matcher object as result — and thus essentially some pointers into + ** the original sequence, which has to be passed in as C++ string_view. + ** + ** An essential concept of this parsing support framework is that each parser + ** can be decorated by a _model transformation functor,_ which gets the result + ** of the wrapped parser and can return _any arbitrary value object._ In essence, + ** this framework does not provide the notion of an _abstract syntax tree_ — yet + ** the user is free to build a custom syntax tree, relying on these model-bindings. + ** + ** \par Syntax building blocks + ** - `accept(SPEC)` builds a clause to accept anything the given SPEC for + ** a parser function would accept and to yield its return model. + ** The `SPEC` argument might be... + ** + a string describing regular expression syntax + ** + a C++ regexp object + ** + any hand-written functor `string_view ⟼ model` + ** + an existing syntax clause (either by RValue or LValue) + ** - `accept_opt` builds a clause to optionally accept in accordance + ** to the given definition; if no match is found, parsing backtracks. + ** - `accept_repeated` builds a clause to accept a repetition of the + ** structure accepted by its argument, optionally with an explicit delimiter + ** and possibly with a limited number of instances. The result values are + ** obviously all from the same type and will be collected into a IterModel, + ** which essentially is a std::vector (note: heap storage!). + ** - `accept_bracket` builds a clause to accept the structure of the + ** given argument, but enclosed in parentheses, or an explicitly defined + ** pair of delimiters. Variants are provided to accept optional bracketing. + ** - `.seq(SPEC)` extends a given syntax clause to accept the structure + ** described by the SPEC _after_ the structure already described by the syntax. + ** Both parts must succeed for the parse to be successful. The result value + ** is packaged into a parse::SeqModel, which essentially is a tuple; when + ** attaching several .seq() specifications, it can become a N-ary tuple. + ** - `.alt(SPEC)` adds an _alternative branch_ to the existing syntax. + ** Either part alone is sufficient for a successful parse. First the existing + ** branch(es) are tried, and only if those do not succeed, backtracking is + ** performed and then the alternative branch is tried. Once some match is + ** found, further branches will _not be attempted._ (short-circuit). + ** Thus there is _always one_ result model, is placed into an AltModel, + ** which is a _variant data type_ with a common inline result buffer. + ** The _selector field_ must be checked to find out which branch of the + ** syntax succeeded, and then the result must be handled with its appropriate + ** type, because the various branches can possibly yield entirely different + ** result value types. + ** - `.repeat()` _sequentially adds_ a repeated clause be accepted + ** _after_ what the existing syntax accepts. The result is thus a SeqModel. + ** - `.bracket()` _sequentially adds_ a bracketing clause to be + ** accepted _after_ parsing with the existing syntax. Again, the result + ** is a SeqModel, with the result-model from the repetition in the last + ** tuple element. The repetition itself yields an IterModel. + ** - `.bind(FUN)` is a postfix-operator and decorates the existing + ** syntax with a result-binding functor `FUN`: The syntax's result value + ** is passed into this functor and whatever this functor returns will + ** become the result value of the compound. + ** - `.bindMatch(n)`: a convenience shortcut to bind to the complete + ** input substring accepted by the underlying parser, or (when directly + ** applied to a reg-exp terminal) it can also extract a match-sub-group. + ** The result value of the resulting syntax clause will thus be a string. + ** + ** ### Recursion + ** A _recursive syntax definition_ is what unleashes the parsing technique's + ** full strength; but recursive grammars can be challenging to master at times + ** and may in fact lead to deadlock due to unlimited recursion. Since this + ** framework is focused on ease of use in simple situations, recursion is + ** considered an advanced usage and thus supported in a way that requires + ** some preparation and help by the user. In essence... + ** - a syntax clause to be referred recursively _must be pre-declared_ + ** - this pre-declaration gives it a known, fixed result type and will + ** internally use a `std::function` as parse function, initially empty. + ** - later the full syntax must be defined, and supplemented with a binding + ** to yield precisely the result-model type as pre-declared + ** - finally this definition is _assigned_ to the pre-declared syntax object. + ** One point to note is that internally a _reference_ to the pre-declared + ** `std::function` is stored — implying that the pre-declared syntax object + ** must **remain in scope and can not be moved**. Other than requiring + ** such a special setup, recursive syntax can be used without limits. + ** + ** ## Structure + ** The util::parse::Syntax object is what the user handles and interacts with. + ** It both holds a result record of type util::parse::Eval, and a parser object + ** of type util::parse::Parser, while also acting as a move-style DSL builder. + ** Here, _move-style_ implies that invoking a DSL extension operator will + ** _move the embedded parser function_ into a new Syntax object with different + ** result type. However, when starting with some of the top-level function-style + ** builders, a given syntax object will be copied. This is important to keep + ** in mind when building complex expressions. Another point worth mentioning + ** is that the basic parse relies on std::string_view, which implies that the + ** original input sequence must remain valid until the parse result is built. + ** + ** At the heart of the combinator mechanics is the class util::part::Connex, + ** which is essentially a parser function with additional configuration, and + ** can be passed to one of the _combinator functions_ `buildXXXConnex(...)`. + ** The construction of compound model results relies on a set of overloaded + ** `_Join` templates, which specify the way how models can be combined and + ** sequences can be extended. + **@see parse-test.cpp */ @@ -41,7 +146,6 @@ #include "lib/meta/function.hpp" #include "lib/meta/trait.hpp" #include "lib/regex.hpp" -#include "lib/test/diagnostic-output.hpp"/////////TODO #include #include @@ -266,7 +370,7 @@ namespace util { else { // defensive fall-back: ignore model, return accepted input part size_t pre = leadingWhitespace (toParse); - return {string{toParse.substr (pre, eval.consumed)} + return {string{toParse.substr (pre, eval.consumed - pre)} ,eval.consumed }; } @@ -377,9 +481,7 @@ namespace util { /** Marker-Tag for the result from a sub-expression, not to be joined */ template struct SubModel - { - RES model; - }; + { /* for metaprogramming only */ }; /** Standard case : combinator of two model branches */ diff --git a/tests/library/parse-test.cpp b/tests/library/parse-test.cpp index 9e039b034..a4d33ba8d 100644 --- a/tests/library/parse-test.cpp +++ b/tests/library/parse-test.cpp @@ -19,15 +19,10 @@ #include "lib/test/run.hpp" #include "lib/test/test-helper.hpp" -#include "lib/parse.hpp" -//#include "lib/format-util.hpp" #include "lib/meta/tuple-helper.hpp" -#include "lib/test/diagnostic-output.hpp"//////////////////TODO -//#include "lib/util.hpp" - -//#include -//#include +#include "lib/parse.hpp" +#include namespace util { @@ -38,35 +33,15 @@ namespace test { using lib::meta::typeSymbol; using lib::meta::is_Tuple; using std::decay_t; + using std::vector; using std::get; -// using util::join; -// using util::isnil; -// using std::vector; -// using std::shared_ptr; -// using std::make_shared; - -// using LERR_(ITER_EXHAUST); -// using LERR_(INDEX_BOUNDS); - - - namespace { // test fixture - -// const uint NUM_ELMS = 10; - -// using Numz = vector; - - } // (END)fixture - - - - /************************************************************************//** - * @test verify helpers and shortcuts for simple recursive descent parsing + /****************************************************//** + * @test verify support for recursive descent parsing * of structured data and specifications. - * * @see parse.hpp * @see proc-node.cpp "usage example" */ @@ -77,6 +52,7 @@ namespace test { run (Arg) { simpleUsage(); + acceptTerminal(); acceptSequential(); acceptAlternatives(); @@ -90,13 +66,32 @@ namespace test { } - /** @test TODO just blah. */ + /** @test demonstrate parsing a function-with-arguments structure. */ void simpleUsage () { + using Model = std::pair>; + + auto word = accept("\\w+").bindMatch(); + auto term = accept(word) + .bracket (accept_repeated(",", word)) + .bind([](auto res){ return Model{get<0>(res),get<1>(res)}; }); + + CHECK (not term.hasResult()); + + term.parse("great (hypertrophy, confusion, deception, profit)"); + CHECK (term.success()); + Model model = term.getResult(); + CHECK (model.first == "great"); + CHECK (model.second[0] == "hypertrophy"); + CHECK (model.second[1] == "confusion" ); + CHECK (model.second[2] == "deception" ); + CHECK (model.second[3] == "profit" ); } + + /** @test define a terminal symbol to match by parse. */ void acceptTerminal() @@ -106,7 +101,7 @@ namespace test { string toParse{"hello vile world of power"}; auto eval = parse (toParse); CHECK (eval.result); - smatch res = *eval.result; // ◁——————————— the »result model« of a terminal parse is the RegExp-Matcher + smatch res = *eval.result; // ◁——————————————————————— »result model« of a terminal parse is the RegExp-Matcher CHECK (res.ready() and not res.empty()); CHECK (res.size() == "2"_expect ); CHECK (res.position() == "0"_expect ); @@ -133,7 +128,7 @@ namespace test { // Going full circle: extract Parser definition from syntax auto parse2 = Parser{syntax2}; - CHECK (eval.result->str(1) == "vile"); + CHECK (eval.result->str(1) == "vile"); // leftover value eval = parse2 (toParse); CHECK (not eval.result); eval = parse2 (bye); @@ -145,7 +140,7 @@ namespace test { * - first demonstrate explicitly how the consecutive parsing works * and how both models are combined into a product model (tuple) * - demonstrate how leading whitespace is skipped automatically - * - then perform the same parse with a Syntax clause build with + * - then perform the same parse with a Syntax clause, built by * the `seq()` builder-DSL * - extend this Syntax by adding a further sequential clause. */ @@ -198,8 +193,8 @@ namespace test { CHECK (not term2.parse(" old ").result); - //___________________________________________________ - // DSL parse clause builder: a sequence of terminals... + //____________________________________________________ + // DSL syntax clause builder: a sequence of terminals... auto syntax = accept("hello").seq("world"); // Perform the same parse as demonstrated above.... @@ -214,8 +209,9 @@ namespace test { // can build extended clause from existing one - auto syntax2 = syntax.seq("trade"); + auto syntax2 = accept(syntax).seq("trade"); // Warning: seq() moves the parse function (but accept() has created a copy) CHECK (not syntax2.hasResult()); + CHECK ( syntax.hasResult()); // ...so the syntax2 is indeed an independent instance now syntax2.parse(s2); CHECK (not syntax2.success()); syntax2.parse(s3); @@ -228,11 +224,12 @@ namespace test { - /** @test define alternative syntax structures to match by parse. + /** @test define alternative syntax clauses to match by parse. * - first demonstrate how a model with alternative branches can be * populated and gradually extended while searching for a match. * - then show explicitly the logic to check and select branches * and construct the corresponding sum-model (variant) + * - finally demonstrate equivalent behaviour using the DSL */ void acceptAlternatives() @@ -340,7 +337,7 @@ namespace test { CHECK (altModel.get<0>().str() == "brazen"); // can build extended clause from existing one - auto syntax2 = syntax.alt("smarmy (\\w+)"); + auto syntax2 = accept(syntax).alt("smarmy (\\w+)"); CHECK (not syntax2.hasResult()); syntax2.parse(s1); CHECK (not syntax2.success()); @@ -370,10 +367,10 @@ namespace test { { //_______________________________________________ // Demonstration: how repetitive sequence works.... auto sep = buildConnex (","); - auto term = buildConnex ("\\w+"); + auto word = buildConnex ("\\w+"); auto parseSeq = [&](StrView toParse) { - using Res = decltype(term)::Result; + using Res = decltype(word)::Result; using IterResult = std::vector; using IterEval = Eval; uint consumed{0}; @@ -389,7 +386,7 @@ namespace test { break; offset += delim.consumed; } - auto eval = term.parse (toParse.substr(offset)); + auto eval = word.parse (toParse.substr(offset)); if (not eval.result) break; offset += eval.consumed; @@ -424,7 +421,7 @@ namespace test { //______________________________________________ // DSL parse clause builder: iterative sequence... - auto syntax1 = accept_repeated(",", term); + auto syntax1 = accept_repeated(",", word); // Perform the same parse as demonstrated above.... CHECK (not syntax1.hasResult()); @@ -442,8 +439,8 @@ namespace test { CHECK (res1[1].str() == "extort" ); CHECK (res1[2].str() == "profit" ); - auto syntax2 = accept_repeated(1,2,",", term); - auto syntax3 = accept_repeated( 4,",", term); + auto syntax2 = accept_repeated(1,2,",", word); + auto syntax3 = accept_repeated( 4,",", word); syntax2.parse(s2); syntax3.parse(s2); CHECK ( syntax2); @@ -460,7 +457,7 @@ namespace test { CHECK (syntax3.getResult()[2].str() == "profit" ); CHECK (syntax3.getResult()[3].str() == "dump" ); - auto syntax4 = accept_repeated(term); + auto syntax4 = accept_repeated(word); syntax4.parse(s1); CHECK (syntax4.success()); CHECK (syntax4.getResult().size() == 2); @@ -743,6 +740,7 @@ namespace test { .alt(quote) .alt(paren)); + // abbreviation for the test... auto apply = [](auto& syntax) { return [&](auto const& str) { return accept(syntax).bindMatch() @@ -751,25 +749,25 @@ namespace test { }; }; -SHOW_EXPR(apply(content)("prey .. haul .. loot")) -SHOW_EXPR(apply(content)("prey .. haul ,. loot")) -SHOW_EXPR(apply(content)("prey .( haul ,. loot")) + CHECK (apply(content)("prey .. haul .. loot") == "prey .. haul .. loot"_expect ); + CHECK (apply(content)("prey .. haul ,. loot") == "prey .. haul "_expect ); + CHECK (apply(content)("prey .( haul ,. loot") == "prey ."_expect ); -SHOW_EXPR(apply(quote)("\"prey .( haul ,\"loot")) -SHOW_EXPR(apply(quote)("\"prey \\ haul ,\"loot")) -SHOW_EXPR(apply(quote)("\"prey\\\"haul ,\"loot")) + CHECK (apply(quote)("\"prey .( haul ,\"loot") == "\"prey .( haul ,\""_expect ); + CHECK (apply(quote)("\"prey \\ haul ,\"loot") == "\"prey \\ haul ,\""_expect ); + CHECK (apply(quote)("\"prey\\\"haul ,\"loot") == "\"prey\\\"haul ,\""_expect ); -SHOW_EXPR(apply(paren)("(prey) .. haul .. loot")) -SHOW_EXPR(apply(paren)("(prey .. haul .. loot)")) -SHOW_EXPR(apply(paren)("(prey(..(haul)..)loot)")) -SHOW_EXPR(apply(paren)("(prey \" haul)\" loot)")) -SHOW_EXPR(apply(paren)("(prey\\( haul)\" loot)")) + CHECK (apply(paren)("(prey) .. haul .. loot") == "(prey)"_expect ); + CHECK (apply(paren)("(prey .. haul .. loot)") == "(prey .. haul .. loot)"_expect ); + CHECK (apply(paren)("(prey(..(haul)..)loot)") == "(prey(..(haul)..)loot)"_expect ); + CHECK (apply(paren)("(prey \" haul)\" loot)") == "(prey \" haul)\" loot)"_expect ); + CHECK (apply(paren)("(prey\\( haul)\" loot)") == "(prey\\( haul)"_expect ); -SHOW_EXPR(apply(spec)("\"prey .( haul ,\"loot!")) -SHOW_EXPR(apply(spec)("\"prey .( haul \",loot!")) -SHOW_EXPR(apply(spec)(" prey .( haul \",loot!")) -SHOW_EXPR(apply(spec)(" prey .( haul )\"loot!")) -SHOW_EXPR(apply(spec)(" (prey\\( haul }, loot)")) + CHECK (apply(spec)("\"prey .( haul ,\"loot!") == "\"prey .( haul ,\"loot!"_expect); + CHECK (apply(spec)("\"prey .( haul \",loot!") == "\"prey .( haul \""_expect ); + CHECK (apply(spec)(" prey .( haul \",loot!") == "prey ."_expect ); + CHECK (apply(spec)(" prey .( haul,)\"loot!") == "prey .( haul,)"_expect ); + CHECK (apply(spec)(" (prey\\( haul }, loot)") == "(prey\\( haul }, loot)"_expect ); } }; diff --git a/wiki/thinkPad.ichthyo.mm b/wiki/thinkPad.ichthyo.mm index bfc6b341f..78675c0f4 100644 --- a/wiki/thinkPad.ichthyo.mm +++ b/wiki/thinkPad.ichthyo.mm @@ -55169,7 +55169,8 @@ - + + @@ -55277,9 +55278,11 @@ - - - + + + + + @@ -55333,7 +55336,8 @@ - + + @@ -55344,12 +55348,22 @@ - +

- Wenn man schon ein zustandsbehaftetes Objekt akzeptiert, könnte auch Zuweisbarkeit bequem sein (man muß dann ja ohnehin aufpassen). Allerdings steht das im Konflikt mit dem Ansatz einer fein-granularen Typisierung, welche die Modell-Struktur abbildet. Und die Entscheidung, die recursive-descent-Struktur als parse-λ einzubetten, verbietet explizit die Zuweisbarkeit, denn man kann (und darf) nicht wissen, was in der Closure steckt + Wenn man schon ein zustandsbehaftetes Objekt akzeptiert, könnte auch Zuweisbarkeit bequem sein (man muß dann ja ohnehin aufpassen). Allerdings steht das im Konflikt mit dem Ansatz einer fein-granularen Typisierung, welche die Modell-Struktur abbildet. Und die Entscheidung, die recursive-descent-Struktur als parse-λ einzubetten, verhindert in den meisten Fällen die Zuweisbarkeit, denn man kann (und darf) nicht wissen, was in der Closure steckt +

+ + + + + + + +

+ ....in der weiteren Entwicklung zeigte sich, daß der Ansatz mit den direkt aufgegriffenen λ-Typen seine Grenzen findet, sobald die Syntax-Klauseln rekursiv werden; in diesem Fall müßen wir explizit die Typisierung abschneiden. Das ist ein Kompromiß, den ich für angemessen halte

@@ -55404,10 +55418,71 @@ + + + + +

+ Das ist eine grundsätzliche Entscheidung, die direkt aus den explizit aufgegriffenen und eingebundenen λ-Typen folgt: +

+
    +
  • + dieses Framework baut selber keinen Syntaxbaum auf +
    • +
  • + dafür aber verwenden wir Metaprogrammierung, um aus den Builder-DSL-Aufrufen einen strukturierten Model-Term aufzubauen, der dem Aufbau der Syntax folgt +
    • +
+ + + - + + + + +

+ ...man ist also nicht in die komplexen, verschachtelten Strukturen hineingezwungen, sondern kann an strategisch günstiger Stelle in einen eigenen Model-Typ übersetzen; für die unterstützung rekursiver Syntax-Klauseln erweist sich das sogar als essentiell +

+ + + + + + + +

+ Aus Sicht der technischen Konstruktion erschien es mir erst sehr naheliegend, die DSL-Syntax weitgehend auf Postfix-Operatoren aufzubauen. Das erwies sich als Trugschluß, denn solche Konstrukte sind in der praktischen Anwendung schwer zu durchschauen. Daher habe ich die Präferenzen unterwegs geändert und die DSL so umstrukturiert, daß sie eine Reihe freier Funktionen als Einstiegspunkt bietet, und diese freien Funktionen auch dazu dienen sollen, verschachtelte Sub-Syntax-Klauseln einzuleiten +

+ + + + + + + +

+ Rekursion ist ein essentielles Element jeden echten Parsers; sie ist aber auch komplex (und potentiell nicht-terminierend). Daher sollen die meisten Anwendungsfälle durch vorgefertigte Syntax-Elemente (Repetition und Klammerung) abgefangen werden. Für die sonstigen Fälle verlangen wir vom Benutzer etwas Vorarbeit: der Model-Result-Type muß in diesem Fall durch ein Binding reduziert werden, so daß das Ergebnis einer rekursiven Referenz bereits feststeht. Damit konnte an der Stelle (mithilfe von std::function) eine elegante Lösung gefunden werden, die nach außen nahezu unsichtbar bleibt: +

+
    +
  • + der Benutzer muß rekursiv zu referenzierende Klauseln vordefinieren, mit angegebenem Ergebnistyp +
    • +
  • + später muß in der Definition der Klausel am Ende ein Model-Binding stehen, das genau diesen Result-Typ liefert +
    • +
  • + schließlich wird diese Definition dem pre-deklarierten Syntax-Objekt zugewiesen +
    • +
+ + + + + - + + @@ -55420,7 +55495,8 @@ - + + @@ -55457,7 +55533,7 @@ - + @@ -55490,7 +55566,8 @@ - + + @@ -55606,7 +55683,7 @@ - + @@ -55630,7 +55707,7 @@ - + @@ -55797,8 +55874,9 @@ - - + + + @@ -55869,8 +55947,8 @@ - - + + @@ -55882,6 +55960,10 @@ + + + + @@ -56023,7 +56105,7 @@ - + @@ -57237,6 +57319,20 @@ + + + + +

+ ...rein nach Bauchgefühl dürfte das in der Praxis dann doch nicht so schlimm werden, sofern man konsistent jede Klausel auch mit einem Binding ausstattet, und die Ergebnistypen systematisch aufbaut. Schließlich ist das ja auch die Aufgabe schlechthin beim Parsen +

+ + + + + + + @@ -57747,9 +57843,10 @@ - + - + + @@ -57788,6 +57885,7 @@ + @@ -57807,6 +57905,7 @@

+ @@ -57817,9 +57916,9 @@ - - - + + + @@ -57828,7 +57927,7 @@

- + @@ -57858,7 +57957,7 @@ - + @@ -57872,11 +57971,73 @@ + + + + +

+ Beobachtung: beide haben zwei führende Leerzeichen +

+ + + + + + + + + + + + + +

+ das liegt daran, daß wir diese Info nicht aufzeichnen, weil generell das Akzeptieren (und Backtracking) durch rekursiven Funktionsaufruf auf Substrings realisiert ist... +

+ + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -57943,8 +58104,31 @@ - - + + + + + + +

+ ...der diese Entwicklung eines Parser-Frameworks angestoßen hat +

+ + + + + + + + + + + + + + + + @@ -57953,6 +58137,9 @@ + + + @@ -105160,9 +105347,9 @@ StM_bind(Builder<R1> b1, Extension<R1,R2> extension) - + - +