benn/lumiera_
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Library: need support for specification parsing

Browse source 
Unfortunately, there are some common syntactic structures, which can not easily be dissected by regular expressions alone, since they entail nested subexpressions. While it is possible to get beyond those fundamental limitations with some trickery, doing so remains precisely that, ''trickery.''

After fighting some inner conflicts, since ''I do know how to write a parser'' —
in the end I have brought myself to just do it.

And indeed, as you'd might expect, I have looked into existing library solutions,
and I would not like to have any one of them as part of the project.
 * I do not want a ''parser engine'' or ''parser generator''
 * I want the directness of recursive-descent, but combined with Regular Expressions as terminal
 * I want to see the structure of the used grammar at the definition site of the custom parser function
 * I want deep integration of ''model bindings'' into the parse process, i.e. binding-λ
 * I do not want to write model-dissecting or pattern-matching code after the parse
 * I do not want to expose ''Monads'' as an interface, since they tend to spread unhealthy structure to surrounding code
 * I do not want to leak technicalities of the parse mechanics into the using code
 * I do not want to impose hard to remember specific conventions onto the user

Thus I've set the following aims:
 * The usage should require only a single header include (ideally header-only)
 * The entrance point should be a small number of DSL-starter functions
 * The parser shall be implemented by recursive-descent, using the parser-combinator technique
 * But I want that wrapped into a DSL, to be able to control what is (not) provided or exposed.
 * I want a stateful, applicative logic, since parsing, by its very nature, is stateful!
 * I want complete compile-time typing, visible to the optimiser, without a virtual »Parser« interface

And last but not least, ''I do not want to create a ticket, since I do not know if those goals can be achieved...''
This commit is contained in:
Fischlurch 2025-01-17 18:40:44 +01:00
parent 55ad44590c
commit 7998c8d724
11 changed files with 1883 additions and 86 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
doc/technical/howto/crackNuts.txt
View file

@ -183,7 +183,7 @@ pick and manipulate individually::
- but sometimes it is easier to use the tried and true technique of the Loki-Typelists, which - but sometimes it is easier to use the tried and true technique of the Loki-Typelists, which
can be programmed recursively, similar to LISP. The »bridge« is to unpack the variadic argument pack can be programmed recursively, similar to LISP. The »bridge« is to unpack the variadic argument pack
into the `lib::meta::Types<ARGS...>` ([yellow-background]#⚠ still broken in 2024# into the `lib::meta::Types<ARGS...>` ([yellow-background]#⚠ still broken in 2024#
see https://issues.lumiera.org/ticket/987[#987], use `lib::meta::TySeq` from 'variadic-helper.hpp' as workaround...) see https://issues.lumiera.org/ticket/987[#987], use `lib::meta::TySeq` from 'meta/typelist.hpp' as workaround...)
+ +
apply functor to each tuple element:: apply functor to each tuple element::
A common trick is to use `std::apply` in combination with a _fold-expression_ A common trick is to use `std::apply` in combination with a _fold-expression_

21
src/lib/meta/typelist.hpp
View file

@ -133,6 +133,25 @@ namespace meta {
}; };
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND -- to be obsoleted
/**
* temporary workaround:
* alternative definition of "type sequence",
* already using variadic template parameters.
* @remarks the problem with our existing type sequence type
* is that it fills the end of each sequence with NullType,
* which was the only way to get a flexible type sequence
* prior to C++11. Unfortunately these trailing NullType
* entries do not play well with other variadic defs.
* @deprecated when we switch our primary type sequence type
* to variadic parameters, this type will be obsoleted. ////////////////////////////////////TICKET #987 : make lib::meta::Types<TYPES...> variadic
*/
template<typename...TYPES>
struct TySeq
{
using Seq = TySeq;
using List = typename Types<TYPES...>::List;
};
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND(End) -- to be obsoleted
}} // namespace lib::meta }} // namespace lib::meta
#endif #endif

40
src/lib/meta/typeseq-util.hpp
View file

@ -143,6 +143,46 @@ namespace meta {
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND -- to be obsoleted
/**
* temporary workaround: additional specialisation for the template
* `Prepend` to work also with the (alternative) variadic TySeq.
* @see typeseq-util.hpp
*/
template<typename T, typename...TYPES>
struct Prepend<T, TySeq<TYPES...>>
{
using Seq = TySeq<T, TYPES...>;
using List = typename Types<T, TYPES...>::List;
};
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND -- to be obsoleted
/**
* temporary workaround: strip trailing NullType entries from a
* type sequence, to make it compatible with new-style variadic
* template definitions.
* @note the result type is a TySec, to keep it apart from our
* legacy (non-variadic) lib::meta::Types
* @deprecated necessary for the transition to variadic sequences ////////////////////////////////////TICKET #987 : make lib::meta::Types<TYPES...> variadic
*/
template<typename SEQ>
struct StripNullType;
template<typename T, typename...TYPES>
struct StripNullType<Types<T,TYPES...>>
{
using TailSeq = typename StripNullType<Types<TYPES...>>::Seq;
using Seq = typename Prepend<T, TailSeq>::Seq;
};
template<typename...TYPES>
struct StripNullType<Types<NullType, TYPES...>>
{
using Seq = TySeq<>; // NOTE: this causes the result to be a TySeq
};
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND(End) -- to be obsoleted

69
src/lib/meta/variadic-helper.hpp
View file

@ -50,74 +50,7 @@
namespace lib { namespace lib {
namespace meta { namespace meta {
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND -- to be obsoleted
/**
* temporary workaround:
* alternative definition of "type sequence",
* already using variadic template parameters.
* @remarks the problem with our existing type sequence type
* is that it fills the end of each sequence with NullType,
* which was the only way to get a flexible type sequence
* prior to C++11. Unfortunately these trailing NullType
* entries do not play well with other variadic defs.
* @deprecated when we switch our primary type sequence type
* to variadic parameters, this type will be obsoleted. ////////////////////////////////////TICKET #987 : make lib::meta::Types<TYPES...> variadic
*/
template<typename...TYPES>
struct TySeq
{
using Seq = TySeq;
using List = typename Types<TYPES...>::List;
};
/**
* temporary workaround: additional specialisation for the template
* `Prepend` to work also with the (alternative) variadic TySeq.
* @see typeseq-util.hpp
*/
template<typename T, typename...TYPES>
struct Prepend<T, TySeq<TYPES...>>
{
using Seq = TySeq<T, TYPES...>;
using List = typename Types<T, TYPES...>::List;
};
/**
* temporary workaround: strip trailing NullType entries from a
* type sequence, to make it compatible with new-style variadic
* template definitions.
* @note the result type is a TySec, to keep it apart from our
* legacy (non-variadic) lib::meta::Types
* @deprecated necessary for the transition to variadic sequences ////////////////////////////////////TICKET #987 : make lib::meta::Types<TYPES...> variadic
*/
template<typename SEQ>
struct StripNullType;
template<typename T, typename...TYPES>
struct StripNullType<Types<T,TYPES...>>
{
using TailSeq = typename StripNullType<Types<TYPES...>>::Seq;
using Seq = typename Prepend<T, TailSeq>::Seq;
};
template<typename...TYPES>
struct StripNullType<Types<NullType, TYPES...>>
{
using Seq = TySeq<>; // NOTE: this causes the result to be a TySeq
};
//////////////////////////////////////////////////////////////////////////////////////////////////////////TICKET #987 temporary WORKAROUND(End) -- to be obsoleted
/* ==== Build Variadic Sequences ==== **/ /* ==== Build Variadic Sequences ==== **/

388
src/lib/parse.hpp Normal file
View file

@ -0,0 +1,388 @@
/*
PARSE.hpp - helpers for parsing textual specifications
Copyright (C)
2024, Hermann Vosseler <Ichthyostega@web.de>
  **Lumiera** is free software; you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by the
  Free Software Foundation; either version 2 of the License, or (at your
  option) any later version. See the file COPYING for further details.
*/
/** @file parse.hpp
** Convenience wrappers and definitions for parsing structured definitions.
** Whenever a specification syntax entails nested structures, extracting contents
** with regular expressions alone becomes tricky. Without much sophistication, a
** directly implemented simple recursive descent parser is often less brittle and
** easier to understand and maintain. With some helper abbreviations, notably
** a combinator scheme to work from building blocks, a hand-written solution
** can benefit from taking short-cuts, especially related to result bindings.
*/
#ifndef LIB_PARSE_H
#define LIB_PARSE_H
#include "lib/iter-adapter.hpp"
#include "lib/meta/function.hpp"
#include "lib/meta/trait.hpp"
#include "lib/regex.hpp"
//#include <regex>
#include <optional>
#include <utility>
namespace util {
namespace parse {
using std::move;
using std::forward;
using std::optional;
using lib::meta::_Fun;
using lib::meta::has_Sig;
using lib::meta::NullType;
using std::decay_t;
using std::tuple;
using StrView = std::string_view;
/**
*/
template<class RES>
struct Eval
{
using Result = RES;
optional<RES> result;
};
template<class FUN>
struct Connex
: util::NonAssign
{
using PFun = FUN;
PFun parse;
using Result = typename _Fun<PFun>::Ret::Result;
Connex (FUN&& pFun)
: parse{move(pFun)}
{ }
};
auto
buildConnex(NullType)
{
return Connex{[](StrView) -> Eval<NullType>
{
return {NullType{}};
}};
}
using NulP = decltype(buildConnex (NullType()));
auto
buildConnex (regex rex)
{
return Connex{[regEx = move(rex)]
(StrView toParse) -> Eval<smatch>
{
return {matchAtStart (toParse,regEx)};
}};
}
using Term = decltype(buildConnex (std::declval<regex>()));
Term
buildConnex (string const& rexDef)
{
return buildConnex (regex{rexDef});
}
template<class FUN>
auto
buildConnex (Connex<FUN> const& anchor)
{
return Connex{anchor};
}
template<class FUN>
auto
buildConnex (Connex<FUN> && anchor)
{
return Connex{move(anchor)};
}
template<class CON, class BIND>
auto
adaptConnex (CON&& connex, BIND&& modelBinding)
{
using R1 = typename CON::Result;
using Arg = lib::meta::_FunArg<BIND>;
static_assert (std::is_constructible_v<Arg,R1>,
"Model binding must accept preceding model result.");
using AdaptedRes = typename _Fun<BIND>::Ret;
return Connex{[origConnex = forward<CON>(connex)
,binding = forward<BIND>(modelBinding)
]
(StrView toParse) -> Eval<AdaptedRes>
{
auto eval = origConnex.parse (toParse);
if (eval.result)
return {binding (move (*eval.result))};
else
return {std::nullopt};
}};
}
/* ===== building structured models ===== */
/**
* Product Model : results from a conjunction of parsing clauses,
* which are to be accepted in sequence, one after the other.
*/
template<typename...RESULTS>
struct SeqModel
: tuple<RESULTS...>
{
static constexpr size_t SIZ = sizeof...(RESULTS);
using Seq = lib::meta::TySeq<RESULTS...>;
using Tup = std::tuple<RESULTS...>;
using Tup::Tup;
};
/**
* Sum Model : results from a disjunction of parsing clauses,
* which are are tested and accepted as alternatives, one at least.
*/
template<typename...CASES>
struct AltModel
{
};
/** Special case Product Model to represent iterative sequence */
template<typename RES>
struct IterModel
{
};
/** Marker-Tag for the result from a sub-expression, not to be joined */
template<typename RES>
struct SubModel
{
};
/** Standard case : combinator of two model branches */
template<template<class...> class TAG, class R1, class R2 =void>
struct _Join
{
using Result = TAG<R1,R2>;
};
/** Generic case : extend a structured model by further branch */
template<template<class...> class TAG, class...RS, class R2>
struct _Join<TAG,TAG<RS...>,R2>
{
using Result = TAG<RS...,R2>;
};
/** Special Case : absorb sub-expression without flattening */
template<template<class...> class TAG, class R1, class R2>
struct _Join<TAG,SubModel<R1>,R2>
{
using Result = TAG<R1,R2>;
};
template<template<class...> class TAG, class R1, class R2>
struct _Join<TAG,R1, SubModel<R2>>
{
using Result = TAG<R1,R2>;
};
template<template<class...> class TAG, class R1, class R2>
struct _Join<TAG,SubModel<R1>,SubModel<R2>>
{
using Result = TAG<R1,R2>;
};
/** Special Case : absorb further similar elements into IterModel */
template<class RES>
struct _Join<IterModel, IterModel<RES>, RES>
{
using Result = IterModel<RES>;
};
template<class C1, class C2>
auto
sequenceConnex (C1&& connex1, C2&& connex2)
{
using R1 = typename decay_t<C1>::Result;
using R2 = typename decay_t<C2>::Result;
using ProductResult = typename _Join<SeqModel, R1, R2>::Result;
using ProductEval = Eval<ProductResult>;
return Connex{[conL = forward<C1>(connex1)
,conR = forward<C2>(connex2)
]
(StrView toParse) -> ProductEval
{
auto eval1 = conL.parse (toParse);
if (eval1.result)
{
uint end1 = eval1.result->length();/////////////////////////OOO pass that via Eval
StrView restInput = toParse.substr(end1);
auto eval2 = conR.parse (restInput);
if (eval2.result)
{
uint end2 = end1 + eval2.result->length();
return ProductEval{ProductResult{move(*eval1.result)
,move(*eval2.result)}};
}
}
return ProductEval{std::nullopt};
}};
}
template<class PAR>
class Syntax;
template<class CON>
class Parser
: public CON
{
using PFun = typename CON::PFun;
static_assert (_Fun<PFun>(), "Connex must define a parse-function");
public:
using Connex = CON;
using Result = typename CON::Result;
using Sigi = typename _Fun<PFun>::Sig;
//lib::test::TypeDebugger<Sigi> buggi;
//lib::test::TypeDebugger<Result> guggi;
static_assert (has_Sig<PFun, Eval<Result>(StrView)>()
,"Signature of the parse-function not suitable");
Eval<Result>
operator() (StrView toParse)
{
return CON::parse (toParse);
}
template<typename SPEC>
Parser (SPEC&& spec)
: CON{buildConnex (forward<SPEC> (spec))}
{ }
// template<class PAR>
// Parser (Syntax<PAR> const& anchor)
// : CON{anchor}
// { }
// template<class PAR>
// Parser (CON const& anchor)
// : CON{anchor}
// { }
};
Parser(NullType) -> Parser<NulP>;
Parser(regex &&) -> Parser<Term>;
Parser(regex const&) -> Parser<Term>;
Parser(string const&) -> Parser<Term>;
template<class FUN>
Parser(Connex<FUN> const&) -> Parser<Connex<FUN>>;
//
// template<class PAR>
// Parser(Syntax<PAR> const&) -> Parser<typename PAR::Connex>;
template<class PAR>
class Syntax
: public Eval<typename PAR::Result>
{
PAR parse_;
public:
using Connex = typename PAR::Connex;
using Result = typename PAR::Result;
bool success() const { return bool(Syntax::result); }
bool hasResult() const { return bool(Syntax::result); }
Result& getResult() { return * Syntax::result; }
Result&& extractResult(){ return move(getResult()); }
Syntax()
: parse_{NullType()}
{ }
explicit
Syntax (PAR&& parser)
: parse_{move (parser)}
{ }
explicit
operator bool() const
{
return success();
}
Syntax&&
parse (StrView toParse)
{
eval() = parse_(toParse);
return move(*this);
}
Connex const&
getConny() const
{
return parse_;
}
template<typename SPEC>
auto
seq (SPEC&& clauseDef)
{
return accept(
sequenceConnex (move(parse_)
,Parser{forward<SPEC> (clauseDef)}));
}
private:
Eval<Result>&
eval()
{
return *this;
}
};
template<typename SPEC>
auto
accept (SPEC&& clauseDef)
{
return Syntax{Parser{forward<SPEC> (clauseDef)}};
}
// template<class PAR>
// Parser(Syntax<PAR> const&) -> Parser<typename PAR::Connex>;
}// namespace parse
using parse::accept;
}// namespace util
namespace lib {
}// namespace lib
#endif/*LIB_PARSE_H*/

20
src/lib/regex.hpp
View file

@ -59,6 +59,24 @@ namespace util {
ENABLE_USE_IN_STD_RANGE_FOR_LOOPS (RegexSearchIter); ENABLE_USE_IN_STD_RANGE_FOR_LOOPS (RegexSearchIter);
}; };
/**
* Helper algorithm to perform a search but require the match to start
* at the beginning of the string or sequence, while accepting trailing content.
* @param toParse a string, string-view or something that can be converted to string.
* @return a std::optional with the match result (`std::smatch`).
*/
template<typename STR>
std::optional<smatch>
matchAtStart (STR&& toParse, regex const& regex)
{
auto search = RegexSearchIter{std::forward<STR> (toParse), regex};
if (search and 0 == search->position(0))
return *search;
else
return std::nullopt;
}
}// namespace util }// namespace util
namespace lib { namespace lib {
@ -66,4 +84,4 @@ namespace lib {
using std::smatch; using std::smatch;
using std::string; using std::string;
}// namespace lib }// namespace lib
#endif/*LIB_STAT_REGEX_H*/ #endif/*LIB_REGEX_H*/

1
src/steam/control/command-def.hpp
View file

@ -79,7 +79,6 @@ namespace control {
using lib::meta::_Fun; using lib::meta::_Fun;
using lib::meta::NullType; using lib::meta::NullType;
using lib::meta::Types; using lib::meta::Types;
using lib::meta::TySeq;
using lib::meta::Tuple; using lib::meta::Tuple;

3
src/steam/engine/proc-node.cpp
View file

@ -24,6 +24,7 @@
#include "lib/iter-explorer.hpp" #include "lib/iter-explorer.hpp"
#include "lib/format-string.hpp" #include "lib/format-string.hpp"
#include "lib/format-util.hpp" #include "lib/format-util.hpp"
#include "lib/regex.hpp"
#include "lib/util.hpp" #include "lib/util.hpp"
#include <boost/functional/hash.hpp> /////////////////////////////////////////////////////TICKET #1391 is boost-hash the proper tool for this task? #include <boost/functional/hash.hpp> /////////////////////////////////////////////////////TICKET #1391 is boost-hash the proper tool for this task?
@ -41,7 +42,7 @@ namespace engine {
using util::contains; using util::contains;
using boost::hash_combine; using boost::hash_combine;
namespace {// Details: registration and symbol table for node spec data... namespace {// Details: parsing, registration and symbol table for node spec data...
std::unordered_set<ProcID> procRegistry; std::unordered_set<ProcID> procRegistry;
std::unordered_set<string> symbRegistry; std::unordered_set<string> symbRegistry;

5
tests/15library.tests
View file

@ -532,6 +532,11 @@ return: 0
END END
TEST "helpers for specification parsing" Parse_test <<END
return: 0
END
TEST "path array abstraction" PathArray_test <<END TEST "path array abstraction" PathArray_test <<END
return: 0 return: 0
END END

192
tests/library/parse-test.cpp Normal file
View file

@ -0,0 +1,192 @@
/*
Parse(Test) - verify parsing textual specifications
Copyright (C)
2024, Hermann Vosseler <Ichthyostega@web.de>
  **Lumiera** is free software; you can redistribute it and/or modify it
  under the terms of the GNU General Public License as published by the
  Free Software Foundation; either version 2 of the License, or (at your
  option) any later version. See the file COPYING for further details.
* *****************************************************************/
/** @file parse-test.cpp
** unit test \ref Parse_test
*/
#include "lib/test/run.hpp"
#include "lib/test/test-helper.hpp"
#include "lib/parse.hpp"
//#include "lib/iter-explorer.hpp"
//#include "lib/format-util.hpp"
#include "lib/meta/tuple-helper.hpp"
#include "lib/test/diagnostic-output.hpp"//////////////////TODO
//#include "lib/util.hpp"
//#include <vector>
//#include <memory>
namespace util {
namespace parse{
namespace test {
using lib::meta::is_Tuple;
using std::get;
// using util::join;
// using util::isnil;
// using std::vector;
// using std::shared_ptr;
// using std::make_shared;
// using LERR_(ITER_EXHAUST);
// using LERR_(INDEX_BOUNDS);
namespace { // test fixture
// const uint NUM_ELMS = 10;
// using Numz = vector<uint>;
} // (END)fixture
/************************************************************************//**
* @test verify helpers and shortcuts for simple recursive descent parsing
* of structured data and specifications.
*
* @see parse.hpp
* @see proc-node.cpp "usage example"
*/
class Parse_test : public Test
{
virtual void
run (Arg)
{
simpleBlah();
acceptTerminal();
acceptSequential();
}
/** @test TODO just blah. */
void
simpleBlah ()
{
}
/** @test TODO define a terminal symbol to match by parse. */
void
acceptTerminal()
{
// set up a parser function to accept some token as terminal
auto parse = Parser{"hello (\\w+) world"};
string toParse{"hello vile world of power"};
auto eval = parse (toParse);
CHECK (eval.result);
auto res = *eval.result;
CHECK (res.ready() and not res.empty());
CHECK (res.size() == "2"_expect );
CHECK (res.position() == "0"_expect );
CHECK (res.str() == "hello vile world"_expect );
CHECK (res[1] == "vile"_expect );
CHECK (res.suffix() == " of power"_expect );
auto syntax = Syntax{move (parse)};
CHECK (not syntax.hasResult());
syntax.parse (toParse);
CHECK (syntax.success());
CHECK (syntax.getResult()[1] == "vile"_expect);
// shorthand notation to start building a syntax
auto syntax2 = accept ("(\\w+) world");
CHECK (not syntax2.hasResult());
syntax2.parse (toParse);
CHECK (not syntax2.success());
string bye{"cruel world"};
syntax2.parse (bye);
CHECK (syntax2.success());
CHECK (syntax2.getResult()[1] == "cruel"_expect);
// going full circle: extract parser def from syntax
// using Conn = decltype(syntax2)::Connex;
// Conn conny{syntax2};
// auto parse2 = Parser{conny};
auto parse2 = Parser{syntax2.getConny()};
CHECK (eval.result->str(1) == "vile");
eval = parse2 (toParse);
CHECK (not eval.result);
eval = parse2 (bye);
CHECK (eval.result->str(1) == "cruel");
}
/** @test TODO define a sequence of syntax structures to match by parse. */
void
acceptSequential()
{
auto term1 = buildConnex ("hello");
auto term2 = buildConnex ("world");
auto parseSeq = [&](StrView toParse)
{
using R1 = decltype(term1)::Result;
using R2 = decltype(term2)::Result;
using ProductResult = std::tuple<R1,R2>;
using ProductEval = Eval<ProductResult>;
auto eval1 = term1.parse (toParse);
if (eval1.result)
{
uint end1 = eval1.result->length();
StrView restInput = toParse.substr(end1);
auto eval2 = term2.parse (restInput);
if (eval2.result)
{
uint end2 = end1 + eval2.result->length();
return ProductEval{ProductResult{move(*eval1.result)
,move(*eval2.result)}};
}
}
return ProductEval{std::nullopt};
};
string s1{"hello millions"};
string s2{"helloworld"};
auto e1 = parseSeq(s1);
CHECK (not e1.result);
auto e2 = parseSeq(s2);
CHECK ( e2.result);
using SeqRes = std::decay_t<decltype(*e2.result)>;
CHECK (is_Tuple<SeqRes>());
auto& [r1,r2] = *e2.result;
CHECK (r1.str() == "hello"_expect);
CHECK (r2.str() == "world"_expect);
auto syntax = accept("hello").seq("world");
CHECK (not syntax.hasResult());
syntax.parse(s1);
CHECK (not syntax.success());
syntax.parse(s2);
CHECK (syntax);
SeqRes seqModel = syntax.getResult();
CHECK (get<0>(seqModel).str() == "hello"_expect);
CHECK (get<1>(seqModel).str() == "world"_expect);
}
};
LAUNCHER (Parse_test, "unit common");
}}} // namespace util::parse::test

1228
wiki/thinkPad.ichthyo.mm
View file

File diff suppressed because it is too large Load diff