1. 12 Feb, 2019 1 commit
    • Eric Myhre's avatar
      traversal as a package! Big diff; docs!, etc. · 69f89f4b
      Eric Myhre authored
      All traversal/transform/selector/path code to date got a rethink.
      
      For the longest time, I was trying to keep all the traversal code in
      the main root IPLD package, thinking/hoping that the number of relevant
      functions would be minimal enough that this would be fine.
      
      Somewhere around the time where I conceded that yes, we *do* need
      separate methods for writable and readonly paths (it's true that
      readonly is a degenerate case of writable -- but they have different
      performance characteristics!), I gave up: the number of functions in
      play is too high to heap alltogether in the root package namespace.
      
      This leads to a number of other surprising results: namely, that
      ipld.Path *isn't* -- it's actually traversal.Path!  We *don't use*
      paths anywhere else except as guidance to some forms of traversal,
      and logs and progress markers during traversal.  Huh!
      Despite being initially surprising, I now regard this as a *good*
      code smell.
      
      ipld.Traversal as an interface and ipld.Path.Traverse are dropped.
      These turned out to be the wrong level of abstraction:
      it's both missing things, and not able to return enough things.
      By the time we would fix both, it's not an abstraction anymore.
      
      Path.Traverse didn't know how to use a link loader func -- and
      that's correct; it shouldn't -- and it also didn't retain and
      return a stack of Nodes it traversed -- which again, it shouldn't,
      because in general that would be useless overkill and resource waste...
      it's just that both these things are essential in some use cases.
      So!  Cut that gordian knot.
      
      The Focus/FocusTransform/Traverse/TraverseTransform methods will now
      do all recursion work themselves, internally.  The transform family
      will keep stacks of nodes encountered along the way, to aid in the
      (presumably) coming tree re-build; the focus family won't.
      All of these methods will be able to support link loading, though
      implementation of that is still upcoming.
      
      There's a few more fields in TraversalProgress, also preparing for when
      we have automatic link loading and traversal: we'll want to be able
      to inform the user's callbacks about load edges.  (I'm not a fan of the
      term 'block' here, but running with it for the moment.)
      
      TraversalConfig is expected to grow.  Namely, to include link
      loaders... and subsequently, probably block writers, as well!  This is
      a big subject, though.  It'll probably take several commits and
      possibly more than one iteration over time to hammer out a good API.
      
      One thing that's interesting to note that neither TraversalConfig nor
      TraversalProgress will grow to contain certain things: for example,
      a map of "seen" CIDs when doing link traversal.  Why?  Well, if you've
      been mentally modelling things as a graph, this is surprising; surely
      any graph traversal needs to remember a set of already-visited nodes.
      The trick is... we're *not* doing a graph traversal -- not exactly!
      Since our visitor gets a (Node,Path) *tuple*... it would be wrong to
      memoize on anything other than the complete tuple; and the Path part
      of the tuple means we're actually doing a *tree* walk rather than
      a graph walk.  (And of course, since we're operating on DAGs,
      termination guarantees are already a non-issue.)  Tada; no "seen" set.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      69f89f4b
  2. 08 Feb, 2019 1 commit
    • Eric Myhre's avatar
      Updated Traversal; introduce TraversalProgress. · 3aa48459
      Eric Myhre authored
      The policy on partial progress results is now explicitly "no".
      It wasn't useful.
      Tests have been updated accordingly.  (The relevant tests *do* still
      still assert that the traversal errors happen where they should; it's
      just via the... well, the error value.  Sense, neh?)
      
      We're introducing a TraversalProgress struct.  This is roughly
      following "Option 2" from the earlier "this composition is imperfect!"
      lamentation block: it allows us to keep carrying Path from existing
      moves across the graph at all times.
      
      (I suspect we're going to incur somewhat unfortunate memory allocation
      thrash in large-scale operation because of all these immutable Path
      objects, which (ultra)unfortunately can't even reuse the n+1 length
      arrays when their prior sibling already allocated them... However,
      I care a lot more about getting the composability and ergonomics of
      these interfaces right first; and we can discuss how to hack shortcuts
      back in later.  (Perhaps the more visit-oriented functions will do
      some fancy footwork, and combine it with warnings that the Path objects
      handed the visitor are only valid for its scope.  This is sharp-edged,
      but seems to be common in practice in this kind of library; basically
      every performant git library I've ever seen has caveats like this on
      its iterators, for example.  Anyway!  Not doing that yet.))
      
      Context is coming along for the ride on TraversalProgress!
      This is discussed in another one of the large comment blocks that's
      disappearing in this commit.
      
      Newly exported Path.Join method.  Does what it says on the tin.
      
      Usage of this whole new TraversalProgress system is limited: we're just
      cutting away the incomplete cruft of first-draft functions in this
      diff whenever they got in the way, and newly designed
      Traverse/Visit/Transform/etc functions which correctly use
      TraversalProgress will be coming up in future diffs.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      3aa48459
  3. 06 Feb, 2019 3 commits
    • Eric Myhre's avatar
      Some comments on Traversal design. · c209676a
      Eric Myhre authored
      Revision needed.  An idea I earlier discarded as too complex seems both
      less complex and more waranted after sleeping on it.
      
      Dunno if I'll get there tonight, but want this pushed for conversation.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      c209676a
    • Eric Myhre's avatar
      fluent.Node.Keys should panic if errors on route. · 9c13d7d6
      Eric Myhre authored
      Like all other methods on fluent.Node that panic the instant a concrete
      type of anything is asserted outside of traversal attempts.
      
      There's not much sense in holding off until trying to start an
      iteration that's most certainly going to fail; and we do assume that
      anyone making the iterator intends to use it.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      9c13d7d6
    • Eric Myhre's avatar
      ipldfree.NodeBuilder, fluent.NodeBuilder, tests! · 1a6107b5
      Eric Myhre authored
      We now have nice, immutable nodes... as soon as we deprecate and remove
      the MutableNode stuff.
      
      All the tests pass.
      
      A few methods for bulk update are still stubbed... as is delete;
      seealso the comment about that in the API file.
      
      NodeBuilders needed replication in the fluent package as well.
      This is perhaps a tad frustrating, as well as borderline surprising
      (because "how can sheer memset have errors anyway?"), but the ability
      for node building to reject invalid calls will become relevant when
      typed.Node is in play and has to enforce type rules while still
      supporting the generic methods.
      
      The builder syntax for maps and lists is based on chaining.
      This works... okay.  I'm not as satisfied with the visual effect of it
      as I'd hoped.  There'll probably be more experiments in this area.
      Things nest -- critical success; but the indentation level doesn't
      quite match what we'd contemporarily consider natural flow.
      
      But look at the diff between the mutableNode test specs and the new
      immutable&fluent nodeBuilder test specs!  The old ones were nuts:
      temp variables everywhere for construction, total visual disorder, etc.
      The new NodeBuilder is much much better for composition, and doesn't
      need named variables to save fragments during construction.
      More refinement on map/list builders still seems possible, but it's
      moving in the right direction.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      1a6107b5
  4. 05 Feb, 2019 6 commits
    • Eric Myhre's avatar
      Test specs for nested map unmarshal. · 5d19d364
      Eric Myhre authored
      Fortunately, that just already worked fine.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      5d19d364
    • Eric Myhre's avatar
      fluent support for KeysIterator, Length, etc. · 0d61265c
      Eric Myhre authored
      Allows uncommenting some fixme's in some tests.
      
      Also, changed free.keyIterator to be unexported.  Exporting that was
      a typo; there's simply no reason for that to be exported.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      0d61265c
    • Eric Myhre's avatar
      ipld.NodeUnmarshaller func interface; 'free' impl. · 35323ff0
      Eric Myhre authored
      Hej, we can finally unmarshal things again.  Plug a cbor parser
      in from the refmt library and go go go!
      
      This doesn't use any of the mutablenode stuff because, as remarked
      upon a (quite a) few commits back, we want to replace that with
      NodeBuilder interfaces; but, furthermore, we actually don't *need*
      those interfaces for anything in an unmarshalling path (because
      I don't expect we'll seriously need to switch result Node impl
      in mid-unmarshalling stream); so.
      
      You can imagine that the cbor.Node system will have a *very* different
      implementation of this interface (and probably another method that
      doesn't match this interface at all, and is more directly byte stream
      based); but that's for later work.
      
      And tests!
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      35323ff0
    • Eric Myhre's avatar
      Node.Keys method now returns iterators. · 5c3bd1af
      Eric Myhre authored
      An "immediate" rather than generative function is also available;
      the docs contain caveats.
      
      At the moment, these distinctions are a bit forced, but we want to be
      ready for when we start getting truly huge maps where the generative
      usage actually *matters* for either memory reasons, latency reasons,
      or both.
      
      Separated Length rather than trying to pull double-duty the Keys
      method; the previous combination was just utterly silly.
      
      The implementations in the bind package are stubs; that package is
      going to take a lot of work, and so the focus is going to be entirely
      on keeping the 'free' package viable; and we'll revisit bind and such
      when there's more dev time budget.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      5c3bd1af
    • Eric Myhre's avatar
    • Eric Myhre's avatar
      Further schema-schema fixes for 'default' values. · 4e3c8def
      Eric Myhre authored
      Continues the work started in 2263295e, with the purpose of fixing
      undesired three-valued logic by disambiguating 'optional' from
      'default'.  Some comments in the schema now address this in detail,
      including description of the membership cardinality.
      
      Some incongruities in the JSON have been fixed;
      and a significant new set of definitions for struct-represented-as-map
      configuration details is introduced.
      
      I gave some brief consideration to eliding StructRepresentation_Map
      and instead simply using a typedef'd map, and it would be possible
      since it only has one field... however, I'm opting to keep it because
      I'm willing to bet we'll come up with more properties someday which
      are for the whole struct rather than just per field, and thus it's
      unlikely that StructRepresentation_Map will stay single-field forever.
      
      The 'default' value itself in the new
      StructRepresentation_Map_FieldDetails type is interesting: it's the
      first time we've had an excuse to use an "Any" cop-out.
      And even more interestingly, it's almost certainly invalid: as
      commented inline in the diff, that field would probably be even better
      represented as a kinded union which makes it clear that only the scalar
      primitives are welcome here.
      We'll almost certainly introduce that union in a future commit, but
      I'm gonna let that thought marinate for a while first.
      There are related interesting questions about how we want to handle
      the DSL parsing in this area, and those might be good to think about
      more deeply before finalizing this part of the schema.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      4e3c8def
  5. 02 Feb, 2019 1 commit
  6. 31 Jan, 2019 3 commits
    • Eric Myhre's avatar
      Experiments with code generation have begun. · fca56c52
      Eric Myhre authored
      I'm using templates for this rather than AST libraries because after
      some review of existing systems, it seems that this is a popular
      choice; and I'm flat out going to assume those people have known
      what they were doing and made their choice rationally.  Protobufs,
      graphql codegen libraries, the ugorji serialization libraries -- almost
      every codegen I've ever come across uses templating.  So, we will too.
      
      And so far, the results seem promising.
      
      The tests here will be shoddy for some time.  I'm outputting code to
      a directory and periodically eyeballing it and making sure the compiler
      likes it.  My aim is to get some fair fraction of codegen bootstrapped
      this way; and after we start codegen'ing the typedeclaration package
      itself, *then* we'll start solidifying things with regression tests.
      
      There's a lot of discovery process to go, here.  See for example the
      comments about ranging versus iterators (namely, that it's not
      possible, which... turns out is going to feed back into our design
      all the way back out at the layer of the ipld.Node interface itself).
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      fca56c52
    • Eric Myhre's avatar
      Clean up typed/declaration/* types; closer to back in line with schema. · 4ee8b402
      Eric Myhre authored
      Removed the 'name' field from all of them.  Doesn't match schema.
      While it's true that we'll want a name property on the reflective
      typesystem.Type objects, that's... well, that's over there.
      
      I'm starting to make fairly wild zigzags through here; with reason:
      I'm trying to get as quickly as possible to a bootstrap of the essenses
      of codegen.  The hope is to get there fast, and power it with a few
      hardcoded values; then get that to churn out a replacement for the
      hand-written declarations package types; and then teach the codegen
      to write more and more of the e.g. ipld.Node interfaces and then
      the serialization bridges and then the NodeBuilder stuff, etc.
      The core coverage for getting the codegen to go, though, I think it's
      actually sufficiently smaller than all those parts that maybe this
      will be reasonable to pull off this way.  Time will tell.  (Shortly!)
      
      Dropping the typed/system/construct.go file temporarily, as the changes
      to the other typedeclaration files has broken it, and maintaining it
      through the shaking seems like it will generate inordinately more
      work than it's worth.  It'll be revived soon; it's mostly on the right
      track, it's just not on the critical path atm.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      4ee8b402
    • Eric Myhre's avatar
      Schema: Introduce 'default'; contrast 'optional'. · 2263295e
      Eric Myhre authored
      This change to the schema-schema in DSL form makes it accurately match
      the schema-schema in JSON form, *AND* fixes the undesired three-valued
      logic at the same time.  Wunderbar.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      2263295e
  7. 28 Jan, 2019 1 commit
  8. 21 Jan, 2019 4 commits
    • Eric Myhre's avatar
      Finish the JSON repr of the schema-schema! · b01956d9
      Eric Myhre authored
      And fix a few bugs detected in the DSL form (one union typo'd into an
      incorrect representation; and one just outright missing its
      representation declaration), but fortunately nothing too serious.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      b01956d9
    • Eric Myhre's avatar
      e9ced454
    • Eric Myhre's avatar
      More optional properties (for optionals!). · 32a7a18e
      Eric Myhre authored
      The JSON representation of the schema shortens up noticeably if the
      properties for "nullable" and "optional" are themselves optional.
      
      Given that the hope is for the in-practice use of these properties to
      be the exceptional case rather than the norm, it's nicer to speak
      less of them.
      
      This edges around an interesting question: we don't actually *want*
      three-valued logic here... but that's precisely what "optional Bool"
      gives you: it may have the value "true", "false", or "undefined"!
      What we *mean* here is of course "true" or "undefined".  It would be
      desirable to have a way to encode that.  Perhaps we can find a way.
      One idea is that this is an issue in the representation of bool;
      and thus perhaps we should configure it in the struct's representation
      block rather than marking it optional in the struct's definition block.
      Let's give this more thought in the future.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      32a7a18e
    • Eric Myhre's avatar
      Another, flatter proposal for TypeTerm, etc. · 75eac952
      Eric Myhre authored
      (An alternative proposal, with the same example DSL but different
      choices in schema representation is in a sibling commit, and we'll
      merge and pick one or the other to continue with shortly.)
      
      This take on TypeTerm keeps things flatter and has fewer schema types:
      the optional and nullable properties are those of the containers
      in the recursive type (i.e., the map itself, list itself, or the fields
      in the case of structs).
      
      As with the alternative proposal, this seems to work and capture all
      of the concepts that we want:
      
      - the inline definitions are possible;
      - they recurse correctly
      - nullability can be inserted in all the correct places;
      - you *can't* have a "nullable nullable" nesting;
      - and bonus, the simple TypeName case is still a string repr.
      
      So, since all the same key properties exist in both approaches,
      in some ways, this is six-of-one/half-a-dozen-of-the-other.
      Both mechanisms of encoding clearly represent the same things.
      It's just question of which is nicer to work with, which seems more
      self-documenting to a human reader, and other intangibles of that sort.
      
      In this approach, we have a less rightward-leaning tree when the schema
      is represented in JSON form; by comparison, the alternative which had
      "nullable" represented as a union experiences deeper nested objects.
      Reducing depth-of-tree is hardly a proof of simplicity, but it's usually
      a good heuristic, so we can count that as a win here.
      
      Another comparison we can make is that the number of types used in this
      diff is 2; the number used in the other approach was 4.  We do this
      at the trade of more fields in existing types, of course, so it's less
      than perfectly clear cut as a win for simplicity... but I tend towards
      thinking it is.  Especially certainly clearer is that we *don't* have
      a union type which immediately nests another different representation
      of union type in this approach!
      
      So, while it's always hard to make absolutionist statements about
      "simplicity", indeed, this seems simpler via several observations.
      
      In terms of what other people are doing...
      One interesting example from another endevour I recently noticed -- the
      Atom project emits a JSON file describing their API:
      https://github.com/atom/atom/releases/download/v1.34.0/atom-api.json
      In this JSON file, they've taken a tack similar to the one emboded in
      this commit: the "isOptional" property is inline with the field defn's.
      So, we can count Atom's choice as an effective a +1 vote for that.
      (It's perhaps interesting they also have *names* in their field obj...
      however, the fields here are in an array, which is necessary since
      they're describing positional arguments.  So, that's entirely
      self-consistent, but also different constraints than we have, and thus
      while interesting, not quite comparable to our choices on that matter.)
      
      Note that the JSON-formatted schema-schema file has not been updated yet.
      We'll update and finish that after merging and selecting a proposal here.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      75eac952
  9. 18 Jan, 2019 18 commits
    • Eric Myhre's avatar
      A proposal for TypeTerm, nullable, InlineDefn, etc · b9ba9dd1
      Eric Myhre authored
      This seems to work and capture all of the concepts that we want:
      
      - the inline definitions are possible;
      - they recurse correctly
      - nullability can be inserted in all the correct places;
      - you *can't* have a "nullable nullable" nesting;
      - and bonus, the simple TypeName case is still a string repr.
      
      The downsides are:
      
      - wow, this *looks* complicated on the schema side;
      - in part because we have a union-union nest, and conceptually we'd
         probably simply prefer to unify those, but it's a representational
         problem (can't clearly combine a keyed and a kinded);
      - the json repr looks to get deep fast.
      
      I wonder if we can do better than this.
      
      And also, we shifted the flagging of optional fields to
      StructRepresentation_Map, then eliding a struct for field properties
      entirely.  This seemed potentially sensible given that we suppose a
      struct can be represented as a list, in which case optionality of
      fields won't work the same.  However, on further thought, I'm also
      extremely dubious of this idea: whether or not a field can be traversed
      and then yield "undefined" is defined by whether it's optional; and
      I'm really suspecting it's going to create a lot of weirdness to make
      that feature conditional upon the StructRepresentation union.
      If anything, we should simply make invalid combinations of optionality
      and representation of that struct, well, erm, invalid.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      b9ba9dd1
    • Eric Myhre's avatar
      Typo fix; and let's propose all uses of TypeTerm. · b200fb86
      Eric Myhre authored
      Even though we having concretely specified TypeTerm yet, it's a pretty
      fair bet that map and array value types will both use it.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      b200fb86
    • Eric Myhre's avatar
      Define TypeStruct and StructField. · 9bfef80c
      Eric Myhre authored
      StructRepresentation also offers at least two options; and in the
      future, we might anticipate adding more options as well (suppose for
      example some kind of struct-to-string conversion systems; of which
      their could be many, but imagine for example one like matching a
      string like the "options" in your /etc/fstab: simple comma separated
      keys and values should be definable enough, no?).
      
      And now we have to wrangle the TypeTerm recursivity to completeness:
      how shall we concisely and correctly represent the declaration of
      inline declarations of map and array kinded types?  I've left this
      a big honking TODO for this commit; it's touchy.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      9bfef80c
    • Eric Myhre's avatar
      Define TypeEnum and expand json schema using it. · 1dd780a7
      Eric Myhre authored
      Not having a "set" definition is kind of irritating.
      
      I'm wondering if we should propose to make a set kind; and simply
      transform these to a map with null values as necessary.  Possibly it
      could even be argued that this could make sense at the Data Model
      level?  I'm not sure.  Something that should be discussed, though.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      1dd780a7
    • Eric Myhre's avatar
      Fix the json form schema not having outer union. · c8d27fac
      Eric Myhre authored
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      c8d27fac
    • Eric Myhre's avatar
      Schema as self-describing JSON! · 65f48df8
      Eric Myhre authored
      To prove that we can.  And now that we've got unions defined, we can
      actually describe most of the core concepts of "Type" itself, etc.
      
      Fixed one bug in the DSL-style schema; the union discriminator keyword
      for what kind of type a type is should be... "kind".  Whew.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      65f48df8
    • Eric Myhre's avatar
      Drop TypeUnion.members field. · 763174ea
      Eric Myhre authored
      It is indeed redundant with the maps that must necessarily exist in
      each of the representation union members.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      763174ea
    • Eric Myhre's avatar
      UnionRepresentation... union. yes. · 4bd496a9
      Eric Myhre authored
      This is actually a fairly big improvement on what we'd written for the
      internal implementation in Go.  The representation *should* be a union,
      in general.  We'll do something similar for structs, and everything
      that has multiple options for representation strategy.
      
      One FIXME bugaboo: it's now absolutely unclear what to put as keys in
      the members map, as although there is a key, which one to use depends
      on the representation kind.  It would feel odd to not have the
      membership be clear before reading through to the representation info,
      but I guess that is defacto where that information lives, so...
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      4bd496a9
    • Eric Myhre's avatar
      fa71fd81
    • Eric Myhre's avatar
      Expand more of the Type member types. · f68ff054
      Eric Myhre authored
      Added some enums that should be useful, and documented the bejezus out
      of them.
      
      Seems to be going well so far.
      
      Documention for TypeUnion.
      
      All the other interesting Type types are still todos, though; and we
      thus far haven't hoisted ourselves back up to the point where typeterm
      recursiveness needs to be fleshed out, which is one of the things we
      already know is going to be Fun.
      
      Fixed what's probably a typo with an extra equals sign.  I'm not sure
      why my brain keeps wanting to do that for unions.
      
      And one interesting note: though I'm keeping TypeKind in the commit,
      see the todo comment about its potential irrelevance?  It's amusing
      that we've just parallelly reproduced that; the same misstep already
      happened once before, just in the golang code: TypeKind there was also
      not needed, because the Type union members already represented it.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      f68ff054
    • Eric Myhre's avatar
      Start of self-describing schema schema. · b492dffb
      Eric Myhre authored
      Everything here so far is probably good; need to continue specifying
      all of the individual type structure next.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      b492dffb
    • Eric Myhre's avatar
      First pass typedeclaration->typesystem construct! · 7a19bc01
      Eric Myhre authored
      Most of the meat is implemented, with a few important exceptions.
      
      For many of the more simple kinds of type, it's pretty cookiecutter:
      e.g., for bools and strings and the other simple scalars, it's just
      copying over the descriptions to the immutable objects in the
      typed/system package, and it's done.
      
      For maps and arrays and other recursive types, a little more code
      is involved: we do a two-pass approach to make sure we get all the
      named type declarations enumerated first, and then go back to review
      the recursive kinds again, and now check and reify their recursive
      references.  That's pretty straightforward as well.
      
      The one problem here is... we didn't account for inline anonymous
      type definitions, e.g. the `struct { field {Foo:[Bar]} }` syntax.
      That requires some significant fixing in the typed/declaration
      structs; they don't have the right recursivity at all right now.
      
      I started punting out to todo comments rather than properly finishing
      the construction of the struct and union types after making that
      realization about anonymous inline type definitions.  I'd like to
      hammer out the correct recursivity in the declarations types before
      coming back and doing a final finishing pass on this.
      
      And one more side-note: I turned TypeName into an alias between the two
      packages because it reduces syntactic irritation and casting.
      
      Up next: well, we did all this first pass work on typedeclarations, but
      something's been under-regarded: a core design goal is to make sure
      these schema declarations are self-describing and serialize clearly
      in the IPLD Data Model formats.  Let's spend a bit of time on that.
      (What I *really* want is to have *codegen* make the whole suite of
      the typed/declaration package contents straight from a schema!  But
      we've got a bit of a bootstrapping cycle there, of course, heh heh.)
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      7a19bc01
    • Eric Myhre's avatar
      Consistency typo fix. · 5370f1c6
      Eric Myhre authored
      The typed/system package calles these members, so the typed/declaration
      package should too.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      5370f1c6
    • Eric Myhre's avatar
      Introduce ContinuedTransform method, for nesting. · c546e580
      Eric Myhre authored
      There are several "REVIEW" comments here, so this will almost certainly
      see another round of iteration.
      
      There are, of course, clear enough ways to unify this and Transform
      into one general function.  The tricky questions are entirely around
      what's the best way to make the simple cases syntactically simple...
      without trading it for either easily-misused APIs or unreasonable
      syntactical complexity for the more complex compositional cases.
      
      (It should also be noted I'm doing morning cleanup commits of diffs
      that were produced in the past when *probably* having been up too late.
      Please be gentle in your thoughts, future reader.)
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      c546e580
    • Eric Myhre's avatar
      Move Transform method to root package. · 28dd4925
      Eric Myhre authored
      Upon a good night's sleep and reexamining the situation, it's extremely
      clear that this function is *not* a fluent thing at all.  (Duh; look at
      the Node types.)
      
      A non-fluent Transform method has tons of virtue.  In many cases, we
      might not actually need the fluent.Node helper features at all when
      using Transform: since all of the intermediate error handling to
      traverse to the reachedNode is already handled, that's half the battle;
      and whether or we need to fight the ease-of-use battle on the interior
      function is... *it depends*: if we use (codegen'd) native types, those
      already address a lot of the error handling; fluent.Node can also be
      used on the inside, but it's simply not needed in all cases.
      
      We'll come back to the subject of writing the fluent equivalent later;
      some example usage development is needed first so we know how to make
      something that's actually helpful.
      It will, of course, mostly be a wrapper; the interesting question is
      whether a fluent.Transform function should use a fluent.Node either,
      since this will make it harder to compose (unless we make fluent.Node
      still embed ipld.Node?)... ahem.  As I said.  We'll come back to that.
      
      One more quick comment about this relocation: I'm a little tepid about
      putting the Transform method in the main package, because that really
      puts it on-screen quite in the forefront.  If it's just Transform and
      Traverse, maybe this is fine; but we end up implementing a large suite
      of optics methods, this will become cluttery.  So, potentially this
      will move to another sub-package again if that becomes an issue. (But
      it still wouldn't be the fluent package.)
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      28dd4925
    • Eric Myhre's avatar
      Move ipld.Path to its own files. · dd8ef869
      Eric Myhre authored
      These tests are all Path.Traverse tests, not general Traversal
      interface tests or specs.
      
      And while having the poor Traversal interface all alone in its own file
      seems a bit lonely, ... well, maybe we'll give it some company in a
      second.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      dd8ef869
    • Eric Myhre's avatar
      Second try on the Transform doc copy. · aea51b42
      Eric Myhre authored
      It's really a mouthful.  I think this is a little better, at least.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      aea51b42
    • Eric Myhre's avatar
      First pass on Transform methods. · da591285
      Eric Myhre authored
      The docs in the diff body do most of the explaining work here.
      
      I'm not sure if these should be in the fluent package or not.
      Probably it will be wiser to have a non-fluent variant first, and wrap
      those in the fluent package.  I wrote them here first, though, so,
      here they stay for this commit.
      
      I'm checkpointing this here, but there's at least one major issue I've
      already realized: with these interfaces, we can't keep correct paths
      when we do *nested* Transform calls.  Perhaps that's survivable; but I
      find it very unattractive.  So, we'll do some additional work to hammer
      that back out -- the tricky part will just be doing it without
      increasing the amount of complexity that's user-facing.
      
      Also -- yes, I know the parent commit says "we'll get rid of
      MutableNode" next -- but it's basically this Transform stuff that's
      been the driving force behind most of the other traversal and builder
      work going on today, so...  We'll nudge both along in tandem for a bit.
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      da591285
  10. 15 Jan, 2019 2 commits
    • Eric Myhre's avatar
      Additional doc about carry powers of NodeBuilder. · 2c4500b9
      Eric Myhre authored
      Specifically in regards to typed nodes, which are the main consumers of
      this that I imagine right now (though we may find others in the
      future).
      
      Typed nodes which are implemented by *native* concrete types (e.g.
      presumably by codegen -- though we haven't implemented any of this yet)
      will *especially* hit this detail: they'll return builders that produce
      new instances of their own concrete native type!
      
      And all of the build methods that actually produce a new Node now also
      have the option to return an error.  This follows from the docs;
      typed nodes actually can and will reject some attempts at invalidly-
      typed creation!
      
      We'll probably want fluent wrappers of NodeBuilder as well, as a result
      of these error returns.  I'll defer that until it's forced.
      Lazy/just-in-time evaluation of the programmer :P
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      2c4500b9
    • Eric Myhre's avatar
      Introduce ipld.NodeBuilder interface. · 8bf090c8
      Eric Myhre authored
      This will replace, deprecate, annihilate, obviate, oblivate, and
      generally nuke the MutableNode interface.  MutableNode was a bad, bad
      idea and made many things terribly complicated.  Immutable interfaces
      are shaping up much, much better and I'm excited to move NodeBuidler
      forward.
      
      I tried for some time to figure out a useful way to do a builder
      pattern in the fluent package, which would work *upon* the MutableNode
      interface... but all attempts to do that ended poorly with code that
      was both messy internally and complex and error prone for callers.
      I've given up all such attempts.
      
      It also seems to be true that we should have NodeBuilder *instances*
      which may be per Node.  This wasn't obvious at first, but I think it'll
      let us carry a lot of important information in a low-fuss way, and it
      solves a *lot* of problems that cropped up when trying to implement
      larger scale generic transformation methods on Node graphs (namely, the
      frequent need to be able to create a new Node in the middle of a
      process; and handing down a factory method to each of those points was
      horrifically verbose and caused complexity that was not essential).
      (Unfortunately, I can't really point at examples of *how* bad this got,
      because I've never committed any of those research spikes; fortunately,
      no one else will now ever have to *see* that mess, since we've got
      better solutions figured out.)
      
      MutableNode will go away in the next handful of commits, but I wanted
      to land the new one first in a smaller diff.  Removing MutableNode
      may be a fair amount of messy work, because of how it's used in nearly
      all of our test fixture setup to date!  Uffdah.
      
      (Node will get a GetBuilder method (or something of similar intent but
      better name) at the same time we get rid of MutableNode; I'm eliding
      it from this commit for the same reasons about diff size avoidance.)
      Signed-off-by: default avatarEric Myhre <hash@exultant.us>
      8bf090c8