GitLab

decouple the DAG traversal logic from the DAG reader (local branch)

This version writes one node at a time instead of first buffering the entire
file contents in a single array (taking up too much memory) before actually
writing it.

Decouple the DAG traversal logic from the `PBDagReader` encapsulating it in the
(new) `Walker` structure.

Collapse PB and Buffer DAG readers into one (`dagReader`) removing the
`bufdagreader.go` and `pbdagreader.go` files, moving all the code to
`dagreader.go`.

Remove `TestSeekAndReadLarge` and `TestReadAndCancel` which operated directly on
the `NodePromise` structure that is now abstracted away in `NavigableIPLDNode`,
in the `go-ipld-format` repo, where they should be recreated.

License: MIT
Signed-off-by: Lucas Molas <schomatis@gmail.com>

Unixfs: enforce refs on files when using nocopy

Notes of removing UnixfsNode:
- `NewLeafNode` will return the `FSNodeOverDag` with the given
  `fsNodeType`. While the old `NewLeaf`: `if data is nil the type
  field will be TRaw (for backwards compatibility), if data is
  defined (but possibly empty) the type field will be TRaw.`. Not
  sure if I should follow this. And because of this, I keep the
  `NewLeafNode` and `NewLeafDataNode`, not rename them to
  `NewLeaf` and `GetNextDataNode`.
- There is no functions in importer/helpers/helpers.go. I am
  thinking if I should move the `FSNodeOverDag` part of
  importer/helpers/dagbuilder.go into importer/helpers/helpers.go.
- `GetDagNode` return FilestoreNode for RawNode. But I do not
  understand how it is used in the `DagBuilderHelper.AddChild`. And
  `FileSize` do not calculate the size of RawNode because there is
  no flag of Raw in `FSNodeOverDag`.

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

This patch is the part of trickledag work which is similar to the
merkledag work in commit 474b77a2bdb1c ("importer: remove `UnixfsNode`
from the balanced builder"). Two helper functions(fillTrickleRecFSNode
and FillFSNodeLayer) is introduced temporarily for modifing the Layout
functions. These two funtions will be removed when all the code of
UnixfsNode is removed in trickledag.go.

Test ipfs add and get commands to check whether get the same hash of
file after the code changes.

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

After fsNodeType in NewLeafNode is supported by commit 85897b3f89301
("dag: add fsNodeType in NewLeafNode and NewLeafDataNode"). Move
comments in NewLeafNode to importer/balanced/builder.go to clarify
why TFile is used by balanced builder as leaves.

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

NewLeafNode and NewLeafDataNode is introduced in commit 474b77a2bdb1c
("importer: remove `UnixfsNode` from the balanced builder"). It is
intended to return ipfs.Node instead of UnixfsNode. But it only
support creating the TFile leaf node for merkledag.

This commit add fsNodeType to above two functions and update the code
in dagbuild.go. Further patches of trickledag will make use of them
and pass TRaw to create leaf node.

License: MIT
Signed-off-by: Bamvor Zhang <jian.zhang@ipfsbit.com>

Fix/handle overflow

I didn't do this before because this datastructure *technically* isn't specific
to sharded directories but, really, that was just even more confusing.

1. It's less confusing. Murmur64 is something our library supports by truncating
a Murmur128 hash.
2. We'll only use the part we need anyways.
directory trees.

If we have a deep enough directory, we could walk off the end of the bit array.

gx publish 1.2.11