GitLab

optimize CheckIfPinned

1. Parallelize fetching from disk.
2. Avoid re-visiting blocks we've already checked. Adding the same data
   over and over with small changes is pretty common.

* Converting from IPLD to datastore-based pins no longer requires loading all dag-storage pins (including indirect pins) into memory
* increase test coverage

feat: store pins in datastore instead of a DAG

Adds a new `/pins` namespace to the given datastore and uses that to store pins as cbor binary, keyed by unique pin ID.

The new datastore pinner stores pins in the datastore as individual key-value items. This is faster than the dag pinner, which stored all pins in a single dag that had to be rewritten every time a pin was added or removed.

The new pinner provides a secondary indexing mechanism that can be used to index any data that a pin has. Secondary indexing logic is provided by the `dsindex` package. The new pinner currently includes indexing by CID.

Both the new datastore pinner (`dspinner` package) and the old dag pinner (`ipldpinner` package) implementations are included to support migration between the two.  Migration logic is provided by the `pinconv` package.

Other features in new pinner:
- Benchmarks are provided to compare performance of between the old and new pinners
- New pinner does not keep in-memory set of pinned CIDs, instead it relies on the datastore
- Separate recursive and direct CID indexes allow searching for pins without having to load pin data to check the mode
- New pinner can rebuild indexes on load, if saved pins appear out of sync with the indexes

fix: don't hold the pin lock while updating pins

fixes https://github.com/ipfs/go-ipfs/issues/6885

Support Async Datastores

update datastore interface to support asynchronous writes to datastores. add datastore Sync during pinner.Flush()

Otherwise, we could abort while fetching the graph and stay in a state where the
direct pin is removed.

fixes #4650

We were pining Y then removing the pin for X. When X == Y, we'd remove the new
pin.

fixes #6648

EnumerateChildrenAsync has been renamed to WalkParallel to reflect the fact
that:

1. It visits the root.
2. It's parallel, not async.

To mirror this change, EnumerateChildren has also been renamed to Walk and now
behaves the same (except that it's not parallel).

Not sure why this didn't show up sooner.

fixes #6418

We do this _just_ to make the error nicer but it's really slow. Additionally, we
do it while holding the pin lock, blocking all other pin operations.

fixes #6295

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

We were canceling the context in `GarbageCollect` but some functions call `GC`
directly. Move the context cancelation down to where we actually _need_ it.

fixes #6279

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Most of these are probably harmless but a few looked like they might actually be
bugs. Most of them are just faulty tests.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Events:

1. User triggers a GC.
2. User aborts the GC.
3. We fail to delete a block when the output channel is already full.

This is really unlikely to happen in practice but it's still incorrect.

Could be related to #6107

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Jakub Sztandera <kubuxu@protonmail.ch>

Fixes the latest batch of bugs found in RC testing.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>