Datastore based pinner (#4)
Andrew Gillis authored
feat: store pins in datastore instead of a DAG

Adds a new `/pins` namespace to the given datastore and uses that to store pins as cbor binary, keyed by unique pin ID.

The new datastore pinner stores pins in the datastore as individual key-value items. This is faster than the dag pinner, which stored all pins in a single dag that had to be rewritten every time a pin was added or removed.

The new pinner provides a secondary indexing mechanism that can be used to index any data that a pin has. Secondary indexing logic is provided by the `dsindex` package. The new pinner currently includes indexing by CID.

Both the new datastore pinner (`dspinner` package) and the old dag pinner (`ipldpinner` package) implementations are included to support migration between the two.  Migration logic is provided by the `pinconv` package.

Other features in new pinner:
- Benchmarks are provided to compare performance of between the old and new pinners
- New pinner does not keep in-memory set of pinned CIDs, instead it relies on the datastore
- Separate recursive and direct CID indexes allow searching for pins without having to load pin data to check the mode
- New pinner can rebuild indexes on load, if saved pins appear out of sync with the indexes
4c920717

go-ipfs-pinner

Coverage Status Travis CI

Background

The pinner system is responsible for keeping track of which objects a user wants to keep stored locally

Install

Via go get:

$ go get github.com/ipfs/go-ipfs-pinner

Requires Go 1.13

Documentation

https://godoc.org/github.com/ipfs/go-ipfs-pinner

Contribute

PRs are welcome!

Small note: If editing the Readme, please conform to the standard-readme specification.

License

This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.