README.md 8.45 KB
Newer Older
1
# go-graphsync
2 3 4 5

[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)
[![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/)
[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)
6
[![Coverage Status](https://codecov.io/gh/ipfs/go-graphsync/branch/master/graph/badge.svg)](https://codecov.io/gh/ipfs/go-graphsync)
hannahhoward's avatar
hannahhoward committed
7
[![Travis CI](https://travis-ci.com/ipfs/go-graphsync.svg?branch=master)](https://travis-ci.com/ipfs/go-graphsync)
8

Dominic Della Valle's avatar
Dominic Della Valle committed
9
> An implementation of the [graphsync protocol](https://github.com/ipld/specs/blob/master/block-layer/graphsync/graphsync.md) in go!
10 11 12 13 14 15

## Table of Contents

- [Background](#background)
- [Install](#install)
- [Usage](#usage)
16
- [Architecture](#architecture)
17 18 19 20 21
- [Contribute](#contribute)
- [License](#license)

## Background

Dominic Della Valle's avatar
Dominic Della Valle committed
22
[GraphSync](https://github.com/ipld/specs/blob/master/block-layer/graphsync/graphsync.md) is a protocol for synchronizing IPLD graphs among peers. It allows a host to make a single request to a remote peer for all of the results of traversing an [IPLD selector](https://github.com/ipld/specs/blob/master/block-layer/selectors/selectors.md) on the remote peer's local IPLD graph. 
23 24 25 26 27 28 29

`go-graphsync` provides an implementation of the Graphsync protocol in go.

### Go-IPLD-Prime

`go-graphsync` relies on `go-ipld-prime` to traverse IPLD Selectors in an IPLD graph. `go-ipld-prime` implements the [IPLD specification](https://github.com/ipld/specs) in go and is an alternative to older implementations such as `go-ipld-format` and `go-ipld-cbor`. In order to use `go-graphsync`, some understanding and use of `go-ipld-prime` concepts is necessary. 

30
If your existing library (i.e. `go-ipfs` or `go-filecoin`) uses these other older libraries, you can largely use go-graphsync without switching to `go-ipld-prime` across your codebase, but it will require some translations
31 32 33 34 35 36 37 38 39 40 41

## Install

`go-graphsync` requires Go >= 1.11 and can be installed using Go modules

## Usage

### Initializing a GraphSync Exchange

```golang
import (
42
  graphsync "github.com/ipfs/go-graphsync/impl"
43 44
  gsnet "github.com/ipfs/go-graphsync/network"
  gsbridge "github.com/ipfs/go-graphsync/ipldbridge"
45
  ipld "github.com/ipld/go-ipld-prime"
46 47 48
)

var ctx context.Context
hannahhoward's avatar
hannahhoward committed
49
var host libp2p.Host
50
var loader ipld.Loader
51
var storer ipld.Storer
52

53 54
network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
55
exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)
56 57 58 59 60
```

Parameter Notes:

1. `context` is just the parent context for all of GraphSync
61 62 63 64
2. `network` is a network abstraction provided to Graphsync on top
of libp2p. This allows graphsync to be tested without the actual network
3. `ipldBridge` is an IPLD abstraction provided to Graphsync on top of  go-ipld-prime. This makes the graphsync library testable in isolation
4. `loader` is used to load blocks from content ids from the local block store. It's used when RESPONDING to requests from other clients. It should conform to the IPLD loader interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
65
5. `storer` is used to store incoming blocks to the local block store. It's used when REQUESTING a graphsync query, to store blocks locally once they are validated as part of the correct response. It should conform to the IPLD storer interface: https://github.com/ipld/go-ipld-prime/blob/master/linking.go
66

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94
### Using GraphSync With An IPFS BlockStore

GraphSync provides two convenience functions in the `storeutil` package for
integrating with BlockStore's from IPFS.

```golang
import (
  graphsync "github.com/ipfs/go-graphsync/impl"
  gsnet "github.com/ipfs/go-graphsync/network"
  gsbridge "github.com/ipfs/go-graphsync/ipldbridge"
  gsbridge "github.com/ipfs/go-graphsync/storeutil"
  ipld "github.com/ipld/go-ipld-prime"
  blockstore "github.com/ipfs/go-ipfs-blockstore"
)

var ctx context.Context
var host libp2p.Host
var bs blockstore.Blockstore
var storer ipld.Storer

network := gsnet.NewFromLibp2pHost(host)
ipldBridge := gsbridge.NewIPLDBridge()
loader := storeutil.LoaderForBlockstore(bs)
storer := storeutil.StorerForBlockstore(bs)

exchange := graphsync.New(ctx, network, ipldBridge, loader, storer)
```

95
### Write A Loader An IPFS BlockStore
96

97 98 99 100 101 102 103 104 105
If you are using a traditional go-ipfs-blockstore, your link loading function looks like this:

```golang
type BlockStore interface {
	Get(lnk cid.Cid) (blocks.Block, error)
}
```

or, more generally:
106

hannahhoward's avatar
hannahhoward committed
107
```golang
108 109 110 111 112
type Cid2BlockFn func (lnk cid.Cid) (blocks.Block, error)
```

in `go-ipld-prime`, the signature for a link loader is as follows:

hannahhoward's avatar
hannahhoward committed
113
```golang
114 115 116
type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)
```

117
`go-ipld-prime` intentionally keeps its interfaces as abstract as possible to limit dependencies on other ipfs/filecoin specific packages. An IPLD Link is an abstraction for a CID, and IPLD expects io.Reader's rather than an actual block. IPLD provides a `cidLink` package for working with Links that use CIDs as the underlying data, and it's safe to assume that's the type in use if your code deals only with CIDs. A conversion would look something like this:
118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139

```golang
import (
   ipld "github.com/ipld/go-ipld-prime"
   cidLink "github.com/ipld/go-ipld-prime/linking/cid"
)

func LoaderFromCid2BlockFn(cid2BlockFn Cid2BlockFn) ipld.Loader {
	return func(lnk ipld.Link, lnkCtx ipld.LinkContext) (io.Reader, error) {
		asCidLink, ok := lnk.(cidlink.Link)
		if !ok {
			return nil, fmt.Errorf("Unsupported Link Type")
		}
		block, err := cid2BlockFn(asCidLink.Cid)
		if err != nil {
			return nil, err
		}
		return bytes.NewReader(block.RawData()), nil
	}
}
```

140 141 142
### Write A Storer From An IPFS BlockStore

If you are using a traditional go-ipfs-blockstore, your storage function looks like this:
143 144

```golang
145 146 147
type BlockStore interface {
	Put(blocks.Block) error
}
148 149
```

150
or, more generally:
151 152

```golang
153
type BlockStoreFn func (blocks.Block) (error)
154 155
```

156
in `go-ipld-prime`, the signature for a link storer is a bit different:
157

158 159 160 161
```golang
type StoreCommitter func(Link) error
type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)
```
162

163
`go-ipld-prime` stores in two parts to support streaming -- the storer is called and returns an IO.Writer and a function to commit changes when finished. Here's how you can write a storer from a traditional block storing signature.
164 165 166

```golang
import (
167 168 169
	blocks "github.com/ipfs/go-block-format"
  ipld "github.com/ipld/go-ipld-prime"
  cidLink "github.com/ipld/go-ipld-prime/linking/cid"
170 171
)

172 173 174 175 176 177 178 179 180 181 182 183
func StorerFromBlockStoreFn(blockStoreFn BlockStoreFn) ipld.Storer {
	return func(lnkCtx ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
		var buffer bytes.Buffer
		committer := func(lnk ipld.Link) error {
			asCidLink, ok := lnk.(cidlink.Link)
			if !ok {
				return fmt.Errorf("Unsupported Link Type")
			}
			block := blocks.NewBlockWithCid(buffer.Bytes(), asCidLink.Cid)
			return blockStoreFn(block)
		}
		return &buffer, committer, nil
184 185 186 187
	}
}
```

188
### Calling Graphsync
189 190

```golang
191 192 193 194 195 196 197 198 199 200
var exchange graphsync.GraphSync
var ctx context.Context
var p peer.ID
var selector ipld.Node
var rootLink ipld.Link

var responseProgress <-chan graphsync.ResponseProgress
var errors <-chan error

responseProgress, errors = exchange.Request(ctx context.Context, p peer.ID, root ipld.Link, selector ipld.Node)
201 202
```

203 204 205 206 207 208
Paramater Notes:
1. `ctx` is the context for this request. To cancel an in progress request, cancel the context.
2. `p` is the peer you will send this request to
3. `link` is an IPLD Link, i.e. a CID (cidLink.Link{Cid})
4. `selector` is an IPLD selector node. Recommend using selector builders from go-ipld-prime to construct these

209 210 211 212 213
### Response Type

```golang

type ResponseProgress struct {
hannahhoward's avatar
hannahhoward committed
214
  Node      ipld.Node // a node which matched the graphsync query
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229
  Path      ipld.Path // the path of that node relative to the traversal start
	LastBlock struct {  // LastBlock stores the Path and Link of the last block edge we had to load. 
		ipld.Path
		ipld.Link
	}
}

```

The above provides both immediate and relevant metadata for matching nodes in a traversal, and is very similar to the information provided by a local IPLD selector traversal in `go-ipld-prime`

## Contribute

PRs are welcome!

230 231
Before doing anything heavy, checkout the [Graphsync Architecture](docs/architecture.md)

232 233 234 235
Small note: If editing the Readme, please conform to the [standard-readme](https://github.com/RichardLitt/standard-readme) specification.

## License

236 237 238
This library is dual-licensed under Apache 2.0 and MIT terms.

Copyright 2019. Protocol Labs, Inc.