GitLab

* fullrt: add experimental FullRT DHT client

The FullRT client periodically crawls the network to fill its local routing table and uses that to perform queries. It supports performing many provides or puts together taking advantage of efficiencies of grouping the DHT operations by where they will occur in the Kademlia space. Additionally, it has a more tunable function for determining when a query is complete that does not require waiting on any individual peer (which could be offline) to respond. This client is experimental and its exposed interfaces should be expected to change and break over time.

* crawler: starting peers with addresses and peers found during a crawl have their addresses extended. logging improved

* dht: move IpfsDHT options to the internal package. Make a breaking change to the filter interfaces to support more DHT implementations

* dht: GetClosestPeers now returns a slice of peers instead of a channel of peers since we have to wait for the query to complete to return the closest peers anyway

* dht: the subscriberNotifiee has been refactored to work more independently of the underlying message sender implementation

refactor: extract messaging components from IpfsDHT into its own struct. create a new struct that manages sending DHT messages that can be used independently from the DHT.

Handle empty keys, both when sent in RPC requests and in the local API.

* Remove deprecated events.
* Fix warning -> warn.
* Use structured logging (some places).

* remove disjoint queries
* remove globally queried check
* removed dht rng field and mutex
* removed the Disjoint option
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>

* feat(query): fully async implementation of Kademlia lookup. peers returned from the lookup are not guaranteed to be alive (i.e. we're only guaranteed to have dialed the closest beta peers to the target), but given stable and correct routing tables the expectation that most of the peers returned are alive is high.
* feat(query): add wrapper lookup followup function to followup after the lookup is completed and ensure that the closest k returned peers from a lookup have been queried even for beta < k
* refactor(query) modified the structure returned from lookups to be a useful subset of the full query state instead of the entire query state
* feat(options): beta parameter exposed as the Resiliency parameter
* feat(routing): do not mark the routing table as updated after a FindPeer query
* feat(routing): FindPeer can return addresses even if not Connected as long as it was either recently connected (CanConnect) or was discovered during the lookup
* feat(bootstrap): bootstrap logic now uses GetClosestPeers instead of FindPeer
* refactor(dht): stopFn no longer takes any state
* fix(test): changed GetClosestPeers test to only assume beta instead of k peers since that is now more appropriate given the query logic changes and that the routing tables in that test are bad, i.e. a ring network with arbitrary peerIDs
Co-authored-by: Petar Maymounkov <petarm@gmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

1. Continue to best-effort provide, but still return an error when we fail to
send provider records to the _best_ peers.
2. Continue returning the best peer's we've found in GetClosestPeers, but also
return an error to indicate that we didn't find the closest ones.

And fix the hang test.

query switches from alpha mode to k mode if no peers closer than one we have heard about (in a given path) has been found

fix Search/GetValue to be Kad compliant. Default quorum is now 0 which means do not abort the query early

This reverts commit 7e68ac3c.

not really an error, it doesn't stop us from logging.

fixes ipfs/go-ipfs#4237

Try to preallocate slices up-front instead of repeatedly reallocating.

Currently keys that get logged are casted from []byte. Which means that log output looks like this.
```
{"event":"getClosestPeersBegin","key":"\u0012 \ufffdᱡ\ufffd`\ufffdF]\ufffd\ufffdᠸCƶ\u0006E\ufffdrb\ufffd\ufffdO\ufffd\ufffdu\ufffd\ufffdj\u000b","system":"dht","time":"2017-07-27T11:12:08.51775836Z"}
```
after fix like this:
```
{"event":"getClosestPeersBegin","key":"QmQyyaB8znrakGydP3ynRsXre96Uw9wc6w6zuP61bbZuNj","system":"dht","time":"2017-07-27T11:16:30.693008112Z"}
```

First, we use Alpha instead of K as the number of peers we grab from the
routing table (as per the kademlia paper).

Second, we don't use a size limited set for the 'GetClosestPeers' query.
We're going to process more than K peers before we find the K closest
peers.

Third, Change GetClosestPeers to actually return the K Closest peers,
not a hodge podge of peers that it found on the way to finding the
closest peers.