GitLab

* fullrt: add experimental FullRT DHT client

The FullRT client periodically crawls the network to fill its local routing table and uses that to perform queries. It supports performing many provides or puts together taking advantage of efficiencies of grouping the DHT operations by where they will occur in the Kademlia space. Additionally, it has a more tunable function for determining when a query is complete that does not require waiting on any individual peer (which could be offline) to respond. This client is experimental and its exposed interfaces should be expected to change and break over time.

* crawler: starting peers with addresses and peers found during a crawl have their addresses extended. logging improved

* dht: move IpfsDHT options to the internal package. Make a breaking change to the filter interfaces to support more DHT implementations

* dht: GetClosestPeers now returns a slice of peers instead of a channel of peers since we have to wait for the query to complete to return the closest peers anyway

* dht: the subscriberNotifiee has been refactored to work more independently of the underlying message sender implementation

refactor: extract messaging components from IpfsDHT into its own struct. create a new struct that manages sending DHT messages that can be used independently from the DHT.

They've told us better ones via identify.

Handle empty keys, both when sent in RPC requests and in the local API.

* Remove deprecated events.
* Fix warning -> warn.
* Use structured logging (some places).

Except deprecated warn calls

* remove disjoint queries
* remove globally queried check
* removed dht rng field and mutex
* removed the Disjoint option
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>

* feat(query): fully async implementation of Kademlia lookup. peers returned from the lookup are not guaranteed to be alive (i.e. we're only guaranteed to have dialed the closest beta peers to the target), but given stable and correct routing tables the expectation that most of the peers returned are alive is high.
* feat(query): add wrapper lookup followup function to followup after the lookup is completed and ensure that the closest k returned peers from a lookup have been queried even for beta < k
* refactor(query) modified the structure returned from lookups to be a useful subset of the full query state instead of the entire query state
* feat(options): beta parameter exposed as the Resiliency parameter
* feat(routing): do not mark the routing table as updated after a FindPeer query
* feat(routing): FindPeer can return addresses even if not Connected as long as it was either recently connected (CanConnect) or was discovered during the lookup
* feat(bootstrap): bootstrap logic now uses GetClosestPeers instead of FindPeer
* refactor(dht): stopFn no longer takes any state
* fix(test): changed GetClosestPeers test to only assume beta instead of k peers since that is now more appropriate given the query logic changes and that the routing tables in that test are bad, i.e. a ring network with arbitrary peerIDs
Co-authored-by: Petar Maymounkov <petarm@gmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

* kpeerset refactoring
* query code cleanup

1. Continue to best-effort provide, but still return an error when we fail to
send provider records to the _best_ peers.
2. Continue returning the best peer's we've found in GetClosestPeers, but also
return an error to indicate that we didn't find the closest ones.

And fix the hang test.

fix Search/GetValue to be Kad compliant. Default quorum is now 0 which means do not abort the query early

* feat: Expose the ProviderManager through the IpfsDht struct
* feat: expose providers prefix so that we do not have to hard code it in other places
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

fixes #274