GitLab

* remove disjoint queries
* remove globally queried check
* removed dht rng field and mutex
* removed the Disjoint option
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>

* feat(query): fully async implementation of Kademlia lookup. peers returned from the lookup are not guaranteed to be alive (i.e. we're only guaranteed to have dialed the closest beta peers to the target), but given stable and correct routing tables the expectation that most of the peers returned are alive is high.
* feat(query): add wrapper lookup followup function to followup after the lookup is completed and ensure that the closest k returned peers from a lookup have been queried even for beta < k
* refactor(query) modified the structure returned from lookups to be a useful subset of the full query state instead of the entire query state
* feat(options): beta parameter exposed as the Resiliency parameter
* feat(routing): do not mark the routing table as updated after a FindPeer query
* feat(routing): FindPeer can return addresses even if not Connected as long as it was either recently connected (CanConnect) or was discovered during the lookup
* feat(bootstrap): bootstrap logic now uses GetClosestPeers instead of FindPeer
* refactor(dht): stopFn no longer takes any state
* fix(test): changed GetClosestPeers test to only assume beta instead of k peers since that is now more appropriate given the query logic changes and that the routing tables in that test are bad, i.e. a ring network with arbitrary peerIDs
Co-authored-by: Petar Maymounkov <petarm@gmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

* kpeerset refactoring
* query code cleanup

options: KValue and AlphaValue global variables no longer used internally. Concurrency option now sets alpha. DisjointPaths option now sets d. Default number of disjoint paths is now bucketSize/2.

query switches from alpha mode to k mode if no peers closer than one we have heard about (in a given path) has been found

fix Search/GetValue to be Kad compliant. Default quorum is now 0 which means do not abort the query early

When the routing table is empty, `Run` would fail but _not_ close the
process (leaking some query goroutines). This patch fixes this in multiple
places by:

1. Not starting queries with no peers.
2. Failing queries with no peers earlier.

Long-running queries can build up large error sets that we never actually use.
This is exacerbated by https://github.com/libp2p/go-libp2p-swarm/pull/115.

fixes https://github.com/libp2p/go-libp2p-swarm/issues/119

When a peer is found, add its addr to the peerstore, just like the addresses for closer peers are added. In particular this is helpful for followup queries by the DHT user.

fixes #276

This patch introduces an adaptive dial queue that spawns a dynamically sized
set of goroutines to preemptively stage dials for later handoff to the DHT
protocol for RPC. It identifies backpressure on both ends (dial consumers and
dial producers), and takes compensating action by adjusting the worker pool.

We start with `DialQueueMinParallelism` number of workers (6), and scale up
and down based on demand and supply of dialled peers.

The following events trigger scaling:
- we scale up when we can't immediately return a successful dial to a new
  consumer.
- we scale down when we've been idle for a while waiting for new dial
  attempts.
- we scale down when we complete a dial and realise nobody was waiting for it.

Dialler throttling (e.g. FD limit exceeded) is a concern, as we can easily
spin up more workers to compensate, and end up adding fuel to the fire. Since
we have no deterministic way to detect this for now, we hard-limit concurrency
to `DialQueueMaxParallelism` (20).

Both, dht.go and dht_bootstrap.go duplicate the package documentation.
The whole block had been moved to doc.go

Also query.go provides documentation for query package but now it
belongs to dht package, so it's been removed.

Without this, one of the workers may hang when trying to re-grab the rate-limit
after dialing. I believe this may be what's causing some of our DHT requests to
stall (evidence: massive goroutine buildup on the gateways).

First, we use Alpha instead of K as the number of peers we grab from the
routing table (as per the kademlia paper).

Second, we don't use a size limited set for the 'GetClosestPeers' query.
We're going to process more than K peers before we find the K closest
peers.

Third, Change GetClosestPeers to actually return the K Closest peers,
not a hodge podge of peers that it found on the way to finding the
closest peers.