GitLab

* fullrt: add experimental FullRT DHT client

The FullRT client periodically crawls the network to fill its local routing table and uses that to perform queries. It supports performing many provides or puts together taking advantage of efficiencies of grouping the DHT operations by where they will occur in the Kademlia space. Additionally, it has a more tunable function for determining when a query is complete that does not require waiting on any individual peer (which could be offline) to respond. This client is experimental and its exposed interfaces should be expected to change and break over time.

* crawler: starting peers with addresses and peers found during a crawl have their addresses extended. logging improved

* dht: move IpfsDHT options to the internal package. Make a breaking change to the filter interfaces to support more DHT implementations

* dht: GetClosestPeers now returns a slice of peers instead of a channel of peers since we have to wait for the query to complete to return the closest peers anyway

* dht: the subscriberNotifiee has been refactored to work more independently of the underlying message sender implementation

dht.msgSender

refactor: rename messageSender to peerMessageSender and added comments to clarify what peerMessageSender and messageManager do

refactor: extract messaging components from IpfsDHT into its own struct. create a new struct that manages sending DHT messages that can be used independently from the DHT.

Update for stream interface changes. See
https://github.com/libp2p/go-libp2p-core/releases/tag/v0.7.0 for details.

These are all critical paths so we should try to do as little work as possible.

* Remove deprecated events.
* Fix warning -> warn.
* Use structured logging (some places).

Except deprecated warn calls

* new RT management policy

This should stop us from waiting on unresponsive peers. This only kicks in when
we've already _sent_ a request so this:

* Shouldn't be blocked on other requests we're planning on making.
* Shouldn't be blocked on dialing.

* upgraded the protocol id to version 2 (i.e. /kad/2.0.0) and made it so v2 peers running in server mode respond to queries from v1 peers. Note: v2 peers will only send queries using the v2 protocol, will only add v2 peers to their routing tables, and will only tell v1 peers about v2 peers.
* to run a forked network we now use network specific protocol prefixes instead of manually setting protocol IDs. Use the ProtocolPrefix option instead of the Protocols option.
* emit errors during initialization if the user misuses the default protocol prefix by setting parameters inconsistent with the default protocol's network specification
* since the Client option has been deprecated it's been removed from the dht's options. While deprecated it is still available in the dht options package. Setting `Client(false)` now puts the node into ModeAuto.

* created Mode(ModeOpt) option for choosing between auto/client/server modes
* Auto mode internally switches the DHT between client and server modes based on the EvtLocalReachabilityChanged event emitted on the event bus (e.g. by AutoNAT)
* routing table management of peers that switch between client and server mode while we are connected to them (i.e. are in auto mode)
* removed Client(bool) option, becoming a DHT client is specified using Mode(ModeClient) instead

* feat: consume identify events to evaluate routing table addition
* fix: routing table no longer gets an update just because new messages have arrived or been sent
* fix: add already connected peers into the routing table before listening to events
Co-authored-by: Raúl Kripalani <raul.kripalani@gmail.com>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>

... otherwise any kinde of error ratio is meaningless.

Related to #453 but not a fix. This will cause us to actually return early when
we start blocking on sending to some peers, but it won't really _unblock_ those
peers. For that, we need to write with a context.

It's currently 10 minutes. This change will kill all inbound DHT streams that
haven't been used in 1 minute.

We keep streams around to avoid the cost of setting up new streams (the
multistream overhead, mostly). However, each one of these takes at least one
goroutine and some other state.

Ideally, this will let us run multiple DHTs side-by-side without keeping one
stream per peer per DHT open.

fixes #389

Credit goes to @sidenaio for finding this.

Update msgio to latest version
Use max size in msgio readers
Fix error handling in reads

This was causing us to build up a bunch of background goroutines. Technically,
we should be resetting all these streams when the DHT stops. However, we don't
actually unregister stream handlers from the host so, really, nothing will
actually stop till the host stops. When it does, it'll kill all the streams
anyways.

Allocate them as-needed and use a pool.

Work towards #322.

Add initial set of metrics exposed via OpenCensus.

We usually _explicitly_ call connect before calling `prep` but we may disconnect
somewhere in-between. We _don't_ want to get stuck here dialing if the context
has been canceled.

This reverts commit 7e68ac3c.