GitLab

* fix: minimize time holding wantlist lock

Instead of holding the lock the entire time we prepare a message, hold the lock
while we retrieve the wantlist entries, process the entries without the lock,
retake the lock, then mark entries as sent.

This means:

1. We never sort entries while holding the lock.
2. We allocate exactly three times while holding the lock (once per entry list).

* fix: address code review

This makes reading the docs much easier as it's clear what's "private" and
what's not.

fixes #238

This commit extends the bitswap protocol with two additional wantlist properties:

* WANT_HAVE/HAVE: Instead of asking for a block, a node can specify that they
  want to know if any peers "have" the block.
* WANT_HAVE_NOT/HAVE_NOT: Instead of waiting for a timeout, a node can explicitly
  request to be told immediately if their peers don't currently have the given
  block.

Additionally, nodes now tell their peers how much data they have queued to send
them when sending messages. This allows peers to better distribute requests,
keeping all peers busy but not overloaded.

Changes in this PR are described in: https://github.com/ipfs/go-bitswap/issues/186

* feat: parallelize reads

* feat: concurent engine task workers and concurrent bstore reads

* fix: lint

* fix: address review comments

* refactor: in BlockstoreManager wait for process.Closing() instead of Context.Done()

* fix: use channel size 0 for BlockstoreManager reads

* fix: change blockstore error logs from warnings to errors

* fix: flaky test

* fix: lint

This patch tracks two usefulness metrics: short-term usefulness and long-term
usefulness. Short-term usefulness is sampled frequently and highly weights new
observations. Long-term usefulness is sampled less frequently and highly weights
on long-term trends.

In practice, we do this by keeping two EWMAs. If we see an interaction within
the sampling period, we record the score, otherwise, we record a 0. The
short-term one has a high alpha and is sampled every shortTerm period. The
long-term one has a low alpha and is sampled every longTermRatio*shortTerm
period.

To calculate the final score, we sum the short-term and long-term scores then
adjust it ±25% based on our debt ratio. Peers that have historically been more
useful to us than we are to them get the highest score.

feat: process response message blocks as a batch

tag peers in connection manager as they have outstanding requests for blocks to serve

fix #114

Uses shared external package peer task queue in place of peer request queue. Shared by graphsync.

Add comments to all exported functions, extract the utils for creating instances in testnet.go,
moves integration tests to bitswap_test

BREAKING CHANGE: removed one constant -- rebroadcastDelay -- which I believe was unused

(fixes linter issues)

We allocate a _lot_ of these.

per https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences

Wantlist/Blocks *copy*.

Updated PeerRequestTask to hold multiple wantlist.Entry(s). This allows Bitswap to send multiple blocks in bulk per a Peer's request. Also, added a metric for how many blocks to put in a given message. Currently: 512 * 1024 bytes. 

License: MIT
Signed-off-by: Jeromy <why@ipfs.io>

Updates:

* go-net
* go-text
* dns
* prometheus
* protobuf (golang, not gogo)

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

alternative to #5243 that updates go-cid and all packages that depend on it

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Updates:

* go-kad-dht: Query performance improvements, DHT client fixes, validates
  records on *local* put.
* go-libp2p-swarm/go-libp2p-transport: Timeout improvements.
* go-multiaddr-net: Exposes useful Conn methods (CloseWrite, CloseRead, etc.)
* go-log: fixes possible panic when enabling/disabling events.
* go-multiaddr: fixes possible panic when stringifying malformed multiaddrs,
  adds support for consuming /p2p/ multiaddrs.

fixes #5113
unblocks #4895

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

* go-log
* sys
* go-crypto

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>

This extracts the blocks/blockstore package and renames the
blocks/blockstore/util package to /blocks/blockstoreutil
(because util depends on Pin and I don't plan to extract
Pin and its depedencies).

The history of blocks/blockstore has been preserved. It has
been gx'ed and imported. Imports have been rewritten accordingly
and re-ordered.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Avoids lots of reallocations under a lock.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>