GitLab

* Add WireTap interface

WireTap interface can be used to access all messages send and received
by Bitswap. This can be used to implement advanced statistics/analysis
logic, which is beyond scope of Bitswap, but can be implemented as IPFS
plugin.

Some examples of potential applications:
 - per CID bandwidth tracker (see: https://gitcoin.co/issue/PinataCloud/apollo/2/100023631)
 - detailed per peer stats
 - intrusion detection system (IDS) implementation

* Add test for WireTap

Also, reduce the overhead from logging.

* fix: ensure broadcast when peer becomes unavailable and all other peers sent DONT_HAVE for CID

* fix: lint warnings

* refactor: simplify session want sender DONT_HAVE list

* fix: flaky test

* test: add session exhausted wants test

* docs: improve sessionWantSender processAvailability docs

This makes reading the docs much easier as it's clear what's "private" and
what's not.

fixes #238

This commit extends the bitswap protocol with two additional wantlist properties:

* WANT_HAVE/HAVE: Instead of asking for a block, a node can specify that they
  want to know if any peers "have" the block.
* WANT_HAVE_NOT/HAVE_NOT: Instead of waiting for a timeout, a node can explicitly
  request to be told immediately if their peers don't currently have the given
  block.

Additionally, nodes now tell their peers how much data they have queued to send
them when sending messages. This allows peers to better distribute requests,
keeping all peers busy but not overloaded.

Changes in this PR are described in: https://github.com/ipfs/go-bitswap/issues/186

Add comments to all exported functions, extract the utils for creating instances in testnet.go,
moves integration tests to bitswap_test

BREAKING CHANGE: removed one constant -- rebroadcastDelay -- which I believe was unused

Goprocess returns a _custom_ context type. Unfortunately, golang has a bunch of
magic type assertions to efficiently handle built-in context types but launches
a new goroutine when deriving a new context from a custom context type.
Otherwise, it has no way to wait on the custom context's channel.

This fix just ensures we only ever have one of goroutines per provide worker by
deriving a (normal) cancelable context up-front and then using that.

Make Bitswap GetBlocks just create a temporary session and use that code

fix #52 fix #49

Also, buffer single-use channels we may walk away from. This was showing
up (rarely) in a go-ipfs test.

Seperates the functions of tracking wants from tracking peers
Unit tests for both peer manager and want manager
Refactor internals of both to address synchonization issues discovered
in tests

Extract session manager from bitswap
Extract session manager & want manager to package
Move want manager message queue to seperate file
Move Message Queue to subpackage
Respond to PR Comments

Updated PeerRequestTask to hold multiple wantlist.Entry(s). This allows Bitswap to send multiple blocks in bulk per a Peer's request. Also, added a metric for how many blocks to put in a given message. Currently: 512 * 1024 bytes. 

License: MIT
Signed-off-by: Jeromy <why@ipfs.io>

alternative to #5243 that updates go-cid and all packages that depend on it

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Updates:

* go-kad-dht: Query performance improvements, DHT client fixes, validates
  records on *local* put.
* go-libp2p-swarm/go-libp2p-transport: Timeout improvements.
* go-multiaddr-net: Exposes useful Conn methods (CloseWrite, CloseRead, etc.)
* go-log: fixes possible panic when enabling/disabling events.
* go-multiaddr: fixes possible panic when stringifying malformed multiaddrs,
  adds support for consuming /p2p/ multiaddrs.

fixes #5113
unblocks #4895

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

* go-log
* sys
* go-crypto

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Base58 encoding cids/peerIDs isn't exactly fast.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>

And updated related dependencies.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

When sending data to another user, the number of bytes sent to that user (saved
by the corresponding Bitswap ledger) was not updated (it was always 0). This
also meant that the debt ratio was also always 0.

The function that updates the `BytesSent` value in the ledger, `MessageSent()`,
was already implemented, however it was not called when the peer was sent data.
To fix this, a call to `MessageSent()` was made in the `taskWorker()` function,
which is where both the message in question and the Bitswap engine were
available to make the call. `MessageSent()` requires the peer's ID and
`BitSwapMessage` as its arguments, the latter of which had to be created by
making a new `BitSwapMessage`, then the block being sent was added to the new
message.

Note that, similar to the analagous call to `MessageReceived()`, records *all*
of the bytes sent to a particular user. At some point, both of these should be
updated to only record the numbers of *useful* bytes sent and received between
peers.

License: MIT
Signed-off-by: David Grisham <dgrisham@mines.edu>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT
Signed-off-by: Jeromy <why@ipfs.io>