GitLab

It's possible to start receiving and processing messages before we get around to starting.

* Separate decision engine ledger on two parts: score and the wantlist

This is the first step to make external decision logic (tagging
peers with score values) possible.

The wantlist still resides in the original `ledger` struct while
sent/received byte accounting and scores are extracted to the new
`scoreledger` struct managed by the original `scoreWorker()` logic.
The accounting is integrated into the `Engine` via `ScoreLedger`
interface making it possible to replace the original `scoreWorker()`
with some other logic. The interface, however, doesn't allow a
score logic to directly touch peer tags: the logic may decide about
score values while tagging itself is still under control of Engine.

Note: with this commit it's yet not possible to replace the original
score logic because there is no public methods for that.

* Added "WithScoreLedger" Bitswap option

New `WithScoreLedger(decision.ScoreLedger)` option in the `bitswap`
package is the way to connect a custom `ScoreLedger` implementation
to the decision engine. The `Engine` now has the corresponding
`UseScoreLedger(ScoreLedger)` method.

The `ScoreLedger` and `ScorePeerFunc` types are exposed from the
internal `decision` package to the public one.

Because its options are processed by the `Bitswap` after construction
of its parts but before starting of the engine, the default
`scoreLedger` initialization is moved from `newEngine()` to
`StartWorkers()`.

New `TestWithScoreLedger` test is added. The test checks for start and
stop of the testing score ledger implementation that is specified via
`WithScoreLedger` option.

* Combine score ledger start with initialization of the score function

Having a separate `Init(ScoreFunc)` method seems redundant (thx
@dirkmc for pointing about that). As a bonus, the two-step ledger
starting process is now enclosed in the `startScoreLedger()` function.

* Let's call Stop() to stop a ScoreLedger

The `Close()` method was there to stop the ledger. Let call it
`Stop()` now.

* Get return of the blank Receipt out of conditional block

Explicitly form it as the final resort.
Co-authored-by: Paul Wolneykien <manowar@altlinux.org>

I'm not sure why we set "full" to true here, but this could be the source of a
whole bunch of bidirectional sync issues. That is, if two peers are syncing off
each other, they could repeatedly "reset" each other's wantlist to "empty".

Quick alternative to #407 to fix the main issue.

* fix: minimize time holding wantlist lock

Instead of holding the lock the entire time we prepare a message, hold the lock
while we retrieve the wantlist entries, process the entries without the lock,
retake the lock, then mark entries as sent.

This means:

1. We never sort entries while holding the lock.
2. We allocate exactly three times while holding the lock (once per entry list).

* fix: address code review

This makes reading the docs much easier as it's clear what's "private" and
what's not.

fixes #238

This commit extends the bitswap protocol with two additional wantlist properties:

* WANT_HAVE/HAVE: Instead of asking for a block, a node can specify that they
  want to know if any peers "have" the block.
* WANT_HAVE_NOT/HAVE_NOT: Instead of waiting for a timeout, a node can explicitly
  request to be told immediately if their peers don't currently have the given
  block.

Additionally, nodes now tell their peers how much data they have queued to send
them when sending messages. This allows peers to better distribute requests,
keeping all peers busy but not overloaded.

Changes in this PR are described in: https://github.com/ipfs/go-bitswap/issues/186

* feat: parallelize reads

* feat: concurent engine task workers and concurrent bstore reads

* fix: lint

* fix: address review comments

* refactor: in BlockstoreManager wait for process.Closing() instead of Context.Done()

* fix: use channel size 0 for BlockstoreManager reads

* fix: change blockstore error logs from warnings to errors

* fix: flaky test

* fix: lint

This patch tracks two usefulness metrics: short-term usefulness and long-term
usefulness. Short-term usefulness is sampled frequently and highly weights new
observations. Long-term usefulness is sampled less frequently and highly weights
on long-term trends.

In practice, we do this by keeping two EWMAs. If we see an interaction within
the sampling period, we record the score, otherwise, we record a 0. The
short-term one has a high alpha and is sampled every shortTerm period. The
long-term one has a low alpha and is sampled every longTermRatio*shortTerm
period.

To calculate the final score, we sum the short-term and long-term scores then
adjust it ±25% based on our debt ratio. Peers that have historically been more
useful to us than we are to them get the highest score.

feat: process response message blocks as a batch

tag peers in connection manager as they have outstanding requests for blocks to serve

fix #114

Uses shared external package peer task queue in place of peer request queue. Shared by graphsync.

Add comments to all exported functions, extract the utils for creating instances in testnet.go,
moves integration tests to bitswap_test

BREAKING CHANGE: removed one constant -- rebroadcastDelay -- which I believe was unused

(fixes linter issues)

We allocate a _lot_ of these.

per https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences

Wantlist/Blocks *copy*.

Updated PeerRequestTask to hold multiple wantlist.Entry(s). This allows Bitswap to send multiple blocks in bulk per a Peer's request. Also, added a metric for how many blocks to put in a given message. Currently: 512 * 1024 bytes. 

License: MIT
Signed-off-by: Jeromy <why@ipfs.io>

Updates:

* go-net
* go-text
* dns
* prometheus
* protobuf (golang, not gogo)

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

alternative to #5243 that updates go-cid and all packages that depend on it

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

Updates:

* go-kad-dht: Query performance improvements, DHT client fixes, validates
  records on *local* put.
* go-libp2p-swarm/go-libp2p-transport: Timeout improvements.
* go-multiaddr-net: Exposes useful Conn methods (CloseWrite, CloseRead, etc.)
* go-log: fixes possible panic when enabling/disabling events.
* go-multiaddr: fixes possible panic when stringifying malformed multiaddrs,
  adds support for consuming /p2p/ multiaddrs.

fixes #5113
unblocks #4895

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

* go-log
* sys
* go-crypto

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>

This extracts the blocks/blockstore package and renames the
blocks/blockstore/util package to /blocks/blockstoreutil
(because util depends on Pin and I don't plan to extract
Pin and its depedencies).

The history of blocks/blockstore has been preserved. It has
been gx'ed and imported. Imports have been rewritten accordingly
and re-ordered.

License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>