GitLab

* log gs traversal

* Apply suggestions from code review
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* add debug logs

* add debug logs
Co-authored-by: dirkmc <dirkmdev@gmail.com>

* fix(allocator): don't let allocs dip below zero

* refactor(allocator): move to top level

move allocator to top level dir since it's used only in messagequeue now

* fix(responsemanager): cleanup requests after network error

* fix(messagequeue): release memory synchronously

refactor messagequeue so that memory is released synchronously as soon as a message is sent or
errors

* refactor(allocator): respond to PR comments

clarify intent via addressing comments on PR

Co-authored-by: acruikshank <acruikshank@example.com>

* WIP

* WIP

* network tests passing

* fix peermanager test

* requestmanager passes

* responsemanager test passing

* peerresponsebuilder test passing

* network and messagequeue tests are passing

* fix imports

* refactor(responsemanager): test cleanup

cleanup response manager test and continue cleaning up names

* refactor(graphsync): add more code documentation

* refactor(responsemanager): finish rename cleanup

remove references to old peerresponsemanager, peerresponsesender

* fix(messagequeue): don't miss queued messages

move large blocks test to message queue where it belongs, make sure we properly handle queued
messages

* Apply suggestions from code review
Co-authored-by: Alex Cruikshank <169613+acruikshank@users.noreply.github.com>

* refactor(responseassembler): rename transaction builder

rename transaction builder to response builder
Co-authored-by: hannahhoward <hannah@hannahhoward.net>
Co-authored-by: acruikshank <acruikshank@example.com>

fix various issues causing network errors not to propogate in many cases

also cleanup imports

* fix: RegisterNetworkErrorListener should fire when there's an error connecting to the peer

* feat(listeners): move listeners to top level folder

* feat(requestmanager): put network error in request manager

Put network error in request manager to cover more cases

* feat(requestmanager): correct for notification refactor

Resolve discripencies with new and old notification interfaces
Co-authored-by: hannahhoward <hannah@hannahhoward.net>

* modify transformable event logic to simply add data

* permit multiple data topics so sent hook gets calle for every block

* rename test to match method rename.
Co-authored-by: dirkmc <dirkmdev@gmail.com>
Co-authored-by: acruikshank <acruikshank@example.com>
Co-authored-by: dirkmc <dirkmdev@gmail.com>

provide a configuration option to change the maximum number of requests processed at once by the
responder

remove peer from peer status list when deallocing peer memory

* test(benchmarks): add large file tests

add tests of large file processing to benchmarks

* feat(allocator): add allocator for memory backpressure

add an allocator that manages global memory allocations on responder and blocks peerresponsesenders
as needed

* feat(allocator): add method to release all peer memory

* feat(allocator): refactor w/ mutex

refactor allocator to remove go routine and address a few PR comments

* Update responsemanager/allocator/allocator.go
Co-authored-by: Steven Allen <steven@stebalien.com>

* fix(impl): update constants in readable form
Co-authored-by: Steven Allen <steven@stebalien.com>

* test(benchmarks): add benchmark for go routine leak

* fix(notifications): shutdown notification queues

make sure all notification publishers are actually shut down, and don't start them automatically

update for 0.3.0 + style fixes

* feat(notifications): build notifications architecture

build an architecture via which we can pass on notifications

* feat(peerresponsesender): add notifications

pass notifications on from peer response sender based on message queue notifications

* feat(responsemanager): handle network errors

Pass on network errors through the response manager, pass block sent errors as well

* fix(notifications): send unique notifications by peer

* feat(responsemanager): subscriber per request

* Revert "fix(notifications): send unique notifications by peer"

This reverts commit 7bf368a05d14229b7523cb25068681d9c7e48246.

* Update go-ipld-prime@v0.5.0

Simultaneously updates go-ipld-prime-proto, which also has changes to
match that version of go-ipld-prime.

* feat(deps): update ipld-prime near master

update to lastest needed sha for filecoin, reinstitute generated node tests

* feat(deps): update go-ipld-prime-prot
Co-authored-by: hannahhoward <hannah@hannahhoward.net>

Use cbor-gen encoding for speed bump on metadata

* feat(benchmarks): initial benchmark infrastructure

* fix(cidlink): mem allocations around link loading

* fix(deps): update to latest deps

use latest go-ipld-prime & go-ipld-prime-proto fixes

* fix(deps): remove unused badger code

* feat(responsemanager): dedup-by-key extension

add deduping requests by a given key to the response manager

* feat(requestmanager): use dedup extension

use dedup extension to not dedup data across stores

* style(imports): fix import formatting

Fix import formatting via script. Also add script for import fixes and release log

* docs(CHANGELOG): add changelog, contributing

Add the changelog and a contributing guidelines doc

* feat(persistenceoptions): add unregister ability

add unregister ability to persistence options so we can create short lived options for single
requests

* fix(lint): fix lint issues

This reverts commit 31cc0d5e.

Needs further exploration

* feat(hooks): add extensions on complete

add the ability to send extensions with a final message

* refactor(hooks): complete listener->hook rename

All changes to date including pause requests & start paused, along with new adds for cleanups and checking of execution (#75)

* WIP

* feat(graphsync): pause/unpause requests

Allow graphsync requests to be paused and unpaused via request cancelling and DoNotSendCIDs
extension

* fix(requestmanager): refactor executor

remove extraneous allocation of closure functions

* feat(graphsync): support external request pauses

allow pausing requests imperatively via PauseRequest function

* fix(lint): fix lint errors

* feat(responsemanager): start requests paused

add the ability for a hook to pause a response right when it's first received

* feat(responsemanager): improve cancellation UX

provide a mechanism for responders to learn a requestor cancelled a request and for requestors to
learn a request was cancelled

* feat(requestmanager): process request cancelled status

process the responder returning a request cancelled error code and also support sentinel errors

* feat(executor): refactor to remove loader

remove loader, also only fire restart request as needed

* fix(asyncloader): load requests synchronously when possible

* fix(responsemanager): fix external pause

Fix external pauses missing a block

* fix(responsemanager): do not delay complete listener

Run complete listener in same thread as response processing, making it less susceptable to
interruption via cancel

* fix(responsemanager): fix context check

fix checking for context cancellation errors based off of the way ipld-prime does not wrap errors

* fix(responsemanager): more precise cancel

make cancels only get recorded if actual blocks are not sent -- otherwise the request is considered
complete -- and the complete hook always runs

* fix(requestmanager): handle non processed pauses

Handler the case where a pause is requested but never actually takes place

* refactor(responsemanager): handle cancels, correctly this time

Properly handle cancels for both paused and unpaused states

* fix(errors): remove regex cause it appears to be very slow

* fix(traverser): fix race condition for shutdown

make sure that the traverser is finished in the request executor

* fix(deps): mod tidy

* fix(executor): add back network error

* feat(responsemanager): add ability to pause response outside of a hook

Add the ability for anyone who knows a requestID & peer to pause a response at any time

* feat(responsemanager): add direct cancellations

add function to directly cancel responses from requestID. also, move query execution code to
seperate struct, internal for now

* fix(responsemanager): minor heap allocation optimization

* feat(responsemanager): support extensions on resume

Support sending extensions when resuming a request

* fix(lint): fix lint errors

Refactor request execution and use IPLD SkipMe functionality for proper partial results on a request (#70)

* refactor(requestmanager): refactor execution of requests

* feat(graphsync): properly use SkipMe from ipld to complete partial selections

* fix(requestmanager): remove unused function

Provides full implementation of do-no-send-cids on responder side

* feat(graphsync): add incoming block hook types

add types (not implemented) for incoming block hook

* feat(asyncloader): identify local loads

add a boolean value to AsyncLoadResult to determine if a request load was local

* fixup! feat(graphsync): add incoming block hook types

* feat(hooks): add block hooks plumbing

* feat(requestmanager): process block hooks

Process block hooks on each block, and connect last response to blocks

* feat(peerresponsemanager): add transactions

add the ability to do multple response operations with a gaurantee they will end up in a single
response

* refactor(lint): fix lint errors

add nil check for an empty task (task deleted but not finished removing from peer task queue

* Updating go-ipld-prime to NodeAssembler era.

The 'testutil/chaintypes' codegen is currently dropped, and I ported
the testchain fixtures to using basicnode instead.  We can restore this
to using codegen soon, but it's easier to complete this set of changes
by temporarily switching to using basicnode.

Tests using 'NewUnencodableSelectorSpec' were discarded -- this more or
less exercised what would happen if an uninitialized zero value got
into a program, and this is now happily statically impossible.

Some tests are not passing; have not sussed yet.

* fix(deps): upgrade ipld-prime with bug fix

use ipld prime with fixed marshalling bug to get tests passing

* fix(lint): fix lint error
Co-authored-by: Eric Myhre <hash@exultant.us>

* feat(graphsync): add update to message

add update to message protobuf. also fix deserialization error with cancel requests

* feat(requestmanager): support updates for response hooks

Support sending request updates in incoming response hooks. Also factor hooks out of request manager

* refactor(hooks): make hooks approach consistent

for ease of use and learning in library, make hooks structure consistent across requestor and
responder

* feat(responsemanager): update request processing

Add the ability to processing update requests as needed

* test(responsemanager): add more tests for update behavior

* test(integration):add update integration test

Add test to verify the whole update flow

* fix(lint): fix lint error

* feat(deps): upgrade IPLD prime to last before refactor

* refactor(responsemanager): refactor traversal

switch to iterative traversal that is more amenable to interruption

* feat(peerresponsemanager): add bytes sent to peer response sender

Add a return value to SendResponse in peer response sender to return the number of block bytes
transmitted. Note: does not count unsent blocks. Also convert block size measurements to uint64

* feat(responsemanager): add response pausing

add first implementation of response pausing in response manager

* refactor(responsemanager): extract classes to simplify response manager

extract hooks as independant functionality

* test(responsemanager): add tests for block hooks and request hooks

* test(graphsync): add pause/resume integration test

* fix(responsemanager): fix directory misname

requesthooks.go directory -> requesthooks

* fix(tests): cleanup races & lint

* test(graphsync): fix flaky test

* feat(graphsync): define hook interfaces, test infrastructure

rename hooks, add outgoing request hook, define chain types for node builder chooser tests

* feat(graphsync): define persistence option

add persistence options to asyncloader & responsemanager + handling

* feat(graphsync): complete persistence / node builder chooser roundtrip

* fix(lint): fix lint errors

* feat(ipldutil): add NodeBuiderChooser to traverse

Add the option to set NodeBuilderChooser on traverse, using default when nil

* feat(responsemanager): set loader and chooser in hooks

Add the ability to set a custom loader and custom node builder chooser in a response hook

* refactor(hooks): refactor default validation as hook, add unregister option

* feat(graphsync): add disable default validation option

* fix(responsemanager): fix mutex  unlocking

cover case where unlocking was not happening

* refactor(tests): introduce testify

Introduce testify, cleanup lots of tests, add channel assertions to simplify channel tests, make
more readable

* refactor(tests): clean up error messages

* fix(deps): remove bitswap

remove accidentally added bitswap

* refactor(tests): put expected in correct position

Make sure for require.equal that the expected value is the first value