GitLab

The last commit did a mv. I intended to do a cp.

Go modules has to support many OSs and filesystems,
so it completely ignores symlinks when creating module zips.
And parts of the Go ecosystem, like pkg.go.dev, consume those zips.

So this symlink actually makes the module technically non-OSS,
since it completely lacks a LICENSE file when downloaded.
pkg.go.dev thus refuses to show its documentation.

One option would be to rename the root LICENSE.md file to just LICENSE,
since cmd/go treats that file in a special way,
including it in subdirectory-module zips.
Unfortunately, we do want the ".md" extension,
since the file is formatted to outline the double license.

As such, simply copy the file. It should be fine as it's mostly static.

Add examples link to the README and refine its structure.

List features of CARv2.

Use stronger language to encourage users to use `v2` over `v0`.

While at it, make v2's license a symlink, so the two stay in sync.
The reason we want the v2 module to have its own license is so that its
module zip includes the file as well.

Besides mentioning v0 and v2, the root README now also links to pkgsite
and lists Masih and myself as default maintainers, given that we're the
ones that did major work on this library last.

Fixes #124.
Fixes #179.

It has unintended side effects, as we leak the inner ReadOnly value as
well as its methods to the parent ReadWrite API.

One unintended consequence is that it was possible to "close" a
ReadWrite, when we only wanted to support finalizing it.

The resumption tests do want to close without finalization,
for the sake of emulating partially-written CARv2 files.
Those can hook into the unexported API via export_test.go.

If a downstream really wants to support closing a ReadWrite blockstore
without finalizing it, two alternatives are outlined in export_test.go.

Implement an iterator over CAR blocks that accepts both v1 and v2. The
reader only requires io.Reader interface to operate and exposes the
version and roots of the underlaying CAR file as fields.

Remove `Next` from CARv2 reader now that the iteration functionality is
provided using `BlockReader`.

Update the benchmarks to reflect the changes and add example usage for
`NewBlockReader`.

Add tests that assert `BlockReader` behaviour in erroneous and
successful cases for both underlying CARv1 and CARv2 payload.

It's technically correct to use panics in these situations,
as the downstream user is clearly breaking the contract documented in
the API's godoc.

However, all these methods return errors already,
so panicking is not our only option here.

There's another reason that makes a panic unfortunate.
ReadWrite.Finalize enables the panic behavior on other methods,
which makes it much easier to run into in production
even when the code and its tests work normally.

Finally, there's some precedent for IO interfaces using errors rather
than panics when they are used after closing.
For example, os.File.Read returns an error if the file is closed.

Note that this change doesn't make panics entirely impossible.
The blockstore could still run into exceptional and unexpected panics,
such as those caused by memory corruption or internal bugs.

BenchmarkReadBlocks uses carv2.OpenReader with its Roots and Next method.
From six runs on a laptop with a i5-8350U and benchstat:

	name          time/op
	ReadBlocks-8    633µs ± 2%

	name          speed
	ReadBlocks-8  824MB/s ± 2%

	name          alloc/op
	ReadBlocks-8   1.32MB ± 0%

	name          allocs/op
	ReadBlocks-8    13.5k ± 0%

OpenReadOnlyV1 uses blockstore.OpenReadOnly with its Get method.
The method is used on all blocks in a shuffled order,
to ensure that the index is working as intended.
The input file lacks an index, so we also generate that.

	name              time/op
	OpenReadOnlyV1-8    899µs ± 1%

	name              speed
	OpenReadOnlyV1-8  534MB/s ± 1%

	name              alloc/op
	OpenReadOnlyV1-8   1.52MB ± 0%

	name              allocs/op
	OpenReadOnlyV1-8    27.2k ± 0%

Both benchmarks use the "sample" v1/v2 CAR files, which weigh about half
a megabyte. It seems like a good size; a significantly larger CAR file
would make each benchmark iteration take tens or hundreds of
milliseconds, making it much slower to obtain many benchmark results,
since we want at least thousands of iterations to avoid noise.

Implement the mechanism to iterate over blocks in CARv1 or CARv2 via the
`carv2.Reader` API.

Implement tests both for the iterator and the rest of the reader API.
Add test files with corrupt sections etc.

Fixes #189

Implement a mechanism that allows the user to hook an error handling
function to the context passed to `AllKeysChan`. The function is then
notified when an error occurs during asynchronous traversal of data
payload.

Add tests that assert the success and failure cases when error handler
is set.

Fixes #177

The CARv2 implementation uses an internal fork of CARv1 due to upstream
dependency issues captured in #104. Propagate the options set in CARv2
APIs for treatment of zero-lenth sections onto internal packages so that
APIs using the internal CARv1 reader behave consistently. Use a `bool`
for setting the option, since it is the only option needed in CARv1.

Update tests to reflect changes.

Add additional tests to internal CARv1 package and ReadOnly blockstore
that assert option is propagated.

Fixes #190

When error is not expected in tests, assert that there is indeed no
error.

Fix a bug in read options where the value of zero-len option is always
set to `true` even if the flag passed is false. This meant if the option
was present it would have always enabled the zero-len as EOF.

Allow the user to specify read options on `ReadOrGenerateIndex`. Pass
the options onto downstream reader/generators.

Generate a CARv1 file with null padding and check it into `testdata`. In
addition check in another such file from ignite just to have two
different samples of this case.

Enhance index generation tests to expect error when zero-len option is
not set, and generate index when it is for files with null padding.

Make RW blockstore test use the fixed test files from `testdata` instead
of generating every time.

Use existing SDK to create temp file in index example.

Avoid writing to files in `testdata` through examples. This is because
when tests that use those file are run in parallel will fail if files
are modified. Instead write to a temporary file in examples, and
whenever opening testdata files in `RW` mode, make a temporary copy.

Thanks to @mvdan for pointing this out.

Fixes #175

Update the tests too, as a few expected whole CIDs.
For now, just make them use the option.

This required index.Index to gain a new method, GetAll, so that when
using whole CIDs, methods like Get can return true if any of the
matching indexed CIDs are an exact whole-CID match, and not just looking
at the first matching indexed CID.

GetAll is akin to an iteration over all matching indexed CIDs, and its
callback returns a boolean to say if the iteration should continue.
This allows stopping as soon as we're done.

We also remove Index.Get, instead replacing it with a helper called
GetFirst, which simply makes simple uses of GetAll a single line.

We remove the non-specced and unused index implementations, too.
They were left in place in case they were useful again,
but they haven't been so far, and their code is still in git.
Keeping them around just means updating more code when refactoring.

While at it, make ZeroLengthSectionAsEOF take a boolean and return an
option, just like the other boolean options, for consistency.

Fixes #130.

We've been working on CARv2 in the wip-v2 branch for some weeks.

In preparation for the first v2 beta, merge all of those commits into master.

The commits are mostly as they were originally, minus some minor squashing to clean up the history.

Implement examples that show how to open a read-only blockstore, and a
read-write blockstore along with resumption from the same file.

The example surfaced a race condition where if the `AllKeysChan` is
partially consumed and file is closed then the gorutie started by chan
population causes the race condition. Locking on Close will make the
closure hang. Better solution is needed but choices are limited due to
`AllKeysChan` signature.

Fixes:
- https://github.com/ipld/go-car/issues/124

It's two lines of code extra and we already have read from and write to
APIs for index.

Remove `hydrate` cli since the assumptions it makes about appending
index are no longer true. header needs updating for example. Removing to
be re-added when needed as part of a propper CARv2 CLI.

If appendage is needed there is already `car.AttachIndex`.

Rename terminology to match what the spec uses to describe the inner
CARv1 payload, i.e. Data payload.

Update docs to use CARv1 and CARv2 consistently.

Fix typo in Options API.

See:
- https://ipld.io/specs/transport/car/carv2/

Remove the memory intensive CARv2 writer implementation since it needs
to be reimplemented anyway and not used. We will release writers if
needed when it comes to SelectiveCar writer implementation. Just now it
is not clear if we want a CARv2 writer that works with the deprecated
`format.NodeGetter`.

Add tests for V1 wrapping functionality.

Reformat code in repo for consistency using `gofumpt`.

Add examples that show how to read and write an index to/from file.

Test marshalling and unmarshalling index files.

Run `gofumt` on repo.

We've agreed to have unified options, since many will be shared between
the root and blockstore packages.

Include docs, and update the tests.

Improve reader type conversion by checking if type satisfies
ReaderAt to avoid unnecessary wrapping.

Move io converters into one place.

Fixes:
- https://github.com/ipld/go-car/issues/145

Fixes #169.

Remove the duplicate test CARv1 file that existed both in `testdata` and
`blockstore/testdata` in favour of the one in root.

Reflect change in tests.

Duplicity checked using `md5` digest:

```sh
$ md5 testdata/sample-v1.car
MD5 (testdata/sample-v1.car) = 14fcffc271e50bc56cfce744d462a9bd

$ md5 blockstore/testdata/test.car
MD5 (blockstore/testdata/test.car) = 14fcffc271e50bc56cfce744d462a9bd
```

Fix an issue where single width index not finding a given key returns
`0` offset instead of `index.ErrNotFound`.

Reflect the changes in blockstore to return appropriate IPFS blockstore
error.

Add tests that asserts error types both in index and blockstore
packages.

Remove redundant TODOs.

Relates to:
- https://github.com/ipld/go-car/pull/158

Use both CID v0 and v1 in testing `ReadWrite` blockstore.

Use consistent import package name `merkledag` across tests.

Add tests that asserts:
 - when padding options are set the payload is as
expected
 - when finalized, blockstore calls panic
 - when resumed from mismatching padding error is as expected
 - when resumed from non-v2 file error is as expected

Remove redundant TODOs in code

All the "New" APIs take IO interfaces, so they don't open any file by
themselves. However, the APIs that take a path to disk and return a
blockstore or a reader need closing, so the prefix "Open" helps clarify
that.

Plus, it makes names more consistent.

Fixes #91.

Unexport the CARv2 writer API until it is reimplemented using WriterAt
or WriteSeeker to be more efficient. This API is not used anyway and we
can postpone releasing it until SelectiveCar writer API is figured out.
For now we unexport it to carry on with interface finalization and alpha
release.

Implement the ability to generate index from a CARv1 payload given only
an `io.Reader`, where the previous implementation required
`io.ReadSeeker`. The rationale is to be minimal in what we expect in the
API, since index generation from a CARv1 payload never need to rewind
the reader and only moves forward in the stream.

Refactor utility IO functions that convert between types in one place.
Implement constructor functions that only instantiate wrappers when the
passed argument does not satisfy a required interface.

Fixes:
- https://github.com/ipld/go-car/issues/146

Relates to:
- https://github.com/ipld/go-car/issues/145

Remove dependency to `bufio.Reader` in internal `carv1` package that
seems to be mainly used for peeking a byte to return appropriate error
when stream abruptly ends, relating to #36. This allows simplification
of code across the repo and remove all unnecessary wrappings of
`io.Reader` with `bufio.Reader`. This will also aid simplify the
internal IO utilities which will be done in future PRs. For now we
simply remove dependency to `bufio.Reader`

See:
- https://github.com/ipld/go-car/pull/36

Support resumption from a complete CARv2 file, effectively implementing
append block to a CARv2 file. Note, on resumption the entire file is
re-indexed. So it is undesirable to do this for complete cars.

Fix a bug where `AllKeysChan` failed on files that contained the index
and resumed from. Fix is done by truncating the CARv2 file, removing the
index if it is present when blockstore is constructed when resuming.
This is because if `AllKeysChan` is called on a file that contains the
index, the ReadOnly backing will continue to read until EOF which will
result in error. Since file is re-indexed on resumption and we require
finalize to be called we truncate the file to remove index.

Write header on every put so that resuming from finalized files that put
more blocks without then finalizing works as expected. Otherwise,
because the last put did not finalize and header is not updated any
blocks put will effectively be lost.

Add tests that assert resume with and without finalization. Add tests
that assert read operations work on resumed read-write blockstore.

Fixes: https://github.com/ipld/go-car/issues/155

Address review comments

Avoid having to write car v2 header on every put by unfinalizing the
blockstore first if resumed from a finalized file.

Fix random seed for tests.

Move index generation functionality to root, because they read/write CAR
files. This leaves the `index` package to only contain the index
implementation itself.

Move `index.Attach` to root while at it following the same logic. Add
TODOs to the implementation of Attach to make it more robust. As is it
_could_ result in corrupt v2 files if offset includes padding since v2
header is not updated in this function.

Add tests to generate index and move existing tests to correspond to the
go file containing generation functionality.