GitLab

Instead or retrying the whole operation, retry the specific parts that can fail.

I believe this also fixes cases where we might end up with a retry loop within a
retry loop? I'm not sure.

cleanup putMany implementation

* Use a map for dirsToSync to avoid syncing the same dir multiple times.
* Keep track of files in a slice, and use offsets into the slice to keep track
of which ones have been closed/removed.

Also, record the fact that we've created a temporary file _before_ we try to
write to it, in case the write fails. That way, we'll try to remove it when we
abort.

feat: read harder

Instead of retrying in get only, retry everywhere we read a file.

fix: remove temporary files when multiple write operations conflict

When multiple write operations conflict, we:

1. Try them in-order till one succeeds.
2. After the fact, re-order them such that the pending operations "happen" after
   the one that succeeds.
3. Return "success" for all the pending write operations for that key. This is
   acceptable because we're claiming that the operation that _actually_ succeeded
   happened "last" so it would have clobbered the other operations.

However, in the case of put, we still need to remove the temporary file that we
didn't end up moving into place.

add fd-overload retries inline

Windows CI + Fixes

fix: close query when finished moving

fix: ensure that we close the diskusage file, even if we fail to rename it

Bump github.com/ipfs/go-datastore from 0.4.2 to 0.4.4

Bump github.com/jbenet/goprocess from 0.1.3 to 0.1.4

feat: put all temporary files in the same directory and clean them up

fix: only log when we find a file we don't expect

Otherwise, if we repeatedly stop the same node, we'll collect a bunch of
temporary files and NEVER DELETE them. This will:

1. Waste space.
2. Slow down queries/GC.

This patch:

1. Moves all temporary files to a single `.temp` directory. The leading `.`
means it can't conflict with any keys and queries (even on older flatfs
versions) will skip it.
2. Adds an "rm -rf flatfs-dir/.temp" call on start.

We _expect_ temporary put- files.

Bumps [github.com/ipfs/go-datastore](https://github.com/ipfs/go-datastore) from 0.4.2 to 0.4.4.
- [Release notes](https://github.com/ipfs/go-datastore/releases)
- [Commits](https://github.com/ipfs/go-datastore/compare/v0.4.2...v0.4.4)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Bumps [github.com/jbenet/goprocess](https://github.com/jbenet/goprocess) from 0.1.3 to 0.1.4.
- [Release notes](https://github.com/jbenet/goprocess/releases)
- [Commits](https://github.com/jbenet/goprocess/compare/v0.1.3...v0.1.4)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Bump github.com/ipfs/go-log from 1.0.2 to 1.0.3

Bumps [github.com/ipfs/go-log](https://github.com/ipfs/go-log) from 1.0.2 to 1.0.3.
- [Release notes](https://github.com/ipfs/go-log/releases)
- [Commits](https://github.com/ipfs/go-log/compare/v1.0.2...v1.0.3)
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Make flatfs robust

It's a no-op if we don't need it.

Only allow keys of the form `/[0-9A-Z+-_=]`. That is, upper-case alphanumeric
keys in the root namespace (plus some special characters).

Why? We don't encode keys before writing them to the filesystem. This change
ensures that:

1. Case sensitivity doesn't matter because we only allow upper-case keys.
2. Path separators and special characters doesn't matter.

For context, go-ipfs only uses flatfs for storing blocks. Every block CID is
encoded as uppercase alphanumeric text (specifically, uppercase base32).

We could be less restrictive, but this is safer and easier to understand.
Unfortunately, we _can't_ allow mixed case (Windows) and can't allow lowercase
because we're already using uppercase keys.

fixes #23

That way, we're finally implementing everything.