GitLab

fix typo in readme.go

os.Rename documentation says: "OS-specific restrictions may apply when
oldpath and newpath are in different directories." On Unix, this means
we can't rename across devices. On Plan 9 however, the functionality is
even more limited: cross-directory renames are not allowed at all.

Add a wrapper around os.Rename for Plan 9, which will copy the file if
we're renaming across directory. All tests seems to pass.

(Aside: I also had to write this wrapper to get go-git working on Plan 9:
https://github.com/go-git/go-billy/blob/v5.0.0/osfs/os_plan9.go#L27
but I notice few issues with that one.)

Fixes #86

move retries lower and retry rename ops

Co-Authored-By: Will <will.scott@protocol.ai>

Retry renames when the error wasn't due to a file not existing. This case can
come up on windows when some other process opens the file for reading while
we're trying to rename it.

Instead or retrying the whole operation, retry the specific parts that can fail.

I believe this also fixes cases where we might end up with a retry loop within a
retry loop? I'm not sure.

cleanup putMany implementation

* Use a map for dirsToSync to avoid syncing the same dir multiple times.
* Keep track of files in a slice, and use offsets into the slice to keep track
of which ones have been closed/removed.

Also, record the fact that we've created a temporary file _before_ we try to
write to it, in case the write fails. That way, we'll try to remove it when we
abort.

feat: read harder

Instead of retrying in get only, retry everywhere we read a file.

fix: remove temporary files when multiple write operations conflict

When multiple write operations conflict, we:

1. Try them in-order till one succeeds.
2. After the fact, re-order them such that the pending operations "happen" after
   the one that succeeds.
3. Return "success" for all the pending write operations for that key. This is
   acceptable because we're claiming that the operation that _actually_ succeeded
   happened "last" so it would have clobbered the other operations.

However, in the case of put, we still need to remove the temporary file that we
didn't end up moving into place.

add fd-overload retries inline

Windows CI + Fixes

fix: close query when finished moving

fix: ensure that we close the diskusage file, even if we fail to rename it

Bump github.com/ipfs/go-datastore from 0.4.2 to 0.4.4

Bump github.com/jbenet/goprocess from 0.1.3 to 0.1.4

feat: put all temporary files in the same directory and clean them up

fix: only log when we find a file we don't expect

Otherwise, if we repeatedly stop the same node, we'll collect a bunch of
temporary files and NEVER DELETE them. This will:

1. Waste space.
2. Slow down queries/GC.

This patch:

1. Moves all temporary files to a single `.temp` directory. The leading `.`
means it can't conflict with any keys and queries (even on older flatfs
versions) will skip it.
2. Adds an "rm -rf flatfs-dir/.temp" call on start.

We _expect_ temporary put- files.