GitLab

Instead of retrying in get only, retry everywhere we read a file.

When multiple write operations conflict, we:

1. Try them in-order till one succeeds.
2. After the fact, re-order them such that the pending operations "happen" after
   the one that succeeds.
3. Return "success" for all the pending write operations for that key. This is
   acceptable because we're claiming that the operation that _actually_ succeeded
   happened "last" so it would have clobbered the other operations.

However, in the case of put, we still need to remove the temporary file that we
didn't end up moving into place.

add fd-overload retries inline

Otherwise, if we repeatedly stop the same node, we'll collect a bunch of
temporary files and NEVER DELETE them. This will:

1. Waste space.
2. Slow down queries/GC.

This patch:

1. Moves all temporary files to a single `.temp` directory. The leading `.`
means it can't conflict with any keys and queries (even on older flatfs
versions) will skip it.
2. Adds an "rm -rf flatfs-dir/.temp" call on start.

We _expect_ temporary put- files.

It's a no-op if we don't need it.

Only allow keys of the form `/[0-9A-Z+-_=]`. That is, upper-case alphanumeric
keys in the root namespace (plus some special characters).

Why? We don't encode keys before writing them to the filesystem. This change
ensures that:

1. Case sensitivity doesn't matter because we only allow upper-case keys.
2. Path separators and special characters doesn't matter.

For context, go-ipfs only uses flatfs for storing blocks. Every block CID is
encoded as uppercase alphanumeric text (specifically, uppercase base32).

We could be less restrictive, but this is safer and easier to understand.
Unfortunately, we _can't_ allow mixed case (Windows) and can't allow lowercase
because we're already using uppercase keys.

fixes #23

That way, we're finally implementing everything.

While this datastore is still quite limited, we might as well support everything
we easily can.

And better handle prefixes.

Merge before: https://github.com/ipfs/go-datastore/pull/120

1. Use []byte instead of interface{}.
2. Handle rename errors.
3. Correctly temporary files.

(and make Close threadsafe)

Closing a query early could leak the query goroutine if the buffer isn't large
enough to fit the rest of the query.

Also sync contents of diskUsage.cache to disk after the initial calculation
and during shutdown.

This simplifies the code with perhaps a slightly higher overhead during
write operations.

This now means that the datastore needs to be properly closed, to clean up
the background thread.