Commits · 8f2d8204121a99344711636f7d9590b2d75acd2b · dms3 / go-dms3

12 Jan, 2016 40 commits

Tommi Virtanen authored May 20, 2015

To test it, set up an S3 bucket (in an AWS region that is not US
Standard, for read-after-write consistency), run `ipfs init`, then
edit `~/.ipfs/config` to say

      "Datastore": {
        "Type": "s3",
        "Region": "us-west-1",
        "Bucket": "mahbukkit",
        "ACL": "private"
      },

with the right values. Set `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` in the environment and you should be able to
run `ipfs add` and `ipfs cat` and see the bucket be populated.

No automated tests exist, unfortunately. S3 is thorny to simulate.

License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>

8f2d8204

thirdparty/s3-datastore: Let caller set ACL, change default to safer "private" · 1174aab8
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
1174aab8
thirdparty/s3-datastore: Datastore keys can be binary, hex encode them for S3 · 33d41285
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
33d41285
Implement pluggable Datastore types, with nothing implemented yet · 9497e26f
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
9497e26f
fsrepo: Refactor to extract datastore internals · b76581d6
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
b76581d6

fsrepo: Detect uninitialized repo by missing config file · dc737440

Tommi Virtanen authored May 19, 2015

Earlier, it also checked checked the leveldb directory. That part
added no crash safety to the application, and just hardcoded
assumptions about the datastore.

If anything, this should rely on the absolute last item created by
fsrepo.Init, and there should be fsync guarantees about ordering.

License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>

dc737440

Remove Config file section "Datastore", it's not used · 6996ce67

Tommi Virtanen authored May 19, 2015

This gives us a clean slate for the new code, avoiding leftovers.

License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>

6996ce67

fsrepo/serialize tests: Stop assuming internals of Config · 2c4a9ab7
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
2c4a9ab7
sharness: Stop assuming leveldb Datastore · d8364462
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
d8364462
core tests: Stop assuming internals of Config · 4a336834
Tommi Virtanen authored May 20, 2015
```
License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>
```
4a336834

gofmt generated assets · a5f98329

Tommi Virtanen authored Jul 15, 2015

The generated file went through some changes because of differing
go-bindata versions.

License: MIT
Signed-off-by: Tommi Virtanen <tv@eagain.net>

a5f98329

update multistream naming of lazyconn · e431f35a
Jeromy authored Jul 10, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
e431f35a
allow multistream to have zero rtt stream opening · 2c4eb609
Jeromy authored Jul 09, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
2c4eb609
dont use searchset for indirect pin checking · 859de514
Jeromy authored Jul 10, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
859de514
pin rm fails appropriately for indirect pins · 879cfeee
Jeromy authored Jul 10, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
879cfeee
addressing comments from CR · 16c4d8cd
Jeromy authored Jul 10, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
16c4d8cd
break up GC logic · 5123857c
Jeromy authored Jul 09, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
5123857c

implement mark and sweep GC · b12ee40a

Jeromy authored Jun 23, 2015

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

dont GC blocks used by pinner

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

comment GC algo

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

add lock to blockstore to prevent GC from eating wanted blocks

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

improve FetchGraph

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

separate interfaces for blockstore and GCBlockstore

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

reintroduce indirect pinning, add enumerateChildren dag method

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

b12ee40a

renamed {R,}Lock -> {Pin,GC}Lock · 27f34b43
Juan Batiz-Benet authored Jul 09, 2015
```
License: MIT
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
```
27f34b43
lock blockstore for pin add · 7b675e87
Jeromy authored Jul 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
7b675e87
move locking out of GC branch · 1a03d5e3
Jeromy authored Jul 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
1a03d5e3
address concerns from PR · dc3b9ed1
Jeromy authored Jul 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
dc3b9ed1

merkledag FetchGraph and EnumerateChildren · b9e5cfaa

Jeromy authored Jul 07, 2015

This commit improves (fixes) the FetchGraph call for recursively
fetching every descendant node of a given merkledag node. This operation
should be the simplest way of ensuring that you have replicated a dag
locally.

This commit also implements a method in the merkledag package called
EnumerateChildren, this method is used to get a set of the keys of every
descendant node of the given node. All keys found are noted in the
passed in KeySet, which may in the future be implemented on disk to
avoid excessive memory consumption.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

b9e5cfaa

Add locking interface to blockstore · da0d48e6

Jeromy authored Jul 07, 2015

The addition of a locking interface to the blockstore allows us to
perform atomic operations on the underlying datastore without having to
worry about different operations happening in the background, such as
garbage collection.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

da0d48e6

using multistream muxer · de50b215

Jeromy authored Jun 17, 2015

* ID service stream
* make the relay service use msmux
* fix nc tests

Note from jbenet: Maybe we should remove the old protocol/muxer
and see what breaks. It shouldn't be used by anything now.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>

de50b215

buffer msgio · 4f34e0ec
Jeromy authored Jun 02, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
4f34e0ec

remove msgio double wrap · c48f456b

Juan Batiz-Benet authored Apr 12, 2015

There was doublewrapping with an unneeded msgio. given that we
use a stream muxer now, msgio is only needed by secureConn -- to
signal the boundaries of an encrypted / mac-ed ciphertext.

Side note: i think including the varint length in the clear is
actually a bad idea that can be exploited by an attacker. it should
be encrypted, too. (TODO)

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

c48f456b

dagmodifier test: Add TODO note about how bad luck can cause test failure · aafebc58
Tommi Virtanen authored Jun 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
aafebc58
dagmodifier: Don't lose pin if old and new key happen to be equal · b24de97b
Tommi Virtanen authored Jun 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
b24de97b
pin: Do not accidentally delete indirect pins on Flush · 7a66a7dc
Tommi Virtanen authored Jun 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
7a66a7dc
bump fsrepo version to 3 · 896601f6
Jeromy authored May 22, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
896601f6
pin: Future-proof against refcount marshaled size changes · 72753e54
Tommi Virtanen authored May 18, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
72753e54

pin: Rewrite to store pins in IPFS objects · e5b8ee48

Tommi Virtanen authored May 11, 2015

WARNING: No migration performed! That needs to come in a separate
commit, perhaps amended into this one.

This is the minimal rewrite, only changing the storage from
JSON(+extra keys) in Datastore to IPFS objects. All of the pinning
state is still loaded in memory, and written from scratch on Flush. To
do more would require API changes, e.g. adding error returns.

Set/Multiset is not cleanly separated into a library, yet, as it's API
is expected to change radically.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

e5b8ee48

sharness: Don't assume we know all things that can create garbage · 531f0579

Tommi Virtanen authored May 11, 2015

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

sharness: Don't assume we know all things that can create garbage

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

531f0579

Typo · fb589a81
Tommi Virtanen authored May 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
fb589a81

Use uint64 for indirect pin refcounts · 96e45c74

Tommi Virtanen authored May 08, 2015

Platform-dependent behavior is not nice, and negative refcounts are
not very useful.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

96e45c74

pin: Remove double bookkeeping of refcount keys · fecfb76c
Tommi Virtanen authored May 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
fecfb76c
pin: Remove dead code · d6a61529
Tommi Virtanen authored May 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
d6a61529

Simplify Pinner interface by folding ManualPinner into Pinner · c9ce2e72

Tommi Virtanen authored May 08, 2015

Pinner had method GetManual that returned a ManualPinner, so every
Pinner had to implement ManualPinner anyway.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

c9ce2e72

blocks/set: Remove now-unused NewDBWrapperSet · c4d2988c
Tommi Virtanen authored May 08, 2015
```
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
```
c4d2988c