datastore.go 7.84 KB
Newer Older
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
1
package datastore
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
2 3

import (
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
4
	"errors"
5
	"io"
6
	"time"
7

Jeromy's avatar
Jeromy committed
8
	query "github.com/ipfs/go-datastore/query"
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
9 10 11
)

/*
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
12
Datastore represents storage for any key-value pair.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Datastores are general enough to be backed by all kinds of different storage:
in-memory caches, databases, a remote datastore, flat files on disk, etc.

The general idea is to wrap a more complicated storage facility in a simple,
uniform interface, keeping the freedom of using the right tools for the job.
In particular, a Datastore can aggregate other datastores in interesting ways,
like sharded (to distribute load) or tiered access (caches before databases).

While Datastores should be written general enough to accept all sorts of
values, some implementations will undoubtedly have to be specific (e.g. SQL
databases where fields should be decomposed into columns), particularly to
support queries efficiently. Moreover, certain datastores may enforce certain
types of values (e.g. requiring an io.Reader, a specific struct, etc) or
serialization formats (JSON, Protobufs, etc).

IMPORTANT: No Datastore should ever Panic! This is a cross-module interface,
and thus it should behave predictably and handle exceptional conditions with
proper error reporting. Thus, all Datastore calls may return errors, which
should be checked by callers.
*/
type Datastore interface {
35 36
	Read
	Write
37
	io.Closer
38 39 40 41
}

// Write is the write-side of the Datastore interface.
type Write interface {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
42 43 44 45 46 47 48 49 50
	// Put stores the object `value` named by `key`.
	//
	// The generalized Datastore interface does not impose a value type,
	// allowing various datastore middleware implementations (which do not
	// handle the values directly) to be composed together.
	//
	// Ultimately, the lowest-level datastore will need to do some value checking
	// or risk getting incorrect values. It may also be useful to expose a more
	// type-safe interface to your application, and do the checking up-front.
51
	Put(key Key, value []byte) error
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
52

53 54 55 56 57 58
	// Delete removes the value for given `key`.
	Delete(key Key) error
}

// Read is the read-side of the Datastore interface.
type Read interface {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
59 60
	// Get retrieves the object `value` named by `key`.
	// Get will return ErrNotFound if the key is not mapped to a value.
61
	Get(key Key) (value []byte, err error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
62

Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
63 64 65 66 67
	// Has returns whether the `key` is mapped to a `value`.
	// In some contexts, it may be much cheaper only to check for existence of
	// a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
	// The default implementation is found in `GetBackedHas`.
	Has(key Key) (exists bool, err error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
68

Steven Allen's avatar
Steven Allen committed
69 70 71 72 73
	// GetSize returns the size of the `value` named by `key`.
	// In some contexts, it may be much cheaper to only get the size of the
	// value rather than retrieving the value itself.
	GetSize(key Key) (size int, err error)

74 75 76 77 78 79
	// Query searches the datastore and returns a query result. This function
	// may return before the query actually runs. To wait for the query:
	//
	//   result, _ := ds.Query(q)
	//
	//   // use the channel interface; result may come in at different times
Vasily Kolobkov's avatar
Vasily Kolobkov committed
80
	//   for entry := range result.Next() { ... }
81
	//
Vasily Kolobkov's avatar
Vasily Kolobkov committed
82 83 84
	//   // or wait for the query to be completely done
	//   entries, _ := result.Rest()
	//   for entry := range entries { ... }
85
	//
86
	Query(q query.Query) (query.Results, error)
Jeromy's avatar
Jeromy committed
87 88
}

89 90 91 92 93 94
// Batching datastores support deferred, grouped updates to the database.
// `Batch`es do NOT have transactional semantics: updates to the underlying
// datastore are not guaranteed to occur in the same iota of time. Similarly,
// batched updates will not be flushed to the underlying datastore until
// `Commit` has been called. `Txn`s from a `TxnDatastore` have all the
// capabilities of a `Batch`, but the reverse is NOT true.
Jeromy's avatar
Jeromy committed
95
type Batching interface {
Jeromy's avatar
Jeromy committed
96
	Datastore
Jeromy's avatar
Jeromy committed
97

Jeromy's avatar
Jeromy committed
98
	Batch() (Batch, error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
99 100
}

101 102
// ErrBatchUnsupported is returned if the by Batch if the Datastore doesn't
// actually support batching.
Jeromy's avatar
Jeromy committed
103 104
var ErrBatchUnsupported = errors.New("this datastore does not support batching")

Juan Batiz-Benet's avatar
linting  
Juan Batiz-Benet committed
105 106
// ThreadSafeDatastore is an interface that all threadsafe datastore should
// implement to leverage type safety checks.
107 108
type ThreadSafeDatastore interface {
	Datastore
109

110 111 112
	IsThreadSafe()
}

113 114 115 116 117 118 119 120
// CheckedDatastore is an interface that should be implemented by datastores
// which may need checking on-disk data integrity.
type CheckedDatastore interface {
	Datastore

	Check() error
}

Łukasz Magiera's avatar
Łukasz Magiera committed
121 122
// CheckedDatastore is an interface that should be implemented by datastores
// which want to provide a mechanism to check data integrity and/or
123
// error correction.
Łukasz Magiera's avatar
Łukasz Magiera committed
124 125 126 127 128 129
type ScrubbedDatastore interface {
	Datastore

	Scrub() error
}

130
// GCDatastore is an interface that should be implemented by datastores which
131
// don't free disk space by just removing data from them.
132 133 134 135 136 137
type GCDatastore interface {
	Datastore

	CollectGarbage() error
}

138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157
// PersistentDatastore is an interface that should be implemented by datastores
// which can report disk usage.
type PersistentDatastore interface {
	Datastore

	// DiskUsage returns the space used by a datastore, in bytes.
	DiskUsage() (uint64, error)
}

// DiskUsage checks if a Datastore is a
// PersistentDatastore and returns its DiskUsage(),
// otherwise returns 0.
func DiskUsage(d Datastore) (uint64, error) {
	persDs, ok := d.(PersistentDatastore)
	if !ok {
		return 0, nil
	}
	return persDs.DiskUsage()
}

158 159 160 161 162
// TTLDatastore is an interface that should be implemented by datastores that
// support expiring entries.
type TTLDatastore interface {
	Datastore

163
	PutWithTTL(key Key, value []byte, ttl time.Duration) error
164
	SetTTL(key Key, ttl time.Duration) error
165
	GetExpiration(key Key) (time.Time, error)
166 167
}

168 169 170 171 172
// Txn extends the Datastore type. Txns allow users to batch queries and
// mutations to the Datastore into atomic groups, or transactions. Actions
// performed on a transaction will not take hold until a successful call to
// Commit has been made. Likewise, transactions can be aborted by calling
// Discard before a successful Commit has been made.
173
type Txn interface {
174 175
	Read
	Write
176

177 178 179
	// Commit finalizes a transaction, attempting to commit it to the Datastore.
	// May return an error if the transaction has gone stale. The presence of an
	// error is an indication that the data was not committed to the Datastore.
180
	Commit() error
181 182 183 184 185
	// Discard throws away changes recorded in a transaction without committing
	// them to the underlying Datastore. Any calls made to Discard after Commit
	// has been successfully called will have no effect on the transaction and
	// state of the Datastore, making it safe to defer.
	Discard()
186 187
}

188
// TxnDatastore is an interface that should be implemented by datastores that
189
// support transactions.
190
type TxnDatastore interface {
191 192
	Datastore

193
	NewTransaction(readOnly bool) (Txn, error)
194 195
}

Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
196
// Errors
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
197 198 199

// ErrNotFound is returned by Get, Has, and Delete when a datastore does not
// map the given key to a value.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
200
var ErrNotFound = errors.New("datastore: key not found")
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
201 202 203 204

// ErrInvalidType is returned by Put when a given value is incopatible with
// the type the datastore supports. This means a conversion (or serialization)
// is needed beforehand.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
205
var ErrInvalidType = errors.New("datastore: invalid type error")
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
206

Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
207 208 209 210 211 212
// GetBackedHas provides a default Datastore.Has implementation.
// It exists so Datastore.Has implementations can use it, like so:
//
// func (*d SomeDatastore) Has(key Key) (exists bool, err error) {
//   return GetBackedHas(d, key)
// }
213
func GetBackedHas(ds Read, key Key) (bool, error) {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
214 215 216 217 218 219 220 221 222
	_, err := ds.Get(key)
	switch err {
	case nil:
		return true, nil
	case ErrNotFound:
		return false, nil
	default:
		return false, err
	}
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
223
}
Jeromy's avatar
Jeromy committed
224

Steven Allen's avatar
Steven Allen committed
225 226 227 228 229 230
// GetBackedSize provides a default Datastore.GetSize implementation.
// It exists so Datastore.GetSize implementations can use it, like so:
//
// func (*d SomeDatastore) GetSize(key Key) (size int, err error) {
//   return GetBackedSize(d, key)
// }
231
func GetBackedSize(ds Read, key Key) (int, error) {
Steven Allen's avatar
Steven Allen committed
232 233 234 235 236 237 238
	value, err := ds.Get(key)
	if err == nil {
		return len(value), nil
	}
	return -1, err
}

Jeromy's avatar
Jeromy committed
239
type Batch interface {
240
	Write
Jeromy's avatar
Jeromy committed
241 242 243

	Commit() error
}