datastore.go 7.83 KB
Newer Older
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
1
package datastore
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
2 3

import (
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
4
	"errors"
5
	"time"
6

Jeromy's avatar
Jeromy committed
7
	query "github.com/ipfs/go-datastore/query"
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
8 9 10
)

/*
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
11
Datastore represents storage for any key-value pair.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

Datastores are general enough to be backed by all kinds of different storage:
in-memory caches, databases, a remote datastore, flat files on disk, etc.

The general idea is to wrap a more complicated storage facility in a simple,
uniform interface, keeping the freedom of using the right tools for the job.
In particular, a Datastore can aggregate other datastores in interesting ways,
like sharded (to distribute load) or tiered access (caches before databases).

While Datastores should be written general enough to accept all sorts of
values, some implementations will undoubtedly have to be specific (e.g. SQL
databases where fields should be decomposed into columns), particularly to
support queries efficiently. Moreover, certain datastores may enforce certain
types of values (e.g. requiring an io.Reader, a specific struct, etc) or
serialization formats (JSON, Protobufs, etc).

IMPORTANT: No Datastore should ever Panic! This is a cross-module interface,
and thus it should behave predictably and handle exceptional conditions with
proper error reporting. Thus, all Datastore calls may return errors, which
should be checked by callers.
*/
type Datastore interface {
34 35 36 37 38 39
	Read
	Write
}

// Write is the write-side of the Datastore interface.
type Write interface {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
40 41 42 43 44 45 46 47 48
	// Put stores the object `value` named by `key`.
	//
	// The generalized Datastore interface does not impose a value type,
	// allowing various datastore middleware implementations (which do not
	// handle the values directly) to be composed together.
	//
	// Ultimately, the lowest-level datastore will need to do some value checking
	// or risk getting incorrect values. It may also be useful to expose a more
	// type-safe interface to your application, and do the checking up-front.
49
	Put(key Key, value []byte) error
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
50

51 52 53 54 55 56
	// Delete removes the value for given `key`.
	Delete(key Key) error
}

// Read is the read-side of the Datastore interface.
type Read interface {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
57 58
	// Get retrieves the object `value` named by `key`.
	// Get will return ErrNotFound if the key is not mapped to a value.
59
	Get(key Key) (value []byte, err error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
60

Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
61 62 63 64 65
	// Has returns whether the `key` is mapped to a `value`.
	// In some contexts, it may be much cheaper only to check for existence of
	// a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
	// The default implementation is found in `GetBackedHas`.
	Has(key Key) (exists bool, err error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
66

Steven Allen's avatar
Steven Allen committed
67 68 69 70 71
	// GetSize returns the size of the `value` named by `key`.
	// In some contexts, it may be much cheaper to only get the size of the
	// value rather than retrieving the value itself.
	GetSize(key Key) (size int, err error)

72 73 74 75 76 77
	// Query searches the datastore and returns a query result. This function
	// may return before the query actually runs. To wait for the query:
	//
	//   result, _ := ds.Query(q)
	//
	//   // use the channel interface; result may come in at different times
Vasily Kolobkov's avatar
Vasily Kolobkov committed
78
	//   for entry := range result.Next() { ... }
79
	//
Vasily Kolobkov's avatar
Vasily Kolobkov committed
80 81 82
	//   // or wait for the query to be completely done
	//   entries, _ := result.Rest()
	//   for entry := range entries { ... }
83
	//
84
	Query(q query.Query) (query.Results, error)
Jeromy's avatar
Jeromy committed
85 86
}

87 88 89 90 91 92
// Batching datastores support deferred, grouped updates to the database.
// `Batch`es do NOT have transactional semantics: updates to the underlying
// datastore are not guaranteed to occur in the same iota of time. Similarly,
// batched updates will not be flushed to the underlying datastore until
// `Commit` has been called. `Txn`s from a `TxnDatastore` have all the
// capabilities of a `Batch`, but the reverse is NOT true.
Jeromy's avatar
Jeromy committed
93
type Batching interface {
Jeromy's avatar
Jeromy committed
94
	Datastore
Jeromy's avatar
Jeromy committed
95

Jeromy's avatar
Jeromy committed
96
	Batch() (Batch, error)
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
97 98
}

99 100
// ErrBatchUnsupported is returned if the by Batch if the Datastore doesn't
// actually support batching.
Jeromy's avatar
Jeromy committed
101 102
var ErrBatchUnsupported = errors.New("this datastore does not support batching")

Juan Batiz-Benet's avatar
linting  
Juan Batiz-Benet committed
103 104
// ThreadSafeDatastore is an interface that all threadsafe datastore should
// implement to leverage type safety checks.
105 106
type ThreadSafeDatastore interface {
	Datastore
107

108 109 110
	IsThreadSafe()
}

111 112 113 114 115 116 117 118
// CheckedDatastore is an interface that should be implemented by datastores
// which may need checking on-disk data integrity.
type CheckedDatastore interface {
	Datastore

	Check() error
}

Łukasz Magiera's avatar
Łukasz Magiera committed
119 120
// CheckedDatastore is an interface that should be implemented by datastores
// which want to provide a mechanism to check data integrity and/or
121
// error correction.
Łukasz Magiera's avatar
Łukasz Magiera committed
122 123 124 125 126 127
type ScrubbedDatastore interface {
	Datastore

	Scrub() error
}

128
// GCDatastore is an interface that should be implemented by datastores which
129
// don't free disk space by just removing data from them.
130 131 132 133 134 135
type GCDatastore interface {
	Datastore

	CollectGarbage() error
}

136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155
// PersistentDatastore is an interface that should be implemented by datastores
// which can report disk usage.
type PersistentDatastore interface {
	Datastore

	// DiskUsage returns the space used by a datastore, in bytes.
	DiskUsage() (uint64, error)
}

// DiskUsage checks if a Datastore is a
// PersistentDatastore and returns its DiskUsage(),
// otherwise returns 0.
func DiskUsage(d Datastore) (uint64, error) {
	persDs, ok := d.(PersistentDatastore)
	if !ok {
		return 0, nil
	}
	return persDs.DiskUsage()
}

156 157 158 159 160
// TTLDatastore is an interface that should be implemented by datastores that
// support expiring entries.
type TTLDatastore interface {
	Datastore

161
	PutWithTTL(key Key, value []byte, ttl time.Duration) error
162
	SetTTL(key Key, ttl time.Duration) error
163
	GetExpiration(key Key) (time.Time, error)
164 165
}

166 167 168 169 170
// Txn extends the Datastore type. Txns allow users to batch queries and
// mutations to the Datastore into atomic groups, or transactions. Actions
// performed on a transaction will not take hold until a successful call to
// Commit has been made. Likewise, transactions can be aborted by calling
// Discard before a successful Commit has been made.
171
type Txn interface {
172
	Datastore
173

174 175 176
	// Commit finalizes a transaction, attempting to commit it to the Datastore.
	// May return an error if the transaction has gone stale. The presence of an
	// error is an indication that the data was not committed to the Datastore.
177
	Commit() error
178 179 180 181 182
	// Discard throws away changes recorded in a transaction without committing
	// them to the underlying Datastore. Any calls made to Discard after Commit
	// has been successfully called will have no effect on the transaction and
	// state of the Datastore, making it safe to defer.
	Discard()
183 184
}

185
// TxnDatastore is an interface that should be implemented by datastores that
186
// support transactions.
187
type TxnDatastore interface {
188 189
	Datastore

190
	NewTransaction(readOnly bool) (Txn, error)
191 192
}

Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
193
// Errors
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
194 195 196

// ErrNotFound is returned by Get, Has, and Delete when a datastore does not
// map the given key to a value.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
197
var ErrNotFound = errors.New("datastore: key not found")
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
198 199 200 201

// ErrInvalidType is returned by Put when a given value is incopatible with
// the type the datastore supports. This means a conversion (or serialization)
// is needed beforehand.
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
202
var ErrInvalidType = errors.New("datastore: invalid type error")
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
203

Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
204 205 206 207 208 209 210
// GetBackedHas provides a default Datastore.Has implementation.
// It exists so Datastore.Has implementations can use it, like so:
//
// func (*d SomeDatastore) Has(key Key) (exists bool, err error) {
//   return GetBackedHas(d, key)
// }
func GetBackedHas(ds Datastore, key Key) (bool, error) {
Juan Batiz-Benet's avatar
go fmt  
Juan Batiz-Benet committed
211 212 213 214 215 216 217 218 219
	_, err := ds.Get(key)
	switch err {
	case nil:
		return true, nil
	case ErrNotFound:
		return false, nil
	default:
		return false, err
	}
Juan Batiz-Benet's avatar
Juan Batiz-Benet committed
220
}
Jeromy's avatar
Jeromy committed
221

Steven Allen's avatar
Steven Allen committed
222 223 224 225 226 227 228 229 230 231 232 233 234 235
// GetBackedSize provides a default Datastore.GetSize implementation.
// It exists so Datastore.GetSize implementations can use it, like so:
//
// func (*d SomeDatastore) GetSize(key Key) (size int, err error) {
//   return GetBackedSize(d, key)
// }
func GetBackedSize(ds Datastore, key Key) (int, error) {
	value, err := ds.Get(key)
	if err == nil {
		return len(value), nil
	}
	return -1, err
}

Jeromy's avatar
Jeromy committed
236
type Batch interface {
237
	Write
Jeromy's avatar
Jeromy committed
238 239 240

	Commit() error
}