With exponential growth in digitized information, innovations in storage
systems have evolved to store an ever growing amount of information with
good performance. While innovations in information retrieval (IR) systems has
enabled indexing and fast retrieval of textual information.
Substantial applications use databases systems to store and manage data
abstracted as structured relational data, or unstructured (document) data.
Retrieval is supported using Structured Query Language (SQL), or other query
languages in the case of NoSql databases. Database scaling and performance
suffers when data growth is unbounded. The data stored by these applications
is not transparent, each application requires embedding a search feature to
provide access to data it collects.
Search engines offer a very fast retrieval technology for the discovery of
web page information on the Internet, and for information stored on a desktop
or within enterprise information systems. Search engines work well with textual
information but don't support the robust data types needed by applications
as supported by databases. Search engine performance also suffers when
data growth is unbounded. Popular web search engines expose a flat view of the
search information space, it is not possible to guide the search request with
context or metadata input.
Furthermore, all of the above discussed information storage and retrieval
systems provide data service guarantees from the storage system perspective.
Every scalable application working with very large data sets is still
burdened with providing its own data guarantees. A data storage abstraction
layer that provides application perspective data service guarantees for all
cooperating applications can provide great benefits to application developers.
DMS3 is a novel solution that combines information storage and retrieval
and offers several advantages to applications developed with the DMS3 API:
With exponential growth in digitized information, innovations in storage systems have evolved to store an ever growing amount of information with good performance. While innovations in information retrieval (IR) systems has enabled indexing and fast retrieval of textual information.
Many modern applications use databases systems to store and manage data abstracted as structured relational data, or unstructured (document) data. Retrieval is supported using Structured Query Language (SQL), or other query languages in the case of NoSql databases. Database scaling and performance suffers when data growth is unbounded. The data stored by these applications is not transparent, each application requires embedding a search feature to provide access to data it collects.
Search engines offer a very fast retrieval technology for the discovery of web page information on the Internet, and for information stored on a desktop or within enterprise information systems. Search engines work well with textual information but don't support the robust data types needed by applications as supported by databases. Search engine performance also suffers when data growth is unbounded. Popular web search engines expose a flat view of the search information space, it is not possible to guide the search request with context or metadata input.
Furthermore, all of the above discussed information storage and retrieval systems provide data service guarantees from the storage system perspective. Every scalable application working with very large data sets is still burdened with providing its own data guarantees. A data storage abstraction layer that provides application perspective data service guarantees for all cooperating applications can provide great benefits to application developers.
DMS3 is a novel solution that combines information storage and retrieval and offers several advantages to applications developed with the DMS3 API:
- Search engine retrieval performance
- Text and application data type support
- Bounded information containers for consistent performance
- Bounded information containers for consistent performance and scaling
- Context sensitive storage and search
- Improved data privacy and access control
- Application data consistency, reliability, safety, and recovery
: This is a working document and captures current design. We reserve the right to change the design at any time as prototype development moves forward.
## Information scale and performance
DMS3 is designed to enable web scale information management (storage and retrieval) and provides applications with an API that abstracts information as documents stored in index repositories managed by a search engine.
Consistent performance is achieved by enforcing configurable upper bound on the number of documents in one index repository. Information scale is achieved by supporting an arbitrary number of index repositories backed by a fault tolerant distributed storage layer.
### Key terms and concepts
Information is organized into index repository **_containers_** [^1] considering multiple criteria:
[^1]:container and repository is used interchangeably in this document to mean the set of files and folders the search engine uses to track volatile and persistent state of a repository.
1. A document **_kind_** suggests a certain structure and semantics context for a class of similar documents. Documents of a certain kind are said to conform to a common schema and contain information of a common theme.
2. An **_infostore_** is a set of containers that collects documents of similar kind. As content is added to a repository and as that repository grows and reaches a configurable document limit, the set is expanded with an additional repository to host additional documents of the same kind. The container-set or **_reposet_** [^2] may expand up to an architectural limit and provides a mechanism to physically shard information. The repository document limit and other life-cycle properties can be configured based on the kind of infospace. An infostore serves to limit search scope of a query target.
[^2]:container-set and reposet is used interchangeably in this document, it represents a generic term that refers to the set of index repositories that form an infostore.
3. A **_metastore_** is a set of index repositories that collects documents providing metadata for corresponding infostores of similar kind. A metastore serves to limit search scope of a query target. Each document in an metastore repository describes the kind of data contained in an associated infostore. A metastore also represents a manifest for hosted information, serves as a testament made by the endpoint user regarding hosted information.
Infostore and metastore life-cycle management is at the purview of a user managing a DMS3 endpoint node, also known as a **_source_** of information.
- A source may create infostores for personal use or to publish and share information with nodes in the peer-to-peer network.
- A metastore is defined by associating (or binding) a locally unique key name with a set of locally hosted infostores of similar kind. Key names are not required be unique in the global DMS3 peer-to-peer network.
- Ideally, the community of sources shares information using commonly accepted key names for aggregating metadata representing similar content semantics into searchable metadata repositories. For example,
- A source that may want to publish and share blogs, uses a metastore under the kind key "blog" to describe the infostores food-blog and product-brand-review-blog, each of which consists of a reposet.
- Food and product review blogs can now be added to the respective infostores.
- The source also creates an metastore under the kind key blog, associated with the food-blog and product-brand-review-blog inforstores. This metastore also consists of a reposet.
- A document is added to the blog metastore to describe the food-blog infostore.
- A separate document is added the blog metastore to describe the product-brand-review-blog.
- A source publishes information by announcing available metastores to the peer-to-peer network.
- Published information is routed through the peer-to-peer network to enable remote search and access.
- Published documents are always on the local endpoint node controlled by the source.
- Routing protocols enable periodic refresh and revocation of published information.
- Published information may represent a broad number of semantic themes, such as:
- tweets, messages, blogs, podcasts, media, news, research, and other kinds of articles
- applications that people can access and run on their client nodes
- data produced by an application that others can search
Any source managing a dms3fs node may decide independently to assume a role as **_curator_** [^3] providing information services on behalf of other peer-to-peer network nodes.
[^3]:curators provide compute and storage hosting resources for published information in return for income.
## Installing the software
DMS3 uses a forked version of the [IPFS](https://ipfs.io/) with modifications to support new capabilities.
In particular dms3fs adds the ability to manage document index repositories and dis-intermediated search.
## Configuring the software
When you initialize dms3fs with the _dms3fs init_ command, dms3fs uses a repository in the local file system.
By default, the repo is located at ~/.dms3-fs. To change the repo location, set the $DMS3FS_PATH environment variable:
```bash
export DMS3FS_PATH=/path/to/dms3fsrepo
```
dms3fs expands the use of local file system by using the _index_ sub-folder within it for managing index repositories.
## Decentralized applications (DApps)
### Base command line interface (CLI)
#### Summary of index management commands
A summary of _index_ repository life-cycle commands are listed below.
- use _config_ to get or set index parameters
- use _mkidx_ to create an index repository
- use _mkdoc_ to generate a document template to be indexed
- use _addoc_ to index a document
- use _rmdoc_ to delete a document added to an index repository
- use _publish_ to share, allowing p2p notes to find, documents in an index
- use _ls_ to display a list of index repositories
- use _show_ to display a document in an index repository
- use _stat_ to display index repository statistics
- use _start_ to run in daemon mode
- use _stop_ to exit daemon mode
- use _restart_ to restart the daemon
- use _recover_ to destructively rebuild a corrupted index repository
When not running in daemon mode, each command affecting an index first performs an integrity check on the index and may detect that the index has been corrupted. Some corruptions may be healed, in other cases the _recover_ command must be used to destroy the corrupted index and rebuild the index from redundant storage. In daemon mode, index integrity checks are performed when the daemon first opens and index repository.
DMS3 uses a forked version of the [Indri Search Engine](https://sourceforge.net/p/lemur/wiki/Home/) with some modifications.
Refer to the following page for [a background on the Indri search engine](https://sourceforge.net/p/lemur/wiki/Language%20Modeling%20and%20Information%20Retrieval%20Background/).
Refer to **_FIXME_** for a discussion of ODM/DES modifications applied to the search engine.
The search engine can service (create or connect to) multiple repositories. Each repository has an active in-memory index where new documents are added to the index dynamically. In-memory indexes are written out to disk for durability, and when the number of indexes grows a background process merges repository indexes consolidating them on disk. All this dynamic index management is performed by the search engine for each of the connected repository contexts.
dms3fs treats each index repository as a container, and manages the growth in information using multiple kinds of reposets to physically chard information in bounded containers.
### Index repository state
Refer to [Indri repository structure](https://sourceforge.net/p/lemur/wiki/Indri%20Repository%20Structure/) for a discussion of the repository on disk structure.
dms3fs uses a path naming convention for each index repository on the local file system within the _index_ sub-folder. In the following example:
```bash
ls ~/.dms3-fs/index/reposet/blog/myblog20/w1543348319-a1-c1-o0/
```
where path components have the following implied semantics
~/.dms3-fs
: initialized local file system dms3fs root
/index
: initialized local file system index repository root
/reposet
: root folder for all resposets on the local file system
/blog
: _kind_ name of the reposet, shared by all repos in the set.
/myblog20
: root folder name for a physical repo instance
/w1543348319-a1-c1-o0
: root folder for a logical grouping repo instance
with name components encoding index metadata:
w
: window start time value, seconds since epoch
a
: area value associated with documents
c
: category value associated with documents
o
: offset value used to encode absolute times
area/category designations provide applications with a logical sharding mechanism.
### Configuring parameters
Refer to [Indri parameters file](https://sourceforge.net/p/lemur/wiki/IndriBuildIndex%20Parameters/) for a discussion of search engine indexer configuration parameters.
### Index metadata and fields
The indexer defines metadata as non-searchable _fields_ and provides forward and reverse document lookup by metadata value. Whereas, fields can be used to qualify query searches.
### Document structure and schema
Fields are used to convey document structure and are used to influence document queries.
### Supported document parsers
The preconfigured file class environment and parsers [indexer file formats](https://sourceforge.net/p/lemur/wiki/Indexer%20File%20Formats/) exhibit the metadata extraction behavior discussed here.
The metadata fields for an index is defined by the parameters supplied
when the index is created. The only metadata required is _docno_, used to
prevent indexing duplicate documents. The metadata field _docno_ also represents the external name or id of the document.
As of this writing, for a document being indexed (using the addFile interface), the parser always adds the following metadata for **html, xml, text, pdf** files. For example:
path
: /home/username/temp/data/sitemap.xml
docno
: /home/username/temp/data/sitemap.xml
For **html, xml, text** documents, the parser additionally adds the metadata. For example:
filetype
: _TEXT_
For **html, pdf** documents, the parser additionally adds the metadata. For example:
url
: /
For **pdf** documents, the parser additionally adds the metadata. For example:
title
: <fromfile,ex:timeoutcdf.eps>
author
: <fromfile,ex:ongardie>
The parser additionally adds the metadata listed within a document, regardless of position in the document tag hierarchy (i.e. whether the metadata fields are wrapped within a <meta> tag or not). the metadata associated with a document is not required to include all the metadata keys specified during index creation.
A file may contain multiple documents wrapped within a <document> tag, in the absence of such a tag, the file represents a single document.
The indexer requires one metadata field for documents: _docno_, the value
of this field must be unique within the index, otherwise the document is
parsed but not indexed.
When an application generated (using the addString interface) document includes a _docno_ field, the index will contain two _docno_ fields. The
first added by the parser, and the second added by the application. The
parser will use the last occurrence as the authoritative _docno_ record.
## Network protocol stack
DMS3 offers two classes of fault tolerant services.
1. Centralized practical byzantine fault tolerant (PBFT) services provide protection for personal private data.
For personal information not publicly shared, DMS3 offers centralized PBFT services for maintaining immutable data integrity. The immutable state for a repository consists of the initial configuration parameters, documents added to the repository corpus, and other repository management state.
2. Decentralized p2p protocol services to provide distribution and access services for shared public data.
For shared public information, this service defrays the hosting and access costs of dedicated compute and storage resources.
The following sub-sections describe these services.
DMS3 allows participants to offer centralized data protection services to protect the users' personal information.
These services protect persisted information by storing dapp consistent redundant copies on multiple endpoints dedicated to one user. The set of endpoints so configured by a user form a user private network that enables access to data from any node within the defined set.
The set of nodes in the private network may be owned by the user or leased from a centralized service provider. Network services encrypt data with the user private key prior to storing on leased equipment.
Nodes and storage scaling capacity of private networks may be restricted to enable reasonable performance expectations. Private networks may be configured to asymmetrically distribute redundant information to compensate for variations of hardware configurations within the defined set of endpoints.
dms3fs implements two PBFS services.
1. Information storage and retrieval, providing index and query services.
2. Network file system (NFS) service, providing backing for node storage.
DMS3 PBFS services automatically recover from a configured number of simultaneous faults.
#### Index fault recovery
The search engine can heal and recover an index from certain faults on its own, for other faults external recovery support using redundant data is required.
There are two options to recover an index repository.
1. Redundant repository state stored in dms3fs. When an index is created, its configuration state is saved external to the search engine. When a document is added to an index, the document is also saved external to the search engine. The _recover_ command can destructively recreate the index from this externally save information. This recovery option does not rely on PBFT.
2. Redundant copies the local file system index repository root. Optional PBFT services automatically keep redundant copies of information in sync. Recovery for simultaneous fault counts exceeding the configured protection level must be handled manually as an exception case.
### Peer to peer (p2p) network services
#### Publishing a repository
Use the _index_ _publish_ command to enable other users on the p2p network to search and access the contents of a repository.
### Published information distribution and access services
High access demand for published information repositories can overburden participant's compute resources, dms3fs enables optional paid p2p services to offload compute and storage onto other p2p nodes.
p2p protocol network enables participants to contribute compute and storage resources to gain income in return to the use of their resources.
#### Information curators
Any source managing a dms3fs node may decide independently to assume a role as **_curator_** providing information services to the peer-to-peer network. These services are independent from locally published information and relate to information published by remote peer sources.
- A curator may choose to operate in limited **_roles_**, offering one or more hosting services to the peer-to-peer network:
- A **_directory_** role indicates a hosting service that enables metastore metadata queries.
- A **_replica_** role indicates a hosting service that enables infostore data queries.
- Each curator peer node decides what information and role, if any, is offered.
- Curators offer source authors information distribution and access services.
- Hosting services are periodically announced to the peer-to-peer network via the routing protocol.
The dms3fs peer-to-peer network will offer **_cryptocurrency_** as incentive gas to drive the information flow economy.
- A Curator may earn coins/tokens for hosting services
- An Author may pay transaction fees for distribution on the network
- An Author may earn tokens for published content in the information flow economy
- Paid content is structured as a pair of documents
- Free promotional (excerpt, abstract) short form content with terms references
- Paid long form content with term enforced by a smart contract
- Smart contracts enforce terms for a variety of payment plans
# Dis-intermediated Multi-State Space Search (DMS3)
## Implementation Notes
## Design Details
The search engine indexer may recover from certain faults on its own, for other faults external recovery support is require and provided by DMS3FS integrated features managing immutable state. The immutable state for a repository consists of the initial configuration parameters, documents added to the repository corpus, and other repository management state. To support recovery as well as, enable document access by remote peer nodes, the immutable state is stored in the DMS3FS filesystem.
To provide consistent search engine performance while scaling to vast and ever increasing amount of information, aggregate data is organized in multiple dimensions:
1. A set of index repositories or ** _infostore_ ** collects documents of similar ** _kind_ ** of content. As content added to a repository and as the repository grows to reach a configurable limit, the set is expanded with an additional repository to host additional documents of the same kind. Repository may expand up to an architectural limit, providing automatic sharding of information. The repository document limit and other infostore life-cycle properties can be configured based on kind specific basis.
2. An ** _metastore_ ** is a set of index repositories that collects documents providing metadata information for infostores of similar kind, hosted at local or remote peer nodes in the network, under a common key (i.e. multiple network nodes may author information of a specific kind). An metastore enables limiting the scope of query search as well as, the query target (data or metadata) of search. Each document in an metastore repository describes the kind of data contained in an associated infostore published by a source.
Infostore and metastore life-cycle management is at the purview of the user managing a DMS3FS node a.k.a the ** _source_ **.
- A source may create infostores for personal use or to publish and share information with nodes in the peer-to-peer network.
- An metastore is defined by associating (or binding) a locally unique key name with a set of locally sourced infostores of similar kind. Key names are not required be unique in the global DMS3FS peer-to-peer network.
- Ideally, the community of sources shares information using commonly accepted key names for aggregating metadata representing similar content semantics into searchable metadata repositories. For example,
- A source that may want to publish and share blogs, creates infostores under the kind key "blog" to describe the infostores food-blog and product-brand-review-blog, each of which consistes of a reposet.
- food and product review blogs can now be added to the respective infostores.
- The source then creates an metastore under the kind key blog, associated with the food-blog and product-brand-review-blog inforstores. This metastore also consists of a reposet.
- A document is added to the blog metastore to describe the food-blog infostore.
- A separate document is added the blog metastore to describe the product-brand-review-blog.
- A source publishes information by announcing metastores to the peer-to-peer network.
- Published information is routed through the peer-to-peer network to enable remote search and access.
- Routing protocols enable period refresh and revocation of published information.
- Any of the following may represent information you publish
- tweets, messages, blogs, podcasts, media, news, research, other articles
- web browser applications that people can access and run on their client nodes
- data generated by your application backend that others can search
- HTML is the only document encoding supported initially
Any source managing a DMS3FS node may decide independently to assume a role as ** _curator_ ** providing information services to the peer-to-peer network. These services are independent from locally published information and relate to information published by remote peer sources.
- A curator may choose to operate in limited ** _roles_ **, offering one or more hosting services to the peer-to-peer network:
- A ** _directory_ ** role indicates a hosting service that enables metastore metadata queries.
- A ** _replica_ ** role indicates a hosting service that enables infostore data queries.
- Each curator peer node decides what information and role, if any, is offered.
- Curators offer source authors information distribution and access services.
- Hosting services are periodically announced to the peer-to-peer network via the routing protocol.
Once developed, the DMS3FS peer-to-peer network will offer ** _cryptocurrency as incentive_ ** gas to drive the information flow economy.
- A Curator may earn tokens for hosting services
- An Author source may pay transaction fees for publishing on the network
- An Author may pay transaction fees for distribution on the network
- An Author may earn tokens for published content in the information flow economy
- Paid content is structured as a pair of documents
- Free promotional (excerpt, abstract) short form content with terms references
- Paid long form content with term enforced by a smart contract
- Smart contracts enforce terms for a variety of payment plans
## Key Components of the Architecture
## Preexisting Component behavior
### Preexisting Component behavior
### Peer Node Builder and Command Execution Environment
Enable new kind of Web Apps and services where generated data stays at the endpoint with the user. Data sharing is fully the purview of the source endpoint user.
Enable data discovery that does not require a centralized intermediary.
### Details
- Dis-intermediated dissemination of information
- Allow participants to self-publish and share information into the peer-to-peer network
-Add information into DMS3 local endpoint repositories for indexing. Information may consist of:
-Index information into DMS3 local endpoint repositories. Information may consist of:
- documents, tweets, messages,
- blogs, podcasts, media,
- news, research, web apps, and
- other articles, etc.
- Publish information in local repositories to allow others on the network to also search and access the information you want to share
- Publish local information to allow others on the network to also search and access information you want to share.
- Dis-intermediated retrieval of information
- Allow participants to find information published into the peer-to-peer network
- Search information in local repositories
- Search information published by others into the network
- Allow participants to find information published into the peer-to-peer network.
- Search local repositories, and
- Search remote repositories published by others.
- Allow participants to opt-in to new information alerts
- kind of information to follow
- peer sources of interest to follow
- Allow participants to opt-in and follow new information via alerts.
- kind of information to follow,
- peer sources of interest to follow.
- Allow participants to filter out unacceptable forms of information
- content matching filters, templates, and policies
- content source filters, templates, and policies
- Allow participants to filter unacceptable forms of information.
- content filters, templates, and policies,
- source filters, templates, and policies.
- Dis-intermediated monetization decisions for the information economy
- Pricing decision is made in isolation, independent of who the purchaser may be
- Purchasing decision is made in isolation, independent of who the seller may be
- Dis-intermediated monetization decisions for the information economy.
- Pricing decision is made in isolation, independent of who the purchaser may be,
- Purchasing decision is made in isolation, independent of who the seller may be.
- [Dis-]intermediated network service
-*No web server involved*
-*Cryptocurrency token* operated network
- smart contract enforced content distribution agreements
- Many services are Dis-intermediated
- Lower barrier, Fair, and competitive markets for intermediated services
- Security in an untrusted network environment
- Censorship free publishing
- Self imposed retrieval censorship based on local norms
- Community based rating of information sources
- Transparent information source ranking rules
- Ex: does published manifest or metadata accurately describe data?
- Dispute resolution inter-mediation process
- A peer-to-peer network architecture
- Blockchain technology
- Smart contracts
- Cryptocurrency economics
-*No web servers involved*
-*Cryptocurrency token* operated network.
- smart contract enforced content distribution agreements,
- Many services are Dis-intermediated,
- Lower barrier to entry,
- fair and competitive markets for intermediated services.
- Security in an untrusted network environment.
- Censorship free publishing,
- Self imposed retrieval censorship based on local norms,
- Community based rating of information sources,
- Transparent information source ranking rules,
- Ex: does published manifest or metadata accurately describe published data?
- Dispute resolution and inter-mediation process.
- A peer-to-peer network architecture,
- Blockchain technology,
- Smart contracts,
- Cryptocurrency economics.
- Commitment to sharing innovation in technology
- Inspired by IPFS
- Inspired by Ethereum
- Open source technology
- Transparent technology
- Audited security
- Open source platform technology
- Transparent platform technology
- Audited platform security
- Permissive platform allows integration and offer of centralized services using proprietary technology enabling entrepreneur and early investor gains at fair market valuations.
