diff --git a/README.md b/README.md index 2a7c0844b212fb5b042e2c1381189fbcbbff8994..eae36fccfcbdb0da7f59b23064cf445198abf56e 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ # https://mermaidjs.github.io/#/ # https://github.com/squidfunk/mkdocs-material/issues/693 # https://cdnjs.com/ -# https://github.com/mermaid-js/mermaid/ +# https://github.com/mermaid-https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-debian-10js/mermaid/ mkdir -p ~/Work/dapp/dms3 mkdir -p ~/Work/dapp/dms3/mkdocs-docs @@ -46,18 +46,37 @@ cp -v dist/mermaid.js* ../mkdocs-docs/docs/js/ cd ../ python --version pip --version -sudo apt-get install python-pip - -wget https://bootstrap.pypa.io/get-pip.py -sudo python get-pip.py +#sudo apt-get install python-pip +# +#wget https://bootstrap.pypa.io/get-pip.py +#sudo python get-pip.py +# follow this instead to setup a virtual python environment +#https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-debian-10 +sudo apt update +sudo apt -y upgrade +python3 -V +sudo apt install -y python3-pip +sudo apt install build-essential libssl-dev libffi-dev python3-dev +sudo apt install -y python3-venv +mkdir python_env +cd python_env +python3.7 -m venv mkdocs_env +ls mkdocs_env +source mkdocs_env/bin/activate +nano hello.py + print("Hello, World!") +python hello.py -sudo pip install mkdocs +pip install mkdocs mkdocs --version mkdocs new mkdocs-docs cd mkdocs-docs mkdocs serve -sudo pip install pymdown-extensions -sudo pip install mkdocs-rtd-dropdown +pip install pymdown-extensions +pip install mkdocs-rtd-dropdown + +mkdocs build +mkdocs serve # to support math rendering: # https://stackoverflow.com/questions/27882261/mkdocs-and-mathjax/31926644 @@ -77,3 +96,7 @@ mkdocs build # local live reload server mkdocs serve # defaults to 127.0.0.1:8000 mkdocs serve --dev-addr=192.168.0.7:8000 + +# deploy +rsync -rv site/* gitlab.dms3.io:~/site/ +/sbin/apach2ctl restart diff --git a/docs/design.md b/docs/design.md index 2a5df9885db10333fffc039838e95f3bf881acc4..dd4fe3472348f99036d216b4db760163c2c2a7b8 100644 --- a/docs/design.md +++ b/docs/design.md @@ -22,13 +22,13 @@ Information is organized into index repository **_containers_** [^1] considering [^2]: container-set and reposet is used interchangeably in this document, it represents a generic term that refers to the set of index repositories that form an infostore. -3. A **_metastore_** is a set of index repositories that collects documents providing metadata for corresponding infostores of similar kind. A metastore serves to limit search scope of a query target. Each document in an metastore repository describes the kind of data contained in an associated infostore. A metastore also represents a manifest for hosted information, serves as a testament made by the endpoint user regarding hosted information. +3. A **_metastore_** is a set of index repositories that collects documents providing metadata for corresponding infostores of similar kind. A metastore serves to limit search scope of a query target. Each document in a metastore repository describes the kind of data contained in an associated infostore. A metastore also represents a manifest for hosted information, serves as a testament made by the endpoint user regarding hosted information. Infostore and metastore life-cycle management is at the purview of a user managing a DMS3 endpoint node, also known as a **_source_** of information. - A source may create infostores for personal use or to publish and share information with nodes in the peer-to-peer network. -- A metastore is defined by associating (or binding) a locally unique key name with a set of locally hosted infostores of similar kind. Key names are not required be unique in the global DMS3 peer-to-peer network. +- A metastore is defined by associating (or binding) a locally unique key name with a set of locally hosted infostores of similar kind. Key names are not required to be unique in the global DMS3 peer-to-peer network. - Ideally, the community of sources shares information using commonly accepted key names for aggregating metadata representing similar content semantics into searchable metadata repositories. For example, @@ -36,7 +36,7 @@ Infostore and metastore life-cycle management is at the purview of a user managi - Food and product review blogs can now be added to the respective infostores. - - The source also creates an metastore under the kind key blog, associated with the food-blog and product-brand-review-blog inforstores. This metastore also consists of a reposet. + - The source also creates a metastore under the kind key blog, associated with the food-blog and product-brand-review-blog inforstores. This metastore also consists of a reposet. - A document is added to the blog metastore to describe the food-blog infostore. @@ -51,29 +51,30 @@ Infostore and metastore life-cycle management is at the purview of a user managi - applications that people can access and run on their client nodes - data produced by an application that others can search -Any source managing a dms3fs node may decide independently to assume a role as **_curator_** [^3] providing information services on behalf of other peer-to-peer network nodes. +Any source managing a dms3 node may decide independently to assume a role as **_curator_** [^3] providing information services on behalf of other peer-to-peer network nodes. [^3]: curators provide compute and storage hosting resources for published information in return for income. ## Installing the software -DMS3 uses a forked version of the [IPFS](https://ipfs.io/) with modifications to support new capabilities. +Installation instructions will be provided at a later time. DMS3 builds on forked versions of key open source software components. A partial list of key components is provided below, additional credits will be provided at a later time. -In particular dms3fs adds the ability to manage document index repositories and dis-intermediated search. +- [IPFS](https://ipfs.io/) with modifications to support new capabilities. In particular dms3 adds the ability to manage document index repositories and dis-intermediated search. +- [Indri Search Engine](https://sourceforge.net/p/lemur/wiki/Home/) as its underlying Search Engine (**_SE_**) with some modifications that allow extensible non-text data type support. ## Configuring the software -When you initialize dms3fs with the _dms3fs init_ command, dms3fs uses a repository in the local file system. +When you initialize dms3 with the _dms3 init_ command, dms3 uses a repository in the local file system. -By default, the repo is located at ~/.dms3-fs. To change the repo location, set the $DMS3FS_PATH environment variable: +By default, the repo is located at ~/.dms3. To change the repo location, set the $DMS3_PATH environment variable: ```bash - export DMS3FS_PATH=/path/to/dms3fsrepo + export DMS3_PATH=/path/to/dms3repo ``` -dms3fs expands the use of local file system by using the _index_ sub-folder within it for managing index repositories. +dms3 expands the use of local file system by using the _index_ sub-folder within it for managing index repositories. ## Decentralized applications (DApps) @@ -98,64 +99,41 @@ A summary of _index_ repository life-cycle commands are listed below. - use _restart_ to restart the daemon - use _recover_ to destructively rebuild a corrupted index repository -When not running in daemon mode, each command affecting an index first performs an integrity check on the index and may detect that the index has been corrupted. Some corruptions may be healed, in other cases the _recover_ command must be used to destroy the corrupted index and rebuild the index from redundant storage. In daemon mode, index integrity checks are performed when the daemon first opens and index repository. +When not running in daemon mode, each command affecting an index, first performs an integrity check on the index and may detect that the index has been corrupted. Some corruptions may be healed, in other cases the _recover_ command must be used to destroy the corrupted index and rebuild the index from redundant storage. In daemon mode, index integrity checks are performed when the daemon first opens and index repository. -```go -$ dms3fs index -h -USAGE - dms3fs index - Interact with DMS3FS index repositories. - - dms3fs index - - 'dms3fs index' is a command used to manipulate DMS3FS index repositories. - -SUBCOMMANDS - dms3fs index addoc - Add document to index repository. - dms3fs index config [] - Get and set dms3fs index config values. - dms3fs index ls - List index repositories. - dms3fs index mkdoc - Make document template for new content. - dms3fs index mkidx []... - Make index repository set. - dms3fs index publish - Publish index repository. - dms3fs index recover - Not yet implemented. - dms3fs index restart - Not yet implemented. - dms3fs index rmdoc - Remove document from index repository. - dms3fs index show - Not yet implemented. - dms3fs index start - Not yet implemented. - dms3fs index stat - Not yet implemented. - dms3fs index stop - Not yet implemented. - -Use 'dms3fs index --help' for more information about this command. - -``` ### Base graphic user interface (GUI) -## Application programming interface (API) +To-Be-Specified ## Search Engine -DMS3 uses a forked version of the [Indri Search Engine](https://sourceforge.net/p/lemur/wiki/Home/) with some modifications. +DMS3 uses a forked version of the open source [Indri Search Engine](https://sourceforge.net/p/lemur/wiki/Home/) as its underlying Search Engine (**_SE_**) with some modifications that allow extensible non-text data type support. Refer to the following page for [a background on the Indri search engine](https://sourceforge.net/p/lemur/wiki/Language%20Modeling%20and%20Information%20Retrieval%20Background/). -Refer to **_FIXME_** for a discussion of ODM/DES modifications applied to the search engine. +Refer to [Indri repository structure](https://sourceforge.net/p/lemur/wiki/Indri%20Repository%20Structure/) for a discussion of the repository on disk structure. + +Refer to [Indri parameters file](https://sourceforge.net/p/lemur/wiki/IndriBuildIndex%20Parameters/) for a discussion of search engine indexer configuration parameters. -The search engine can service (create or connect to) multiple repositories. Each repository has an active in-memory index where new documents are added to the index dynamically. In-memory indexes are written out to disk for durability, and when the number of indexes grows a background process merges repository indexes consolidating them on disk. All this dynamic index management is performed by the search engine for each of the connected repository contexts. +## Application programming interface (API) -dms3fs treats each index repository as a container, and manages the growth in information using multiple kinds of reposets to physically chard information in bounded containers. +DMS3 provides an API to simplify dapp developments that target the platform. Details of the dms3api will be provided at a later time. -### Index repository state +The API helps dapps create or connect to multiple repositories. Each repository has an active in-memory index where new documents are added to the index dynamically. In-memory indexes are written out to disk for durability, and when the number of indexes grows a background process merges repository indexes consolidating them on disk. All this dynamic index management is performed by the search engine for each of the connected repository contexts. -Refer to [Indri repository structure](https://sourceforge.net/p/lemur/wiki/Indri%20Repository%20Structure/) for a discussion of the repository on disk structure. +dms3 treats each index repository as a container, and manages the growth in information using multiple kinds of reposets to physically chard information in bounded containers. -dms3fs uses a path naming convention for each index repository on the local file system within the _index_ sub-folder. In the following example: +### Index repository state + +dms3 uses a path naming convention for each index repository on the local file system within the _index_ sub-folder. In the following example: ```bash -ls ~/.dms3-fs/index/reposet/blog/myblog20/w1543348319-a1-c1-o0/ +ls ~/.dms3/index/reposet/blog/myblog20/w1543348319-a1-c1-o0/ ``` where path components have the following implied semantics -~/.dms3-fs -: initialized local file system dms3fs root +~/.dms3 +: initialized local file system dms3 root /index : initialized local file system index repository root @@ -188,10 +166,6 @@ where path components have the following implied semantics area/category designations provide applications with a logical sharding mechanism. -### Configuring parameters - -Refer to [Indri parameters file](https://sourceforge.net/p/lemur/wiki/IndriBuildIndex%20Parameters/) for a discussion of search engine indexer configuration parameters. - ### Index metadata and fields The indexer defines metadata as non-searchable _fields_ and provides forward and reverse document lookup by metadata value. Whereas, fields can be used to qualify query searches. @@ -271,12 +245,12 @@ The set of nodes in the private network may be owned by the user or leased from Nodes and storage scaling capacity of private networks may be restricted to enable reasonable performance expectations. Private networks may be configured to asymmetrically distribute redundant information to compensate for variations of hardware configurations within the defined set of endpoints. -dms3fs implements two PBFS services. +dms3 implements two PBFS services. 1. Information storage and retrieval, providing index and query services. 2. Network file system (NFS) service, providing backing for node storage. -DMS3 PBFS services automatically recover from a configured number of simultaneous faults. +DMS3 PBFS services automatically recover from a configured number of arbitrary simultaneous faults. #### Index fault recovery @@ -285,7 +259,7 @@ The search engine can heal and recover an index from certain faults on its own, There are two options to recover an index repository. -1. Redundant repository state stored in dms3fs. When an index is created, its configuration state is saved external to the search engine. When a document is added to an index, the document is also saved external to the search engine. The _recover_ command can destructively recreate the index from this externally save information. This recovery option does not rely on PBFT. +1. Redundant repository state stored in dms3. When an index is created, its configuration state is saved external to the search engine. When a document is added to an index, the document is also saved external to the search engine. The _recover_ command can destructively recreate the index from this externally save information. This recovery option does not rely on PBFT. 2. Redundant copies the local file system index repository root. Optional PBFT services automatically keep redundant copies of information in sync. Recovery for simultaneous fault counts exceeding the configured protection level must be handled manually as an exception case. @@ -296,15 +270,15 @@ There are two options to recover an index repository. Use the _index_ _publish_ command to enable other users on the p2p network to search and access the contents of a repository. -### Published information distribution and access services +### Published information distribution and protection services -High access demand for published information repositories can overburden participant's compute resources, dms3fs enables optional paid p2p services to offload compute and storage onto other p2p nodes. +High demand for published information can overburden participant's compute resources, dms3 enables optional paid p2p services to offload compute and storage onto other p2p nodes. p2p protocol network enables participants to contribute compute and storage resources to gain income in return to the use of their resources. #### Information curators -Any source managing a dms3fs node may decide independently to assume a role as **_curator_** providing information services to the peer-to-peer network. These services are independent from locally published information and relate to information published by remote peer sources. +Any source managing a dms3 node may decide independently to assume a role as **_curator_** providing information services to the peer-to-peer network. These services are independent from locally published information and relate to information published by remote peer sources. - A curator may choose to operate in limited **_roles_**, offering one or more hosting services to the peer-to-peer network: - A **_directory_** role indicates a hosting service that enables metastore metadata queries. @@ -313,7 +287,7 @@ Any source managing a dms3fs node may decide independently to assume a role as * - Curators offer source authors information distribution and access services. - Hosting services are periodically announced to the peer-to-peer network via the routing protocol. -The dms3fs peer-to-peer network will offer **_cryptocurrency_** as incentive gas to drive the information flow economy. +The dms3 peer-to-peer network will offer **_cryptocurrency_** as incentive gas to drive the information flow economy. - A Curator may earn coins/tokens for hosting services - An Author may pay transaction fees for distribution on the network - An Author may earn tokens for published content in the information flow economy diff --git a/docs/how.md b/docs/how.md index ee1659ef7c2bf48d26d76958828e7341f0cbba5e..ebe4720abd625e488a0af700fa777d309f500012 100644 --- a/docs/how.md +++ b/docs/how.md @@ -7,6 +7,7 @@ - No unsolicited information flow, receiving endpoint must pull or opt-in - Each source independently chooses information to share publicly - Centralized services are optional + - Data protection services - Data distribution capacity, availability, and reliability services - Independent centralized service sources @@ -21,7 +22,7 @@ - Strong privacy and security - Anonymous public distributed ledger - transactions are transparent, yet anonymous - - identity and personal activity is not a product + - identity and personal activity is not a product for sale - Each participant independently self-censors and decides acceptable speech - participants can independently validate claims - network facilitates claims management diff --git a/docs/intro.md b/docs/intro.md index 540b2b8942908cbc9e8ee89c35f21e5d772b4c86..7c1292938042b084dab2f92f354ec8df8ba10548 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -75,3 +75,23 @@ Acceptable speech is defined by each endpoint and in accordance to locally interpreted cultural norms. The network protocols provide mechanisms to quarantine and penalize peer nodes that violate the acceptable use policy. + +#### Balancing individual and group rights + +In an ideal governance environment, information flow and sharing should be maximized and be free for the benefit of the group. + +Innovation accumulates knowledge, sharing drives and accelerates wealth and economic growth. + +Individuals share more of their knowledge and wealth when they perceive their basic needs are secure and growth objectives achievable. Individual needs include: + +1. Food +2. Shelter +3. Health +4. Security, safety, privacy +5. Fair share +6. Continuity +7. Acceptance + +Due to inefficiencies in our current governance practices and lack of guarantees on individual needs, information flow is sub-optimal and free sharing often (not always) limited to self promotion and gain. Where self may be an individual or a segment of society or group. + +DMS3 will be open source software with certain proprietary components and services developed and non-free to meet individual needs. diff --git a/docs/why.md b/docs/why.md index c35746d20bf8af0301f5b83116c7d306f7cade9f..c5c6d7700f80c4de5abd3f636f0616fdbfde4d86 100644 --- a/docs/why.md +++ b/docs/why.md @@ -187,3 +187,29 @@ Prof: "The returns to capital are significantly outpacing the returns to labor." "Place defines enemies, fears, actions and, above all, limits." "Societies and people run on different clocks. This is the fundamental tension between a nation and an individual." + +##### [Trends in Income From 1975 to 2018](https://www.rand.org/content/dam/rand/pubs/working_papers/WRA500/WRA516-1/RAND_WRA516-1.pdf) + +"By Carter C. Price and Kathryn Edwards +November 20th, 2020" + +"The three decades following the Second World War saw a period of economic growth that was +shared across the income distribution, but inequality in taxable income has increased +substantially over the last four decades. This work seeks to quantify the scale of income gap +created by rising inequality compared to a counterfactual in which growth was shared more +broadly." + +"For the two decades following the Second World War, income grew at a rate close to the +economy-wide growth rate across the full income distribution, which reduced income inequality +by most measures. Anemic growth from 1969 to 1974 further reduced inequality. But since then, +the benefits of growth have not been evenly shared. Multiple studies have found that labor, +capital, pre-tax, and post-tax income has been increasingly concentrated at the top of the +distribution since the middle of the 20th century." + +##### [The 'Ugly Truth' About Facebook: How The Company's Policies Cause Its Biggest Problems](https://www.wbur.org/hereandnow/2021/07/19/an-ugly-truth-facebook) + +In "An Ugly Truth: Inside Facebook's Battle for Domination," New York Times reporters Sheera Frenkel and Cecilia Kang chronicle a series of scandals between 2016 and 2021 over how Facebook policies its platform. + +The new book explores in-depth the inner workings of the company and its top executives. The word ugly in the title comes from a memo written by one of Facebook's own vice presidents, with Frenkel and Kang’s reporting highlighting that many of the platform’s perceived flaws are deliberate design choices. + +“So many of Facebook's problems are built into the way that they do business,” Frenkel says. “The very business model that they're premised on … is to keep you online.”