Commit 7a572606 authored by tavit ohanian's avatar tavit ohanian

Merge remote-tracking branch 'upstream/master' into reference

parents 26076264 f10507e3
Pipeline #53 failed with stages
in 0 seconds
blank_issues_enabled: false
contact_links:
- name: Getting Help on IPFS
url: https://ipfs.io/help
about: All information about how and where to get help on IPFS.
- name: IPFS Official Forum
url: https://discuss.ipfs.io
about: Please post general questions, support requests, and discussions here.
---
name: Open an issue
about: Only for actionable issues relevant to this repository.
title: ''
labels: need/triage
assignees: ''
---
<!--
Hello! To ensure this issue is correctly addressed as soon as possible by the IPFS team, please try to make sure:
- This issue is relevant to this repository's topic or codebase.
- A clear description is provided. It should includes as much relevant information as possible and clear scope for the issue to be actionable.
FOR GENERAL DISCUSSION, HELP OR QUESTIONS, please see the options at https://ipfs.io/help or head directly to https://discuss.ipfs.io.
(you can delete this section after reading)
-->
# Configuration for welcome - https://github.com/behaviorbot/welcome
# Configuration for new-issue-welcome - https://github.com/behaviorbot/new-issue-welcome
# Comment to be posted to on first time issues
newIssueWelcomeComment: >
Thank you for submitting your first issue to this repository! A maintainer
will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the
necessary information to make this process easy! Any information that can
help save additional round trips is useful! We currently aim to give
initial feedback within **two business days**. If this does not happen, feel
free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an
overview of priorities, assignments and additional actions requested by the
maintainers:
- "Priority" labels will show how urgent this is for the team.
- "Status" labels will show if this is ready to be worked on, blocked, or in progress.
- "Need" labels will indicate if additional input or analysis is required.
Finally, remember to use https://discuss.ipfs.io if you just need general
support.
# Configuration for new-pr-welcome - https://github.com/behaviorbot/new-pr-welcome
# Comment to be posted to on PRs from first time contributors in your repository
newPRWelcomeComment: >
Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure
that:
* The context for this PR is clear, with relevant discussion, decisions
and stakeholders linked/mentioned.
* Your contribution itself is clear (code comments, self-review for the
rest) and in its best form. Follow the [code contribution
guidelines](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md#code-contribution-guidelines)
if they apply.
Getting other community members to do a review would be great help too on
complex PRs (you can ask in the chats/forums). If you are unsure about
something, just leave us a comment.
Next steps:
* A maintainer will triage and assign priority to this PR, commenting on
any missing things and potentially assigning a reviewer for high
priority items.
* The PR gets reviews, discussed and approvals as needed.
* The PR is merged by maintainers when it has been approved and comments addressed.
We currently aim to provide initial feedback/triaging within **two business
days**. Please keep an eye on any labelling actions, as these will indicate
priorities and status of your contribution.
We are very grateful for your contribution!
# Configuration for first-pr-merge - https://github.com/behaviorbot/first-pr-merge
# Comment to be posted to on pull requests merged by a first time user
# Currently disabled
#firstPRMergeComment: ""
os:
- linux
language: go
go:
- 1.11.x
env:
global:
- GOTFLAGS="-race"
matrix:
- BUILD_DEPTYPE=gomod
# disable travis install
install:
- true
script:
- bash <(curl -s https://raw.githubusercontent.com/ipfs/ci-helpers/master/travis-ci/run-standard-tests.sh)
cache:
directories:
- $GOPATH/src/gx
- $GOPATH/pkg/mod
- $HOME/.cache/go-build
notifications:
email: false
The MIT License (MIT)
Copyright (c) 2016 Jeromy Johnson
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
all: build
build:
go build ./...
.PHONY: build
test:
go test ./...
.PHONY: test
benchmark:
go test -bench=./...
.PHONY: benchmark
# go-dms3ld-cbor go-ipld-cbor
==================
dms3 ld cbor [![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)
\ No newline at end of file [![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://libp2p.io/)
[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)
[![Coverage Status](https://coveralls.io/repos/github/libp2p/js-libp2p-floodsub/badge.svg?branch=master)](https://coveralls.io/github/libp2p/js-libp2p-floodsub?branch=master)
[![Travis CI](https://travis-ci.org/libp2p/js-libp2p-floodsub.svg?branch=master)](https://travis-ci.org/libp2p/js-libp2p-floodsub)
> An implementation of a cbor encoded merkledag object.
## Status
This library **has alternatives available**: For new projects, prefer using the [cbor codec](https://github.com/ipld/go-ipld-prime/tree/master/codec/dagcbor) included with [go-ipld-prime](https://github.com/ipld/go-ipld-prime).
This library is in **standby** mode. It works, but we recommend migrating to alternatives if possible. New features are unlikely to be added here.
## Lead Maintainer
[Eric Myhre](https://github.com/warpfork)
## Table of Contents
- [Install](#install)
- [Usage](#usage)
- [API](#api)
- [Contribute](#contribute)
- [License](#license)
## Install
```sh
make install
```
## Usage
Note: This package isn't the easiest to use.
```go
// Make an object
obj := map[interface{}]interface{}{
"foo": "bar",
"baz": &Link{
Target: myCid,
},
}
// Parse it into an ipldcbor node
nd, err := WrapMap(obj)
fmt.Println(nd.Links())
```
## Contribute
PRs are welcome!
Small note: If editing the Readme, please conform to the [standard-readme](https://github.com/RichardLitt/standard-readme) specification.
## License
MIT © Jeromy Johnson
coverage:
range: "50...100"
comment: off
package encoding
import (
"sync"
refmt "github.com/polydawn/refmt"
"github.com/polydawn/refmt/obj/atlas"
)
// PooledCloner is a thread-safe pooled object cloner.
type PooledCloner struct {
pool sync.Pool
}
// NewPooledCloner returns a PooledCloner with the given atlas. Do not copy
// after use.
func NewPooledCloner(atl atlas.Atlas) PooledCloner {
return PooledCloner{
pool: sync.Pool{
New: func() interface{} {
return refmt.NewCloner(atl)
},
},
}
}
type selfCloner interface {
Clone(b interface{}) error
}
// Clone clones a into b using a cloner from the pool.
func (p *PooledCloner) Clone(a, b interface{}) error {
if self, ok := a.(selfCloner); ok {
return self.Clone(b)
}
c := p.pool.Get().(refmt.Cloner)
err := c.Clone(a, b)
p.pool.Put(c)
return err
}
package encoding
import (
"bytes"
"io"
"sync"
cbor "github.com/polydawn/refmt/cbor"
"github.com/polydawn/refmt/obj/atlas"
)
type proxyWriter struct {
w io.Writer
}
func (w *proxyWriter) Write(b []byte) (int, error) {
return w.w.Write(b)
}
// Marshaller is a reusbale CBOR marshaller.
type Marshaller struct {
marshal *cbor.Marshaller
writer proxyWriter
}
// NewMarshallerAtlased constructs a new cbor Marshaller using the given atlas.
func NewMarshallerAtlased(atl atlas.Atlas) *Marshaller {
m := new(Marshaller)
m.marshal = cbor.NewMarshallerAtlased(&m.writer, atl)
return m
}
type cborMarshaler interface {
MarshalCBOR(w io.Writer) error
}
// Encode encodes the given object to the given writer.
func (m *Marshaller) Encode(obj interface{}, w io.Writer) error {
m.writer.w = w
var err error
selfMarshaling, ok := obj.(cborMarshaler)
if ok {
err = selfMarshaling.MarshalCBOR(w)
} else {
err = m.marshal.Marshal(obj)
}
m.writer.w = nil
return err
}
// Marshal marshels the given object to a byte slice.
func (m *Marshaller) Marshal(obj interface{}) ([]byte, error) {
var buf bytes.Buffer
if err := m.Encode(obj, &buf); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// PooledMarshaller is a thread-safe pooled CBOR marshaller.
type PooledMarshaller struct {
pool sync.Pool
}
// NewPooledMarshaller returns a PooledMarshaller with the given atlas. Do not
// copy after use.
func NewPooledMarshaller(atl atlas.Atlas) PooledMarshaller {
return PooledMarshaller{
pool: sync.Pool{
New: func() interface{} {
return NewMarshallerAtlased(atl)
},
},
}
}
// Marshal marshals the passed object using the pool of marshallers.
func (p *PooledMarshaller) Marshal(obj interface{}) ([]byte, error) {
m := p.pool.Get().(*Marshaller)
bts, err := m.Marshal(obj)
p.pool.Put(m)
return bts, err
}
// Encode encodes the passed object to the given writer using the pool of
// marshallers.
func (p *PooledMarshaller) Encode(obj interface{}, w io.Writer) error {
m := p.pool.Get().(*Marshaller)
err := m.Encode(obj, w)
p.pool.Put(m)
return err
}
package encoding
import (
"bytes"
"io"
"sync"
cbor "github.com/polydawn/refmt/cbor"
"github.com/polydawn/refmt/obj/atlas"
)
type proxyReader struct {
r io.Reader
}
func (r *proxyReader) Read(b []byte) (int, error) {
return r.r.Read(b)
}
// Unmarshaller is a reusable CBOR unmarshaller.
type Unmarshaller struct {
unmarshal *cbor.Unmarshaller
reader proxyReader
}
// NewUnmarshallerAtlased creates a new reusable unmarshaller.
func NewUnmarshallerAtlased(atl atlas.Atlas) *Unmarshaller {
m := new(Unmarshaller)
m.unmarshal = cbor.NewUnmarshallerAtlased(cbor.DecodeOptions{CoerceUndefToNull: true}, &m.reader, atl)
return m
}
type cborUnmarshaler interface {
UnmarshalCBOR(r io.Reader) error
}
// Decode reads a CBOR object from the given reader and decodes it into the
// given object.
func (m *Unmarshaller) Decode(r io.Reader, obj interface{}) (err error) {
m.reader.r = r
selfUnmarshaler, ok := obj.(cborUnmarshaler)
if ok {
err = selfUnmarshaler.UnmarshalCBOR(r)
} else {
err = m.unmarshal.Unmarshal(obj)
}
m.reader.r = nil
return err
}
// Unmarshal unmarshals the given CBOR byte slice into the given object.
func (m *Unmarshaller) Unmarshal(b []byte, obj interface{}) error {
return m.Decode(bytes.NewReader(b), obj)
}
// PooledUnmarshaller is a thread-safe pooled CBOR unmarshaller.
type PooledUnmarshaller struct {
pool sync.Pool
}
// NewPooledUnmarshaller returns a PooledUnmarshaller with the given atlas. Do
// not copy after use.
func NewPooledUnmarshaller(atl atlas.Atlas) PooledUnmarshaller {
return PooledUnmarshaller{
pool: sync.Pool{
New: func() interface{} {
return NewUnmarshallerAtlased(atl)
},
},
}
}
// Decode decodes an object from the passed reader into the given object using
// the pool of unmarshallers.
func (p *PooledUnmarshaller) Decode(r io.Reader, obj interface{}) error {
u := p.pool.Get().(*Unmarshaller)
err := u.Decode(r, obj)
p.pool.Put(u)
return err
}
// Unmarshal unmarshals the passed object using the pool of unmarshallers.
func (p *PooledUnmarshaller) Unmarshal(b []byte, obj interface{}) error {
u := p.pool.Get().(*Unmarshaller)
err := u.Unmarshal(b, obj)
p.pool.Put(u)
return err
}
module github.com/ipfs/go-ipld-cbor
require (
github.com/ipfs/go-block-format v0.0.2
github.com/ipfs/go-cid v0.0.3
github.com/ipfs/go-ipfs-util v0.0.1
github.com/ipfs/go-ipld-format v0.0.1
github.com/multiformats/go-multihash v0.0.10
github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992
github.com/smartystreets/goconvey v0.0.0-20190222223459-a17d461953aa // indirect
github.com/warpfork/go-wish v0.0.0-20180510122957-5ad1f5abf436 // indirect
github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158
)
go 1.13
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU=
github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48=
github.com/ipfs/go-block-format v0.0.2 h1:qPDvcP19izTjU8rgo6p7gTXZlkMkF5bz5G3fqIsSCPE=
github.com/ipfs/go-block-format v0.0.2/go.mod h1:AWR46JfpcObNfg3ok2JHDUfdiHRgWhJgCQF+KIgOPJY=
github.com/ipfs/go-cid v0.0.1 h1:GBjWPktLnNyX0JiQCNFpUuUSoMw5KMyqrsejHYlILBE=
github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.3 h1:UIAh32wymBpStoe83YCzwVQQ5Oy/H0FdxvUS6DJDzms=
github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-ipfs-util v0.0.1 h1:Wz9bL2wB2YBJqggkA4dD7oSmqB4cAnpNbGrlHJulv50=
github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc=
github.com/ipfs/go-ipld-format v0.0.1 h1:HCu4eB/Gh+KD/Q0M8u888RFkorTWNIL3da4oc5dwc80=
github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dCDnkOJhcZkms=
github.com/jtolds/gls v4.2.1+incompatible h1:fSuqC+Gmlu6l/ZYAoZzx2pyucC8Xza35fpRVWLVmUEE=
github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16 h1:5W7KhL8HVF3XCFOweFD3BNESdnO8ewyYTFT2R+/b8FQ=
github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U=
github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771 h1:MHkK1uRtFbVqvAgvWxafZe54+5uBxLluGylDiKgdhwo=
github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
github.com/mr-tron/base58 v1.1.0 h1:Y51FGVJ91WBqCEabAi5OPUz38eAx8DakuAm5svLcsfQ=
github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8=
github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78=
github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI=
github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA=
github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA=
github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
github.com/multiformats/go-multihash v0.0.1 h1:HHwN1K12I+XllBCrqKnhX949Orn4oawPkegHMu2vDqQ=
github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
github.com/multiformats/go-multihash v0.0.10 h1:lMoNbh2Ssd9PUF74Nz008KGzGPlfeV6wH3rit5IIGCM=
github.com/multiformats/go-multihash v0.0.10/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992 h1:bzMe+2coZJYHnhGgVlcQKuRy4FSny4ds8dLQjw5P1XE=
github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992/go.mod h1:uIp+gprXxxrWSjjklXD+mN4wed/tMfjMMmN/9+JsA9o=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/goconvey v0.0.0-20190222223459-a17d461953aa h1:E+gaaifzi2xF65PbDmuKI3PhLWY6G5opMLniFq8vmXA=
github.com/smartystreets/goconvey v0.0.0-20190222223459-a17d461953aa/go.mod h1:2RVY1rIf+2J2o/IM9+vPq9RzmHDSseB7FoXiSNIUsoU=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/warpfork/go-wish v0.0.0-20180510122957-5ad1f5abf436 h1:qOpVTI+BrstcjTZLm2Yz/3sOnqkzj3FQoh0g+E5s3Gc=
github.com/warpfork/go-wish v0.0.0-20180510122957-5ad1f5abf436/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw=
github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158 h1:WXhVOwj2USAXB5oMDwRl3piOux2XMV9TANaYxXHdkoE=
github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158/go.mod h1:Xj/M2wWU+QdTdRbu/L/1dIZY8/Wb2K9pAhtroQuxJJI=
golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67 h1:ng3VDlRp5/DHpSWl02R4rM9I+8M2rhmsuLwAMmkLQWE=
golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8 h1:1wopBVtVdWnn03fZelqdXTqk7U7zPQCb+T4rbU9ZEoU=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190219092855-153ac476189d h1:Z0Ahzd7HltpJtjAHHxX8QFP3j1yYgiuvjbjRzDj/KH0=
golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
package cbornode
import (
"encoding/json"
"errors"
"io"
"math"
"strconv"
"strings"
blocks "github.com/ipfs/go-block-format"
cid "github.com/ipfs/go-cid"
node "github.com/ipfs/go-ipld-format"
mh "github.com/multiformats/go-multihash"
)
// CBORTagLink is the integer used to represent tags in CBOR.
const CBORTagLink = 42
// Node represents an IPLD node.
type Node struct {
obj interface{}
tree []string
links []*node.Link
raw []byte
cid cid.Cid
}
// Compile time check to make sure Node implements the node.Node interface
var _ node.Node = (*Node)(nil)
var (
// ErrNoSuchLink is returned when no link with the given name was found.
ErrNoSuchLink = errors.New("no such link found")
ErrNonLink = errors.New("non-link found at given path")
ErrInvalidLink = errors.New("link value should have been bytes")
ErrInvalidKeys = errors.New("map keys must be strings")
ErrArrayOutOfRange = errors.New("array index out of range")
ErrNoLinks = errors.New("tried to resolve through object that had no links")
ErrEmptyLink = errors.New("link value was empty")
ErrInvalidMultibase = errors.New("invalid multibase on IPLD link")
ErrNonStringLink = errors.New("link should have been a string")
)
// DecodeBlock decodes a CBOR encoded Block into an IPLD Node.
//
// This method *does not* canonicalize and *will* preserve the CID. As a matter
// of fact, it will assume that `block.Cid()` returns the correct CID and will
// make no effort to validate this assumption.
//
// In general, you should not be calling this method directly. Instead, you
// should be calling the `Decode` method from the `go-ipld-format` package. That
// method will pick the right decoder based on the Block's CID.
//
// Note: This function keeps a reference to `block` and assumes that it is
// immutable.
func DecodeBlock(block blocks.Block) (node.Node, error) {
return decodeBlock(block)
}
func decodeBlock(block blocks.Block) (*Node, error) {
var m interface{}
if err := DecodeInto(block.RawData(), &m); err != nil {
return nil, err
}
return newObject(block, m)
}
func newObject(block blocks.Block, m interface{}) (*Node, error) {
tree, links, err := compute(m)
if err != nil {
return nil, err
}
return &Node{
obj: m,
tree: tree,
links: links,
raw: block.RawData(),
cid: block.Cid(),
}, nil
}
var _ node.DecodeBlockFunc = DecodeBlock
// Decode decodes a CBOR object into an IPLD Node.
//
// If passed a non-canonical CBOR node, this function will canonicalize it.
// Therefore, `bytes.Equal(b, Decode(b).RawData())` may not hold. If you already
// have a CID for this data and want to ensure that it doesn't change, you
// should use `DecodeBlock`.
// mhType is multihash code to use for hashing, for example mh.SHA2_256
//
// Note: This function does not hold onto `b`. You may reuse it.
func Decode(b []byte, mhType uint64, mhLen int) (*Node, error) {
var m interface{}
if err := DecodeInto(b, &m); err != nil {
return nil, err
}
// We throw away `b` here to ensure that we canonicalize the encoded
// CBOR object.
return WrapObject(m, mhType, mhLen)
}
// DecodeInto decodes a serialized IPLD cbor object into the given object.
func DecodeInto(b []byte, v interface{}) error {
return unmarshaller.Unmarshal(b, v)
}
func DecodeReader(r io.Reader, v interface{}) error {
return unmarshaller.Decode(r, v)
}
// WrapObject converts an arbitrary object into a Node.
func WrapObject(m interface{}, mhType uint64, mhLen int) (*Node, error) {
data, err := marshaller.Marshal(m)
if err != nil {
return nil, err
}
var obj interface{}
err = cloner.Clone(m, &obj)
if err != nil {
return nil, err
}
if mhType == math.MaxUint64 {
mhType = mh.SHA2_256
}
hash, err := mh.Sum(data, mhType, mhLen)
if err != nil {
return nil, err
}
c := cid.NewCidV1(cid.DagCBOR, hash)
block, err := blocks.NewBlockWithCid(data, c)
if err != nil {
// TODO: Shouldn't this just panic?
return nil, err
}
// No need to deserialize. We can just deep copy.
return newObject(block, obj)
}
// Resolve resolves a given path, and returns the object found at the end, as well
// as the possible tail of the path that was not resolved.
func (n *Node) Resolve(path []string) (interface{}, []string, error) {
var cur interface{} = n.obj
for i, val := range path {
switch curv := cur.(type) {
case map[string]interface{}:
next, ok := curv[val]
if !ok {
return nil, nil, ErrNoSuchLink
}
cur = next
case map[interface{}]interface{}:
next, ok := curv[val]
if !ok {
return nil, nil, ErrNoSuchLink
}
cur = next
case []interface{}:
n, err := strconv.Atoi(val)
if err != nil {
return nil, nil, err
}
if n < 0 || n >= len(curv) {
return nil, nil, ErrArrayOutOfRange
}
cur = curv[n]
case cid.Cid:
return &node.Link{Cid: curv}, path[i:], nil
default:
return nil, nil, ErrNoLinks
}
}
lnk, ok := cur.(cid.Cid)
if ok {
return &node.Link{Cid: lnk}, nil, nil
}
jsonish, err := convertToJSONIsh(cur)
if err != nil {
return nil, nil, err
}
return jsonish, nil, nil
}
// Copy creates a copy of the Node.
func (n *Node) Copy() node.Node {
links := make([]*node.Link, len(n.links))
copy(links, n.links)
raw := make([]byte, len(n.raw))
copy(raw, n.raw)
tree := make([]string, len(n.tree))
copy(tree, n.tree)
return &Node{
obj: copyObj(n.obj),
links: links,
raw: raw,
tree: tree,
cid: n.cid,
}
}
func copyObj(i interface{}) interface{} {
switch i := i.(type) {
case map[string]interface{}:
out := make(map[string]interface{})
for k, v := range i {
out[k] = copyObj(v)
}
return out
case map[interface{}]interface{}:
out := make(map[interface{}]interface{})
for k, v := range i {
out[k] = copyObj(v)
}
return out
case []interface{}:
var out []interface{}
for _, v := range i {
out = append(out, copyObj(v))
}
return out
default:
// TODO: do not be lazy
// being lazy for now
// use caution
return i
}
}
// ResolveLink resolves a path and returns the raw Link at the end, as well as
// the possible tail of the path that was not resolved.
func (n *Node) ResolveLink(path []string) (*node.Link, []string, error) {
obj, rest, err := n.Resolve(path)
if err != nil {
return nil, nil, err
}
lnk, ok := obj.(*node.Link)
if !ok {
return nil, rest, ErrNonLink
}
return lnk, rest, nil
}
// Tree returns a flattend array of paths at the given path for the given depth.
func (n *Node) Tree(path string, depth int) []string {
if path == "" && depth == -1 {
return n.tree
}
var out []string
for _, t := range n.tree {
if !strings.HasPrefix(t, path) {
continue
}
sub := strings.TrimLeft(t[len(path):], "/")
if sub == "" {
continue
}
if depth < 0 {
out = append(out, sub)
continue
}
parts := strings.Split(sub, "/")
if len(parts) <= depth {
out = append(out, sub)
}
}
return out
}
func compute(obj interface{}) (tree []string, links []*node.Link, err error) {
err = traverse(obj, "", func(name string, val interface{}) error {
if name != "" {
tree = append(tree, name[1:])
}
if lnk, ok := val.(cid.Cid); ok {
links = append(links, &node.Link{Cid: lnk})
}
return nil
})
if err != nil {
return nil, nil, err
}
return tree, links, nil
}
// Links lists all known links of the Node.
func (n *Node) Links() []*node.Link {
return n.links
}
func traverse(obj interface{}, cur string, cb func(string, interface{}) error) error {
if err := cb(cur, obj); err != nil {
return err
}
switch obj := obj.(type) {
case map[string]interface{}:
for k, v := range obj {
this := cur + "/" + k
if err := traverse(v, this, cb); err != nil {
return err
}
}
return nil
case map[interface{}]interface{}:
for k, v := range obj {
ks, ok := k.(string)
if !ok {
return errors.New("map key was not a string")
}
this := cur + "/" + ks
if err := traverse(v, this, cb); err != nil {
return err
}
}
return nil
case []interface{}:
for i, v := range obj {
this := cur + "/" + strconv.Itoa(i)
if err := traverse(v, this, cb); err != nil {
return err
}
}
return nil
default:
return nil
}
}
// RawData returns the raw bytes that represent the Node as serialized CBOR.
func (n *Node) RawData() []byte {
return n.raw
}
// Cid returns the canonical Cid of the NOde.
func (n *Node) Cid() cid.Cid {
return n.cid
}
// Loggable returns a loggable representation of the Node.
func (n *Node) Loggable() map[string]interface{} {
return map[string]interface{}{
"node_type": "cbor",
"cid": n.Cid(),
}
}
// Size returns the size of the binary representation of the Node.
func (n *Node) Size() (uint64, error) {
return uint64(len(n.RawData())), nil
}
// Stat returns stats about the Node.
// TODO: implement?
func (n *Node) Stat() (*node.NodeStat, error) {
return &node.NodeStat{}, nil
}
// String returns the string representation of the CID of the Node.
func (n *Node) String() string {
return n.Cid().String()
}
// MarshalJSON converts the Node into its JSON representation.
func (n *Node) MarshalJSON() ([]byte, error) {
out, err := convertToJSONIsh(n.obj)
if err != nil {
return nil, err
}
return json.Marshal(out)
}
// DumpObject marshals any object into its CBOR serialized byte representation
// TODO: rename
func DumpObject(obj interface{}) (out []byte, err error) {
return marshaller.Marshal(obj)
}
func toSaneMap(n map[interface{}]interface{}) (interface{}, error) {
if lnk, ok := n["/"]; ok && len(n) == 1 {
lnkb, ok := lnk.([]byte)
if !ok {
return nil, ErrInvalidLink
}
c, err := cid.Cast(lnkb)
if err != nil {
return nil, err
}
return map[string]interface{}{"/": c}, nil
}
out := make(map[string]interface{})
for k, v := range n {
ks, ok := k.(string)
if !ok {
return nil, ErrInvalidKeys
}
obj, err := convertToJSONIsh(v)
if err != nil {
return nil, err
}
out[ks] = obj
}
return out, nil
}
func convertToJSONIsh(v interface{}) (interface{}, error) {
switch v := v.(type) {
case map[interface{}]interface{}:
return toSaneMap(v)
case []interface{}:
var out []interface{}
if len(v) == 0 && v != nil {
return []interface{}{}, nil
}
for _, i := range v {
obj, err := convertToJSONIsh(i)
if err != nil {
return nil, err
}
out = append(out, obj)
}
return out, nil
default:
return v, nil
}
}
// FromJSON converts incoming JSON into a Node.
func FromJSON(r io.Reader, mhType uint64, mhLen int) (*Node, error) {
var m interface{}
err := json.NewDecoder(r).Decode(&m)
if err != nil {
return nil, err
}
obj, err := convertToCborIshObj(m)
if err != nil {
return nil, err
}
return WrapObject(obj, mhType, mhLen)
}
func convertToCborIshObj(i interface{}) (interface{}, error) {
switch v := i.(type) {
case map[string]interface{}:
if len(v) == 0 && v != nil {
return v, nil
}
if lnk, ok := v["/"]; ok && len(v) == 1 {
// special case for links
vstr, ok := lnk.(string)
if !ok {
return nil, ErrNonStringLink
}
return cid.Decode(vstr)
}
for a, b := range v {
val, err := convertToCborIshObj(b)
if err != nil {
return nil, err
}
v[a] = val
}
return v, nil
case []interface{}:
if len(v) == 0 && v != nil {
return v, nil
}
var out []interface{}
for _, o := range v {
obj, err := convertToCborIshObj(o)
if err != nil {
return nil, err
}
out = append(out, obj)
}
return out, nil
default:
return v, nil
}
}
func castBytesToCid(x []byte) (cid.Cid, error) {
if len(x) == 0 {
return cid.Cid{}, ErrEmptyLink
}
// TODO: manually doing multibase checking here since our deps don't
// support binary multibase yet
if x[0] != 0 {
return cid.Cid{}, ErrInvalidMultibase
}
c, err := cid.Cast(x[1:])
if err != nil {
return cid.Cid{}, ErrInvalidLink
}
return c, nil
}
func castCidToBytes(link cid.Cid) ([]byte, error) {
if !link.Defined() {
return nil, ErrEmptyLink
}
return append([]byte{0}, link.Bytes()...), nil
}
package cbornode
import (
"bytes"
"encoding/hex"
"encoding/json"
"fmt"
"io/ioutil"
"math"
"math/big"
"sort"
"strings"
"testing"
blocks "github.com/ipfs/go-block-format"
cid "github.com/ipfs/go-cid"
u "github.com/ipfs/go-ipfs-util"
mh "github.com/multiformats/go-multihash"
)
func init() {
RegisterCborType(BigIntAtlasEntry)
}
func assertCid(c cid.Cid, exp string) error {
if c.String() != exp {
return fmt.Errorf("expected cid of %s, got %s", exp, c)
}
return nil
}
func TestNonObject(t *testing.T) {
nd, err := WrapObject("", mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(nd.Cid(), "bafyreiengp2sbi6ez34a2jctv34bwyjl7yoliteleaswgcwtqzrhmpyt2m"); err != nil {
t.Fatal(err)
}
back, err := Decode(nd.Copy().RawData(), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(back.Cid(), "bafyreiengp2sbi6ez34a2jctv34bwyjl7yoliteleaswgcwtqzrhmpyt2m"); err != nil {
t.Fatal(err)
}
}
func TestDecodeInto(t *testing.T) {
nd, err := WrapObject(map[string]string{
"name": "foo",
}, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
var m map[string]string
err = DecodeInto(nd.RawData(), &m)
if err != nil {
t.Fatal(err)
}
if len(m) != 1 || m["name"] != "foo" {
t.Fatal("failed to decode object")
}
}
func TestDecodeIntoNonObject(t *testing.T) {
nd, err := WrapObject("foobar", mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
var s string
err = DecodeInto(nd.RawData(), &s)
if err != nil {
t.Fatal(err)
}
if s != "foobar" {
t.Fatal("strings don't match")
}
}
func TestBasicMarshal(t *testing.T) {
c := cid.NewCidV0(u.Hash([]byte("something")))
obj := map[string]interface{}{
"name": "foo",
"bar": c,
}
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(nd.Cid(), "bafyreib4hmpkwa7zyzoxmpwykof6k7akxnvmsn23oiubsey4e2tf6gqlui"); err != nil {
t.Fatal(err)
}
back, err := Decode(nd.RawData(), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(back.Cid(), "bafyreib4hmpkwa7zyzoxmpwykof6k7akxnvmsn23oiubsey4e2tf6gqlui"); err != nil {
t.Fatal(err)
}
lnk, _, err := back.ResolveLink([]string{"bar"})
if err != nil {
t.Fatal(err)
}
if !lnk.Cid.Equals(c) {
t.Fatal("expected cid to match")
}
if !nd.Cid().Equals(back.Cid()) {
t.Fatal("re-serialize failed to generate same cid")
}
}
func TestMarshalRoundtrip(t *testing.T) {
c1 := cid.NewCidV0(u.Hash([]byte("something1")))
c2 := cid.NewCidV0(u.Hash([]byte("something2")))
c3 := cid.NewCidV0(u.Hash([]byte("something3")))
obj := map[string]interface{}{
"foo": "bar",
"hello": c1,
"baz": []interface{}{
c1,
c2,
},
"cats": map[string]interface{}{
"qux": c3,
},
}
nd1, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(nd1.Cid(), "bafyreibgx4rjaqolj7c32c7ibxc5tedhisc4d23ihx5t4tgamuvy2hvwjm"); err != nil {
orig, err1 := json.Marshal(obj)
if err1 != nil {
t.Fatal(err1)
}
js, err1 := nd1.MarshalJSON()
if err1 != nil {
t.Fatal(err1)
}
t.Fatalf("%s != %s\n%s", orig, js, err)
}
if len(nd1.Links()) != 4 {
t.Fatal("didnt have enough links")
}
nd2, err := Decode(nd1.RawData(), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if !nd1.Cid().Equals(nd2.Cid()) {
t.Fatal("objects didnt match between marshalings")
}
lnk, rest, err := nd2.ResolveLink([]string{"baz", "1", "bop"})
if err != nil {
t.Fatal(err)
}
if !lnk.Cid.Equals(c2) {
t.Fatal("expected c2")
}
if len(rest) != 1 || rest[0] != "bop" {
t.Fatal("should have had one path element remaning after resolve")
}
out, err := nd1.MarshalJSON()
if err != nil {
t.Fatal(err)
}
t.Log(string(out))
}
func assertStringsEqual(t *testing.T, a, b []string) {
if len(a) != len(b) {
t.Fatal("lengths differed: ", a, b)
}
sort.Strings(a)
sort.Strings(b)
for i, v := range a {
if v != b[i] {
t.Fatal("got mismatch: ", a, b)
}
}
}
func TestTree(t *testing.T) {
c1 := cid.NewCidV0(u.Hash([]byte("something1")))
c2 := cid.NewCidV0(u.Hash([]byte("something2")))
c3 := cid.NewCidV0(u.Hash([]byte("something3")))
c4 := cid.NewCidV0(u.Hash([]byte("something4")))
obj := map[string]interface{}{
"foo": c1,
"baz": []interface{}{c2, c3, "c"},
"cats": map[string]interface{}{
"qux": map[string]interface{}{
"boo": 1,
"baa": c4,
"bee": 3,
"bii": 4,
"buu": map[string]string{
"coat": "rain",
},
},
},
}
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(nd.Cid(), "bafyreicp66zmx7grdrnweetu23anx3e5zguda7646iwyothju6nhgqykgq"); err != nil {
t.Fatal(err)
}
full := []string{
"foo",
"baz",
"baz/0",
"baz/1",
"baz/2",
"cats",
"cats/qux",
"cats/qux/boo",
"cats/qux/baa",
"cats/qux/bee",
"cats/qux/bii",
"cats/qux/buu",
"cats/qux/buu/coat",
}
assertStringsEqual(t, full, nd.Tree("", -1))
cats := []string{
"qux",
"qux/boo",
"qux/baa",
"qux/bee",
"qux/bii",
"qux/buu",
"qux/buu/coat",
}
assertStringsEqual(t, cats, nd.Tree("cats", -1))
toplevel := []string{
"foo",
"baz",
"cats",
}
assertStringsEqual(t, toplevel, nd.Tree("", 1))
assertStringsEqual(t, []string{}, nd.Tree("", 0))
}
func TestParsing(t *testing.T) {
// This shouldn't pass
// Debug representation from cbor.io is
//
// D9 0102 # tag(258)
// 58 25 # bytes(37)
// A503221220659650FC3443C916428048EFC5BA4558DC863594980A59F5CB3C4D84867E6D31 # "\xA5\x03\"\x12 e\x96P\xFC4C\xC9\x16B\x80H\xEF\xC5\xBAEX\xDC\x865\x94\x98\nY\xF5\xCB<M\x84\x86~m1"
//
t.Skip()
b := []byte("\xd9\x01\x02\x58\x25\xa5\x03\x22\x12\x20\x65\x96\x50\xfc\x34\x43\xc9\x16\x42\x80\x48\xef\xc5\xba\x45\x58\xdc\x86\x35\x94\x98\x0a\x59\xf5\xcb\x3c\x4d\x84\x86\x7e\x6d\x31")
n, err := Decode(b, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(n.Cid(), "bafyreib4jqzv5rpohiu2vi56uggpnshhhnqrsclx2mji67wnihkrfboox4"); err != nil {
t.Fatal(err)
}
n, err = Decode(b, mh.SHA2_512, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(n.Cid(), "bafyrgqcomxa52fzyrx6d46sgposgxrzdyqpbtu5adh7fpjjhqo7mwftoexd26ghhhxcq53hczq3dobudmvyegjhamahzdu2k66jklr5uys3tq"); err != nil {
t.Fatal(err)
}
}
func TestFromJson(t *testing.T) {
data := `{
"something": {"/":"bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq"},
"cats": "not cats",
"cheese": [
{"/":"bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq"},
{"/":"bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq"},
{"/":"bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq"},
{"/":"bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq"}
]
}`
n, err := FromJSON(bytes.NewReader([]byte(data)), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(n.Cid(), "bafyreicnokmhmrnlp2wjhyk2haep4tqxiptwfrp2rrs7rzq7uk766chqvq"); err != nil {
t.Fatal(err)
}
c, ok := n.obj.(map[string]interface{})["something"].(cid.Cid)
if !ok {
fmt.Printf("%#v\n", n.obj)
t.Fatal("expected a cid")
}
if c.String() != "bafkreifvxooyaffa7gy5mhrb46lnpdom34jvf4r42mubf5efbodyvzeujq" {
t.Fatal("cid unmarshaled wrong")
}
}
func TestResolvedValIsJsonable(t *testing.T) {
data := `{
"foo": {
"bar": 1,
"baz": 2
}
}`
n, err := FromJSON(strings.NewReader(data), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if err := assertCid(n.Cid(), "bafyreiahcy6ewqmabbh7lcjhxrillpf72zlu3vqcovckanvj2fwdtenvbe"); err != nil {
t.Fatal(err)
}
val, _, err := n.Resolve([]string{"foo"})
if err != nil {
t.Fatal(err)
}
out, err := json.Marshal(val)
if err != nil {
t.Fatal(err)
}
if string(out) != `{"bar":1,"baz":2}` {
t.Fatal("failed to get expected json")
}
}
func TestExamples(t *testing.T) {
examples := map[string]string{
"[null]": "bafyreigtpkiih7wr7wb7ts6j5aunnotxiff3yqkx33rs4k4xhskprx5tui",
"[]": "bafyreidwx2fvfdiaox32v2mnn6sxu3j4qoxeqcuenhtgrv5qv6litfnmoe",
"{}": "bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua",
"null": "bafyreifqwkmiw256ojf2zws6tzjeonw6bpd5vza4i22ccpcq4hjv2ts7cm",
"1": "bafyreihtx752fmf3zafbys5dtr4jxohb53yi3qtzfzf6wd5274jwtn5agu",
"[1]": "bafyreihwrdqkjomfjaoqe5hbpfjzqoxkhptohvoa5u362s6obgpvxcw45q",
"true": "bafyreibhvppn37ufanewvxvwendgzksh3jpwhk6sxrx2dh3m7s3t5t7noa",
`{"a":"IPFS"}`: "bafyreihyyz2badz34h5pvcgof4fj3qwwr7mopoucejwbnpzs7soorkrct4",
`{"a":"IPFS","b":null,"c":[1]}`: "bafyreigg2gcszayx2lywb3edqfoftyvus7gxeanmudqla3e6eh2okei25a",
`{"a":[]}`: "bafyreian4t6wau4jdqt6nys76dfvsn6g7an4ulbv326yzutdgnrr5cjpui",
}
for originalJSON, expcid := range examples {
t.Run(originalJSON, func(t *testing.T) {
check := func(err error) {
if err != nil {
t.Fatalf("for object %s: %s", originalJSON, err)
}
}
n, err := FromJSON(bytes.NewReader([]byte(originalJSON)), mh.SHA2_256, -1)
check(err)
check(assertCid(n.Cid(), expcid))
cbor := n.RawData()
_, err = Decode(cbor, mh.SHA2_256, -1)
check(err)
node, err := Decode(cbor, mh.SHA2_256, -1)
check(err)
jsonBytes, err := node.MarshalJSON()
check(err)
json := string(jsonBytes)
if json != originalJSON {
t.Fatalf("marshaled to incorrect JSON: %s != %s", originalJSON, json)
}
})
}
}
func TestObjects(t *testing.T) {
raw, err := ioutil.ReadFile("test_objects/expected.json")
if err != nil {
t.Fatal(err)
}
var cases map[string]map[string]string
err = json.Unmarshal(raw, &cases)
if err != nil {
t.Fatal(err)
}
for k, c := range cases {
t.Run(k, func(t *testing.T) {
in, err := ioutil.ReadFile(fmt.Sprintf("test_objects/%s.json", k))
if err != nil {
t.Fatal(err)
}
expected, err := ioutil.ReadFile(fmt.Sprintf("test_objects/%s.cbor", k))
if err != nil {
t.Fatal(err)
}
nd, err := FromJSON(bytes.NewReader(in), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
cExp, err := cid.Decode(c["/"])
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(nd.RawData(), expected) {
t.Fatalf("bytes do not match: %x != %x", nd.RawData(), expected)
}
if !nd.Cid().Equals(cExp) {
t.Fatalf("cid missmatch: %s != %s", nd.String(), cExp.String())
}
})
}
}
func TestCanonicalize(t *testing.T) {
b, err := ioutil.ReadFile("test_objects/non-canon.cbor")
if err != nil {
t.Fatal(err)
}
nd1, err := Decode(b, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if bytes.Equal(b, nd1.RawData()) {
t.Fatal("failed to canonicalize node")
}
if err := assertCid(nd1.Cid(), "bafyreiawx7ona7oa2ptcoh6vwq4q6bmd7x2ibtkykld327bgb7t73ayrqm"); err != nil {
t.Fatal(err)
}
nd2, err := Decode(nd1.RawData(), mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
if !nd2.Cid().Equals(nd1.Cid()) || !bytes.Equal(nd2.RawData(), nd1.RawData()) {
t.Fatal("re-decoding a canonical node should be idempotent")
}
}
func TestStableCID(t *testing.T) {
b, err := ioutil.ReadFile("test_objects/non-canon.cbor")
if err != nil {
t.Fatal(err)
}
hash, err := mh.Sum(b, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
c := cid.NewCidV1(cid.DagCBOR, hash)
badBlock, err := blocks.NewBlockWithCid(b, c)
if err != nil {
t.Fatal(err)
}
badNode, err := DecodeBlock(badBlock)
if err != nil {
t.Fatal(err)
}
if !badBlock.Cid().Equals(badNode.Cid()) {
t.Fatal("CIDs not stable")
}
}
func TestCidAndBigInt(t *testing.T) {
type Foo struct {
B *big.Int
A cid.Cid
}
RegisterCborType(Foo{})
nd, err := WrapObject("", mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
c := nd.Cid()
_, err = WrapObject(&Foo{
A: c,
B: big.NewInt(1),
}, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
}
func TestEmptyCid(t *testing.T) {
type Foo struct {
A cid.Cid
}
type Bar struct {
A cid.Cid `refmt:",omitempty"`
}
RegisterCborType(Foo{})
RegisterCborType(Bar{})
_, err := WrapObject(&Foo{}, mh.SHA2_256, -1)
if err == nil {
t.Fatal("should have failed to encode an object with an empty but non-omitted CID")
}
_, err = WrapObject(&Bar{}, mh.SHA2_256, -1)
if err != nil {
t.Fatal(err)
}
}
func TestCanonicalStructEncoding(t *testing.T) {
type Foo struct {
Zebra string
Dog int
Cats float64
Whale string
Cat bool
}
RegisterCborType(Foo{})
s := Foo{
Zebra: "seven",
Dog: 15,
Cats: 1.519,
Whale: "never",
Cat: true,
}
m, err := WrapObject(s, math.MaxUint64, -1)
if err != nil {
t.Fatal(err)
}
expraw, err := hex.DecodeString("a563636174f563646f670f6463617473fb3ff84dd2f1a9fbe7657768616c65656e65766572657a6562726165736576656e")
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(expraw, m.RawData()) {
t.Fatal("not canonical")
}
}
type TestMe struct {
Hello *big.Int
World big.Int
Hi int
}
func TestBigIntRoundtrip(t *testing.T) {
RegisterCborType(TestMe{})
one := TestMe{
Hello: big.NewInt(100),
World: *big.NewInt(99),
}
bytes, err := DumpObject(&one)
if err != nil {
t.Fatal(err)
}
var oneBack TestMe
if err := DecodeInto(bytes, &oneBack); err != nil {
t.Fatal(err)
}
if one.Hello.Cmp(oneBack.Hello) != 0 {
t.Fatal("failed to roundtrip *big.Int")
}
if one.World.Cmp(&oneBack.World) != 0 {
t.Fatal("failed to roundtrip big.Int")
}
list := map[string]*TestMe{
"hello": &TestMe{Hello: big.NewInt(10), World: *big.NewInt(101), Hi: 1},
"world": &TestMe{Hello: big.NewInt(9), World: *big.NewInt(901), Hi: 3},
}
bytes, err = DumpObject(list)
if err != nil {
t.Fatal(err)
}
var listBack map[string]*TestMe
if err := DecodeInto(bytes, &listBack); err != nil {
t.Fatal(err)
}
t.Log(listBack["hello"])
t.Log(listBack["world"])
if list["hello"].Hello.Cmp(listBack["hello"].Hello) != 0 {
t.Fatalf("failed to roundtrip *big.Int: %s != %s", list["hello"].Hello, listBack["hello"].Hello)
}
if list["hello"].World.Cmp(&listBack["hello"].World) != 0 {
t.Fatalf("failed to roundtrip big.Int: %s != %s", &list["hello"].World, &listBack["hello"].World)
}
if list["world"].Hello.Cmp(listBack["world"].Hello) != 0 {
t.Fatalf("failed to roundtrip *big.Int: %s != %s", list["world"].Hello, listBack["world"].Hello)
}
if list["world"].World.Cmp(&listBack["world"].World) != 0 {
t.Fatalf("failed to roundtrip big.Int: %s != %s", &list["world"].World, &listBack["world"].World)
}
}
package cbornode
import (
"bufio"
"bytes"
cbor "github.com/polydawn/refmt/cbor"
"github.com/polydawn/refmt/pretty"
"github.com/polydawn/refmt/shared"
)
//HumanReadable returns a string representation of a CBOR blob
func HumanReadable(blob []byte) (string, error) {
reader := bytes.NewReader(blob)
var buf bytes.Buffer
writer := bufio.NewWriter(&buf)
err := shared.TokenPump{
TokenSource: cbor.NewDecoder(cbor.DecodeOptions{}, reader),
TokenSink: pretty.NewEncoder(writer),
}.Run()
if err != nil {
return "", err
}
if err = writer.Flush(); err != nil {
return "", err
}
return buf.String(), nil
}
package cbornode
import (
"math/big"
cid "github.com/ipfs/go-cid"
encoding "github.com/ipfs/go-ipld-cbor/encoding"
"github.com/polydawn/refmt/obj/atlas"
)
// This atlas describes the CBOR Tag (42) for IPLD links, such that refmt can marshal and unmarshal them
var cidAtlasEntry = atlas.BuildEntry(cid.Cid{}).
UseTag(CBORTagLink).
Transform().
TransformMarshal(atlas.MakeMarshalTransformFunc(
castCidToBytes,
)).
TransformUnmarshal(atlas.MakeUnmarshalTransformFunc(
castBytesToCid,
)).
Complete()
// BigIntAtlasEntry gives a reasonable default encoding for big.Int. It is not
// included in the entries by default.
var BigIntAtlasEntry = atlas.BuildEntry(big.Int{}).Transform().
TransformMarshal(atlas.MakeMarshalTransformFunc(
func(i big.Int) ([]byte, error) {
return i.Bytes(), nil
})).
TransformUnmarshal(atlas.MakeUnmarshalTransformFunc(
func(x []byte) (big.Int, error) {
return *big.NewInt(0).SetBytes(x), nil
})).
Complete()
// CborAtlas is the refmt.Atlas used by the CBOR IPLD decoder/encoder.
var CborAtlas atlas.Atlas
var cborSortingMode = atlas.KeySortMode_RFC7049
var atlasEntries = []*atlas.AtlasEntry{cidAtlasEntry}
var (
cloner encoding.PooledCloner
unmarshaller encoding.PooledUnmarshaller
marshaller encoding.PooledMarshaller
)
func init() {
rebuildAtlas()
}
func rebuildAtlas() {
CborAtlas = atlas.MustBuild(atlasEntries...).
WithMapMorphism(atlas.MapMorphism{KeySortMode: atlas.KeySortMode_RFC7049})
marshaller = encoding.NewPooledMarshaller(CborAtlas)
unmarshaller = encoding.NewPooledUnmarshaller(CborAtlas)
cloner = encoding.NewPooledCloner(CborAtlas)
}
// RegisterCborType allows to register a custom cbor type
func RegisterCborType(i interface{}) {
var entry *atlas.AtlasEntry
if ae, ok := i.(*atlas.AtlasEntry); ok {
entry = ae
} else {
entry = atlas.BuildEntry(i).StructMap().AutogenerateWithSortingScheme(atlas.KeySortMode_RFC7049).Complete()
}
atlasEntries = append(atlasEntries, entry)
rebuildAtlas()
}
package cbornode
import (
"bytes"
"context"
"fmt"
block "github.com/ipfs/go-block-format"
cid "github.com/ipfs/go-cid"
mh "github.com/multiformats/go-multihash"
recbor "github.com/polydawn/refmt/cbor"
atlas "github.com/polydawn/refmt/obj/atlas"
cbg "github.com/whyrusleeping/cbor-gen"
)
// IpldStore wraps a Blockstore and provides an interface for storing and retrieving CBOR encoded data.
type IpldStore interface {
Get(ctx context.Context, c cid.Cid, out interface{}) error
Put(ctx context.Context, v interface{}) (cid.Cid, error)
}
// IpldBlockstore defines a subset of the go-ipfs-blockstore Blockstore interface providing methods
// for storing and retrieving block-centered data.
type IpldBlockstore interface {
Get(cid.Cid) (block.Block, error)
Put(block.Block) error
}
// IpldBlockstoreViewer is a trait that enables zero-copy access to blocks in
// a blockstore.
type IpldBlockstoreViewer interface {
// View provides zero-copy access to blocks in a blockstore. The callback
// function will be invoked with the value for the key. The user MUST not
// modify the byte array, as it could be memory-mapped.
View(cid.Cid, func([]byte) error) error
}
// BasicIpldStore wraps and IpldBlockstore and implements the IpldStore interface.
type BasicIpldStore struct {
Blocks IpldBlockstore
Viewer IpldBlockstoreViewer
Atlas *atlas.Atlas
}
var _ IpldStore = &BasicIpldStore{}
// NewCborStore returns an IpldStore implementation backed by the provided IpldBlockstore.
func NewCborStore(bs IpldBlockstore) *BasicIpldStore {
viewer, _ := bs.(IpldBlockstoreViewer)
return &BasicIpldStore{Blocks: bs, Viewer: viewer}
}
// Get reads and unmarshals the content at `c` into `out`.
func (s *BasicIpldStore) Get(ctx context.Context, c cid.Cid, out interface{}) error {
if s.Viewer != nil {
// zero-copy path.
return s.Viewer.View(c, func(b []byte) error {
return s.decode(b, out)
})
}
blk, err := s.Blocks.Get(c)
if err != nil {
return err
}
return s.decode(blk.RawData(), out)
}
func (s *BasicIpldStore) decode(b []byte, out interface{}) error {
cu, ok := out.(cbg.CBORUnmarshaler)
if ok {
if err := cu.UnmarshalCBOR(bytes.NewReader(b)); err != nil {
return NewSerializationError(err)
}
return nil
}
if s.Atlas == nil {
return DecodeInto(b, out)
} else {
return recbor.UnmarshalAtlased(recbor.DecodeOptions{}, b, out, *s.Atlas)
}
}
type cidProvider interface {
Cid() cid.Cid
}
// Put marshals and writes content `v` to the backing blockstore returning its CID.
func (s *BasicIpldStore) Put(ctx context.Context, v interface{}) (cid.Cid, error) {
mhType := uint64(mh.BLAKE2B_MIN + 31)
mhLen := -1
codec := uint64(cid.DagCBOR)
var expCid cid.Cid
if c, ok := v.(cidProvider); ok {
expCid := c.Cid()
pref := expCid.Prefix()
mhType = pref.MhType
mhLen = pref.MhLength
codec = pref.Codec
}
cm, ok := v.(cbg.CBORMarshaler)
if ok {
buf := new(bytes.Buffer)
if err := cm.MarshalCBOR(buf); err != nil {
return cid.Undef, NewSerializationError(err)
}
pref := cid.Prefix{
Codec: codec,
MhType: mhType,
MhLength: mhLen,
Version: 1,
}
c, err := pref.Sum(buf.Bytes())
if err != nil {
return cid.Undef, err
}
blk, err := block.NewBlockWithCid(buf.Bytes(), c)
if err != nil {
return cid.Undef, err
}
if err := s.Blocks.Put(blk); err != nil {
return cid.Undef, err
}
blkCid := blk.Cid()
if expCid != cid.Undef && blkCid != expCid {
return cid.Undef, fmt.Errorf("your object is not being serialized the way it expects to")
}
return blkCid, nil
}
nd, err := WrapObject(v, mhType, mhLen)
if err != nil {
return cid.Undef, err
}
if err := s.Blocks.Put(nd); err != nil {
return cid.Undef, err
}
ndCid := nd.Cid()
if expCid != cid.Undef && ndCid != expCid {
return cid.Undef, fmt.Errorf("your object is not being serialized the way it expects to")
}
return ndCid, nil
}
func NewSerializationError(err error) error {
return SerializationError{err}
}
type SerializationError struct {
err error
}
func (se SerializationError) Error() string {
return se.err.Error()
}
func (se SerializationError) Unwrap() error {
return se.err
}
func (se SerializationError) Is(o error) bool {
_, ok := o.(*SerializationError)
return ok
}
func NewMemCborStore() IpldStore {
return NewCborStore(newMockBlocks())
}
type mockBlocks struct {
data map[cid.Cid]block.Block
}
func newMockBlocks() *mockBlocks {
return &mockBlocks{make(map[cid.Cid]block.Block)}
}
func (mb *mockBlocks) Get(c cid.Cid) (block.Block, error) {
d, ok := mb.data[c]
if ok {
return d, nil
}
return nil, fmt.Errorf("Not Found")
}
func (mb *mockBlocks) Put(b block.Block) error {
mb.data[b.Cid()] = b
return nil
}
[{"/":"QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"}]
\ No newline at end of file
 
\ No newline at end of file
{
"array-link": {"/":"bafyreidj7kdv2gkmhycdq5bagiloawte5z6egwztbwovsk6v3xookbpyuu"},
"empty-array": {"/":"bafyreidwx2fvfdiaox32v2mnn6sxu3j4qoxeqcuenhtgrv5qv6litfnmoe"},
"empty-obj": {"/":"bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua"},
"foo": {"/":"bafyreid56m6jpnmimcltiqpdaujthpqfl722vd4kr2xjp6rpuhx66iri64"},
"foo2": {"/":"bafyreid56m6jpnmimcltiqpdaujthpqfl722vd4kr2xjp6rpuhx66iri64"},
"obj-with-link": {"/":"bafyreibvjvcv745gig4mvqs4hctx4zfkono4rjejm2ta6gtyzkqxfjeily"},
"obj-no-link": {"/":"bafyreiduyr3p3ppndxmoc4q2it3vmezjx2ndt5oakudtplfgyyfb2k7swm"}
}
{
"foo": "bar",
"cats": [
{"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"},
{
"something": "interesting"
},
[
"fish",
{"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"},
9
]
],
"other": {"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"}
}
{
"other": {"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"},
"cats": [
{"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"},
{
"something": "interesting"
},
[
"fish",
{"/": "QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"},
9
]
],
"foo": "bar"
}
isassafrashand cats
\ No newline at end of file
{
"sassafras": "and cats"
}
{
"foo": {"/":"QmRgutAxd8t7oGkSm4wmeuByG6M51wcTso6cubDdQtuEfL"}
}
package cbornode
import (
"sync"
"testing"
mh "github.com/multiformats/go-multihash"
)
type MyStruct struct {
Items map[string]MyStruct
Foo string
Bar []byte
Baz []int
}
func init() {
RegisterCborType(MyStruct{})
}
func testStruct() MyStruct {
return MyStruct{
Items: map[string]MyStruct{
"Foo": {
Foo: "Foo",
Bar: []byte("Bar"),
Baz: []int{1, 2, 3, 4},
},
"Bar": {
Bar: []byte("Bar"),
Baz: []int{1, 2, 3, 4},
},
},
Baz: []int{5, 1, 2},
}
}
func BenchmarkWrapObject(b *testing.B) {
obj := testStruct()
b.ResetTimer()
for i := 0; i < b.N; i++ {
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
b.Fatal(err, nd)
}
}
}
func BenchmarkDecodeBlock(b *testing.B) {
obj := testStruct()
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
b.Fatal(err, nd)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
nd2, err := DecodeBlock(nd)
if err != nil {
b.Fatal(err, nd2)
}
}
}
func BenchmarkWrapObjectParallel(b *testing.B) {
obj := testStruct()
b.ResetTimer()
var wg sync.WaitGroup
wg.Add(100)
for j := 0; j < 100; j++ {
go func() {
defer wg.Done()
for i := 0; i < b.N; i++ {
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
b.Fatal(err, nd)
}
}
}()
}
wg.Wait()
}
func BenchmarkDecodeBlockParallel(b *testing.B) {
obj := testStruct()
nd, err := WrapObject(obj, mh.SHA2_256, -1)
if err != nil {
b.Fatal(err, nd)
}
b.ResetTimer()
var wg sync.WaitGroup
wg.Add(100)
for j := 0; j < 100; j++ {
go func() {
defer wg.Done()
for i := 0; i < b.N; i++ {
nd2, err := DecodeBlock(nd)
if err != nil {
b.Fatal(err, nd2)
}
}
}()
}
wg.Wait()
}
func BenchmarkDumpObject(b *testing.B) {
obj := testStruct()
for i := 0; i < b.N; i++ {
bytes, err := DumpObject(obj)
if err != nil {
b.Fatal(err, bytes)
}
}
}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment