├── Makefile ├── bitmap ├── bitmap_test.go ├── bitmap.go ├── slice.go ├── toarray.go ├── join.go ├── ofmany.go ├── of.go ├── join_test.go ├── toarray_test.go ├── fmt_test.go ├── mask.go ├── next_bench_test.go ├── fromstr32.go ├── README.md ├── get.go ├── select_bench_test.go ├── fmt.go ├── next.go ├── builder.go ├── slice_test.go ├── rank_bench_test.go ├── ofmany_test.go ├── rank_test.go ├── tailbitmap.go ├── next_test.go ├── of_test.go ├── get_test.go ├── builder_test.go ├── fromstr32_test.go ├── rank.go └── tailbitmap_test.go ├── scripts ├── requirements.txt └── build_md.py ├── typehelper ├── typehelper.go ├── toslice.go └── toslice_test.go ├── .github ├── settings.yml ├── ISSUE_TEMPLATE │ ├── github-does-not-support-choosing-PR-template-on-web-yet │ ├── doc_improve.md │ ├── feature_request.md │ ├── enhancement.md │ ├── testing_request.md │ ├── refactor_request.md │ └── bug_report.md ├── settings-sample.yml ├── dependabot.yml ├── workflows │ ├── test.yml │ └── golangci-lint.yml ├── PULL_REQUEST_TEMPLATE.md └── CODE_OF_CONDUCT.md ├── pbcmpl ├── errors.go ├── header_test.go ├── header.go ├── pbcmpl.go └── pbcmpl_test.go ├── bmtree ├── pathlen.go ├── height.go ├── pathheight.go ├── pathbits.go ├── pathstr.go ├── bitmappath_check.go ├── bitmap_check.go ├── decode.go ├── pathcheck_release_test.go ├── height_test.go ├── pathcheck.go ├── pathheight_test.go ├── pathcheck_test.go ├── index_bench_test.go ├── pathstr_test.go ├── partial_tree_test.go ├── partial_tree.go ├── newpath.go ├── allpaths.go ├── bitmappath_check_test.go ├── bitmap_check_test.go ├── decode_test.go ├── bmtree.go ├── allpaths_test.go ├── pathlen_test.go ├── pathbits_test.go ├── index_debug_test.go ├── index_test.go └── newpath_test.go ├── .gitignore ├── go.mod ├── bitword ├── example_test.go ├── bitword.go └── bitword_test.go ├── vers ├── vers_debug_test.go ├── vers.go └── vers_test.go ├── .gitsubrepo ├── sigbits ├── sigbits_countprefixes.go ├── sigbits_test.go ├── sigbits.go ├── countprefixes.go ├── sharding.go ├── firstdiff.go ├── countprefixes_test.go ├── sigbits_countprefixes_test.go ├── firstdiff_test.go └── sharding_test.go ├── size ├── exmaple_size_test.go └── sizeof.go ├── appveyor.yml ├── mathext ├── zipf │ ├── zipf_example_access_test.go │ ├── zipf_example_perfect_test.go │ ├── zipf_example_rand_test.go │ └── zipf_test.go └── util │ ├── util_test.go │ └── util.go ├── .travis.yml ├── docs └── badges.md ├── LICENSE ├── README.md ├── tree ├── tree_test.go └── tree.go ├── iohelper └── iohelper.go ├── common.mk ├── bitstr └── bitstr.go └── go.sum /Makefile: -------------------------------------------------------------------------------- 1 | include common.mk 2 | -------------------------------------------------------------------------------- /bitmap/bitmap_test.go: -------------------------------------------------------------------------------- 1 | package bitmap_test 2 | -------------------------------------------------------------------------------- /scripts/requirements.txt: -------------------------------------------------------------------------------- 1 | semantic_version==2.6.0 2 | jinja2==2.11.3 3 | PyYAML==5.4 4 | -------------------------------------------------------------------------------- /typehelper/typehelper.go: -------------------------------------------------------------------------------- 1 | // Package typehelper provides with some type conversion utility functions. 2 | package typehelper 3 | -------------------------------------------------------------------------------- /.github/settings.yml: -------------------------------------------------------------------------------- 1 | _extends: gh-config 2 | 3 | repository: 4 | name: low 5 | description: low level data type and utils in Golang. 6 | topics: go, golang, datastructure, lowlevel, util 7 | -------------------------------------------------------------------------------- /pbcmpl/errors.go: -------------------------------------------------------------------------------- 1 | package pbcmpl 2 | 3 | import "errors" 4 | 5 | var ( 6 | // ErrInvalidHeaderSize indicates the size in the header is incorrect 7 | ErrInvalidHeaderSize = errors.New("headersize is incorrect") 8 | ) 9 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/github-does-not-support-choosing-PR-template-on-web-yet: -------------------------------------------------------------------------------- 1 | Choosing PR template can be done with URL query. 2 | 3 | For now github will only read PR template from `.github/PULL_REQUEST_TEMPLATE.md` only. 4 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/doc_improve.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Document 3 | about: Add document 4 | title: '' 5 | labels: 'doc' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **What document to add** 11 | 12 | **Additional context** 13 | -------------------------------------------------------------------------------- /.github/settings-sample.yml: -------------------------------------------------------------------------------- 1 | # Usage copy this to settings.yml 2 | 3 | _extends: gh-config 4 | 5 | 6 | repository: 7 | name: name 8 | description: desc 9 | homepage: https://openacid.github.io/ 10 | topics: go, golang 11 | -------------------------------------------------------------------------------- /bitmap/bitmap.go: -------------------------------------------------------------------------------- 1 | // Package bitmap provides basic bitmap operations. 2 | // A bitmap uses []uint64 as storage. 3 | // 4 | // Since 0.1.8 5 | package bitmap 6 | 7 | func init() { 8 | initMasks() 9 | initSelectLookup() 10 | } 11 | -------------------------------------------------------------------------------- /bmtree/pathlen.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "math/bits" 5 | ) 6 | 7 | // PathLen returns the number of bits for this varbit. 8 | // 9 | // Since 0.1.9 10 | func PathLen(p uint64) int32 { 11 | return int32(bits.OnesCount32(uint32(p))) 12 | } 13 | -------------------------------------------------------------------------------- /bmtree/height.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import "math/bits" 4 | 5 | // Height returns the tree height of this bitmap. 6 | // 7 | // Since 0.1.9 8 | func Height(bitmapSize int32) int32 { 9 | return int32(31 - bits.LeadingZeros32(uint32(bitmapSize))) 10 | } 11 | -------------------------------------------------------------------------------- /bmtree/pathheight.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import "math/bits" 4 | 5 | // PathHeight returns the tree height of a searching path. 6 | // 7 | // Since 0.1.9 8 | func PathHeight(path uint64) int32 { 9 | return int32(32 - bits.LeadingZeros32(uint32(path))) 10 | } 11 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: gomod 4 | directory: "/" 5 | schedule: 6 | interval: daily 7 | open-pull-requests-limit: 10 8 | ignore: 9 | - dependency-name: github.com/golang/protobuf 10 | versions: 11 | - 1.4.3 12 | - 1.5.1 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.exe~ 4 | *.dll 5 | *.so 6 | *.dylib 7 | 8 | # Test binary, built with `go test -c` 9 | *.test 10 | 11 | # Output of the go coverage tool, specifically when used with LiteIDE 12 | *.out 13 | 14 | # Dependency directories (remove the comment below to include it) 15 | # vendor/ 16 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/openacid/low 2 | 3 | go 1.14 4 | 5 | require ( 6 | github.com/blang/semver v3.5.1+incompatible 7 | github.com/golang/protobuf v1.4.2 8 | github.com/openacid/errors v0.8.1 9 | github.com/openacid/must v0.1.3 10 | github.com/openacid/testkeys v0.1.7 11 | github.com/pkg/errors v0.9.1 // indirect 12 | github.com/stretchr/testify v1.8.1 13 | ) 14 | -------------------------------------------------------------------------------- /bitword/example_test.go: -------------------------------------------------------------------------------- 1 | package bitword 2 | 3 | import "fmt" 4 | 5 | func Example() { 6 | bw4 := BitWord[4] 7 | 8 | fmt.Println(bw4.FromStr("abc")) 9 | fmt.Println(bw4.ToStr([]byte{6, 1, 6, 2, 6, 3})) 10 | fmt.Println(bw4.Get("abc", 1)) 11 | fmt.Println(bw4.FirstDiff("abc", "abd", 0, -1)) 12 | 13 | // Output: 14 | // [6 1 6 2 6 3] 15 | // abc 16 | // 1 17 | // 5 18 | 19 | } 20 | -------------------------------------------------------------------------------- /bmtree/pathbits.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | // PathBits returns the searching bit, e.g., discard the height info. 4 | // 5 | // Since 0.1.12 6 | func PathBits(path uint64) uint64 { 7 | return path >> 32 8 | } 9 | 10 | // PathMask returns the mask, e.g., discard the searching bits. 11 | // 12 | // Since 0.1.12 13 | func PathMask(path uint64) uint64 { 14 | return path & 0xffffffff 15 | } 16 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: 'feature' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | 12 | **Describe the solution you'd like** 13 | 14 | **Describe alternatives you've considered** 15 | 16 | **Additional context** 17 | -------------------------------------------------------------------------------- /bmtree/pathstr.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | // PathStr creates a human-readable string of a node searching path. 8 | // 9 | // Since 0.1.9 10 | func PathStr(path uint64) string { 11 | 12 | treeHeight := PathHeight(path) 13 | 14 | l := PathLen(path) 15 | if l == 0 { 16 | return "" 17 | } 18 | 19 | return fmt.Sprintf("%0[1]*[2]b", l, path>>uint(32+treeHeight-l)) 20 | } 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/enhancement.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Enhancement 3 | about: Enhancement to a feature/API 4 | title: '' 5 | labels: 'enhancement' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is the requested enhancement related to a problem? Please describe.** 11 | 12 | **Describe the solution** 13 | 14 | **Describe alternatives you've considered** 15 | 16 | **Additional context** 17 | 18 | **Affect other component or side effect** 19 | -------------------------------------------------------------------------------- /bmtree/bitmappath_check.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "github.com/openacid/must" 5 | ) 6 | 7 | func bitmapPathMustHaveEqualHeight(bitmapSize int32, path uint64) { 8 | 9 | // TODO remove these, add to Height() and PathHeight() 10 | bitmapSizeCheck(bitmapSize) 11 | pathCheck(path) 12 | if uint32(path) != 0 { 13 | must.Be.Equal(Height(bitmapSize), PathHeight(path), 14 | "bitmap height == path height") 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/testing_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Testing 3 | about: Improve test suite. 4 | title: '' 5 | labels: 'testing' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is the requested test related to a known/potential problem? Please describe.** 11 | 12 | **Describe what test should be added/modified** 13 | 14 | **Describe alternatives you've considered** 15 | 16 | **Additional context** 17 | 18 | **Affect other component or side effect** 19 | -------------------------------------------------------------------------------- /bitmap/slice.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // Slice returns a new bitmap which is a "slice" of the input bitmap. 4 | // 5 | // Since 0.1.9 6 | func Slice(words []uint64, from, to int32) []uint64 { 7 | 8 | l := ((to - from) + 63) & (^63) 9 | r := make([]uint64, l) 10 | 11 | for i := from; i < to; i++ { 12 | if words[i>>6]&(1<>6] |= 1 << uint(j&63) 15 | } 16 | } 17 | 18 | return r 19 | } 20 | -------------------------------------------------------------------------------- /bitmap/toarray.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // ToArray creates a new slice of int32 containing all of the integers stored in 4 | // the bitmap in sorted order. 5 | // 6 | // Since 0.1.9 7 | func ToArray(words []uint64) []int32 { 8 | 9 | r := make([]int32, 0) 10 | l := int32(len(words) * 64) 11 | 12 | for i := int32(0); i < l; i++ { 13 | if words[i>>6]&(1<>6) 11 | for i, e := range subs { 12 | j := i * int(size) 13 | r[j>>6] |= (e & Mask[size]) << uint(j&63) 14 | 15 | } 16 | return r 17 | } 18 | -------------------------------------------------------------------------------- /bmtree/bitmap_check.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "math/bits" 5 | 6 | "github.com/openacid/must" 7 | ) 8 | 9 | func bitmapSizeCheck(bitmapSize int32) { 10 | 11 | height := int32(31 - bits.LeadingZeros32(uint32(bitmapSize))) 12 | 13 | must.Be.True(height <= 30) 14 | must.Be.NotEqual(int32(0), bitmapSize) 15 | } 16 | 17 | func bitmapMustHaveLevel(bitmapSize, l int32) { 18 | must.Be.Equal(int32(1), (bitmapSize>>uint(l))&1, 19 | "level[pathlen] must be stored by bitmap") 20 | } 21 | -------------------------------------------------------------------------------- /bmtree/decode.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | // Decode a encoded bitmap, retrieves all paths in it. 4 | // 5 | // Since 0.1.9 6 | func Decode(bitmapSize int32, bm []uint64) []uint64 { 7 | 8 | rst := make([]uint64, 0) 9 | 10 | paths := AllPaths(bitmapSize, 0, 1<<63) 11 | 12 | for _, p := range paths { 13 | idx := PathToIndex(bitmapSize, p) 14 | 15 | wordI := idx >> 6 16 | 17 | if int32(len(bm)) > wordI && bm[wordI]&(1<> 32) 29 | 30 | must.Be.Equal(uint32(0), ^pmask&pbits) 31 | 32 | } 33 | -------------------------------------------------------------------------------- /bmtree/pathheight_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestMaxbits(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input uint64 15 | want int32 16 | }{ 17 | {0x0000000000000000, 0}, 18 | {0x0000000000000040, 7}, 19 | {0x0000000000000070, 7}, 20 | {0x0000007000000070, 7}, 21 | {0x0000006000000070, 7}, 22 | {0x0000000500000070, 7}, 23 | {0x000000050000007f, 7}, 24 | {0x0000000b0000007f, 7}, 25 | {0x00000000ffffffff, 32}, 26 | {0x00000012ffffffff, 32}, 27 | {0x00000012fffffffc, 32}, 28 | } 29 | 30 | for i, c := range cases { 31 | got := PathHeight(c.input) 32 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /appveyor.yml: -------------------------------------------------------------------------------- 1 | version: build-{build}.{branch} 2 | 3 | clone_folder: C:\gopath\src\github.com\openacid\low 4 | shallow_clone: true # for startup speed 5 | 6 | environment: 7 | GOPATH: C:\gopath 8 | 9 | platform: 10 | - x64 11 | 12 | # http://www.appveyor.com/docs/installed-software 13 | install: 14 | # some helpful output for debugging builds 15 | - go version 16 | - go env 17 | # pre-installed MinGW at C:\MinGW is 32bit only 18 | # but MSYS2 at C:\msys64 has mingw64 19 | - set PATH=C:\msys64\mingw64\bin;%PATH% 20 | - gcc --version 21 | - g++ --version 22 | 23 | build_script: 24 | - go install -v ./... 25 | 26 | test_script: 27 | - set PATH=C:\gopath\bin;%PATH% 28 | - go test -v ./... 29 | 30 | #artifacts: 31 | # - path: '%GOPATH%\bin\*.exe' 32 | deploy: off 33 | -------------------------------------------------------------------------------- /mathext/zipf/zipf_example_access_test.go: -------------------------------------------------------------------------------- 1 | package zipf 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | ) 7 | 8 | func ExampleAccesses() { 9 | 10 | n := 10 11 | 12 | // y = C * x**(1-s), x ∈ [a, b) 13 | a := float64(1) 14 | s := float64(1.5) 15 | 16 | got := Accesses(a, s, n, 50, nil) 17 | fmt.Println(got) 18 | 19 | arr := make([]int, n) 20 | for _, idx := range got { 21 | arr[idx]++ 22 | } 23 | for _, v := range arr { 24 | fmt.Println("|" + strings.Repeat("*", v)) 25 | } 26 | 27 | // Output: 28 | // 29 | // [0 3 5 6 6 5 8 7 7 7 0 8 6 6 0 6 4 0 7 6 6 6 0 6 2 6 5 6 0 6 8 5 7 3 0 4 0 8 7 0 6 3 6 0 0 4 6 6 7 7] 30 | // |*********** 31 | // | 32 | // |* 33 | // |*** 34 | // |*** 35 | // |**** 36 | // |**************** 37 | // |******** 38 | // |**** 39 | // | 40 | } 41 | -------------------------------------------------------------------------------- /bmtree/pathcheck_test.go: -------------------------------------------------------------------------------- 1 | // +build debug 2 | 3 | package bmtree 4 | 5 | import ( 6 | "testing" 7 | 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestPathCheck(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | cases := []struct { 16 | path uint64 17 | wantpanic bool 18 | }{ 19 | // path must be consecutive "1"s 20 | {5, true}, 21 | 22 | // path mask shorter than path bits 23 | {0xf<<32 + 0xe, true}, 24 | 25 | // path length must <=31 26 | {1 << 31, true}, 27 | {1 << 30, true}, 28 | 29 | {0x7, false}, 30 | } 31 | 32 | for _, c := range cases { 33 | if c.wantpanic { 34 | ta.Panics(func() { pathCheck(c.path) }, "%064b", c.path) 35 | } else { 36 | ta.NotPanics(func() { pathCheck(c.path) }, "%064b", c.path) 37 | 38 | } 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /bmtree/index_bench_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "fmt" 5 | "testing" 6 | ) 7 | 8 | var Input int32 = 23 9 | var Output int 10 | 11 | var IndexOfInput uint64 12 | 13 | func BenchmarkIndexOf(b *testing.B) { 14 | 15 | var s int32 = 0 16 | IndexOfInput = 0x0000aa000000ff00 17 | bitmapSize := int32(0xffff) 18 | 19 | for i := 0; i < b.N; i++ { 20 | s += PathToIndex(bitmapSize, IndexOfInput) 21 | } 22 | Output = int(s) 23 | } 24 | 25 | func BenchmarkPathOf(b *testing.B) { 26 | 27 | var s uint64 = 0 28 | 29 | for treeHeight := 1; treeHeight < 16; treeHeight++ { 30 | b.Run(fmt.Sprintf("treeHeight=%d", treeHeight), func(b *testing.B) { 31 | for i := 0; i < b.N; i++ { 32 | p := IndexToPath(4, Input) 33 | s += p 34 | } 35 | }) 36 | 37 | } 38 | Output = int(s) 39 | } 40 | -------------------------------------------------------------------------------- /sigbits/sigbits_test.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestNew(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input []string 15 | want []int32 16 | }{ 17 | {[]string{ 18 | "", 19 | "12345678", 20 | "12345678a", 21 | "12345678aa", 22 | "12345678aab", 23 | "12345678aac", 24 | "a", 25 | "aa", 26 | "aaa", 27 | "aac", 28 | "ab", 29 | "b", 30 | }, 31 | []int32{ 32 | 0, 33 | 64, 34 | 72, 35 | 80, 36 | 87, 37 | 1, 38 | 8, 39 | 16, 40 | 22, 41 | 14, 42 | 6, 43 | }, 44 | }, 45 | } 46 | 47 | for i, c := range cases { 48 | got := New(c.input) 49 | ta.Equal(c.want, got.sigbits, "%d-th: case: %+v", i+1, c) 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /bitmap/of.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // Of creates a bitmap of bits set to 1 at specified positions. 4 | // 5 | // An optional argument n can be provided to make the result bitmap 6 | // have at least n bits. 7 | // 8 | // Since 0.1.9 9 | func Of(bitPositions []int32, opts ...int32) []uint64 { 10 | 11 | n := int32(0) 12 | 13 | // The first opts is specified number of result bits. 14 | if len(opts) > 0 { 15 | n = opts[0] 16 | } 17 | 18 | if len(bitPositions) > 0 { 19 | max := bitPositions[len(bitPositions)-1] + 1 20 | if n < max { 21 | n = max 22 | } 23 | } 24 | if n < 0 { 25 | n = 0 26 | } 27 | 28 | nWords := (n + 63) >> 6 29 | words := make([]uint64, nWords) 30 | 31 | for _, i := range bitPositions { 32 | wordI := i >> 6 33 | i = i & 63 34 | words[wordI] |= 1 << uint(i) 35 | } 36 | return words 37 | 38 | } 39 | -------------------------------------------------------------------------------- /bmtree/pathstr_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestStr(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input uint64 15 | want string 16 | }{ 17 | {0x0000000000000000, ""}, 18 | {0x0000000000000040, "0"}, 19 | {0x0000000000000070, "000"}, 20 | {0x0000007000000070, "111"}, 21 | {0x0000006000000070, "110"}, 22 | {0x0000000500000070, "000"}, 23 | {0x000000050000007f, "0000101"}, 24 | {0x0000000b0000007f, "0001011"}, 25 | {0x00000000ffffffff, "00000000000000000000000000000000"}, 26 | {0x00000012ffffffff, "00000000000000000000000000010010"}, 27 | {0x00000012fffffffc, "000000000000000000000000000100"}, 28 | } 29 | 30 | for i, c := range cases { 31 | got := PathStr(c.input) 32 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /bitmap/join_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestJoin(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | subs []uint64 15 | size int32 16 | wantwords []uint64 17 | }{ 18 | {[]uint64{}, 0, []uint64{}}, 19 | {[]uint64{}, 1, []uint64{}}, 20 | {[]uint64{}, 2, []uint64{}}, 21 | 22 | {[]uint64{1, 2}, 1, []uint64{1}}, 23 | {[]uint64{1, 1, 0, 1, 0, 1}, 1, []uint64{43}}, 24 | {[]uint64{1, 2}, 2, []uint64{9}}, 25 | {[]uint64{1, 2}, 32, []uint64{1 + 2<<32}}, 26 | {[]uint64{1, 2, 3}, 32, []uint64{1 + 2<<32, 3}}, 27 | {[]uint64{1, 2, 3, 4}, 32, []uint64{1 + 2<<32, 3 + 4<<32}}, 28 | } 29 | 30 | for i, c := range cases { 31 | 32 | got := Join(c.subs, c.size) 33 | 34 | ta.Equal(c.wantwords, got, 35 | "%d-th: case: %+v", 36 | i+1, c) 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /mathext/zipf/zipf_example_perfect_test.go: -------------------------------------------------------------------------------- 1 | package zipf 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | ) 7 | 8 | func ExampleZipf_perfect() { 9 | 10 | n := 20 11 | 12 | // y = C * x**(1-s), x ∈ [a, b) 13 | a := float64(1) 14 | b := float64(20) 15 | s := float64(1.5) 16 | 17 | z := New(a, b, s) 18 | sampleCnt := float64(100) 19 | 20 | sample := make([]int, n) 21 | for u := float64(0); u < 1; u += 1 / sampleCnt { 22 | x := int(z.Float64(u)) 23 | sample[x]++ 24 | } 25 | 26 | for _, v := range sample { 27 | fmt.Println("|" + strings.Repeat("*", v)) 28 | } 29 | 30 | // Output: 31 | // 32 | // | 33 | // |************************************** 34 | // |***************** 35 | // |********** 36 | // |******* 37 | // |***** 38 | // |**** 39 | // |*** 40 | // |** 41 | // |*** 42 | // |* 43 | // |** 44 | // |** 45 | // |* 46 | // |* 47 | // |* 48 | // |* 49 | // |* 50 | // |* 51 | // | 52 | } 53 | -------------------------------------------------------------------------------- /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: test 2 | on: 3 | push: 4 | pull_request: 5 | 6 | jobs: 7 | test: 8 | strategy: 9 | matrix: 10 | go-version: 11 | - 1.14.x 12 | - 1.15.x 13 | os: 14 | - ubuntu-latest 15 | - macos-latest 16 | - windows-latest 17 | 18 | runs-on: ${{ matrix.os }} 19 | 20 | steps: 21 | - name: Install Go 22 | uses: actions/setup-go@v2 23 | with: 24 | go-version: ${{ matrix.go-version }} 25 | 26 | - name: checkout 27 | uses: actions/checkout@v2 28 | 29 | - name: cache 30 | uses: actions/cache@v2 31 | with: 32 | path: ~/go/pkg/mod 33 | key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} 34 | restore-keys: | 35 | ${{ runner.os }}-go- 36 | 37 | - name: test 38 | run: go test ./... 39 | -------------------------------------------------------------------------------- /bitmap/toarray_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestToArray(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input []int32 15 | wantwords []uint64 16 | }{ 17 | { 18 | []int32{}, 19 | []uint64{}, 20 | }, 21 | { 22 | []int32{0}, 23 | []uint64{1}, 24 | }, 25 | { 26 | []int32{0, 1, 2}, 27 | []uint64{7}, 28 | }, 29 | { 30 | []int32{0, 1, 2, 63}, 31 | []uint64{(1 << 63) + 7}, 32 | }, 33 | { 34 | []int32{64}, 35 | []uint64{0, 1}, 36 | }, 37 | { 38 | []int32{1, 2, 3, 64, 129}, 39 | []uint64{0x0e, 1, 2}, 40 | }, 41 | } 42 | 43 | for i, c := range cases { 44 | 45 | words := Of(c.input) 46 | 47 | ta.Equal(c.wantwords, words, 48 | "%d-th: case: %+v", 49 | i+1, c) 50 | 51 | ta.Equal(c.input, ToArray(words), 52 | "%d-th: case: %+v", 53 | i+1, c) 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /bmtree/partial_tree_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestMult(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | a, b, w uint64 15 | want uint64 16 | }{ 17 | {1, 1, 0, 1}, 18 | {2, 3, 1, 3}, // 10 x 11 19 | {3, 3, 1, 4}, 20 | {5, 3, 2, 3}, // 101 x 011 21 | {5, 5, 2, 6}, // 101 x 101 22 | {0xb9, 0xe2, 7, 0x145}, // 10111001 x 11100010 = 101000101 23 | } 24 | 25 | for i, c := range cases { 26 | got := shiftMulti(c.a, c.b, c.w) 27 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 28 | } 29 | } 30 | 31 | var ( 32 | A, B, C, D uint64 33 | ) 34 | 35 | func BenchmarkMult(b *testing.B) { 36 | 37 | A = 0x112233b9 38 | B = 0x445566e2 39 | B = 0x40000601 40 | // B = 0x10001 41 | C = 32 42 | 43 | var s uint64 44 | for i := 0; i < b.N; i++ { 45 | s += shiftMulti(A, B, C) 46 | } 47 | D = s 48 | } 49 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | 3 | go: 4 | - 1.14.x 5 | - 1.15.x 6 | # - tip 7 | 8 | script: 9 | - make travis 10 | 11 | after_success: 12 | - make coveralls 13 | 14 | # private repo requires a crypted token: 15 | # env: 16 | # global: 17 | # # coveralls token 18 | # secure: fJOPOuwBaZ59iQA1VskxZ3h08Nt5CjGbz5PdrZVT/v5UCG7DOLuVTx3x0Tb+gR9AG9lB8Fqpsnm0jjbBAPvOnyn1KIJDuK9Xj2PvKT78vhJ/SyCnn0BAinmxu9hZqghvyWIzeM8RrA3IrvmnoSUTdE1jnTC7McJ7np6cTRGO9Xe6b4mOO1xQOHJFMyTBFvA84uSKZPbuUHCrh19YH7NKrA4MKunX49R+niEFlFEM4oNM/2FXMca+4+OdlGNJmPkG0kV5exP87ihfqI3Q++9v3Z8SR0KOblL6yRBspaRDHmfKxuGx/YEf71pu0yu7nyT7uVeIABTz5SLrqX2Fhb/cpKb7iqCBQ+ifvgpd86pkfhrPUOsIO9N6pieNxmb+aCNm5WBJ2AaT1zrrfthpfbXvEl66K209rUDL0PV1n/u1pAgY5q7DQD5YuOnyAJNPBNQYYzJnZ+X1GjSNrHKOQPjXmrgwkq7KPVlDoqiaJAh97YwUmjXaULKYOm9JBPwVaToEUeCxzK82ZZRwa4YiYl3MLpJb+SvDMl97hgc58lolfg01wHgLYAT901bbq+qsrQZY4pkW9nDGvBuJg0Mru1bu6hqk/tUA7G4amh2y/5lJxxELednfnyzQ6fBeXKb0FVOTN9xRuFBkpRL1Drmbz3y6J2flAcdpJ4KgAMUP/941J6o= 19 | -------------------------------------------------------------------------------- /sigbits/sigbits.go: -------------------------------------------------------------------------------- 1 | // Package sigbits extracts significant bits from a sorted list of strings. 2 | // Significant bits are the minimal bit set to distinguish all of the strings. 3 | // E.g. we have 3 strings: 4 | // 5 | // ab // bin: 0110 0001 0110 0010 6 | // ac // bin: 0110 0001 0110 0011 7 | // b // bin: 0110 0010 8 | // 9 | // The significant bits are [15, 7], which means the first different bits of 10 | // "ab" and "ac" is the 15-th bit, and the 7-th bit for "ac" and "b". 11 | // 12 | // Since 0.1.9 13 | package sigbits 14 | 15 | // SigBits is a helper to get information about how many bits are required to 16 | // distinguish a set of keys. 17 | // 18 | // Since 0.1.9 19 | type SigBits struct { 20 | keys []string 21 | sigbits []int32 22 | } 23 | 24 | // New creates a SigBits struct. 25 | // 26 | // Since 0.1.9 27 | func New(keys []string) *SigBits { 28 | sb := &SigBits{ 29 | keys: keys, 30 | sigbits: FirstDiffBits(keys), 31 | } 32 | 33 | return sb 34 | } 35 | -------------------------------------------------------------------------------- /docs/badges.md: -------------------------------------------------------------------------------- 1 | [![Travis](https://travis-ci.com/openacid/{{ name }}.svg?branch=main)](https://travis-ci.com/openacid/{{ name }}) 2 | 3 | ![test](https://github.com/openacid/{{ name }}/workflows/test/badge.svg) 4 | 5 | [![Report card](https://goreportcard.com/badge/github.com/openacid/{{ name }})](https://goreportcard.com/report/github.com/openacid/{{ name }}) 6 | [![Coverage Status](https://coveralls.io/repos/github/openacid/{{ name }}/badge.svg?branch=main&service=github)](https://coveralls.io/github/openacid/{{ name }}?branch=main&service=github) 7 | 8 | [![GoDoc](https://godoc.org/github.com/openacid/{{ name }}?status.svg)](http://godoc.org/github.com/openacid/{{ name }}) 9 | [![Sourcegraph](https://sourcegraph.com/github.com/openacid/{{ name }}/-/badge.svg)](https://sourcegraph.com/github.com/openacid/{{ name }}?badge) 10 | -------------------------------------------------------------------------------- /mathext/zipf/zipf_example_rand_test.go: -------------------------------------------------------------------------------- 1 | package zipf 2 | 3 | import ( 4 | "fmt" 5 | "math/rand" 6 | "strings" 7 | ) 8 | 9 | func ExampleZipf() { 10 | 11 | n := 20 12 | 13 | // y = C * x**(1-s), x ∈ [a, b) 14 | a := float64(1) 15 | b := float64(20) 16 | s := float64(1.5) 17 | 18 | z := New(a, b, s) 19 | sampleCnt := float64(100) 20 | 21 | sample := make([]int, n) 22 | r := rand.New(rand.NewSource(44)) 23 | for u := float64(0); u < 1; u += 1 / sampleCnt { 24 | v := r.Float64() 25 | x := int(z.Float64(v)) 26 | sample[x]++ 27 | } 28 | 29 | for _, v := range sample { 30 | fmt.Println("|" + strings.Repeat("*", v)) 31 | } 32 | 33 | // Output: 34 | // 35 | // | 36 | // |******************************* 37 | // |**************** 38 | // |**************** 39 | // |******* 40 | // |** 41 | // |***** 42 | // |*** 43 | // |*** 44 | // |******* 45 | // |* 46 | // |** 47 | // |* 48 | // |*** 49 | // | 50 | // | 51 | // | 52 | // |* 53 | // |* 54 | // |* 55 | } 56 | -------------------------------------------------------------------------------- /bmtree/partial_tree.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "math/bits" 5 | ) 6 | 7 | // shiftMulti maps a tree node to bitmap position. 8 | // The bitmap is with node at some level not stored, and of size 110011... 9 | // A "0" in bitmap size indicates an absent level. 10 | // 11 | // It is defined as: 12 | // sum = 0 13 | // for i = 0; i> h 15 | // } 16 | // 17 | // t[h] t[h-1].. t[1] t[0] 18 | // X p[h-1]...p[1] p[0] 19 | // ------------------------------- 20 | // t[h] t[h-1].. t[1] t[0] x p[0] 21 | // t[h] t[h-1].. t[1] t[0] x p[1] 22 | // ... 23 | // ------------------------------- 24 | // | 25 | // 26 | // Since ??? 27 | func shiftMulti(a, b, shift uint64) uint64 { 28 | 29 | rst := uint64(0) 30 | 31 | n := bits.TrailingZeros64(b) 32 | b >>= uint(n) 33 | shift -= uint64(n) 34 | 35 | for b != 0 { 36 | rst += (a >> shift) 37 | n := bits.TrailingZeros64(b - 1) 38 | b >>= uint(n) 39 | shift -= uint64(n) 40 | } 41 | 42 | return rst 43 | 44 | } 45 | -------------------------------------------------------------------------------- /.github/workflows/golangci-lint.yml: -------------------------------------------------------------------------------- 1 | name: golangci-lint 2 | on: 3 | push: 4 | tags: 5 | - v* 6 | branches: 7 | - '*' 8 | pull_request: 9 | jobs: 10 | golangci: 11 | name: lint 12 | runs-on: ubuntu-latest 13 | steps: 14 | - uses: actions/checkout@v2 15 | 16 | - name: golangci-lint 17 | uses: golangci/golangci-lint-action@v3 18 | with: 19 | # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version. 20 | version: v1.29 21 | 22 | # Optional: working directory, useful for monorepos 23 | # working-directory: somedir 24 | 25 | # disable staticcheck: 26 | # SA1019: package github.com/golang/protobuf/proto is deprecated: Use the "google.golang.org/protobuf/proto" package instead 27 | args: --issues-exit-code=0 --exclude SA1019 28 | 29 | # Optional: show only new issues if it's a pull request. The default value is `false`. 30 | # only-new-issues: true 31 | -------------------------------------------------------------------------------- /bitmap/fmt_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestFmt(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input interface{} 15 | want string 16 | }{ 17 | {int8(7), "11100000"}, 18 | {uint8(7), "11100000"}, 19 | {int16(0x0507), "11100000 10100000"}, 20 | {uint16(0x0507), "11100000 10100000"}, 21 | {int32(0x01030507), "11100000 10100000 11000000 10000000"}, 22 | {uint32(0x01030507), "11100000 10100000 11000000 10000000"}, 23 | {int64(0x0f01030507), "11100000 10100000 11000000 10000000 11110000 00000000 00000000 00000000"}, 24 | {uint64(0x0f01030507), "11100000 10100000 11000000 10000000 11110000 00000000 00000000 00000000"}, 25 | 26 | {[]int8{7}, "11100000"}, 27 | {[]uint16{0x0507, 0x0102}, "11100000 10100000,01000000 10000000"}, 28 | } 29 | 30 | for i, c := range cases { 31 | got := Fmt(c.input) 32 | ta.Equal(c.want, got, 33 | "%d-th: input: %#v; want: %#v; got: %#v", 34 | i+1, c.input, c.want, got) 35 | 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /bitmap/mask.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | var ( 4 | // Mask are pre-calculated width-indexed bit masks. 5 | // E.g. Mask[1] is 63 "0" and 1 "1": 000..01 . 6 | // 7 | // Since 0.1.9 8 | Mask [65]uint64 9 | 10 | // RMask are pre-calculated reverse mask of Mask. 11 | // 12 | // Since 0.1.9 13 | RMask [65]uint64 14 | 15 | // MaskUpto are mask with bits set upto i-th bit(include i-th bit). 16 | // E.g. MaskUpto[1] == Mask[2] == 000..011 . 17 | // 18 | // Since 0.1.9 19 | MaskUpto [64]uint64 20 | 21 | // RMaskUpto are reverse of MaskUpto. 22 | // 23 | // Since 0.1.9 24 | RMaskUpto [64]uint64 25 | 26 | // Bit set i-th bit to 1. 27 | // 28 | // Since 0.1.9 29 | Bit [64]uint64 30 | 31 | // RBit are reverse of Bit. 32 | // 33 | // Since 0.1.9 34 | RBit [64]uint64 35 | ) 36 | 37 | func initMasks() { 38 | for i := 0; i < 65; i++ { 39 | Mask[i] = (1 << uint(i)) - 1 40 | RMask[i] = ^Mask[i] 41 | } 42 | 43 | for i := 0; i < 64; i++ { 44 | MaskUpto[i] = (1 << uint(i+1)) - 1 45 | RMaskUpto[i] = ^MaskUpto[i] 46 | Bit[i] = 1 << uint(i) 47 | RBit[i] = ^Bit[i] 48 | } 49 | } 50 | -------------------------------------------------------------------------------- /bitmap/next_bench_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import "testing" 4 | 5 | var OutputNextOne int 6 | 7 | func BenchmarkNextOne(b *testing.B) { 8 | bm := []uint64{0xff, 0xff, 0x33, 0x66} 9 | 10 | var s int32 11 | 12 | b.Run("FoundInCurrentWord", func(b *testing.B) { 13 | for i := 0; i < b.N; i++ { 14 | s += NextOne(bm, 0, int32(i&0x7)) 15 | } 16 | OutputNextOne = int(s) 17 | }) 18 | 19 | bm = []uint64{0, 0, 0, 0x66} 20 | 21 | b.Run("EnumerateAllBits", func(b *testing.B) { 22 | for i := 0; i < b.N; i++ { 23 | s += NextOne(bm, 1, 64*4) 24 | } 25 | OutputNextOne = int(s) 26 | }) 27 | } 28 | 29 | func BenchmarkPrevOne(b *testing.B) { 30 | bm := []uint64{0xff, 0xff, 0x33, 0x66} 31 | 32 | var s int32 33 | 34 | b.Run("FoundInCurrentWord", func(b *testing.B) { 35 | for i := 0; i < b.N; i++ { 36 | s += PrevOne(bm, 0, int32(i&0x7)+1) 37 | } 38 | OutputNextOne = int(s) 39 | }) 40 | 41 | bm = []uint64{3, 0, 0, 0} 42 | 43 | b.Run("EnumerateAllBits", func(b *testing.B) { 44 | for i := 0; i < b.N; i++ { 45 | s += PrevOne(bm, 1, 64*4) 46 | } 47 | OutputNextOne = int(s) 48 | }) 49 | } 50 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 openacid 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /bmtree/newpath.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import "github.com/openacid/low/bitmap" 4 | 5 | // NewPath creates a path, which is a uint64, from node searching path, path 6 | // length and tree height. 7 | // 8 | // Since 0.1.9 9 | func NewPath(searchingBits uint64, length, height int32) uint64 { 10 | return (searchingBits << 32) | (bitmap.Mask[length] << uint(height-length)) 11 | } 12 | 13 | // PathOf creates a path, which is a uint64, from a string and sepcified 14 | // starting bit position and height. 15 | // 16 | // Since 0.1.9 17 | func PathOf(s string, frombit int32, height int32) uint64 { 18 | plen, path := bitmap.FromStr32(s, frombit, frombit+height) 19 | return NewPath(path, plen, height) 20 | } 21 | 22 | // PathsOf creates more than one node search paths at a time. 23 | // 24 | // Since 0.1.9 25 | func PathsOf(keys []string, frombit int32, height int32, dedup bool) []uint64 { 26 | l := len(keys) 27 | rst := make([]uint64, 0, l) 28 | prev := ^uint64(0) 29 | for _, s := range keys { 30 | p := PathOf(s, frombit, height) 31 | if !dedup || p != prev { 32 | rst = append(rst, p) 33 | } 34 | prev = p 35 | } 36 | return rst 37 | } 38 | -------------------------------------------------------------------------------- /sigbits/countprefixes.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | // countPrefixes counts the number of n-bit prefixes: 4 | // 5 | // It returns the minimal bit index where there is a different bit 6 | // and a []int32 of length maxitem + 1. 7 | // The ith element represents the number of distinct i-bits words. 8 | // E.g. 9 | // There is only 1 0-bit word. 10 | // There are at most 2 1-bit words. 11 | // 12 | // The first argument is made by FirstDiffBits(keys). 13 | // 14 | // Since 0.1.9 15 | func countPrefixes(firstdiffs []int32, maxitem int32) (int32, []int32) { 16 | 17 | min := int32(0x7fffffff) 18 | for _, d := range firstdiffs { 19 | if min > d { 20 | min = d 21 | } 22 | } 23 | 24 | // counts[i] is number of i-th bits that is the first diff bit. 25 | counts := make([]int32, maxitem-1) 26 | for _, d := range firstdiffs { 27 | d -= min 28 | if d < maxitem-1 { 29 | counts[d]++ 30 | } 31 | } 32 | 33 | // rst[i] means how many distinct i-bit words there are there are 34 | rst := make([]int32, maxitem) 35 | rst[0] = 1 36 | for i := int32(0); i < maxitem-1; i++ { 37 | rst[i+1] = rst[i] + counts[i] 38 | } 39 | 40 | return min, rst 41 | } 42 | -------------------------------------------------------------------------------- /bmtree/allpaths.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "math/bits" 5 | 6 | "github.com/openacid/low/bitmap" 7 | ) 8 | 9 | // AllPaths returns all possible searching path a bitmap can have. 10 | // 11 | // Since 0.1.9 12 | func AllPaths(bitmapSize int32, from, to uint64) []uint64 { 13 | 14 | height := Height(bitmapSize) 15 | 16 | paths := make([]uint64, 0) 17 | 18 | fullPathCnt := bitmap.Bit[height] 19 | fullPathMask := bitmap.Mask[height] 20 | 21 | var t uint64 22 | t = to>>32 + 1 23 | 24 | if t > fullPathCnt { 25 | t = fullPathCnt 26 | } 27 | 28 | for i := from >> 32; i < t; i++ { 29 | tz := int32(bits.TrailingZeros64(i)) 30 | if tz > height { 31 | tz = height 32 | } 33 | 34 | for ; tz >= 0; tz-- { 35 | if bitmapSize&int32(bitmap.Bit[height-tz]) == 0 { 36 | continue 37 | } 38 | m := bitmap.Mask[tz] 39 | p := (i << 32) | (fullPathMask ^ m) 40 | if p < from { 41 | // i is full searching bits without length, that it may generate 42 | // smaller path. 43 | continue 44 | } 45 | if p >= to { 46 | return paths 47 | } 48 | paths = append(paths, p) 49 | } 50 | } 51 | 52 | return paths 53 | } 54 | -------------------------------------------------------------------------------- /bitmap/fromstr32.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // FromStr32 returns a bit array from string and put them in the least 4 | // significant "to-from" bits of a uint64. 5 | // "to-from" must be smaller than or equal 32. 6 | // 7 | // It returns actual number of bits used from the string, and a uint64. 8 | // 9 | // Since 0.1.9 10 | func FromStr32(s string, frombit, tobit int32) (int32, uint64) { 11 | 12 | size := tobit - frombit 13 | spanSize := tobit - (frombit & ^7) 14 | 15 | blen := int32(len(s)<<3) - frombit 16 | 17 | if blen > size { 18 | blen = size 19 | } 20 | 21 | if blen <= 0 { 22 | return 0, 0 23 | } 24 | 25 | l := int32(len(s)) 26 | toByte := (tobit + 7) >> 3 27 | if l > toByte { 28 | l = toByte 29 | } 30 | 31 | i := frombit >> 3 32 | b := uint64(0) 33 | 34 | if i < l { 35 | b |= uint64(s[i]) << 32 36 | i++ 37 | if i < l { 38 | b |= uint64(s[i]) << 24 39 | i++ 40 | if i < l { 41 | b |= uint64(s[i]) << 16 42 | i++ 43 | if i < l { 44 | b |= uint64(s[i]) << 8 45 | i++ 46 | if i < l { 47 | b |= uint64(s[i]) 48 | } 49 | } 50 | } 51 | } 52 | } 53 | 54 | return blen, (b >> uint(40-spanSize)) & Mask[size] 55 | } 56 | -------------------------------------------------------------------------------- /bitmap/README.md: -------------------------------------------------------------------------------- 1 | # bitmap - bitmap operation creation, query, rank/select impl 2 | 3 | A bitmap is a `[]uint64`. 4 | 5 | 6 | ``` 7 | BenchmarkFromStr32-4 147230166 8.07 ns/op 8 | BenchmarkStringToBytes-4 201391240 5.95 ns/op 9 | BenchmarkNextOne/FoundInCurrentWord-4 334609323 3.56 ns/op 10 | BenchmarkNextOne/EnumerateAllBits-4 294496100 4.03 ns/op 11 | BenchmarkPrevOne/FoundInCurrentWord-4 334022793 3.59 ns/op 12 | BenchmarkPrevOne/EnumerateAllBits-4 286702246 4.12 ns/op 13 | BenchmarkRank64_5_bits-4 672754074 1.77 ns/op 14 | BenchmarkRank128_5_bits-4 486679581 2.49 ns/op 15 | BenchmarkRank64_64k_bits-4 881509602 1.35 ns/op 16 | BenchmarkRank128_64k_bits-4 424685930 2.84 ns/op 17 | BenchmarkSelect-4 70999674 16.1 ns/op 18 | BenchmarkSelect32-4 141631495 8.54 ns/op 19 | BenchmarkSelect32R64-4 153702387 7.85 ns/op 20 | BenchmarkSelectU64Indexed-4 365140676 3.29 ns/op 21 | ``` 22 | -------------------------------------------------------------------------------- /bmtree/bitmappath_check_test.go: -------------------------------------------------------------------------------- 1 | // +build debug 2 | 3 | package bmtree 4 | 5 | import ( 6 | "testing" 7 | 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestBitmapPathMustHaveEqualHeight(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | cases := []struct { 16 | bitmapSize int32 17 | path uint64 18 | wantpanic bool 19 | }{ 20 | 21 | // path is too long 22 | {5, 4, true}, 23 | 24 | {7, 4, true}, 25 | {7, 5, true}, 26 | {7, 6, true}, 27 | {7, 7, true}, 28 | {7, 8, true}, 29 | 30 | // path is too short 31 | {7, 1, true}, 32 | 33 | // path must be consecutive "1"s 34 | {0xf, 5, true}, 35 | 36 | // path mask shorter than path bits 37 | {0xf, 0xf<<32 + 0xe, true}, 38 | 39 | // bitmap height <=30 40 | {-1, 0, true}, 41 | 42 | // path height <=30 43 | {1, 1 << 31, true}, 44 | {1, 1 << 30, true}, 45 | 46 | // path points to present level 47 | {7, 0, false}, 48 | {7, 2, false}, 49 | {7, 3, false}, 50 | 51 | {5, 0, false}, 52 | {5, 3, false}, 53 | } 54 | 55 | for _, c := range cases { 56 | if c.wantpanic { 57 | ta.Panics(func() { bitmapPathMustHaveEqualHeight(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 58 | } else { 59 | ta.NotPanics(func() { bitmapPathMustHaveEqualHeight(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 60 | 61 | } 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /bitmap/get.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // Get returns a uint64 with the (i%64)-th bit set. 4 | // 5 | // Since 0.1.9 6 | func Get(bm []uint64, i int32) uint64 { 7 | return (bm[i>>6] & Bit[i&63]) 8 | } 9 | 10 | // Get1 returns a uint64 with the least significant bit set if (i%64)-th bit is 11 | // set. 12 | // 13 | // Since 0.1.9 14 | func Get1(bm []uint64, i int32) uint64 { 15 | return (bm[i>>6] >> uint(i&63)) & 1 16 | } 17 | 18 | // Getw returns the i-th w-bit word in the least significant w bits of a 19 | // uint64. 20 | // "w" must be one of 1, 2, 4, 8, 16, 32 and 64. 21 | // 22 | // Since 0.1.9 23 | func Getw(bm []uint64, i int32, w int32) uint64 { 24 | i *= w 25 | return (bm[i>>6] >> uint(i&63)) & Mask[w] 26 | } 27 | 28 | // SafeGet is same as Get() except it return 0 instead of a panic when index out 29 | // of boundary. 30 | // 31 | // Since 0.1.9 32 | func SafeGet(bm []uint64, i int32) uint64 { 33 | wordI := i >> 6 34 | bitI := i & 63 35 | if wordI < 0 || wordI >= int32(len(bm)) { 36 | return 0 37 | } 38 | return (bm[wordI] & Bit[bitI]) 39 | } 40 | 41 | // SafeGet1 is same as Get1() except it return 0 instead of a panic when index 42 | // out of boundary. 43 | // 44 | // Since 0.1.9 45 | func SafeGet1(bm []uint64, i int32) uint64 { 46 | wordI := i >> 6 47 | bitI := i & 63 48 | if wordI < 0 || wordI >= int32(len(bm)) { 49 | return 0 50 | } 51 | return (bm[wordI] >> uint(bitI)) & 1 52 | } 53 | -------------------------------------------------------------------------------- /bitmap/select_bench_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | var InputI32 int32 = 35 8 | var OutputI32 int32 = 0 9 | 10 | func BenchmarkSelect(b *testing.B) { 11 | words := []uint64{0xffffffff, 0xffffffff, 1} 12 | idx := IndexSelect32(words) 13 | var s int32 14 | b.ResetTimer() 15 | for i := 0; i < b.N; i++ { 16 | s += select32single(words, idx, InputI32) 17 | s += select32single(words, idx, InputI32+1) 18 | } 19 | OutputI32 = s 20 | } 21 | 22 | func BenchmarkSelect32(b *testing.B) { 23 | words := []uint64{0xffffffff, 0xffffffff, 1} 24 | idx := IndexSelect32(words) 25 | var s int32 26 | b.ResetTimer() 27 | for i := 0; i < b.N; i++ { 28 | v1, _ := Select32(words, idx, InputI32) 29 | s += v1 30 | } 31 | OutputI32 = s 32 | } 33 | 34 | func BenchmarkSelect32R64(b *testing.B) { 35 | words := []uint64{0xffffffff, 0xffffffff, 1} 36 | sidx, ridx := IndexSelect32R64(words) 37 | var s int32 38 | b.ResetTimer() 39 | for i := 0; i < b.N; i++ { 40 | v1, _ := Select32R64(words, sidx, ridx, InputI32) 41 | s += v1 42 | } 43 | OutputI32 = s 44 | } 45 | 46 | func BenchmarkSelectU64Indexed(b *testing.B) { 47 | 48 | word := uint64(0xffffffffffffffff) 49 | 50 | idx := indexSelectU64(word) 51 | 52 | var s int32 53 | 54 | b.ResetTimer() 55 | 56 | for i := 0; i < b.N; i++ { 57 | v1, _ := selectU64Indexed(word, idx, uint64(InputI32)) 58 | s += v1 59 | } 60 | OutputI32 = s 61 | } 62 | -------------------------------------------------------------------------------- /bitmap/fmt.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "fmt" 5 | "math/bits" 6 | "reflect" 7 | "strings" 8 | ) 9 | 10 | // Fmt converts bitmap in an integer or in a slice of integer to binary format string, 11 | // with significant bits at right. 12 | // E.g.: 13 | // int32(0x0102) --> 01000000 10000000 14 | // 15 | // Since 0.1.11 16 | func Fmt(x interface{}) string { 17 | 18 | rst := []string{} 19 | 20 | v := reflect.ValueOf(x) 21 | if v.Kind() == reflect.Slice { 22 | n := v.Len() 23 | for i := 0; i < n; i++ { 24 | rst = append(rst, intFmt(v.Index(i).Interface())) 25 | } 26 | return strings.Join(rst, ",") 27 | 28 | } else { 29 | return intFmt(x) 30 | } 31 | } 32 | 33 | func intFmt(i interface{}) string { 34 | 35 | sz, v := intSize(i) 36 | 37 | rst := []string{} 38 | for i := 0; i < sz; i++ { 39 | b := uint8(v >> uint(i*8)) 40 | s := fmt.Sprintf("%08b", bits.Reverse8(b)) 41 | rst = append(rst, s) 42 | } 43 | return strings.Join(rst, " ") 44 | } 45 | 46 | func intSize(i interface{}) (int, uint64) { 47 | switch i := i.(type) { 48 | case int8: 49 | return 1, uint64(i) 50 | case uint8: 51 | return 1, uint64(i) 52 | case int16: 53 | return 2, uint64(i) 54 | case uint16: 55 | return 2, uint64(i) 56 | case int32: 57 | return 4, uint64(i) 58 | case uint32: 59 | return 4, uint64(i) 60 | case int64: 61 | return 8, uint64(i) 62 | case uint64: 63 | return 8, i 64 | } 65 | 66 | panic(fmt.Sprintf("not a int type: %v", i)) 67 | } 68 | -------------------------------------------------------------------------------- /bitmap/next.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "math/bits" 5 | ) 6 | 7 | // NextOne find the next "1" in range `[i, end)`. 8 | // If there is no "1" found, it returns `end`. 9 | // 10 | // Since 0.1.11 11 | func NextOne(bm []uint64, i, end int32) int32 { 12 | 13 | wordIdx := i >> 6 14 | bitIdx := i & 63 15 | var nxt int32 = -1 16 | 17 | word := bm[wordIdx] & RMask[bitIdx] 18 | if word != 0 { 19 | nxt = wordIdx<<6 + int32(bits.TrailingZeros64(word)) 20 | } else { 21 | 22 | i = (i + 63) & ^63 23 | 24 | for ; i < end; i += 64 { 25 | 26 | word := bm[i>>6] 27 | if word != 0 { 28 | nxt = i + int32(bits.TrailingZeros64(word)) 29 | break 30 | } 31 | } 32 | } 33 | if nxt >= end { 34 | return -1 35 | } 36 | return nxt 37 | } 38 | 39 | // PrevOne find the previous "1" in range `[i, end)` 40 | // If there is no "1" found, it returns -1. 41 | // 42 | // Since 0.1.11 43 | func PrevOne(bm []uint64, i, end int32) int32 { 44 | 45 | end-- 46 | wordIdx := end >> 6 47 | bitIdx := end & 63 48 | var prv int32 = -1 49 | 50 | word := bm[wordIdx] & MaskUpto[bitIdx] 51 | if word != 0 { 52 | prv = wordIdx<<6 + 63 - int32(bits.LeadingZeros64(word)) 53 | } else { 54 | end = (end & ^63) - 1 55 | 56 | for ; end >= i; end -= 64 { 57 | 58 | word := bm[end>>6] 59 | if word != 0 { 60 | prv = end - int32(bits.LeadingZeros64(word)) 61 | break 62 | } 63 | } 64 | } 65 | 66 | if prv < i { 67 | return -1 68 | } 69 | 70 | return prv 71 | } 72 | -------------------------------------------------------------------------------- /vers/vers.go: -------------------------------------------------------------------------------- 1 | // Package vers provides version checking functionalities. 2 | package vers 3 | 4 | import ( 5 | "strings" 6 | 7 | "github.com/blang/semver" 8 | "github.com/openacid/must" 9 | ) 10 | 11 | // VersionGetter defines GetVersion() 12 | // 13 | // Since 0.1.7 14 | type VersionGetter interface { 15 | GetVersion() string 16 | } 17 | 18 | // IsCompatible checks if a verion "ver" satisfies semantic version spec. 19 | // 20 | // `>1.0.0 <2.0.0 || >3.0.0 !4.2.1` would match `1.2.3`, `1.9.9`, `3.1.1`. 21 | // Not `4.2.1`, `2.1.1` 22 | // 23 | // Deprecated: Should use Check() which would panic if version is invalid. 24 | // 25 | // Since 0.1.7 26 | func IsCompatible(ver string, spec []string) bool { 27 | 28 | sp := strings.Join(spec, " || ") 29 | 30 | v, err := semver.Parse(ver) 31 | if err != nil { 32 | return false 33 | } 34 | 35 | chk, err := semver.ParseRange(sp) 36 | if err != nil { 37 | return false 38 | } 39 | 40 | return chk(v) 41 | } 42 | 43 | // Check checks if a verion "ver" satisfies any of the semantic versions in "spec". 44 | // 45 | // `>1.0.0 <2.0.0 || >3.0.0 !4.2.1` would match `1.2.3`, `1.9.9`, `3.1.1`. 46 | // Not `4.2.1`, `2.1.1` 47 | // 48 | // Since 0.1.9 49 | func Check(ver string, spec ...string) bool { 50 | 51 | sp := strings.Join(spec, " || ") 52 | 53 | v, err := semver.Parse(ver) 54 | must.Be.NoError(err, "ver must be valid but: %q", ver) 55 | 56 | chk, err := semver.ParseRange(sp) 57 | must.Be.NoError(err, "spec must be valid but: %q", spec) 58 | 59 | return chk(v) 60 | } 61 | -------------------------------------------------------------------------------- /bmtree/bitmap_check_test.go: -------------------------------------------------------------------------------- 1 | // +build debug 2 | 3 | package bmtree 4 | 5 | import ( 6 | "testing" 7 | 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestBitmapSizeCheck(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | cases := []struct { 16 | bitmapSize int32 17 | wantpanic bool 18 | }{ 19 | 20 | // bitmap height <=30 21 | {-1, true}, 22 | 23 | // bitmap must not be 0 24 | {0, true}, 25 | 26 | // valid bitmapSize 27 | {0x01, false}, 28 | {0x02, false}, 29 | {0x03, false}, 30 | {0x04, false}, 31 | {0x05, false}, 32 | {0x06, false}, 33 | {0x07, false}, 34 | {0x7fffffff, false}, 35 | } 36 | 37 | for _, c := range cases { 38 | if c.wantpanic { 39 | ta.Panics(func() { bitmapSizeCheck(c.bitmapSize) }, "%032b", c.bitmapSize) 40 | } else { 41 | ta.NotPanics(func() { bitmapSizeCheck(c.bitmapSize) }, "%032b", c.bitmapSize) 42 | 43 | } 44 | } 45 | 46 | } 47 | 48 | func TestBitmapMustHaveLevel(t *testing.T) { 49 | 50 | ta := require.New(t) 51 | 52 | cases := []struct { 53 | bitmapSize int32 54 | level int32 55 | wantpanic bool 56 | }{ 57 | 58 | {0x0d, 0, false}, 59 | {0x0d, 1, true}, 60 | {0x0d, 2, false}, 61 | {0x0d, 3, false}, 62 | {0x0d, 4, true}, 63 | } 64 | 65 | for _, c := range cases { 66 | if c.wantpanic { 67 | ta.Panics(func() { bitmapMustHaveLevel(c.bitmapSize, c.level) }, "%032b, %d", c.bitmapSize, c.level) 68 | } else { 69 | ta.NotPanics(func() { bitmapMustHaveLevel(c.bitmapSize, c.level) }, "%032b %d", c.bitmapSize, c.level) 70 | 71 | } 72 | } 73 | 74 | } 75 | -------------------------------------------------------------------------------- /bitmap/builder.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // Builder provides continous bitmap building. 4 | // 5 | // Since 0.1.19 6 | type Builder struct { 7 | Words []uint64 8 | Offset int32 9 | } 10 | 11 | // NewBuilder creates a new Builder with preallocated n bits. 12 | // 13 | // Since 0.1.19 14 | func NewBuilder(n int32) *Builder { 15 | b := &Builder{ 16 | Words: make([]uint64, 0, n>>6), 17 | Offset: 0, 18 | } 19 | 20 | return b 21 | } 22 | 23 | // Extend add bits into builder, the bit position are relative to current 24 | // Builder.Offset and the size in bit of the bitmap is size. 25 | // 26 | // Since 0.1.19 27 | func (b *Builder) Extend(bitPositions []int32, size int32) { 28 | end := b.Offset + size 29 | if len(bitPositions) > 0 { 30 | bitEnd := bitPositions[len(bitPositions)-1] 31 | if bitEnd >= size { 32 | end = b.Offset + bitEnd + 1 33 | } 34 | } 35 | for int(end) > len(b.Words)<<6 { 36 | b.Words = append(b.Words, 0) 37 | } 38 | 39 | for _, i := range bitPositions { 40 | idx := b.Offset + i 41 | b.Words[idx>>6] |= 1 << uint(idx&63) 42 | } 43 | 44 | b.Offset += size 45 | } 46 | 47 | // Set a bit to `value` in the bitmap builder. 48 | // Builder.Offset is updated to the last set bit if it is smaller. 49 | // 50 | // Since 0.1.19 51 | func (b *Builder) Set(bitPosition int32, value int32) { 52 | 53 | for int(bitPosition>>6) >= len(b.Words) { 54 | b.Words = append(b.Words, 0) 55 | } 56 | 57 | b.Words[bitPosition>>6] |= uint64(value&1) << uint(bitPosition&63) 58 | 59 | // update to next unused bit 60 | if b.Offset <= bitPosition { 61 | b.Offset = bitPosition + 1 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /bitmap/slice_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestSlice(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | ta.Panics(func() { Slice(Of([]int32{0}), 0, 65) }) 14 | ta.Panics(func() { Slice(Of([]int32{0}), 64, 65) }) 15 | 16 | cases := []struct { 17 | input []int32 18 | from, to int32 19 | wantarr []int32 20 | }{ 21 | { 22 | []int32{}, 23 | 0, 0, 24 | []int32{}, 25 | }, 26 | { 27 | []int32{0}, 28 | 0, 0, 29 | []int32{}, 30 | }, 31 | { 32 | []int32{0}, 33 | 0, 1, 34 | []int32{0}, 35 | }, 36 | { 37 | []int32{0}, 38 | 0, 2, 39 | []int32{0}, 40 | }, 41 | { 42 | []int32{0}, 43 | 0, 64, 44 | []int32{0}, 45 | }, 46 | { 47 | []int32{0, 1, 2}, 48 | 0, 3, 49 | []int32{0, 1, 2}, 50 | }, 51 | { 52 | []int32{0, 1, 2}, 53 | 0, 64, 54 | []int32{0, 1, 2}, 55 | }, 56 | { 57 | []int32{0, 1, 2}, 58 | 1, 64, 59 | []int32{0, 1}, 60 | }, 61 | { 62 | []int32{0, 1, 2}, 63 | 2, 64, 64 | []int32{0}, 65 | }, 66 | { 67 | []int32{0, 1, 2}, 68 | 3, 64, 69 | []int32{}, 70 | }, 71 | { 72 | []int32{0, 1, 2}, 73 | 4, 64, 74 | []int32{}, 75 | }, 76 | { 77 | []int32{64, 66}, 78 | 63, 67, 79 | []int32{1, 3}, 80 | }, 81 | { 82 | []int32{64, 66}, 83 | 64, 67, 84 | []int32{0, 2}, 85 | }, 86 | } 87 | 88 | for i, c := range cases { 89 | 90 | words := Of(c.input) 91 | sl := Slice(words, c.from, c.to) 92 | gotarr := ToArray(sl) 93 | 94 | ta.Equal(c.wantarr, gotarr, "%d-th: case: %+v", i+1, c) 95 | } 96 | } 97 | -------------------------------------------------------------------------------- /bmtree/decode_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/openacid/low/bitmap" 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestDecode(t *testing.T) { 11 | 12 | ta := require.New(t) 13 | 14 | cases := []struct { 15 | bitmapSize int32 16 | bitmapIndex []int32 17 | want []string 18 | }{ 19 | {0x1, []int32{}, []string{}}, 20 | {0x3, []int32{}, []string{}}, 21 | {0x3, []int32{0}, []string{""}}, 22 | {0x3, []int32{1}, []string{"0"}}, 23 | {0x3, []int32{1, 2}, []string{"0", "1"}}, 24 | {0x7, []int32{1, 2, 3, 6}, []string{"0", "00", "01", "11"}}, 25 | {0xd, []int32{0, 1, 3, 4, 9, 12}, []string{"", "00", "001", "01", "101", "111"}}, 26 | 27 | {0xf08, []int32{0, 63, 64, 65}, []string{"000", "0000010000", "00000100000", "00000100001"}}, 28 | } 29 | 30 | for i, c := range cases { 31 | bm := bitmap.Of(c.bitmapIndex) 32 | ps := Decode(c.bitmapSize, bm) 33 | got := []string{} 34 | for _, p := range ps { 35 | got = append(got, PathStr(p)) 36 | } 37 | 38 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 39 | } 40 | } 41 | 42 | func TestDecode_fulltree_height_7(t *testing.T) { 43 | 44 | ta := require.New(t) 45 | 46 | bmidx := []int32{ 47 | 0x01, 48 | 0x0f, 49 | 0x1a, 50 | 0x30, 51 | 0x5c, 52 | 0x65, 53 | 0x89, 54 | 0xc6, 55 | } 56 | 57 | paths := []uint64{ 58 | 0x0000000000000040, 59 | 0x000000050000007f, 60 | 0x0000000b0000007f, 61 | 0x000000160000007f, 62 | 0x0000002c0000007f, 63 | 0x000000300000007f, 64 | 0x000000420000007f, 65 | 0x000000610000007f, 66 | } 67 | 68 | got := Decode(1<<(7+1)-1, bitmap.Of(bmidx)) 69 | ta.Equal(paths, got) 70 | } 71 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | # Description 2 | 3 | 7 | 8 | Fixes # (issue) 9 | 10 | ## Type of change 11 | 12 | 13 | 14 | - **Bug fix** 15 | - **New feature** 16 | - **Breaking change** 17 | - **Refactoring** 18 | - **Document changes** 19 | - **Test changes** 20 | 21 | 22 | ## How to reproduce it (if it is a bug-fix PR) 23 | 24 | - Env: x86-64, CentOS-7.4, kernel-3.10.0, GO-1.10.1, etc. 25 | 26 | - Step-1: 27 | - Step-2: 28 | 29 | 30 | ## The solution (to fix a bug, implement a new feature etc.) 31 | 32 | 33 | 34 | # Checklist: 35 | 36 | - [ ] **Style**: My code follows the **style guidelines** of this project 37 | - [ ] **Self-review**: I have performed a **self-review** of my own code 38 | - [ ] **Comment**: I have **commented my code**, particularly in hard-to-understand areas 39 | - [ ] **Doc**: I have made corresponding changes to the **documentation** 40 | - [ ] **No-warnings**: My changes generate **no new warnings** 41 | - [ ] **Add-test**: I have added **tests** that prove my fix is effective or that my feature works 42 | - [ ] **Pass**: New and existing **unit tests pass** locally with my changes 43 | - [ ] **Dep**: Any **dependent** changes have been merged and published in downstream modules 44 | -------------------------------------------------------------------------------- /bitmap/rank_bench_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import "testing" 4 | 5 | var DsOutput int 6 | 7 | func BenchmarkRank64_5_bits(b *testing.B) { 8 | 9 | bm := Of([]int32{1, 2, 3, 64, 129}) 10 | idx := IndexRank64(bm) 11 | 12 | b.ResetTimer() 13 | 14 | var gotrank int32 15 | for i := 0; i < b.N; i++ { 16 | upto, _ := Rank64(bm, idx, int32(i&127)) 17 | gotrank += upto 18 | } 19 | 20 | DsOutput = int(gotrank) 21 | } 22 | 23 | func BenchmarkRank128_5_bits(b *testing.B) { 24 | 25 | bm := Of([]int32{1, 2, 3, 64, 129}) 26 | idx := IndexRank128(bm) 27 | 28 | b.ResetTimer() 29 | 30 | var gotrank int32 31 | for i := 0; i < b.N; i++ { 32 | upto, _ := Rank128(bm, idx, int32(i&127)) 33 | gotrank += upto 34 | } 35 | 36 | DsOutput = int(gotrank) 37 | } 38 | 39 | func BenchmarkRank64_64k_bits(b *testing.B) { 40 | 41 | n := 64 * 1024 42 | mask := n - 1 43 | indexes := make([]int32, n) 44 | for i := 0; i < n; i++ { 45 | indexes[i] = int32(i * 2) 46 | } 47 | bm := Of(indexes) 48 | idx := IndexRank64(bm) 49 | 50 | b.ResetTimer() 51 | 52 | var gotrank int32 53 | for i := 0; i < b.N; i++ { 54 | upto, _ := Rank64(bm, idx, int32(i&mask)) 55 | gotrank += upto 56 | } 57 | 58 | DsOutput = int(gotrank) 59 | } 60 | 61 | func BenchmarkRank128_64k_bits(b *testing.B) { 62 | 63 | n := 64 * 1024 64 | mask := n - 1 65 | indexes := make([]int32, n) 66 | for i := 0; i < n; i++ { 67 | indexes[i] = int32(i * 2) 68 | } 69 | bm := Of(indexes) 70 | idx := IndexRank128(bm) 71 | 72 | b.ResetTimer() 73 | 74 | var gotrank int32 75 | for i := 0; i < b.N; i++ { 76 | upto, _ := Rank128(bm, idx, int32(i&mask)) 77 | gotrank += upto 78 | } 79 | 80 | DsOutput = int(gotrank) 81 | } 82 | -------------------------------------------------------------------------------- /sigbits/sharding.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | // ShardByPrefix splits a sorted string slice into shards each of which has at most maxSize elts. 4 | // 5 | // The rule is that every shard has a unique prefix. All prefixes are still a 6 | // sorted string slice. 7 | // 8 | // It returns a slice of prefix lengths in bytes of each shard, and a slice of 9 | // the starting index of each shard. 10 | // 11 | // Since 0.1.20 12 | func ShardByPrefix(keys []string, maxSize int32) ([]int32, []int32) { 13 | 14 | firstDiffs := FirstDiffBits(keys) 15 | 16 | n := int32(len(firstDiffs) + 1) 17 | 18 | prefixes := make([]int32, 0, len(keys)/32) 19 | keyCnts := make([]int32, 0, len(keys)/32) 20 | keyCnts = append(keyCnts, 0) 21 | 22 | var dfs func(s, e int32) 23 | 24 | dfs = func(s, e int32) { 25 | if e-s <= maxSize { 26 | 27 | // find the longest prefix: 28 | min := int32(len(keys[s])) 29 | for i := s; i < e-1; i++ { 30 | if min > firstDiffs[i]>>3 { 31 | min = firstDiffs[i] >> 3 32 | } 33 | } 34 | 35 | prefixes = append(prefixes, min) 36 | keyCnts = append(keyCnts, e) 37 | return 38 | } 39 | 40 | // range is too large, split it 41 | 42 | longest := int32(len(keys[s])) 43 | 44 | // endsAt collects where to split current range 45 | endsAt := make([]int32, 0, 256) 46 | 47 | for i := s; i < e-1; i++ { 48 | prefixLen := firstDiffs[i] >> 3 49 | 50 | if prefixLen < longest { 51 | longest = prefixLen 52 | endsAt = endsAt[0:0] 53 | endsAt = append(endsAt, i+1) 54 | } else if prefixLen == longest { 55 | endsAt = append(endsAt, i+1) 56 | } 57 | } 58 | 59 | endsAt = append(endsAt, e) 60 | 61 | for i := 0; i < len(endsAt); i++ { 62 | end := endsAt[i] 63 | dfs(s, end) 64 | s = end 65 | } 66 | } 67 | 68 | dfs(0, n) 69 | 70 | return prefixes, keyCnts 71 | } 72 | -------------------------------------------------------------------------------- /mathext/util/util_test.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestMinMaxClap(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | ta.Equal(int(1), MinI(1, 2)) 14 | ta.Equal(int8(1), MinI8(1, 2)) 15 | ta.Equal(int16(1), MinI16(1, 2)) 16 | ta.Equal(int32(1), MinI32(1, 2)) 17 | ta.Equal(int64(1), MinI64(1, 2)) 18 | 19 | ta.Equal(uint(1), MinU(1, 2)) 20 | ta.Equal(uint8(1), MinU8(1, 2)) 21 | ta.Equal(uint16(1), MinU16(1, 2)) 22 | ta.Equal(uint32(1), MinU32(1, 2)) 23 | ta.Equal(uint64(1), MinU64(1, 2)) 24 | 25 | ta.Equal(int(2), MaxI(1, 2)) 26 | ta.Equal(int8(2), MaxI8(1, 2)) 27 | ta.Equal(int16(2), MaxI16(1, 2)) 28 | ta.Equal(int32(2), MaxI32(1, 2)) 29 | ta.Equal(int64(2), MaxI64(1, 2)) 30 | 31 | ta.Equal(uint(2), MaxU(1, 2)) 32 | ta.Equal(uint8(2), MaxU8(1, 2)) 33 | ta.Equal(uint16(2), MaxU16(1, 2)) 34 | ta.Equal(uint32(2), MaxU32(1, 2)) 35 | ta.Equal(uint64(2), MaxU64(1, 2)) 36 | 37 | ta.Equal(int(2), ClapI(1, 2, 3)) 38 | ta.Equal(int(3), ClapI(4, 2, 3)) 39 | 40 | ta.Equal(int8(2), ClapI8(1, 2, 3)) 41 | ta.Equal(int8(3), ClapI8(4, 2, 3)) 42 | 43 | ta.Equal(int16(2), ClapI16(1, 2, 3)) 44 | ta.Equal(int16(3), ClapI16(4, 2, 3)) 45 | 46 | ta.Equal(int32(2), ClapI32(1, 2, 3)) 47 | ta.Equal(int32(3), ClapI32(4, 2, 3)) 48 | 49 | ta.Equal(int64(2), ClapI64(1, 2, 3)) 50 | ta.Equal(int64(3), ClapI64(4, 2, 3)) 51 | 52 | ta.Equal(uint(2), ClapU(1, 2, 3)) 53 | ta.Equal(uint(3), ClapU(4, 2, 3)) 54 | 55 | ta.Equal(uint8(2), ClapU8(1, 2, 3)) 56 | ta.Equal(uint8(3), ClapU8(4, 2, 3)) 57 | 58 | ta.Equal(uint16(2), ClapU16(1, 2, 3)) 59 | ta.Equal(uint16(3), ClapU16(4, 2, 3)) 60 | 61 | ta.Equal(uint32(2), ClapU32(1, 2, 3)) 62 | ta.Equal(uint32(3), ClapU32(4, 2, 3)) 63 | 64 | ta.Equal(uint64(2), ClapU64(1, 2, 3)) 65 | ta.Equal(uint64(3), ClapU64(4, 2, 3)) 66 | 67 | } 68 | -------------------------------------------------------------------------------- /scripts/build_md.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | 4 | import os 5 | import jinja2 6 | import subprocess 7 | 8 | def render_j2(tmpl_path, tmpl_vars, output_path): 9 | template_loader = jinja2.FileSystemLoader(searchpath='./') 10 | template_env = jinja2.Environment(loader=template_loader, 11 | undefined=jinja2.StrictUndefined) 12 | template = template_env.get_template(tmpl_path) 13 | 14 | txt = template.render(tmpl_vars) 15 | 16 | with open(output_path, 'w') as f: 17 | f.write(txt) 18 | 19 | def command(cmd, *arguments, **options): 20 | 21 | close_fds = options.get('close_fds', True) 22 | cwd = options.get('cwd', None) 23 | shell = options.get('shell', False) 24 | env = options.get('env', None) 25 | if env is not None: 26 | env = dict(os.environ, **env) 27 | stdin = options.get('stdin', None) 28 | 29 | subproc = subprocess.Popen([cmd] + list(arguments), 30 | close_fds=close_fds, 31 | shell=shell, 32 | cwd=cwd, 33 | env=env, 34 | encoding='utf-8', 35 | stdin=subprocess.PIPE, 36 | stdout=subprocess.PIPE, 37 | stderr=subprocess.PIPE, ) 38 | 39 | out, err = subproc.communicate(input=stdin) 40 | 41 | subproc.wait() 42 | 43 | if subproc.returncode != 0: 44 | raise Exception(subproc.returncode, out, err) 45 | 46 | return out 47 | 48 | if __name__ == "__main__": 49 | pkg = command('go', 'list', '.') 50 | name = pkg.strip().split('/')[-1] 51 | tmpl_vars = { 52 | "name": name 53 | } 54 | 55 | fns = os.listdir('docs') 56 | for fn in fns: 57 | if fn.endswith('.md.j2'): 58 | render_j2('docs/'+fn, tmpl_vars, fn[:-3]) 59 | -------------------------------------------------------------------------------- /vers/vers_test.go: -------------------------------------------------------------------------------- 1 | package vers 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestIsCompatible(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input string 15 | spec []string 16 | want bool 17 | }{ 18 | {"1.0.0", []string{"==1.0.0"}, true}, 19 | {"1.0.0", []string{"==1.0.1"}, false}, 20 | {"1.0.0", []string{"==1.0.0", "==1.0.1"}, true}, 21 | {"1.2.3", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 22 | {"1.9.9", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 23 | {"3.1.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 24 | {"2.1.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, false}, 25 | {"4.2.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, false}, 26 | 27 | // invalid version 28 | {"abc.e", []string{"1.0.0"}, false}, 29 | {"1.0.0", []string{"a.b.c"}, false}, 30 | {"1.0.0", []string{"1.2.3", "a.b.c"}, false}, 31 | } 32 | 33 | for i, c := range cases { 34 | got := IsCompatible(c.input, c.spec) 35 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 36 | } 37 | } 38 | 39 | func TestCheck_valid(t *testing.T) { 40 | 41 | ta := require.New(t) 42 | 43 | cases := []struct { 44 | input string 45 | spec []string 46 | want bool 47 | }{ 48 | {"1.0.0", []string{"==1.0.0"}, true}, 49 | {"1.0.0", []string{"==1.0.1"}, false}, 50 | {"1.0.0", []string{"==1.0.0", "==1.0.1"}, true}, 51 | {"1.0.0", []string{"1.0.0", "1.0.2"}, true}, 52 | {"1.2.3", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 53 | {"1.9.9", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 54 | {"3.1.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, true}, 55 | {"2.1.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, false}, 56 | {"4.2.1", []string{`>1.0.0 <2.0.0`, `>3.0.0 !4.2.1`}, false}, 57 | } 58 | 59 | for i, c := range cases { 60 | got := Check(c.input, c.spec...) 61 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /sigbits/firstdiff.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "math/bits" 5 | ) 6 | 7 | // sFirstDiffBit returns the first different bit position. 8 | // If a or b is a prefix of the other, it returns the smaller length times 8. 9 | // 10 | // Since 0.1.9 11 | func sFirstDiffBit(a, b string) int32 { 12 | 13 | la := len(a) 14 | lb := len(b) 15 | 16 | l1 := len(a) * 8 17 | l2 := len(b) * 8 18 | 19 | minl := l1 20 | if minl > l2 { 21 | minl = l2 22 | } 23 | 24 | for i := 0; i < la && i < lb; i += 8 { 25 | 26 | au := get64Bits(a[i:]) 27 | bu := get64Bits(b[i:]) 28 | 29 | first := bits.LeadingZeros64(au ^ bu) 30 | 31 | if first < 64 { 32 | first = i<<3 + first 33 | if first < minl { 34 | return int32(first) 35 | } else { 36 | return int32(minl) 37 | } 38 | } 39 | } 40 | 41 | return int32(minl) 42 | } 43 | 44 | // FirstDiffBits find the first different bit position of every two adjacent 45 | // keys. 46 | // 47 | // Since 0.1.9 48 | func FirstDiffBits(keys []string) []int32 { 49 | l := len(keys) 50 | 51 | ds := make([]int32, l-1) 52 | for i := 0; i < l-1; i++ { 53 | ds[i] = sFirstDiffBit(keys[i], keys[i+1]) 54 | } 55 | 56 | return ds 57 | } 58 | 59 | // get64Bits converts a string of length upto 8 to a uint64, 60 | // in big endian. 61 | // Less than 8 byte string will be filled with trailing 0. 62 | // More than 8 bytes will be ignored. 63 | // 64 | // Since 0.1.9 65 | func get64Bits(s string) uint64 { 66 | 67 | if len(s) >= 8 { 68 | 69 | return ((uint64(s[0]) << 56) + 70 | (uint64(s[1]) << 48) + 71 | (uint64(s[2]) << 40) + 72 | (uint64(s[3]) << 32) + 73 | (uint64(s[4]) << 24) + 74 | (uint64(s[5]) << 16) + 75 | (uint64(s[6]) << 8) + 76 | (uint64(s[7]))) 77 | 78 | } else { 79 | 80 | bs := make([]byte, 8) 81 | copy(bs, s) 82 | return ((uint64(bs[0]) << 56) + 83 | (uint64(bs[1]) << 48) + 84 | (uint64(bs[2]) << 40) + 85 | (uint64(bs[3]) << 32) + 86 | (uint64(bs[4]) << 24) + 87 | (uint64(bs[5]) << 16) + 88 | (uint64(bs[6]) << 8) + 89 | (uint64(bs[7]))) 90 | } 91 | } 92 | -------------------------------------------------------------------------------- /bmtree/bmtree.go: -------------------------------------------------------------------------------- 1 | // Package bmtree encode a binary tree into a bitmap. 2 | // Present nodes are set to 1 in bitmap, absent nodes are set to 0. 3 | // 4 | // A node is identified with its searching path: 5 | // E.g. a path 0110 means going from root node and then choose left, right, 6 | // right and left branch. 7 | // 8 | // "bmtree" supports encoding a binary tree of up to 30 levels and the result 9 | // bitmap size is 0~2^31-1. 10 | // E.g. the following three nodes "0", "01", "10" can be encoded into a 7 bits 11 | // bitmap: "0101010". 12 | // The mapping of all nodes in a 2 level tree are: 13 | // 14 | // path index in bitmap 15 | // "" 0 16 | // "0" 1 17 | // "00" 2 18 | // "01" 3 19 | // "1" 4 20 | // "10" 5 21 | // "11" 6 22 | // 23 | // How it works 24 | // 25 | // If there is a binary tree: 26 | // an empty bit array is mapped to root node, 27 | // "01" is mapped to node 01, etc. 28 | // 29 | // root h=2 30 | // / \ 31 | // 0 1 h=1 32 | // / \ / \ 33 | // 00 01 10 11 h=0 34 | // 35 | // To assign a node to a index in bitmap, 36 | // We places them in "pre-order"(or NLR: parent node comes before left child, 37 | // then right child), 38 | // thus "0" comes before "00" in the bitmap: 39 | // 40 | // Put a node at index x, 41 | // then start from x+1 to put its left-subtree, 42 | // then right-subtree, recursively. 43 | // 44 | // Root node is at the 0-th bit. 45 | // 46 | // Path: 47 | // 48 | // A path is for selecting a node. 49 | // E.g.: a "0" in a path for selecting left child, a "1" for right child. 50 | // 51 | // The representation of a path is a uint64, 52 | // the higher 32 bits are path mask, 53 | // the lower 32 bits are searching bits: 54 | // 0...011111000 0...0xxxxx000 55 | // <-- significant bits 56 | // 57 | // The Left most bit in path-mask indicates the tree height. 58 | // 59 | // A full binary tree has 2^(h+1)-1 nodes thus the result bitmap has 2^(h+1)-1 60 | // bits. 61 | // 62 | // Since 0.1.9 63 | package bmtree 64 | -------------------------------------------------------------------------------- /pbcmpl/header_test.go: -------------------------------------------------------------------------------- 1 | package pbcmpl 2 | 3 | import ( 4 | "testing" 5 | 6 | proto "github.com/golang/protobuf/proto" 7 | 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestVerStr(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | var str string = verStr(nil) 16 | ta.Equal("", str) 17 | 18 | ta.Equal("", verStr([]byte{})) 19 | ta.Equal("abc", verStr([]byte{'a', 'b', 'c'})) 20 | ta.Equal("1.0.0", verStr([]byte{'1', '.', '0', '.', '0', 0})) 21 | 22 | bBuf := []byte{'1', '.', '0', '.', '0', 0} 23 | str = verStr(bBuf) 24 | bBuf[0] = '2' 25 | ta.Equal("1.0.0", str) 26 | } 27 | 28 | func TestNewHeader(t *testing.T) { 29 | 30 | ta := require.New(t) 31 | 32 | ver := "0.0.1" 33 | dataSize := uint64(1000) 34 | headerSize := uint64(32) 35 | h := newHeader(ver, dataSize) 36 | 37 | ta.Equal(dataSize, h.BodySize) 38 | ta.Equal(headerSize, h.HeaderSize) 39 | ta.Equal(ver, verStr(h.Version[:])) 40 | 41 | ta.Equal("0.0.1 32 1000", h.String()) 42 | 43 | // 16 byte ver, no panic 44 | _ = newHeader("111111111111.2.6", 10) 45 | ta.Panics(func() { newHeader("111111111111.2.62", 10) }) 46 | } 47 | 48 | func TestHeader_MarshalUnmarshal(t *testing.T) { 49 | 50 | ta := require.New(t) 51 | 52 | ver := "0.0.1" 53 | dataSize := uint64(0xffff01) 54 | h := newHeader(ver, dataSize) 55 | 56 | b, err := proto.Marshal(h) 57 | ta.Nil(err) 58 | 59 | want := append([]byte("0.0.1"), make([]byte, 11)...) 60 | want = append(want, 61 | 32, 0, 0, 0, 62 | 0, 0, 0, 0) 63 | want = append(want, 64 | 1, 0xff, 0xff, 0, 65 | 0, 0, 0, 0) 66 | ta.Equal(want, b) 67 | 68 | h2 := &header{} 69 | err = proto.Unmarshal(b, h2) 70 | ta.Nil(err) 71 | ta.Equal(h, h2) 72 | 73 | h.ProtoMessage() 74 | } 75 | 76 | func TestReadHeader(t *testing.T) { 77 | 78 | ta := require.New(t) 79 | 80 | msg := &vh{} 81 | rw := &IncomleteReaderWriter{} 82 | n, err := Marshal(rw, msg) 83 | ta.Nil(err) 84 | ta.Equal(int64(64), n) 85 | 86 | n, h, err := ReadHeader(rw) 87 | 88 | ta.Nil(err) 89 | ta.Equal(int64(32), n) 90 | ta.Equal("1.2.3", h.GetVersion()) 91 | ta.Equal(int64(32), h.GetHeaderSize()) 92 | ta.Equal(int64(32), h.GetBodySize()) 93 | } 94 | -------------------------------------------------------------------------------- /bmtree/allpaths_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestAllPaths(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | bitmapSize int32 15 | want []string 16 | }{ 17 | {1, []string{""}}, 18 | {2, []string{"0", "1"}}, 19 | {3, []string{"", "0", "1"}}, 20 | {4, []string{"00", "01", "10", "11"}}, 21 | {5, []string{"", "00", "01", "10", "11"}}, 22 | {6, []string{"0", "00", "01", "1", "10", "11"}}, 23 | {7, []string{"", "0", "00", "01", "1", "10", "11"}}, 24 | {8, []string{"000", "001", "010", "011", "100", "101", "110", "111"}}, 25 | {9, []string{"", "000", "001", "010", "011", "100", "101", "110", "111"}}, 26 | } 27 | 28 | for i, c := range cases { 29 | ps := AllPaths(c.bitmapSize, 0, 1<<63) 30 | got := make([]string, 0) 31 | for _, p := range ps { 32 | got = append(got, PathStr(p)) 33 | } 34 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 35 | } 36 | } 37 | 38 | func TestAllPaths_range(t *testing.T) { 39 | 40 | ta := require.New(t) 41 | 42 | cases := []struct { 43 | bitmapSize int32 44 | from, to uint64 45 | want []string 46 | }{ 47 | {3, 0x0000000000000001, 0x0000000100000002, []string{"0", "1"}}, 48 | {9, 0x0000000000000000, 0x0000000f00000010, []string{"", "000", "001", "010", "011", "100", "101", "110", "111"}}, 49 | {9, 0x0000000000000000, 0x0000000100000007, []string{"", "000"}}, 50 | {0x7fffffff, 0x3fffffff3fffffff, 0x4000000000000000, []string{"111111111111111111111111111111"}}, 51 | } 52 | 53 | for i, c := range cases { 54 | ps := AllPaths(c.bitmapSize, c.from, c.to) 55 | got := make([]string, 0) 56 | for _, p := range ps { 57 | got = append(got, PathStr(p)) 58 | } 59 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 60 | } 61 | } 62 | 63 | func TestAllPaths_order(t *testing.T) { 64 | 65 | ta := require.New(t) 66 | 67 | for bitmapSize := int32(1); bitmapSize < 1024; bitmapSize++ { 68 | ps := AllPaths(bitmapSize, 0, 1<<63) 69 | prev := uint64(0) 70 | for i, p := range ps { 71 | if i > 0 { 72 | ta.True(p > prev) 73 | } 74 | prev = p 75 | } 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /bitmap/ofmany_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestOfMany(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input [][]int32 15 | inputsizes []int32 16 | wantwords []uint64 17 | }{ 18 | { 19 | [][]int32{}, 20 | []int32{}, 21 | []uint64{}, 22 | }, 23 | { 24 | [][]int32{{0}}, 25 | []int32{1}, 26 | []uint64{1}, 27 | }, 28 | { 29 | [][]int32{{0}}, 30 | []int32{2}, 31 | []uint64{1}, 32 | }, 33 | { 34 | [][]int32{{0}}, 35 | []int32{64}, 36 | []uint64{1}, 37 | }, 38 | { 39 | [][]int32{{0}}, 40 | []int32{65}, 41 | []uint64{1, 0}, 42 | }, 43 | { 44 | [][]int32{{0, 1, 2}}, 45 | []int32{0}, 46 | []uint64{7}, 47 | }, 48 | { 49 | [][]int32{{0, 1, 2}}, 50 | []int32{1}, 51 | []uint64{7}, 52 | }, 53 | { 54 | [][]int32{{0}, {0, 1}}, 55 | []int32{1, 2}, 56 | []uint64{7}, 57 | }, 58 | { 59 | [][]int32{{0}, {1, 2}}, 60 | []int32{1, 2}, 61 | []uint64{13}, 62 | }, 63 | { 64 | [][]int32{{0}, {1, 2}}, 65 | []int32{5, 2}, 66 | []uint64{193}, 67 | }, 68 | { 69 | [][]int32{{0}, {1, 2}}, 70 | []int32{64, 64}, 71 | []uint64{1, 6}, 72 | }, 73 | { 74 | [][]int32{{0, 1}, {2, 63}}, 75 | []int32{64, 0}, 76 | []uint64{3, (1 << 63) + 4}, 77 | }, 78 | { 79 | [][]int32{{64}}, 80 | []int32{0}, 81 | []uint64{0, 1}, 82 | }, 83 | { 84 | [][]int32{{64}, {0}}, 85 | []int32{65, 1}, 86 | []uint64{0, 3}, 87 | }, 88 | { 89 | [][]int32{{1}, {2}, {3, 64, 129}}, 90 | []int32{1, 1, 129}, 91 | []uint64{42, 4, 8}, 92 | }, 93 | { // extend to contain all bitmap 94 | [][]int32{{1}, {2}}, 95 | []int32{1, 127}, 96 | []uint64{0xa, 0}, 97 | }, 98 | { // extend to contain all bitmap 99 | [][]int32{{1}, {2}}, 100 | []int32{1, 128}, 101 | []uint64{0xa, 0, 0}, 102 | }, 103 | } 104 | 105 | for i, c := range cases { 106 | 107 | got := OfMany(c.input, c.inputsizes) 108 | 109 | ta.Equal(c.wantwords, got, 110 | "%d-th: case: %+v", 111 | i+1, c) 112 | } 113 | } 114 | -------------------------------------------------------------------------------- /bmtree/pathlen_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "strconv" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestPathLen(t *testing.T) { 11 | 12 | ta := require.New(t) 13 | 14 | cases := []struct { 15 | path string 16 | wantlen int32 17 | }{ 18 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 19 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 20 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 21 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 22 | {"00000000000000000000000000000000" + "00000000000000000000000000000001", 1}, 23 | {"00000000000000000000000000000001" + "00000000000000000000000000000011", 2}, 24 | {"00000000000000000000000000001100" + "00000000000000000000000000011111", 5}, 25 | {"00000000000000000000000000011000" + "00000000000000000000000000111111", 6}, 26 | {"00000000000000000000000000110000" + "00000000000000000000000001111111", 7}, 27 | {"00000000000000000000000001100001" + "00000000000000000000000011111111", 8}, 28 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 29 | {"00000000000000000000000000000001" + "00000000000000000000000000000001", 1}, 30 | {"00000000000000000000000000000010" + "00000000000000000000000000000011", 2}, 31 | {"00000000000000000000000000010000" + "00000000000000000000000000011111", 5}, 32 | {"00000000000000000000000000100001" + "00000000000000000000000000111111", 6}, 33 | {"00000000000000000000000010000100" + "00000000000000000000000011111100", 6}, 34 | {"10001001100011000000000000000000" + "11111111111111000000000000000000", 14}, 35 | {"00000000000000000000000010000000" + "00000000000000000000000010000000", 1}, 36 | {"00110010101100110011001110110100" + "11111111111111111111111111111111", 32}, 37 | {"01100101011001100110011101101000" + "11111111111111111111111111111111", 32}, 38 | {"11001010110011001100111011010000" + "11111111111111111111111111111110", 31}, 39 | } 40 | 41 | for i, c := range cases { 42 | p := u64(c.path) 43 | gotlen := PathLen(p) 44 | ta.Equal(c.wantlen, gotlen, "%d-th: case: %+v", i+1, c) 45 | } 46 | } 47 | 48 | func u64(s string) uint64 { 49 | i, _ := strconv.ParseUint(s, 2, 64) 50 | return i 51 | } 52 | -------------------------------------------------------------------------------- /bmtree/pathbits_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "strconv" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestPathBitsAndMask(t *testing.T) { 11 | 12 | ta := require.New(t) 13 | 14 | u64 := func(s string) uint64 { 15 | i, _ := strconv.ParseUint(s, 2, 64) 16 | return i 17 | } 18 | 19 | cases := []struct { 20 | path string 21 | wantlen int32 22 | }{ 23 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 24 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 25 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 26 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 27 | {"00000000000000000000000000000000" + "00000000000000000000000000000001", 1}, 28 | {"00000000000000000000000000000001" + "00000000000000000000000000000011", 2}, 29 | {"00000000000000000000000000001100" + "00000000000000000000000000011111", 5}, 30 | {"00000000000000000000000000011000" + "00000000000000000000000000111111", 6}, 31 | {"00000000000000000000000000110000" + "00000000000000000000000001111111", 7}, 32 | {"00000000000000000000000001100001" + "00000000000000000000000011111111", 8}, 33 | {"00000000000000000000000000000000" + "00000000000000000000000000000000", 0}, 34 | {"00000000000000000000000000000001" + "00000000000000000000000000000001", 1}, 35 | {"00000000000000000000000000000010" + "00000000000000000000000000000011", 2}, 36 | {"00000000000000000000000000010000" + "00000000000000000000000000011111", 5}, 37 | {"00000000000000000000000000100001" + "00000000000000000000000000111111", 6}, 38 | {"00000000000000000000000010000100" + "00000000000000000000000011111100", 6}, 39 | {"10001001100011000000000000000000" + "11111111111111000000000000000000", 14}, 40 | {"00000000000000000000000010000000" + "00000000000000000000000010000000", 1}, 41 | {"00110010101100110011001110110100" + "11111111111111111111111111111111", 32}, 42 | {"01100101011001100110011101101000" + "11111111111111111111111111111111", 32}, 43 | {"11001010110011001100111011010000" + "11111111111111111111111111111110", 31}, 44 | } 45 | 46 | for i, c := range cases { 47 | p := u64(c.path) 48 | gotbits := PathBits(p) 49 | gotmask := PathMask(p) 50 | ta.Equal(p>>32, gotbits, "%d-th: case: %+v", i+1, c) 51 | ta.Equal(p&0xffffffff, gotmask, "%d-th: case: %+v", i+1, c) 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /sigbits/countprefixes_test.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestCountPrefixes(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | keys []string 15 | to int32 16 | wantmin int32 17 | want []int32 18 | }{ 19 | // a= 0110 0001 20 | // b= 0110 0011 21 | 22 | {[]string{"a", "b"}, 1, 23 | 6, []int32{1}}, 24 | 25 | {[]string{"a", "b"}, 8, 26 | 6, []int32{1, 2, 2, 2, 27 | 2, 2, 2, 2}}, 28 | 29 | {[]string{"a", "b"}, 16, 30 | 6, []int32{1, 2, 2, 2, 31 | 2, 2, 2, 2, 32 | 2, 2, 2, 2, 33 | 2, 2, 2, 2}}, 34 | 35 | {[]string{"aa", "ab"}, 1, 36 | 14, []int32{1}}, 37 | 38 | {[]string{"aa", "ab"}, 8, 39 | 14, []int32{1, 2, 2, 2, 40 | 2, 2, 2, 2}}, 41 | 42 | {[]string{"aa", "ab"}, 16, 43 | 14, []int32{1, 2, 2, 2, 44 | 2, 2, 2, 2, 45 | 2, 2, 2, 2, 46 | 2, 2, 2, 2}}, 47 | 48 | {[]string{"aa", "ab"}, 1, 49 | 14, []int32{1}}, 50 | } 51 | 52 | for i, c := range cases { 53 | firstdiffs := FirstDiffBits(c.keys) 54 | gotmin, got := countPrefixes(firstdiffs, c.to) 55 | ta.Equal(c.wantmin, gotmin, "%d-th: case: %+v", i+1, c) 56 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 57 | } 58 | } 59 | 60 | func TestCountPrefixes_2uint64(t *testing.T) { 61 | 62 | ta := require.New(t) 63 | 64 | keys := []string{ 65 | "aa", 66 | "ab", 67 | "abc", 68 | "abd", 69 | "ac", 70 | "b", 71 | "bbbbbbbb", 72 | "bbbbbbbba", 73 | "bbbbbbbbaa", 74 | "bbbbbbbbab", 75 | "bbbbbbbbc", 76 | "c", 77 | } 78 | 79 | wantfull := []int32{1, 80 | // 1~6 are empty 81 | 2, 3, 82 | 4, 4, 4, 4, 83 | 4, 4, 5, 6, 84 | 7, 7, 7, 7, 85 | 7, 8, 8, 8, 86 | 8, 8, 8, 8, 87 | 8, 8, 8, 8, 88 | 89 | // 4-th byte 90 | 8, 8, 8, 8, 91 | 8, 8, 8, 8, 92 | 8, 8, 8, 8, 93 | 8, 8, 8, 8, 94 | 8, 8, 8, 8, 95 | 8, 8, 8, 8, 96 | 8, 8, 8, 8, 97 | 8, 8, 8, 8, 98 | 99 | // 8-th byte 100 | 9, 9, 9, 9, 101 | 9, 9, 10, 10, 102 | 11, 11, 11, 11, 103 | 11, 11, 12, 12, 104 | 12, 12, 12, 12, 105 | 12, 12, 12, 12, 106 | 12, 12, 12, 12, 107 | 12, 12, 12, 12, 108 | } 109 | 110 | for i := 1; i < len(wantfull)+1; i++ { 111 | firstdiffs := FirstDiffBits(keys) 112 | gotmin, got := countPrefixes(firstdiffs, int32(i)) 113 | want := wantfull[:i] 114 | 115 | ta.Equal(int32(6), gotmin) 116 | ta.Equal(want, got, "%d-th: case: %+v", i, i) 117 | } 118 | } 119 | -------------------------------------------------------------------------------- /sigbits/sigbits_countprefixes_test.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestSigBits_CountPrefixes(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | keys []string 15 | to int32 16 | wantmin int32 17 | want []int32 18 | }{ 19 | // a= 0110 0001 20 | // b= 0110 0011 21 | 22 | {[]string{"a", "b"}, 1, 23 | 6, []int32{1}}, 24 | 25 | {[]string{"a", "b"}, 8, 26 | 6, []int32{1, 2, 2, 2, 27 | 2, 2, 2, 2}}, 28 | 29 | {[]string{"a", "b"}, 16, 30 | 6, []int32{1, 2, 2, 2, 31 | 2, 2, 2, 2, 32 | 2, 2, 2, 2, 33 | 2, 2, 2, 2}}, 34 | 35 | {[]string{"aa", "ab"}, 1, 36 | 14, []int32{1}}, 37 | 38 | {[]string{"aa", "ab"}, 8, 39 | 14, []int32{1, 2, 2, 2, 40 | 2, 2, 2, 2}}, 41 | 42 | {[]string{"aa", "ab"}, 16, 43 | 14, []int32{1, 2, 2, 2, 44 | 2, 2, 2, 2, 45 | 2, 2, 2, 2, 46 | 2, 2, 2, 2}}, 47 | 48 | {[]string{"aa", "ab"}, 1, 49 | 14, []int32{1}}, 50 | } 51 | 52 | for i, c := range cases { 53 | sb := New(c.keys) 54 | gotmin, got := sb.CountPrefixes(0, int32(len(c.keys)), c.to) 55 | ta.Equal(c.wantmin, gotmin, "%d-th: case: %+v", i+1, c) 56 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 57 | } 58 | } 59 | 60 | func TestSigBigs_CountPrefixes_2uint64(t *testing.T) { 61 | 62 | ta := require.New(t) 63 | 64 | keys := []string{ 65 | "aa", 66 | "ab", 67 | "abc", 68 | "abd", 69 | "ac", 70 | "b", 71 | "bbbbbbbb", 72 | "bbbbbbbba", 73 | "bbbbbbbbaa", 74 | "bbbbbbbbab", 75 | "bbbbbbbbc", 76 | "c", 77 | } 78 | 79 | wantfull := []int32{1, 80 | // 1~6 are empty 81 | 2, 3, 82 | 4, 4, 4, 4, 83 | 4, 4, 5, 6, 84 | 7, 7, 7, 7, 85 | 7, 8, 8, 8, 86 | 8, 8, 8, 8, 87 | 8, 8, 8, 8, 88 | 89 | // 4-th byte 90 | 8, 8, 8, 8, 91 | 8, 8, 8, 8, 92 | 8, 8, 8, 8, 93 | 8, 8, 8, 8, 94 | 8, 8, 8, 8, 95 | 8, 8, 8, 8, 96 | 8, 8, 8, 8, 97 | 8, 8, 8, 8, 98 | 99 | // 8-th byte 100 | 9, 9, 9, 9, 101 | 9, 9, 10, 10, 102 | 11, 11, 11, 11, 103 | 11, 11, 12, 12, 104 | 12, 12, 12, 12, 105 | 12, 12, 12, 12, 106 | 12, 12, 12, 12, 107 | 12, 12, 12, 12, 108 | } 109 | 110 | for i := 1; i < len(wantfull)+1; i++ { 111 | sb := New(keys) 112 | gotmin, got := sb.CountPrefixes(0, int32(len(keys)), int32(i)) 113 | want := wantfull[:i] 114 | 115 | ta.Equal(int32(6), gotmin) 116 | ta.Equal(want, got, "%d-th: case: %+v", i, i) 117 | } 118 | } 119 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # low 2 | low level data type and utils in Golang. 3 | 4 | [![Travis-CI](https://api.travis-ci.org/openacid/low.svg?branch=master)](https://travis-ci.org/openacid/low) 5 | [![AppVeyor](https://ci.appveyor.com/api/projects/status/1jnttodaenbrv3va/branch/master?svg=true)](https://ci.appveyor.com/project/drmingdrmer/low/branch/master) 6 | [![GoDoc](https://godoc.org/github.com/openacid/low?status.svg)](http://godoc.org/github.com/openacid/low) 7 | [![Report card](https://goreportcard.com/badge/github.com/openacid/low)](https://goreportcard.com/report/github.com/openacid/low) 8 | [![GolangCI](https://golangci.com/badges/github.com/openacid/low.svg)](https://golangci.com/r/github.com/openacid/low) 9 | [![Sourcegraph](https://sourcegraph.com/github.com/openacid/low/-/badge.svg)](https://sourcegraph.com/github.com/openacid/low?badge) 10 | [![Coverage Status](https://coveralls.io/repos/github/openacid/low/badge.svg?branch=master)](https://coveralls.io/github/openacid/low?branch=master) 11 | 12 | A stable low level function set is the basis of a robust architecture. 13 | It focuses on stability and requires high test coverage. 14 | 15 | # Status 16 | 17 | ![stability-stable](https://img.shields.io/badge/stability-stable-green.svg) 18 | 19 | This project has been supporting all of our other go projects. 20 | 21 | # Install 22 | 23 | ```sh 24 | go get github.com/openacid/low/... 25 | ``` 26 | 27 | # Modules 28 | 29 | - `bitmap` provides bitmap operations. [![GoDoc](https://godoc.org/github.com/openacid/low/bitmap?status.svg)](http://godoc.org/github.com/openacid/low/bitmap) 30 | - `bitword` provides n-bit word conversion to and from string. [![GoDoc](https://godoc.org/github.com/openacid/low/bitword?status.svg)](http://godoc.org/github.com/openacid/low/bitword) 31 | - `bmtree` encode a binary tree into a bitmap. [![GoDoc](https://godoc.org/github.com/openacid/low/bmtree?status.svg)](http://godoc.org/github.com/openacid/low/bmtree) 32 | - `iohelper` provides extra interfaces than package io. [![GoDoc](https://godoc.org/github.com/openacid/low/iohelper?status.svg)](http://godoc.org/github.com/openacid/low/iohelper) 33 | - `pbcmpl` adds a header for proto.Message to make it self-described. [![GoDoc](https://godoc.org/github.com/openacid/low/pbcmpl?status.svg)](http://godoc.org/github.com/openacid/low/pbcmpl) 34 | - `sigbits` extracts significant bits from a sorted list of strings.. [![GoDoc](https://godoc.org/github.com/openacid/low/sigbits?status.svg)](http://godoc.org/github.com/openacid/low/sigbits) 35 | - `size` provides value size operations. [![GoDoc](https://godoc.org/github.com/openacid/low/size?status.svg)](http://godoc.org/github.com/openacid/low/size) 36 | - `tree` provides abstract tree operations. [![GoDoc](https://godoc.org/github.com/openacid/low/tree?status.svg)](http://godoc.org/github.com/openacid/low/tree) 37 | -------------------------------------------------------------------------------- /bitmap/rank_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestIndexRank(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | bm []uint64 15 | want64 []int32 16 | want128 []int32 17 | }{ 18 | { 19 | []uint64{}, 20 | []int32{0}, 21 | []int32{0}, 22 | }, 23 | { 24 | []uint64{0}, 25 | []int32{0, 0}, 26 | []int32{0}, 27 | }, 28 | { 29 | []uint64{1}, 30 | []int32{0, 1}, 31 | []int32{0}, 32 | }, 33 | { 34 | []uint64{0xffffffffffffffff}, 35 | []int32{0, 64}, 36 | []int32{0}, 37 | }, 38 | { 39 | []uint64{0xffffffffffffffff, 1}, 40 | []int32{0, 64, 65}, 41 | []int32{0, 65}, 42 | }, 43 | { 44 | []uint64{0xffffffffffffffff, 1, 1}, 45 | []int32{0, 64, 65, 66}, 46 | []int32{0, 65}, 47 | }, 48 | { 49 | []uint64{0xffffffffffffffff, 1, 1, 3}, 50 | []int32{0, 64, 65, 66, 68}, 51 | []int32{0, 65, 68}, 52 | }, 53 | { 54 | []uint64{0xffffffffffffffff, 1, 1, 3, 4}, 55 | []int32{0, 64, 65, 66, 68, 69}, 56 | []int32{0, 65, 68}, 57 | }, 58 | } 59 | 60 | for i, c := range cases { 61 | 62 | noLast := c.want64[:len(c.want64)-1] 63 | idx64 := IndexRank64(c.bm) 64 | ta.Equal(noLast, idx64, "%d-th: case: %+v", i+1, c) 65 | 66 | idx64 = IndexRank64(c.bm, false) 67 | ta.Equal(noLast, idx64, "%d-th: case: %+v", i+1, c) 68 | 69 | idx64 = IndexRank64(c.bm, true) 70 | ta.Equal(c.want64, idx64, "%d-th: case: %+v", i+1, c) 71 | 72 | idx128 := IndexRank128(c.bm) 73 | ta.Equal(c.want128, idx128, "%d-th: case: %+v", i+1, c) 74 | 75 | // test Rank64 and Rank128 76 | cnt := int32(0) 77 | cntExcludeI := int32(0) 78 | for j := 0; j < len(c.bm)*64; j++ { 79 | if c.bm[j>>6]&(1< 1 --> 3 -->5 64 | // `->6 65 | // `-> 2 --> 4 66 | tr := &T{ 67 | branches: map[int][]int{ 68 | 0: {1, 2}, 69 | 1: {3}, 70 | 2: {4}, 71 | 3: {5, 6}, 72 | }, 73 | } 74 | 75 | rst := tree.String(tr) 76 | want := ` 77 | #00(foo)*2 78 | -0->#01(foo) 79 | -0->#03(foo)*2 80 | -0->#05(foo)=leaf 81 | -1->#06(foo)=leaf 82 | -1->#02(foo) 83 | -0->#04(foo)=leaf`[1:] 84 | if want != rst { 85 | t.Fatalf("expect: \n%v\n; but: \n%v\n", want, rst) 86 | } 87 | } 88 | 89 | func TestDepthFirst(t *testing.T) { 90 | 91 | ta := require.New(t) 92 | 93 | tr := &T{ 94 | branches: map[int][]int{ 95 | 0: {1, 2}, 96 | 1: {3}, 97 | 2: {4}, 98 | 3: {5, 6}, 99 | }, 100 | } 101 | 102 | got := []string{} 103 | 104 | tree.DepthFirst(tr, func(tr tree.Tree, parent, label, node interface{}) { 105 | var b int 106 | if label == nil { 107 | b = 0 108 | } else { 109 | b = label.(int) 110 | } 111 | s := fmt.Sprintf("p:%s-b:%d->n:%s", tr.NodeID(parent), b, tr.NodeID(node)) 112 | got = append(got, s) 113 | }) 114 | 115 | want := []string{ 116 | "p:03-b:0->n:05", 117 | "p:03-b:1->n:06", 118 | "p:01-b:0->n:03", 119 | "p:00-b:0->n:01", 120 | "p:02-b:0->n:04", 121 | "p:00-b:1->n:02", 122 | "p:00-b:0->n:00", 123 | } 124 | 125 | ta.Equal(want, got) 126 | } 127 | -------------------------------------------------------------------------------- /bmtree/index_debug_test.go: -------------------------------------------------------------------------------- 1 | // +build debug 2 | 3 | package bmtree 4 | 5 | import ( 6 | "testing" 7 | 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestPathToIndex_input(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | cases := []struct { 16 | bitmapSize int32 17 | path uint64 18 | wantpanic bool 19 | }{ 20 | // path to a root but tree does not store. 21 | {4, 1, true}, 22 | {4, 2, true}, 23 | 24 | // path to absent level 25 | {5, 1, true}, 26 | {5, 2, true}, 27 | 28 | // longer 29 | {5, 4, true}, 30 | 31 | // path is longer than expected 32 | {7, 8, true}, 33 | {7, 4, true}, 34 | {7, 5, true}, 35 | {7, 6, true}, 36 | {7, 7, true}, 37 | 38 | // path must be consecutive "1"s 39 | {0xf, 5, true}, 40 | 41 | // path mask shorter than path bits 42 | {0xf, 0xf<<32 + 0xe, true}, 43 | 44 | // bitmap length <=31 45 | {-1, 0, true}, 46 | 47 | // path length <=31 48 | {1, 1 << 31, true}, 49 | 50 | // path points to present level 51 | {7, 0, false}, 52 | {7, 2, false}, 53 | {7, 3, false}, 54 | 55 | {5, 0, false}, 56 | {5, 3, false}, 57 | } 58 | 59 | for _, c := range cases { 60 | if c.wantpanic { 61 | ta.Panics(func() { PathToIndex(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 62 | } else { 63 | ta.NotPanics(func() { PathToIndex(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 64 | 65 | } 66 | } 67 | 68 | } 69 | 70 | func TestPathToIndexLoose_input(t *testing.T) { 71 | 72 | ta := require.New(t) 73 | 74 | cases := []struct { 75 | bitmapSize int32 76 | path uint64 77 | wantpanic bool 78 | }{ 79 | // path to a root but tree does not store. 80 | {4, 1, true}, 81 | {4, 2, true}, 82 | 83 | // path to absent level 84 | {5, 1, true}, 85 | {5, 2, true}, 86 | 87 | // longer 88 | {5, 4, true}, 89 | 90 | // path is longer than expected 91 | {7, 8, true}, 92 | {7, 4, true}, 93 | {7, 5, true}, 94 | {7, 6, true}, 95 | {7, 7, true}, 96 | 97 | // path must be consecutive "1"s 98 | {0xf, 5, true}, 99 | 100 | // path mask shorter than path bits 101 | {0xf, 0xf<<32 + 0xe, true}, 102 | 103 | // bitmap length <=31 104 | {-1, 0, true}, 105 | 106 | // path length <=31 107 | {1, 1 << 31, true}, 108 | 109 | // path points to present level 110 | {7, 0, false}, 111 | {7, 2, false}, 112 | {7, 3, false}, 113 | 114 | {5, 0, false}, 115 | {5, 3, false}, 116 | } 117 | 118 | for _, c := range cases { 119 | if c.wantpanic { 120 | ta.Panics(func() { PathToIndex(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 121 | } else { 122 | ta.NotPanics(func() { PathToIndex(c.bitmapSize, c.path) }, "%032b, %064b", c.bitmapSize, c.path) 123 | 124 | } 125 | } 126 | 127 | } 128 | -------------------------------------------------------------------------------- /iohelper/iohelper.go: -------------------------------------------------------------------------------- 1 | // Package iohelper provides extra interfaces than package io. 2 | package iohelper 3 | 4 | import ( 5 | "errors" 6 | "io" 7 | ) 8 | 9 | const ( 10 | maxOffset int64 = 0x7fffffffffffffff 11 | ) 12 | 13 | // AtToWriter convert a WriterAt to a Writer 14 | // 15 | // Since 0.1.6 16 | func AtToWriter(w io.WriterAt, offset int64) io.Writer { 17 | return NewSectionWriter(w, offset, maxOffset-offset) 18 | } 19 | 20 | // AtToReader convert a ReaderAt to a Reader 21 | // 22 | // Since 0.1.6 23 | func AtToReader(r io.ReaderAt, offset int64) io.Reader { 24 | return io.NewSectionReader(r, offset, maxOffset-offset) 25 | } 26 | 27 | // NewSectionWriter returns a SectionWriter that writes to w 28 | // starting at offset off and stops with io.ErrShortWrite after n bytes. 29 | // 30 | // Since 0.1.6 31 | func NewSectionWriter(w io.WriterAt, off int64, n int64) *SectionWriter { 32 | return &SectionWriter{w, off, off, off + n} 33 | } 34 | 35 | // SectionWriter implements Write, Seek, and WriteAt on a section 36 | // of an underlying io.WriterAt. 37 | // 38 | // Since 0.1.6 39 | type SectionWriter struct { 40 | w io.WriterAt 41 | base int64 42 | off int64 43 | limit int64 44 | } 45 | 46 | // Write buf "p". 47 | // 48 | // Since 0.1.6 49 | func (s *SectionWriter) Write(p []byte) (n int, err error) { 50 | if s.off >= s.limit { 51 | return 0, io.ErrShortWrite 52 | } 53 | if max := s.limit - s.off; int64(len(p)) > max { 54 | p = p[0:max] 55 | err = io.ErrShortWrite 56 | } 57 | n, err2 := s.w.WriteAt(p, s.off) 58 | s.off += int64(n) 59 | 60 | if err2 != nil { 61 | err = err2 62 | } 63 | 64 | return 65 | } 66 | 67 | var errWhence = errors.New("Seek: invalid whence") 68 | var errOffset = errors.New("Seek: invalid offset") 69 | 70 | // Seek seeks to relative position by offset. 71 | // 72 | // Since 0.1.6 73 | func (s *SectionWriter) Seek(offset int64, whence int) (int64, error) { 74 | switch whence { 75 | default: 76 | return 0, errWhence 77 | case io.SeekStart: 78 | offset += s.base 79 | case io.SeekCurrent: 80 | offset += s.off 81 | case io.SeekEnd: 82 | offset += s.limit 83 | } 84 | if offset < s.base { 85 | return 0, errOffset 86 | } 87 | s.off = offset 88 | return offset - s.base, nil 89 | } 90 | 91 | // WriteAt write buf p at relative position off. 92 | // 93 | // Since 0.1.6 94 | func (s *SectionWriter) WriteAt(p []byte, off int64) (n int, err error) { 95 | if off < 0 || off >= s.limit-s.base { 96 | return 0, io.ErrShortWrite 97 | } 98 | off += s.base 99 | if max := s.limit - off; int64(len(p)) > max { 100 | p = p[0:max] 101 | n, err = s.w.WriteAt(p, off) 102 | if err == nil { 103 | err = io.ErrShortWrite 104 | } 105 | return n, err 106 | } 107 | return s.w.WriteAt(p, off) 108 | } 109 | 110 | // Size returns the size of the section in bytes. 111 | // 112 | // Since 0.1.6 113 | func (s *SectionWriter) Size() int64 { return s.limit - s.base } 114 | -------------------------------------------------------------------------------- /pbcmpl/header.go: -------------------------------------------------------------------------------- 1 | package pbcmpl 2 | 3 | import ( 4 | "bytes" 5 | "encoding/binary" 6 | "fmt" 7 | ) 8 | 9 | const ( 10 | versionLen = 16 11 | 12 | // DefaultVer is the default version if the message to marshal does not 13 | // provide version. 14 | // 15 | // Since v0.1.6 16 | DefaultVer = "1.0.0" 17 | ) 18 | 19 | var ( 20 | fixedSize = binary.Size(&header{}) 21 | endian = binary.LittleEndian 22 | ) 23 | 24 | // Header defines header info retrieving APIs. 25 | // 26 | // Since 0.1.7 27 | type Header interface { 28 | GetVersion() string 29 | GetHeaderSize() int64 30 | GetBodySize() int64 31 | } 32 | 33 | type headerInfo struct { 34 | *header 35 | } 36 | 37 | func (hi *headerInfo) GetVersion() string { 38 | return verStr(hi.Version[:]) 39 | } 40 | 41 | func (hi *headerInfo) GetHeaderSize() int64 { 42 | return int64(hi.HeaderSize) 43 | } 44 | 45 | func (hi *headerInfo) GetBodySize() int64 { 46 | return int64(hi.BodySize) 47 | } 48 | 49 | // header is a fixed-size structure that defines the header format of a 50 | // marshaled byte stream. 51 | // 52 | // It contains version, header size(size of this struct) and data size(size of 53 | // the user data). 54 | // 55 | // NEVER change this structure. 56 | // 57 | // Since v0.1.6 58 | type header struct { 59 | // version of this marshaled data, for compatibility check. 60 | // It is a semantic version in form of ..;. 61 | // As long as version is a string, its max size is 16. 62 | // 63 | // See: https://semver.org/ 64 | Version [versionLen]byte 65 | 66 | // HeaderSize is the marshaled size in byte for header. 67 | HeaderSize uint64 68 | 69 | // BodySize is the size in byte of user data. 70 | BodySize uint64 71 | } 72 | 73 | // Marshal header implement proto.Message 74 | // 75 | // Since 0.1.6 76 | func (h *header) Marshal() ([]byte, error) { 77 | b := &bytes.Buffer{} 78 | err := binary.Write(b, endian, h) 79 | return b.Bytes(), err 80 | } 81 | 82 | // Unmarshal header implement proto.Message 83 | // 84 | // Since 0.1.6 85 | func (h *header) Unmarshal(buf []byte) error { 86 | r := bytes.NewReader(buf) 87 | return binary.Read(r, endian, h) 88 | } 89 | 90 | // Reset header implement proto.Message 91 | // 92 | // Since 0.1.6 93 | func (h *header) Reset() { 94 | *h = header{} 95 | } 96 | 97 | // String header implement proto.Message 98 | // 99 | // Since 0.1.6 100 | func (h *header) String() string { 101 | return fmt.Sprintf("%s %d %d", 102 | verStr(h.Version[:]), 103 | h.HeaderSize, 104 | h.BodySize) 105 | } 106 | 107 | // ProtoMessage header implement proto.Message 108 | // 109 | // Since 0.1.6 110 | func (h *header) ProtoMessage() {} 111 | 112 | func newHeader(ver string, bodysize uint64) *header { 113 | 114 | if len(ver) > versionLen { 115 | panic("version length overflow") 116 | } 117 | 118 | h := &header{ 119 | HeaderSize: uint64(fixedSize), 120 | BodySize: bodysize, 121 | } 122 | copy(h.Version[:], ver) 123 | return h 124 | } 125 | -------------------------------------------------------------------------------- /sigbits/firstdiff_test.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestGet64Bits(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input string 15 | want uint64 16 | }{ 17 | {"", 0}, 18 | {"a", 0x61 << 56}, 19 | {"abc", 0x616263 << 40}, 20 | {"abcd1234", 0x6162636431323334}, 21 | {"abcd1234x", 0x6162636431323334}, 22 | } 23 | 24 | for i, c := range cases { 25 | got := get64Bits(c.input) 26 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 27 | } 28 | } 29 | 30 | var ( 31 | Get64BitsInput string = "abcd1234x" 32 | Get64BitsOutput int 33 | ) 34 | 35 | func BenchmarkGet64Bits_9(b *testing.B) { 36 | Get64BitsInput = "abcd1234x" 37 | var s uint64 = 0 38 | for i := 0; i < b.N; i++ { 39 | s += get64Bits(Get64BitsInput) 40 | } 41 | Get64BitsOutput = int(s) 42 | } 43 | 44 | func BenchmarkGet64Bits_7(b *testing.B) { 45 | Get64BitsInput = "abcd123" 46 | var s uint64 = 0 47 | for i := 0; i < b.N; i++ { 48 | s += get64Bits(Get64BitsInput) 49 | } 50 | Get64BitsOutput = int(s) 51 | } 52 | 53 | func TestFirstDiffBit(t *testing.T) { 54 | 55 | ta := require.New(t) 56 | 57 | cases := []struct { 58 | a, b string 59 | want int32 60 | }{ 61 | {"", "", 0}, 62 | {"a", "b", 6}, 63 | {"aa", "ab", 14}, 64 | {"aa", "aa", 16}, 65 | {"aa", "aab", 16}, 66 | {"aac", "aa", 16}, 67 | {"aac", "ab", 14}, 68 | {"aaa", "aaa", 24}, 69 | {"12345678", "12345678", 64}, 70 | {"12345678a", "12345678", 64}, 71 | {"12345678", "12345678a", 64}, 72 | {"12345678a", "12345678a", 72}, 73 | {"12345678a", "12345678b", 70}, 74 | {"12345678aab", "12345678aa", 80}, 75 | {"12345678aa", "12345678aab", 80}, 76 | {"12345678aac", "12345678aab", 87}, 77 | } 78 | 79 | for i, c := range cases { 80 | got := sFirstDiffBit(c.a, c.b) 81 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 82 | } 83 | } 84 | 85 | func TestFirstDiffBits(t *testing.T) { 86 | 87 | ta := require.New(t) 88 | 89 | ta.Panics(func() { FirstDiffBits([]string{}) }) 90 | 91 | cases := []struct { 92 | keys []string 93 | want []int32 94 | }{ 95 | {[]string{""}, []int32{}}, 96 | {[]string{"a"}, []int32{}}, 97 | {[]string{"ab"}, []int32{}}, 98 | 99 | {[]string{"a", "b"}, []int32{6}}, 100 | {[]string{"aa", "ab"}, []int32{14}}, 101 | {[]string{"a", "aa"}, []int32{8}}, 102 | {[]string{"aa", "b"}, []int32{6}}, 103 | 104 | {[]string{"aa", "aab", "aac"}, []int32{16, 23}}, 105 | 106 | {[]string{ 107 | "aa", 108 | "ab", 109 | "abc", 110 | "abd", 111 | "ac", 112 | "b", 113 | "bbbbbbbb", 114 | "bbbbbbbba", 115 | "bbbbbbbbaa", 116 | "bbbbbbbbab", 117 | "bbbbbbbbc", 118 | "c", 119 | }, []int32{ 120 | 14, 121 | 16, 122 | 21, 123 | 15, 124 | 6, 125 | 8, 126 | 64, 127 | 72, 128 | 78, 129 | 70, 130 | 7, 131 | }}, 132 | } 133 | 134 | for i, c := range cases { 135 | got := FirstDiffBits(c.keys) 136 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 137 | } 138 | } 139 | -------------------------------------------------------------------------------- /bitmap/tailbitmap.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | // TailBitmap is a special case bitmap in which the bits before a certain index 4 | // are all set to `1`. 5 | // Thus we only need to store the sparse tail part. 6 | // 7 | // The data structure is as the following described: 8 | // 9 | // reclaimed 10 | // | 11 | // | Offset 12 | // | | 13 | // v v 14 | // ..... X ... 01010...00111 00... 15 | // bitIndex: 0123... ^ ^ 16 | // | | 17 | // Words[0] Words[1] 18 | // 19 | // Since 0.1.22 20 | type TailBitmap struct { 21 | // The bit index from which there are a `0` in it. 22 | // Before Offset all bits are set to `1` thus we donot need to store them. 23 | // Offset must be a `n*64`. 24 | Offset int64 25 | // Words is the tail part bitmap in which there are `0`. 26 | Words []uint64 27 | 28 | // reclaimed is the bit index from which the memory in Words are reclaimed. 29 | reclaimed int64 30 | } 31 | 32 | // reclaimThreshold is the size threshold in bit for reclamation of `Words`. 33 | var reclaimThreshold = int64(1024) * 64 34 | 35 | // NewTailBitmap creates an TailBitmap with a preset Offset and an empty 36 | // tail bitmap. 37 | // 38 | // Since 0.1.22 39 | func NewTailBitmap(offset int64) *TailBitmap { 40 | tb := &TailBitmap{ 41 | Offset: offset, 42 | reclaimed: offset, 43 | Words: make([]uint64, 0, reclaimThreshold>>6), 44 | } 45 | return tb 46 | } 47 | 48 | // Compact all leading all-ones words in the bitmap. 49 | // 50 | // Since 0.1.22 51 | func (tb *TailBitmap) Compact() { 52 | 53 | allOnes := uint64(0xffffffffffffffff) 54 | 55 | for len(tb.Words) > 0 && tb.Words[0] == allOnes { 56 | tb.Offset += 64 57 | tb.Words = tb.Words[1:] 58 | } 59 | 60 | if tb.Offset-tb.reclaimed >= reclaimThreshold { 61 | l := len(tb.Words) 62 | newWords := make([]uint64, l, l*2) 63 | 64 | copy(newWords, tb.Words) 65 | tb.reclaimed = tb.Offset 66 | } 67 | } 68 | 69 | // Set the bit at `idx` to `1`. 70 | // 71 | // Since 0.1.22 72 | func (tb *TailBitmap) Set(idx int64) { 73 | if idx < tb.Offset { 74 | return 75 | } 76 | 77 | idx = idx - tb.Offset 78 | wordIdx := idx >> 6 79 | 80 | for int(wordIdx) >= len(tb.Words) { 81 | tb.Words = append(tb.Words, 0) 82 | } 83 | 84 | tb.Words[wordIdx] |= Bit[idx&63] 85 | 86 | if wordIdx == 0 { 87 | tb.Compact() 88 | } 89 | } 90 | 91 | // Get retrieves a bit at its 64-based offset. 92 | // 93 | // Since 0.1.22 94 | func (tb *TailBitmap) Get(idx int64) uint64 { 95 | if idx < tb.Offset { 96 | return Bit[idx&63] 97 | } 98 | 99 | idx = idx - tb.Offset 100 | return tb.Words[idx>>6] & Bit[idx&63] 101 | } 102 | 103 | // Get1 retrieves a bit and returns a 1-bit word, i.e., putting the bit in the 104 | // lowest bit. 105 | // 106 | // Since 0.1.22 107 | func (tb *TailBitmap) Get1(idx int64) uint64 { 108 | if idx < tb.Offset { 109 | return 1 110 | } 111 | idx = idx - tb.Offset 112 | return (tb.Words[idx>>6] >> uint(idx&63)) & 1 113 | } 114 | -------------------------------------------------------------------------------- /mathext/zipf/zipf_test.go: -------------------------------------------------------------------------------- 1 | package zipf_test 2 | 3 | import ( 4 | "math/rand" 5 | "strings" 6 | "testing" 7 | 8 | "github.com/openacid/low/mathext/zipf" 9 | "github.com/stretchr/testify/require" 10 | ) 11 | 12 | func TestZipf(t *testing.T) { 13 | 14 | ta := require.New(t) 15 | 16 | n := 100 17 | 18 | a := float64(1) 19 | b := float64(n) 20 | s := float64(1.5) 21 | 22 | got := make([]int, n) 23 | z := zipf.New(a, b, s) 24 | 25 | want := []int{0, 3255, 1442, 859, 587, 433, 336, 271, 225, 190, 163, 143, 26 | 126, 112, 101, 91, 83, 76, 70, 64, 60, 56, 52, 49, 45, 44, 40, 39, 36, 35, 27 | 33, 31, 30, 29, 27, 27, 25, 24, 23, 23, 21, 21, 20, 20, 18, 18, 18, 17, 16, 28 | 16, 16, 15, 14, 15, 13, 14, 13, 13, 12, 12, 12, 11, 12, 11, 10, 11, 10, 10, 29 | 10, 10, 9, 9, 9, 9, 9, 8, 8, 9, 8, 7, 8, 8, 7, 7, 7, 7, 7, 7, 7, 6, 7, 6, 6, 30 | 7, 6, 6, 5, 6, 6, 6} 31 | 32 | sampleCnt := float64(10000) 33 | for u := float64(0); u < 1; u += 1 / sampleCnt { 34 | x := int(z.Float64(float64(u))) 35 | got[x]++ 36 | } 37 | 38 | ta.Equal(want, got) 39 | } 40 | 41 | func TestAccess(t *testing.T) { 42 | 43 | ta := require.New(t) 44 | 45 | a := float64(1) 46 | s := float64(1.5) 47 | 48 | want := ` 49 | ** 50 | ** 51 | ** 52 | *** 53 | ************************************************************************ 54 | ***** 55 | 56 | ** 57 | ************ 58 | ***************** 59 | * 60 | ** 61 | * 62 | ******************** 63 | ****************************** 64 | ******** 65 | ****** 66 | ** 67 | ****** 68 | *******`[1:] 69 | 70 | got := zipf.Accesses(a, s, 20, 200, nil) 71 | _, g := makeSample(20, got) 72 | ta.Equal(want, g) 73 | } 74 | 75 | func makeSample(n int, accesses []int) ([]int, string) { 76 | arr := make([]int, n) 77 | for _, idx := range accesses { 78 | arr[idx]++ 79 | } 80 | 81 | lines := make([]string, n) 82 | for i, v := range arr { 83 | lines[i] = strings.Repeat("*", v) 84 | } 85 | 86 | return arr, strings.Join(lines, "\n") 87 | } 88 | 89 | func TestOfficalZipf(t *testing.T) { 90 | 91 | // Not to run. Only when I want. 92 | t.Skip() 93 | 94 | start := 1 95 | end := 100 96 | n := end 97 | s := float64(1.5) 98 | 99 | ta := require.New(t) 100 | _ = ta 101 | 102 | sample := make([]int, n) 103 | r := rand.New(rand.NewSource(44)) 104 | 105 | sampleCnt := float64(10000) 106 | rz := rand.NewZipf(r, s, float64(start), uint64(end-start)) 107 | for i := 0; i < int(sampleCnt); i++ { 108 | x := rz.Uint64() 109 | sample[x]++ 110 | } 111 | } 112 | 113 | var Output float64 114 | 115 | func BenchmarkZipf(b *testing.B) { 116 | start := 10 117 | end := 10000 118 | a := float64(start) 119 | bb := float64(end) 120 | s := float64(1.5) 121 | 122 | z := zipf.New(a, bb, s) 123 | ss := float64(0) 124 | for i := 0; i < b.N; i++ { 125 | v := z.Float64(float64(i) / float64(b.N)) 126 | ss += v 127 | } 128 | 129 | Output = ss 130 | } 131 | 132 | func BenchmarkOfficialZipf(b *testing.B) { 133 | 134 | start := 10 135 | end := 10000 136 | s := float64(1.5) 137 | r := rand.New(rand.NewSource(44)) 138 | 139 | rz := rand.NewZipf(r, s, float64(start), uint64(end-start)) 140 | ss := uint64(0) 141 | for i := 0; i < b.N; i++ { 142 | x := rz.Uint64() 143 | ss += x 144 | } 145 | 146 | Output = float64(ss) 147 | } 148 | -------------------------------------------------------------------------------- /common.mk: -------------------------------------------------------------------------------- 1 | # Usage: echo 'include common.mk' > Makefile 2 | 3 | TOP_PKG := $(shell go list .) 4 | NAME := $(shell go list . | awk -F/ '{print $$NF}') 5 | PKGS := $(shell go list ./... | grep -v "^$(TOP_PKG)/\(vendor\|prototype\)") 6 | 7 | # PKGS := github.com/openacid/slimarray/array \ 8 | # github.com/openacid/slimarray/bit \ 9 | # github.com/openacid/slimarray/trie \ 10 | 11 | SRCDIRS := $(shell go list -f '{{.Dir}}' $(PKGS)) 12 | 13 | # gofmt check vendor dir. we need to skip vendor manually 14 | GOFILES := $(shell find $(SRCDIRS) -not -path "*/vendor/*" -name "*.go") 15 | GO := go 16 | 17 | check: test vet gofmt misspell unconvert staticcheck ineffassign unparam 18 | 19 | travis: vet gofmt misspell unconvert ineffassign unparam test 20 | 21 | test: 22 | # fail fast with severe bugs 23 | $(GO) test -short $(PKGS) 24 | $(GO) test -tags debug $(PKGS) 25 | # test release version and generate coverage data for task `coveralls`. 26 | $(GO) test -covermode=count -coverprofile=coverage.out $(PKGS) 27 | 28 | vet: 29 | $(GO) vet $(PKGS) 30 | 31 | staticcheck: 32 | $(GO) get honnef.co/go/tools/cmd/staticcheck 33 | # ST1016: methods on the same type should have the same receiver name 34 | # .pb.go have this issue. 35 | staticcheck -checks all,-ST1016 $(PKGS) 36 | 37 | misspell: 38 | $(GO) get github.com/client9/misspell/cmd/misspell 39 | find $(SRCDIRS) -name '*.go' -or -name '*.md' | grep -v "\bvendor/" | xargs misspell \ 40 | -locale US \ 41 | -error 42 | misspell \ 43 | -locale US \ 44 | -error \ 45 | *.md *.go 46 | 47 | unconvert: 48 | $(GO) get github.com/mdempsky/unconvert 49 | unconvert -v $(PKGS) 50 | 51 | ineffassign: 52 | $(GO) get github.com/gordonklaus/ineffassign 53 | find $(SRCDIRS) -name '*.go' | grep -v "\bvendor/" | xargs ineffassign 54 | 55 | pedantic: check errcheck 56 | 57 | unparam: 58 | $(GO) get mvdan.cc/unparam 59 | unparam ./... 60 | 61 | errcheck: 62 | $(GO) get github.com/kisielk/errcheck 63 | errcheck $(PKGS) 64 | 65 | gofmt: 66 | @echo Checking code is gofmted 67 | @test -z "$(shell gofmt -s -l -d -e $(GOFILES) | tee /dev/stderr)" 68 | 69 | ben: test 70 | $(GO) test ./... -run=none -bench=. -benchmem 71 | 72 | gen: 73 | $(GO) generate ./... 74 | 75 | doc: 76 | # build a markdown version of package doc to embed to README.md 77 | # $(GO) get github.com/robertkrimen/godocdown/godocdown 78 | godocdown . > docs/$(NAME).md 79 | # "package" is the first phrase in a go doc. 80 | # "## Usage" is the start of API section. 81 | cat docs/$(NAME).md | awk '/^package /,/^## Usage/' | grep -v '^## Usage' > docs/$(NAME)-package.md 82 | 83 | 84 | readme: doc 85 | python ./scripts/build_md.py 86 | # brew install nodejs 87 | # npm install -g doctoc 88 | doctoc --title '' --github README.md 89 | 90 | fix: 91 | gofmt -s -w $(GOFILES) 92 | unconvert -v -apply $(PKGS) 93 | 94 | 95 | # local coverage 96 | coverage: 97 | $(GO) test -covermode=count -coverprofile=coverage.out $(PKGS) 98 | go tool cover -func=coverage.out 99 | # go tool cover -html=coverage.out 100 | 101 | # send coverage to coveralls 102 | coveralls: 103 | # this job relies on the output of task test: `-coverprofile=coverage.out` 104 | $(GO) get golang.org/x/tools/cmd/cover 105 | $(GO) get github.com/mattn/goveralls 106 | goveralls -ignore='*.pb.go' -coverprofile=coverage.out -service=travis-ci 107 | -------------------------------------------------------------------------------- /bitstr/bitstr.go: -------------------------------------------------------------------------------- 1 | package bitstr 2 | 3 | import ( 4 | "bytes" 5 | "math/bits" 6 | "unsafe" 7 | 8 | "github.com/openacid/low/bitmap" 9 | ) 10 | 11 | // New extracts bits from fromBit to toBit and save it into bytes. 12 | // The fromBit must be aligned to 8, or it will be truncated to 8*n. 13 | // The bits after toBit are discard. 14 | // 15 | // To describe non-8 aligned bits, a trailing byte of the effective number of 16 | // bits in the last payload byte is appended. 17 | // 18 | // E.g.: 19 | // // "abc" = 0110 0001, 0110 0010, 0110 0011 20 | // New("abc", 5, 12) 21 | // // returns 3 byte: 22 | // // 0110 0001, 0110 0000, 1111, 0000 23 | // // --------------- ---------- 24 | // // payload trailing byte: 25 | // // the higher 4 bit in the last(2nd) byte. 26 | // 27 | // Since 0.1.20 28 | func New(s string, fromBit, toBit int32) []byte { 29 | 30 | if fromBit == toBit && fromBit&7 == 0 { 31 | return []byte{0xff} 32 | } 33 | 34 | fromByte := fromBit >> 3 35 | toByte := (toBit + 7) >> 3 36 | 37 | l := toByte - fromByte 38 | 39 | bitStr := make([]byte, l+1) 40 | copy(bitStr, s[fromBit>>3:toByte]) 41 | 42 | mask := byte(bitmap.RMask[(8-toBit)&7]) 43 | 44 | // truncate bits after `toBit` 45 | bitStr[l-1] &= mask 46 | 47 | // the last byte stores number of effective bits in the last payload byte. 48 | bitStr[l] = mask 49 | 50 | return bitStr 51 | } 52 | 53 | // Cmp compares two bitStr and returns -1, 0, 1 for less-than, equal, greater-than. 54 | // 55 | // Since 0.1.20 56 | func Cmp(a, b []byte) int { 57 | la, lb := len(a), len(b) 58 | if la == lb { 59 | return bytes.Compare(a, b) 60 | } 61 | 62 | return bytes.Compare(a[:la-1], b[:lb-1]) 63 | } 64 | 65 | func cmpBytes(a []byte, b []byte) int { 66 | 67 | la, lb := len(a), len(b) 68 | if la < 8 { 69 | var i int 70 | for i = 0; i < la; i++ { 71 | if a[i] < b[i] { 72 | return -1 73 | } else if a[i] > b[i] { 74 | return 1 75 | } 76 | } 77 | 78 | if i < lb { 79 | return -1 80 | } 81 | return 0 82 | } 83 | return bytes.Compare(a, b) 84 | } 85 | 86 | // CmpUpto compares a normal []byte `a` and a bitStr `b` with `a` being 87 | // TRUNCATED upto `Len(b)`. 88 | // 89 | // Performance 90 | // >=8 bytes: TODO 91 | // <8 bytes: TODO 92 | // 93 | // Since 0.1.20 94 | func CmpUpto(a, b []byte) int { 95 | la, lb := len(a), len(b) 96 | if lb == 1 { 97 | // an empty bitStr 98 | return 0 99 | } 100 | 101 | if la < lb-1 { 102 | return cmpBytes(a, b[:lb-1]) 103 | } 104 | 105 | // la >= lb-1 106 | 107 | la = lb - 1 108 | 109 | rst := cmpBytes(a[:lb-2], b[:lb-2]) 110 | if rst != 0 { 111 | return rst 112 | } 113 | 114 | // compare the last byte 115 | 116 | bytea := a[la-1] & b[lb-1] 117 | byteb := b[la-1] 118 | 119 | if bytea > byteb { 120 | return 1 121 | } else if bytea < byteb { 122 | return -1 123 | } 124 | 125 | return 0 126 | } 127 | 128 | // StrCmpUpto is similar to CmpUpto, except the first arg is a string instead of 129 | // []byte. 130 | // 131 | // Since 0.1.20 132 | func StrCmpUpto(a string, b []byte) int { 133 | return CmpUpto(*(*[]byte)(unsafe.Pointer(&a)), b) 134 | } 135 | 136 | // Len returns the number of payload bits in a bitStr. 137 | // 138 | // Since 0.1.20 139 | func Len(bs []byte) int32 { 140 | l := len(bs) 141 | 142 | // -16: exclude the tailing byte, and the last payload byte 143 | return int32(l)<<3 - 16 + int32(bits.OnesCount8(bs[l-1])) 144 | } 145 | -------------------------------------------------------------------------------- /bitmap/next_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestNextOne(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | bm []uint64 15 | from int32 16 | n int32 17 | want int32 18 | }{ 19 | {[]uint64{0}, 0, 1, -1}, 20 | {[]uint64{0}, 0, 63, -1}, 21 | {[]uint64{0}, 0, 64, -1}, 22 | 23 | {[]uint64{1}, 0, 0, -1}, 24 | {[]uint64{1}, 0, 1, 0}, 25 | {[]uint64{1}, 0, 2, 0}, 26 | {[]uint64{1}, 1, 2, -1}, 27 | 28 | {[]uint64{0x02}, 0, 0, -1}, 29 | {[]uint64{0x02}, 0, 1, -1}, 30 | {[]uint64{0x02}, 0, 2, 1}, 31 | {[]uint64{0x02}, 1, 2, 1}, 32 | {[]uint64{0x02}, 2, 2, -1}, 33 | {[]uint64{0x02}, 2, 63, -1}, 34 | {[]uint64{0x02}, 2, 64, -1}, 35 | 36 | {[]uint64{0xff, 0x82}, 0, 72, 0}, 37 | {[]uint64{0xff, 0x82}, 1, 72, 1}, 38 | {[]uint64{0xff, 0x82}, 63, 64, -1}, 39 | {[]uint64{0xff, 0x82}, 64, 65, -1}, 40 | {[]uint64{0xff, 0x82}, 64, 66, 65}, 41 | {[]uint64{0xff, 0x82}, 65, 66, 65}, 42 | {[]uint64{0xff, 0x82}, 65, 72, 65}, 43 | {[]uint64{0xff, 0x82}, 66, 71, -1}, 44 | {[]uint64{0xff, 0x82}, 71, 72, 71}, 45 | {[]uint64{0xff, 0x82}, 72, 128, -1}, 46 | } 47 | 48 | for i, c := range cases { 49 | got := NextOne(c.bm, c.from, c.n) 50 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 51 | } 52 | } 53 | 54 | func TestPrevOne(t *testing.T) { 55 | 56 | ta := require.New(t) 57 | 58 | cases := []struct { 59 | bm []uint64 60 | from int32 61 | n int32 62 | want int32 63 | }{ 64 | {[]uint64{0}, 0, 1, -1}, 65 | {[]uint64{0}, 0, 63, -1}, 66 | {[]uint64{0}, 0, 64, -1}, 67 | 68 | {[]uint64{1}, 0, 1, 0}, 69 | {[]uint64{1}, 0, 2, 0}, 70 | {[]uint64{1}, 1, 2, -1}, 71 | 72 | {[]uint64{0x02}, 0, 1, -1}, 73 | {[]uint64{0x02}, 0, 2, 1}, 74 | {[]uint64{0x02}, 1, 1, -1}, 75 | {[]uint64{0x02}, 1, 2, 1}, 76 | {[]uint64{0x02}, 1, 3, 1}, 77 | {[]uint64{0x02}, 2, 2, -1}, 78 | {[]uint64{0x02}, 2, 3, -1}, 79 | {[]uint64{0x02}, 2, 63, -1}, 80 | {[]uint64{0x02}, 2, 64, -1}, 81 | 82 | {[]uint64{0xff, 0x82}, 0, 63, 7}, 83 | {[]uint64{0xff, 0x82}, 0, 64, 7}, 84 | {[]uint64{0xff, 0x82}, 0, 65, 7}, 85 | {[]uint64{0xff, 0x82}, 0, 66, 65}, 86 | {[]uint64{0xff, 0x82}, 0, 72, 71}, 87 | {[]uint64{0xff, 0x82}, 1, 72, 71}, 88 | {[]uint64{0xff, 0x82}, 1, 71, 65}, 89 | {[]uint64{0xff, 0x82}, 1, 64, 7}, 90 | {[]uint64{0xff, 0x82}, 63, 64, -1}, 91 | {[]uint64{0xff, 0x82}, 64, 65, -1}, 92 | {[]uint64{0xff, 0x82}, 64, 66, 65}, 93 | {[]uint64{0xff, 0x82}, 65, 66, 65}, 94 | {[]uint64{0xff, 0x82}, 65, 72, 71}, 95 | {[]uint64{0xff, 0x82}, 66, 71, -1}, 96 | {[]uint64{0xff, 0x82}, 71, 72, 71}, 97 | {[]uint64{0xff, 0x82}, 72, 128, -1}, 98 | 99 | {[]uint64{0xffffffffffffffff, 0x82}, 63, 64, 63}, 100 | {[]uint64{0xffffffffffffffff, 0x82}, 63, 65, 63}, 101 | {[]uint64{0xffffffffffffffff, 0x82}, 63, 66, 65}, 102 | } 103 | 104 | for i, c := range cases { 105 | got := PrevOne(c.bm, c.from, c.n) 106 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 107 | } 108 | } 109 | 110 | func TestPrevOne_panic(t *testing.T) { 111 | 112 | // zero-width range will panic 113 | 114 | ta := require.New(t) 115 | 116 | cases := []struct { 117 | bm []uint64 118 | from int32 119 | n int32 120 | want int32 121 | }{ 122 | {[]uint64{0}, 0, 0, -1}, 123 | {[]uint64{1}, 0, 0, -1}, 124 | } 125 | 126 | for _, c := range cases { 127 | ta.Panics(func(){ 128 | PrevOne(c.bm,c.from,c.n) 129 | }, "case: %v", c) 130 | } 131 | } 132 | -------------------------------------------------------------------------------- /bitmap/of_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestOf(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | bitPositions []int32 15 | wantwords []uint64 16 | }{ 17 | { 18 | []int32{}, 19 | []uint64{}, 20 | }, 21 | { 22 | []int32{0}, 23 | []uint64{1}, 24 | }, 25 | { 26 | []int32{0, 1, 2}, 27 | []uint64{7}, 28 | }, 29 | { 30 | []int32{0, 1, 2, 63}, 31 | []uint64{(1 << 63) + 7}, 32 | }, 33 | { 34 | []int32{64}, 35 | []uint64{0, 1}, 36 | }, 37 | { 38 | []int32{1, 2, 3, 64, 129}, 39 | []uint64{0x0e, 1, 2}, 40 | }, 41 | } 42 | 43 | for i, c := range cases { 44 | 45 | got := Of(c.bitPositions) 46 | 47 | ta.Equal(c.wantwords, got, 48 | "%d-th: case: %+v", 49 | i+1, c) 50 | } 51 | } 52 | 53 | func TestOf_with_n(t *testing.T) { 54 | 55 | ta := require.New(t) 56 | 57 | cases := []struct { 58 | bitPositions []int32 59 | n int32 60 | wantwords []uint64 61 | }{ 62 | // 0 bit 63 | { 64 | []int32{}, 0, 65 | []uint64{}, 66 | }, 67 | { 68 | []int32{}, -1, 69 | []uint64{}, 70 | }, 71 | { 72 | []int32{}, -100, 73 | []uint64{}, 74 | }, 75 | { 76 | []int32{}, 1, 77 | []uint64{0}, 78 | }, 79 | { 80 | []int32{}, 63, 81 | []uint64{0}, 82 | }, 83 | { 84 | []int32{}, 64, 85 | []uint64{0}, 86 | }, 87 | { 88 | []int32{}, 65, 89 | []uint64{0, 0}, 90 | }, 91 | // 1 bit 92 | { 93 | []int32{0}, 0, 94 | []uint64{1}, 95 | }, 96 | { 97 | []int32{0}, -1, 98 | []uint64{1}, 99 | }, 100 | { 101 | []int32{0}, -100, 102 | []uint64{1}, 103 | }, 104 | { 105 | []int32{0}, 1, 106 | []uint64{1}, 107 | }, 108 | { 109 | []int32{0}, 2, 110 | []uint64{1}, 111 | }, 112 | { 113 | []int32{0}, 63, 114 | []uint64{1}, 115 | }, 116 | { 117 | []int32{0}, 64, 118 | []uint64{1}, 119 | }, 120 | { 121 | []int32{0}, 65, 122 | []uint64{1, 0}, 123 | }, 124 | 125 | // more than one bit 126 | { 127 | []int32{0, 1, 2}, 0, 128 | []uint64{7}, 129 | }, 130 | { 131 | []int32{0, 1, 2}, -1, 132 | []uint64{7}, 133 | }, 134 | { 135 | []int32{0, 1, 2}, -100, 136 | []uint64{7}, 137 | }, 138 | { 139 | []int32{0, 1, 2}, 63, 140 | []uint64{7}, 141 | }, 142 | { 143 | []int32{0, 1, 2}, 64, 144 | []uint64{7}, 145 | }, 146 | { 147 | []int32{0, 1, 2}, 65, 148 | []uint64{7, 0}, 149 | }, 150 | { 151 | []int32{0, 1, 2}, 127, 152 | []uint64{7, 0}, 153 | }, 154 | { 155 | []int32{0, 1, 2}, 128, 156 | []uint64{7, 0}, 157 | }, 158 | { 159 | []int32{0, 1, 2}, 129, 160 | []uint64{7, 0, 0}, 161 | }, 162 | 163 | // 1 bit in 2nd word 164 | { 165 | []int32{64}, 0, 166 | []uint64{0, 1}, 167 | }, 168 | { 169 | []int32{64}, -1, 170 | []uint64{0, 1}, 171 | }, 172 | { 173 | []int32{64}, -100, 174 | []uint64{0, 1}, 175 | }, 176 | { 177 | []int32{64}, 1, 178 | []uint64{0, 1}, 179 | }, 180 | { 181 | []int32{64}, 65, 182 | []uint64{0, 1}, 183 | }, 184 | { 185 | []int32{64}, 127, 186 | []uint64{0, 1}, 187 | }, 188 | { 189 | []int32{64}, 128, 190 | []uint64{0, 1}, 191 | }, 192 | { 193 | []int32{64}, 129, 194 | []uint64{0, 1, 0}, 195 | }, 196 | } 197 | 198 | for i, c := range cases { 199 | 200 | got := Of(c.bitPositions, c.n) 201 | 202 | ta.Equal(c.wantwords, got, 203 | "%d-th: case: %+v", 204 | i+1, c) 205 | } 206 | } 207 | -------------------------------------------------------------------------------- /bitmap/get_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestGet(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | ta.Panics(func() { Get(nil, 0) }) 14 | ta.Panics(func() { Get([]uint64{}, 0) }) 15 | ta.Panics(func() { Get([]uint64{}, 1) }) 16 | ta.Panics(func() { Get([]uint64{}, -1) }) 17 | ta.Panics(func() { Get([]uint64{0}, -1) }) 18 | ta.Panics(func() { Get([]uint64{0}, 64) }) 19 | 20 | cases := []struct { 21 | input []uint64 22 | n int32 23 | want uint64 24 | }{ 25 | {[]uint64{0}, 0, 0}, 26 | {[]uint64{0}, 1, 0}, 27 | {[]uint64{1}, 0, 1}, 28 | {[]uint64{1}, 1, 0}, 29 | {[]uint64{12}, 1, 0}, 30 | {[]uint64{12}, 2, 4}, 31 | {[]uint64{12}, 3, 8}, 32 | {[]uint64{32, 12}, 5, 32}, 33 | {[]uint64{32, 12}, 64, 0}, 34 | {[]uint64{32, 12}, 66, 4}, 35 | {[]uint64{32, 12}, 67, 8}, 36 | } 37 | 38 | for i, c := range cases { 39 | got := Get(c.input, c.n) 40 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 41 | 42 | got = Get1(c.input, c.n) 43 | want := uint64(0) 44 | if c.want != 0 { 45 | want = 1 46 | } 47 | ta.Equal(want, got, "%d-th: case: %+v", i+1, c) 48 | } 49 | } 50 | 51 | func TestGetw(t *testing.T) { 52 | 53 | ta := require.New(t) 54 | 55 | ta.Panics(func() { Getw(nil, 0, 1) }) 56 | ta.Panics(func() { Getw([]uint64{}, 0, 1) }) 57 | ta.Panics(func() { Getw([]uint64{}, 1, 1) }) 58 | ta.Panics(func() { Getw([]uint64{}, -1, 1) }) 59 | ta.Panics(func() { Getw([]uint64{0}, -1, 1) }) 60 | ta.Panics(func() { Getw([]uint64{0}, 64, 1) }) 61 | 62 | cases := []struct { 63 | bm []uint64 64 | n int32 65 | w int32 66 | want uint64 67 | }{ 68 | {[]uint64{0}, 0, 1, 0}, 69 | {[]uint64{0}, 1, 1, 0}, 70 | {[]uint64{1}, 0, 1, 1}, 71 | {[]uint64{1}, 1, 1, 0}, 72 | {[]uint64{12}, 1, 1, 0}, 73 | {[]uint64{12}, 2, 1, 1}, 74 | {[]uint64{12}, 3, 1, 1}, 75 | {[]uint64{32, 12}, 5, 1, 1}, 76 | {[]uint64{32, 12}, 64, 1, 0}, 77 | {[]uint64{32, 12}, 66, 1, 1}, 78 | {[]uint64{32, 12}, 67, 1, 1}, 79 | 80 | {[]uint64{0}, 0, 2, 0}, 81 | {[]uint64{0}, 1, 2, 0}, 82 | {[]uint64{1}, 0, 2, 1}, 83 | {[]uint64{1}, 1, 2, 0}, 84 | {[]uint64{12}, 0, 2, 0}, 85 | {[]uint64{12}, 1, 2, 3}, 86 | {[]uint64{12}, 2, 2, 0}, 87 | {[]uint64{32, 12}, 1, 2, 0}, 88 | {[]uint64{32, 12}, 2, 2, 2}, 89 | {[]uint64{32, 12}, 3, 2, 0}, 90 | {[]uint64{32, 12}, 32, 2, 0}, 91 | {[]uint64{32, 12}, 33, 2, 3}, 92 | {[]uint64{32, 12}, 34, 2, 0}, 93 | } 94 | 95 | for i, c := range cases { 96 | got := Getw(c.bm, c.n, c.w) 97 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 98 | } 99 | } 100 | 101 | func TestSafeGet(t *testing.T) { 102 | 103 | ta := require.New(t) 104 | 105 | cases := []struct { 106 | input []uint64 107 | n int32 108 | want uint64 109 | }{ 110 | {nil, 0, 0}, 111 | {[]uint64{}, 0, 0}, 112 | {[]uint64{}, 1, 0}, 113 | {[]uint64{}, -1, 0}, 114 | {[]uint64{0}, -1, 0}, 115 | {[]uint64{0}, 64, 0}, 116 | 117 | {[]uint64{0}, 0, 0}, 118 | {[]uint64{0}, 1, 0}, 119 | {[]uint64{1}, 0, 1}, 120 | {[]uint64{1}, 1, 0}, 121 | {[]uint64{12}, 1, 0}, 122 | {[]uint64{12}, 2, 4}, 123 | {[]uint64{12}, 3, 8}, 124 | {[]uint64{32, 12}, 5, 32}, 125 | {[]uint64{32, 12}, 64, 0}, 126 | {[]uint64{32, 12}, 66, 4}, 127 | {[]uint64{32, 12}, 67, 8}, 128 | } 129 | 130 | for i, c := range cases { 131 | got := SafeGet(c.input, c.n) 132 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 133 | 134 | got = SafeGet1(c.input, c.n) 135 | want := uint64(0) 136 | if c.want != 0 { 137 | want = 1 138 | } 139 | ta.Equal(want, got, "%d-th: case: %+v", i+1, c) 140 | } 141 | } 142 | -------------------------------------------------------------------------------- /sigbits/sharding_test.go: -------------------------------------------------------------------------------- 1 | package sigbits 2 | 3 | import ( 4 | "strings" 5 | "testing" 6 | 7 | "github.com/openacid/testkeys" 8 | "github.com/stretchr/testify/require" 9 | ) 10 | 11 | func TestShardByPrefix(t *testing.T) { 12 | 13 | ta := require.New(t) 14 | 15 | split := func(ks string) []string { 16 | ks = strings.Trim(ks, "\n ") 17 | return strings.Split(ks, "\n") 18 | } 19 | 20 | makeRst := func(keys []string, prefLens, starts []int32) []string { 21 | rst := make([]string, 0, len(keys)) 22 | for j := 0; j < len(starts)-1; j++ { 23 | s := starts[j] 24 | e := starts[j+1] 25 | prefLen := prefLens[j] 26 | 27 | rst = append(rst, "="+keys[s]) 28 | 29 | for k := s + 1; k < e; k++ { 30 | rst = append(rst, "-"+strings.Repeat(" ", int(prefLen))+keys[k][prefLen:]) 31 | } 32 | 33 | } 34 | 35 | return rst 36 | } 37 | 38 | keysAbe_Abo := ` 39 | Abe 40 | Abel 41 | Abelia 42 | Abelian 43 | Abelicea 44 | Abelite 45 | Abelmoschus 46 | Abelonian 47 | Abencerrages 48 | Aberdeen 49 | Aberdonian 50 | Aberia 51 | Abhorson 52 | Abie 53 | Abies 54 | Abietineae 55 | Abiezer 56 | Abigail 57 | Abipon 58 | Abitibi 59 | Abkhas 60 | Abkhasian 61 | Ablepharus 62 | Abnaki 63 | Abner 64 | Abo 65 | ` 66 | 67 | keysAaron_Abuta := ` 68 | Aaron 69 | Aaronic 70 | Aaronical 71 | Aaronite 72 | Aaronitic 73 | Aaru 74 | Ab 75 | Ababdeh 76 | Ababua 77 | Abadite 78 | Abama 79 | Abanic 80 | Abantes 81 | Abarambo 82 | Abaris 83 | Abasgi 84 | Abassin 85 | Abatua 86 | Abba 87 | Abbadide 88 | Abbasside 89 | Abbie 90 | Abby 91 | Abderian 92 | Abderite 93 | Abdiel 94 | Abdominales 95 | Abe 96 | Abel 97 | Abelia 98 | Abelian 99 | Abelicea 100 | Abelite 101 | Abelmoschus 102 | Abelonian 103 | Abencerrages 104 | Aberdeen 105 | Aberdonian 106 | Aberia 107 | Abhorson 108 | Abie 109 | Abies 110 | Abietineae 111 | Abiezer 112 | Abigail 113 | Abipon 114 | Abitibi 115 | Abkhas 116 | Abkhasian 117 | Ablepharus 118 | Abnaki 119 | Abner 120 | Abo 121 | Abobra 122 | Abongo 123 | Abraham 124 | Abrahamic 125 | Abrahamidae 126 | Abrahamite 127 | Abrahamitic 128 | Abram 129 | Abramis 130 | Abranchiata 131 | Abrocoma 132 | Abroma 133 | Abronia 134 | Abrus 135 | Absalom 136 | Absaroka 137 | Absi 138 | Absyrtus 139 | Abu 140 | Abundantia 141 | Abuta 142 | ` 143 | 144 | cases := []struct { 145 | keys []string 146 | maxSize int32 147 | }{ 148 | { 149 | keys: split(keysAbe_Abo), 150 | maxSize: 5, 151 | }, 152 | { 153 | keys: split(keysAbe_Abo), 154 | maxSize: 7, 155 | }, 156 | { 157 | keys: split(keysAaron_Abuta), 158 | maxSize: 7, 159 | }, 160 | { 161 | keys: testkeys.Load("200kweb2"), 162 | maxSize: 16, 163 | }, 164 | { 165 | keys: testkeys.Load("200kweb2"), 166 | maxSize: 64, 167 | }, 168 | } 169 | 170 | for _, c := range cases { 171 | keys := c.keys 172 | prefLens, starts := ShardByPrefix(keys, c.maxSize) 173 | 174 | got := makeRst(keys, prefLens, starts) 175 | 176 | // fmt.Println(strings.Join(got, "\n")) 177 | 178 | ta.Equal(len(keys), len(got)) 179 | 180 | // check if prefixes are all in ascending order. 181 | for j := 1; j < len(prefLens); j++ { 182 | prev := keys[starts[j-1]][:prefLens[j-1]] 183 | shd := keys[starts[j]][:prefLens[j]] 184 | 185 | ta.GreaterOrEqual(shd, prev, "compare %d th pref with %d th", j-1, j) 186 | } 187 | 188 | // check keys can be rebuilt. 189 | for j := 0; j < len(prefLens); j++ { 190 | 191 | ta.LessOrEqual(starts[j+1]-starts[j], c.maxSize) 192 | 193 | pref := keys[starts[j]][:prefLens[j]] 194 | 195 | for k := starts[j]; k < starts[j+1]; k++ { 196 | ta.Equal(pref, keys[k][:len(pref)], "%d th key", k) 197 | } 198 | } 199 | 200 | // fmt.Println(len(keys) / len(prefLens)) 201 | } 202 | } 203 | -------------------------------------------------------------------------------- /tree/tree.go: -------------------------------------------------------------------------------- 1 | // Package tree provides abstract tree operations. 2 | // 3 | // Since 0.1.5 4 | package tree 5 | 6 | import ( 7 | "fmt" 8 | "strings" 9 | ) 10 | 11 | // Tree defines required functions to convert a tree-like data structure into 12 | // string. 13 | // 14 | // A nil indicates root node. 15 | // 16 | // Since 0.1.5 17 | type Tree interface { 18 | 19 | // Child returns the child of a node 20 | // 21 | // Since 0.1.5 22 | Child(node, label interface{}) interface{} 23 | 24 | // Labels returns the outgoing branch labels from a node. 25 | // 26 | // Since 0.1.5 27 | Labels(node interface{}) []interface{} 28 | 29 | // NodeID returns the a node id string. 30 | // 31 | // Since 0.1.5 32 | NodeID(node interface{}) string 33 | 34 | // LabelInfo returns a string representation of a branch label. 35 | // 36 | // Since 0.1.5 37 | LabelInfo(label interface{}) string 38 | 39 | // NodeInfo returns string describing the node. 40 | // 41 | // Since 0.1.5 42 | NodeInfo(node interface{}) string 43 | 44 | // LeafVal returns the value if the node is leaf. 45 | // 46 | // Since 0.1.5 47 | LeafVal(node interface{}) (interface{}, bool) 48 | } 49 | 50 | // String converts a Tree into a multiline string. 51 | // 52 | // A tree node is in form of 53 | // ->+*= 54 | // E.g.: 55 | // 000->#000+2*3 56 | // 001->#001+4*2 57 | // 003->#004+1=0 58 | // 006->#007+2=1 59 | // 004->#005+1=2 60 | // 006->#008+2=3 61 | // 002->#002+3=4 62 | // 006->#006+2=5 63 | // 006->#009+2=6 64 | // 003->#003+5=7`[1:] 65 | // 66 | // Since 0.1.5 67 | func String(t Tree) string { 68 | lines := toStrings(t, nil, nil) 69 | return strings.Join(lines, "\n") 70 | } 71 | 72 | // DepthFirst walks through a tree in a depth-first manner: process children in 73 | // order then their parent. 74 | // 75 | // Since 0.1.5 76 | func DepthFirst(t Tree, np func(t Tree, parent, label, node interface{})) { 77 | depthFirst(t, nil, nil, t.Child(nil, nil), np) 78 | } 79 | 80 | func depthFirst(t Tree, parent, label, node interface{}, np func(t Tree, parent, label, node interface{})) { 81 | 82 | for _, b := range t.Labels(node) { 83 | child := t.Child(node, b) 84 | depthFirst(t, node, b, child, np) 85 | } 86 | np(t, parent, label, node) 87 | } 88 | 89 | // toStrings converts a node and its subtree into multi strings. 90 | // 91 | // Since 0.1.5 92 | func toStrings(t Tree, inbranch, node interface{}) []string { 93 | 94 | line, ind := nodeStr(t, inbranch, node) 95 | 96 | indent := strings.Repeat(" ", ind) 97 | 98 | rst := make([]string, 0, 64) 99 | rst = append(rst, line) 100 | 101 | for _, b := range t.Labels(node) { 102 | sub := toStrings(t, b, t.Child(node, b)) 103 | for _, s := range sub { 104 | rst = append(rst, indent+s) 105 | } 106 | } 107 | return rst 108 | } 109 | 110 | // nodeStr returns a string representation of a node, and subtree indent. 111 | // 112 | // Since 0.1.5 113 | func nodeStr(t Tree, inbranch, node interface{}) (string, int) { 114 | 115 | var line string 116 | 117 | if inbranch != nil { 118 | line += fmt.Sprintf("-%v->", t.LabelInfo(inbranch)) 119 | } 120 | 121 | nodeid := t.NodeID(node) 122 | if nodeid != "" { 123 | line += "#" + nodeid 124 | } 125 | 126 | indent := len(line) 127 | 128 | line += t.NodeInfo(node) 129 | 130 | brCnt := len(t.Labels(node)) 131 | if brCnt > 1 { 132 | line += fmt.Sprintf("*%d", brCnt) 133 | } 134 | 135 | v, isLeaf := t.LeafVal(node) 136 | if isLeaf { 137 | line += fmt.Sprintf("=%v", v) 138 | } 139 | return line, indent 140 | } 141 | -------------------------------------------------------------------------------- /bmtree/index_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/openacid/low/bitmap" 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestPathToIndex_fulltree_height_0_to_6(t *testing.T) { 11 | 12 | ta := require.New(t) 13 | 14 | cases := []struct { 15 | treeheight int32 16 | }{ 17 | {0}, 18 | {1}, 19 | {2}, 20 | {3}, 21 | {4}, 22 | {5}, 23 | {6}, 24 | } 25 | 26 | for _, c := range cases { 27 | paths := AllPaths(int32(bitmap.MaskUpto[c.treeheight]), 0, 1<<63) 28 | 29 | bitmapSize := bitmap.MaskUpto[c.treeheight] 30 | 31 | // test if generated index are consecutive ints 32 | prev := int32(0) 33 | for i, mp := range paths { 34 | got := PathToIndex(int32(bitmapSize), mp) 35 | ta.Equal(prev, got, "%d-th: treeheight: %d mp: %064b", i+1, c.treeheight, mp) 36 | 37 | got, has := PathToIndexLoose(int32(bitmapSize), mp) 38 | ta.Equal(prev, got, "%d-th: treeheight: %d mp: %064b", i+1, c.treeheight, mp) 39 | ta.Equal(int32(1), has) 40 | 41 | gotpath := IndexToPath(c.treeheight, got) 42 | ta.Equal(mp, gotpath, "%d-th: treeheight: %d mp: %064b", i+1, c.treeheight, mp) 43 | 44 | prev++ 45 | } 46 | } 47 | } 48 | 49 | func TestPathToIndex_halftree_height_0_to_6(t *testing.T) { 50 | 51 | ta := require.New(t) 52 | 53 | cases := []struct { 54 | treeheight int32 55 | }{ 56 | {0}, 57 | {1}, 58 | {2}, 59 | {3}, 60 | {4}, 61 | {5}, 62 | {6}, 63 | } 64 | 65 | for _, c := range cases { 66 | bitmapSize := bitmap.Bit[c.treeheight] 67 | 68 | // test if generated index are consecutive ints 69 | 70 | for i := uint64(0); i < bitmapSize; i++ { 71 | mask := (uint64(1) << uint(c.treeheight)) - 1 72 | mp := i<<32 | mask 73 | got := PathToIndex(int32(bitmapSize), mp) 74 | ta.Equal(int32(i), got, "%d-th: treeheight: %d mp: %064b", i+1, c.treeheight, mp) 75 | 76 | got, has := PathToIndexLoose(int32(bitmapSize), mp) 77 | ta.Equal(int32(i), got, "%d-th: treeheight: %d mp: %064b", i+1, c.treeheight, mp) 78 | ta.Equal(int32(1), has) 79 | } 80 | } 81 | } 82 | 83 | func TestPathToIndex_partialtree(t *testing.T) { 84 | 85 | ta := require.New(t) 86 | 87 | cases := []struct { 88 | bitmapSize int32 89 | }{ 90 | {0x01}, 91 | {0x3}, 92 | {0x5}, 93 | {0x72}, 94 | {0xfd49}, 95 | } 96 | 97 | for _, c := range cases { 98 | 99 | paths := AllPaths(c.bitmapSize, 0, 1<<63) 100 | h := Height(c.bitmapSize) 101 | 102 | // test if generated index are consecutive ints 103 | prev := int32(0) 104 | for i, mp := range paths { 105 | l := PathLen(mp) 106 | if c.bitmapSize&int32(bitmap.Bit[l]) == 0 { 107 | continue 108 | } 109 | got := PathToIndex(c.bitmapSize, mp) 110 | ta.Equal(prev, got, "%d-th: bitmapSize: %032b mp: %064b height: %d, pathlen:%d", i+1, c.bitmapSize, mp, h, l) 111 | 112 | got, has := PathToIndexLoose(c.bitmapSize, mp) 113 | ta.Equal(prev, got, "%d-th: bitmapSize: %032b mp: %064b height: %d, pathlen:%d", i+1, c.bitmapSize, mp, h, l) 114 | ta.Equal(int32(1), has) 115 | 116 | prev++ 117 | } 118 | } 119 | } 120 | 121 | func TestPathToIndexLoose_absentLevel(t *testing.T) { 122 | 123 | ta := require.New(t) 124 | 125 | cases := []struct { 126 | bitmapSize int32 127 | path uint64 128 | wantindex int32 129 | wanthas int32 130 | }{ 131 | {0x5, 0x000000000, 0, 1}, 132 | {0x5, 0x000000002, 1, 0}, 133 | {0x5, 0x000000003, 1, 1}, 134 | {0x5, 0x100000003, 2, 1}, 135 | {0x5, 0x200000002, 3, 0}, 136 | {0x5, 0x200000003, 3, 1}, 137 | {0x5, 0x300000003, 4, 1}, 138 | } 139 | 140 | for i, c := range cases { 141 | 142 | got, has := PathToIndexLoose(c.bitmapSize, c.path) 143 | 144 | ta.Equal(c.wantindex, got, "%d-th: %+v", i+1, c) 145 | ta.Equal(c.wanthas, has, "%d-th: %+v", i+1, c) 146 | 147 | } 148 | } 149 | -------------------------------------------------------------------------------- /bitmap/builder_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestBuilder_Extend(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input [][]int32 15 | inputSizes []int32 16 | wantWords []uint64 17 | }{ 18 | { 19 | [][]int32{}, 20 | []int32{}, 21 | []uint64{}, 22 | }, 23 | { 24 | [][]int32{{0}}, 25 | []int32{1}, 26 | []uint64{1}, 27 | }, 28 | { 29 | [][]int32{{0}}, 30 | []int32{2}, 31 | []uint64{1}, 32 | }, 33 | { 34 | [][]int32{{0}}, 35 | []int32{64}, 36 | []uint64{1}, 37 | }, 38 | { 39 | [][]int32{{0}}, 40 | []int32{65}, 41 | []uint64{1, 0}, 42 | }, 43 | { 44 | [][]int32{{0, 1, 2}}, 45 | []int32{0}, 46 | []uint64{7}, 47 | }, 48 | { 49 | [][]int32{{0, 1, 2}}, 50 | []int32{1}, 51 | []uint64{7}, 52 | }, 53 | { 54 | [][]int32{{0}, {0, 1}}, 55 | []int32{1, 2}, 56 | []uint64{7}, 57 | }, 58 | { 59 | [][]int32{{0}, {1, 2}}, 60 | []int32{1, 2}, 61 | []uint64{13}, 62 | }, 63 | { 64 | [][]int32{{0}, {1, 2}}, 65 | []int32{5, 2}, 66 | []uint64{193}, 67 | }, 68 | { 69 | [][]int32{{0}, {1, 2}}, 70 | []int32{64, 64}, 71 | []uint64{1, 6}, 72 | }, 73 | { 74 | [][]int32{{0, 1}, {2, 63}}, 75 | []int32{64, 0}, 76 | []uint64{3, (1 << 63) + 4}, 77 | }, 78 | { 79 | [][]int32{{64}}, 80 | []int32{0}, 81 | []uint64{0, 1}, 82 | }, 83 | { 84 | [][]int32{{64}, {0}}, 85 | []int32{65, 1}, 86 | []uint64{0, 3}, 87 | }, 88 | { 89 | [][]int32{{1}, {2}, {3, 64, 129}}, 90 | []int32{1, 1, 129}, 91 | []uint64{42, 4, 8}, 92 | }, 93 | { // extend to contain all bitmap 94 | [][]int32{{1}, {2}}, 95 | []int32{1, 127}, 96 | []uint64{0xa, 0}, 97 | }, 98 | { // extend to contain all bitmap 99 | [][]int32{{1}, {2}}, 100 | []int32{1, 128}, 101 | []uint64{0xa, 0, 0}, 102 | }, 103 | } 104 | 105 | for i, c := range cases { 106 | 107 | { 108 | b := NewBuilder(0) 109 | for i, sz := range c.inputSizes { 110 | pos := c.input[i] 111 | b.Extend(pos, sz) 112 | } 113 | 114 | ta.Equal(c.wantWords, b.Words, 115 | "%d-th: case: %+v", 116 | i+1, c) 117 | } 118 | 119 | { 120 | 121 | b := NewBuilder(100) 122 | for i, sz := range c.inputSizes { 123 | pos := c.input[i] 124 | b.Extend(pos, sz) 125 | } 126 | 127 | ta.Equal(c.wantWords, b.Words, 128 | "%d-th: case: %+v", 129 | i+1, c) 130 | } 131 | } 132 | } 133 | 134 | func TestBuilder_Set(t *testing.T) { 135 | 136 | ta := require.New(t) 137 | 138 | cases := []struct { 139 | input []int32 140 | value []int32 141 | wantWords []uint64 142 | wantOffset int32 143 | }{ 144 | { 145 | []int32{0}, 146 | []int32{1}, 147 | []uint64{1}, 148 | 1, 149 | }, 150 | { 151 | []int32{0, 5}, 152 | []int32{1, 0}, 153 | []uint64{1}, 154 | 6, 155 | }, 156 | { 157 | []int32{0, 5, 63}, 158 | []int32{1, 0, 1}, 159 | []uint64{1 + 1<<63}, 160 | 64, 161 | }, 162 | { 163 | []int32{0, 5, 63, 5}, 164 | []int32{1, 0, 1, 1}, 165 | []uint64{1 + 1<<5 + 1<<63}, 166 | 64, 167 | }, 168 | { 169 | []int32{0, 5, 63, 5, 64}, 170 | []int32{1, 0, 1, 1, 0}, 171 | []uint64{1 + 1<<5 + 1<<63, 0}, 172 | 65, 173 | }, 174 | { 175 | []int32{0, 5, 63, 5, 64}, 176 | []int32{1, 0, 1, 1, 1}, 177 | []uint64{1 + 1<<5 + 1<<63, 1}, 178 | 65, 179 | }, 180 | } 181 | 182 | for i, c := range cases { 183 | 184 | b := NewBuilder(0) 185 | for i, pos := range c.input { 186 | b.Set(pos, c.value[i]) 187 | } 188 | 189 | ta.Equal(c.wantWords, b.Words, 190 | "%d-th: words, case: %+v", 191 | i+1, c) 192 | ta.Equal(c.wantOffset, b.Offset, 193 | "%d-th: offset, case: %+v", 194 | i+1, c) 195 | } 196 | } 197 | -------------------------------------------------------------------------------- /bmtree/newpath_test.go: -------------------------------------------------------------------------------- 1 | package bmtree 2 | 3 | import ( 4 | "fmt" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestNewPath_compare(t *testing.T) { 11 | 12 | ta := require.New(t) 13 | 14 | ta.True(NewPath(0, 0, 7) < NewPath(0, 1, 7)) 15 | ta.True(NewPath(0, 1, 7) < NewPath(0, 2, 7)) 16 | ta.True(NewPath(12, 0, 7) < NewPath(12, 1, 7)) 17 | ta.True(NewPath(12, 1, 7) < NewPath(12, 2, 7)) 18 | 19 | ta.True(NewPath(7, 2, 3) > NewPath(6, 3, 3)) 20 | } 21 | 22 | func TestNewPath(t *testing.T) { 23 | 24 | ta := require.New(t) 25 | 26 | cases := []struct { 27 | input string 28 | from, height int32 29 | wantbm string 30 | }{ 31 | // a: 0110 0001 32 | {"", 0, 0, "00000000000000000000000000000000" + "00000000000000000000000000000000"}, 33 | {"", 0, 1, "00000000000000000000000000000000" + "00000000000000000000000000000000"}, 34 | {"", 0, 32, "00000000000000000000000000000000" + "00000000000000000000000000000000"}, 35 | {"a", 0, 0, "00000000000000000000000000000000" + "00000000000000000000000000000000"}, 36 | {"a", 0, 1, "00000000000000000000000000000000" + "00000000000000000000000000000001"}, 37 | {"a", 0, 2, "00000000000000000000000000000001" + "00000000000000000000000000000011"}, 38 | {"a", 0, 5, "00000000000000000000000000001100" + "00000000000000000000000000011111"}, 39 | {"a", 0, 6, "00000000000000000000000000011000" + "00000000000000000000000000111111"}, 40 | {"a", 0, 7, "00000000000000000000000000110000" + "00000000000000000000000001111111"}, 41 | {"a", 0, 8, "00000000000000000000000001100001" + "00000000000000000000000011111111"}, 42 | {"a", 2, 0, "00000000000000000000000000000000" + "00000000000000000000000000000000"}, 43 | {"a", 2, 1, "00000000000000000000000000000001" + "00000000000000000000000000000001"}, 44 | {"a", 2, 2, "00000000000000000000000000000010" + "00000000000000000000000000000011"}, 45 | {"a", 2, 5, "00000000000000000000000000010000" + "00000000000000000000000000011111"}, 46 | {"a", 2, 6, "00000000000000000000000000100001" + "00000000000000000000000000111111"}, 47 | {"a", 2, 8, "00000000000000000000000010000100" + "00000000000000000000000011111100"}, 48 | 49 | // "to" over string end 50 | {"abc", 10, 32, "10001001100011000000000000000000" + "11111111111111000000000000000000"}, 51 | {"abc", 23, 8, "00000000000000000000000010000000" + "00000000000000000000000010000000"}, 52 | {"abcdefgh", 31, 32, "00110010101100110011001110110100" + "11111111111111111111111111111111"}, 53 | {"abcdefgh", 32, 32, "01100101011001100110011101101000" + "11111111111111111111111111111111"}, 54 | {"abcdefgh", 33, 32, "11001010110011001100111011010000" + "11111111111111111111111111111110"}, 55 | } 56 | 57 | for i, c := range cases { 58 | gotbm64 := PathOf(c.input, c.from, c.height) 59 | gotlen := PathLen(gotbm64) 60 | gotbm := fmt.Sprintf("%064b", gotbm64) 61 | 62 | ta.Equal(c.wantbm, gotbm, "%d-th: case: %+v", i+1, c) 63 | ta.Equal(gotbm64, NewPath(gotbm64>>32, gotlen, c.height)) 64 | } 65 | } 66 | 67 | func TestPathsOf(t *testing.T) { 68 | 69 | ta := require.New(t) 70 | 71 | got := PathsOf([]string{"abc", "abc", "abd"}, 10, 32, false) 72 | gots := toStr(got) 73 | 74 | want := []string{ 75 | "10001001100011000000000000000000" + "11111111111111000000000000000000", 76 | "10001001100011000000000000000000" + "11111111111111000000000000000000", 77 | "10001001100100000000000000000000" + "11111111111111000000000000000000", 78 | } 79 | ta.Equal(want, gots) 80 | 81 | got = PathsOf([]string{"abc", "abc", "abd"}, 10, 32, true) 82 | gots = toStr(got) 83 | 84 | want = []string{ 85 | "10001001100011000000000000000000" + "11111111111111000000000000000000", 86 | "10001001100100000000000000000000" + "11111111111111000000000000000000", 87 | } 88 | ta.Equal(want, gots) 89 | } 90 | 91 | func toStr(vbs []uint64) []string { 92 | gots := make([]string, len(vbs)) 93 | for i, g := range vbs { 94 | gots[i] = fmt.Sprintf("%064b", g) 95 | } 96 | return gots 97 | } 98 | -------------------------------------------------------------------------------- /bitmap/fromstr32_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestFromStr32(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | abcdabcd := uint64(0x6162636461626364) 14 | mask24 := (uint64(1) << 24) - 1 15 | mask25 := (uint64(1) << 25) - 1 16 | mask32 := (uint64(1) << 32) - 1 17 | 18 | cases := []struct { 19 | input string 20 | from, to int32 21 | wantlen int32 22 | wantbm uint64 23 | }{ 24 | // a: 0110 0001 25 | {"", 0, 0, 0, 0}, 26 | {"", 0, 1, 0, 0}, 27 | {"", 0, 32, 0, 0}, 28 | {"a", 0, 0, 0, 0}, 29 | {"a", 0, 1, 1, 0x00}, 30 | {"a", 0, 2, 2, 0x01}, 31 | {"a", 0, 5, 5, 0x0c}, 32 | {"a", 0, 6, 6, 0x18}, 33 | {"a", 0, 7, 7, 0x30}, 34 | {"a", 0, 8, 8, 0x61}, 35 | {"a", 2, 2, 0, 0x00}, 36 | {"a", 2, 3, 1, 0x01}, 37 | {"a", 2, 4, 2, 0x02}, 38 | {"a", 2, 7, 5, 0x10}, 39 | {"a", 2, 8, 6, 0x21}, 40 | {"a", 2, 10, 6, 0x84}, 41 | 42 | // "to" over string end 43 | {"abc", 10, 42, 14, 0x2263 << 18}, 44 | {"abc", 23, 31, 1, 1 << 7}, 45 | {"abcdefgh", 31, 63, 32, (0x65666768 >> 1)}, 46 | {"abcdefgh", 32, 64, 32, 0x65666768}, 47 | {"abcdefgh", 33, 65, 31, (0x65666768 << 1)}, 48 | 49 | {"abcdabcdabcd", 0, 32, 32, (abcdabcd >> 32) & mask32}, 50 | {"abcdabcdabcd", 1, 33, 32, (abcdabcd >> 31) & mask32}, 51 | {"abcdabcdabcd", 2, 34, 32, (abcdabcd >> 30) & mask32}, 52 | {"abcdabcdabcd", 3, 35, 32, (abcdabcd >> 29) & mask32}, 53 | {"abcdabcdabcd", 4, 36, 32, (abcdabcd >> 28) & mask32}, 54 | {"abcdabcdabcd", 5, 37, 32, (abcdabcd >> 27) & mask32}, 55 | {"abcdabcdabcd", 6, 38, 32, (abcdabcd >> 26) & mask32}, 56 | {"abcdabcdabcd", 7, 39, 32, (abcdabcd >> 25) & mask32}, 57 | {"abcdabcdabcd", 8, 40, 32, (abcdabcd >> 24) & mask32}, 58 | {"abcdabcdabcd", 9, 41, 32, (abcdabcd >> 23) & mask32}, 59 | 60 | {"abcdabcdabcd", 0, 24, 24, (abcdabcd >> 40) & mask24}, 61 | {"abcdabcdabcd", 1, 25, 24, (abcdabcd >> 39) & mask24}, 62 | {"abcdabcdabcd", 2, 26, 24, (abcdabcd >> 38) & mask24}, 63 | {"abcdabcdabcd", 3, 27, 24, (abcdabcd >> 37) & mask24}, 64 | {"abcdabcdabcd", 4, 28, 24, (abcdabcd >> 36) & mask24}, 65 | {"abcdabcdabcd", 5, 29, 24, (abcdabcd >> 35) & mask24}, 66 | {"abcdabcdabcd", 6, 30, 24, (abcdabcd >> 34) & mask24}, 67 | {"abcdabcdabcd", 7, 31, 24, (abcdabcd >> 33) & mask24}, 68 | {"abcdabcdabcd", 8, 32, 24, (abcdabcd >> 32) & mask24}, 69 | {"abcdabcdabcd", 9, 33, 24, (abcdabcd >> 31) & mask24}, 70 | 71 | {"abcdabcdabcd", 0, 25, 25, (abcdabcd >> 39) & mask25}, 72 | {"abcdabcdabcd", 1, 26, 25, (abcdabcd >> 38) & mask25}, 73 | {"abcdabcdabcd", 2, 27, 25, (abcdabcd >> 37) & mask25}, 74 | {"abcdabcdabcd", 3, 28, 25, (abcdabcd >> 36) & mask25}, 75 | {"abcdabcdabcd", 4, 29, 25, (abcdabcd >> 35) & mask25}, 76 | {"abcdabcdabcd", 5, 30, 25, (abcdabcd >> 34) & mask25}, 77 | {"abcdabcdabcd", 6, 31, 25, (abcdabcd >> 33) & mask25}, 78 | {"abcdabcdabcd", 7, 32, 25, (abcdabcd >> 32) & mask25}, 79 | {"abcdabcdabcd", 8, 33, 25, (abcdabcd >> 31) & mask25}, 80 | {"abcdabcdabcd", 9, 34, 25, (abcdabcd >> 30) & mask25}, 81 | } 82 | 83 | for i, c := range cases { 84 | gotlen, gotbm := FromStr32(c.input, c.from, c.to) 85 | ta.Equal(c.wantlen, gotlen, "%d-th: case: %+v", i+1, c) 86 | ta.Equal(c.wantbm, gotbm, "%d-th: case: %+v", i+1, c) 87 | } 88 | } 89 | 90 | var ( 91 | FromStr32Output int 92 | FromStr32Input string 93 | ) 94 | 95 | func BenchmarkFromStr32(b *testing.B) { 96 | FromStr32Input = "abcdabcdabcd" 97 | var s uint64 = 0 98 | for i := 0; i < b.N; i++ { 99 | _, b := FromStr32(FromStr32Input, 9, 41) 100 | s += b 101 | } 102 | FromStr32Output = int(s) 103 | } 104 | 105 | func BenchmarkStringToBytes(b *testing.B) { 106 | FromStr32Input = "abcdabcdabcd" 107 | var s uint64 = 0 108 | for i := 0; i < b.N; i++ { 109 | b := []byte(FromStr32Input) 110 | s += uint64(b[2]) 111 | } 112 | FromStr32Output = int(s) 113 | } 114 | -------------------------------------------------------------------------------- /mathext/util/util.go: -------------------------------------------------------------------------------- 1 | // Package util provides most used math functions that golang does not provided. 2 | // 3 | // Since 0.1.21 4 | package util 5 | 6 | func MinI(a, b int) int { 7 | if a < b { 8 | return a 9 | } else { 10 | return b 11 | } 12 | } 13 | 14 | func MaxI(a, b int) int { 15 | if a > b { 16 | return a 17 | } else { 18 | return b 19 | } 20 | } 21 | 22 | func MinI8(a, b int8) int8 { 23 | if a < b { 24 | return a 25 | } else { 26 | return b 27 | } 28 | } 29 | 30 | func MaxI8(a, b int8) int8 { 31 | if a > b { 32 | return a 33 | } else { 34 | return b 35 | } 36 | } 37 | 38 | func MinI16(a, b int16) int16 { 39 | if a < b { 40 | return a 41 | } else { 42 | return b 43 | } 44 | } 45 | 46 | func MaxI16(a, b int16) int16 { 47 | if a > b { 48 | return a 49 | } else { 50 | return b 51 | } 52 | } 53 | 54 | func MinI32(a, b int32) int32 { 55 | if a < b { 56 | return a 57 | } else { 58 | return b 59 | } 60 | } 61 | 62 | func MaxI32(a, b int32) int32 { 63 | if a > b { 64 | return a 65 | } else { 66 | return b 67 | } 68 | } 69 | 70 | func MinI64(a, b int64) int64 { 71 | if a < b { 72 | return a 73 | } else { 74 | return b 75 | } 76 | } 77 | 78 | func MaxI64(a, b int64) int64 { 79 | if a > b { 80 | return a 81 | } else { 82 | return b 83 | } 84 | } 85 | 86 | // uint: 87 | 88 | func MinU(a, b uint) uint { 89 | if a < b { 90 | return a 91 | } else { 92 | return b 93 | } 94 | } 95 | 96 | func MaxU(a, b uint) uint { 97 | if a > b { 98 | return a 99 | } else { 100 | return b 101 | } 102 | } 103 | 104 | func MinU8(a, b uint8) uint8 { 105 | if a < b { 106 | return a 107 | } else { 108 | return b 109 | } 110 | } 111 | 112 | func MaxU8(a, b uint8) uint8 { 113 | if a > b { 114 | return a 115 | } else { 116 | return b 117 | } 118 | } 119 | 120 | func MinU16(a, b uint16) uint16 { 121 | if a < b { 122 | return a 123 | } else { 124 | return b 125 | } 126 | } 127 | 128 | func MaxU16(a, b uint16) uint16 { 129 | if a > b { 130 | return a 131 | } else { 132 | return b 133 | } 134 | } 135 | 136 | func MinU32(a, b uint32) uint32 { 137 | if a < b { 138 | return a 139 | } else { 140 | return b 141 | } 142 | } 143 | 144 | func MaxU32(a, b uint32) uint32 { 145 | if a > b { 146 | return a 147 | } else { 148 | return b 149 | } 150 | } 151 | 152 | func MinU64(a, b uint64) uint64 { 153 | if a < b { 154 | return a 155 | } else { 156 | return b 157 | } 158 | } 159 | 160 | func MaxU64(a, b uint64) uint64 { 161 | if a > b { 162 | return a 163 | } else { 164 | return b 165 | } 166 | } 167 | 168 | // clap 169 | 170 | func ClapI(n, min, max int) int { 171 | if n < min { 172 | n = min 173 | } 174 | if n > max { 175 | n = max 176 | } 177 | return n 178 | } 179 | 180 | func ClapI8(n, min, max int8) int8 { 181 | if n < min { 182 | n = min 183 | } 184 | if n > max { 185 | n = max 186 | } 187 | return n 188 | } 189 | 190 | func ClapI16(n, min, max int16) int16 { 191 | if n < min { 192 | n = min 193 | } 194 | if n > max { 195 | n = max 196 | } 197 | return n 198 | } 199 | 200 | func ClapI32(n, min, max int32) int32 { 201 | if n < min { 202 | n = min 203 | } 204 | if n > max { 205 | n = max 206 | } 207 | return n 208 | } 209 | 210 | func ClapI64(n, min, max int64) int64 { 211 | if n < min { 212 | n = min 213 | } 214 | if n > max { 215 | n = max 216 | } 217 | return n 218 | } 219 | 220 | func ClapU(n, min, max uint) uint { 221 | if n < min { 222 | n = min 223 | } 224 | if n > max { 225 | n = max 226 | } 227 | return n 228 | } 229 | 230 | func ClapU8(n, min, max uint8) uint8 { 231 | if n < min { 232 | n = min 233 | } 234 | if n > max { 235 | n = max 236 | } 237 | return n 238 | } 239 | 240 | func ClapU16(n, min, max uint16) uint16 { 241 | if n < min { 242 | n = min 243 | } 244 | if n > max { 245 | n = max 246 | } 247 | return n 248 | } 249 | 250 | func ClapU32(n, min, max uint32) uint32 { 251 | if n < min { 252 | n = min 253 | } 254 | if n > max { 255 | n = max 256 | } 257 | return n 258 | } 259 | 260 | func ClapU64(n, min, max uint64) uint64 { 261 | if n < min { 262 | n = min 263 | } 264 | if n > max { 265 | n = max 266 | } 267 | return n 268 | } 269 | -------------------------------------------------------------------------------- /bitmap/rank.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "math/bits" 5 | ) 6 | 7 | // IndexRank64 creates a rank index for a bitmap. 8 | // rank(i) is defined as number of "1" upto position i, excluding i. 9 | // 10 | // It returns an index of []int32. 11 | // Every element in it is rank(i*64) 12 | // 13 | // Since 0.1.11 14 | // An optional bool specifies whether to add a last index entry of count of all 15 | // "1". 16 | // 17 | // Since 0.1.8 18 | func IndexRank64(words []uint64, opts ...bool) []int32 { 19 | 20 | trailing := false 21 | if len(opts) > 0 { 22 | trailing = opts[0] 23 | } 24 | 25 | l := len(words) 26 | if trailing { 27 | l++ 28 | } 29 | 30 | idx := make([]int32, l) 31 | n := int32(0) 32 | for i := 0; i < len(words); i++ { 33 | idx[i] = n 34 | n += int32(bits.OnesCount64(words[i])) 35 | } 36 | 37 | if trailing { 38 | idx[len(words)] = n 39 | } 40 | 41 | return idx 42 | } 43 | 44 | // IndexRank128 creates a rank index for a bitmap. 45 | // rank(i) is defined as number of "1" upto position i, excluding i. 46 | // 47 | // It returns an index of []int32. 48 | // Every element in it is rank(i*128). 49 | // 50 | // It also adds a last index item if len(words) % 2 == 0, in order to make the 51 | // distance from any bit to the closest index be less than 64. 52 | // 53 | // 54 | // Since 0.1.8 55 | func IndexRank128(words []uint64) []int32 { 56 | 57 | idx := make([]int32, 0) 58 | n := int32(0) 59 | for i := 0; i < len(words); i += 2 { 60 | idx = append(idx, n) 61 | n += int32(bits.OnesCount64(words[i])) 62 | if i < len(words)-1 { 63 | n += int32(bits.OnesCount64(words[i+1])) 64 | } 65 | } 66 | 67 | // Need a last index to let distance from every bit to its closest index 68 | // <=64 69 | if len(words)&1 == 0 { 70 | idx = append(idx, n) 71 | } 72 | 73 | // clone to reduce cap to len 74 | idx = append(idx[:0:0], idx...) 75 | 76 | return idx 77 | } 78 | 79 | // Rank128 returns the count of 1 up to i excluding i and bit value at i, 80 | // with the help of a 128-bit index. 81 | // 82 | // It takes about 2~3 ns/op. 83 | // Rank128 is about 50% slower than Rank64 84 | // 85 | // Since 0.1.9 86 | func Rank128(words []uint64, rindex []int32, i int32) (int32, int32) { 87 | 88 | // A precaculated count serves for two 64-bit word on the left and right. 89 | // 90 | // Idx[1] = OnesCount(bits[0:128]) 91 | // Idx[1] serves for rank query from 64 to 192 92 | // 93 | // 0 64 128 192 256 94 | // |-------+-------+-------+-------+ 95 | // Idx[0] Idx[1] Idx[2] 96 | // 97 | // let j = i % 128 98 | // 99 | // If 0 <= j < 64: use index at i/128 100 | // If 64 <= j < 128: use index at i/128 + 1 101 | 102 | wordI := i >> 6 103 | j := uint32(i & 63) 104 | atRight := wordI & 1 105 | 106 | n := rindex[(i+64)>>7] 107 | w := words[wordI] 108 | 109 | cnt1 := int32(bits.OnesCount64(w)) 110 | c1 := n - atRight*cnt1 + int32(bits.OnesCount64(w&Mask[j])) 111 | return c1, int32(w>>uint(j)) & 1 112 | } 113 | 114 | // Rank64 returns the count of 1 up to i excluding i and the bit value at i, 115 | // with the help of a 64-bit index. 116 | // 117 | // It takes about 2~3 ns/op. 118 | // 119 | // Since 0.1.9 120 | func Rank64(words []uint64, rindex []int32, i int32) (int32, int32) { 121 | 122 | wordI := i >> 6 123 | j := uint32(i & 63) 124 | 125 | n := rindex[wordI] 126 | w := words[wordI] 127 | 128 | c1 := n + int32(bits.OnesCount64(w&Mask[j])) 129 | return c1, int32(w>>uint(j)) & 1 130 | } 131 | 132 | // Tip: use static mask to speed up rank() 133 | // 134 | // calculate mask ((1<= 0 && buf[i] == 0; i-- { 150 | } 151 | return string(buf[:i+1]) 152 | } 153 | -------------------------------------------------------------------------------- /bitword/bitword.go: -------------------------------------------------------------------------------- 1 | // Package bitword provides string operations functions 2 | package bitword 3 | 4 | // Interface defines operations for n-bit word. 5 | // 6 | // Since 0.1.4 7 | type Interface interface { 8 | // FromStr split a string into a slice of `n`-bit words in byte. 9 | // 10 | // Since 0.1.4 11 | FromStr(string) []byte 12 | // FromStrs converts a `[]string` to a n-bit word `[][]byte`. 13 | // 14 | // Since 0.1.4 15 | FromStrs([]string) [][]byte 16 | // ToStr is the reverse of FromStr. 17 | // 18 | // Since 0.1.4 19 | ToStr([]byte) string 20 | // ToStrs converts a `[][]byte` back to a `[]string`. 21 | // 22 | // Since 0.1.4 23 | ToStrs([][]byte) []string 24 | // Get returns i-th n-bit word from a string. 25 | // 26 | // Since 0.1.4 27 | Get(string, int) byte 28 | // FirstDiff returns the index of the first different n-bit word, 29 | // from "start" upto "end". 30 | // 31 | // Since 0.1.4 32 | FirstDiff(a, b string, start, end int) int 33 | } 34 | 35 | // bitWord implements `n`-bit word operations. 36 | // 37 | // Since 0.1.4 38 | type bitWord struct { 39 | // width is the word width. 40 | width int 41 | // byteCap defines how many words a byte contains. 42 | // It is 8 / width. 43 | byteCap int 44 | // wordMask is bit mask. 45 | // It sets the least siginicant "width" bit to 1. 46 | wordMask byte 47 | } 48 | 49 | func newBW(n int) Interface { 50 | return &bitWord{ 51 | width: n, 52 | byteCap: 8 / n, 53 | wordMask: (1 << uint(n)) - 1, 54 | } 55 | } 56 | 57 | var ( 58 | // BitWord pre-defines n-bit words operations for 1, 2, 4, 8 59 | BitWord = map[int]Interface{ 60 | 1: newBW(1), 61 | 2: newBW(2), 62 | 4: newBW(4), 63 | 8: newBW(8), 64 | } 65 | ) 66 | 67 | // FromStr split a string into a slice of byte. 68 | // A byte in string is split into 8/`n` `n`-bit words 69 | // Value of every byte is in range [0, 2^n-1]. 70 | // 71 | // Significant bits in a byte is place at left. 72 | // Thus the result byte slice keeps order with the original string. 73 | // 74 | // Since 0.1.4 75 | func (w *bitWord) FromStr(s string) []byte { 76 | 77 | // number of words per byte 78 | m := w.byteCap 79 | lenSrc := len(s) 80 | words := make([]byte, lenSrc*m) 81 | 82 | for i := 0; i < lenSrc; i++ { 83 | b := s[i] 84 | 85 | for j := 0; j < m; j++ { 86 | words[i*m+j] = (b >> uint(8-w.width*j-w.width)) & w.wordMask 87 | } 88 | } 89 | return words 90 | } 91 | 92 | // FromStrs converts a `[]string` to a n-bit word `[][]byte`. 93 | // 94 | // Since 0.1.4 95 | func (w *bitWord) FromStrs(strs []string) [][]byte { 96 | rst := make([][]byte, len(strs)) 97 | for i, s := range strs { 98 | rst[i] = w.FromStr(s) 99 | } 100 | return rst 101 | } 102 | 103 | // ToStr is the reverse of FromStr. 104 | // 105 | // Since 0.1.4 106 | func (w *bitWord) ToStr(bs []byte) string { 107 | 108 | // number of words per byte 109 | m := w.byteCap 110 | sz := (len(bs) + m - 1) / m 111 | strbs := make([]byte, sz) 112 | 113 | var b byte 114 | for i := 0; i < len(strbs); i++ { 115 | b = 0 116 | for j := 0; j < m; j++ { 117 | if i*m+j < len(bs) { 118 | b = (b << uint(w.width)) + bs[i*m+j] 119 | } else { 120 | b = b << uint(w.width) 121 | } 122 | } 123 | strbs[i] = b 124 | } 125 | 126 | return string(strbs) 127 | } 128 | 129 | // ToStrs converts a `[][]byte` back to a `[]string`. 130 | // 131 | // Since 0.1.4 132 | func (w *bitWord) ToStrs(bytesslice [][]byte) []string { 133 | rst := make([]string, len(bytesslice)) 134 | for i, s := range bytesslice { 135 | rst[i] = w.ToStr(s) 136 | } 137 | return rst 138 | } 139 | 140 | // Get returns i-th n-bit word from a string. 141 | // 142 | // Since 0.1.4 143 | func (w *bitWord) Get(s string, ith int) byte { 144 | i := w.width * ith 145 | end := (i + w.width - 1) & 7 146 | 147 | word := s[i>>3] 148 | return (word >> uint(7-end)) & w.wordMask 149 | } 150 | 151 | // FirstDiff returns the index of the first different n-bit word, 152 | // that ge "from" and lt "end". 153 | // If "end" is -1 it means to look up upto end of a or b. 154 | // 155 | // Since 0.1.4 156 | func (w *bitWord) FirstDiff(a, b string, from, end int) int { 157 | la, lb := len(a)*w.byteCap, len(b)*w.byteCap 158 | 159 | if end == -1 { 160 | end = la 161 | } 162 | 163 | if end > la { 164 | end = la 165 | } 166 | 167 | if end > lb { 168 | end = lb 169 | } 170 | 171 | for i := from; i < end; i++ { 172 | if w.Get(a, i) != w.Get(b, i) { 173 | return i 174 | } 175 | } 176 | return end 177 | } 178 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= 2 | github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= 3 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 4 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 5 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 6 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 7 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 8 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 9 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 10 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 11 | github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= 12 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 13 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 14 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 15 | github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= 16 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 17 | github.com/openacid/errors v0.8.1 h1:Hrj9WENDoj5jP27ZfF60SY5LShbxei+sxKZa0EP+oDw= 18 | github.com/openacid/errors v0.8.1/go.mod h1:GUQEJJOJE3W9skHm8E8Y4phdl2LLEN8iD7c5gcGgdx0= 19 | github.com/openacid/must v0.1.3 h1:deanGZVyVwV+ozfwNFbRU5YF7czXeQ67s8GVyZxzKW4= 20 | github.com/openacid/must v0.1.3/go.mod h1:luPiXCuJlEo3UUFQngVQokV0MPGryeYvtCbQPs3U1+I= 21 | github.com/openacid/testkeys v0.1.7 h1:8mai/cJLsVvBob8K9RrXilkatK4oahfCZdxDJ8CUK7I= 22 | github.com/openacid/testkeys v0.1.7/go.mod h1:MfA7cACzBpbiwekivj8StqX0WIRmqlMsci1c37CA3Do= 23 | github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= 24 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 25 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 26 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 27 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 28 | github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= 29 | github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= 30 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 31 | github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 32 | github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 33 | github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= 34 | github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= 35 | github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= 36 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= 37 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 38 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 39 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 40 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 41 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 42 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 43 | google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM= 44 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 45 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 46 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 47 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 48 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 49 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 50 | -------------------------------------------------------------------------------- /size/sizeof.go: -------------------------------------------------------------------------------- 1 | // Package size provides value size operations. 2 | package size 3 | 4 | import ( 5 | "fmt" 6 | "reflect" 7 | "strings" 8 | "unsafe" 9 | ) 10 | 11 | // const uintSize = 32 << (^uint(0) >> 32 & 1) // 32 or 64 12 | 13 | // Of returns the size in byte a value(not type) costs. 14 | // 15 | // Since 0.1.0 16 | func Of(data interface{}) int { 17 | if data == nil { 18 | return 0 19 | } 20 | return sizeof(reflect.ValueOf(data)) 21 | } 22 | 23 | // Opt defines options for size.Stat() 24 | // 25 | // Since 0.1.11 26 | type Opt struct { 27 | 28 | // AvgOf being a non-zero value indicates to display average. 29 | // 30 | // Since 0.1.11 31 | AvgOf int 32 | 33 | // AvgUnit specifies the unit of size to use. 34 | // By default it is byte. 35 | // Set AvgUnit to 1.0/8 to display size in bit. 36 | // 37 | // Since 0.1.11 38 | AvgUnit float64 39 | } 40 | 41 | // Stat returns a multi line string describe size of every component of a 42 | // vavlue, such as slice element, struct member: 43 | // 44 | // size_test.my: 48 45 | // a: []int32: 36 46 | // 0: int32: 4 47 | // 1: int32: 4 48 | // 2: int32: 4 49 | // b: [3]int32: 12 50 | // 0: int32: 4 51 | // 1: int32: 4 52 | // 2: int32: 4 53 | // 54 | // Since 0.1.0 55 | func Stat(v interface{}, depth int, maxItem int, opts ...interface{}) string { 56 | opt := Opt{} 57 | if len(opts) > 0 { 58 | opt = opts[0].(Opt) 59 | } 60 | lines := stat(reflect.ValueOf(v), depth, maxItem, opt) 61 | return strings.Join(lines, "\n") 62 | } 63 | 64 | var ( 65 | mapsize = int(unsafe.Sizeof(map[int]int{})) // 8 66 | slicesize = int(unsafe.Sizeof([]int8{})) // 24 67 | stringsize = int(unsafe.Sizeof("")) // 16 68 | pointersize = int(unsafe.Sizeof(&mapsize)) // 8 69 | interfacesize = int(unsafe.Sizeof(interface{}(nil))) // 16 70 | ) 71 | 72 | func stat(v reflect.Value, depth int, maxItem int, opt Opt) []string { 73 | if !v.IsValid() { 74 | return []string{""} 75 | } 76 | 77 | var header string 78 | if opt.AvgOf > 0 { 79 | s := sizeof(v) 80 | avg := float64(s) / float64(opt.AvgOf) 81 | unit := opt.AvgUnit 82 | if unit == 0 { 83 | unit = 1 84 | } 85 | avg /= unit 86 | header = fmt.Sprintf("%s: %d /n = %.3f", v.Type(), s, avg) 87 | } else { 88 | header = fmt.Sprintf("%s: %d", v.Type(), sizeof(v)) 89 | } 90 | if depth == 0 { 91 | return []string{header} 92 | } 93 | 94 | depth -= 1 95 | 96 | lines := []string{header} 97 | 98 | switch v.Kind() { 99 | case reflect.Map: 100 | keys := v.MapKeys() 101 | for i := 0; i < len(keys) && i < maxItem; i++ { 102 | mapkey := keys[i] 103 | subs := stat(v.MapIndex(mapkey), depth, maxItem, opt) 104 | subs[0] = fmt.Sprintf("%s: ", mapkey) + subs[0] 105 | 106 | lines = append(lines, subs...) 107 | } 108 | case reflect.Slice, reflect.Array: 109 | for i, n := 0, v.Len(); i < n && i < maxItem; i++ { 110 | subs := stat(v.Index(i), depth, maxItem, opt) 111 | subs[0] = fmt.Sprintf("%d: ", i) + subs[0] 112 | lines = append(lines, subs...) 113 | } 114 | 115 | case reflect.Ptr: 116 | p := (*[]byte)(unsafe.Pointer(v.Pointer())) 117 | if p != nil { 118 | lines = append(lines, stat(v.Elem(), depth, maxItem, opt)...) 119 | } 120 | case reflect.Interface: 121 | lines = append(lines, stat(v.Elem(), depth, maxItem, opt)...) 122 | case reflect.Struct: 123 | for i, n := 0, v.NumField(); i < n; i++ { 124 | subs := stat(v.Field(i), depth, maxItem, opt) 125 | subs[0] = fmt.Sprintf("%s: ", v.Type().Field(i).Name) + subs[0] 126 | lines = append(lines, subs...) 127 | } 128 | } 129 | for i, s := range lines { 130 | if i > 0 { 131 | lines[i] = " " + s 132 | } 133 | } 134 | 135 | return lines 136 | } 137 | 138 | func sizeof(v reflect.Value) int { 139 | if !v.IsValid() { 140 | return 0 141 | } 142 | 143 | sum := 0 144 | switch v.Kind() { 145 | case reflect.Map: 146 | keys := v.MapKeys() 147 | for i := 0; i < len(keys); i++ { 148 | mapkey := keys[i] 149 | s := sizeof(mapkey) 150 | sum += s 151 | s = sizeof(v.MapIndex(mapkey)) 152 | sum += s 153 | } 154 | case reflect.Slice, reflect.Array: 155 | for i, n := 0, v.Len(); i < n; i++ { 156 | s := sizeof(v.Index(i)) 157 | sum += s 158 | } 159 | 160 | case reflect.String: 161 | for i, n := 0, v.Len(); i < n; i++ { 162 | s := sizeof(v.Index(i)) 163 | sum += s 164 | } 165 | 166 | case reflect.Ptr: 167 | p := (*[]byte)(unsafe.Pointer(v.Pointer())) 168 | if p == nil { 169 | sum = 0 170 | } else { 171 | sum = sizeof(v.Elem()) 172 | } 173 | case reflect.Interface: 174 | sum = sizeof(v.Elem()) 175 | case reflect.Struct: 176 | for i, n := 0, v.NumField(); i < n; i++ { 177 | s := sizeof(v.Field(i)) 178 | sum += s 179 | } 180 | 181 | case reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, 182 | reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, 183 | reflect.Float32, reflect.Float64, reflect.Complex64, reflect.Complex128, 184 | reflect.Int: 185 | sum = int(v.Type().Size()) 186 | case reflect.Bool: 187 | sum = int(v.Type().Size()) 188 | default: 189 | panic(fmt.Sprintf("unknown kind: %s", v.Kind())) 190 | } 191 | 192 | switch v.Kind() { 193 | case reflect.Map: 194 | sum += mapsize 195 | case reflect.Slice: 196 | sum += slicesize 197 | case reflect.String: 198 | sum += stringsize 199 | case reflect.Ptr: 200 | sum += pointersize 201 | case reflect.Interface: 202 | sum += interfacesize 203 | } 204 | return sum 205 | } 206 | -------------------------------------------------------------------------------- /bitword/bitword_test.go: -------------------------------------------------------------------------------- 1 | package bitword 2 | 3 | import ( 4 | "reflect" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/require" 8 | ) 9 | 10 | func TestToAndFromStr(t *testing.T) { 11 | cases := []struct { 12 | src string 13 | n int 14 | want []byte 15 | }{ 16 | {"", 1, 17 | []byte{}}, 18 | {"", 2, 19 | []byte{}}, 20 | {"", 4, 21 | []byte{}}, 22 | {"", 8, 23 | []byte{}}, 24 | {"a", 1, 25 | []byte{0, 1, 1, 0, 0, 0, 0, 1}}, 26 | {"a", 2, 27 | []byte{0x1, 0x2, 0x0, 0x1}}, 28 | {"a", 4, 29 | []byte{0x6, 0x1}}, 30 | {"a", 8, 31 | []byte{0x61}}, 32 | {"\x00", 4, 33 | []byte{0, 0}}, 34 | {"\x01\x02\xff", 1, 35 | []byte{ 36 | 0, 0, 0, 0, 0, 0, 0, 1, 37 | 0, 0, 0, 0, 0, 0, 1, 0, 38 | 1, 1, 1, 1, 1, 1, 1, 1}}, 39 | {"\x01\x02\xff", 2, 40 | []byte{0, 0, 0, 1, 0, 0, 0, 2, 3, 3, 3, 3}}, 41 | {"\x01\x02\xff", 4, 42 | []byte{0, 1, 0, 2, 0xf, 0xf}}, 43 | {"\x01\x02\xff", 8, 44 | []byte{1, 2, 0xff}}, 45 | {"我", 1, 46 | []byte{ 47 | 1, 1, 1, 0, 0, 1, 1, 0, 48 | 1, 0, 0, 0, 1, 0, 0, 0, 49 | 1, 0, 0, 1, 0, 0, 0, 1, 50 | }, 51 | }, 52 | {"我", 2, 53 | []byte{ 54 | 3, 2, 1, 2, 55 | 2, 0, 2, 0, 56 | 2, 1, 0, 1, 57 | }, 58 | }, 59 | {"我", 4, 60 | []byte{0xe, 0x6, 0x8, 0x8, 0x9, 0x1}}, 61 | {"我", 8, 62 | []byte{0xe6, 0x88, 0x91}, 63 | }, 64 | } 65 | 66 | for i, c := range cases { 67 | res := BitWord[c.n].FromStr(c.src) 68 | 69 | if !reflect.DeepEqual(res, c.want) { 70 | t.Errorf("test %d: got %#v, want %#v", 71 | i+1, res, c.want) 72 | } 73 | 74 | str := BitWord[c.n].ToStr(res) 75 | if str != c.src { 76 | t.Fatalf(" expect: %v; but: %v", c.src, str) 77 | } 78 | } 79 | } 80 | 81 | func TestToStrIncomplete(t *testing.T) { 82 | ta := require.New(t) 83 | got := BitWord[4].ToStr([]byte{1, 2, 3}) 84 | want := "\x12\x30" 85 | ta.Equal(want, got) 86 | } 87 | 88 | func TestToAndFromStrs(t *testing.T) { 89 | 90 | cases := []struct { 91 | input []string 92 | n int 93 | want [][]byte 94 | }{ 95 | {[]string{"a", "bc", "d"}, 4, 96 | [][]byte{ 97 | {6, 1}, 98 | {6, 2, 6, 3}, 99 | {6, 4}, 100 | }, 101 | }, 102 | {[]string{"a", "bc", "d"}, 2, 103 | [][]byte{ 104 | {1, 2, 0, 1}, 105 | {1, 2, 0, 2, 1, 2, 0, 3}, 106 | {1, 2, 1, 0}, 107 | }, 108 | }, 109 | } 110 | 111 | for i, c := range cases { 112 | rst := BitWord[c.n].FromStrs(c.input) 113 | if !reflect.DeepEqual(c.want, rst) { 114 | t.Fatalf("%d-th: input: %v; want: %v; actual: %v", 115 | i+1, c.input, c.want, rst) 116 | } 117 | 118 | strs := BitWord[c.n].ToStrs(rst) 119 | if !reflect.DeepEqual(c.input, strs) { 120 | t.Fatalf("%d-th expect: %v; but: %v", i+1, c.input, strs) 121 | } 122 | } 123 | } 124 | 125 | func TestGet(t *testing.T) { 126 | 127 | ta := require.New(t) 128 | 129 | type getInput struct { 130 | s string 131 | n int 132 | ith int 133 | } 134 | 135 | cases := []struct { 136 | input getInput 137 | want byte 138 | }{ 139 | {getInput{"a", 1, 0}, 0}, 140 | {getInput{"a", 1, 1}, 1}, 141 | {getInput{"a", 1, 2}, 1}, 142 | {getInput{"a", 1, 3}, 0}, 143 | {getInput{"a", 1, 4}, 0}, 144 | {getInput{"a", 1, 5}, 0}, 145 | {getInput{"a", 1, 6}, 0}, 146 | {getInput{"a", 1, 7}, 1}, 147 | 148 | {getInput{"a", 2, 0}, 1}, 149 | {getInput{"a", 2, 1}, 2}, 150 | {getInput{"a", 2, 2}, 0}, 151 | {getInput{"a", 2, 3}, 1}, 152 | 153 | {getInput{"a", 4, 0}, 6}, 154 | {getInput{"a", 4, 1}, 1}, 155 | 156 | {getInput{"a", 8, 0}, 0x61}, 157 | 158 | {getInput{"abc", 4, 0}, 6}, 159 | {getInput{"abc", 4, 1}, 1}, 160 | {getInput{"abc", 4, 2}, 6}, 161 | {getInput{"abc", 4, 3}, 2}, 162 | {getInput{"abc", 4, 4}, 6}, 163 | {getInput{"abc", 4, 5}, 3}, 164 | } 165 | 166 | for i, c := range cases { 167 | got := BitWord[c.input.n].Get(c.input.s, c.input.ith) 168 | ta.Equal(c.want, got, 169 | "%d-th: input: %#v; want: %#v; got: %#v", 170 | i+1, c.input, c.want, got) 171 | } 172 | } 173 | 174 | func TestGet_panic(t *testing.T) { 175 | 176 | ta := require.New(t) 177 | 178 | cases := []struct { 179 | s string 180 | n int 181 | ith int 182 | }{ 183 | {"", 1, 0}, 184 | {"", 2, 0}, 185 | {"", 3, 0}, 186 | {"", 3, 1}, 187 | {"a", 1, 8}, 188 | {"a", 1, 9}, 189 | {"ab", 1, 16}, 190 | 191 | {"a", 2, 4}, 192 | {"a", 4, 2}, 193 | {"a", 8, 1}, 194 | {"a", 8, 2}, 195 | } 196 | 197 | for i, c := range cases { 198 | ta.Panics(func() { BitWord[c.n].Get(c.s, c.ith) }, "%d-th: %v", i+1, c) 199 | } 200 | } 201 | 202 | func TestFirstDiff(t *testing.T) { 203 | 204 | ta := require.New(t) 205 | 206 | cases := []struct { 207 | a, b string 208 | n int 209 | from, end int 210 | want int 211 | }{ 212 | {"", "", 1, 0, 0, 0}, 213 | 214 | // 0x61 0x62 215 | {"a", "b", 1, 0, 0, 0}, 216 | {"a", "b", 1, 0, 1, 1}, 217 | {"a", "b", 1, 0, 2, 2}, 218 | {"a", "b", 1, 0, 6, 6}, 219 | {"a", "b", 1, 0, 7, 6}, 220 | 221 | {"aa", "ab", 1, 5, 7, 7}, 222 | {"aa", "ab", 1, 5, 14, 14}, 223 | {"aa", "ab", 1, 5, 15, 14}, 224 | 225 | {"aa", "ab", 1, 15, 15, 15}, 226 | {"aa", "ab", 2, 2, 7, 7}, 227 | {"aa", "ab", 4, 0, 4, 3}, 228 | {"aa", "ab", 8, 0, 1000, 1}, 229 | 230 | {"aa", "aa", 4, 0, 4, 4}, 231 | 232 | {"aac", "aa", 4, 0, 4, 4}, 233 | {"aac", "ab", 4, 0, 4, 3}, 234 | {"aac", "ab", 4, 0, 100, 3}, 235 | 236 | // bug in 0.1.2 237 | {"aaa", "aaa", 4, 0, -1, 6}, 238 | } 239 | 240 | for i, c := range cases { 241 | got := BitWord[c.n].FirstDiff(c.a, c.b, c.from, c.end) 242 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 243 | } 244 | } 245 | -------------------------------------------------------------------------------- /bitmap/tailbitmap_test.go: -------------------------------------------------------------------------------- 1 | package bitmap 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/require" 7 | ) 8 | 9 | func TestNewTailBitmap(t *testing.T) { 10 | 11 | ta := require.New(t) 12 | 13 | cases := []struct { 14 | input int64 15 | want *TailBitmap 16 | }{ 17 | { 18 | input: 0, 19 | want: &TailBitmap{ 20 | Offset: 0, 21 | Words: make([]uint64, 0, 1024), 22 | reclaimed: 0, 23 | }, 24 | }, 25 | { 26 | input: 999999, 27 | want: &TailBitmap{ 28 | Offset: 999999, 29 | Words: make([]uint64, 0, 1024), 30 | reclaimed: 999999, 31 | }, 32 | }, 33 | } 34 | 35 | for i, c := range cases { 36 | got := NewTailBitmap(c.input) 37 | ta.Equal(c.want, got, "%d-th: case: %+v", i+1, c) 38 | } 39 | } 40 | 41 | func TestTailBitmap_Compact(t *testing.T) { 42 | 43 | ta := require.New(t) 44 | 45 | allOnes1024 := make([]uint64, 1024) 46 | for i, _ := range allOnes1024 { 47 | allOnes1024[i] = 0xffffffffffffffff 48 | } 49 | 50 | cases := []struct { 51 | input *TailBitmap 52 | want *TailBitmap 53 | }{ 54 | { 55 | input: &TailBitmap{ 56 | Offset: 0, 57 | Words: []uint64{0xffffffffffffffff}, 58 | reclaimed: 0, 59 | }, 60 | want: &TailBitmap{ 61 | Offset: 64, 62 | Words: []uint64{}, 63 | reclaimed: 0, 64 | }, 65 | }, 66 | { 67 | input: &TailBitmap{ 68 | Offset: 64, 69 | Words: []uint64{0xffffffffffffffff}, 70 | reclaimed: 0, 71 | }, 72 | want: &TailBitmap{ 73 | Offset: 64 * 2, 74 | Words: []uint64{}, 75 | reclaimed: 0, 76 | }, 77 | }, 78 | { 79 | input: &TailBitmap{ 80 | Offset: 64, 81 | Words: []uint64{0xffffffffffffffff, 1}, 82 | reclaimed: 0, 83 | }, 84 | want: &TailBitmap{ 85 | Offset: 64 * 2, 86 | Words: []uint64{1}, 87 | reclaimed: 0, 88 | }, 89 | }, 90 | { 91 | input: &TailBitmap{ 92 | Offset: 64, 93 | Words: allOnes1024, 94 | reclaimed: 0, 95 | }, 96 | want: &TailBitmap{ 97 | Offset: 64 * 1025, 98 | Words: []uint64{}, 99 | reclaimed: 64 * 1025, 100 | }, 101 | }, 102 | } 103 | 104 | for i, c := range cases { 105 | c.input.Compact() 106 | ta.Equal(c.want, c.input, "%d-th: case: %+v", i+1, c) 107 | } 108 | } 109 | 110 | func TestTailBitmap_Set(t *testing.T) { 111 | 112 | ta := require.New(t) 113 | 114 | allOnes1024 := make([]uint64, 1024) 115 | for i, _ := range allOnes1024 { 116 | allOnes1024[i] = 0xffffffffffffffff 117 | } 118 | 119 | cases := []struct { 120 | input *TailBitmap 121 | set int64 122 | want *TailBitmap 123 | }{ 124 | { 125 | input: &TailBitmap{ 126 | Offset: 0, 127 | Words: []uint64{}, 128 | reclaimed: 0, 129 | }, 130 | set: 0, 131 | want: &TailBitmap{ 132 | Offset: 0, 133 | Words: []uint64{1}, 134 | reclaimed: 0, 135 | }, 136 | }, 137 | { 138 | input: &TailBitmap{ 139 | Offset: 64, 140 | Words: []uint64{}, 141 | reclaimed: 0, 142 | }, 143 | set: 65, 144 | want: &TailBitmap{ 145 | Offset: 64, 146 | Words: []uint64{2}, 147 | reclaimed: 0, 148 | }, 149 | }, 150 | { 151 | input: &TailBitmap{ 152 | Offset: 64 * 2, 153 | Words: []uint64{1}, 154 | reclaimed: 0, 155 | }, 156 | set: 5, 157 | want: &TailBitmap{ 158 | Offset: 64 * 2, 159 | Words: []uint64{1}, 160 | reclaimed: 0, 161 | }, 162 | }, 163 | { 164 | input: &TailBitmap{ 165 | Offset: 64 * 2, 166 | Words: []uint64{1}, 167 | reclaimed: 0, 168 | }, 169 | set: 64*2 + 1, 170 | want: &TailBitmap{ 171 | Offset: 64 * 2, 172 | Words: []uint64{3}, 173 | reclaimed: 0, 174 | }, 175 | }, 176 | { 177 | input: &TailBitmap{ 178 | Offset: 64 * 2, 179 | Words: []uint64{1}, 180 | reclaimed: 0, 181 | }, 182 | set: 64*3 + 2, 183 | want: &TailBitmap{ 184 | Offset: 64 * 2, 185 | Words: []uint64{1, 4}, 186 | reclaimed: 0, 187 | }, 188 | }, 189 | { 190 | input: &TailBitmap{ 191 | Offset: 64, 192 | Words: []uint64{0xffffffffffffff7f, 1}, 193 | reclaimed: 0, 194 | }, 195 | set: 64 + 7, 196 | want: &TailBitmap{ 197 | Offset: 64 * 2, 198 | Words: []uint64{1}, 199 | reclaimed: 0, 200 | }, 201 | }, 202 | { 203 | input: &TailBitmap{ 204 | Offset: 64 * 1023, 205 | Words: []uint64{0xffffffffffffff7f, 1}, 206 | reclaimed: 0, 207 | }, 208 | set: 64*1023 + 7, 209 | want: &TailBitmap{ 210 | Offset: 64 * 1024, 211 | Words: []uint64{1}, 212 | reclaimed: 64 * 1024, 213 | }, 214 | }, 215 | } 216 | 217 | for i, c := range cases { 218 | c.input.Set(c.set) 219 | ta.Equal(c.want, c.input, "%d-th: case: %+v", i+1, c) 220 | } 221 | } 222 | 223 | func TestTailBitmap_Get(t *testing.T) { 224 | 225 | ta := require.New(t) 226 | 227 | allOnes1024 := make([]uint64, 1024) 228 | for i, _ := range allOnes1024 { 229 | allOnes1024[i] = 0xffffffffffffffff 230 | } 231 | 232 | cases := []struct { 233 | input *TailBitmap 234 | get int64 235 | want uint64 236 | }{ 237 | { 238 | input: &TailBitmap{ 239 | Offset: 64, 240 | Words: []uint64{}, 241 | reclaimed: 0, 242 | }, 243 | get: 0, 244 | want: 1, 245 | }, 246 | { 247 | input: &TailBitmap{ 248 | Offset: 64, 249 | Words: []uint64{}, 250 | reclaimed: 0, 251 | }, 252 | get: 1, 253 | want: 2, 254 | }, 255 | { 256 | input: &TailBitmap{ 257 | Offset: 64, 258 | Words: []uint64{}, 259 | reclaimed: 0, 260 | }, 261 | get: 63, 262 | want: 1 << 63, 263 | }, 264 | 265 | { 266 | input: &TailBitmap{ 267 | Offset: 64, 268 | Words: []uint64{0xffffffffffffff7f, 1}, 269 | reclaimed: 0, 270 | }, 271 | get: 64 + 7, 272 | want: 0, 273 | }, 274 | { 275 | input: &TailBitmap{ 276 | Offset: 64, 277 | Words: []uint64{0xffffffffffffff7f, 1}, 278 | reclaimed: 0, 279 | }, 280 | get: 64 + 6, 281 | want: 1 << 6, 282 | }, 283 | { 284 | input: &TailBitmap{ 285 | Offset: 64, 286 | Words: []uint64{0xffffffffffffff7f, 1}, 287 | reclaimed: 0, 288 | }, 289 | get: 64 + 8, 290 | want: 1 << 8, 291 | }, 292 | { 293 | input: &TailBitmap{ 294 | Offset: 64, 295 | Words: []uint64{0xffffffffffffff7f, 1}, 296 | reclaimed: 0, 297 | }, 298 | get: 64*2 + 0, 299 | want: 1, 300 | }, 301 | } 302 | 303 | for i, c := range cases { 304 | got := c.input.Get(c.get) 305 | ta.Equal(c.want, got, "%d-th: Get case: %+v", i+1, c) 306 | 307 | got1 := c.input.Get1(c.get) 308 | if c.want != 0 { 309 | ta.Equal(uint64(1), got1, "%d-th: Get1 case: %+v", i+1, c) 310 | } else { 311 | ta.Equal(uint64(0), got1, "%d-th: Get1 case: %+v", i+1, c) 312 | } 313 | } 314 | } 315 | -------------------------------------------------------------------------------- /pbcmpl/pbcmpl_test.go: -------------------------------------------------------------------------------- 1 | package pbcmpl 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/openacid/errors" 7 | 8 | "github.com/openacid/low/iohelper" 9 | "github.com/stretchr/testify/require" 10 | ) 11 | 12 | func TestMarshalUnMarshal(t *testing.T) { 13 | 14 | ta := require.New(t) 15 | 16 | // use a header as a proto.Message 17 | msg := newHeader("123", 0xff) 18 | 19 | ta.Equal(32, HeaderSize(msg)) 20 | ta.Equal(64, Size(msg)) 21 | 22 | rw := newBuf(512) 23 | 24 | n, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 25 | ta.Nil(err) 26 | ta.Equal(int64(64), n) 27 | 28 | // unmarshal 29 | 30 | m2 := &header{} 31 | 32 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 33 | ta.Nil(err) 34 | ta.Equal(int64(64), n) 35 | ta.Equal("1.0.0", ver, "default ver") 36 | ta.Equal(uint64(0xff), m2.BodySize) 37 | } 38 | 39 | type vh struct { 40 | header 41 | } 42 | 43 | func (h *vh) GetVersion() string { 44 | return "1.2.3" 45 | } 46 | 47 | func TestMarshalUnMarshal_withVersion(t *testing.T) { 48 | 49 | ta := require.New(t) 50 | 51 | msg := &vh{ 52 | header: header{ 53 | Version: [16]byte{1, 2, 3, 4}, 54 | HeaderSize: 456, 55 | BodySize: 789, 56 | }, 57 | } 58 | 59 | rw := newBuf(512) 60 | 61 | n, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 62 | ta.Nil(err) 63 | ta.Equal(int64(64), n) 64 | 65 | // unmarshal 66 | 67 | m2 := &vh{} 68 | 69 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 70 | ta.Nil(err) 71 | ta.Equal(int64(64), n) 72 | ta.Equal("1.2.3", ver) 73 | 74 | ta.Equal([16]byte{1, 2, 3, 4}, m2.Version) 75 | ta.Equal(uint64(456), m2.HeaderSize) 76 | ta.Equal(uint64(789), m2.BodySize) 77 | } 78 | 79 | type IncomleteReaderWriter struct { 80 | Buf []byte 81 | } 82 | 83 | func (rw *IncomleteReaderWriter) Read(p []byte) (n int, err error) { 84 | // Read 1 byte per Read() 85 | b := rw.Buf[0] 86 | rw.Buf = rw.Buf[1:] 87 | p[0] = b 88 | return 1, nil 89 | } 90 | 91 | func (rw *IncomleteReaderWriter) Write(p []byte) (n int, err error) { 92 | rw.Buf = append(rw.Buf, p...) 93 | return len(p), nil 94 | } 95 | 96 | type badMarshal struct { 97 | header 98 | } 99 | 100 | func (b *badMarshal) Marshal() ([]byte, error) { 101 | return nil, errors.New("badMarshal") 102 | } 103 | 104 | type badUnmarshal struct { 105 | header 106 | } 107 | 108 | func (b *badUnmarshal) Unmarshal(buf []byte) error { 109 | return errors.New("badUnmarshal") 110 | } 111 | 112 | func TestMarshal_Error(t *testing.T) { 113 | 114 | ta := require.New(t) 115 | 116 | msg := &badMarshal{ 117 | header: header{ 118 | Version: [16]byte{1, 2, 3, 4}, 119 | HeaderSize: 456, 120 | BodySize: 789, 121 | }, 122 | } 123 | 124 | rw := newBuf(512) 125 | n, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 126 | ta.Equal(int64(0), n) 127 | ta.Equal("badMarshal", err.Error()) 128 | } 129 | 130 | func TestMarshal_WriteError(t *testing.T) { 131 | 132 | ta := require.New(t) 133 | 134 | msg := &vh{ 135 | header: header{ 136 | Version: [16]byte{1, 2, 3, 4}, 137 | HeaderSize: 456, 138 | BodySize: 789, 139 | }, 140 | } 141 | 142 | // fail at first write 143 | 144 | rw := newBuf(31) 145 | n, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 146 | ta.Equal(int64(31), n) 147 | ta.Equal("OutOfBoundary", err.Error()) 148 | 149 | // fail at second write 150 | 151 | rw = newBuf(33) 152 | n, err = Marshal(iohelper.AtToWriter(rw, 0), msg) 153 | ta.Equal(int64(33), n) 154 | ta.Equal("OutOfBoundary", err.Error()) 155 | } 156 | 157 | func TestUnmarshal_Error(t *testing.T) { 158 | 159 | ta := require.New(t) 160 | 161 | msg := &badUnmarshal{} 162 | 163 | rw := newBuf(64) 164 | _, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 165 | ta.Nil(err) 166 | 167 | m2 := &badUnmarshal{} 168 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 169 | ta.Equal(int64(64), n) 170 | ta.Equal("1.0.0", ver) 171 | ta.Equal("badUnmarshal", err.Error()) 172 | } 173 | 174 | func TestUnmarshal_ReadError(t *testing.T) { 175 | 176 | ta := require.New(t) 177 | 178 | msg := &vh{ 179 | header: header{ 180 | Version: [16]byte{1, 2, 3, 4}, 181 | HeaderSize: 456, 182 | BodySize: 789, 183 | }, 184 | } 185 | 186 | rw := newBuf(64) 187 | _, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 188 | ta.Nil(err) 189 | 190 | { 191 | // shrink buf to make a Read error at body 192 | tmp := rw.b 193 | rw.b = make([]byte, 33) 194 | copy(rw.b, tmp) 195 | 196 | m2 := &vh{} 197 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 198 | ta.Equal(int64(33), n) 199 | ta.Equal("1.2.3", ver) 200 | ta.Equal("EOF", err.Error()) 201 | } 202 | 203 | { 204 | // shrink buf to make a Read error at body 205 | tmp := rw.b 206 | rw.b = make([]byte, 31) 207 | copy(rw.b, tmp) 208 | 209 | m2 := &vh{} 210 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 211 | ta.Equal(int64(31), n) 212 | ta.Equal("", ver) 213 | ta.Equal("EOF", err.Error()) 214 | } 215 | 216 | } 217 | 218 | func TestUnmarshal_invalidHeaderSize(t *testing.T) { 219 | 220 | ta := require.New(t) 221 | 222 | msg := &vh{ 223 | header: header{ 224 | Version: [16]byte{1, 2, 3, 4}, 225 | HeaderSize: 456, 226 | BodySize: 789, 227 | }, 228 | } 229 | 230 | rw := newBuf(64) 231 | _, err := Marshal(iohelper.AtToWriter(rw, 0), msg) 232 | ta.Nil(err) 233 | 234 | // headerSize starts from 16-th byte 235 | rw.b[16] = 0x21 236 | 237 | m2 := &vh{} 238 | n, ver, err := Unmarshal(iohelper.AtToReader(rw, 0), m2) 239 | ta.Equal(int64(32), n, "unmarshaled header but yet not body") 240 | ta.Equal("1.2.3", ver, "version is still correct") 241 | ta.Equal(ErrInvalidHeaderSize, errors.Cause(err)) 242 | } 243 | 244 | func TestUnMarshal_IncompleteReader(t *testing.T) { 245 | 246 | ta := require.New(t) 247 | 248 | // Marshal() must work correctly 249 | // with an io.Reader:Read(p []byte) 250 | // returns n < len(p). 251 | 252 | msg := &vh{ 253 | header: header{ 254 | Version: [16]byte{1, 2, 3, 4}, 255 | HeaderSize: 456, 256 | BodySize: 789, 257 | }, 258 | } 259 | 260 | rw := &IncomleteReaderWriter{} 261 | 262 | n, err := Marshal(rw, msg) 263 | ta.Nil(err) 264 | ta.Equal(int64(64), n) 265 | 266 | // unmarshal 267 | 268 | m2 := &vh{} 269 | 270 | n, ver, err := Unmarshal(rw, m2) 271 | ta.Nil(err) 272 | ta.Equal(int64(64), n) 273 | ta.Equal("1.2.3", ver) 274 | 275 | ta.Equal([16]byte{1, 2, 3, 4}, m2.Version) 276 | ta.Equal(uint64(456), m2.HeaderSize) 277 | ta.Equal(uint64(789), m2.BodySize) 278 | } 279 | 280 | func newBuf(n int) *atBuffer { 281 | return &atBuffer{ 282 | b: make([]byte, n), 283 | } 284 | } 285 | 286 | type atBuffer struct { 287 | b []byte 288 | } 289 | 290 | func (t *atBuffer) WriteAt(b []byte, off int64) (n int, err error) { 291 | length := len(b) 292 | for i := 0; i < length; i++ { 293 | if i+int(off) >= len(t.b) { 294 | return i, errors.New("OutOfBoundary") 295 | } 296 | t.b[int64(i)+off] = b[i] 297 | } 298 | return length, nil 299 | } 300 | 301 | func (t *atBuffer) ReadAt(b []byte, off int64) (n int, err error) { 302 | length := len(b) 303 | 304 | min := off + int64(length) 305 | if min > int64(len(t.b)) { 306 | min = int64(len(t.b)) 307 | } 308 | 309 | n = copy(b, t.b[off:min]) 310 | if n < length { 311 | return n, errors.New("EOF") 312 | } 313 | return length, nil 314 | } 315 | -------------------------------------------------------------------------------- /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # OpenACID's Code of Conduct 2 | 3 | 4 | 5 | 6 | 7 | - [Why have a Code of Conduct?](#why-have-a-code-of-conduct) 8 | - [Our Standards](#our-standards) 9 | - [Social Rules](#social-rules) 10 | - [No feigning surprise](#no-feigning-surprise) 11 | - [No condescending well-actually’s](#no-condescending-well-actually%E2%80%99s) 12 | - [No backseat driving](#no-backseat-driving) 13 | - [No subtle -isms](#no-subtle--isms) 14 | - [Giving and Receiving Feedback](#giving-and-receiving-feedback) 15 | - [Enforcement](#enforcement) 16 | 17 | 18 | 19 | ## Why have a Code of Conduct? 20 | 21 | This Code of Conduct is designed to help all of us build a pleasant, productive, 22 | and fearless community. 23 | 24 | We are striving to make our community a great group of people to work with. 25 | 26 | This Code of Conduct applies both within project spaces and in public spaces 27 | when an individual is representing the project or its community. 28 | 29 | 30 | ## Our Standards 31 | 32 | Examples of behavior that contributes to creating a positive environment 33 | include: 34 | 35 | * Using welcoming and inclusive language 36 | * Being respectful of differing viewpoints and experiences 37 | * Gracefully accepting constructive criticism 38 | * Focusing on what is best for the community 39 | * Showing empathy towards other community members 40 | 41 | Examples of unacceptable behavior by participants include: 42 | 43 | * The use of sexualized language or imagery and unwelcome sexual attention or 44 | advances 45 | * Trolling, insulting/derogatory comments, and personal or political attacks 46 | * Public or private harassment 47 | * Publishing others' private information, such as a physical or electronic 48 | address, without explicit permission 49 | * Other conduct which could reasonably be considered inappropriate in a 50 | professional setting 51 | 52 | 53 | ## Social Rules 54 | 55 | 56 | ### No feigning surprise 57 | 58 | **The first rule means you shouldn't act surprised when people say they don't know 59 | something**. 60 | 61 | This applies to both technical things ("What?! I can't believe you 62 | don't know what the stack is!") and non-technical things ("You don't know who 63 | RMS is?!"). 64 | 65 | Feigning surprise has absolutely no social or educational benefit: When people 66 | feign surprise, it's usually to make them feel better about themselves and 67 | others feel worse. 68 | And even when that's not the intention, it's almost always the effect. 69 | 70 | As you've probably already guessed, this rule is tightly coupled to our belief 71 | in the importance of people **feeling comfortable saying "I don't know" and "I 72 | don't understand."** 73 | 74 | 75 | ### No condescending well-actually’s 76 | 77 | **A well-actually happens when someone says something that's almost— but not 78 | entirely— correct, and you say, "well, actually…" and then give a minor 79 | correction**. 80 | 81 | Even in complicated environments where small details and edge-cases can be 82 | forgotten, unless they are critical, they should not be interjected. 83 | 84 | If they are critical to the conversation phrasing can be the difference between 85 | a valuable clarification and condescension e.g. instead of “well actually …” a 86 | simple change to “don’t forget …” or “it’s easy to forget …” 87 | 88 | 89 | ### No backseat driving 90 | 91 | **If you overhear people working through a problem, you shouldn't intermittently 92 | lob advice across the room**. 93 | 94 | This can lead to the "too many cooks" problem, but more important, it can be 95 | rude and disruptive to half-participate in a conversation. 96 | This is particularly true in a distributed environment involving conversations 97 | in Slack. 98 | The occasional interjection to an on-going conversation, particularly based on 99 | backscroll, can be very disruptive. 100 | 101 | This isn't to say you shouldn't help, offer advice, or join conversations. 102 | On the contrary, we encourage all those things. 103 | Rather, it just means that when you want to help out or work with others, you 104 | should fully engage and not just interject sporadically. 105 | 106 | 107 | ### No subtle -isms 108 | 109 | **Subtle -isms, also called microaggressions, are small things that make others 110 | feel uncomfortable**, for example, saying "It's so easy my grandmother could do 111 | it" is a subtle -ism, as it is both subtly sexist and ageist. 112 | 113 | The "subtle" in "subtle -isms" means that it's probably not obvious to everyone 114 | right away what was wrong with the comment, even people in the group otherwise 115 | affected by the comment. 116 | And, even though they are subtle, might seem insignificant, and are 117 | often unintentional, a steady stream of them compounds to make people in 118 | under-represented groups feel less welcome. 119 | 120 | 121 | ## Giving and Receiving Feedback 122 | 123 | **Give constructive, not critical feedback**. 124 | 125 | Feedback is negatively critical when it surfaces something wrong with someone or 126 | something they produced, especially without any mention of ways to make their 127 | behavior or their product better. 128 | 129 | - **Critical feedback** on work often looks like "you don't write enough tests" or 130 | "your code quality isn't good enough". 131 | Personal criticism can be more severe and often looks like "you should be less 132 | judgemental" or "you are a burden because you ask too many questions”. 133 | 134 | - **Constructive feedback** is more about how a person can do better rather than what 135 | they are doing wrong. 136 | If you want someone to do something better, you should tell them what better 137 | looks like. 138 | Ask a question to get a discussion rolling, to gain context, and then if you see 139 | room for improvement give declarative feedback to that effect. 140 | 141 | This creates an environment where people understand what success looks like 142 | instead of just feeling like they are unsuccessful. 143 | 144 | **Code, configurations, and their reviews are also mechanisms for communication**. 145 | 146 | Just as you shouldn't interact with people poorly in person, do not interact 147 | poorly through code or code review. 148 | 149 | **You are not your products**. 150 | Technical critiques are integral, and should be hard on the product, not on the 151 | producer. 152 | While it is important to care about your work and producing the best thing you 153 | can, this can make review difficult. 154 | It is important to realize that it’s better to find errors in review than in 155 | production and recognize that your work fits into a larger whole. 156 | 157 | **Go about your review under the assumption that the decisions were made for a 158 | reason, not in a vacuum**. 159 | Ask about circumstances if you’re confused. 160 | 161 | Be pragmatic, ask for context, don’t filibuster, don’t block on style not 162 | explicitly covered in DO’s style guides. 163 | 164 | Code, configurations, architecture, platforms, frameworks will need to be 165 | changed. Fight for your way if you think it’s right, **but not only to be right**. 166 | 167 | 168 | ## Enforcement 169 | 170 | Enforcement of the Code of Conduct is essential. 171 | 172 | If there is no enforcement, then the Code of Conduct becomes a feel-good 173 | document without value. 174 | --------------------------------------------------------------------------------