├── R ├── cnpy.R └── utils.R ├── src ├── Makevars ├── Makevars.win ├── RcppExports.cpp ├── cnpyMod.cpp ├── cnpy.cpp └── cnpy.h ├── tests ├── arr.npy ├── fmat.npy ├── fvec.npy ├── imat.npy ├── ivec.npy ├── fmat.npy.gz ├── loadFiles.py ├── createFiles.py ├── loadFiles.R ├── loadFiles.Rout.save ├── saveAndLoad.R └── saveAndLoad.Rout.save ├── demo ├── 00Index └── timings.R ├── paper ├── paper.pdf ├── codemeta.json ├── paper.bib ├── paper.md └── joss │ ├── 2016-09-22-more-comments.md │ ├── 2016-09-20-review.md │ └── 2016-09-21-reply.md ├── local ├── global_signals.npy └── global_signals.npy.zip ├── vignettes ├── pdf │ ├── RcppCNPy-intro.pdf │ └── UsingReticulate.pdf ├── RcppCNPy-intro.Rnw ├── UsingReticulate.Rnw └── rmd │ ├── Makefile │ ├── RcppCNPy.bib │ ├── UsingReticulate.Rmd.orig │ ├── UsingReticulate.Rmd │ └── RcppCNPy-intro.Rmd ├── inst ├── THANKS ├── CITATION ├── cnpy-LICENSE └── NEWS.Rd ├── NAMESPACE ├── .gitignore ├── .Rbuildignore ├── cleanup ├── .github └── workflows │ └── ci.yaml ├── DESCRIPTION ├── man └── RcppCNPy-package.Rd ├── README.md ├── ChangeLog └── LICENSE /R/cnpy.R: -------------------------------------------------------------------------------- 1 | 2 | loadModule("cnpy", TRUE) 3 | 4 | 5 | -------------------------------------------------------------------------------- /src/Makevars: -------------------------------------------------------------------------------- 1 | ## We need the compression library 2 | PKG_LIBS = -lz 3 | -------------------------------------------------------------------------------- /src/Makevars.win: -------------------------------------------------------------------------------- 1 | ## We need the compression library 2 | PKG_LIBS = -lz 3 | -------------------------------------------------------------------------------- /tests/arr.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/arr.npy -------------------------------------------------------------------------------- /demo/00Index: -------------------------------------------------------------------------------- 1 | timings Benchmark comparison of accessing ascii, npy and npy.gz data 2 | -------------------------------------------------------------------------------- /paper/paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/paper/paper.pdf -------------------------------------------------------------------------------- /tests/fmat.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/fmat.npy -------------------------------------------------------------------------------- /tests/fvec.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/fvec.npy -------------------------------------------------------------------------------- /tests/imat.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/imat.npy -------------------------------------------------------------------------------- /tests/ivec.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/ivec.npy -------------------------------------------------------------------------------- /tests/fmat.npy.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/tests/fmat.npy.gz -------------------------------------------------------------------------------- /local/global_signals.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/local/global_signals.npy -------------------------------------------------------------------------------- /local/global_signals.npy.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/local/global_signals.npy.zip -------------------------------------------------------------------------------- /vignettes/pdf/RcppCNPy-intro.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/vignettes/pdf/RcppCNPy-intro.pdf -------------------------------------------------------------------------------- /vignettes/pdf/UsingReticulate.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eddelbuettel/rcppcnpy/HEAD/vignettes/pdf/UsingReticulate.pdf -------------------------------------------------------------------------------- /inst/THANKS: -------------------------------------------------------------------------------- 1 | Carl Rogers for writing the terrific cnpy library 2 | Gong-Yi Liao for most useful help with integer support 3 | -------------------------------------------------------------------------------- /NAMESPACE: -------------------------------------------------------------------------------- 1 | import(methods) 2 | importFrom("Rcpp", "loadModule") 3 | useDynLib(RcppCNPy, .registration=TRUE) 4 | exportPattern("^[[:alpha:]]+") 5 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .Rproj.user 2 | .Rhistory 3 | .RData 4 | src/*.o 5 | src/*.so 6 | src/*.dll 7 | paper/joss/*.html 8 | *.tar.gz 9 | GPATH 10 | GRTAGS 11 | GTAGS 12 | paper/runMe.sh 13 | -------------------------------------------------------------------------------- /.Rbuildignore: -------------------------------------------------------------------------------- 1 | ^.*\.Rproj$ 2 | ^\.Rproj\.user$ 3 | LICENSE 4 | ^\.travis\.yml$ 5 | ^paper 6 | ^local 7 | ^.*\.tar\.gz$ 8 | vignettes/UsingReticulate.Rmd.orig 9 | vignettes/rmd 10 | ^\.github 11 | ^GPATH 12 | ^GRTAGS 13 | ^GTAGS 14 | -------------------------------------------------------------------------------- /cleanup: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | rm -f src/*.o src/*.so src/symbols.rds 4 | 5 | rm -rf vignettes/auto/ vignettes/rmd/pinp.cls vignettes/rmd/jss.bst 6 | 7 | for e in log aux out tex blg bbl xwm; do 8 | rm -f vignettes/rmd/${e} 9 | done 10 | 11 | find . -name \*~ -exec rm {} \; 12 | 13 | -------------------------------------------------------------------------------- /tests/loadFiles.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import os 4 | import numpy as np 5 | 6 | # simple float and integer arrays 7 | print np.load("fmat.npy") 8 | print np.load("imat.npy") 9 | 10 | # simple float and integer vectors 11 | print np.load("fvec.npy") 12 | print np.load("ivec.npy") 13 | -------------------------------------------------------------------------------- /vignettes/RcppCNPy-intro.Rnw: -------------------------------------------------------------------------------- 1 | \documentclass{article} 2 | \usepackage{pdfpages} 3 | %\VignetteIndexEntry{RcppCNPy-intro} 4 | %\VignetteKeywords{Python, NumPy, R, data transfer} 5 | %\VignettePackage{RcppCNPy} 6 | %\VignetteEncoding{UTF-8} 7 | 8 | \begin{document} 9 | \includepdf[pages=-, fitpaper=true]{pdf/RcppCNPy-intro.pdf} 10 | \end{document} 11 | -------------------------------------------------------------------------------- /vignettes/UsingReticulate.Rnw: -------------------------------------------------------------------------------- 1 | \documentclass{article} 2 | \usepackage{pdfpages} 3 | %\VignetteIndexEntry{RcppCNPy-reticulate} 4 | %\VignetteKeywords{Python, NumPy, R, data transfer} 5 | %\VignettePackage{RcppCNPy} 6 | %\VignetteEncoding{UTF-8} 7 | 8 | \begin{document} 9 | \includepdf[pages=-, fitpaper=true]{pdf/UsingReticulate.pdf} 10 | \end{document} 11 | -------------------------------------------------------------------------------- /vignettes/rmd/Makefile: -------------------------------------------------------------------------------- 1 | 2 | ## This Makefile is not included in the package sources so we can use GNUmake idioms. Yay. 3 | 4 | rmdsources := $(wildcard *.Rmd) 5 | rmdvignettes := $(rmdsources:.Rmd=.pdf) 6 | 7 | %.pdf: %.Rmd 8 | Rscript -e 'rmarkdown::render("$<")' 9 | Rscript -e 'tools::compactPDF("$@", gs_quality="ebook")' 10 | cp -vax $@ ../pdf 11 | 12 | all: ${rmdvignettes} 13 | 14 | clean: 15 | @rm -rf *.aux *.log *.out *.toc *.tex *.pdf 16 | -------------------------------------------------------------------------------- /R/utils.R: -------------------------------------------------------------------------------- 1 | 2 | ## We would like to do this at the C++ level too, alas file-system operations 3 | ## are still very dependent on the operating system. C++17 is going to help, 4 | ## but it is not yet widely enough available (and certainly not on CRAN anytime 5 | ## soon). Boost has functions in its filesystem module, but those require 6 | ## linking with the system and filesystem libraries so they cannot be used in 7 | ## header-only mode from BH 8 | 9 | .checkPath <- function(filename) dir.exists(dirname(filename)) 10 | 11 | -------------------------------------------------------------------------------- /tests/createFiles.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import os 4 | import numpy as np 5 | 6 | # simple float and integer arrays 7 | np.save("fmat.npy", np.arange(12).reshape(3,4) * 1.1) 8 | np.save("imat.npy", np.arange(12).reshape(3,4)) 9 | 10 | # simple float and integer vectors 11 | np.save("fvec.npy", np.arange(5) * 1.1) 12 | np.save("ivec.npy", np.arange(5)) 13 | 14 | # three dimensional array 15 | arr = np.array(list(range(24))).reshape(4,3,2) 16 | np.save("arr.npy", arr) 17 | #print(arr[:,:,0]) 18 | #print(arr[:,:,1]) 19 | -------------------------------------------------------------------------------- /src/RcppExports.cpp: -------------------------------------------------------------------------------- 1 | // Generated by using Rcpp::compileAttributes() -> do not edit by hand 2 | // Generator token: 10BE3573-1514-4C36-9D1C-5A225CD40393 3 | 4 | #include 5 | 6 | using namespace Rcpp; 7 | 8 | 9 | RcppExport SEXP _rcpp_module_boot_cnpy(); 10 | 11 | static const R_CallMethodDef CallEntries[] = { 12 | {"_rcpp_module_boot_cnpy", (DL_FUNC) &_rcpp_module_boot_cnpy, 0}, 13 | {NULL, NULL, 0} 14 | }; 15 | 16 | RcppExport void R_init_RcppCNPy(DllInfo *dll) { 17 | R_registerRoutines(dll, NULL, CallEntries, NULL, NULL); 18 | R_useDynamicSymbols(dll, FALSE); 19 | } 20 | -------------------------------------------------------------------------------- /tests/loadFiles.R: -------------------------------------------------------------------------------- 1 | #!/usr/bin/Rscript 2 | 3 | library(RcppCNPy) 4 | 5 | fmat <- npyLoad("fmat.npy") 6 | print(fmat) 7 | stopifnot(all.equal(fmat, t(matrix(seq(0,11) * 1.1, 4, 3)))) 8 | 9 | fmat <- npyLoad("fmat.npy.gz") 10 | print(fmat) 11 | stopifnot(all.equal(fmat, t(matrix(seq(0,11) * 1.1, 4, 3)))) 12 | 13 | imat <- npyLoad("imat.npy", "integer") 14 | print(imat) 15 | stopifnot(all.equal(imat, t(matrix(seq(0,11), 4, 3)))) 16 | 17 | fvec <- npyLoad("fvec.npy") 18 | print(fvec) 19 | stopifnot(all.equal(fvec, seq(0,4) * 1.1)) 20 | 21 | ivec <- npyLoad("ivec.npy", "integer") 22 | print(ivec) 23 | stopifnot(all.equal(ivec, seq(0,4))) 24 | 25 | 26 | -------------------------------------------------------------------------------- /.github/workflows/ci.yaml: -------------------------------------------------------------------------------- 1 | # Run CI for R using https://eddelbuettel.github.io/r-ci/ 2 | 3 | name: ci 4 | 5 | on: 6 | push: 7 | pull_request: 8 | 9 | env: 10 | _R_CHECK_FORCE_SUGGESTS_: "false" 11 | 12 | jobs: 13 | ci: 14 | strategy: 15 | matrix: 16 | include: 17 | #- {os: macOS-latest} 18 | - {os: ubuntu-latest} 19 | 20 | runs-on: ${{ matrix.os }} 21 | 22 | steps: 23 | - uses: actions/checkout@v4 24 | 25 | - name: Setup 26 | uses: eddelbuettel/github-actions/r-ci@master 27 | 28 | - name: Dependencies 29 | run: ./run.sh install_deps 30 | 31 | - name: Test 32 | run: ./run.sh run_tests 33 | 34 | #- name: Coverage 35 | # if: ${{ matrix.os == 'ubuntu-latest' }} 36 | # run: ./run.sh coverage 37 | -------------------------------------------------------------------------------- /inst/CITATION: -------------------------------------------------------------------------------- 1 | bibentry("Manual", 2 | other = unlist(citation(auto = meta), recursive = FALSE)) 3 | 4 | bibentry("Article", 5 | title = "{RcppCNPy}: Read-Write Support for NumPy Files in R", 6 | author = c(person("Dirk", "Eddelbuettel", 7 | email = "edd@debian.org", 8 | comment = c(ORCID = "0000-0001-6419-907X")), 9 | person("Wush Chi-Hsuan", "Wu", 10 | comment = c(ORCID = "0000-0001-5180-0567"))), 11 | journal = "Journal of Open Source Software", 12 | publisher = "The Open Journal", 13 | year = "2016", 14 | volume = "1", 15 | issue = "5", 16 | month = "September", 17 | doi = "10.21105/joss.00055") 18 | -------------------------------------------------------------------------------- /demo/timings.R: -------------------------------------------------------------------------------- 1 | 2 | library(RcppCNPy) 3 | library(rbenchmark) 4 | 5 | ## expensive: N <- 1e5 6 | ## cheaper: 7 | n <- 1e4 8 | k <- 50 9 | 10 | M <- matrix(seq(1.0, n*k, by=1.0), n, k) 11 | 12 | txtfile <- tempfile(fileext=".txt") 13 | write.table(M, file=txtfile) 14 | 15 | pyfile <- tempfile(fileext=".npy") 16 | npySave(pyfile, M) 17 | 18 | pygzfile <- tempfile(fileext=".npy.gz") 19 | npySave(pygzfile, M) 20 | 21 | print(do.call(rbind, (lapply(c(txtfile, pyfile, pygzfile), 22 | function(f) file.info(f)["size"])))) 23 | 24 | res <- benchmark(read.table(txtfile), 25 | npyLoad(pyfile), 26 | npyLoad(pygzfile), 27 | order="relative", 28 | columns=c("test", "replications", "elapsed", "relative"), 29 | replications=10) 30 | print(res) 31 | 32 | unlink(txtfile) 33 | unlink(pyfile) 34 | unlink(pygzfile) 35 | 36 | 37 | 38 | -------------------------------------------------------------------------------- /paper/codemeta.json: -------------------------------------------------------------------------------- 1 | { 2 | "@context": "https://raw.githubusercontent.com/mbjones/codemeta/master/codemeta.jsonld", 3 | "@type": "Code", 4 | "author": [ 5 | { 6 | "@id": "http://orcid.org/0000-0001-6419-907X", 7 | "@type": "Person", 8 | "email": "edd@debian.org", 9 | "name": "Dirk Eddelbuettel", 10 | "affiliation": "Debian and R Projects" 11 | }, 12 | { 13 | "@id": "http://orcid.org/0000-0001-5180-0567", 14 | "@type": "Person", 15 | "email": "wush978@gmail.com", 16 | "name": "Wush Wu", 17 | "affiliation": "Institute of Electrical Engineering, National Taiwan University" 18 | } 19 | ], 20 | "identifier": "", 21 | "codeRepository": "https://github.com/eddelbuettel/rcppcnpy", 22 | "datePublished": "2016-08-27", 23 | "dateModified": "2016-08-27", 24 | "dateCreated": "2016-08-27", 25 | "description": "Rcpp bindings for NumPy files", 26 | "keywords": "Python, NumPy, R, data transfer", 27 | "license": "GPL-2", 28 | "title": "RcppCNPy", 29 | "version": "0.2.5" 30 | } 31 | -------------------------------------------------------------------------------- /inst/cnpy-LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) Carl Rogers, 2011 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Package: RcppCNPy 2 | Type: Package 3 | Title: Read-Write Support for 'NumPy' Files via 'Rcpp' 4 | Version: 0.2.14 5 | Date: 2025-11-03 6 | Authors@R: c(person("Dirk", "Eddelbuettel", role = c("aut", "cre"), email = "edd@debian.org", 7 | comment = c(ORCID = "0000-0001-6419-907X")), 8 | person("Wush", "Wu", role="aut", comment = c(ORCID = "0000-0001-5180-0567")), 9 | person("Carl", "Rogers", role="aut", comment = "Author of CNPy")) 10 | Description: The 'cnpy' library written by Carl Rogers provides read and write 11 | facilities for files created with (or for) the 'NumPy' extension for 'Python'. 12 | Vectors and matrices of numeric types can be read or written to and from 13 | files as well as compressed files. Support for integer files is available if 14 | the package has been built with as C++11 which should be the default on 15 | all platforms since the release of R 3.3.0. 16 | URL: https://github.com/eddelbuettel/rcppcnpy, https://dirk.eddelbuettel.com/code/rcpp.cnpy.html 17 | BugReports: https://github.com/eddelbuettel/rcppcnpy/issues 18 | License: GPL (>= 2) 19 | Depends: R (>= 3.1.0) 20 | Imports: methods, Rcpp 21 | LinkingTo: Rcpp 22 | Suggests: reticulate, rbenchmark 23 | -------------------------------------------------------------------------------- /tests/loadFiles.Rout.save: -------------------------------------------------------------------------------- 1 | 2 | R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet" 3 | Copyright (C) 2014 The R Foundation for Statistical Computing 4 | Platform: x86_64-pc-linux-gnu (64-bit) 5 | 6 | R is free software and comes with ABSOLUTELY NO WARRANTY. 7 | You are welcome to redistribute it under certain conditions. 8 | Type 'license()' or 'licence()' for distribution details. 9 | 10 | R is a collaborative project with many contributors. 11 | Type 'contributors()' for more information and 12 | 'citation()' on how to cite R or R packages in publications. 13 | 14 | Type 'demo()' for some demos, 'help()' for on-line help, or 15 | 'help.start()' for an HTML browser interface to help. 16 | Type 'q()' to quit R. 17 | 18 | > #!/usr/bin/Rscript 19 | > 20 | > library(RcppCNPy) 21 | > 22 | > fmat <- npyLoad("fmat.npy") 23 | > print(fmat) 24 | [,1] [,2] [,3] [,4] 25 | [1,] 0.0 1.1 2.2 3.3 26 | [2,] 4.4 5.5 6.6 7.7 27 | [3,] 8.8 9.9 11.0 12.1 28 | > stopifnot(all.equal(fmat, t(matrix(seq(0,11) * 1.1, 4, 3)))) 29 | > 30 | > fmat <- npyLoad("fmat.npy.gz") 31 | > print(fmat) 32 | [,1] [,2] [,3] [,4] 33 | [1,] 0.0 1.1 2.2 3.3 34 | [2,] 4.4 5.5 6.6 7.7 35 | [3,] 8.8 9.9 11.0 12.1 36 | > stopifnot(all.equal(fmat, t(matrix(seq(0,11) * 1.1, 4, 3)))) 37 | > 38 | > imat <- npyLoad("imat.npy", "integer") 39 | > print(imat) 40 | [,1] [,2] [,3] [,4] 41 | [1,] 0 1 2 3 42 | [2,] 4 5 6 7 43 | [3,] 8 9 10 11 44 | > stopifnot(all.equal(imat, t(matrix(seq(0,11), 4, 3)))) 45 | > 46 | > fvec <- npyLoad("fvec.npy") 47 | > print(fvec) 48 | [1] 0.0 1.1 2.2 3.3 4.4 49 | > stopifnot(all.equal(fvec, seq(0,4) * 1.1)) 50 | > 51 | > ivec <- npyLoad("ivec.npy", "integer") 52 | > print(ivec) 53 | [1] 0 1 2 3 4 54 | > stopifnot(all.equal(ivec, seq(0,4))) 55 | > 56 | > 57 | > 58 | > proc.time() 59 | user system elapsed 60 | 0.408 0.025 0.418 61 | -------------------------------------------------------------------------------- /paper/paper.bib: -------------------------------------------------------------------------------- 1 | @Book{Eddelbuettel:2013:Rcpp, 2 | author = {Dirk Eddelbuettel}, 3 | title = {Seamless R and C++ Integration with Rcpp}, 4 | publisher = {Springer}, 5 | series = {Use R!}, 6 | year = 2013, 7 | address = {New York}, 8 | isbn = {978-1-4614-6867-7}, 9 | doi = {10.1007/978-1-4614-6868-4} 10 | } 11 | 12 | @Manual{CRAN:Rcpp, 13 | title = {{Rcpp}: Seamless {R} and {C++} Integration}, 14 | author = {Dirk Eddelbuettel and Romain Fran\c{c}ois and JJ 15 | Allaire and Kevin Ushey and Qiang Kou and 16 | John Chambers and Douglas Bates}, 17 | year = 2016, 18 | note = {R package version 0.12.7}, 19 | url = {https://CRAN.R-Project.org/package=Rcpp} 20 | } 21 | 22 | @Manual{CRAN:RcppCNPy, 23 | title = {RcppCNPy: Read-Write Support for 'NumPy' Files via 'Rcpp'}, 24 | author = {Dirk Eddelbuettel and Wush Wu}, 25 | year = 2016, 26 | note = {R package version 0.2.5}, 27 | url = {https://CRAN.R-Project.org/package=RcppCNPy} 28 | } 29 | 30 | @Article{NumPy, 31 | author = {Stéfan {van der Walt} and S. Chris Colbert and 32 | Gaël Varoquaux}, 33 | title = {The NumPy Array: A Structure for Efficient Numerical 34 | Computation}, 35 | journal = {Computing in Science & Engineering}, 36 | year = 2011, 37 | volume = 13, 38 | number = 2, 39 | pages = {22-30}, 40 | url = {http://scitation.aip.org/content/aip/journal/cise/13/2/10.1109/MCSE.2011.37}, 41 | doi = {10.1109/MCSE.2011.37} 42 | } 43 | 44 | @misc{CNPy, 45 | author = {Carl Rogers}, 46 | title = {cnpy}, 47 | year = 2015, 48 | publisher = {GitHub}, 49 | journal = {GitHub repository}, 50 | howpublished = {\url{https://github.com/rogersce/cnpy}}, 51 | commit = {4f57d6a0e4c030202a07a60bc1bb1ed1544bf679} 52 | } -------------------------------------------------------------------------------- /vignettes/rmd/RcppCNPy.bib: -------------------------------------------------------------------------------- 1 | @Manual{CRAN:Rcpp, 2 | title = {{Rcpp}: Seamless {R} and {C++} Integration}, 3 | author = {Dirk Eddelbuettel and Romain Fran\c{c}ois and JJ 4 | Allaire and Kevin Ushey and Qiang Kou and Nathan 5 | Russel and John Chambers and Douglas Bates}, 6 | year = 2018, 7 | note = {R package version 0.12.17}, 8 | url = CRAN # "package=Rcpp" 9 | } 10 | 11 | @Manual{CRAN:reticulate, 12 | title = {reticulate: Interface to 'Python'}, 13 | author = {JJ Allaire and Kevin Ushey and Yuan Tang}, 14 | year = 2018, 15 | note = {R package version 1.9}, 16 | url = {https://CRAN.R-project.org/package=reticulate}, 17 | } 18 | 19 | @Book{Eddelbuettel:2013:Rcpp, 20 | author = {Dirk Eddelbuettel}, 21 | title = {Seamless R and C++ Integration with Rcpp}, 22 | publisher = {Springer}, 23 | series = {Use R!}, 24 | year = 2013, 25 | address = {New York}, 26 | isbn = {978-1-4614-6867-7} 27 | } 28 | 29 | @Article{JOSS:RcppCNPy, 30 | title = {{RcppCNPy}: Read-Write Support for NumPy Files in R}, 31 | author = {Dirk Eddelbuettel and Wush Wu}, 32 | journal = {Journal of Open Source Software}, 33 | publisher = {The Open Journal}, 34 | year = 2016, 35 | volume = 1, 36 | issue = 5, 37 | month = {September}, 38 | url = {http://dx.doi.org/10.21105/joss.00055}, 39 | } 40 | 41 | @Article{JSS:Rcpp, 42 | title = {{Rcpp}: Seamless {R} and {C++} Integration}, 43 | author = {Dirk Eddelbuettel and Romain Fran\c{c}ois}, 44 | journal = {Journal of Statistical Software}, 45 | year = 2011, 46 | volume = 40, 47 | number = 8, 48 | pages = {1--18}, 49 | url = {http://www.jstatsoft.org/v40/i08/}, 50 | } 51 | -------------------------------------------------------------------------------- /tests/saveAndLoad.R: -------------------------------------------------------------------------------- 1 | #!/usr/bin/Rscript 2 | 3 | library(RcppCNPy) 4 | 5 | set.seed(42) 6 | M1 <- matrix(rnorm(1e6), 1e3, 1e3) 7 | 8 | ## double precision floating point, uncompressed 9 | tempmatfile <- tempfile(pattern="npymat", fileext=".npy") 10 | invisible(npySave(tempmatfile, M1)) 11 | M2 <- npyLoad(tempmatfile) 12 | identical(M1, M2) 13 | 14 | ## double precision floating point, compressed 15 | tempmatfile <- tempfile(pattern="npymat", fileext=".npy.gz") 16 | invisible(npySave(tempmatfile, M1)) 17 | M3 <- npyLoad(tempmatfile) 18 | identical(M1, M3) 19 | 20 | ## double precision floating point vector 21 | dim(M1) <- NULL 22 | tempmatfile <- tempfile(pattern="npyvec", fileext=".npy") 23 | invisible(npySave(tempmatfile, M1)) 24 | M2 <- npyLoad(tempmatfile) 25 | identical(M1, M2) 26 | 27 | ## double precision floating point vector, compressed 28 | tempmatfile <- tempfile(pattern="npyvec", fileext=".npy.gz") 29 | invisible(npySave(tempmatfile, M1)) 30 | M3 <- npyLoad(tempmatfile) 31 | identical(M1, M3) 32 | 33 | ## integer, uncompressed 34 | M4 <- matrix(as.integer(round(M1)), 1e3, 1e3) 35 | tempmatfile <- tempfile(pattern="intnpymat", fileext=".npy") 36 | invisible(npySave(tempmatfile, M4)) 37 | M5 <- npyLoad(tempmatfile, "integer") 38 | identical(M4, M5) 39 | 40 | ## integer, compressed 41 | tempmatfile <- tempfile(pattern="intnpymat", fileext=".npy.gz") 42 | invisible(npySave(tempmatfile, M4)) 43 | M6 <- npyLoad(tempmatfile, "integer") 44 | identical(M4, M6) 45 | 46 | ## integer vector, uncompressed 47 | dim(M4) <- NULL 48 | tempmatfile <- tempfile(pattern="intnpyvec", fileext=".npy") 49 | invisible(npySave(tempmatfile, M4)) 50 | M5 <- npyLoad(tempmatfile, "integer") 51 | identical(M4, M5) 52 | 53 | ## integer vector, compressed 54 | tempmatfile <- tempfile(pattern="intnpyvec", fileext=".npy.gz") 55 | invisible(npySave(tempmatfile, M4)) 56 | M6 <- npyLoad(tempmatfile, "integer") 57 | identical(M4, M6) 58 | -------------------------------------------------------------------------------- /paper/paper.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: 'RcppCNPy: Read-Write Support for NumPy Files in R' 3 | tags: 4 | - Python 5 | - NumPy 6 | - R 7 | - data transfer 8 | authors: 9 | - name: Dirk Eddelbuettel 10 | orcid: 0000-0001-6419-907X 11 | affiliation: Debian and R Projects 12 | - name: Wush Wu 13 | orcid: 0000-0001-5180-0567 14 | affiliation: Institute of Electrical Engineering, National Taiwan University 15 | date: 26 August 2016 16 | bibliography: paper.bib 17 | --- 18 | 19 | # Summary 20 | 21 | [Python](https://https://www.python.org/) is a widely-used and popular programming 22 | language. It is deployed in use cases ranging from simple scripting to larger-scale 23 | application development. [Python](https://https://www.python.org/) is also popular for 24 | quantitative and scientific application due to the existence of extension modules such as 25 | [NumPy](http://www.numpy.org/) (which is shorthand for Numeric Python) and many other 26 | packages for data analysis. 27 | 28 | [NumPy](http://www.numpy.org/) [@NumPy] is used to efficiently represent N-dimensional arrays, and 29 | provides an efficient binary storage model for these files. In practice, N is often equal 30 | to two, and matrices processed or generated in [Python](https://https://www.python.org/) 31 | can be stored in this form. As [NumPy](http://www.numpy.org/) is popular, many project 32 | utilize this file format. 33 | 34 | [R](https://www.r-project.org) has no dedicated reading or writing functionality for these 35 | type of files. However, Carl Rogers has provided a small C++ library called 36 | [cnpy](https://github.com/rogersce/cnpy) [@CNPy]. Using the _Rcpp modules_ feature in 37 | [Rcpp](http://dirk.eddelbuettel.com/code/rcpp.html) [@Eddelbuettel:2013:Rcpp,@CRAN:Rcpp], 38 | we provide (some) features of this library to [R](https://www.r-project.org) via the 39 | [RcppCNPy](https://dirk.eddelbuettel.com/code/rcpp.cnpy.html) package [@CRAN:RcppCNPy]. 40 | 41 | # References 42 | 43 | -------------------------------------------------------------------------------- /tests/saveAndLoad.Rout.save: -------------------------------------------------------------------------------- 1 | 2 | R version 4.4.1 (2024-06-14) -- "Race for Your Life" 3 | Copyright (C) 2024 The R Foundation for Statistical Computing 4 | Platform: x86_64-pc-linux-gnu 5 | 6 | R is free software and comes with ABSOLUTELY NO WARRANTY. 7 | You are welcome to redistribute it under certain conditions. 8 | Type 'license()' or 'licence()' for distribution details. 9 | 10 | R is a collaborative project with many contributors. 11 | Type 'contributors()' for more information and 12 | 'citation()' on how to cite R or R packages in publications. 13 | 14 | Type 'demo()' for some demos, 'help()' for on-line help, or 15 | 'help.start()' for an HTML browser interface to help. 16 | Type 'q()' to quit R. 17 | 18 | > #!/usr/bin/Rscript 19 | > 20 | > library(RcppCNPy) 21 | > 22 | > set.seed(42) 23 | > M1 <- matrix(rnorm(1e6), 1e3, 1e3) 24 | > 25 | > ## double precision floating point, uncompressed 26 | > tempmatfile <- tempfile(pattern="npymat", fileext=".npy") 27 | > invisible(npySave(tempmatfile, M1)) 28 | > M2 <- npyLoad(tempmatfile) 29 | > identical(M1, M2) 30 | [1] TRUE 31 | > 32 | > ## double precision floating point, compressed 33 | > tempmatfile <- tempfile(pattern="npymat", fileext=".npy.gz") 34 | > invisible(npySave(tempmatfile, M1)) 35 | > M3 <- npyLoad(tempmatfile) 36 | > identical(M1, M3) 37 | [1] TRUE 38 | > 39 | > ## double precision floating point vector 40 | > dim(M1) <- NULL 41 | > tempmatfile <- tempfile(pattern="npyvec", fileext=".npy") 42 | > invisible(npySave(tempmatfile, M1)) 43 | Saving Numeric Vector 44 | > M2 <- npyLoad(tempmatfile) 45 | > identical(M1, M2) 46 | [1] TRUE 47 | > 48 | > ## double precision floating point vector, compressed 49 | > tempmatfile <- tempfile(pattern="npyvec", fileext=".npy.gz") 50 | > invisible(npySave(tempmatfile, M1)) 51 | Saving Numeric Vector 52 | > M3 <- npyLoad(tempmatfile) 53 | > identical(M1, M3) 54 | [1] TRUE 55 | > 56 | > ## integer, uncompressed 57 | > M4 <- matrix(as.integer(round(M1)), 1e3, 1e3) 58 | > tempmatfile <- tempfile(pattern="intnpymat", fileext=".npy") 59 | > invisible(npySave(tempmatfile, M4)) 60 | > M5 <- npyLoad(tempmatfile, "integer") 61 | > identical(M4, M5) 62 | [1] TRUE 63 | > 64 | > ## integer, compressed 65 | > tempmatfile <- tempfile(pattern="intnpymat", fileext=".npy.gz") 66 | > invisible(npySave(tempmatfile, M4)) 67 | > M6 <- npyLoad(tempmatfile, "integer") 68 | > identical(M4, M6) 69 | [1] TRUE 70 | > 71 | > ## integer vector, uncompressed 72 | > dim(M4) <- NULL 73 | > tempmatfile <- tempfile(pattern="intnpyvec", fileext=".npy") 74 | > invisible(npySave(tempmatfile, M4)) 75 | > M5 <- npyLoad(tempmatfile, "integer") 76 | > identical(M4, M5) 77 | [1] TRUE 78 | > 79 | > ## integer vector, compressed 80 | > tempmatfile <- tempfile(pattern="intnpyvec", fileext=".npy.gz") 81 | > invisible(npySave(tempmatfile, M4)) 82 | > M6 <- npyLoad(tempmatfile, "integer") 83 | > identical(M4, M6) 84 | [1] TRUE 85 | > 86 | > proc.time() 87 | user system elapsed 88 | 1.544 0.506 1.390 89 | -------------------------------------------------------------------------------- /paper/joss/2016-09-22-more-comments.md: -------------------------------------------------------------------------------- 1 | Thank you for the quick and thorough review, @mdnunez! 2 | 3 | I have a few additional comments/suggestions before acceptance. 4 | 5 | (1) paper.md didn't pull the author information properly (hopefully @arfon can look into that), however, the package vignette on CRAN (https://cran.r-project.org/web/packages/RcppCNPy/vignettes/RcppCNPy-intro.pdf) includes a second author, also listed in the README.md, that isn't included in the JOSS submission. They should probably also be included in paper.md. 6 | 7 | (2) Since the package vignette in CRAN includes more information, it would be good to also reference/point-to that as well (in paper.md). 8 | 9 | (3) README.md should have a small section for the installation instructions. I think it should include both the install.packages("RcppCNPy"), as well as install.packages("eddelbuettel/rcppcnpy"), should the CRAN package be a slightly older version than that found on GitHub. 10 | 11 | (4) README.md should also include a sentence pointing to the tests directory. A short description of each test (in README.md or in a separate file) would also be beneficial. 12 | 13 | (5) It would be good to cite NumPy in the paper, see https://www.scipy.org/citing.html. 14 | 15 | 16 | 17 | 18 | 19 | Briefly 20 | 21 | (1) I will add Wush as he is a contributor to the package. 22 | (2) Yes, that occurred to me too, and I saw other JOSS submission do it. Will do. 23 | (3) I will add the usual two-liner about install.packages(). The delta is temporary and are the changes I made in the 24 hours in response to this very issue ticket / review, and are in a branch. (I consider CRAN to be the primary and always push releases. GitHub is my development repository. This policy has been consistent and transparent for ten+ years across two dozen packages. I do not plan to change that.) 24 | (4) No. It is a CRAN package, and R users understand how to tickle tests. The README in the repository is not a replacement for the R manuals. 25 | (5) I look for such citation information, but found it wanting. If anything, the cnpy repository should be cited (and I will add this). RcppCNPy does not contain NumPy code or functionality so this is a bit of a stretch. But I will add it. 26 | 27 | 28 | 29 | (3) I still think it would be good to add install.packages("RcppCNPy") to the README.md, to be overtly clear how the package can be installed from CRAN. This also may be useful in preventing confusion related to the case-sensitivity (also since the GitHub repo is named in all lowercase). 30 | 31 | (5) Despite the package not using any NumPy code, it is built as a means to read-write from the NumPy data format--that's why I think citing NumPy makes sense. I hope the rationale is more clear now. 32 | 33 | 34 | 35 | 36 | 37 | (3) Yes I agree (and I edited the above commented twice). I checked, and this package does not have the short block most my other packages have -- ie preference for install.packages(). I will add that. 38 | 39 | (5) Yes. No point to debate this to death. Taken further, we could also cite R and Python. But NumPy is appropriate, and cnpy even more so. Any tips on citing 'unpublished' github repos? 40 | -------------------------------------------------------------------------------- /man/RcppCNPy-package.Rd: -------------------------------------------------------------------------------- 1 | \name{RcppCNPy-package} 2 | \alias{RcppCNPy-package} 3 | \alias{RcppCNPy} 4 | \alias{npyLoad} 5 | \alias{npySave} 6 | \alias{npyHasIntegerSupport} 7 | \docType{package} 8 | \title{ 9 | File access to data files written by (or for) NumPy (Numeric Python) modules 10 | } 11 | \description{ 12 | This package provides access to the \code{cnpy} library by Carl Rogers 13 | which provides read and write facilities for files created with (or for) the 14 | NumPy extension for Python. 15 | 16 | Support is provided to reading and writing of either vectors 17 | or matrices of numeric or integer types. 18 | 19 | Files with \code{gzip} compression can be transparently read and 20 | written as well. 21 | } 22 | \usage{ 23 | npyLoad(filename, type="numeric", dotranspose=TRUE) 24 | npySave(filename, object, mode="w", checkPath=TRUE) 25 | npyHasIntegerSupport() 26 | } 27 | \arguments{ 28 | \item{filename}{string with (path and) filename for a \code{npy} 29 | object file. If the string ends with \code{.gz}, compressed files 30 | can be read or written.} 31 | \item{type}{string with type 'numeric' (default) or 'integer'.} 32 | \item{object}{an R object, currently limited to a vector or matrix of 33 | either integer or numeric type} 34 | \item{dotranspose}{a boolean variable indicating whether a 35 | two-dimensional object should be transposed after reading, default 36 | is true} 37 | \item{mode}{a one-character string indicating whether files are 38 | appended to ("a") or written ("w", the default). In case of writing 39 | \code{gzip}-ed file, this option is not supported as such files can 40 | only be (over-)written, and bot appended. 41 | } 42 | \item{checkPath}{a boolean variable indicating whether a path implied 43 | in the \code{filename} argument is to be checked for existing 44 | directories, default is true. 45 | } 46 | } 47 | \details{ 48 | The package uses Rcpp modules to provide R bindings \code{npyLoad()} 49 | and \code{npySave()} which wrap the \code{npy_load()} and 50 | \code{npy_save()} functions. Currently, only one- and two-dimensional 51 | vectors and matrices are suppported; higher-dimensional arrays could 52 | be added. 53 | 54 | Integer support requires access to the \code{long long} type which is 55 | available if the package is built using the C++11 standard; this is 56 | the default since release 0.2.3 which came out after R 3.1.0 permitted 57 | use of C++11 in CRAN packages. 58 | 59 | Note that R uses only one \code{integer} type (which uses 32 bits) and 60 | one \code{double} floating point type (which uses 64 bits). If Python 61 | data of either type with a different bitsize is to be shared with R, 62 | it has be cast to the corresponding width used by R first. 63 | } 64 | \author{ 65 | Dirk Eddelbuettel provided the binding to R (using the Rcpp package). 66 | 67 | Carl Rogers wrote the underlying \code{cnpy} library, which is 68 | released under the MIT license. 69 | 70 | Maintainer: Dirk Eddelbuettel 71 | } 72 | \references{ 73 | Rcpp, in particular the Rcpp modules documentation. 74 | 75 | The \code{cnpy} repository: \url{https://github.com/rogersce/cnpy} 76 | } 77 | \keyword{package} 78 | \seealso{ 79 | \code{\link[Rcpp:Rcpp-package]{Rcpp}} 80 | } 81 | \examples{ 82 | \dontrun{ 83 | library(RcppCNPy) 84 | 85 | ## load NumPy file with floating-point data 86 | fmat <- npyLoad("fmat.npy") 87 | 88 | ## load NumPy file with integer data 89 | imat <- npyLoad("imat.npy", "integer") 90 | 91 | ## save floating-point data: matrix and vector 92 | M <- matrix(0:11, 3, 4, byrow=TRUE) * 1.1 93 | v <- v <- 0:4 * 1.1 94 | npySave("fmat.npy", M) 95 | npySave("fvec.npy", v) 96 | 97 | ## save integer data: matrix and vector 98 | M <- matrix(0:11, 3, 4, byrow=TRUE) 99 | v <- v <- 0:4 100 | npySave("imat.npy", M) 101 | npySave("ivec.npy", v) 102 | 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## RcppCNPy: Rcpp bindings for NumPy files 2 | 3 | [![CI](https://github.com/eddelbuettel/rcppcnpy/workflows/ci/badge.svg)](https://github.com/eddelbuettel/rcppcnpy/actions?query=workflow%3Aci) 4 | [![License](https://img.shields.io/badge/license-GPL%20%28%3E=%202%29-brightgreen.svg?style=flat)](https://www.r-project.org/Licenses/GPL-2) 5 | [![CRAN](https://www.r-pkg.org/badges/version/RcppCNPy)](https://cran.r-project.org/package=RcppCNPy) 6 | [![Dependencies](https://tinyverse.netlify.app/badge/RcppCNPy)](https://cran.r-project.org/package=RcppCNPy) 7 | [![Downloads](https://cranlogs.r-pkg.org/badges/RcppCNPy?color=brightgreen)](https://www.r-pkg.org:443/pkg/RcppCNPy) 8 | [![Last Commit](https://img.shields.io/github/last-commit/eddelbuettel/rcppcnpy)](https://github.com/eddelbuettel/rcppcnpy) 9 | [![status](https://joss.theoj.org/papers/10.21105/joss.00055/status.svg)](https://joss.theoj.org/papers/10.21105/joss.00055) 10 | [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.155066.svg)](https://doi.org/10.5281/zenodo.155066) 11 | 12 | ### About 13 | 14 | This package uses the [cnpy](https://github.com/rogersce/cnpy) library 15 | written by Carl Rogers to provide read and write facilities for files created 16 | with (or for) the NumPy extension for Python. Vectors and matrices of 17 | numeric types can be read or written to and from files as well as compressed 18 | files. Support for integer files is available if the package has been built 19 | with `-std=c++11` which is the default starting with release 0.2.3 following 20 | the release of R 3.1.0, and available on all platforms following the release 21 | of R 3.3.0 with the updated 'Rtools'. 22 | 23 | ### Example 24 | 25 | The following Python code 26 | 27 | ```{.python} 28 | >>> import numpy as np 29 | >>> fm = np.arange(12).reshape(3,4) * 1.1 30 | >>> fm 31 | array([[ 0. , 1.1, 2.2, 3.3], 32 | [ 4.4, 5.5, 6.6, 7.7], 33 | [ 8.8, 9.9, 11. , 12.1]]) 34 | >>> np.save("fmat.npy", fm) 35 | >>> 36 | >>> im = np.arange(12).reshape(3,4) 37 | >>> im 38 | array([[ 0, 1, 2, 3], 39 | [ 4, 5, 6, 7], 40 | [ 8, 9, 10, 11]]) 41 | >>> np.save("imat.npy", im) 42 | >>> 43 | ``` 44 | 45 | saves two matrices in floating-point and integer representation. 46 | 47 | With this R code we can read and assign the files: 48 | 49 | ```{.r} 50 | R> library(RcppCNPy) 51 | R> fmat <- npyLoad("fmat.npy") 52 | R> fmat 53 | [,1] [,2] [,3] [,4] 54 | [1,] 0.0 1.1 2.2 3.3 55 | [2,] 4.4 5.5 6.6 7.7 56 | [3,] 8.8 9.9 11.0 12.1 57 | R> 58 | R> imat <- npyLoad("imat.npy", "integer") 59 | R> imat 60 | [,1] [,2] [,3] [,4] 61 | [1,] 0 1 2 3 62 | [2,] 4 5 6 7 63 | [3,] 8 9 10 11 64 | R> 65 | ``` 66 | 67 | Going the opposite way by saving in R and reading in Python works equally 68 | well. An extension not present in [CNPy](https://github.com/rogersce/cnpy) 69 | allows reading and writing of gzip-compressed files. 70 | 71 | The package has been tested and used on several architecture, and copes 72 | correctly with little-vs-big endian switches. 73 | 74 | More details are available in the [package vignette](https://cran.r-project.org/package=RcppCNPy/vignettes/RcppCNPy-intro.pdf). 75 | 76 | ### Installation 77 | 78 | The package is on [CRAN](https://cran.r-project.org) and can be installed per: 79 | 80 | ```{r} 81 | R> install.packages("RcppCNPy") 82 | ``` 83 | 84 | ### Status 85 | 86 | On [CRAN](https://cran.r-project.org/package=RcppCNPy), stable and mostly 87 | feature-complete. 88 | 89 | ### Alternative: [reticulate](https://github.com/rstudio/reticulate) 90 | 91 | The [reticulate](https://github.com/rstudio/reticulate) package can also provide easy and comprehensive access 92 | to NumPy data; see the 93 | [additional vignette in RcppCNPy](https://cran.r-project.org/package=RcppCNPy/vignettes/UsingReticulate.pdf) 94 | for examples and more details. 95 | 96 | ### Feedback 97 | 98 | Contributions are welcome, please use the GitHub 99 | [issue tracker](https://github.com/eddelbuettel/rcppcnpy/issues) for 100 | bug reports, feature requests or general discussions before sending 101 | [pull requests](https://github.com/eddelbuettel/rcppcnpy/pulls). 102 | 103 | ### Author 104 | 105 | Dirk Eddelbuettel and Wush Wu 106 | 107 | ### License 108 | 109 | GPL (>= 2) 110 | -------------------------------------------------------------------------------- /paper/joss/2016-09-20-review.md: -------------------------------------------------------------------------------- 1 | # Review of RcppCNPy: Read-Write Support for NumPy Files in R 2 | 3 | 4 | ## Reviewer questions 5 | 6 | ### Conflict of interest 7 | 8 | - [x] As the reviewer I confirm that there are no conflicts of interest for me to review this work (such as being a major contributor to the software). 9 | 10 | ### General checks 11 | 12 | - [x] **Repository:** Is the source code for this software available at the repository url? 13 | 14 | > Yes, it is also available from CRAN with install.packages('rcppcnpy') 15 | 16 | - [x] **License:** Does the repository contain a plain-text LICENSE file with the contents of an [OSI approved](https://opensource.org/licenses/alphabetical) software license? 17 | 18 | > Yes, except it is GNU 2, not GNU 3 or later. Also, should GNU statement be at the top of each script? 19 | 20 | - [x] **Version:** Does the release version given match the GitHub release (0.2.5)? 21 | 22 | ### Functionality 23 | 24 | - [ ] **Installation:** Does installation proceed as outlined in the documentation? 25 | 26 | > README does not list installation instructions, easy fix 27 | > library(rcppcnpy) did not work for me 28 | > It seems library('RcppCNPy') command is case sensitive. 29 | 30 | - [x] **Functionality:** Have the functional claims of the software been confirmed? 31 | 32 | - [x] **Performance:** Have any performance claims of the software been confirmed? 33 | 34 | > N/A? I did not see any performance claims in README, I found performance claims in https://cran.r-project.org/web/packages/RcppCNPy/vignettes/RcppCNPy-intro.pdf 35 | 36 | ### Documentation 37 | 38 | - [x] **A statement of need:** Do the authors clearly state what problems the software is designed to solve and who the target audience is? 39 | 40 | > Target audience could be better stated in README and vignette 41 | 42 | - [x] **Installation instructions:** Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution. 43 | 44 | > I'm not sure if the authors would also like to include the list of Python dependencies since Python example code is given in the README 45 | 46 | - [x] **Example usage:** Do the authors include examples of how to use the software (ideally to solve real-world analysis problems). 47 | 48 | - [x] **Functionality documentation:** Is the core functionality of the software documented to a satisfactory level (e.g. API method documentation)? 49 | 50 | > Functionality is better stated in vignette. README does not contain exhaustive list of functionality 51 | 52 | - [ ] **Automated tests:** Are there automated tests or manual steps described so that the function of the software can be verified? 53 | 54 | > Tests were performed, but I did not find a .md or README if that is required by this journal 55 | 56 | - [ ] **Community guidelines:** Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support 57 | 58 | > Community guidelines for contributions could be better reported 59 | 60 | ### Software paper 61 | 62 | Paper PDF: [10.21105.joss.00055.pdf](https://github.com/openjournals/joss-reviews/files/441216/10.21105.joss.00055.pdf) 63 | 64 | - [x] **Authors:** Does the `paper.md` file include a list of authors with their affiliations? 65 | 66 | > Author names present in paper.md but did not compile to pdf 67 | 68 | - [x] **A statement of need:** Do the authors clearly state what problems the software is designed to solve and who the target audience is? 69 | 70 | - [ ] **References:** Do all archival references that should have a DOI list one (e.g. papers, datasets, software)? 71 | 72 | > References do not have a DOI in pdf 73 | 74 | ### Recommendations to editor 75 | 76 | While the software works well and quickly, some additional documentation is required by JOSS for publication. 77 | 78 | ### Recommendations to author 79 | 80 | This reviewer would like to see reading and writing capability of .npz in the future. npyLoad of integer arrays produces incorrect values when not given the flag "integer." Multidimensional arrays should also be easily loadable. I was impressed that arrays of sizes (24,) , (1,24), and (24,1) all loaded correctly in R with the existing functionality. 81 | 82 | ### Recommendations to JOSS 83 | 84 | Reviewers should be left with the choice to be anonymous 85 | 86 | 87 | -------------------------------------------------------------------------------- /vignettes/rmd/UsingReticulate.Rmd.orig: -------------------------------------------------------------------------------- 1 | --- 2 | title: Using \pkg{reticulate} to read and write \pkg{NumPy} files 3 | 4 | author: 5 | - name: Dirk Eddelbuettel 6 | affiliation: a 7 | address: 8 | - code: a 9 | address: \url{http://dirk.eddelbuettel.com} 10 | 11 | lead_author_surname: Eddelbuettel 12 | 13 | doi: "https://cran.r-project.org/package=RcppCNPy" 14 | 15 | abstract: | 16 | This vignette shows how to use the \pkg{reticulate} package to directly 17 | access the \pkg{NumPy} module for \proglang{Python}. 18 | 19 | footer_contents: "RcppCNPy Vignette" 20 | 21 | output: pinp::pinp 22 | 23 | header-includes: 24 | \newcommand{\proglang}[1]{\textsf{#1}} 25 | \newcommand{\pkg}[1]{\textbf{#1}} 26 | \newcommand{\R}{\proglang{R}\ } 27 | \newcommand{\Rns}{\proglang{R}} 28 | 29 | include-after: | 30 | \begin{thebibliography}{4} 31 | \newcommand{\enquote}[1]{``#1''} 32 | \providecommand{\natexlab}[1]{#1} 33 | \providecommand{\url}[1]{\texttt{#1}} 34 | \providecommand{\urlprefix}{URL } 35 | \expandafter\ifx\csname urlstyle\endcsname\relax 36 | \providecommand{\doi}[1]{doi:\discretionary{}{}{}#1}\else 37 | \providecommand{\doi}{doi:\discretionary{}{}{}\begingroup 38 | \urlstyle{rm}\Url}\fi 39 | \providecommand{\eprint}[2][]{\url{#2}} 40 | 41 | \bibitem[{Allaire \emph{et~al.}(2018)Allaire, Ushey, RStudio, Tang, Eddelbuettel, 42 | Lewis, and Geelnard}]{CRAN:reticulate} 43 | Allaire JJ, Ushey K, RStudio, Tang Y, Eddelbuettel D, Lewis B, Geelnard M (2018). 44 | \newblock \emph{{reticulate}: Interface to {Python}}. 45 | \newblock R package version 1.5, 46 | \urlprefix\url{http://CRAN.R-Project.org/package=reticulate}. 47 | 48 | \bibitem[{Eddelbuettel and Wu(2016)}]{JOSS:RcppCNPy} 49 | Eddelbuettel D, Wu W (2016). 50 | \newblock \enquote{{RcppCNPy}: Read-Write Support for {NumPy} Files in R.} 51 | \newblock \emph{The Journal of Open Source Software}, \textbf{1}(5). 52 | \newblock \doi{10.21105/joss.00055}. 53 | \newblock \urlprefix\url{https://doi.org/10.21105/joss.00055}. 54 | 55 | \end{thebibliography} 56 | 57 | 58 | vignette: > 59 | %\VignetteIndexEntry{RcppCNPy-reticulate} 60 | %\VignetteKeywords{Python, NumPy, R, data transfer} 61 | %\VignettePackage{RcppCNPy} 62 | %\VignetteEngine{knitr::rmarkdown} 63 | --- 64 | 65 | 66 | ```{r echo = FALSE, message = FALSE} 67 | knitr::opts_chunk$set(python.reticulate = FALSE) 68 | ## knitr::opts_chunk$set(eval = FALSE) 69 | ``` 70 | 71 | # Motivation 72 | 73 | The RcppCNPy package by \citet{JOSS:RcppCNPy} provides a simple and 74 | reliable access to NumPy files. It does not require Python as it relies on 75 | the CNPy library which is connected to R with the help of Rcpp. Now, thanks 76 | to the reticulate package by \citet{CRAN:reticulate}, we can also consider an 77 | alternative which does not require CNPy--but which requires Python. Thanks 78 | to reticulate, we can (on a correctly set up machine, how to do that is 79 | beyond the scope of this note but described in the reticulate documentation) 80 | use Python to read NumPy data. And reticulate will faithfully transfer the 81 | data for us. 82 | 83 | ```{r prep, include=FALSE} 84 | file.copy("~/git/rcppcnpy/tests/fmat.npy", ".") 85 | file.copy("~/git/rcppcnpy/tests/fmat.npy.gz", ".") 86 | file.copy("~/git/rcppcnpy/tests/fvec.npy", ".") 87 | file.copy("~/git/rcppcnpy/tests/imat.npy", ".") 88 | file.copy("~/git/rcppcnpy/tests/ivec.npy", ".") 89 | options(width=50) 90 | ``` 91 | 92 | 93 | # Simple Examples 94 | 95 | ```{r ex1} 96 | ### load reticulate and use it to load numpy 97 | library(reticulate) 98 | np <- import("numpy") 99 | 100 | ## data reading 101 | (mat <- np$load("fmat.npy")) 102 | (vec <- np$load("fvec.npy")) 103 | ``` 104 | 105 | Integer data can be read the same way: 106 | 107 | ```{r ex2} 108 | (imat <- np$load("imat.npy")) 109 | ``` 110 | 111 | # Compressed Files 112 | 113 | The gzip Python module allows us to access compressed files. 114 | 115 | ```{r ex3} 116 | ## compressed data: import gzip 117 | gz <- import("gzip") 118 | 119 | ## use it to create handle to uncompressed file 120 | (mat2 <- np$load(gz$GzipFile("fmat.npy.gz","r"))) 121 | ``` 122 | 123 | # Saving Files 124 | 125 | Similarly, files can be saved via reticulate access to NumPy. 126 | 127 | ```{r ex4} 128 | tfile <- tempfile(fileext=".npy") 129 | 130 | set.seed(42) 131 | (m <- matrix(sort(rnorm(6)), 3, 2)) 132 | 133 | np$save(tfile, m) 134 | 135 | (m2 <- np$load(tfile)) 136 | all.equal(m, m2) 137 | ``` 138 | 139 | # Savez Array Files 140 | 141 | We can also access `savez` files. First we save two vectors two different 142 | ways: 143 | 144 | ```{r ex5} 145 | x <- seq(1, 10) 146 | y <- sin(x) 147 | np$savez("file1.npz", x, y) 148 | np$savez("file2.npz", x=x, y=y) 149 | ``` 150 | 151 | We can access these files with and without names: 152 | 153 | ```{r ex6} 154 | npz1 <- np$load("file1.npz") 155 | npz1$files 156 | npz1$f[["arr_0"]] 157 | npz1$f[["arr_1"]] 158 | 159 | 160 | npz2 <- np$load("file2.npz") 161 | npz2$files 162 | npz2$f[["x"]] 163 | npz2$f[["y"]] 164 | 165 | 166 | 167 | ```{r cleanup, include=FALSE} 168 | unlink("fmat.npy") 169 | unlink("fmat.npy.gz") 170 | unlink("fvec.npy") 171 | unlink("imat.npy") 172 | unlink("ivec.npy") 173 | unlink("file1.npz") 174 | unlink("file2.npz") 175 | ``` 176 | -------------------------------------------------------------------------------- /vignettes/rmd/UsingReticulate.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: Using \pkg{reticulate} to read and write \pkg{NumPy} files 3 | 4 | author: 5 | - name: Dirk Eddelbuettel 6 | affiliation: 1 7 | address: 8 | - code: 1 9 | address: http://dirk.eddelbuettel.com 10 | 11 | lead_author_surname: Eddelbuettel 12 | 13 | doi: "https://cran.r-project.org/package=RcppCNPy" 14 | 15 | abstract: | 16 | This vignette shows how to use the \pkg{reticulate} package to directly 17 | access the \pkg{NumPy} module for \proglang{Python}. 18 | 19 | footer_contents: "RcppCNPy Vignette" 20 | 21 | skip_final_break: true 22 | 23 | output: 24 | pinp::pinp: 25 | collapse: true 26 | 27 | header-includes: 28 | \newcommand{\proglang}[1]{\textsf{#1}} 29 | \newcommand{\pkg}[1]{\textbf{#1}} 30 | \newcommand{\R}{\proglang{R}\ } 31 | \newcommand{\Rns}{\proglang{R}} 32 | 33 | bibliography: RcppCNPy.bib 34 | 35 | vignette: > 36 | %\VignetteIndexEntry{RcppCNPy-reticulate} 37 | %\VignetteKeywords{Python, NumPy, R, data transfer} 38 | %\VignettePackage{RcppCNPy} 39 | %\VignetteEngine{knitr::rmarkdown} 40 | --- 41 | 42 | 43 | ```{r echo = FALSE, message = FALSE} 44 | knitr::opts_chunk$set(python.reticulate = TRUE) 45 | if (identical(Sys.info()[['sysname']], "Windows")) { 46 | knitr::opts_chunk$set(eval = FALSE) 47 | msg <- paste("Note: Some examples in this vignette require Python", 48 | "but you are running this vignette on Windows where Python", 49 | "is much less likely to be present, or even known to be", 50 | "missing (i.e. win-builder) so examples will not be evaluated.") 51 | msg <- paste(strwrap(msg), collapse="\n") 52 | message(msg) 53 | } 54 | options(width=50) 55 | ``` 56 | 57 | # Motivation 58 | 59 | The \pkg{RcppCNPy} package by \citet{JOSS:RcppCNPy} provides a simple and 60 | reliable access to \pkg{NumPy} files. It does not require \proglang{Python} as it 61 | relies on the \pkg{cnpy} library which is connected to \proglang{R} with the help 62 | of \pkg{Rcpp} \pkg{Rcpp} \shortcites{CRAN:Rcpp} 63 | \citep{JSS:Rcpp,Eddelbuettel:2013:Rcpp,CRAN:Rcpp}. 64 | 65 | Now, thanks to the \pkg{reticulate} package by \citet{CRAN:reticulate}, we 66 | can consider an alternative which does not require \pkg{cnpy}--but which requires 67 | Python. We can (on a correctly set up machine, how to do that is beyond the 68 | scope of this note but described in the reticulate documentation) use Python 69 | to read \pkg{NumPy} data via \pkg{reticulate}. 70 | 71 | ```{r prep, include=FALSE, eval=TRUE} 72 | file.copy("../tests/fmat.npy", ".") 73 | file.copy("../tests/fmat.npy.gz", ".") 74 | file.copy("../tests/fvec.npy", ".") 75 | file.copy("../tests/imat.npy", ".") 76 | file.copy("../tests/ivec.npy", ".") 77 | file.copy("../tests/arr.npy", ".") 78 | ``` 79 | 80 | This short note reproduces all the examples in the primary \pkg{RcppCNPy} 81 | vignette, but using \pkg{reticulate} instead of \pkg{cnpy}. 82 | 83 | # Simple Examples 84 | 85 | ```{r ex1} 86 | # load reticulate and use it to load numpy 87 | library(reticulate) 88 | np <- import("numpy") 89 | 90 | # data reading 91 | mat <- np$load("fmat.npy") 92 | mat 93 | 94 | vec <- np$load("fvec.npy") 95 | vec 96 | ``` 97 | 98 | Integer data can be read the same way: 99 | 100 | ```{r ex2} 101 | imat <- np$load("imat.npy") 102 | imat 103 | ``` 104 | 105 | # Compressed Files 106 | 107 | The gzip \proglang{Python} module allows us to access compressed files. 108 | 109 | ```{r ex3} 110 | # use the gzip modules for compressed data 111 | gz <- import("gzip") 112 | # use it to create handle to uncompressed file 113 | mat2 <- np$load(gz$GzipFile("fmat.npy.gz","r")) 114 | mat2 115 | ``` 116 | 117 | 118 | # Saving Files 119 | 120 | Similarly, files can be saved via reticulate access to \pkg{NumPy}. 121 | 122 | ```{r ex4} 123 | tfile <- tempfile(fileext=".npy") 124 | set.seed(42) 125 | m <- matrix(sort(rnorm(6)), 3, 2) 126 | m 127 | np$save(tfile, m) 128 | 129 | m2 <- np$load(tfile) 130 | m2 131 | 132 | all.equal(m, m2) 133 | ``` 134 | 135 | 136 | # Savez Array Files 137 | 138 | We can also access `savez` files. 139 | 140 | First we save two vectors two different ways: 141 | 142 | ```{r ex5} 143 | x <- seq(1, 10) 144 | y <- sin(x) 145 | np$savez("file1.npz", x, y) 146 | np$savez("file2.npz", x=x, y=y) 147 | ``` 148 | 149 | We can access these files with and without names: 150 | 151 | ```{r ex6} 152 | npz1 <- np$load("file1.npz") 153 | npz1$files 154 | npz1$f[["arr_0"]] 155 | npz1$f[["arr_1"]] 156 | ``` 157 | 158 | Ditto for the second file: 159 | 160 | ```{r ex7} 161 | npz2 <- np$load("file2.npz") 162 | npz2$files 163 | npz2$f[["x"]] 164 | npz2$f[["y"]] 165 | ``` 166 | 167 | # Three-dimentional Arrays 168 | 169 | We can also import three-dimensional array from \pkg{NumPy} as the 170 | next example shows. 171 | 172 | ```{r ex8} 173 | arr <- np$load("arr.npy") 174 | arr 175 | ``` 176 | 177 | ```{r cleanup, include=FALSE} 178 | unlink("fmat.npy") 179 | unlink("fmat.npy.gz") 180 | unlink("fvec.npy") 181 | unlink("imat.npy") 182 | unlink("ivec.npy") 183 | unlink("file1.npz") 184 | unlink("file2.npz") 185 | unlink("arr.npy") 186 | ``` 187 | 188 | # Summary 189 | 190 | While the \pkg{RcppCNPy} package provides functions for the simple reading 191 | and writing of \pkg{NumPy} files, we can also use the \pkg{reticulate} 192 | package to access the \pkg{NumPy} functionality directly from \Rns. 193 | 194 | \pkg{RcppCNPy} remains an attractive option for simple, direct and lighter-weight 195 | file imports whereas \pkg{reticulate} by by \citet{CRAN:reticulate} shines a more 196 | full-featured access to many more aspects of \proglang{Python}. 197 | 198 | 199 | -------------------------------------------------------------------------------- /paper/joss/2016-09-21-reply.md: -------------------------------------------------------------------------------- 1 | # Reply to Review Comments on _RcppCNPy: Read-Write Support for NumPy Files in R_ 2 | 3 | Thank you for your very detailed review and comments. 4 | 5 | The package and paper have been updated updated to reflect these suggestions, and the 6 | additional documentation is very beneficial. 7 | 8 | This reply contains two major section. The first is 'open issues' and addresess items 9 | which were brought up as required improvements or changes. The second is 'comments' and 10 | either responds to some minor items, or provides other clarification. 11 | 12 | 13 | ## Open Issues 14 | 15 | ### Functionality: Installation 16 | 17 | R packages need to be loaded, so `library("RcppCNPy")` (which is case-sensitive) is indeed 18 | required. 19 | 20 | We do have a section in the `README.md` showing this as floating point (shown below) and 21 | also integer: 22 | 23 | ```r 24 | R> library(RcppCNPy) 25 | R> fmat <- npyLoad("fmat.npy") 26 | ``` 27 | 28 | The package contains a demo file (which can be executed via `demo("timings", package="RcppCNPy"`) 29 | showing this, and the vignette also shows it. 30 | 31 | To be even more explicit we added this also to the 'Examples' sections of the help pages. 32 | 33 | That provides four different instances showing how to load the package and perform a few 34 | first steps which should be sufficient for users of the package. 35 | 36 | ### Documentation: Automated Tests 37 | 38 | The package already contains tests which are executed each time `R CMD check` runs. The 39 | standard R mechanism of using files in the top-level directory `tests/` is used. 40 | 41 | There are two levels of tests. 42 | 43 | Files ending in `.Rout.save` provide (persistent) reference output which is compared 44 | character-by-character to freshly created output during a test. Any differences are 45 | reported (but are not fatal). 46 | 47 | Additionally, the file `tests/loadFiles.R` contains five `stopifnot()` tests which would 48 | abort unless the tested condition (of comparing read content to expected content) are met 49 | exactly. That provides stringent unit testing. 50 | 51 | It is unclear whether an additional markdown or README file is required as the package 52 | conforms to standard R and CRAN practice. 53 | 54 | ### Documentation: Community Guidelines 55 | 56 | The `DESCRIPTION` file lists the standard GitHub facilities: 57 | 58 | ``` 59 | BugReports: https://github.com/eddelbuettel/rcppcnpy/issues 60 | ``` 61 | 62 | By being on GitHub and listing the repository, the standard and well-known participation 63 | venues offered by GitHub are available. 64 | 65 | Following this suggestion, we made this more explicit by adding a short section to the 66 | README.md as well. 67 | 68 | ### Software Paper: References 69 | 70 | We added the DOIs (as supplied by the publisher) for one of the two references. The 71 | second reference does not yet have a DOI. 72 | 73 | We also added the DOI for this submission as a 'badge' to the main `README.md`. 74 | 75 | 76 | 77 | ## Comments 78 | 79 | ### General Checks: Repository 80 | 81 | The installation command for R is case-sensitive so the most recent released version can 82 | be installed via `install.packages("RcppCNPy"). 83 | 84 | Should this be desired, R also allows installation directly from the GitHub repository by 85 | supplying `eddelbuettel/rcppcnpy` as the (unique) 'author/repo' token. But as RcppCNPy is 86 | a CRAN package so we prefer the more common first approach using `install.packages()`. 87 | 88 | ### General Check: License 89 | 90 | [Line 17 of `DESCRIPTION`](https://github.com/eddelbuettel/rcppcnpy/blob/master/DESCRIPTION#L17) 91 | clearly states `License: GPL (>= 2)`. This exact form is described in the corresponding R 92 | manuals, and _e.g._ expands on the 93 | [CRAN webpage for the package](https://cloud.r-project.org/web/packages/RcppCNPy/index.html) 94 | to 'GPL-2 | GPL-3'. Both licenses are 95 | [OSI approved](https://opensource.org/licenses/alphabetical), and used by R itself in the same form. 96 | 97 | The GNU statement was already present at `src/cnpyMod.cpp`, the main C++ intgegration 98 | script. Files `src/cnpy.cpp` and `src/cnpy.h` carry both a statement for the MIT license 99 | of the underlying cnpy library by Rogers, and the a short sentence stating GPL for the 100 | additional integration provided by this package. 101 | 102 | The remaining source file `R/cnpy.R` is a one-liner so that adding a copyright header 103 | seemed excessive. 104 | 105 | 106 | ### Functionality: Performance 107 | 108 | In general, code from a vignette can always be extracted via `Stangle()`. Here, we chose 109 | to keep the result fixed and static to not endure the (still short) simulation on each 110 | latex-compilation of the vignette. However, the package contains a demo detailing this 111 | which can be seen via `demo("timings", package="RcppCNPy")`. It contains the same 112 | benchmark. This permits each user to validate performance locally. 113 | 114 | ### Documentation: Statement of Need 115 | 116 | The package provides an input/ouput facility so that R users can read and/or write files 117 | in the NumPy format. As such, it the target audience is not restricted to any class or 118 | set of users, members of discipline, or methodological school. 119 | 120 | The key focus is stated is stated in the package description, in the first full sentence 121 | in the README.md, and in the first line of the abstract of the vignette. 122 | 123 | ### Documentation: Installation Instructions 124 | 125 | As an R package, there should be rather a limited need for the Python specifics 126 | particularly as every NumPy version appears to provide the identical output. As such, we 127 | have not seen any need to provide more detail. 128 | 129 | ### Documentation: Functionality Documentation 130 | 131 | As an R package, primary documentation is provided via the help pages, and then the vignette. Both detail how to use the package. 132 | 133 | ### Recommendation To The Author 134 | 135 | Adding `.npz` files has long been noted as possible extensions, this is also explicitly 136 | stated in the vignette. Development of the package was needs-driven: we needed to process 137 | certain files, and the `npz` was not a format we used. Pull-requests by users with the 138 | need to process such files are therefore strongly encouraged. 139 | 140 | The need to explicitly request `integer` format is a limitation of the format and the 141 | underlying library. There appears to be no meta-data signalling the content. 142 | 143 | ### Recommendation To JOSS 144 | 145 | I second this recommendation. 146 | -------------------------------------------------------------------------------- /inst/NEWS.Rd: -------------------------------------------------------------------------------- 1 | \name{NEWS} 2 | \title{News for Package \pkg{RcppCNPy}} 3 | \newcommand{\cpkg}{\href{http://CRAN.R-project.org/package=#1}{\pkg{#1}}} 4 | 5 | \section{Changes in version 0.2.14 (2024-11-03)}{ 6 | \itemize{ 7 | \item The \pkg{rbenchmark} package is now a Suggests: as it appears in 8 | \code{demo} 9 | \item The continuous integration setup now uses \code{r-ci} with its 10 | embedded setup step 11 | \item The URL used for the GPL-2 is now the R Project copy 12 | } 13 | } 14 | 15 | \section{Changes in version 0.2.13 (2024-09-03)}{ 16 | \itemize{ 17 | \item A test script was updated to account for the fact that it now 18 | returns a few instances of NULL under current \pkg{Rcpp}. 19 | \item Small package maintenance updates have been made to the README and 20 | DESCRIPTION files as well as to the continuous integration setup. 21 | } 22 | } 23 | 24 | \section{Changes in version 0.2.12 (2023-11-27)}{ 25 | \itemize{ 26 | \item The continuous integration workflow received a trivial update, 27 | twice. 28 | \item The C++ compilation standard is now implicit per CRAN and R 29 | preference. 30 | \item The CITATION file format has been updated for the current usage. 31 | \item Two print format string issues reported by current R-devel have 32 | been addressed. 33 | } 34 | } 35 | 36 | \section{Changes in version 0.2.11 (2022-03-24)}{ 37 | \itemize{ 38 | \item The reticulate vignette has new section on 3d-arrays. 39 | \item Added clarification to the manual page that the default types 40 | are 32-bit integer and 64-bit double (as we are working with R here). 41 | \item Several updates have been made to the CI system; it now runs 42 | \href{https://eddelbuettel.github.io/r-ci/}{r-ci}. 43 | \item The README.md was updated with some new badges. 44 | \item The vignettes are now pre-made to avoid any external 45 | dependencies. 46 | } 47 | } 48 | 49 | \section{Changes in version 0.2.10 (2018-07-29)}{ 50 | \itemize{ 51 | \item The vignettes have been updated using \sQuote{collapse} mode 52 | and edited. 53 | \item The README.md now refers to \CRANpkg{reticulate} as an 54 | alternative and points to the \dQuote{Using reticulate} vignette. 55 | \item The file \code{src/RcppExports.cpp} is used for package 56 | registration instead of \code{src/init.c}. 57 | } 58 | } 59 | 60 | \section{Changes in version 0.2.9 (2018-03-22)}{ 61 | \itemize{ 62 | \item The \code{npySave} function has a new option to check the path 63 | in the given filename. 64 | \item A new vignette was added showing how the \pkg{reticulate} 65 | package can be used instead. 66 | } 67 | } 68 | 69 | \section{Changes in version 0.2.8 (2018-01-04)}{ 70 | \itemize{ 71 | \item Vignette sets knitr option \code{python.reticulate=FALSE} to 72 | avoid another dependency just for the vignette [CRAN request] 73 | } 74 | } 75 | 76 | \section{Changes in version 0.2.7 (2017-09-22)}{ 77 | \itemize{ 78 | \item Vignette updated to Rmd and use of \code{pinp} package 79 | \item File \code{src/init.c} added for dynamic registration 80 | } 81 | } 82 | 83 | \section{Changes in version 0.2.6 (2016-09-25)}{ 84 | \itemize{ 85 | \item Expanded documentation in README.md 86 | \item Added examples to help page 87 | \item Added CITATION file for JOSS paper 88 | } 89 | } 90 | 91 | \section{Changes in version 0.2.5 (2016-08-26)}{ 92 | \itemize{ 93 | \item Synchronized code with the \code{cnpy} repository 94 | \item Updated vignette 95 | \item Expanded DESCRIPTION 96 | } 97 | } 98 | 99 | \section{Changes in version 0.2.4 (2015-01-05)}{ 100 | \itemize{ 101 | \item Support for saving integer objects was not correct and has been 102 | fixed 103 | \item Support for loading and saving on 'big endian' systems was 104 | incomplete, has been greatly expanded and corrected, thanks in large 105 | part to very diligent testing as well as patching by Wush Wu 106 | \item The implementation now uses const iterators, thanks to a pull 107 | request by Romain Francois 108 | \item The vignette no longer assumes that one can call \code{gzip} 109 | via \code{system} as the world's leading consumer OS may disagree. 110 | } 111 | } 112 | 113 | \section{Changes in version 0.2.3 (2014-04-10)}{ 114 | \itemize{ 115 | \item \code{src/Makevars} now sets \code{CXX_STD = CXX11} which also 116 | provides the \code{long long} type on all platforms, so integer file 117 | support is no longer conditional 118 | \item Consequently, code conditional on \code{RCPP_HAS_LONG_LONG_TYPES} 119 | has been simplified and is no longer conditional. 120 | \item The package now depends on R 3.1.0 or later to allow this. 121 | \item The vignette has been updated and refreshed to reflect this. 122 | } 123 | } 124 | 125 | \section{Changes in version 0.2.2 (2013-11-29)}{ 126 | \itemize{ 127 | \item Switched to using the result from the compile-time configuration 128 | for R to determine big or little endian (as needed for the \code{NPy} 129 | headers) 130 | \item Added a new test (and test validation result file) for a complete 131 | save-reload cycle and comparison 132 | } 133 | } 134 | 135 | \section{Changes in version 0.2.1 (2013-11-28)}{ 136 | \itemize{ 137 | \item Synchronized code with the \code{cnpy} repository 138 | \item Added new function to test from R whether integers supported 139 | \item Updated tests for integer support, if available 140 | \item Updated vignette and discussion about need for rebuilding only 141 | \pkg{RcppCNPy} with the \code{-std=c++11} flag if integer support is 142 | desired 143 | \item Updated tests for integer support, if available 144 | \item Updated THANKS file 145 | \item Ensure that inclusion of \code{Rinternal.h} does not affect 146 | other headers by adding RF_NO_REMAP 147 | } 148 | } 149 | 150 | \section{Changes in version 0.2.0 (2012-07-30)}{ 151 | \itemize{ 152 | \item Support for writing of \code{gzip}-ed \code{npy} files has 153 | been added. 154 | \item A new option \code{dotranspose} has been added to 155 | \code{npyLoad()} to support data sets that do not need to be 156 | transposed to be used in R. 157 | \item A memory leak in reading files has been corrected. 158 | } 159 | } 160 | \section{Changes in version 0.1.0 (2012-07-07)}{ 161 | \itemize{ 162 | \item Added automatic use of transpose to automagically account for 163 | Fortran-vs-C major storage defaults between Python and R. 164 | \item Support for integer types in dependent on the \code{int64_t} 165 | type which is available only when the \code{-std=c++0x} switch is 166 | used at build-time (and CRAN still discourages use of it) 167 | \item Added support for reading gzip'ed files ending in ".npy.gz" 168 | \item Added regression tests in directory \code{tests/} 169 | \item Added a vignette describing the package 170 | \item Added a timing benchmark in demo/timings.R 171 | } 172 | } 173 | \section{Changes in version 0.0.1 (2012-07-04)}{ 174 | \itemize{ 175 | \item Initial version, as a straightforward Rcpp modules wrap around 176 | the \code{cpny} library by Carl Rogers (on github under a MIT 177 | license). 178 | \item At present, \code{npy} files can be read and written for 179 | vectors and matrices of either \code{numeric} or \code{integer} type. 180 | Note however that matrices are currently \emph{transposed} because 181 | of the default Fortran ordering done by numpy. 182 | } 183 | } 184 | -------------------------------------------------------------------------------- /vignettes/rmd/RcppCNPy-intro.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: | 3 | | \pkg{RcppCNPy}: 4 | | Reading and writing \pkg{NumPy} binary files 5 | 6 | author: 7 | - name: Dirk Eddelbuettel 8 | affiliation: a 9 | - name: Wush Wu 10 | affiliation: b 11 | address: 12 | - code: a 13 | address: http://dirk.eddelbuettel.com 14 | - code: b 15 | address: https://github.com/wush978 16 | 17 | lead_author_surname: Eddelbuettel and Wu 18 | 19 | doi: "https://cran.r-project.org/package=RcppCNPy" 20 | 21 | abstract: | 22 | This vignette introduces the \pkg{RcppCNPy} package for reading 23 | and writing files created by or for the \pkg{NumPy} module for 24 | \proglang{Python}. 25 | 26 | acknowledgements: | 27 | This paper can be cited as \citet{JOSS:RcppCNPy}; 28 | see the `citation("RcppCNPy")` command in R for details. 29 | 30 | footer_contents: "RcppCNPy Vignette" 31 | 32 | output: 33 | pinp::pinp: 34 | collapse: true 35 | 36 | header-includes: 37 | \newcommand{\proglang}[1]{\textsf{#1}} 38 | \newcommand{\pkg}[1]{\textbf{#1}} 39 | \newcommand{\R}{\proglang{R}\ } 40 | \newcommand{\Rns}{\proglang{R}} 41 | 42 | bibliography: RcppCNPy.bib 43 | 44 | vignette: > 45 | %\VignetteIndexEntry{RcppCNPy-intro} 46 | %\VignetteKeywords{Python, NumPy, R, data transfer} 47 | %\VignettePackage{RcppCNPy} 48 | %\VignetteEngine{knitr::rmarkdown} 49 | --- 50 | 51 | 52 | ```{r echo = FALSE, message = FALSE} 53 | knitr::opts_chunk$set(python.reticulate = TRUE) 54 | if (identical(Sys.info()[['sysname']], "Windows")) { 55 | knitr::opts_chunk$set(eval = FALSE) 56 | msg <- paste("Note: Some examples in this vignette require Python", 57 | "but you are running this vignette on Windows where Python", 58 | "is much less likely to be present, or even known to be", 59 | "missing (i.e. win-builder) so examples will not be evaluated.") 60 | msg <- paste(strwrap(msg), collapse="\n") 61 | message(msg) 62 | } 63 | ``` 64 | 65 | # Motivation 66 | 67 | \proglang{Python}\footnote{\url{http://www.python.org}} is a widely-used 68 | and popular programming language. It is deployed in use cases ranging from simple 69 | scripting to larger-scale application development. \proglang{Python} is also popular 70 | for quantitative and scientific application due to the existence of extension 71 | modules such as \pkg{NumPy}\footnote{\url{http://numpy.scipy.org/}} (which is 72 | shorthand for Numeric Python) and many other packages for data analysis. 73 | 74 | \pkg{NumPy} is used to efficiently represent $N$-dimensional arrays, 75 | and provides an efficient binary storage model for these files. In 76 | practice, $N$ is often equal to two, and matrices processed or 77 | generated in \proglang{Python} can be stored in this form. As 78 | \pkg{NumPy} is popular, many project utilize this file format. 79 | 80 | \R has no dedicated reading or writing functionality for these type of files. 81 | However, Carl Rogers has provided a small \proglang{Cpp} library called 82 | \pkg{cnpy}\footnote{\url{https://github.com/rogersce/cnpy}} which is released 83 | under the MIT license. Using the `Rcpp modules' feature in 84 | \pkg{Rcpp} \shortcites{CRAN:Rcpp} \citep{JSS:Rcpp,Eddelbuettel:2013:Rcpp,CRAN:Rcpp}, we provide 85 | (some) features of this library to \Rns. 86 | 87 | # Examples 88 | 89 | ## Data creation in \proglang{Python} 90 | 91 | The first code example simply creates two files in \proglang{Python}: a 92 | two-dimensional rectangular array as well as a vector. 93 | 94 | ```{python} 95 | import numpy as np 96 | 97 | mat = np.arange(12).reshape(3,4) * 1.1 98 | np.save("fmat.npy", mat) 99 | print(mat) 100 | 101 | vec = np.arange(5) * 1.1 102 | np.save("fvec.npy", vec) 103 | print(vec) 104 | ``` 105 | 106 | As illustrated, \proglang{Python} uses the \proglang{Fortran} convention for storing 107 | matrices and higher-dimensional arrays: a matrix constructed from a single 108 | sequence has its first consecutive elements in its first row---whereas \Rns, 109 | following the \proglang{C} convention, has these first few values in its first 110 | column. This shows that to go back and forth we need to transpose these 111 | matrices (which represented internally as two-dimensional arrays). 112 | 113 | ## Data reading in \R 114 | 115 | We can read the same data in \R using the \code{npyLoad()} function provided 116 | by the \pkg{RcppCNPy} package: 117 | 118 | ```{r rex1} 119 | library(RcppCNPy) 120 | mat <- npyLoad("fmat.npy") 121 | mat 122 | vec <- npyLoad("fvec.npy") 123 | vec 124 | ``` 125 | 126 | The \proglang{Fortran}-order of the matrix is preserved; we obtain the exact 127 | same data as we stored. 128 | 129 | ## Reading compressed data in \R 130 | 131 | A useful extension to the \pkg{cnpy} library is the support of 132 | \pkg{gzip}-compressed data. 133 | 134 | ```{r rex2, eval=FALSE} 135 | mat2 <- npyLoad("fmat.npy.gz") 136 | mat2 137 | ``` 138 | 139 | Support for writing compressed files has been added in version 0.2.0. 140 | 141 | ## Data writing in \R 142 | 143 | Matrices and vectors can be written to files using the \code{npySave()} 144 | function. 145 | 146 | ```{r rex3} 147 | set.seed(42) 148 | m <- matrix(sort(rnorm(6)), 3, 2) 149 | m 150 | npySave("randmat.npy", m) 151 | v <- seq(10, 12) 152 | v 153 | npySave("simplevec.npy", v) 154 | ``` 155 | 156 | ## Data reading in \proglang{Python} 157 | 158 | Reading the data back in \proglang{Python} is also straightforward as shown in 159 | the following example: 160 | 161 | ```{python pyex2} 162 | import numpy as np 163 | m = np.load("randmat.npy") 164 | print(m) 165 | v = np.load("simplevec.npy") 166 | print(v) 167 | ``` 168 | 169 | ## Integer support 170 | 171 | Support for integer data types has been conditional on use of either 172 | the \code{-std=c++0x} or the \code{-std=c++11} compiler extensions. 173 | Only these standards support the \code{long long int} type needed to 174 | represent \code{int64} data on a 32-bit OS. Following the release of 175 | \R 3.1.0, it has been enabled by default in \pkg{RcppCNPy} (whereas it 176 | previously required a manual rebuild), and following the release of R 177 | 3.3.0 with its updated Windows toolchain, C++11 is now available on 178 | all common R platforms. Consequently, support for large integers in 179 | \pkg{RcppCNPy} is no longer just a compile-time option for some 180 | platforms, but generally available on all (current) R installations. 181 | 182 | 183 | ## Performance 184 | 185 | The \R script \code{timing} in the \code{demo/} directory of the package 186 | \pkg{RcppCNPy} provides a simple benchmark. Given two values $n$ and $k$, a 187 | matrix of size $n \times k$ is created with $n$ rows and $k$ columns. It is 188 | written to temporary files in 189 | i) ascii format using \code{write.table()}; 190 | ii) \code{NumPy} format using \code{npySave()}; and 191 | iii) \code{NumPy} format using \code{npySave()} with compression via 192 | the \code{zlib} library (used also by \code{gzip}). 193 | 194 | Table \ref{tab:benchmark} shows some timing comparisons for a matrix with 195 | five million elements. Reading the \code{npy} data is clearly fastest as it 196 | required only parsing of the header, followed by a single large binary read 197 | (and the transpose required to translate the representation used by \Rns). The 198 | compressed file requires only one-fourth of the disk space, but takes 199 | approximately 2.5 times as long to read as the binary stream has be 200 | transformed. Lastly, the default ascii reading mode is clearly by far the 201 | slowest. 202 | 203 | \begin{table}[bt] 204 | \begin{center} 205 | \begin{small} 206 | \begin{tabular}{rrr} 207 | \toprule 208 | {\bf Access method \phantom{X}} & {\bf Time in sec.} & {\bf Relative to best} \\ 209 | \cmidrule(r){1-3} 210 | \code{npyLoad(pyfile)} & 0.074 & 1.000 \\ 211 | \code{npyLoad(pygzfile)} & 0.190 & 2.568 \\ 212 | \code{read.table(}txtfile) & 4.189 & 56.608 \\ 213 | \bottomrule 214 | \end{tabular} 215 | \end{small} 216 | \caption{Performance comparison of data reads using a matrix of size 217 | $10^5 \times 50$. File size are 39.7mb for ascii, 40.0mb for npy and 218 | 10.8mb for npy.gz. Ten replications were performed, and total times 219 | are shown. R 3.3.1 was used on a laptop with an SSD disk. } 220 | \label{tab:benchmark} 221 | \end{center} 222 | \end{table} 223 | 224 | 225 | # Limitations 226 | 227 | ## Higher-dimensional arrays 228 | 229 | \pkg{Rcpp} supports three-dimensional arrays, this could be support in 230 | \pkg{RcppCNPy} as well. 231 | 232 | ## \code{npz} files 233 | 234 | The \pkg{cnpy} library supports reading and writing of sets of arrays; this 235 | feature could also be exported. 236 | 237 | # Summary 238 | 239 | The \pkg{RcppCNPy} package provides simple reading and writing of 240 | \pkg{NumPy} files, using the \pkg{cnpy} library. Reading of compressed 241 | files is also supported as an extension, offering more compact storage 242 | at the cost of slightly longer read times. 243 | 244 | ```{r rex4, echo=FALSE} 245 | unlink("fmat.npy") 246 | unlink("fvec.npy") 247 | unlink("randmat.npy") 248 | unlink("simplevec.npy") 249 | ``` 250 | 251 | -------------------------------------------------------------------------------- /src/cnpyMod.cpp: -------------------------------------------------------------------------------- 1 | // -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*- 2 | // 3 | // cnpyMod.cpp: Rcpp R/C++ modules interface to cnpy 4 | // 5 | // Copyright (C) 2012 - 2014 Dirk Eddelbuettel 6 | // 7 | // This file is part of RcppCNPy. 8 | // 9 | // RcppCNPy is free software: you can redistribute it and/or modify it 10 | // under the terms of the GNU General Public License as published by 11 | // the Free Software Foundation, either version 2 of the License, or 12 | // (at your option) any later version. 13 | // 14 | // RcppCNPy is distributed in the hope that it will be useful, but 15 | // WITHOUT ANY WARRANTY; without even the implied warranty of 16 | // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | // GNU General Public License for more details. 18 | // 19 | // You should have received a copy of the GNU General Public License 20 | // along with RcppCNPy. If not, see . 21 | 22 | #include // need to include the main Rcpp header file only 23 | #include "cnpy.h" // (local copy of) header for cnpy library 24 | 25 | template 26 | T transpose(const T & m) { // tranpose for IntegerMatrix / NumericMatrix, see array.c in R 27 | int k = m.rows(), n = m.cols(); 28 | //Rcpp::Rcout << "Transposing " << n << " by " << k << std::endl; 29 | T z(n, k); 30 | int sz1 = n*k-1; 31 | typename T::const_iterator mit ; 32 | typename T::iterator zit; 33 | for (mit = m.begin(), zit = z.begin(); mit != m.end(); mit++, zit += n) { 34 | if (zit >= z.end()) zit -= sz1; 35 | *zit = *mit; 36 | } 37 | return(z); 38 | } 39 | 40 | // cf stackoverflow.com/questions/874134 41 | bool hasEnding(std::string const &full, std::string const &ending) { 42 | if (full.length() >= ending.length()) { 43 | return(0 == full.compare(full.length() - ending.length(), ending.length(), ending)); 44 | } else { 45 | return false; 46 | } 47 | } 48 | 49 | Rcpp::RObject npyLoad(const std::string & filename, const std::string & type, const bool dotranspose) { 50 | 51 | cnpy::NpyArray arr; 52 | 53 | if (hasEnding(filename, ".gz")) { 54 | arr = cnpy::npy_gzload(filename); 55 | } else { 56 | arr = cnpy::npy_load(filename); 57 | } 58 | 59 | std::vector shape = arr.shape; 60 | SEXP ret = R_NilValue; // allows us to assign either int or numeric 61 | if (shape.size() == 1) { 62 | if (type == "numeric") { 63 | double *p = reinterpret_cast(arr.data); 64 | #ifdef WORDS_BIGENDIAN 65 | std::transform(p, p + shape[0], p, swap_endian); 66 | #endif 67 | ret = Rcpp::NumericVector(p, p + shape[0]); 68 | } else if (type == "integer") { 69 | int64_t *p = reinterpret_cast(arr.data); 70 | #ifdef WORDS_BIGENDIAN 71 | std::transform(p, p + shape[0], p, swap_endian); 72 | #endif 73 | ret = Rcpp::IntegerVector(p, p + shape[0]); 74 | } else { 75 | arr.destruct(); 76 | Rf_error("Unsupported type in npyLoad"); 77 | } 78 | } else if (shape.size() == 2) { 79 | if (type == "numeric") { 80 | double *p = reinterpret_cast(arr.data); 81 | #ifdef WORDS_BIGENDIAN 82 | std::transform(p, p + shape[0] * shape[1], p, swap_endian); 83 | #endif 84 | // invert dimension for creation, and then tranpose to correct Fortran-vs-C storage 85 | if (dotranspose) { 86 | ret = transpose(Rcpp::NumericMatrix(shape[1], shape[0], p)); 87 | } else { 88 | ret = Rcpp::NumericMatrix(shape[0], shape[1], p); 89 | } 90 | } else if (type == "integer") { 91 | int64_t *p = reinterpret_cast(arr.data); 92 | #ifdef WORDS_BIGENDIAN 93 | std::transform(p, p + shape[0] * shape[1], p, swap_endian); 94 | #endif 95 | // invert dimension for creation, and then tranpose to correct Fortran-vs-C storage 96 | if (dotranspose) { 97 | ret = transpose(Rcpp::IntegerMatrix(shape[1], shape[0], p)); 98 | } else { 99 | ret = Rcpp::IntegerMatrix(shape[0], shape[1], p); 100 | } 101 | } else { 102 | arr.destruct(); 103 | Rf_error("Unsupported type in npyLoad"); 104 | } 105 | } else { 106 | arr.destruct(); 107 | Rf_error("Unsupported dimension in npyLoad"); 108 | } 109 | arr.destruct(); 110 | return ret; 111 | } 112 | 113 | void npySave(std::string filename, Rcpp::RObject x, std::string mode, bool checkPath) { 114 | if (checkPath) { 115 | Rcpp::Environment ns = Rcpp::Environment::namespace_env("RcppCNPy"); 116 | Rcpp::Function checkPath = ns[".checkPath"]; 117 | bool res = Rcpp::as(checkPath(filename)); 118 | if (!res) Rcpp::stop("Filename contains non-existing directory."); 119 | } 120 | if (::Rf_isMatrix(x)) { 121 | if (::Rf_isInteger(x)) { 122 | Rcpp::IntegerMatrix mat = transpose(Rcpp::IntegerMatrix(x)); 123 | std::vector vec(mat.ncol()*mat.nrow()); 124 | std::copy(mat.begin(), mat.end(), vec.begin()); 125 | #ifdef WORDS_BIGENDIAN 126 | std::transform(vec.begin(), vec.end(), vec.begin(), swap_endian); 127 | #endif 128 | std::vector shape = 129 | Rcpp::as >(Rcpp::IntegerVector::create(mat.ncol(), mat.nrow())); 130 | if (hasEnding(filename, ".gz")) { 131 | cnpy::npy_gzsave(filename, &(vec[0]), &(shape[0]), 2); // no mode, overwrite only 132 | } else { 133 | cnpy::npy_save(filename, &(vec[0]), &(shape[0]), 2, mode); 134 | } 135 | } else if (::Rf_isNumeric(x)) { 136 | Rcpp::NumericMatrix mat = transpose(Rcpp::NumericMatrix(x)); 137 | #ifdef WORDS_BIGENDIAN 138 | std::transform(mat.begin(), mat.end(), mat.begin(), swap_endian); 139 | #endif 140 | std::vector shape = 141 | Rcpp::as >(Rcpp::IntegerVector::create(mat.ncol(), mat.nrow())); 142 | 143 | if (hasEnding(filename, ".gz")) { 144 | cnpy::npy_gzsave(filename, mat.begin(), &(shape[0]), 2); // no mode, overwrite only 145 | } else { 146 | cnpy::npy_save(filename, mat.begin(), &(shape[0]), 2, mode); 147 | } 148 | } else { 149 | Rf_error("Unsupported matrix type\n"); 150 | } 151 | } else if (::Rf_isVector(x)) { 152 | if (::Rf_isInteger(x)) { 153 | Rcpp::IntegerVector vec(x); 154 | std::vector v(vec.size()); 155 | std::copy(vec.begin(), vec.end(), v.begin()); 156 | #ifdef WORDS_BIGENDIAN 157 | std::transform(v.begin(), v.end(), v.begin(), swap_endian); 158 | #endif 159 | std::vector shape = 160 | Rcpp::as >(Rcpp::IntegerVector::create(vec.length())); 161 | if (hasEnding(filename, ".gz")) { 162 | cnpy::npy_gzsave(filename, &(v[0]), &(shape[0]), 1); // no mode, append only 163 | } else { 164 | cnpy::npy_save(filename, &(v[0]), &(shape[0]), 1, mode); 165 | } 166 | } else if (::Rf_isNumeric(x)) { 167 | Rcpp::Rcout << "Saving Numeric Vector\n"; 168 | #ifdef WORDS_BIGENDIAN 169 | Rcpp::NumericVector vec(Rcpp::clone(x)); // ensures a deep copy 170 | std::transform(vec.begin(), vec.end(), vec.begin(), swap_endian); 171 | #else 172 | Rcpp::NumericVector vec(x); 173 | #endif 174 | std::vector shape = 175 | Rcpp::as >(Rcpp::IntegerVector::create(vec.length())); 176 | if (hasEnding(filename, ".gz")) { 177 | cnpy::npy_gzsave(filename, vec.begin(), &(shape[0]), 1); // no mode, append only 178 | } else { 179 | cnpy::npy_save(filename, vec.begin(), &(shape[0]), 1, mode); 180 | } 181 | } else { 182 | Rf_error("Unsupported vector type\n"); 183 | } 184 | } else { 185 | Rf_error("Unsupported type\n"); 186 | } 187 | } 188 | 189 | bool npyHasIntegerSupport() { 190 | return true; // no longer conditional 191 | } 192 | 193 | RCPP_MODULE(cnpy){ 194 | 195 | using namespace Rcpp; 196 | 197 | function("npyLoad", // name of the identifier at the R level 198 | &npyLoad, // function pointer to helper function defined above 199 | List::create( Named("filename"), // function arguments including default value 200 | Named("type") = "numeric", 201 | Named("dotranspose") = true), 202 | "read an npy file into a numeric or integer vector or matrix"); 203 | 204 | function("npySave", // name of the identifier at the R level 205 | &npySave, // function pointer to helper function defined above 206 | List::create( Named("filename"), // function arguments including default value 207 | Named("object"), 208 | Named("mode") = "w", 209 | Named("checkPath") = true), 210 | "save an R object (vector or matrix of type integer or numeric) to an npy file"); 211 | 212 | function("npyHasIntegerSupport", &npyHasIntegerSupport, 213 | "return logical value indicating whether package has integer support (which need C++11)"); 214 | } 215 | -------------------------------------------------------------------------------- /src/cnpy.cpp: -------------------------------------------------------------------------------- 1 | // -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*- 2 | 3 | //Copyright (C) 2011 Carl Rogers 4 | //Released under MIT License 5 | //license available in LICENSE file, or at http://www.opensource.org/licenses/mit-license.php 6 | 7 | // Changes for RcppCNPy are 8 | // Copyright (C) 2012 - 2016 Dirk Eddelbuettel 9 | // and licensed under GNU GPL (>= 2) 10 | 11 | #include"cnpy.h" 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | 18 | #include // use R configuration test result for endianness 19 | 20 | char cnpy::map_type(const std::type_info& t) 21 | { 22 | if(t == typeid(float) ) return 'f'; 23 | if(t == typeid(double) ) return 'f'; 24 | if(t == typeid(long double) ) return 'f'; 25 | 26 | if(t == typeid(int) ) return 'i'; 27 | if(t == typeid(char) ) return 'i'; 28 | if(t == typeid(short) ) return 'i'; 29 | if(t == typeid(long) ) return 'i'; 30 | if(t == typeid(long long) ) return 'i'; 31 | 32 | if(t == typeid(unsigned char) ) return 'u'; 33 | if(t == typeid(unsigned short) ) return 'u'; 34 | if(t == typeid(unsigned long) ) return 'u'; 35 | if(t == typeid(unsigned long long) ) return 'u'; 36 | if(t == typeid(unsigned int) ) return 'u'; 37 | 38 | if(t == typeid(bool) ) return 'b'; 39 | 40 | if(t == typeid(std::complex) ) return 'c'; 41 | if(t == typeid(std::complex) ) return 'c'; 42 | if(t == typeid(std::complex) ) return 'c'; 43 | 44 | else return '?'; 45 | } 46 | 47 | template<> std::vector& cnpy::operator+=(std::vector& lhs, const std::string rhs) { 48 | lhs.insert(lhs.end(),rhs.begin(),rhs.end()); 49 | return lhs; 50 | } 51 | 52 | template<> std::vector& cnpy::operator+=(std::vector& lhs, const char* rhs) { 53 | //write in little endian 54 | size_t len = strlen(rhs); 55 | lhs.reserve(len); 56 | for(size_t byte = 0; byte < len; byte++) { 57 | lhs.push_back(rhs[byte]); 58 | } 59 | return lhs; 60 | } 61 | 62 | void cnpy::parse_npy_header(FILE* fp, unsigned int& word_size, unsigned int*& shape, unsigned int& ndims, bool& fortran_order) { 63 | char buffer[256]; 64 | size_t res = fread(buffer,sizeof(char),11,fp); 65 | if (res != 11) 66 | Rf_error("cnpy::parse_npy_header read discprepancy"); 67 | std::string header = fgets(buffer,256,fp); 68 | Rassert(header[header.size()-1] == '\n', "header ended improperly"); 69 | 70 | int loc1, loc2; 71 | 72 | //fortran order 73 | loc1 = header.find("fortran_order")+16; 74 | fortran_order = (header.substr(loc1,5) == "True" ? true : false); 75 | 76 | //shape 77 | loc1 = header.find("("); 78 | loc2 = header.find(")"); 79 | std::string str_shape = header.substr(loc1+1,loc2-loc1-1); 80 | if(str_shape[str_shape.size()-1] == ',') ndims = 1; 81 | else ndims = std::count(str_shape.begin(),str_shape.end(),',')+1; 82 | shape = new unsigned int[ndims]; 83 | for(unsigned int i = 0;i < ndims;i++) { 84 | loc1 = str_shape.find(","); 85 | shape[i] = atoi(str_shape.substr(0,loc1).c_str()); 86 | str_shape = str_shape.substr(loc1+1); 87 | } 88 | 89 | //endian, word size, data type 90 | //byte order code | stands for not applicable. 91 | //not sure when this applies except for byte array 92 | loc1 = header.find("descr")+9; 93 | bool littleEndian = (header[loc1] == '<' || header[loc1] == '|' ? true : false); 94 | Rassert(littleEndian, "littleEndian error"); 95 | 96 | //char type = header[loc1+1]; 97 | //assert(type == map_type(T)); 98 | 99 | std::string str_ws = header.substr(loc1+2); 100 | loc2 = str_ws.find("'"); 101 | word_size = atoi(str_ws.substr(0,loc2).c_str()); 102 | } 103 | 104 | void cnpy::parse_zip_footer(FILE* fp, unsigned short& nrecs, unsigned int& global_header_size, unsigned int& global_header_offset) 105 | { 106 | std::vector footer(22); 107 | fseek(fp,-22,SEEK_END); 108 | size_t res = fread(&footer[0],sizeof(char),22,fp); 109 | if (res != 22) 110 | Rf_error("cnpy::parse_zip_footer read discprepancy"); 111 | 112 | unsigned short disk_no, disk_start, nrecs_on_disk, comment_len; 113 | disk_no = *(unsigned short*) &footer[4]; 114 | disk_start = *(unsigned short*) &footer[6]; 115 | nrecs_on_disk = *(unsigned short*) &footer[8]; 116 | nrecs = *(unsigned short*) &footer[10]; 117 | global_header_size = *(unsigned int*) &footer[12]; 118 | global_header_offset = *(unsigned int*) &footer[16]; 119 | comment_len = *(unsigned short*) &footer[20]; 120 | 121 | Rassert(disk_no == 0, "disk_no is != 0"); 122 | Rassert(disk_start == 0, "disk_start != 0"); 123 | Rassert(nrecs_on_disk == nrecs, "nrecs_on_disk != nrecs"); 124 | Rassert(comment_len == 0, "comment_len != 0"); 125 | } 126 | 127 | cnpy::NpyArray load_the_npy_file(FILE* fp) { 128 | unsigned int* shape; 129 | unsigned int ndims, word_size; 130 | bool fortran_order; 131 | cnpy::parse_npy_header(fp,word_size,shape,ndims,fortran_order); 132 | unsigned long long size = 1; //long long so no overflow when multiplying by word_size 133 | for(unsigned int i = 0;i < ndims;i++) size *= shape[i]; 134 | 135 | cnpy::NpyArray arr; 136 | arr.word_size = word_size; 137 | arr.shape = std::vector(shape,shape+ndims); 138 | delete[] shape; 139 | arr.data = new char[size*word_size]; 140 | arr.fortran_order = fortran_order; 141 | size_t nread = fread(arr.data,word_size,size,fp); 142 | if (nread != size) 143 | Rf_error("cnpy::load_the_npy_file read size discrepancy"); 144 | return arr; 145 | } 146 | 147 | cnpy::NpyArray gzload_the_npy_file(gzFile fp) { 148 | unsigned int* shape; 149 | unsigned int ndims, word_size; 150 | bool fortran_order; 151 | cnpy::parse_npy_gzheader(fp,word_size,shape,ndims,fortran_order); 152 | unsigned long long size = 1; //long long so no overflow when multiplying by word_size 153 | for(unsigned int i = 0;i < ndims;i++) size *= shape[i]; 154 | 155 | cnpy::NpyArray arr; 156 | arr.word_size = word_size; 157 | arr.shape = std::vector(shape,shape+ndims); 158 | delete[] shape; 159 | arr.data = new char[size*word_size]; 160 | arr.fortran_order = fortran_order; 161 | size_t nread = gzread(fp,arr.data,word_size*size); 162 | if (nread != size*word_size) 163 | Rf_error("cnpy::gzload_the_npy_file read size discrepancy"); 164 | return arr; 165 | } 166 | 167 | cnpy::npz_t cnpy::npz_load(std::string fname) { 168 | FILE* fp = fopen(fname.c_str(),"rb"); 169 | 170 | if(!fp) Rf_error("npz_load: Error! Unable to open file %s!\n",fname.c_str()); 171 | Rassert(fp, "fp error"); 172 | 173 | cnpy::npz_t arrays; 174 | 175 | while(1) { 176 | std::vector local_header(30); 177 | size_t headerres = fread(&local_header[0],sizeof(char),30,fp); 178 | if (headerres != 30) 179 | Rf_error("cnpy::npz_load read discprepancy on header"); 180 | 181 | //if we've reached the global header, stop reading 182 | if(local_header[2] != 0x03 || local_header[3] != 0x04) break; 183 | 184 | //read in the variable name 185 | unsigned short name_len = *(unsigned short*) &local_header[26]; 186 | std::string varname(name_len,' '); 187 | size_t vname_res = fread(&varname[0],sizeof(char),name_len,fp); 188 | if (vname_res != name_len) 189 | Rf_error("cnpy::npz_load read discprepancy on name_len"); 190 | 191 | //erase the lagging .npy 192 | varname.erase(varname.end()-4,varname.end()); 193 | 194 | //read in the extra field 195 | unsigned short extra_field_len = *(unsigned short*) &local_header[28]; 196 | if(extra_field_len > 0) { 197 | std::vector buff(extra_field_len); 198 | size_t efield_res = fread(&buff[0],sizeof(char),extra_field_len,fp); 199 | if (efield_res != extra_field_len) 200 | Rf_error("cnpy::npz_load read discprepancy on extra_field_len"); 201 | } 202 | 203 | arrays[varname] = load_the_npy_file(fp); 204 | } 205 | 206 | fclose(fp); 207 | return arrays; 208 | } 209 | 210 | cnpy::NpyArray cnpy::npz_load(std::string fname, std::string varname) { 211 | FILE* fp = fopen(fname.c_str(),"rb"); 212 | 213 | if(!fp) { 214 | Rf_error("npz_load: Error! Unable to open file %s!\n",fname.c_str()); 215 | } 216 | 217 | while(1) { 218 | std::vector local_header(30); 219 | size_t header_res = fread(&local_header[0],sizeof(char),30,fp); 220 | if (header_res != 30) 221 | Rf_error("cnpy::npz_load read discprepancy on header"); 222 | 223 | //if we've reached the global header, stop reading 224 | if(local_header[2] != 0x03 || local_header[3] != 0x04) break; 225 | 226 | //read in the variable name 227 | unsigned short name_len = *(unsigned short*) &local_header[26]; 228 | std::string vname(name_len,' '); 229 | size_t vname_res = fread(&vname[0],sizeof(char),name_len,fp); 230 | if (vname_res != name_len) 231 | Rf_error("cnpy::npz_load read discprepancy on name_len"); 232 | vname.erase(vname.end()-4,vname.end()); //erase the lagging .npy 233 | 234 | //read in the extra field 235 | unsigned short extra_field_len = *(unsigned short*) &local_header[28]; 236 | fseek(fp,extra_field_len,SEEK_CUR); //skip past the extra field 237 | 238 | if(vname == varname) { 239 | NpyArray array = load_the_npy_file(fp); 240 | fclose(fp); 241 | return array; 242 | } 243 | else { 244 | //skip past the data 245 | unsigned int size = *(unsigned int*) &local_header[22]; 246 | fseek(fp,size,SEEK_CUR); 247 | } 248 | } 249 | 250 | fclose(fp); 251 | Rf_error("npz_load: Error! Variable name %s not found in %s!\n",varname.c_str(),fname.c_str()); 252 | // never reached -- not satisfying -Wall -pedantic 253 | cnpy::NpyArray unused; 254 | unused.word_size = 0; 255 | unused.data = NULL; 256 | return unused; 257 | } 258 | 259 | cnpy::NpyArray cnpy::npy_load(std::string fname) { 260 | 261 | FILE* fp = fopen(fname.c_str(), "rb"); 262 | 263 | if(!fp) { 264 | Rf_error("npy_load: Error! Unable to open file %s!\n",fname.c_str()); 265 | } 266 | 267 | NpyArray arr = load_the_npy_file(fp); 268 | 269 | fclose(fp); 270 | return arr; 271 | } 272 | 273 | cnpy::NpyArray cnpy::npy_gzload(std::string fname) { 274 | gzFile fp = gzopen(fname.c_str(), "rb"); 275 | if(!fp) { 276 | Rf_error("npy_gzload: Error! Unable to open file %s!\n",fname.c_str()); 277 | } 278 | NpyArray arr = gzload_the_npy_file(fp); 279 | gzclose(fp); 280 | return arr; 281 | } 282 | 283 | void cnpy::parse_npy_gzheader(gzFile fp, unsigned int& word_size, unsigned int*& shape, unsigned int& ndims, bool& fortran_order) { 284 | char buffer[256]; 285 | size_t res = gzread(fp,buffer,sizeof(char)*11); 286 | if (res != 11) 287 | Rf_error("cnpy::parse_npy_gzheader read discprepancy"); 288 | std::string header = gzgets(fp, buffer,256); 289 | Rassert(header[header.size()-1] == '\n', "header ended improperly"); 290 | 291 | int loc1, loc2; 292 | 293 | //fortran order 294 | loc1 = header.find("fortran_order")+16; 295 | fortran_order = (header.substr(loc1,5) == "True" ? true : false); 296 | 297 | //shape 298 | loc1 = header.find("("); 299 | loc2 = header.find(")"); 300 | std::string str_shape = header.substr(loc1+1,loc2-loc1-1); 301 | if(str_shape[str_shape.size()-1] == ',') ndims = 1; 302 | else ndims = std::count(str_shape.begin(),str_shape.end(),',')+1; 303 | shape = new unsigned int[ndims]; 304 | for(unsigned int i = 0;i < ndims;i++) { 305 | loc1 = str_shape.find(","); 306 | shape[i] = atoi(str_shape.substr(0,loc1).c_str()); 307 | str_shape = str_shape.substr(loc1+1); 308 | } 309 | 310 | //endian, word size, data type 311 | loc1 = header.find("descr")+9; 312 | bool littleEndian = (header[loc1] == '<' ? true : false); 313 | Rassert(littleEndian, "littleEndian error"); 314 | 315 | //char type = header[loc1+1]; 316 | //assert(type == map_type(T); 317 | 318 | std::string str_ws = header.substr(loc1+2); 319 | loc2 = str_ws.find("'"); 320 | word_size = atoi(str_ws.substr(0,loc2).c_str()); 321 | } 322 | -------------------------------------------------------------------------------- /src/cnpy.h: -------------------------------------------------------------------------------- 1 | // -*- mode: C++; c-indent-level: 4; c-basic-offset: 4; indent-tabs-mode: nil; -*- 2 | 3 | //Copyright (C) 2011 Carl Rogers 4 | //Released under MIT License 5 | //license available in LICENSE file, or at http://www.opensource.org/licenses/mit-license.php 6 | 7 | // Changes for RcppCNPy are 8 | // Copyright (C) 2012 - 2013 Dirk Eddelbuettel 9 | // and licensed under GNU GPL (>= 2) 10 | 11 | #ifndef LIBCNPY_H_ 12 | #define LIBCNPY_H_ 13 | 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | 25 | #include // for std::int64_t, needs c++11 switch 26 | 27 | #define R_NO_REMAP 28 | #include // for Rf_error 29 | 30 | // cf http://stackoverflow.com/a/4956493/143305 31 | template 32 | T swap_endian(T u) { 33 | union { 34 | T u; 35 | unsigned char u8[sizeof(T)]; 36 | } source, dest; 37 | 38 | source.u = u; 39 | 40 | for (size_t k = 0; k < sizeof(T); k++) 41 | dest.u8[k] = source.u8[sizeof(T) - k - 1]; 42 | 43 | return dest.u; 44 | } 45 | 46 | namespace cnpy { 47 | 48 | inline void Rassert(bool val, std::string txt) { 49 | if ( ! val) Rf_error("%s", txt.c_str()); 50 | } 51 | 52 | struct NpyArray { 53 | char* data; 54 | std::vector shape; 55 | unsigned int word_size; 56 | bool fortran_order; 57 | void destruct() {delete[] data;} 58 | }; 59 | 60 | struct npz_t : public std::map 61 | { 62 | void destruct() 63 | { 64 | npz_t::iterator it = this->begin(); 65 | for(; it != this->end(); ++it) (*it).second.destruct(); 66 | } 67 | }; 68 | 69 | char map_type(const std::type_info& t); 70 | template std::vector create_npy_header(const T* data, const unsigned int* shape, const unsigned int ndims); 71 | void parse_npy_header(FILE* fp,unsigned int& word_size, unsigned int*& shape, unsigned int& ndims, bool& fortran_order); 72 | void parse_zip_footer(FILE* fp, unsigned short& nrecs, unsigned int& global_header_size, unsigned int& global_header_offset); 73 | npz_t npz_load(std::string fname); 74 | NpyArray npz_load(std::string fname, std::string varname); 75 | NpyArray npy_load(std::string fname); 76 | NpyArray npy_gzload(std::string fname); 77 | void parse_npy_gzheader(gzFile fp,unsigned int& word_size, unsigned int*& shape, unsigned int& ndims, bool& fortran_order); 78 | 79 | template std::vector& operator+=(std::vector& lhs, const T rhs) { 80 | //write in little endian 81 | for(unsigned char byte = 0; byte < sizeof(T); byte++) { 82 | char val = *((char*)&rhs+byte); 83 | lhs.push_back(val); 84 | } 85 | return lhs; 86 | } 87 | 88 | template<> std::vector& operator+=(std::vector& lhs, const std::string rhs); 89 | template<> std::vector& operator+=(std::vector& lhs, const char* rhs); 90 | 91 | 92 | template std::string tostring(T i, int pad = 0, char padval = ' ') { 93 | std::stringstream s; 94 | s << i; 95 | return s.str(); 96 | } 97 | 98 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w") { 99 | FILE* fp = NULL; 100 | 101 | if(mode == "a") fp = fopen(fname.c_str(),"r+b"); 102 | 103 | if(fp) { 104 | //file exists. we need to append to it. read the header, modify the array size 105 | unsigned int word_size, tmp_dims; 106 | unsigned int* tmp_shape = 0; 107 | bool fortran_order; 108 | parse_npy_header(fp,word_size,tmp_shape,tmp_dims,fortran_order); 109 | Rassert(!fortran_order, "Data in Fortran order"); 110 | 111 | if(word_size != sizeof(T)) { 112 | Rf_error("cnpy error: %s has word size %u but npy_save appending data sized %u\n", fname.c_str(), word_size, static_cast(sizeof(T))); 113 | //assert( word_size == sizeof(T) ); 114 | } 115 | if(tmp_dims != ndims) { 116 | Rf_error("cnpy error: npy_save attempting to append misdimensioned data to %s\n", fname.c_str()); 117 | //assert(tmp_dims == ndims); 118 | } 119 | 120 | for(unsigned int i = 1; i < ndims; i++) { 121 | if(shape[i] != tmp_shape[i]) { 122 | Rf_error("cnpy error: npy_save attempting to append misshaped data to %s\n", fname.c_str()); 123 | //assert(shape[i] == tmp_shape[i]); 124 | } 125 | } 126 | tmp_shape[0] += shape[0]; 127 | 128 | fseek(fp,0,SEEK_SET); 129 | std::vector header = create_npy_header(data,tmp_shape,ndims); 130 | fwrite(&header[0],sizeof(char),header.size(),fp); 131 | fseek(fp,0,SEEK_END); 132 | 133 | delete[] tmp_shape; 134 | } 135 | else { 136 | fp = fopen(fname.c_str(),"wb"); 137 | std::vector header = create_npy_header(data,shape,ndims); 138 | fwrite(&header[0],sizeof(char),header.size(),fp); 139 | } 140 | 141 | unsigned int nels = 1; 142 | for(unsigned int i = 0;i < ndims;i++) nels *= shape[i]; 143 | 144 | fwrite(data,sizeof(T),nels,fp); 145 | fclose(fp); 146 | } 147 | 148 | template void npy_gzsave(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims) { 149 | gzFile fp = gzopen(fname.c_str(),"wb"); 150 | if(!fp) { 151 | Rf_error("npy_gzsave: Error! Unable to open file %s!\n",fname.c_str()); 152 | } 153 | std::vector header = create_npy_header(data,shape,ndims); 154 | gzwrite(fp, &header[0], sizeof(char) * header.size()); 155 | 156 | unsigned int nels = 1; 157 | for (unsigned int i = 0;i < ndims;i++) nels *= shape[i]; 158 | 159 | gzwrite(fp, data, sizeof(T)*nels); 160 | gzclose(fp); 161 | } 162 | 163 | template void npz_save(std::string zipname, std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w") 164 | { 165 | //first, append a .npy to the fname 166 | fname += ".npy"; 167 | 168 | //now, on with the show 169 | FILE* fp = NULL; 170 | unsigned short nrecs = 0; 171 | unsigned int global_header_offset = 0; 172 | std::vector global_header; 173 | 174 | if(mode == "a") fp = fopen(zipname.c_str(),"r+b"); 175 | 176 | if(fp) { 177 | //zip file exists. we need to add a new npy file to it. 178 | //first read the footer. this gives us the offset and size of the global header 179 | //then read and store the global header. 180 | //below, we will write the the new data at the start of the global header then append the global header and footer below it 181 | unsigned int global_header_size; 182 | parse_zip_footer(fp,nrecs,global_header_size,global_header_offset); 183 | fseek(fp,global_header_offset,SEEK_SET); 184 | global_header.resize(global_header_size); 185 | size_t res = fread(&global_header[0],sizeof(char),global_header_size,fp); 186 | if(res != global_header_size){ 187 | throw std::runtime_error("npz_save: header read error while adding to existing zip"); 188 | } 189 | fseek(fp,global_header_offset,SEEK_SET); 190 | } 191 | else { 192 | fp = fopen(zipname.c_str(),"wb"); 193 | } 194 | 195 | std::vector npy_header = create_npy_header(data,shape,ndims); 196 | 197 | unsigned long nels = 1; 198 | for (unsigned int m=0; m local_header; 207 | local_header += "PK"; //first part of sig 208 | local_header += (unsigned short) 0x0403; //second part of sig 209 | local_header += (unsigned short) 20; //min version to extract 210 | local_header += (unsigned short) 0; //general purpose bit flag 211 | local_header += (unsigned short) 0; //compression method 212 | local_header += (unsigned short) 0; //file last mod time 213 | local_header += (unsigned short) 0; //file last mod date 214 | local_header += (unsigned int) crc; //crc 215 | local_header += (unsigned int) nbytes; //compressed size 216 | local_header += (unsigned int) nbytes; //uncompressed size 217 | local_header += (unsigned short) fname.size(); //fname length 218 | local_header += (unsigned short) 0; //extra field length 219 | local_header += fname; 220 | 221 | //build global header 222 | global_header += "PK"; //first part of sig 223 | global_header += (unsigned short) 0x0201; //second part of sig 224 | global_header += (unsigned short) 20; //version made by 225 | global_header.insert(global_header.end(),local_header.begin()+4,local_header.begin()+30); 226 | global_header += (unsigned short) 0; //file comment length 227 | global_header += (unsigned short) 0; //disk number where file starts 228 | global_header += (unsigned short) 0; //internal file attributes 229 | global_header += (unsigned int) 0; //external file attributes 230 | global_header += (unsigned int) global_header_offset; //relative offset of local file header, since it begins where the global header used to begin 231 | global_header += fname; 232 | 233 | //build footer 234 | std::vector footer; 235 | footer += "PK"; //first part of sig 236 | footer += (unsigned short) 0x0605; //second part of sig 237 | footer += (unsigned short) 0; //number of this disk 238 | footer += (unsigned short) 0; //disk where footer starts 239 | footer += (unsigned short) (nrecs+1); //number of records on this disk 240 | footer += (unsigned short) (nrecs+1); //total number of records 241 | footer += (unsigned int) global_header.size(); //nbytes of global headers 242 | footer += (unsigned int) (global_header_offset + nbytes + local_header.size()); //offset of start of global headers, since global header now starts after newly written array 243 | footer += (unsigned short) 0; //zip file comment length 244 | 245 | //write everything 246 | fwrite(&local_header[0],sizeof(char),local_header.size(),fp); 247 | fwrite(&npy_header[0],sizeof(char),npy_header.size(),fp); 248 | fwrite(data,sizeof(T),nels,fp); 249 | fwrite(&global_header[0],sizeof(char),global_header.size(),fp); 250 | fwrite(&footer[0],sizeof(char),footer.size(),fp); 251 | fclose(fp); 252 | } 253 | 254 | template std::vector create_npy_header(const T* data, const unsigned int* shape, const unsigned int ndims) { 255 | 256 | std::vector dict; 257 | dict += "{'descr': '"; 258 | dict += '<'; // always write little endian 259 | dict += map_type(typeid(T)); 260 | dict += tostring(sizeof(T)); 261 | dict += "', 'fortran_order': False, 'shape': ("; 262 | dict += tostring(shape[0]); 263 | for(unsigned int i = 1;i < ndims;i++) { 264 | dict += ", "; 265 | dict += tostring(shape[i]); 266 | } 267 | if(ndims == 1) dict += ","; 268 | dict += "), }"; 269 | //pad with spaces so that preamble+dict is modulo 16 bytes. preamble is 10 bytes. dict needs to end with \n 270 | int remainder = 16 - (10 + dict.size()) % 16; 271 | dict.insert(dict.end(),remainder,' '); 272 | dict.back() = '\n'; 273 | 274 | std::vector header; 275 | header += (char) 0x93; 276 | header += "NUMPY"; 277 | header += (char) 0x01; //major version of numpy format 278 | header += (char) 0x00; //minor version of numpy format 279 | #ifdef WORDS_BIGENDIAN 280 | header += swap_endian((unsigned short) dict.size()); 281 | #else 282 | header += (unsigned short) dict.size(); 283 | #endif 284 | header.insert(header.end(),dict.begin(),dict.end()); 285 | return header; 286 | } 287 | 288 | 289 | } 290 | 291 | #endif 292 | -------------------------------------------------------------------------------- /ChangeLog: -------------------------------------------------------------------------------- 1 | 2025-11-03 Dirk Eddelbuettel 2 | 3 | * DESCRIPTION (Version, Date): Release 0.2.14 4 | 5 | * README.md: Use R Project URL for GPL-2 link in badge 6 | 7 | 2025-04-02 Dirk Eddelbuettel 8 | 9 | * DESCRIPTION (Version, Date): Roll micro version and date 10 | 11 | * DESCRIPTION (Suggests): Add rbenchmark because of demo, remove pinp 12 | as vignettes have long been pre-made 13 | 14 | 2025-03-14 Dirk Eddelbuettel 15 | 16 | * .github/workflows/ci.yaml: Switch to r-ci with implicit bootstrap 17 | 18 | 2024-09-03 Dirk Eddelbuettel 19 | 20 | * DESCRIPTION (Version, Date): Release 0.2.13 21 | 22 | * DESCRIPTION (Authors@R): Added 23 | 24 | * tests/saveAndLoad.R: Wrap invisisble() around npySave() calls as 25 | the function returns NULL under current Rcpp 26 | * tests/saveAndLoad.Rout.save: Idem 27 | 28 | 2024-07-15 Dirk Eddelbuettel 29 | 30 | * README.md: Switch some URLs from http to https 31 | 32 | 2024-06-20 Dirk Eddelbuettel 33 | 34 | * .github/workflows/ci.yaml (jobs): Update to r-ci-setup action 35 | 36 | 2024-05-21 Dirk Eddelbuettel 37 | 38 | * README.md: Use tinyverse.netlify.app for dependency badge 39 | 40 | 2023-11-27 Dirk Eddelbuettel 41 | 42 | * DESCRIPTION (Version, Date): Release 0.2.12 43 | 44 | * src/cnpy.h (Rassert): Add format string to Rf_error call 45 | * src/cnpy.h (npy_save): Correct printf string for long size 46 | 47 | * .github/workflows/ci.yaml (jobs): Update to actions/checkout@v4 48 | 49 | 2023-02-08 Dirk Eddelbuettel 50 | 51 | * DESCRIPTION (Version, Date): Roll minor version 52 | 53 | * src/Makevars: No longer set CXX_STD 54 | * src/Makevars.win: Idem 55 | 56 | * inst/CITATION: Update to new format preferred by r-devel 57 | 58 | 2022-11-10 Dirk Eddelbuettel 59 | 60 | * .github/workflows/ci.yaml (jobs): Update to actions/checkout@v3 61 | 62 | 2022-03-24 Dirk Eddelbuettel 63 | 64 | * DESCRIPTION (Version, Date): Release 0.2.11 65 | 66 | * vignettes/*.Rnw: Created to Rnw wrappers for pdf vignettes 67 | * vignettes/rmd/*: Moved Rmd and bib files from vignettes/ 68 | * vignettes/pdf/*: Moved pdf vignettes/ created from Rmd 69 | 70 | * DESCRIPTION (Suggests): Removed knitr and rmarkdown 71 | * DESCRIPTION (VignetteBuilder): Removed 72 | * .Rbuildignore: Expanded 73 | 74 | * cleanup: Removed bashishm 75 | 76 | 2021-12-08 Dirk Eddelbuettel 77 | 78 | * README.md: Remove unused continuous integration artifact and badge 79 | 80 | 2021-04-24 Dirk Eddelbuettel 81 | 82 | * DESCRIPTION (URL): Add repo 83 | 84 | 2021-04-15 Dirk Eddelbuettel 85 | 86 | * README.md: Update two URLs to https 87 | 88 | 2021-01-03 Dirk Eddelbuettel 89 | 90 | * .github/workflows/ci.yaml: Add CI runner using r-ci 91 | * README.md: Add new badge 92 | 93 | 2020-06-20 Dirk Eddelbuettel 94 | 95 | * .travis.yml: Upgrade to 'bionic' and R 4.0.0 96 | 97 | 2020-06-19 Dirk Eddelbuettel 98 | 99 | * README.md: Add 'last commit' badge 100 | 101 | 2019-04-05 Dirk Eddelbuettel 102 | 103 | * README.md: Add dependency badge 104 | 105 | * vignettes/UsingReticulate.Rmd: Added brief section on 3d-arrays 106 | 107 | * tests/createFiles.py: Created three-dim array for vignette 108 | 109 | 2018-11-16 Dirk Eddelbuettel 110 | 111 | * man/RcppCNPy-package.Rd: Add that 32bit integer and 64bit floats 112 | are the key types, and casting may be needed from other width. 113 | 114 | 2018-09-02 Dirk Eddelbuettel 115 | 116 | * .travis.yml: Switch Travis CI to R 3.5 repo 117 | 118 | 2018-07-29 Dirk Eddelbuettel 119 | 120 | * DESCRIPTION (Version, Date): Release 0.2.10 121 | 122 | * README.md: Also mention reticulate and vignette 123 | 124 | * src/RcppExports.cpp: Added autogenerated file 125 | * src/init.c: Removed, content in src/RcppExports.cpp 126 | 127 | 2018-07-22 Dirk Eddelbuettel 128 | 129 | * vignettes/RcppCNPy-intro.Rmd: Updated, now with collapse option 130 | * vignettes/UsingReticulate.Rmd: Idem 131 | 132 | * vignettes/RcppCNPy.bib: New file to allow pandoc version 2.* use 133 | 134 | * .travis.yml: Also install r-cran-reticulate 135 | 136 | 2018-03-22 Dirk Eddelbuettel 137 | 138 | * DESCRIPTION (Version, Date): Release 0.2.9 139 | 140 | 2018-03-21 Dirk Eddelbuettel 141 | 142 | * vignettes/UsingReticulate.Rmd: New vignette showing how to use the 143 | reticulate package (and vignette results 'fixed' to not add dependency) 144 | 145 | 2018-02-28 Dirk Eddelbuettel 146 | 147 | * src/cnpyMod.cpp (npySave): Operate on bool from R; default to TRUE 148 | * R/utils.R (.checkPath): Now just returns the dir.exists() result 149 | 150 | * man/RcppCNPy-package.Rd: Updated documentation for new argument 151 | 152 | 2018-02-27 Dirk Eddelbuettel 153 | 154 | * src/cnpyMod.cpp (npySave): Add new option to check directory 155 | and path before attempting to save 156 | 157 | * R/utils.R (.checkPath): New helper function to check if a 158 | directory contained in a path exists 159 | 160 | 2018-01-04 Dirk Eddelbuettel 161 | 162 | * DESCRIPTION (Version, Date): Release 0.2.8 163 | 164 | * vignettes/RcppCNPy-intro.Rmd: Set knitr chunk option 165 | 'python.reticulate=FALSE' to avoid vignette dependency 166 | 167 | 2017-09-22 Dirk Eddelbuettel 168 | 169 | * DESCRIPTION (Version, Date): Release 0.2.7 170 | 171 | * cleanup: Expanded cleanup 172 | 173 | * vignettes/RcppCNPy-intro.Rmd: Re-built under pinp 0.0.2; also 174 | disable evaluation if Sys.info() detects the wonders of Windows 175 | 176 | 2017-09-17 Dirk Eddelbuettel 177 | 178 | * DESCRIPTION (Version, Date): Roll minor version 179 | 180 | * vignettes/RcppCNPy-intro.Rmd: Renamed, updated, uses pinp 181 | 182 | * .travis.yml (install): Add edd/misc ppa for r-cran-pinp 183 | 184 | * src/init.c: Added to register dynamic routines 185 | 186 | 2017-08-26 Dirk Eddelbuettel 187 | 188 | * .travis.yml (before_install): Use https for curl fetch 189 | 190 | 2016-09-25 Dirk Eddelbuettel 191 | 192 | * DESCRIPTION (Version, Date): Version 0.2.6 193 | 194 | * inst/CITATION: Added with new JOSS paper 195 | 196 | * README.md: One more canonical URL 197 | 198 | 2016-09-24 Dirk Eddelbuettel 199 | 200 | * README.md: Added Zenodo doi reference 201 | 202 | 2016-09-22 Dirk Eddelbuettel 203 | 204 | * README.md: Remove one redundant code line 205 | 206 | 2016-09-21 Dirk Eddelbuettel 207 | 208 | * README.md: Added 'Feedback' section, added JOSS badge 209 | 210 | * man/RcppCNPy-package.Rd: Add examples 211 | 212 | 2016-08-26 Dirk Eddelbuettel 213 | 214 | * DESCRIPTION: Version 0.2.5 215 | 216 | * vignettes/RcppCNPy-intro.Rnw: Updated vignette, added Wush 217 | 218 | * README.md: Updated, uses canonical URLs for CRAN 219 | 220 | 2016-08-24 Dirk Eddelbuettel 221 | 222 | * src/cnpy.cpp: Carry one small fix over from cnpy upstream; also 223 | applied to gzfile variant; revert to using 'long long' under C++11 224 | 225 | * .travis.yml: Switch to using run.sh for Travis CI 226 | 227 | 2015-02-12 Dirk Eddelbuettel 228 | 229 | * .travis.yml: Use ppa:edd/misc to apt-get install Rcpp and highlight 230 | 231 | 2015-01-05 Dirk Eddelbuettel 232 | 233 | * DESCRIPTION: Version 0.2.4 234 | 235 | * inst/NEWS.Rd: Updated with release news 236 | 237 | 2015-01-05 Wush Wu 238 | 239 | * src/cnpyMod.cpp: Correct the clone() instantiation for big endian 240 | 241 | 2015-01-04 Dirk Eddelbuettel 242 | 243 | * DESCRIPTION: Add Wush to Author:, bump Date: and Version: 244 | 245 | * src/cnpy.h: Remove BigEndianTest() 246 | * src/cnpy.cpp: Always set header entry to little endian 247 | * src/cnpyMod.cpp: Use Rcpp::clone() for deep copy 248 | * tests/saveAndLoad.R: Some simplifications 249 | * tests/saveAndLoad.Rout.save: Updated corresponding reference output 250 | 251 | 2015-01-04 Wush Wu 252 | 253 | * src/cnpy.cpp: Make BigEndianTest() unconditional 254 | * src/cnpy.h: Move swapEndian() here and use in header 255 | * src/cnpyMod.cpp: 256 | - Correct saved size for matrices, 257 | - Ensure deep copy prior to swapping endianness 258 | * tests/saveAndLoad.R: Add more tests 259 | 260 | 2015-01-03 Dirk Eddelbuettel 261 | 262 | * DESCRIPTION: Bump Date: and Version: 263 | 264 | * src/cnpyMod.cpp: Several corrections on file save: 265 | - Correct saving on integer matrices and vectors by 266 | testing for integer before testing for numeric (which is always true 267 | for int as well) 268 | - Convert integer object to int64_t and save as int64_t 269 | - For good measure, properly specify template type when saving 270 | - [So far untested] On big endian systems, convert before writing 271 | 272 | * tests/saveAndLoad.R: Added tests for saving (and loading) of both 273 | uncompressed and compressed integer matrices 274 | * tests/saveAndLoad.Rout.saev: Updated corresponding reference output 275 | 276 | 2015-01-02 Dirk Eddelbuettel 277 | 278 | * src/cnpyMod.cpp: Apply correction by Wush Wu to correct (if needed) 279 | for endianness before instantiating vectors when reading from file 280 | 281 | 2015-01-01 Dirk Eddelbuettel 282 | 283 | * src/cnpyMod.cpp: Move byte-swap call off into helper functions 284 | 285 | 2014-12-31 Dirk Eddelbuettel 286 | 287 | * src/cnpyMod.cpp: Correct call to byte swap for big endian systems 288 | 289 | 2014-12-28 Dirk Eddelbuettel 290 | 291 | * DESCRIPTION: Edited Title: and Description:; bumped Date: and Version: 292 | 293 | * tests/loadFiles.R: Print results; remove test for [known] int support 294 | * tests/loadFiles.Rout.save: Updated as well 295 | 296 | 2014-12-01 Romain Francois 297 | 298 | * src/cnpyMod.cpp: Use const_iterator 299 | 300 | 2014-08-05 Dirk Eddelbuettel 301 | 302 | * src/cnpyMod.cpp: Attempt to deal with big endian for int64 and double 303 | 304 | 2014-08-04 Dirk Eddelbuettel 305 | 306 | * vignettes/RcppCNPy-intro.Rnw: Do not assume 'system("gzip ...")' 307 | works because, y'a know, there is Windoze... 308 | 309 | 2014-04-10 Dirk Eddelbuettel 310 | 311 | * DESCRIPTION: Version 0.2.3 312 | * inst/NEWS.Rd: Updated with release news 313 | 314 | 2014-04-09 Dirk Eddelbuettel 315 | 316 | * vignettes/RcppCNPy-intro.Rnw: 317 | - Updated vignette to reflect integer support on all 318 | platforms thanks to 'long long' 319 | - Switched to font used in Rcpp vignettes 320 | - Switched to vignette processing via highlight 321 | 322 | * man/RcppCNPy-package.Rd: Updated to reflect integer support 323 | 324 | * src/Makevars.win: Updated as well 325 | 326 | * .travis.yml: Added _R_CHECK_FORCE_SUGGESTS_=FALSE to not 327 | stumble over Suggests: highlight not being installed 328 | 329 | 2014-04-08 Dirk Eddelbuettel 330 | 331 | * src/Makevars: Set 'CXX_STD = CXX11' which gets us 'long long' 332 | * DESCRIPTION: Add depends on R (>= 3.1.0) 333 | 334 | * src/cnpy.h: No longer test for RCPP_HAS_LONG_LONG_TYPES 335 | * src/cnpy.cpp: Idem 336 | * src/cnpyMod.cpp: Idem 337 | 338 | 2013-11-29 Dirk Eddelbuettel 339 | 340 | * DESCRIPTION: Version 0.2.2 341 | 342 | * src/cnpy.cpp (BigEndianTest): Re-use the R compile-time 343 | configuration result to switch between little and big endian 344 | 345 | * tests/saveAndLoad.R: New test file for save and load cycle 346 | * tests/saveAndLoad.Rout.save: And result for comparison 347 | 348 | 2013-11-28 Dirk Eddelbuettel 349 | 350 | * DESCRIPTION: Version 0.2.1 351 | 352 | * DESCRIPTION: Import methods and Rcpp 353 | * NAMESPACE: Idem 354 | 355 | * src/cnpy.h: Synchronized with version in cnpy repo 356 | * src/cnpy.cpp: Idem 357 | 358 | 2013-02-20 Dirk Eddelbuettel 359 | 360 | * src/cnpyMod.cpp (npyHasIntegerSupport): New function to test if 361 | integer support was compiled in, also exported via Rcpp Modules to R 362 | * man/RcppCNPy-package.Rd: Added documentation entry 363 | 364 | * tests/loadFiles.R: Updated regression test to use run-time integer 365 | support check, and test integer support if available 366 | * tests/loadFiles.Rout.save: Updated accordingly 367 | 368 | * RcppCNPy-intro.Rnw: Updated the discussion of integer support and 369 | clarified that only RcppCNPy needs to be rebuilt with -std=c++11 370 | * DESCRIPTION: Idem 371 | 372 | * inst/THANKS: Added to show thanks to Carl and Gong 373 | 374 | 2013-02-19 Dirk Eddelbuettel 375 | 376 | * src/cnpy.h: Define R_NO_REMAP before including Rinternals to not 377 | define length(), error(), ... but rather Rf_length(), Rf_error() 378 | 379 | * src/cnpy.cpp: Add a bogus unused variable at the never-reached end 380 | of code (after an error exit) to silence the pedantic mode of g++ 381 | 382 | 2012-07-30 Dirk Eddelbuettel 383 | 384 | * DESCRIPTION: Version 0.2.0 385 | 386 | * src/cnpyMod.cpp: 387 | (npySave): Support gzip'ed files for saving 388 | (npyLoad): Support new argument 'dotranspose', plug a memory leak 389 | 390 | * src/cnpy.h (cnpy): Added npy_gzsave() function 391 | 392 | * man/RcppCNPy-package.Rd: Document new gzip-compression 393 | 394 | * demo/timings.R: Use new gzip-compression feature 395 | 396 | * vignettes/RcppCNPy-intro.Rnw: Updated as well 397 | 398 | * inst/NEWS.Rd: Updated as well 399 | 400 | 2012-07-07 Dirk Eddelbuettel 401 | 402 | * DESCRIPTION: Version 0.1.0 403 | 404 | * vignettes/RcppCNPy-intro.Rnw: Added vignette documentation 405 | 406 | * demo/timings.R: Added simple timing benchmark demo 407 | 408 | 2012-07-06 Dirk Eddelbuettel 409 | 410 | * src/cnpy.h: Include cstdint for int64_t if C++0x has been enabled 411 | * src/cnpyMod.cpp: Support integer types if C++0x available 412 | 413 | * tests/: Simple set of regression tests added 414 | 415 | 2012-07-05 Dirk Eddelbuettel 416 | 417 | * src/cnpyMod.cpp: Added transpose() method to transparently deal 418 | with the Fortran-vs-C storage order difference between Python and R. 419 | Also added support for reading vectors. 420 | 421 | * src/cnpy.{cpp,h}: Added support for loading from .npy.gz files 422 | which is automagically enabled if the filename ends in ".gz" 423 | 424 | 2012-07-04 Dirk Eddelbuettel 425 | 426 | * Initial version 0.0.1 427 | 428 | * src/cnpy.{cpp,h}: Numerous minor changes to conform to CRAN Policy 429 | 430 | * src/cnpyMod.cpp: Rcpp modules wrapping of two function npyLoad and 431 | npySave which work on one- and two-dimensional objects of integer or 432 | numeric type. Note that Matrices need a transpose due to numpy 433 | Fortran ordering. 434 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | {description} 294 | Copyright (C) {year} {fullname} 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | {signature of Ty Coon}, 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | --------------------------------------------------------------------------------