├── requirements.txt ├── tests ├── __init__.py ├── conftest.py ├── test_io.py └── test_juicefs.py ├── docs ├── _static │ └── .gitkeep ├── readme.rst ├── changelog.rst ├── juicefs.rst ├── index.rst └── conf.py ├── juicefs ├── lib │ └── __init__.py ├── version.py ├── __init__.py ├── utils.py ├── libjfs.py ├── io.py └── juicefs.py ├── CONTRIBUTROS ├── CHANGELOG.md ├── pytest.ini ├── .gitignore ├── requirements-dev.txt ├── BUILD.md ├── setup.cfg ├── .github └── workflows │ ├── publish-docs.yml │ ├── on-push.yaml │ └── release.yaml ├── setup.py ├── Makefile ├── README.md ├── CHECKLIST.md └── LICENSE /requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/_static/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /juicefs/lib/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /juicefs/version.py: -------------------------------------------------------------------------------- 1 | VERSION = "0.0.4" 2 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../README.md 2 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. mdinclude:: ../CHANGELOG.md 2 | -------------------------------------------------------------------------------- /CONTRIBUTROS: -------------------------------------------------------------------------------- 1 | Contributors, order by family name: 2 | 3 | 董瑞晓 Ruixiao Dong 4 | 李阳 Yang Li 5 | 解彦博 Xie Yanbo 6 | 周君栋 Jundong Zhou 7 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | CHANGELOG 2 | ========= 3 | 4 | ## 0.0.1 - 2021.07.24 5 | 6 | - first release, including `os` / `io` compatible JuiceFS SDK 7 | -------------------------------------------------------------------------------- /docs/juicefs.rst: -------------------------------------------------------------------------------- 1 | API Reference 2 | ============= 3 | 4 | .. automodule:: juicefs 5 | :members: 6 | :undoc-members: 7 | :show-inheritance: 8 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | addopts = -v -s --cov=juicefs --no-cov-on-fail --cov-report=html:html_cov/ --cov-report=term-missing 3 | filterwarnings = 4 | error 5 | -------------------------------------------------------------------------------- /juicefs/__init__.py: -------------------------------------------------------------------------------- 1 | from juicefs.io import FileIO, open 2 | from juicefs.juicefs import JuiceFS, JuiceFSPath 3 | from juicefs.version import VERSION as __version__ 4 | 5 | __all__ = ["JuiceFS", "JuiceFSPath", "FileIO", "open"] 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /build 2 | /dist 3 | /*.egg-info 4 | /.pytest_cache 5 | /.pytype_output 6 | __pycache__/ 7 | *.pyc 8 | *.swp 9 | .coverage 10 | html_cov/ 11 | html_doc/ 12 | 13 | juicefs/lib/juicefs 14 | juicefs/lib/libjfs.so 15 | wheelhouse/ 16 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | Welcome to juicefs-python's documentation! 2 | ========================================== 3 | 4 | .. toctree:: 5 | :maxdepth: 4 6 | :caption: Contents: 7 | 8 | readme 9 | juicefs 10 | changelog 11 | 12 | 13 | Indices and tables 14 | ================== 15 | 16 | * :ref:`genindex` 17 | * :ref:`modindex` 18 | * :ref:`search` 19 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | setuptools 2 | wheel 3 | 4 | pytest 5 | pytest-cov 6 | pytest-mock 7 | pytest-socket 8 | 9 | pytype; sys_platform != 'win32' and python_version == '3.6' 10 | 11 | isort ~= 5.0; sys_platform != 'win32' and python_version == '3.6' 12 | black ~= 21.5b0; sys_platform != 'win32' and python_version == '3.6' 13 | 14 | Sphinx >= 4.0.0; sys_platform != 'win32' and python_version == '3.6' 15 | sphinx-rtd-theme; sys_platform != 'win32' and python_version == '3.6' 16 | m2r2; sys_platform != 'win32' and python_version == '3.6' 17 | -------------------------------------------------------------------------------- /BUILD.md: -------------------------------------------------------------------------------- 1 | ## Build libjfs.so 2 | 3 | ### Install go 4 | 5 | ```shell 6 | sudo add-apt-repository ppa:longsleep/golang-backports 7 | sudo apt update 8 | sudo apt install golang-go 9 | ``` 10 | 11 | ### Download JuiceFS Source Code 12 | 13 | ```shell 14 | git clone https://github.com/juicedata/juicefs.git 15 | ``` 16 | 17 | Better to checkout to the last release version. 18 | 19 | ### Install Dependencies 20 | 21 | ```shell 22 | go mod download 23 | ``` 24 | 25 | ### Build 26 | 27 | ```shell 28 | cd sdk/java/libjfs 29 | make libjfs.so 30 | ``` 31 | 32 | -------------------------------------------------------------------------------- /juicefs/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | from typing import Optional 3 | 4 | 5 | def create_os_error( 6 | code: int, filename: Optional[str] = None, filename2: Optional[str] = None 7 | ): 8 | args = [code, os.strerror(code)] 9 | if filename is not None: 10 | args.append(filename) 11 | if filename2 is not None: 12 | args.append(None) 13 | args.append(filename2) 14 | return OSError(*args) 15 | 16 | 17 | def check_juicefs_error( 18 | code: int, filename: Optional[str] = None, filename2: Optional[str] = None 19 | ): 20 | if code < 0: 21 | # juicefs 返回的错误码是负数 22 | raise create_os_error(-code, filename, filename2) 23 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [tool.black] 2 | line-length = 88 3 | target-version = ['py36'] 4 | include_trailing_comma = True 5 | 6 | [isort] 7 | # Ignore line width limit for import 8 | line_length = 88 9 | profile = black 10 | 11 | 12 | [report] 13 | exclude_lines = 14 | pragma: no cover 15 | 16 | [pytype] 17 | # Space-separated list of files or directories to exclude. 18 | exclude = 19 | tests/ 20 | 21 | # Keep going past errors to analyze as many files as possible. 22 | keep_going = True 23 | 24 | # All pytype output goes here. 25 | output = .pytype_output 26 | 27 | # Paths to source code directories, separated by ':'. 28 | pythonpath = 29 | . 30 | 31 | # Python version (major.minor) of the target code. 32 | python_version = 3.6 33 | 34 | # Comma separated list of error names to ignore. 35 | disable = pyi-error 36 | 37 | # Don't report errors. 38 | report_errors = True 39 | 40 | # Experimental: solve unknown types to label with structural types. 41 | protocols = False 42 | 43 | # Experimental: Only load submodules that are explicitly imported. 44 | strict_import = False 45 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | import sys 4 | 5 | import pytest 6 | 7 | import juicefs 8 | from juicefs import JuiceFS 9 | 10 | NAME = "test-jfs" 11 | BUCKET = "/tmp" 12 | META = "/tmp/test-jfs.db" 13 | META_URL = "sqlite3:///tmp/test-jfs.db" 14 | 15 | if sys.platform == "win32": 16 | BUCKET = os.path.join(__file__, "..", "tmp") 17 | META = os.path.join(__file__, "..", "test-jfs.db") 18 | META_URL = r"sqlite3:///{}".format(os.path.join(__file__, "..", "test-jfs.db")) 19 | 20 | 21 | @pytest.fixture(scope="session") 22 | def jfs(): 23 | return JuiceFS(NAME, {"meta": META_URL}) 24 | 25 | 26 | def format_juicefs(): 27 | if os.path.exists(META): 28 | os.unlink(META) 29 | if os.path.exists(os.path.join(BUCKET, NAME)): 30 | shutil.rmtree(os.path.join(BUCKET, NAME)) 31 | 32 | juicefs_binary = os.path.normpath( 33 | os.path.join(juicefs.__file__, "..", "lib", "juicefs") 34 | ) 35 | 36 | commands = [ 37 | juicefs_binary, 38 | "format", 39 | "--bucket=%s" % BUCKET, 40 | "--force", 41 | META_URL, 42 | NAME, 43 | ] 44 | 45 | command = " ".join(commands) 46 | print("run command:", command) 47 | os.system(command) 48 | 49 | 50 | format_juicefs() 51 | -------------------------------------------------------------------------------- /.github/workflows/publish-docs.yml: -------------------------------------------------------------------------------- 1 | name: docs 2 | on: 3 | workflow_dispatch: 4 | release: 5 | types: [published] 6 | 7 | jobs: 8 | publish-doc: 9 | runs-on: ${{ matrix.os }} 10 | strategy: 11 | matrix: 12 | os: [ubuntu-18.04] 13 | python-version: [3.7] 14 | 15 | steps: 16 | - name: Checkout Github Repository 17 | uses: actions/checkout@v2 18 | - name: Set up Python 19 | uses: actions/setup-python@v2 20 | with: 21 | python-version: ${{ matrix.python-version }} 22 | - name: Cache pip 23 | id: pip-cache 24 | uses: actions/cache@v2 25 | with: 26 | path: /opt/hostedtoolcache/Python 27 | key: ${{ matrix.os }}-python${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }} 28 | restore-keys: | 29 | ${{ matrix.os }}-pip- 30 | ${{ matrix.os }}- 31 | - name: Install package dependencies 32 | if: steps.pip-cache.outputs.cache-hit != 'true' 33 | run: | 34 | python -m pip install --upgrade pip 35 | pip install -r requirements-dev.txt 36 | pip install -r requirements.txt 37 | - name: Run build-sphinx 38 | run: | 39 | make doc 40 | 41 | - name: Deploy 42 | uses: peaceiris/actions-gh-pages@v3 43 | with: 44 | github_token: ${{ secrets.GITHUB_TOKEN }} 45 | publish_dir: ./html_doc/html 46 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import sys 2 | from importlib.machinery import SourceFileLoader 3 | 4 | from setuptools import find_packages, setup 5 | 6 | 7 | def load_version(filename): 8 | loader = SourceFileLoader(filename, filename) 9 | return loader.load_module().VERSION 10 | 11 | 12 | def load_requirements(filename): 13 | with open(filename) as fd: 14 | return fd.readlines() 15 | 16 | 17 | requirements = load_requirements("requirements.txt") 18 | test_requirements = load_requirements("requirements-dev.txt") 19 | 20 | package_data = {'juicefs.lib': ['libjfs.so', 'juicefs']} 21 | if sys.platform == 'win32': 22 | package_data = {'juicefs.lib': ['libjfs.dll', 'juicefs.exe']} 23 | 24 | setup( 25 | name="juicefs", 26 | description="JuiceFS Python SDK", 27 | version=load_version("juicefs/version.py"), 28 | author="r-eng", 29 | author_email="r-eng@megvii.com", 30 | url="https://github.com/megvii-research/juicefs-python", 31 | packages=find_packages(exclude=("tests",)), 32 | package_data=package_data, 33 | classifiers=[ 34 | 'Development Status :: 4 - Beta', 35 | 'Environment :: Console', 36 | 'Intended Audience :: Developers', 37 | 'License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)', # noqa 38 | 'Operating System :: POSIX :: Linux', 39 | 'Programming Language :: Python', 40 | 'Programming Language :: Python :: 3', 41 | 'Programming Language :: Python :: 3 :: Only', 42 | 'Programming Language :: Python :: 3.5', 43 | 'Programming Language :: Python :: 3.6', 44 | 'Programming Language :: Python :: 3.7', 45 | 'Programming Language :: Python :: 3.8', 46 | 'Programming Language :: Python :: 3.9', 47 | 'Programming Language :: Python :: 3.10', 48 | 'Topic :: Software Development', 49 | 'Topic :: Utilities' 50 | ], 51 | tests_require=test_requirements, 52 | install_requires=requirements, 53 | python_requires=">=3.5", 54 | ) 55 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # http://www.sphinx-doc.org/en/master/config 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | import os 14 | import sys 15 | 16 | sys.path.insert(0, "./../") 17 | 18 | # -- Project information ----------------------------------------------------- 19 | 20 | project = "juicefs-python" 21 | copyright = "2021, Megvii Research Engineering Team" 22 | author = "r-eng" 23 | 24 | # -- General configuration --------------------------------------------------- 25 | 26 | # Add any Sphinx extension module names here, as strings. They can be 27 | # extensions coming with Sphinx (named "sphinx.ext.*") or your custom 28 | # ones. 29 | extensions = [ 30 | "sphinx.ext.autodoc", 31 | "sphinx.ext.viewcode", 32 | "sphinx.ext.todo", 33 | "sphinx_rtd_theme", 34 | "m2r2", 35 | ] 36 | 37 | # Add any paths that contain templates here, relative to this directory. 38 | templates_path = ["_templates"] 39 | 40 | # The language for content autogenerated by Sphinx. Refer to documentation 41 | # for a list of supported languages. 42 | # 43 | # This is also used if you do content translation via gettext catalogs. 44 | # Usually you set "language" from the command line for these cases. 45 | language = "en" 46 | 47 | # List of patterns, relative to source directory, that match files and 48 | # directories to ignore when looking for source files. 49 | # This pattern also affects html_static_path and html_extra_path. 50 | exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] 51 | 52 | # -- Options for HTML output ------------------------------------------------- 53 | 54 | # The theme to use for HTML and HTML Help pages. See the documentation for 55 | # a list of builtin themes. 56 | # 57 | html_theme = "sphinx_rtd_theme" 58 | 59 | # Add any paths that contain custom static files (such as style sheets) here, 60 | # relative to this directory. They are copied after the builtin static files, 61 | # so a file named "default.css" will overwrite the builtin "default.css". 62 | html_static_path = ["_static"] 63 | 64 | # -- Extension configuration ------------------------------------------------- 65 | 66 | # -- Options for todo extension ---------------------------------------------- 67 | 68 | # If true, `todo` and `todoList` produce output, else they produce nothing. 69 | todo_include_todos = True 70 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | PACKAGE := juicefs 2 | VERSION := $(shell cat juicefs/version.py | sed -n -E 's/^VERSION = "(.+?)"/\1/p') 3 | JUICEFS_VERSION := 0.14.2 4 | 5 | clean: 6 | rm -rf dist build *.egg-info .pytype .pytest_cache .pytype_output 7 | 8 | build_libjfs_so: 9 | rm -rf build && mkdir build 10 | cd build \ 11 | && wget https://github.com/juicedata/juicefs/archive/refs/tags/v${JUICEFS_VERSION}.zip \ 12 | && unzip v${JUICEFS_VERSION}.zip 13 | cd build/juicefs-${JUICEFS_VERSION}/sdk/java/libjfs \ 14 | && make libjfs.so \ 15 | && cp libjfs.so ../../../../../juicefs/lib/libjfs.so 16 | cd build/juicefs-${JUICEFS_VERSION} \ 17 | && make juicefs \ 18 | && cp juicefs ../../juicefs/lib/juicefs 19 | 20 | build_libjfs_dll: 21 | rm -rf build && mkdir build 22 | cd build \ 23 | && curl -L -O https://github.com/juicedata/juicefs/archive/refs/tags/v${JUICEFS_VERSION}.zip \ 24 | && unzip v${JUICEFS_VERSION}.zip 25 | cd build/juicefs-${JUICEFS_VERSION}/sdk/java/libjfs \ 26 | && make libjfs.dll \ 27 | && realpath ./ \ 28 | && realpath ../../../../../juicefs/lib/libjfs.dll \ 29 | && test -f libjfs.dll; echo $? \ 30 | && cp libjfs.dll ../../../../../juicefs/lib/libjfs.dll \ 31 | && test -f ../../../../../juicefs/lib/libjfs.dll; echo $? 32 | cd build/juicefs-${JUICEFS_VERSION} \ 33 | && make juicefs.exe \ 34 | && realpath ./ \ 35 | && realpath ../../juicefs/lib/juicefs.exe \ 36 | && test -f juicefs.exe; echo $? \ 37 | && cp juicefs.exe ../../juicefs/lib/juicefs.exe \ 38 | && test -f ../../juicefs/lib/juicefs.exe; echo $? 39 | 40 | print_libjfs_version: 41 | echo ${JUICEFS_VERSION} 42 | 43 | build_wheel: 44 | python3 setup.py bdist_wheel 45 | 46 | static_check: 47 | pytype ${PACKAGE} 48 | 49 | test: 50 | pytest -s --cov=${PACKAGE} --no-cov-on-fail --cov-report=html:html_cov/ --cov-report=term-missing tests 51 | 52 | style_check: 53 | isort --check --diff ${PACKAGE} 54 | black --check --diff ${PACKAGE} 55 | 56 | format: 57 | autoflake --remove-unused-variables --remove-all-unused-imports --ignore-init-module-imports -r -i ${PACKAGE} tests 58 | isort ${PACKAGE} tests 59 | black ${PACKAGE} tests 60 | 61 | doc: 62 | python3 setup.py build_sphinx --fresh-env --build-dir html_doc/ 63 | 64 | release: 65 | # git tag ${VERSION} 66 | # git push origin ${VERSION} 67 | 68 | rm -rf build dist 69 | python3 setup.py bdist_wheel 70 | auditwheel repair --plat manylinux2014_x86_64 dist/${PACKAGE}-${VERSION}-py3-none-any.whl 71 | 72 | devpi login ${PYPI_USERNAME} --password=${PYPI_PASSWORD} 73 | devpi upload wheelhouse/${PACKAGE}-${VERSION}-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 74 | twine upload wheelhouse/${PACKAGE}-${VERSION}-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl --username=${PYPI_USERNAME_2} --password=${PYPI_PASSWORD_2} 75 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | JuiceFS Python SDK 2 | === 3 | 4 | [![Build](https://github.com/megvii-research/juicefs-python/actions/workflows/on-push.yaml/badge.svg?branch=main)](https://github.com/megvii-research/juicefs-python/actions/workflows/on-push.yaml) 5 | [![Documents](https://github.com/megvii-research/juicefs-python/actions/workflows/publish-docs.yml/badge.svg)](https://github.com/megvii-research/juicefs-python/actions/workflows/publish-docs.yml) 6 | [![Code Coverage](https://img.shields.io/codecov/c/gh/megvii-research/juicefs-python)](https://app.codecov.io/gh/megvii-research/juicefs-python/) 7 | [![Latest version](https://img.shields.io/pypi/v/juicefs.svg)](https://pypi.org/project/juicefs/) 8 | [![Support python versions](https://img.shields.io/pypi/pyversions/juicefs.svg)](https://pypi.org/project/juicefs/) 9 | [![License](https://img.shields.io/pypi/l/juicefs.svg)](https://github.com/megvii-research/juicefs-python/blob/master/LICENSE) 10 | 11 | - Docs: https://megvii-research.github.io/juicefs-python/ 12 | 13 | ## What is this ? 14 | 15 | 16 | [JuiceFS](https://github.com/juicedata/juicefs) is a high-performance POSIX file system released under GNU Affero General Public License v3.0. 17 | 18 | `juicefs-python` is JuiceFS Python SDK, provides a majority of APIs in `os` module, complete APIs of `io.FileIO` and `io.open()`, based on JuiceFS APIs. 19 | 20 | `juicefs-python` works on Linux, macOS and Windows, you can install it via PyPI, where the wheel package also includes `libjfs.so`. 21 | 22 | ## Installation 23 | 24 | ### PyPI 25 | 26 | Use pip install JuiceFS Python SDK package: 27 | ``` 28 | pip install juicefs 29 | ``` 30 | 31 | ### Build from Source 32 | 33 | Clone the repository: 34 | 35 | ``` 36 | git clone git@github.com:megvii-research/juicefs-python.git 37 | ``` 38 | 39 | And then build `juicefs` and `libjfs.so` and install requirements: 40 | 41 | ``` 42 | cd juicefs-python 43 | make build_juicefs 44 | pip install -r requirements.txt 45 | ``` 46 | 47 | If you want to develop based on JuiceFS Python SDK package, you may want to `pip install -r requirements-dev.txt`. 48 | 49 | ## How to Contribute 50 | 51 | * You can help to improve juicefs-python in many ways: 52 | * Write code. 53 | * Improve [documentation](https://github.com/megvii-research/juicefs-python/blob/main/docs). 54 | * Report or investigate [bugs and issues](https://github.com/megvii-research/juicefs-python/issues). 55 | * Review [pull requests](https://github.com/megvii-research/juicefs-python/pulls). 56 | * Star juicefs-python repo. 57 | * Recommend juicefs-python to your friends. 58 | * Any other form of contribution is welcomed. 59 | * We are happy to see your contribution to juicefs-python. Before contributing to this project, you should follow these rules: 60 | * **Code format**: Use `make format` to format the code before pushing your code to repository. 61 | * **Test**:`pytest` is used to test the code in this project. You should use `make test` to do the test the code. This should be done before pushing your code, asuring bug-free code based on complete tests. 62 | * **Static check**:We use `pytype` to do the static check. `make static_check` can help finish static check. 63 | * **Others**: You can get more details in `Makefile` at the root path. 64 | 65 | ## Resources 66 | 67 | - [JuiceFS](https://github.com/juicedata/juicefs) 68 | - [JuiceFS Hadoop Java SDK](https://github.com/juicedata/juicefs/blob/main/docs/en/hadoop_java_sdk.md) 69 | - [Build libjfs.so](https://github.com/megvii-research/juicefs-python/blob/main/BUILD.md) 70 | 71 | 72 | ## License 73 | 74 | `juicefs-python` is open-sourced under GNU AGPL v3.0, see [LICENSE](https://github.com/megvii-research/juicefs-python/blob/main/LICENSE). 75 | -------------------------------------------------------------------------------- /CHECKLIST.md: -------------------------------------------------------------------------------- 1 | ### JuiceFS SDK 2 | 3 | - [x] int jfs_access(long pid, long h, String path, int flags); 4 | - [x] int jfs_chmod(long pid, long h, String path, int mode); 5 | - [x] int jfs_chown(long pid, long h, String path); 6 | - [x] int jfs_close(long pid, int fd); 7 | - [x] int jfs_concat(long pid, long h, String path, Pointer buf, int bufsize); 8 | - [x] int jfs_create(long pid, long h, String path, short mode); 9 | - [x] int jfs_delete(long pid, long h, String path); 10 | - [x] int jfs_flush(long pid, int fd); 11 | - [x] int jfs_fsync(long pid, int fd); 12 | - [x] long jfs_init(String name, String jsonConf, String user, String group, String superuser, String supergroup); 13 | - [x] int jfs_listdir(long pid, long h, String path, int offset, Pointer buf, int size); 14 | - [x] long jfs_lseek(long pid, int fd, long pos, int whence); 15 | - [x] int jfs_lstat1(long pid, long h, String path, Pointer buf); 16 | - [x] int jfs_mkdir(long pid, long h, String path, short mode); 17 | - [x] int jfs_open(long pid, long h, String path, int flags); 18 | - [x] int jfs_pread(long pid, int fd, Pointer b, int len, long offset); 19 | - [x] int jfs_readlink(long pid, long h, String path, Pointer buf, int bufsize); 20 | - [x] int jfs_rename(long pid, long h, String src, String dst); 21 | - [x] int jfs_rmr(long pid, long h, String path); 22 | - [x] int jfs_stat1(long pid, long h, String path, Pointer buf); 23 | - [x] int jfs_statvfs(long pid, long h, Pointer buf); 24 | - [x] int jfs_summary(long pid, long h, String path, Pointer buf); 25 | - [x] int jfs_symlink(long pid, long h, String target, String path); 26 | - [x] int jfs_term(long pid, long h); 27 | - [x] int jfs_truncate(long pid, long h, String path, long length); 28 | - [x] int jfs_utime(long pid, long h, String path, long mtime, long atime); 29 | - [x] int jfs_write(long pid, int fd, Pointer b, int len); 30 | - [ ] void jfs_update_uid_grouping(long h, String uidstr, String grouping); 31 | - int jfs_getXattr(long pid, long h, String path, String name, Pointer buf, int size); 32 | - int jfs_listXattr(long pid, long h, String path, Pointer buf, int size); 33 | - int jfs_removeXattr(long pid, long h, String path, String name); 34 | - int jfs_setOwner(long pid, long h, String path, String user, String group); 35 | - int jfs_setXattr(long pid, long h, String path, String name, Pointer value, int vlen, int mode); 36 | 37 | ### Python SDK 38 | 39 | - [x] os.fdopen(fd) 40 | - [ ] os.close(fd) 41 | - os.fchmod(fd) 42 | - os.fchown(fd) 43 | - os.fstat(fd) 44 | - os.fstatvfs(fd) 45 | - [ ] os.fsync(fd) 46 | - [x] os.ftruncate(fd, length) 47 | - [ ] os.lseek(fd, pos, how) 48 | - [ ] os.pread(fd, n, offset) 49 | - os.pwrite(fd, str, offset) 50 | - [ ] os.read(fd, n) 51 | - [ ] os.write(fd, str) 52 | - [x] os.access(path, mode) 53 | - [x] os.chmod(path, mode) 54 | - os.chown(path, user, group) 55 | - os.getxattr(path, attribute) 56 | - os.lchmod(path, mode) 57 | - os.lchown(path, user, group) 58 | - [x] os.listdir(path) 59 | - os.listxattr(path=None) 60 | - [x] os.lstat(path) 61 | - [x] os.mkdir(path, mode=0o777) 62 | - [ ] os.open(path, flags, mode=0o777) 63 | - [x] os.readlink(path) 64 | - [x] os.remove(path) 65 | - os.removexattr(path, attribute) 66 | - [x] os.rmdir(path) 67 | - [x] os.scandir(path) 68 | - os.setxattr(path, attribute, value, flags=0) 69 | - [x] os.stat(path) 70 | - [ ] os.statvfs() 71 | - [x] os.truncate(path, length) 72 | - [x] os.unlink(path) 73 | - [x] os.utime(path, times=None) 74 | - os.sendfile(out_fd, in_fd, offset, count) 75 | - os.link(src, dst) 76 | - [x] os.makedirs(name, mode=0o777, exist_ok=False) 77 | - [x] os.removedirs(name) 78 | - [x] os.rename(src, dst) 79 | - os.renames(old, new) 80 | - [x] os.replace(src, dst) 81 | - [x] os.symlink(src, dst) 82 | - [x] os.walk(top, topdown=True, onerror=None) 83 | - [x] os.path.exists(path) 84 | - [x] os.path.lexists(path) 85 | - [x] os.path.getatime(path) 86 | - [x] os.path.getmtime(path) 87 | - os.path.getctime(path) 88 | - [x] os.path.getsize(path) 89 | - os.path.samefile(src, dst) 90 | - [x] os.path.isfile(path) 91 | - [x] os.path.isdir(path) 92 | - [x] os.path.islink(path) 93 | -------------------------------------------------------------------------------- /juicefs/libjfs.py: -------------------------------------------------------------------------------- 1 | import getpass 2 | import json 3 | import os 4 | import stat 5 | import struct 6 | from ctypes import CDLL 7 | from io import BytesIO 8 | from pathlib import Path 9 | from threading import current_thread 10 | from typing import Callable, List, Optional 11 | 12 | from juicefs.utils import check_juicefs_error 13 | 14 | 15 | def read_cstring(buffer: BytesIO) -> bytearray: 16 | res = bytearray() 17 | while True: 18 | b = buffer.read(1) 19 | if not b: 20 | return res 21 | c = ord(b) 22 | if c == 0: 23 | break 24 | res.append(c) 25 | return res 26 | 27 | 28 | def parse_stat_mode(mode: int) -> int: 29 | # https://github.com/juicedata/juicefs/blob/main/pkg/fs/fs.go#L78 30 | # func (fs *FileStat) Mode() os.FileMode 31 | # https://golang.org/pkg/io/fs/#FileMode 32 | res = mode & 0o777 33 | if mode & (1 << 31): # ModeDir 34 | res |= stat.S_IFDIR 35 | elif mode & (1 << 27): # ModeSymlink 36 | res |= stat.S_IFLNK 37 | else: 38 | res |= stat.S_IFREG 39 | if mode & (1 << 23): # ModeSetuid 40 | res |= stat.S_ISUID 41 | if mode & (1 << 22): # ModeSetgid 42 | res |= stat.S_ISGID 43 | if mode & (1 << 20): # ModeSticky 44 | res |= stat.S_ISVTX 45 | return res 46 | 47 | 48 | def parse_xattrs(data: bytes, length: int) -> List[str]: 49 | res = [] 50 | buffer = BytesIO(data) 51 | while buffer.tell() < length: 52 | res.append(read_cstring(buffer).decode()) 53 | return res 54 | 55 | 56 | def create_stat_result(data: bytes, length: int) -> os.stat_result: 57 | buffer = BytesIO(data) 58 | mode, size, mtime, atime = struct.unpack(" os.statvfs_result: 84 | blocks, bavail = struct.unpack("" % self.name 117 | 118 | @property 119 | def path(self): 120 | return Path(os.path.join(self.root, self.name)).as_posix() 121 | 122 | def inode(self): 123 | return self._stat.st_ino 124 | 125 | def is_dir(self): 126 | return stat.S_ISDIR(self._stat.st_mode) 127 | 128 | def is_file(self): 129 | return stat.S_ISREG(self._stat.st_mode) 130 | 131 | def is_symlink(self): 132 | return stat.S_ISLNK(self._stat.st_mode) 133 | 134 | def stat(self): 135 | return self._stat 136 | 137 | 138 | class DirSummary: 139 | def __init__(self, size: int, files: int, dirs: int): 140 | self.size = size # type: int 141 | self.files = files # type: int 142 | self.dirs = dirs # type: int 143 | 144 | def __repr__(self): 145 | return "" % ( 146 | self.size, 147 | self.files, 148 | self.dirs, 149 | ) 150 | 151 | 152 | class LibJuiceFSFunction: 153 | def __init__(self, func: Callable, handle: int, nargs: Optional[int] = None): 154 | self._func = func 155 | self._handle = handle 156 | self._nargs = nargs 157 | 158 | @property 159 | def _ident(self) -> int: 160 | return current_thread().ident 161 | 162 | def __call__(self, *args): 163 | jfs_func = self._func 164 | jfs_args = [self._ident, self._handle] 165 | for arg in args: 166 | if isinstance(arg, str): 167 | arg = arg.encode() 168 | jfs_args.append(arg) 169 | code = jfs_func(*jfs_args) 170 | if self._nargs is not None: 171 | check_juicefs_error(code, *args[: self._nargs]) 172 | return code 173 | 174 | def __getitem__(self, nargs: int): 175 | return LibJuiceFSFunction(self._func, self._handle, nargs) 176 | 177 | 178 | class LibJuiceFSHandle: 179 | def __init__(self, lib, handle: int): 180 | self._lib = lib 181 | self._handle = handle 182 | 183 | def __getattr__(self, name: str): 184 | func = getattr(self._lib, "jfs_%s" % name) 185 | return LibJuiceFSFunction(func, self._handle) 186 | 187 | 188 | class LibJuiceFS(LibJuiceFSHandle): 189 | def __init__(self, path, name: str, config: dict): 190 | self._lib = CDLL(path) 191 | self._handle = self.init(name, config) 192 | 193 | def init(self, name: str, config: dict): 194 | handle = self._lib.jfs_init( 195 | name.encode(), # name 196 | json.dumps(config).encode(), # conf 197 | getpass.getuser().encode(), # user 198 | b"nogroup", # group 199 | b"root", # superuser 200 | b"nogroup", # supergroup 201 | ) 202 | if handle <= 0: 203 | raise IOError("JuiceFS initialized failed for jfs://%s" % name) 204 | return handle 205 | 206 | def __getitem__(self, handle: int): 207 | return LibJuiceFSHandle(self._lib, handle) 208 | 209 | def __del__(self): 210 | if hasattr(self, "_handle"): 211 | self.term() 212 | -------------------------------------------------------------------------------- /.github/workflows/on-push.yaml: -------------------------------------------------------------------------------- 1 | name: build 2 | on: 3 | workflow_dispatch: 4 | push: 5 | pull_request: 6 | 7 | jobs: 8 | build-binaries: 9 | name: build juicefs (${{ matrix.target }}) 10 | runs-on: ${{ matrix.os }} 11 | strategy: 12 | matrix: 13 | include: 14 | - target: linux 15 | os: ubuntu-18.04 16 | - target: windows 17 | os: ubuntu-18.04 18 | - target: macos 19 | os: macos-latest 20 | 21 | steps: 22 | - name: Checkout Github Repository 23 | uses: actions/checkout@v2 24 | 25 | - name: Get JuiceFS Version 26 | id: jfs-version 27 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 28 | 29 | - name: Cache binary files 30 | uses: actions/cache@v2 31 | id: jfs-cache 32 | with: 33 | path: | 34 | ./juicefs/lib/ 35 | !./juicefs/lib/*.py 36 | key: cache-jfs-${{ matrix.target }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 37 | 38 | - name: Set up Golang 39 | uses: actions/setup-go@v2 40 | if: ${{ steps.jfs-cache.outputs.cache-hit != 'true' }} 41 | 42 | - name: Build for linux or macos 43 | if: ${{ steps.jfs-cache.outputs.cache-hit != 'true' && matrix.target != 'Windows' }} 44 | run: | 45 | make build_libjfs_so 46 | 47 | - name: Install MinGW GCC 48 | if: ${{ steps.jfs-cache.outputs.cache-hit != 'true' && matrix.target == 'Windows' }} 49 | run: | 50 | sudo apt-get update 51 | sudo apt-get install gcc-mingw-w64-x86-64 52 | 53 | - name: Build for windows 54 | if: ${{ steps.jfs-cache.outputs.cache-hit != 'true' && matrix.target == 'Windows' }} 55 | run: | 56 | make build_libjfs_dll 57 | 58 | # Artifact and cache are separated, and artifact lacks the ability to check 59 | # if it exists, so we upload the artifact repeatedly every time. 60 | - name: Upload binary artifact 61 | uses: actions/upload-artifact@v2 62 | with: 63 | name: jfs-binary-${{ matrix.target }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 64 | path: | 65 | ./juicefs/lib/libjfs* 66 | ./juicefs/lib/juicefs* 67 | retention-days: 1 68 | 69 | test: 70 | needs: build-binaries 71 | runs-on: ${{ matrix.os }} 72 | strategy: 73 | matrix: 74 | os: [macos-latest, ubuntu-18.04, windows-latest] 75 | python-version: ["3.5", "3.6", "3.7", "3.8", "3.9", "3.10"] 76 | 77 | include: 78 | - os: ubuntu-18.04 79 | target: linux 80 | pip-path: /opt/hostedtoolcache/Python 81 | - os: macos-latest 82 | target: macos 83 | pip-path: /Users/runner/hostedtoolcache/Python 84 | - os: windows-latest 85 | target: windows 86 | pip-path: C:\hostedtoolcache\windows\Python 87 | 88 | steps: 89 | - name: Checkout Github Repository 90 | uses: actions/checkout@v2 91 | 92 | - name: Get JuiceFS Version 93 | id: jfs-version 94 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 95 | 96 | - name: Download jfs binary artifact 97 | uses: actions/download-artifact@v2 98 | with: 99 | name: jfs-binary-${{ matrix.target }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 100 | path: ./juicefs/lib/ 101 | 102 | - name: Make juicefs executable 103 | run: | 104 | chmod 755 ./juicefs/lib/juicefs* 105 | ls -l ./juicefs/lib/ 106 | 107 | - name: Set up Python 108 | uses: actions/setup-python@v2 109 | with: 110 | python-version: ${{ matrix.python-version }} 111 | 112 | # - name: Cache Pip 113 | # id: pip-cache 114 | # uses: actions/cache@v2 115 | # with: 116 | # path: ${{ matrix.pip-path }} 117 | # key: ${{ matrix.os }}-python${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }} 118 | 119 | - name: Install Package Dependencies 120 | # if: steps.pip-cache.outputs.cache-hit != 'true' 121 | run: | 122 | python -m pip install --upgrade pip 123 | pip install -r requirements-dev.txt 124 | pip install -r requirements.txt 125 | 126 | - name: Run unit-test 127 | run: | 128 | make test 129 | 130 | - name: "Upload coverage to Codecov" 131 | uses: codecov/codecov-action@v1 132 | with: 133 | fail_ci_if_error: true 134 | 135 | check: 136 | name: static check (${{ matrix.os }}, ${{ matrix.python-version }}) 137 | needs: build-binaries 138 | runs-on: ${{ matrix.os }} 139 | strategy: 140 | matrix: 141 | os: [ubuntu-18.04] 142 | python-version: ["3.6"] 143 | include: 144 | - os: ubuntu-18.04 145 | target: linux 146 | pip-path: /opt/hostedtoolcache/Python 147 | 148 | steps: 149 | - name: Checkout Github Repository 150 | uses: actions/checkout@v2 151 | 152 | - name: Set up Python 153 | uses: actions/setup-python@v2 154 | with: 155 | python-version: ${{ matrix.python-version }} 156 | 157 | # - name: Cache Pip 158 | # id: pip-cache 159 | # uses: actions/cache@v2 160 | # with: 161 | # path: ${{ matrix.pip-path }} 162 | # key: ${{ matrix.os }}-python${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }} 163 | 164 | - name: Install Package Dependencies 165 | #if: steps.pip-cache.outputs.cache-hit != 'true' 166 | run: | 167 | python -m pip install --upgrade pip 168 | pip install -r requirements-dev.txt 169 | pip install -r requirements.txt 170 | 171 | - name: Get JuiceFS Version 172 | id: jfs-version 173 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 174 | 175 | - name: Download jfs binary artifact 176 | uses: actions/download-artifact@v2 177 | with: 178 | name: jfs-binary-${{ matrix.target }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 179 | path: ./juicefs/lib/ 180 | 181 | - name: Make juicefs executable 182 | run: | 183 | chmod 755 ./juicefs/lib/juicefs* 184 | ls -l ./juicefs/lib/ 185 | 186 | - name: Run style-check 187 | run: | 188 | make style_check 189 | 190 | - name: Run static-check 191 | run: | 192 | make static_check 193 | -------------------------------------------------------------------------------- /.github/workflows/release.yaml: -------------------------------------------------------------------------------- 1 | name: release wheel 2 | on: 3 | workflow_dispatch: 4 | branches: [main] 5 | 6 | jobs: 7 | build-juicefs-libjfs: 8 | runs-on: ${{ matrix.os }} 9 | strategy: 10 | matrix: 11 | os: [ubuntu-18.04, macos-latest] 12 | 13 | steps: 14 | - name: Checkout Github Repository 15 | uses: actions/checkout@v2 16 | - name: Set up Golang 17 | uses: actions/setup-go@v2 18 | - name: Get JuiceFS Version 19 | id: jfs-version 20 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 21 | 22 | - name: Cache juicefs and libjfs binary linux/mac 23 | uses: actions/cache@v2 24 | id: jfs-cache 25 | with: 26 | path: | 27 | ./juicefs/lib/libjfs.so 28 | ./juicefs/lib/juicefs 29 | key: ${{ matrix.os }}-jfs-binary-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 30 | 31 | - name: Cache juicefs and libjfs binary windows 32 | if: ${{ matrix.os == 'ubuntu-18.04' }} 33 | uses: actions/cache@v2 34 | id: jfs-cache-win 35 | with: 36 | path: | 37 | ./juicefs/lib/libjfs.dll 38 | ./juicefs/lib/juicefs.exe 39 | key: ${{ matrix.os }}-jfs-binary-win-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 40 | 41 | 42 | - name: Run build-libjfs linux/mac 43 | if: ${{ steps.jfs-cache.outputs.cache-hit != 'true' }} 44 | run: | 45 | make build_libjfs_so 46 | 47 | 48 | - name: Install MinGW GCC 49 | if: ${{ steps.jfs-cache-win.outputs.cache-hit != 'true' && matrix.os == 'ubuntu-18.04'}} 50 | run: | 51 | sudo apt-get update 52 | sudo apt-get install gcc-mingw-w64-x86-64 53 | 54 | - name: Run build-libjfs windows 55 | if: ${{ steps.jfs-cache-win.outputs.cache-hit != 'true' && matrix.os == 'ubuntu-18.04'}} 56 | run: | 57 | make build_libjfs_dll 58 | 59 | - name: Upload linux jfs binary 60 | if: matrix.os == 'ubuntu-18.04' 61 | uses: actions/upload-artifact@v2 62 | with: 63 | name: jfs-binary-linux-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 64 | path: | 65 | ./juicefs/lib/libjfs.so 66 | ./juicefs/lib/juicefs 67 | retention-days: 1 68 | - name: Upload macos jfs binary 69 | if: matrix.os == 'macos-latest' 70 | uses: actions/upload-artifact@v2 71 | with: 72 | name: jfs-binary-mac-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 73 | path: | 74 | ./juicefs/lib/libjfs.so 75 | ./juicefs/lib/juicefs 76 | retention-days: 1 77 | - name: Upload windows jfs binary 78 | if: matrix.os == 'ubuntu-18.04' 79 | uses: actions/upload-artifact@v2 80 | with: 81 | name: jfs-binary-win-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 82 | path: | 83 | ./juicefs/lib/libjfs.dll 84 | ./juicefs/lib/juicefs.exe 85 | retention-days: 1 86 | 87 | build_release: 88 | needs: build-juicefs-libjfs 89 | runs-on: ${{ matrix.os }} 90 | strategy: 91 | matrix: 92 | os: [macos-latest, ubuntu-18.04, windows-latest] 93 | python-version: [3.6] 94 | include: 95 | - os: ubuntu-18.04 96 | os-key: linux 97 | pip-path: /opt/hostedtoolcache/Python 98 | plat-name: manylinux2014_x86_64 99 | - os: macos-latest 100 | os-key: mac 101 | pip-path: /Users/runner/hostedtoolcache/Python 102 | plat-name: macosx_10_15_x86_64 103 | 104 | - os: windows-latest 105 | os-key: win 106 | pip-path: C:\hostedtoolcache\windows\Python 107 | plat-name: win_amd64 108 | 109 | 110 | steps: 111 | - name: Checkout Github Repository 112 | uses: actions/checkout@v2 113 | - name: Set up Python 114 | uses: actions/setup-python@v2 115 | with: 116 | python-version: ${{ matrix.python-version }} 117 | # - name: Cache Pip 118 | # id: pip-cache 119 | # uses: actions/cache@v2 120 | # with: 121 | # path: ${{ matrix.pip-path }} 122 | # key: ${{ matrix.os }}-python${{ matrix.python-version }}-pip-${{ hashFiles('requirements.txt') }}-${{ hashFiles('requirements-dev.txt') }} 123 | - name: Install Package Dependencies 124 | # if: steps.pip-cache.outputs.cache-hit != 'true' 125 | run: | 126 | python -m pip install --upgrade pip 127 | pip install -r requirements-dev.txt 128 | pip install -r requirements.txt 129 | - name: Get JuiceFS Version 130 | id: jfs-version 131 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 132 | 133 | - name: Download jfs binary artifact Linux/Mac/Windows 134 | uses: actions/download-artifact@v2 135 | with: 136 | name: jfs-binary-${{ matrix.os-key }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 137 | path: ./juicefs/lib/ 138 | - name: Chmod to make juicefs executable linux/mac 139 | if: ${{ matrix.os != 'windows-latest' }} 140 | run: | 141 | chmod 755 ./juicefs/lib/juicefs 142 | ls -l ./juicefs/lib/juicefs 143 | 144 | 145 | - name: Chmod to make juicefs executable windows 146 | if: ${{ matrix.os == 'windows-latest' }} 147 | run: | 148 | chmod 755 ./juicefs/lib/juicefs.exe 149 | ls -l ./juicefs/lib/juicefs.exe 150 | 151 | - name: Build wheel 152 | run: | 153 | python setup.py bdist_wheel --plat-name ${{ matrix.plat-name }} 154 | - name: Upload Wheel 155 | uses: actions/upload-artifact@v2 156 | with: 157 | name: jfs-wheel-${{ matrix.os-key }}-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 158 | path: | 159 | ./dist/* 160 | retention-days: 1 161 | 162 | publish_release: 163 | needs: build_release 164 | runs-on: ${{ matrix.os }} 165 | strategy: 166 | matrix: 167 | os: [ubuntu-18.04] 168 | 169 | steps: 170 | - name: Checkout Github Repository 171 | uses: actions/checkout@v2 172 | - name: Get JuiceFS Version 173 | id: jfs-version 174 | run: echo "::set-output name=JUICEFS_VERSION::$(make -s print_libjfs_version)" 175 | - name: Download jfs wheel artifact Linux 176 | uses: actions/download-artifact@v2 177 | with: 178 | name: jfs-wheel-linux-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 179 | path: ./dist/ 180 | - name: Download jfs wheel artifact Mac 181 | uses: actions/download-artifact@v2 182 | with: 183 | name: jfs-wheel-mac-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 184 | path: ./dist/ 185 | 186 | - name: Download jfs wheel artifact Windows 187 | uses: actions/download-artifact@v2 188 | with: 189 | name: jfs-wheel-win-${{ steps.jfs-version.outputs.JUICEFS_VERSION }} 190 | path: ./dist/ 191 | 192 | - name: Publish package to PyPI 193 | uses: pypa/gh-action-pypi-publish@release/v1 194 | with: 195 | password: ${{ secrets.JUICEFS_PYPI_TOKEN }} 196 | -------------------------------------------------------------------------------- /tests/test_io.py: -------------------------------------------------------------------------------- 1 | import io 2 | import os 3 | import sys 4 | 5 | import pytest 6 | 7 | from juicefs.io import FileIO, open 8 | 9 | CONTENT = b"block0\n block1\n block2" 10 | 11 | 12 | def remove_file(jfs, path): 13 | if jfs.path.isfile(path): 14 | jfs.unlink(path) 15 | 16 | 17 | def create_tempfile(jfs, path): 18 | remove_file(jfs, path) 19 | yield path 20 | remove_file(jfs, path) 21 | 22 | 23 | @pytest.fixture() 24 | def filename(jfs): 25 | yield from create_tempfile(jfs, "/test.file") 26 | 27 | 28 | def test_fileio_close(jfs, filename): 29 | jfs.create(filename) 30 | fd = jfs.open(filename, os.W_OK) 31 | fio = FileIO(jfs, fd) 32 | assert fio.closed is False 33 | assert repr(fio) == "" 34 | 35 | fio.close() 36 | 37 | assert fio.closed is True 38 | assert repr(fio) == '' 39 | 40 | 41 | def remove_local(path): 42 | if os.path.isfile(path): 43 | os.unlink(path) 44 | 45 | 46 | def test_fileio_del(jfs, filename): 47 | jfs.create(filename) 48 | fd = jfs.open(filename, os.W_OK) 49 | fio = FileIO(jfs, fd) 50 | assert fio.closed is False 51 | 52 | with pytest.warns(ResourceWarning): 53 | del fio 54 | 55 | 56 | @pytest.mark.filterwarnings("ignore:unclosed file") 57 | def test_pickle(jfs, filename): 58 | jfs.create(filename) 59 | fd = jfs.open(filename, os.W_OK) 60 | fio = FileIO(jfs, fd) 61 | 62 | import pickle 63 | with pytest.raises(TypeError): 64 | pickle.dumps(fio) 65 | 66 | 67 | @pytest.fixture() 68 | def local_filename(): 69 | path = "/tmp/test.file" 70 | if sys.platform == "win32": 71 | path = os.path.join(__file__, "..", "tmp", "test.file") 72 | remove_local(path) 73 | yield path 74 | remove_local(path) 75 | 76 | 77 | def test_fileio_close_wb(jfs, filename): 78 | writer = open(jfs, filename, "wb") 79 | assert writer.closed is False 80 | assert repr(writer) == "" 81 | assert writer.mode == "wb" 82 | 83 | writer.close() 84 | 85 | assert writer.closed is True 86 | 87 | 88 | def test_fileio_close_rb(jfs, filename): 89 | open(jfs, filename, "wb").close() 90 | reader = open(jfs, filename, "rb") 91 | assert reader.closed is False 92 | assert repr(reader) == "<_io.BufferedReader name='/test.file'>" 93 | assert reader.mode == "rb" 94 | 95 | reader.close() 96 | 97 | assert reader.closed is True 98 | 99 | 100 | def test_fileio_read(jfs, filename): 101 | jfs.create(filename) 102 | fd = jfs.open(filename, os.W_OK) 103 | jfs.write(fd, CONTENT) 104 | jfs.close(fd) 105 | 106 | with open(jfs, filename, "rb") as reader: 107 | assert reader.readline() == b"block0\n" 108 | assert reader.read() == b" block1\n block2" 109 | 110 | with open(jfs, filename, "rb") as reader: 111 | assert reader.read(5) == CONTENT[:5] 112 | 113 | 114 | def test_reader_unwritable(jfs, filename): 115 | jfs.create(filename) 116 | fd = jfs.open(filename, os.W_OK) 117 | jfs.close(fd) 118 | with open(jfs, filename, "rb") as reader: 119 | with pytest.raises(io.UnsupportedOperation): 120 | reader.write(b'test') 121 | 122 | 123 | def test_fileio_write(jfs, filename): 124 | with open(jfs, filename, "wb") as writer: 125 | writer.write(CONTENT) 126 | 127 | fd = jfs.open(filename, os.R_OK) 128 | content = jfs.read(fd, len(CONTENT)) 129 | assert content == CONTENT 130 | 131 | with open(jfs, filename, "rb") as reader: 132 | assert reader.readline() == b"block0\n" 133 | assert reader.read() == b" block1\n block2" 134 | 135 | with open(jfs, filename, "rb") as reader: 136 | assert reader.read(5) == CONTENT[:5] 137 | 138 | 139 | def test_writer_unreadable(jfs, filename): 140 | with open(jfs, filename, "wb") as writer: 141 | with pytest.raises(io.UnsupportedOperation): 142 | writer.read() 143 | 144 | 145 | def test_mode(jfs, filename): 146 | with open(jfs, filename, "xb") as writer: 147 | assert writer.mode == "xb" 148 | remove_file(jfs, filename) 149 | with open(jfs, filename, "xb+") as writer: 150 | assert writer.mode == "xb+" 151 | with open(jfs, filename, "ab") as writer: 152 | assert writer.mode == "ab" 153 | with open(jfs, filename, "ab+") as writer: 154 | assert writer.mode == "ab+" 155 | 156 | 157 | def test_open_pandora_box(jfs, filename): 158 | with pytest.raises(TypeError): 159 | open(jfs, 0xbad) 160 | with pytest.raises(TypeError): 161 | open(jfs, filename, mode=0xbad) 162 | with pytest.raises(TypeError): 163 | open(jfs, filename, buffering='bad') 164 | with pytest.raises(TypeError): 165 | open(jfs, filename, encoding=0xbad) 166 | with pytest.raises(TypeError): 167 | open(jfs, filename, errors=0xbad) 168 | with pytest.raises(ValueError): 169 | open(jfs, filename, mode="bad") 170 | with open(jfs, filename, "wb") as writer: 171 | pass 172 | with pytest.warns(DeprecationWarning): 173 | open(jfs, filename, mode="Urb") 174 | for mode in ["rrr", "Uw", "tb", "xrwa", "b"]: 175 | with pytest.raises(ValueError): 176 | open(jfs, filename, mode=mode) 177 | with pytest.raises(ValueError): 178 | open(jfs, filename, mode="wb", encoding="utf8") 179 | with pytest.raises(ValueError): 180 | open(jfs, filename, mode="wb", errors="ignore") 181 | with pytest.raises(ValueError): 182 | open(jfs, filename, mode="wb", newline="\r\n") 183 | 184 | 185 | def test_open_text(jfs, filename): 186 | with open(jfs, filename, mode="tw", encoding="utf8") as writer: 187 | writer.write("hello") 188 | 189 | 190 | def test_fileio_append(jfs, filename): 191 | with open(jfs, filename, "ab") as writer: 192 | writer.write(CONTENT) 193 | 194 | with open(jfs, filename, "ab") as writer: 195 | writer.write(CONTENT) 196 | 197 | with open(jfs, filename, "rb") as reader: 198 | assert reader.read() == CONTENT * 2 199 | 200 | with open(jfs, filename, "rb") as reader: 201 | assert reader.read(20) == (CONTENT * 2)[:20] 202 | 203 | 204 | def test_fileio_flush(jfs, filename): 205 | with open(jfs, filename, "wb") as writer: 206 | writer.write(CONTENT) 207 | writer.flush() 208 | with open(jfs, filename, "rb") as reader: 209 | assert reader.read() == CONTENT 210 | writer.write(CONTENT[::-1]) 211 | writer.flush() 212 | with open(jfs, filename, "rb") as reader: 213 | assert reader.read() == CONTENT + CONTENT[::-1] 214 | 215 | 216 | def test_truncate(jfs, filename): 217 | with open(jfs, filename, "wb") as writer: 218 | writer.write(b"hello") 219 | 220 | with open(jfs, filename, "wb") as writer: 221 | writer.truncate() 222 | 223 | with open(jfs, filename, "rb") as reader: 224 | assert reader.read() == b"" 225 | 226 | 227 | def write_no_assert(fp1, fp2, buffer): 228 | fp1.write(buffer) 229 | fp2.write(buffer) 230 | 231 | 232 | def test_seek_write(jfs, filename, local_filename): 233 | with open(jfs, filename, "wb") as writer: 234 | writer.write(CONTENT) 235 | 236 | with io.open(local_filename, "wb") as writer: 237 | writer.write(CONTENT) 238 | 239 | with io.open(local_filename, "wb") as fp1, open(jfs, filename, "wb") as fp2: 240 | assert_ability(fp1, fp2) 241 | write_no_assert(fp1, fp2, CONTENT) 242 | assert_seek(fp1, fp2, 0, 0) 243 | write_no_assert(fp1, fp2, CONTENT) 244 | assert_seek(fp1, fp2, 0, 1) 245 | write_no_assert(fp1, fp2, CONTENT) 246 | assert_seek(fp1, fp2, 0, 2) 247 | write_no_assert(fp1, fp2, CONTENT) 248 | 249 | with io.open(local_filename, "rb") as fp1, open(jfs, filename, "rb") as fp2: 250 | assert fp1.read() == fp2.read() 251 | 252 | 253 | def assert_ability(fp1, fp2): 254 | assert fp1.seekable() == fp2.seekable() 255 | assert fp1.readable() == fp2.readable() 256 | assert fp1.writable() == fp2.writable() 257 | 258 | 259 | def assert_read(fp1, fp2, size): 260 | assert fp1.read(size) == fp2.read(size) 261 | 262 | 263 | def assert_seek(fp1, fp2, cookie, whence): 264 | fp1.seek(cookie, whence) 265 | fp2.seek(cookie, whence) 266 | assert fp1.tell() == fp2.tell() 267 | 268 | 269 | def load_content(fp): 270 | print("call flush in load_content") 271 | fp.flush() 272 | if isinstance(fp.raw, FileIO): 273 | with open(fp.raw._jfs, fp.name, "rb") as fpr: 274 | return fpr.read() 275 | with io.open(fp.name, "rb") as fpr: 276 | return fpr.read() 277 | 278 | 279 | def assert_write(fp1, fp2, buffer): 280 | fp1.write(buffer) 281 | fp2.write(buffer) 282 | assert load_content(fp1) == load_content(fp2) 283 | 284 | 285 | def test_fileio_mode_rb(jfs, filename, local_filename): 286 | with open(jfs, filename, "wb") as writer: 287 | writer.write(CONTENT) 288 | 289 | with io.open(local_filename, "wb") as writer: 290 | writer.write(CONTENT) 291 | 292 | with io.open(local_filename, "rb") as fp1, open(jfs, filename, "rb") as fp2: 293 | assert_ability(fp1, fp2) 294 | assert_read(fp1, fp2, 5) 295 | assert_seek(fp1, fp2, 0, 0) 296 | assert_read(fp1, fp2, 5) 297 | assert_seek(fp1, fp2, 0, 1) 298 | assert_read(fp1, fp2, 5) 299 | assert_seek(fp1, fp2, 0, 2) 300 | assert_read(fp1, fp2, 5) 301 | 302 | fp2.seek(0) 303 | assert fp2.readline() == b"block0\n" 304 | assert list(fp2.readlines()) == [b" block1\n", b" block2"] 305 | 306 | with pytest.raises(IOError): 307 | fp2.write(b"") 308 | 309 | 310 | def test_fileio_mode_wb(jfs, filename, local_filename): 311 | with open(jfs, filename, "wb") as writer: 312 | writer.write(CONTENT) 313 | 314 | with io.open(local_filename, "wb") as writer: 315 | writer.write(CONTENT) 316 | 317 | with io.open(local_filename, "wb") as fp1, open(jfs, filename, "wb") as fp2: 318 | assert_ability(fp1, fp2) 319 | assert_write(fp1, fp2, CONTENT) 320 | assert_seek(fp1, fp2, 0, 0) 321 | assert_write(fp1, fp2, CONTENT) 322 | assert_seek(fp1, fp2, 0, 1) 323 | assert_write(fp1, fp2, CONTENT) 324 | assert_seek(fp1, fp2, 0, 2) 325 | assert_write(fp1, fp2, CONTENT) 326 | 327 | with pytest.raises(IOError): 328 | fp2.read() 329 | with pytest.raises(IOError): 330 | fp2.readline() 331 | with pytest.raises(IOError): 332 | fp2.readlines() 333 | 334 | 335 | def test_fileio_mode_ab(jfs, filename, local_filename): 336 | with open(jfs, filename, "wb") as writer: 337 | writer.write(CONTENT) 338 | 339 | with io.open(local_filename, "wb") as writer: 340 | writer.write(CONTENT) 341 | 342 | with io.open(local_filename, "ab") as fp1, open(jfs, filename, "ab") as fp2: 343 | assert_ability(fp1, fp2) 344 | assert_write(fp1, fp2, CONTENT) 345 | assert_seek(fp1, fp2, 0, 0) 346 | assert_write(fp1, fp2, CONTENT) 347 | assert_seek(fp1, fp2, 0, 1) 348 | assert_write(fp1, fp2, CONTENT) 349 | assert_seek(fp1, fp2, 0, 2) 350 | assert_write(fp1, fp2, CONTENT) 351 | 352 | 353 | # TODO: read-write mode not supported in this version 354 | # def test_fileio_mode_rbp(jfs, filename, local_filename): 355 | # with open(jfs, filename, "wb") as writer: 356 | # writer.write(CONTENT) 357 | 358 | # with io.open(local_filename, "wb") as writer: 359 | # writer.write(CONTENT) 360 | 361 | # with io.open(local_filename, "rb+") as fp1, open(jfs, filename, "rb+") as fp2: 362 | # assert_ability(fp1, fp2) 363 | # assert_read(fp1, fp2, 5) 364 | # assert_write(fp1, fp2, CONTENT) 365 | # assert_seek(fp1, fp2, 0, 0) 366 | # assert_read(fp1, fp2, 5) 367 | # assert_write(fp1, fp2, CONTENT) 368 | # assert_seek(fp1, fp2, 0, 1) 369 | # assert_read(fp1, fp2, 5) 370 | # assert_write(fp1, fp2, CONTENT) 371 | # assert_seek(fp1, fp2, 0, 2) 372 | # assert_read(fp1, fp2, 5) 373 | 374 | 375 | # TODO: read-write mode not supported in this version 376 | # def test_fileio_mode_wbp(jfs, filename, local_filename): 377 | # with open(jfs, filename, "wb") as writer: 378 | # writer.write(CONTENT) 379 | 380 | # with io.open(local_filename, "wb") as writer: 381 | # writer.write(CONTENT) 382 | 383 | # with io.open(local_filename, "wb+") as fp1, open(jfs, filename, "wb+") as fp2: 384 | # assert_ability(fp1, fp2) 385 | # assert_read(fp1, fp2, 5) 386 | # assert_write(fp1, fp2, CONTENT) 387 | # assert_seek(fp1, fp2, 0, 0) 388 | # assert_read(fp1, fp2, 5) 389 | # assert_write(fp1, fp2, CONTENT) 390 | # assert_seek(fp1, fp2, 0, 1) 391 | # assert_read(fp1, fp2, 5) 392 | # assert_write(fp1, fp2, CONTENT) 393 | # assert_seek(fp1, fp2, 0, 2) 394 | # assert_read(fp1, fp2, 5) 395 | 396 | 397 | # TODO: 'a' mode not implemented 398 | # def test_s3_cached_handler_mode_abp(jfs, filename, local_filename): 399 | # with open(jfs, filename, "wb") as writer: 400 | # writer.write(CONTENT) 401 | 402 | # with io.open(local_filename, "wb") as writer: 403 | # writer.write(CONTENT) 404 | 405 | # with io.open(local_filename, "ab+") as fp1, open(jfs, filename, "ab+") as fp2: 406 | # assert_ability(fp1, fp2) 407 | # assert_read(fp1, fp2, 5) 408 | # assert_write(fp1, fp2, CONTENT) 409 | # assert_seek(fp1, fp2, 0, 0) 410 | # assert_read(fp1, fp2, 5) 411 | # assert_write(fp1, fp2, CONTENT) 412 | # assert_seek(fp1, fp2, 0, 1) 413 | # assert_read(fp1, fp2, 5) 414 | # assert_write(fp1, fp2, CONTENT) 415 | # assert_seek(fp1, fp2, 0, 2) 416 | # assert_read(fp1, fp2, 5) 417 | -------------------------------------------------------------------------------- /juicefs/io.py: -------------------------------------------------------------------------------- 1 | import errno 2 | import os 3 | from io import DEFAULT_BUFFER_SIZE 4 | from io import BufferedRandom as _BufferedRandom 5 | from io import BufferedReader 6 | from io import BufferedWriter as _BufferedWriter 7 | from io import RawIOBase, TextIOWrapper, UnsupportedOperation 8 | from typing import Optional 9 | 10 | 11 | class BufferedWriter(_BufferedWriter): 12 | def flush(self): 13 | super().flush() 14 | self.raw.flush() 15 | 16 | 17 | class BufferedRandom(_BufferedRandom): 18 | def flush(self): 19 | super().flush() 20 | self.raw.flush() 21 | 22 | 23 | class FileIO(RawIOBase): 24 | _fd = -1 25 | _created = False 26 | _readable = False 27 | _writable = False 28 | _appending = False 29 | _seekable = None 30 | 31 | def __init__(self, jfs, fd: int): 32 | path, flags, jfs_flags = jfs._fetch_path_by_fd(fd) 33 | self._jfs = jfs 34 | self._fd = fd 35 | self.name = path 36 | self._flags = flags 37 | self._created = flags & os.O_EXCL != 0 38 | self._readable = jfs_flags & os.R_OK != 0 39 | self._writable = jfs_flags & os.W_OK != 0 40 | self._appending = flags & os.O_APPEND != 0 41 | 42 | def __del__(self): 43 | if self._fd >= 0 and not self.closed: 44 | import warnings 45 | 46 | warnings.warn("unclosed file %r" % self, ResourceWarning, stacklevel=2) 47 | self.close() 48 | 49 | def __getstate__(self): 50 | raise TypeError("cannot pickle '{}' object".format(self.__class__.__name__)) 51 | 52 | def __repr__(self) -> str: 53 | class_name = "%s.%s" % (self.__class__.__module__, self.__class__.__qualname__) 54 | if self.closed: 55 | return "<%s [closed]>" % class_name 56 | try: 57 | name = self.name 58 | except AttributeError: 59 | return "<%s fd=%d mode=%r>" % (class_name, self._fd, self.mode) 60 | else: 61 | return "<%s name=%r mode=%r>" % (class_name, name, self.mode) 62 | 63 | def _checkReadable(self): 64 | if not self._readable: 65 | raise UnsupportedOperation("File not open for reading") 66 | 67 | def _checkWritable(self, msg=None): 68 | if not self._writable: 69 | raise UnsupportedOperation("File not open for writing") 70 | 71 | def read(self, size: Optional[int] = None) -> Optional[bytes]: 72 | """Read at most *size* bytes, returned as bytes. 73 | Only makes one system call, so less data may be returned than requested 74 | In non-blocking mode, returns None if no data is available. 75 | Return an empty bytes object at EOF. 76 | """ 77 | self._checkClosed() 78 | self._checkReadable() 79 | if size is None or size < 0: 80 | return self.readall() 81 | try: 82 | return self._jfs.read(self._fd, size) 83 | except BlockingIOError: 84 | return None 85 | 86 | def readall(self) -> Optional[bytes]: 87 | """Read all data from the file, returned as bytes. 88 | In non-blocking mode, returns as much as is immediately available, 89 | or None if no data is available. Return an empty bytes object at EOF. 90 | """ 91 | self._checkClosed() 92 | self._checkReadable() 93 | bufsize = DEFAULT_BUFFER_SIZE 94 | try: 95 | pos = self._jfs.lseek(self._fd, 0, os.SEEK_CUR) 96 | end = self._jfs.fstat(self._fd).st_size 97 | if end >= pos: 98 | bufsize = end - pos + 1 99 | except OSError: 100 | pass 101 | 102 | result = bytearray() 103 | while True: 104 | if len(result) >= bufsize: 105 | bufsize = len(result) 106 | bufsize += max(bufsize, DEFAULT_BUFFER_SIZE) 107 | n = bufsize - len(result) 108 | try: 109 | chunk = self._jfs.read(self._fd, n) 110 | except BlockingIOError: 111 | if result: 112 | break 113 | return None 114 | if not chunk: # reached the end of the file 115 | break 116 | result += chunk 117 | 118 | return bytes(result) 119 | 120 | def readinto(self, b): 121 | """Same as RawIOBase.readinto().""" 122 | m = memoryview(b).cast("B") # pytype: disable=attribute-error 123 | data = self.read(len(m)) 124 | n = len(data) 125 | m[:n] = data 126 | return n 127 | 128 | def write(self, b: bytes) -> Optional[int]: 129 | """Write bytes *b* to file, return number written. 130 | Only makes one system call, so not all of the data may be written. 131 | The number of bytes actually written is returned. In non-blocking mode, 132 | returns None if the write would block. 133 | """ 134 | self._checkClosed() 135 | self._checkWritable() 136 | try: 137 | if isinstance(b, memoryview): 138 | b = bytes(b) 139 | if self._appending: 140 | self._jfs.lseek(self._fd, 0, os.SEEK_END) 141 | return self._jfs.write(self._fd, b) 142 | except BlockingIOError: 143 | return None 144 | 145 | def seek(self, pos: int, whence: int = os.SEEK_SET) -> int: 146 | """Move to new file position. 147 | Argument *offset* is a byte count. Optional argument *whence* defaults to 148 | SEEK_SET or 0 (offset from start of file, offset should be >= 0); other values 149 | are SEEK_CUR or 1 (move relative to current position, positive or negative), 150 | and SEEK_END or 2 (move relative to end of file, usually negative, although 151 | many platforms allow seeking beyond the end of a file). 152 | Note that not all file objects are seekable. 153 | """ 154 | if isinstance(pos, float): 155 | raise TypeError("an integer is required") 156 | self._checkClosed() 157 | return self._jfs.lseek(self._fd, pos, whence) 158 | 159 | def tell(self) -> int: 160 | """tell() -> int. Current file position. 161 | Can raise OSError for non seekable files.""" 162 | self._checkClosed() 163 | return self._jfs.lseek(self._fd, 0, os.SEEK_CUR) 164 | 165 | def truncate(self, size: Optional[int] = None) -> int: 166 | """Truncate the file to at most *size* bytes. 167 | *size* defaults to the current file position, as returned by tell(). 168 | The current file position is changed to the value of size. 169 | """ 170 | self._checkClosed() 171 | self._checkWritable() 172 | if size is None: 173 | size = self.tell() 174 | self._jfs.ftruncate(self._fd, size) 175 | return size 176 | 177 | def flush(self): 178 | """Flush write buffers, if applicable. 179 | This is not implemented for read-only and non-blocking streams. 180 | """ 181 | self._checkClosed() 182 | self._jfs.flush(self._fd) 183 | 184 | def close(self): 185 | """Close the file. 186 | A closed file cannot be used for further I/O operations. close() may be 187 | called more than once without error. 188 | """ 189 | if not self.closed: 190 | try: 191 | super().close() 192 | finally: 193 | self._jfs.close(self._fd) 194 | 195 | def seekable(self) -> bool: 196 | """True if file supports random-access.""" 197 | self._checkClosed() 198 | if self._seekable is None: 199 | try: 200 | self.tell() 201 | except OSError: 202 | self._seekable = False 203 | else: 204 | self._seekable = True 205 | return self._seekable 206 | 207 | def readable(self) -> bool: 208 | """True if file was opened in a read mode.""" 209 | self._checkClosed() 210 | return self._readable 211 | 212 | def writable(self) -> bool: 213 | """True if file was opened in a write mode.""" 214 | self._checkClosed() 215 | return self._writable 216 | 217 | def fileno(self) -> int: 218 | """Return the underlying file descriptor (an integer).""" 219 | self._checkClosed() 220 | return self._fd 221 | 222 | def isatty(self) -> bool: 223 | """True if the file is connected to a TTY device.""" 224 | self._checkClosed() 225 | return False 226 | 227 | @property 228 | def mode(self) -> str: 229 | """String giving the file mode""" 230 | if self._created: 231 | if self._readable: 232 | return "xb+" 233 | else: 234 | return "xb" 235 | elif self._appending: 236 | if self._readable: 237 | return "ab+" 238 | else: 239 | return "ab" 240 | elif self._readable: 241 | if self._writable: 242 | return "rb+" 243 | else: 244 | return "rb" 245 | else: 246 | return "wb" 247 | 248 | 249 | def open( 250 | jfs, 251 | path: str, 252 | mode: str = "r", 253 | buffering: int = -1, 254 | encoding: Optional[str] = None, 255 | errors: Optional[str] = None, 256 | newline: Optional[str] = None, 257 | ): 258 | """Open a juicefs file. 259 | 260 | The *mode* can be 'r' (default), 'w', 'x' or 'a' for reading, writing, 261 | exclusive creation or appending. 262 | 263 | The file will be created if it doesn't exist when opened for writing or 264 | appending; it will be truncated when opened for writing. 265 | 266 | A FileExistsError will be raised if it already exists when opened for 267 | creating. Opening a file for creating implies writing so this mode behaves 268 | in a similar way to 'w'. Add a '+' to the mode to allow simultaneous 269 | reading and writing. 270 | """ 271 | 272 | if not isinstance(path, str): 273 | raise TypeError("invalid path: %r" % path) 274 | if not isinstance(mode, str): 275 | raise TypeError("invalid mode: %r" % mode) 276 | if not isinstance(buffering, int): 277 | raise TypeError("invalid buffering: %r" % buffering) 278 | if encoding is not None and not isinstance(encoding, str): 279 | raise TypeError("invalid encoding: %r" % encoding) 280 | if errors is not None and not isinstance(errors, str): 281 | raise TypeError("invalid errors: %r" % errors) 282 | modes = set(mode) 283 | if modes - set("axrwb+tU") or len(mode) > len(modes): 284 | raise ValueError("invalid mode: %r" % mode) 285 | creating = "x" in modes 286 | reading = "r" in modes 287 | writing = "w" in modes 288 | appending = "a" in modes 289 | updating = "+" in modes 290 | text = "t" in modes 291 | binary = "b" in modes 292 | 293 | if "U" in modes: 294 | if creating or writing or appending or updating: 295 | raise ValueError("mode U cannot be combined with 'x', 'w', 'a', or '+'") 296 | import warnings 297 | 298 | warnings.warn("'U' mode is deprecated", DeprecationWarning, 2) 299 | reading = True 300 | if text and binary: 301 | raise ValueError("can't have text and binary mode at once") 302 | if creating + reading + writing + appending > 1: 303 | raise ValueError("can't have read/write/append mode at once") 304 | if not (creating or reading or writing or appending): 305 | raise ValueError("must have exactly one of read/write/append mode") 306 | if binary and encoding is not None: 307 | raise ValueError("binary mode doesn't take an encoding argument") 308 | if binary and errors is not None: 309 | raise ValueError("binary mode doesn't take an errors argument") 310 | if binary and newline is not None: 311 | raise ValueError("binary mode doesn't take a newline argument") 312 | 313 | readable = False 314 | writable = False 315 | if creating: 316 | writable = True 317 | flags = os.O_EXCL | os.O_CREAT 318 | elif reading: 319 | readable = True 320 | flags = 0 321 | elif writing: 322 | writable = True 323 | flags = os.O_CREAT | os.O_TRUNC 324 | elif appending: 325 | writable = True 326 | flags = os.O_APPEND | os.O_CREAT 327 | 328 | if updating: 329 | readable = True 330 | writable = True 331 | 332 | if readable and writable: 333 | flags |= os.O_RDWR 334 | elif readable: 335 | flags |= os.O_RDONLY 336 | else: 337 | flags |= os.O_WRONLY 338 | 339 | flags |= getattr(os, "O_BINARY", 0) 340 | 341 | fd = None 342 | try: 343 | fd = jfs.open(path, flags) 344 | 345 | if appending: 346 | # For consistent behaviour, we explicitly seek to the 347 | # end of file (otherwise, it might be done only on the 348 | # first write()). 349 | try: 350 | jfs.lseek(fd, 0, os.SEEK_END) 351 | except OSError as e: 352 | if e.errno != errno.ESPIPE: 353 | raise 354 | except: 355 | if fd is not None: 356 | jfs.close(fd) 357 | raise 358 | 359 | raw = FileIO(jfs, fd) 360 | result = raw 361 | try: 362 | line_buffering = False 363 | if buffering == 1 or buffering < 0 and raw.isatty(): 364 | buffering = -1 365 | line_buffering = True 366 | if buffering < 0: 367 | buffering = DEFAULT_BUFFER_SIZE 368 | try: 369 | bs = os.fstat(raw.fileno()).st_blksize 370 | except (OSError, AttributeError): 371 | pass 372 | else: 373 | if bs > 1: 374 | buffering = bs 375 | if buffering < 0: 376 | raise ValueError("invalid buffering size") 377 | if buffering == 0: 378 | if binary: 379 | return result 380 | raise ValueError("can't have unbuffered text I/O") 381 | if updating: 382 | buffer = BufferedRandom(raw, buffering) 383 | elif creating or writing or appending: 384 | buffer = BufferedWriter(raw, buffering) 385 | elif reading: 386 | buffer = BufferedReader(raw, buffering) 387 | else: 388 | raise ValueError("unknown mode: %r" % mode) 389 | result = buffer 390 | if binary: 391 | return result 392 | text = TextIOWrapper(buffer, encoding, errors, newline, line_buffering) 393 | result = text 394 | text.mode = mode 395 | return result 396 | except: 397 | result.close() 398 | raise 399 | -------------------------------------------------------------------------------- /juicefs/juicefs.py: -------------------------------------------------------------------------------- 1 | import errno 2 | import os 3 | import posixpath 4 | import stat 5 | import struct 6 | import sys 7 | import time 8 | from ctypes import create_string_buffer 9 | from io import BytesIO 10 | from pathlib import Path 11 | from typing import Dict, Iterator, List, Optional, Tuple, Union 12 | 13 | from juicefs.io import FileIO 14 | from juicefs.libjfs import ( 15 | DirEntry, 16 | LibJuiceFS, 17 | create_stat_result, 18 | create_statvfs_result, 19 | create_summary, 20 | parse_xattrs, 21 | ) 22 | from juicefs.utils import check_juicefs_error, create_os_error 23 | 24 | DEFAULT_FILE_MODE = 0o777 25 | DEFAULT_DIRECOTRY_MODE = 0o777 26 | DEFAULT_CONFIG = { 27 | "accessLog": "", 28 | "autoCreate": True, 29 | "cacheDir": "memory", 30 | "cacheFullBlock": True, 31 | "cacheSize": 100, 32 | "debug": True, 33 | "fastResolve": True, 34 | "freeSpace": "0.1", 35 | "getTimeout": 5, 36 | "maxUploads": 20, 37 | "memorySize": 300, 38 | "meta": "", 39 | "noUsageReport": True, 40 | "opencache": False, 41 | # "openCache": 0.0, # v0.15.0 变成了 float 42 | "prefetch": 1, 43 | "pushAuth": "", 44 | "pushGateway": "", 45 | "pushInterval": 10, 46 | "putTimeout": 60, 47 | "readahead": 0, 48 | "readOnly": False, 49 | "uploadLimit": 0, 50 | "writeback": False, 51 | } 52 | 53 | 54 | def juicefs_stat(stat_func, path): 55 | buf = create_string_buffer(130) 56 | return buf, stat_func(path, buf) 57 | 58 | 59 | def juicefs_stat_result(stat_func, path): 60 | buf, code = juicefs_stat(stat_func[1], path) 61 | return create_stat_result(buf.raw, code) 62 | 63 | 64 | def juicefs_exist(stat_func, path, type_func=None): 65 | buf, code = juicefs_stat(stat_func, path) 66 | if code < 0: 67 | return False 68 | if type_func is None: 69 | return True 70 | st = create_stat_result(buf.raw, code) 71 | return type_func(st.st_mode) 72 | 73 | 74 | class JuiceFS: 75 | def __init__( 76 | self, 77 | name: str, 78 | config: Dict[str, Union[str, bool, int, float]] = {}, 79 | ): 80 | """JuiceFS Session 81 | 82 | :param str name: Redis URI used for this session 83 | :param dict config: JuiceFS configuration, available keys: https://github.com/juicedata/juicefs/blob/main/sdk/java/libjfs/main.go#L215 84 | """ 85 | jfs_config = DEFAULT_CONFIG.copy() 86 | jfs_config.update(config) 87 | 88 | path = os.path.normpath(os.path.join(__file__, "..", "lib", "libjfs.so")) 89 | if sys.platform == "win32": 90 | path = os.path.normpath(os.path.join(__file__, "..", "lib", "libjfs.dll")) 91 | 92 | self._lib = LibJuiceFS(path, name, jfs_config) 93 | self._name = name 94 | self._config = config 95 | self._fds = {} # type: Dict[int, Tuple[str, int, int]] 96 | self.path = JuiceFSPath(self._lib) 97 | 98 | # TODO: remove this 99 | # https://github.com/juicedata/juicefs/issues/646 100 | self._flens = {} # type: Dict[int, int] 101 | 102 | def _fetch_path_by_fd( 103 | self, fd: int 104 | ) -> Tuple[str, int, int]: # (name, py_flags, jfs_flags) 105 | if fd not in self._fds: 106 | raise create_os_error(errno.EBADF) 107 | return self._fds[fd] 108 | 109 | def statvfs(self) -> os.statvfs_result: 110 | """Return an os.statvfs_result object. 111 | 112 | The return value is an object whose attributes describe the filesystem 113 | on the given path, and correspond to the members of the statvfs structure, 114 | namely: 115 | 116 | f_bsize, f_frsize, f_blocks, f_bfree, f_bavail, f_files, f_ffree, f_favail, f_flag, f_namemax. 117 | """ 118 | buf = create_string_buffer(16) 119 | self._lib.statvfs(buf) 120 | return create_statvfs_result(buf.raw) 121 | 122 | def summary(self, path: str): 123 | """Returns a juicefs.libjfs.DirSummary object, which contains number of size, 124 | files and dirs on the path. 125 | """ 126 | buf = create_string_buffer(24) 127 | self._lib.summary[1](path, buf) 128 | return create_summary(buf.raw) 129 | 130 | def mkdir(self, path: str, mode: int = DEFAULT_DIRECOTRY_MODE): 131 | """Create a directory named *path* with numeric mode *mode*. 132 | 133 | If the directory already exists, FileExistsError is raised. 134 | """ 135 | self._lib.mkdir[1](path, mode) 136 | 137 | def makedirs( 138 | self, path: str, mode: int = DEFAULT_DIRECOTRY_MODE, exist_ok: bool = False 139 | ): 140 | """Recursive directory creation function. Like mkdir(), but makes all 141 | intermediate-level directories needed to contain the leaf directory. 142 | 143 | The *mode* parameter is passed to mkdir(); see the mkdir() description for 144 | how it is interpreted. 145 | If *exist_ok* is False (the default), an OSError is raised if the target 146 | directory already exists. 147 | """ 148 | code = self._lib.mkdir(path, mode) 149 | if code == -errno.EEXIST and exist_ok: 150 | return 0 151 | if code == -errno.ENOENT: 152 | self.makedirs(Path(os.path.dirname(path)).as_posix(), mode, exist_ok) 153 | code = self._lib.mkdir(path, mode) 154 | check_juicefs_error(code) 155 | 156 | def rmdir(self, path: str): 157 | """Remove (delete) the directory *path*. 158 | 159 | Only works when the directory is empty, otherwise, OSError is raised. 160 | In order to remove whole directory trees, rmtree() can be used. 161 | """ 162 | if not self.path.isdir(path): 163 | raise create_os_error(errno.ENOTDIR, path) 164 | self._lib.delete[1](path) 165 | 166 | def removedirs(self, path: str): 167 | """Remove directories recursively. 168 | 169 | Works like rmdir() except that, if the leaf directory is successfully 170 | removed, removedirs() tries to successively remove every parent directory 171 | mentioned in *path* until an error is raised (which is ignored, because it 172 | generally means that a parent directory is not empty). 173 | 174 | For example, ``JuiceFS.removedirs('/foo/bar/baz')`` will first remove the 175 | directory '/foo/bar/baz', and then remove '/foo/bar' and '/foo' if they 176 | are empty. 177 | 178 | Raises OSError if the leaf directory could not be successfully removed. 179 | """ 180 | self.rmdir(path) 181 | head, tail = posixpath.split(path) 182 | if not tail: 183 | head, tail = posixpath.split(head) 184 | while head and tail: 185 | try: 186 | self.rmdir(head) 187 | except OSError: 188 | break 189 | head, tail = posixpath.split(head) 190 | 191 | def scandir(self, path: str) -> Iterator[DirEntry]: 192 | """Return an iterator of juicefs.libjfs.DirEntry objects corresponding to 193 | the entries in the directory given by *path*. 194 | 195 | The entries are yielded in arbitrary order, and the special entries '.' and 196 | '..' are not included. 197 | """ 198 | bufsize = 32 << 10 199 | buf = create_string_buffer(bufsize) 200 | code = self._lib.listdir(path, 0, buf, bufsize) 201 | while code > 0: 202 | buffer = BytesIO(buf.raw) 203 | while buffer.tell() < code: 204 | name = buffer.read(ord(buffer.read(1))).decode() 205 | length = ord(buffer.read(1)) 206 | stat = create_stat_result(buffer.read(length), length) 207 | yield DirEntry(name, path, stat) 208 | left = struct.unpack(" List[str]: 216 | """Return a list containing the names of the entries in the directory given 217 | by *path*. 218 | 219 | The list is in arbitrary order, and does not include the special entries '.' 220 | and '..' even if they are present in the directory. 221 | """ 222 | return list(entry.name for entry in self.scandir(path)) 223 | 224 | def symlink(self, src_path: str, dst_path: str): 225 | """Create a symbolic link pointing to *src_path* named *dst_path*.""" 226 | self._lib.symlink[2](src_path, dst_path) 227 | 228 | def readlink(self, path: str) -> str: 229 | """Return a string representing the path to which the symbolic link points. 230 | 231 | The result will be a relative pathname to the path of symbolic link; 232 | it may be converted to an absolute pathname using ``os.path.join(os.path.dirname(path), result)``. 233 | """ 234 | bufsize = 4096 235 | buf = create_string_buffer(bufsize) 236 | size = self._lib.readlink[1](path, buf, bufsize) 237 | return buf.raw[:size].decode() 238 | 239 | def open(self, path: str, flags: int, mode: int = DEFAULT_FILE_MODE) -> int: # fd 240 | """Open the file *path* and set various flags according to *flags* and 241 | possibly its mode according to *mode*. 242 | 243 | When computing mode, the current umask value is first masked out. 244 | 245 | Return the file descriptor for the newly opened file. 246 | """ 247 | buf, code = juicefs_stat(self._lib.stat1, path) 248 | if code > 0: # self.path.exists 249 | st = create_stat_result(buf.raw, code) 250 | if stat.S_ISDIR(st.st_mode): # self.path.isdir 251 | raise create_os_error(errno.EISDIR, path) 252 | if flags & os.O_EXCL: 253 | raise create_os_error(errno.EEXIST, path) 254 | if flags & os.O_TRUNC: 255 | self.truncate(path, 0) 256 | else: 257 | if flags & os.O_CREAT: 258 | self.create(path, mode) 259 | 260 | jfs_flags = os.R_OK 261 | if flags & os.O_WRONLY: 262 | jfs_flags = os.W_OK 263 | elif flags & os.O_RDWR: 264 | jfs_flags = os.R_OK | os.W_OK 265 | 266 | fd = self._lib.open[1](path, jfs_flags) 267 | self._fds[fd] = (path, flags, jfs_flags) 268 | self._flens[fd] = self.path.getsize(path) 269 | return fd 270 | 271 | def close(self, fd: int): 272 | """Close file descriptor *fd*.""" 273 | self._lib[fd].close[0]() 274 | 275 | def flush(self, fd: int): 276 | """Flush the write buffers of the stream if applicable. This does nothing 277 | for read-only and non-blocking streams. 278 | """ 279 | self._lib[fd].flush[0]() 280 | 281 | def fsync(self, fd: int): 282 | """Force write of *fd* to disk.""" 283 | self._lib[fd].fsync[0]() 284 | 285 | def lseek(self, fd: int, offset: int, whence: int) -> int: 286 | """Set the current position of file descriptor *fd* to position pos, 287 | modified by *whence*: 288 | 289 | os.SEEK_SET or 0 to set the position relative to the beginning of the file; 290 | os.SEEK_CUR or 1 to set it relative to the current position; 291 | os.SEEK_END or 2 to set it relative to the end of the file. 292 | 293 | Return the new cursor position in bytes, starting from the beginning. 294 | """ 295 | if whence == os.SEEK_END: 296 | whence = os.SEEK_SET 297 | offset += self._flens[fd] 298 | return self._lib[fd].lseek[0](offset, whence) 299 | 300 | def read(self, fd: int, size: int) -> bytes: 301 | """Read at most *size* bytes from file descriptor *fd*. 302 | 303 | Return a bytestring containing the bytes read. 304 | 305 | If the end of the file referred to by fd has been reached, an empty bytes object is returned. 306 | """ 307 | buf = create_string_buffer(size) 308 | size = self._lib[fd].read[0](buf, size) 309 | return buf.raw[:size] 310 | 311 | def pread(self, fd: int, size: int, offset: int) -> bytes: 312 | """Read from a file descriptor, *fd*, at a position of *offset*. 313 | 314 | It will read up to *size* number of bytes. The file offset remains unchanged. 315 | """ 316 | buf = create_string_buffer(size) 317 | size = self._lib[fd].pread[0](buf, size, offset) 318 | return buf.raw[:size] 319 | 320 | def write(self, fd: int, content: bytes) -> int: 321 | """Write the *content* to file descriptor *fd*. 322 | 323 | Return the number of bytes actually written. 324 | """ 325 | buf = create_string_buffer(content) 326 | bufsize = len(content) 327 | length = min(self._flens[fd], self.lseek(fd, 0, os.SEEK_CUR)) 328 | code = self._lib[fd].write[0](buf, bufsize) 329 | self._flens[fd] = max(self._flens[fd], length + code) 330 | return code 331 | 332 | def create(self, path: str, mode: int = DEFAULT_FILE_MODE): 333 | """Create a file with with numeric mode *mode*.""" 334 | self._lib.create[1](path, mode) 335 | 336 | def remove(self, path: str): 337 | """Remove (delete) the file path. If *path* is a directory, OSError is 338 | raised. Use rmdir() to remove directories. 339 | 340 | This function is semantically identical to unlink(). 341 | """ 342 | if self.path.isdir(path): 343 | raise create_os_error(errno.EISDIR, path) 344 | self._lib.delete[1](path) 345 | 346 | unlink = remove 347 | 348 | def rename(self, src_path: str, dst_path: str): 349 | """Rename the file or directory *src_path* to *dst_path*. 350 | 351 | If *dst_path* is a directory or file, OSError will be raised. 352 | """ 353 | self._lib.rename[2](src_path, dst_path) 354 | 355 | replace = rename 356 | 357 | def truncate(self, path: str, length: int): 358 | """Truncate the file corresponding to *path*, so that it is at most *length* 359 | bytes in size. 360 | """ 361 | self._lib.truncate[1](path, length) 362 | 363 | def concat(self, path: str, *other_paths: str): 364 | """Concat the file content of *other_paths* with *path*'s.""" 365 | content = b"" 366 | for other_path in other_paths: 367 | if not self.path.exists(other_path): 368 | raise create_os_error(errno.ENOENT, other_path) 369 | content += other_path.encode() + b"\x00" 370 | buf = create_string_buffer(content) 371 | bufsize = len(content) 372 | self._lib.concat[1](path, buf, bufsize) 373 | 374 | def delete(self, path: str): 375 | """Delete a file, symbolic link or empty directory.""" 376 | # 删除一个文件，一个 symlink，或者空目录 377 | self._lib.delete[1](path) 378 | 379 | def rmtree(self, path: str): 380 | """Delete a file, symbolic link or recursively delete a directory.""" 381 | # 删除一个文件，一个 symlink，或者递归删除目录 382 | self._lib.rmr[1](path) 383 | 384 | def access(self, path: str, flags: int) -> bool: 385 | """Use the real uid/gid to test for access to *path*. 386 | 387 | The *flags* can be the inclusive OR of one or more of R_OK, W_OK, and X_OK 388 | to test permissions. 389 | 390 | Return True if access is allowed, False if not. 391 | """ 392 | return self._lib.access(path, flags) == 0 393 | 394 | def lstat(self, path: str) -> os.stat_result: 395 | """Similar to stat(), but does not follow symbolic links. 396 | 397 | Return an os.stat_result object. 398 | """ 399 | return juicefs_stat_result(self._lib.lstat1, path) 400 | 401 | def stat(self, path: str) -> os.stat_result: 402 | """Get the status of a file by *path*. 403 | 404 | This function normally follows symlinks. 405 | 406 | Return an os.stat_result object. 407 | """ 408 | return juicefs_stat_result(self._lib.stat1, path) 409 | 410 | def chmod(self, path: str, mode: int): 411 | """Change the mode of *path* to the numeric mode. 412 | 413 | *mode* may take one of values defined in the stat module or bitwise ORed 414 | combinations of them. 415 | """ 416 | self._lib.chmod[1](path, mode) 417 | 418 | def chown(self, path: str, user: str, group: str): 419 | """Change the user and group name of path to the user and group name.""" 420 | self._lib.setOwner[1](path, user, group) 421 | 422 | def utime(self, path: str, times: Optional[Tuple[float, float]] = None): 423 | """Set the access and modified times of the file specified by path. 424 | 425 | *times* should be a tuple like (atime, mtime) or None. 426 | if *times* is None, atime = mtime = time.time() 427 | """ 428 | if times is None: 429 | atime = mtime = time.time() 430 | else: 431 | atime, mtime = times 432 | 433 | atime, mtime = int(atime * 1000), int(mtime * 1000) 434 | self._lib.utime[1](path, mtime, atime) 435 | 436 | def getxattr(self, path: str, attribute: str) -> Optional[bytes]: 437 | """Return the value of the extended filesystem attribute attribute for *path*. 438 | 439 | Return a bytes object. 440 | """ 441 | bufsize = 16 << 10 442 | while True: 443 | bufsize *= 2 444 | buf = create_string_buffer(bufsize) 445 | code = self._lib.getXattr[2](path, attribute, buf, bufsize) 446 | if code != bufsize: 447 | break 448 | if code == -errno.EPROTONOSUPPORT or code == -errno.ENODATA: 449 | return None # attr not found 450 | return buf.raw[:code] 451 | 452 | def setxattr(self, path: str, attribute: str, value: bytes, flags: int = 0): 453 | """Set the extended filesystem attribute *attribute* on *path* to *value*. 454 | 455 | *attribute* must be a *str* encoded with the filesystem encoding. 456 | *flags* may be XATTR_REPLACE or XATTR_CREATE. 457 | 458 | If XATTR_REPLACE is given and the *attribute* does not exist, EEXISTS will be raised. 459 | If XATTR_CREATE is given and the *attribute* already exists, the *attribute* will not be created and ENODATA will be raised. 460 | """ 461 | buf = create_string_buffer(value) 462 | bufsize = len(value) 463 | self._lib.setXattr[2](path, attribute, buf, bufsize, flags) 464 | 465 | def removexattr(self, path: str, attribute: str): 466 | """Removes the extended filesystem attribute *attribute* from *path*.""" 467 | self._lib.removeXattr[2](path, attribute) 468 | 469 | def listxattr(self, path: str) -> List[str]: 470 | """Return a list of the extended filesystem attributes on *path*.""" 471 | bufsize = 1024 472 | res = [] 473 | while True: 474 | bufsize *= 2 475 | buf = create_string_buffer(bufsize) 476 | code = self._lib.listXattr[1](path, buf, bufsize) 477 | res.extend(parse_xattrs(buf.raw, code)) 478 | if code != bufsize: 479 | break 480 | return res 481 | 482 | def walk(self, top, topdown: bool = True): 483 | """Directory tree generator. 484 | 485 | For each directory in the directory tree rooted at top (including top 486 | itself, but excluding '.' and '..'), yields a 3-tuple (dirpath, dirnames, filenames) 487 | 488 | dirpath is a string, the path to the directory. dirnames is a list of 489 | the names of the subdirectories in dirpath (excluding '.' and '..'). 490 | filenames is a list of the names of the non-directory files in dirpath. 491 | Note that the names in the lists are just names, with no path components. 492 | To get a full path (which begins with top) to a file or directory in 493 | dirpath, do os.path.join(dirpath, name). 494 | 495 | If optional arg 'topdown' is true or not specified, the triple for a 496 | directory is generated before the triples for any of its subdirectories 497 | (directories are generated top down). If topdown is false, the triple 498 | for a directory is generated after the triples for all of its 499 | subdirectories (directories are generated bottom up). 500 | 501 | When topdown is true, the caller can modify the dirnames list in-place 502 | (e.g., via del or slice assignment), and walk will only recurse into the 503 | subdirectories whose names remain in dirnames; this can be used to prune the 504 | search, or to impose a specific order of visiting. Modifying dirnames when 505 | topdown is false is ineffective, since the directories in dirnames have 506 | already been generated by the time dirnames itself is generated. No matter 507 | the value of topdown, the list of subdirectories is retrieved before the 508 | tuples for the directory and its subdirectories are generated. 509 | 510 | Caution: if you pass a relative pathname for top, don't change the 511 | current working directory between resumptions of walk. walk never 512 | changes the current directory, and assumes that the client doesn't 513 | either. 514 | 515 | Example: 516 | :: 517 | 518 | from juicefs import JuiceFS 519 | jfs = JuiceFS("test") 520 | for root, dirs, files in jfs.walk('/python/Lib/email'): 521 | # do somthing 522 | pass 523 | 524 | """ 525 | dirs = [] 526 | files = [] 527 | walk_dirs = [] 528 | 529 | # We may not have read permission for top, in which case we can't 530 | # get a list of the files the directory contains. os.walk 531 | # always suppressed the exception then, rather than blow up for a 532 | # minor reason when (say) a thousand readable directories are still 533 | # left to visit. That logic is copied here. 534 | for entry in self.scandir(top): 535 | is_dir = entry.is_dir() 536 | 537 | if is_dir: 538 | dirs.append(entry.name) 539 | else: 540 | files.append(entry.name) 541 | 542 | if not topdown and is_dir: 543 | walk_dirs.append(entry.path) 544 | 545 | # Yield before recursion if going top down 546 | if topdown: 547 | yield top, dirs, files 548 | # Recurse into sub-directories 549 | for dirname in dirs: 550 | new_path = Path(os.path.join(top, dirname)).as_posix() 551 | if not self.path.islink(new_path): 552 | yield from self.walk(new_path, topdown) 553 | else: 554 | # Recurse into sub-directories 555 | for new_path in walk_dirs: 556 | yield from self.walk(new_path, topdown) 557 | # Yield after recursion if going bottom up 558 | yield top, dirs, files 559 | 560 | def fstat(self, fd: int) -> os.stat_result: 561 | """Get the status of the file descriptor *fd*. 562 | 563 | Return a stat_result object. 564 | """ 565 | path = self._fetch_path_by_fd(fd)[0] 566 | return self.stat(path) 567 | 568 | def ftruncate(self, fd: int, length: int): 569 | """Truncate the file corresponding to file descriptor *fd*, so that it is at 570 | most *length* bytes in size. 571 | """ 572 | path = self._fetch_path_by_fd(fd)[0] 573 | self._lib.truncate[1](path, length) 574 | self._flens[fd] = min(self._flens[fd], length) 575 | 576 | def fdopen(self, fd: int) -> FileIO: 577 | """Return an open file object connected to the file descriptor *fd*. 578 | 579 | This is an alias of the juicefs.io.open() function and accepts the same arguments. 580 | The only difference is that the first argument of fdopen() must always be an integer. 581 | """ 582 | return FileIO(self, fd) 583 | 584 | 585 | class JuiceFSPath: 586 | def __init__(self, lib: LibJuiceFS): 587 | self._lib = lib 588 | 589 | def lexists(self, path: str) -> bool: 590 | """Return True if *path* refers to an existing path. 591 | 592 | Returns True for broken symbolic links. 593 | """ 594 | return juicefs_exist(self._lib.lstat1, path) 595 | 596 | def exists(self, path: str) -> bool: 597 | """Return True if *path* refers to an existing path or an open file 598 | descriptor. 599 | 600 | Returns False for broken symbolic links. 601 | """ 602 | return juicefs_exist(self._lib.stat1, path) 603 | 604 | def isdir(self, path: str) -> bool: 605 | """Return True if *path* is an existing directory. 606 | 607 | This follows symbolic links, so both islink() and isdir() can be true for the same path. 608 | """ 609 | return juicefs_exist(self._lib.stat1, path, stat.S_ISDIR) 610 | 611 | def isfile(self, path: str) -> bool: 612 | """Return True if *path* is an existing regular file. 613 | 614 | This follows symbolic links, so both islink() and isfile() can be true for the same path. 615 | """ 616 | return juicefs_exist(self._lib.stat1, path, stat.S_ISREG) 617 | 618 | def islink(self, path: str) -> bool: 619 | """Return True if *path* refers to an existing directory entry that is a 620 | symbolic link. 621 | """ 622 | return juicefs_exist(self._lib.lstat1, path, stat.S_ISLNK) 623 | 624 | def getatime(self, path) -> float: 625 | """Return the time of last access of *path*. 626 | 627 | The return value is a number giving the number of seconds since the epoch (see the time module). 628 | 629 | Raise OSError if the file does not exist or is inaccessible. 630 | """ 631 | return juicefs_stat_result(self._lib.stat1, path).st_atime 632 | 633 | def getmtime(self, path) -> float: 634 | """Return the time of last modification of *path*. 635 | 636 | The return value is a number giving the number of seconds since the epoch (see the time module). 637 | 638 | Raise OSError if the file does not exist or is inaccessible. 639 | """ 640 | return juicefs_stat_result(self._lib.stat1, path).st_mtime 641 | 642 | def getsize(self, path) -> int: 643 | """Return the size, in bytes, of *path*. 644 | 645 | Raise OSError if the file does not exist or is inaccessible. 646 | """ 647 | return juicefs_stat_result(self._lib.stat1, path).st_size 648 | -------------------------------------------------------------------------------- /tests/test_juicefs.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import shutil 4 | import sys 5 | import time 6 | from pathlib import Path 7 | 8 | import pytest 9 | 10 | 11 | def remove_os_file(path): 12 | if os.path.exists(path): 13 | os.chmod(path, 0o777) 14 | if os.path.isdir(path): 15 | shutil.rmtree(path) 16 | else: 17 | os.remove(path) 18 | if os.path.lexists(path): 19 | os.remove(path) 20 | 21 | 22 | def create_os_tempfile(path): 23 | remove_os_file(path) 24 | yield path 25 | remove_os_file(path) 26 | 27 | 28 | def remove_file(jfs, path): 29 | if jfs.path.exists(path): 30 | jfs.chmod(path, 0o777) 31 | if jfs.path.lexists(path): 32 | jfs.rmtree(path) 33 | 34 | 35 | def create_tempfile(jfs, path): 36 | remove_file(jfs, path) 37 | yield path 38 | remove_file(jfs, path) 39 | 40 | 41 | @pytest.fixture() 42 | def dirname(jfs): 43 | yield from create_tempfile(jfs, "/test.dir") 44 | 45 | 46 | @pytest.fixture() 47 | def dirname2(jfs): 48 | yield from create_tempfile(jfs, "/test.dir/dir2") 49 | 50 | 51 | @pytest.fixture() 52 | def dirname3(jfs): 53 | yield from create_tempfile(jfs, "/test.dir/dir2/dir3") 54 | 55 | 56 | @pytest.fixture() 57 | def filename(jfs): 58 | yield from create_tempfile(jfs, "/test.file") 59 | 60 | 61 | @pytest.fixture() 62 | def filename2(jfs): 63 | yield from create_tempfile(jfs, "/test.file.2") 64 | 65 | 66 | @pytest.fixture() 67 | def filename3(jfs): 68 | yield from create_tempfile(jfs, "/test.file.3") 69 | 70 | 71 | @pytest.fixture() 72 | def os_filename(): 73 | yield from create_os_tempfile("./test.file") 74 | 75 | 76 | @pytest.fixture() 77 | def os_filename2(): 78 | yield from create_os_tempfile("./test.file.2") 79 | 80 | 81 | @pytest.fixture() 82 | def os_dirname(jfs): 83 | yield from create_os_tempfile("./test.dir") 84 | 85 | 86 | CONTENT = b"text" 87 | CONTENT2 = b"text/text" 88 | CONTENT_WITH_ZERO = ( 89 | b"\x05\x00\x00\x00\x00\x00\x00\x00\x98\x00\x00\x00\x00\x00\x00\x00-\x01" 90 | ) 91 | 92 | 93 | def test_rename(jfs, filename, filename2, dirname): 94 | jfs.mkdir(dirname) 95 | jfs.create(filename) 96 | with pytest.raises(FileExistsError): 97 | jfs.rename(filename, dirname) 98 | 99 | assert jfs.path.exists(filename2) is False 100 | assert jfs.path.exists(filename) is True 101 | jfs.rename(filename, filename2) 102 | assert jfs.path.exists(filename2) is True 103 | assert jfs.path.exists(filename) is False 104 | 105 | jfs.create(filename, 0o777) 106 | assert jfs.path.exists(filename) is True 107 | assert jfs.path.exists(filename2) is True 108 | with pytest.raises(FileExistsError): 109 | jfs.rename(filename, filename2) 110 | 111 | 112 | def test_replace(jfs, filename, filename2, dirname): 113 | test_rename(jfs, filename, filename2, dirname) 114 | 115 | 116 | def test_symlink(jfs, filename, filename2): 117 | assert jfs.path.islink(filename2) is False 118 | jfs.create(filename) 119 | jfs.symlink(filename, filename2) 120 | assert jfs.path.islink(filename2) is True 121 | assert jfs.stat(filename) == jfs.stat(filename2) 122 | 123 | 124 | def test_unlink(jfs, filename, dirname): 125 | assert jfs.path.exists(filename) is False 126 | jfs.create(filename) 127 | assert jfs.path.exists(filename) is True 128 | jfs.unlink(filename) 129 | assert jfs.path.exists(filename) is False 130 | 131 | jfs.mkdir(dirname) 132 | with pytest.raises(IsADirectoryError): 133 | jfs.unlink(dirname) 134 | 135 | 136 | def test_remove(jfs, filename, dirname): 137 | test_unlink(jfs, filename, dirname) 138 | 139 | 140 | def test_makedirs(jfs, dirname, dirname2, dirname3): 141 | assert jfs.path.isdir(dirname) is False 142 | jfs.makedirs(dirname) 143 | 144 | assert jfs.path.isdir(dirname) is True 145 | jfs.makedirs(dirname, exist_ok=True) 146 | 147 | with pytest.raises(OSError): 148 | jfs.makedirs(dirname) 149 | 150 | assert jfs.path.isdir(dirname2) is False 151 | jfs.makedirs(dirname2) 152 | 153 | assert jfs.path.isdir(dirname2) is True 154 | jfs.makedirs(dirname2, exist_ok=True) 155 | 156 | with pytest.raises(OSError): 157 | jfs.makedirs(dirname2) 158 | 159 | assert jfs.path.isdir(dirname3) is False 160 | jfs.makedirs(dirname3) 161 | 162 | assert jfs.path.isdir(dirname3) is True 163 | jfs.makedirs(dirname3, exist_ok=True) 164 | 165 | with pytest.raises(OSError): 166 | jfs.makedirs(dirname3) 167 | 168 | 169 | def test_removedirs(jfs, dirname, dirname2, dirname3, filename): 170 | jfs.makedirs(dirname2) 171 | with pytest.raises(OSError): 172 | jfs.removedirs(dirname) 173 | 174 | jfs.removedirs(dirname2) 175 | assert jfs.path.exists(dirname2) is False 176 | 177 | jfs.makedirs(dirname2) 178 | jfs.create(dirname + "/" + filename, 0o777) 179 | jfs.removedirs(dirname2) 180 | assert jfs.path.exists(dirname2) is False 181 | assert jfs.path.exists(dirname + "/" + filename) is True 182 | jfs.unlink(dirname + "/" + filename) 183 | 184 | 185 | # def test_utime(jfs, filename): 186 | # jfs.create(filename) 187 | # start = int(time.time()) 188 | # jfs.utime(filename) 189 | # stop = int(time.time()) 190 | # assert jfs.path.getatime(filename) == jfs.path.getmtime(filename) 191 | # assert start <= int(jfs.path.getatime(filename)) <= stop 192 | # assert start <= int(jfs.path.getmtime(filename)) <= stop 193 | 194 | 195 | def test_mkdir(jfs, dirname): 196 | jfs.mkdir(dirname) 197 | assert jfs.path.exists(dirname) is True 198 | 199 | with pytest.raises(FileExistsError): 200 | jfs.mkdir(dirname) 201 | 202 | jfs.chmod(dirname, 0o444) # readonly 203 | with pytest.raises(PermissionError): 204 | jfs.mkdir(Path(os.path.join(dirname, "test")).as_posix()) 205 | 206 | 207 | def test_rmdir(jfs, filename, dirname): 208 | jfs.mkdir(dirname) 209 | assert jfs.path.exists(dirname) is True 210 | jfs.rmdir(dirname) 211 | assert jfs.path.exists(dirname) is False 212 | 213 | jfs.create(filename) 214 | with pytest.raises(NotADirectoryError): 215 | jfs.rmdir(filename) 216 | 217 | jfs.mkdir(dirname) 218 | jfs.create(dirname + "/test.file", 0o777) 219 | with pytest.raises(OSError): # Directory not empty 220 | jfs.rmdir(dirname) 221 | jfs.unlink(dirname + "/test.file") 222 | 223 | 224 | def test_exists(jfs, filename, dirname): 225 | assert jfs.path.exists(filename) is False 226 | jfs.create(filename) 227 | assert jfs.path.exists(filename) is True 228 | 229 | assert jfs.path.exists(dirname) is False 230 | jfs.mkdir(dirname) 231 | assert jfs.path.exists(dirname) is True 232 | 233 | 234 | def test_lexists(jfs, filename, filename2, filename3, dirname): 235 | assert jfs.path.lexists(filename) is False 236 | jfs.create(filename) 237 | assert jfs.path.lexists(filename) is True 238 | jfs.symlink(filename, filename2) 239 | assert jfs.path.lexists(filename2) is True 240 | jfs.unlink(filename) 241 | assert jfs.path.lexists(filename2) is True 242 | jfs.unlink(filename2) 243 | assert jfs.path.lexists(filename2) is False 244 | 245 | assert jfs.path.lexists(dirname) is False 246 | jfs.mkdir(dirname) 247 | assert jfs.path.lexists(dirname) is True 248 | jfs.symlink(dirname, filename3) 249 | assert jfs.path.lexists(filename3) is True 250 | jfs.rmdir(dirname) 251 | assert jfs.path.lexists(filename3) is True 252 | jfs.unlink(filename3) 253 | assert jfs.path.lexists(filename3) is False 254 | 255 | 256 | def test_access_dir(jfs, dirname): 257 | jfs.mkdir(dirname) 258 | assert jfs.access(dirname, os.R_OK) is True 259 | assert jfs.access(dirname, os.W_OK) is True 260 | assert jfs.access(dirname, os.X_OK) is True 261 | 262 | jfs.chmod(dirname, 0o444) # readonly 263 | assert jfs.access(dirname, os.W_OK) is False 264 | assert jfs.access(dirname, os.X_OK) is False 265 | 266 | 267 | def test_access_file(jfs, filename): 268 | jfs.create(filename) 269 | assert jfs.access(filename, os.R_OK) is True 270 | assert jfs.access(filename, os.W_OK) is True 271 | assert jfs.access(filename, os.X_OK) is True 272 | 273 | jfs.chmod(filename, 0o444) # readonly 274 | assert jfs.access(filename, os.W_OK) is False 275 | assert jfs.access(filename, os.X_OK) is False 276 | 277 | 278 | def test_access_follow_symlinks(jfs, filename, filename2): 279 | jfs.symlink(filename, filename2) 280 | assert jfs.access(filename2, os.R_OK) is False 281 | assert jfs.access(filename2, os.W_OK) is False 282 | assert jfs.access(filename2, os.X_OK) is False 283 | 284 | jfs.create(filename) 285 | assert jfs.access(filename2, os.R_OK) is True 286 | assert jfs.access(filename2, os.W_OK) is True 287 | assert jfs.access(filename2, os.X_OK) is True 288 | 289 | 290 | def test_scandir(jfs, filename, filename2, dirname): 291 | jfs.symlink(filename, filename2) 292 | jfs.mkdir(dirname) 293 | 294 | entries = list(jfs.scandir("/")) 295 | assert len(entries) == 2 296 | 297 | for entry in entries: 298 | if entry.path == filename2: 299 | assert entry.is_file() is False 300 | assert entry.is_symlink() is True 301 | elif entry.path == dirname: 302 | assert entry.is_dir() is True 303 | assert entry.is_symlink() is False 304 | else: 305 | assert False, entry 306 | 307 | 308 | def test_scandir_follow_symlinks(jfs, filename, filename2, dirname): 309 | jfs.create(filename) 310 | jfs.symlink(filename, filename2) 311 | jfs.mkdir(dirname) 312 | 313 | entries = list(jfs.scandir("/")) 314 | assert len(entries) == 3 315 | 316 | for entry in entries: 317 | if entry.path == filename: 318 | assert entry.is_file() is True 319 | assert entry.is_symlink() is False 320 | elif entry.path == filename2: 321 | assert entry.is_file() is False 322 | assert entry.is_symlink() is True 323 | elif entry.path == dirname: 324 | assert entry.is_dir() is True 325 | assert entry.is_symlink() is False 326 | else: 327 | assert False, entry 328 | 329 | 330 | def test_listdir(jfs, filename, filename2, dirname): 331 | jfs.create(filename) 332 | jfs.symlink(filename, filename2) 333 | jfs.mkdir(dirname) 334 | 335 | names = list(jfs.listdir("/")) 336 | assert len(names) == 3 337 | assert set(names) == set( 338 | [ 339 | filename[1:], 340 | filename2[1:], 341 | dirname[1:], 342 | ] 343 | ) 344 | 345 | 346 | def test_walk(jfs, dirname): 347 | jfs.mkdir(dirname) 348 | jfs.create(Path(os.path.join(dirname, "file")).as_posix()) 349 | jfs.mkdir(Path(os.path.join(dirname, "dir")).as_posix()) 350 | jfs.symlink( 351 | Path(os.path.join(dirname, "file")).as_posix(), 352 | Path(os.path.join(dirname, "dir", "link")).as_posix(), 353 | ) 354 | 355 | assert list(jfs.walk(dirname)) == [ 356 | (dirname, ["dir"], ["file"]), 357 | (Path(os.path.join(dirname, "dir")).as_posix(), [], ["link"]), 358 | ] 359 | 360 | assert list(jfs.walk(dirname, topdown=False)) == [ 361 | (Path(os.path.join(dirname, "dir")).as_posix(), [], ["link"]), 362 | (dirname, ["dir"], ["file"]), 363 | ] 364 | 365 | 366 | def test_isfile(jfs, filename, filename2, dirname): 367 | assert jfs.path.isfile(filename) is False 368 | assert jfs.path.isfile(filename2) is False 369 | assert jfs.path.isfile(dirname) is False 370 | 371 | jfs.symlink(filename, filename2) 372 | assert jfs.path.isfile(filename2) is False 373 | 374 | jfs.create(filename) 375 | assert jfs.path.isfile(filename) is True 376 | assert jfs.path.isfile(filename2) is True 377 | 378 | jfs.mkdir(dirname) 379 | assert jfs.path.isfile(dirname) is False 380 | 381 | 382 | def test_isdir(jfs, filename, filename2, dirname): 383 | assert jfs.path.isdir(dirname) is False 384 | assert jfs.path.isdir(filename) is False 385 | assert jfs.path.isdir(filename2) is False 386 | 387 | jfs.symlink(dirname, filename2) 388 | assert jfs.path.isdir(filename2) is False 389 | jfs.mkdir(dirname) 390 | assert jfs.path.isdir(dirname) is True 391 | assert jfs.path.isdir(dirname + "/") is True 392 | assert jfs.path.isdir(filename2) is True 393 | 394 | 395 | def test_islink(jfs, filename, filename2, filename3, dirname): 396 | assert jfs.path.islink(filename2) is False 397 | 398 | jfs.create(filename) 399 | jfs.symlink(filename, filename2) 400 | assert jfs.path.islink(filename2) is True 401 | 402 | jfs.unlink(filename2) 403 | assert jfs.path.islink(filename2) is False 404 | 405 | jfs.mkdir(dirname) 406 | jfs.symlink(dirname, filename2) 407 | assert jfs.path.islink(filename2) is True 408 | assert jfs.path.islink(filename3) is False 409 | 410 | jfs.symlink(filename2, filename3) 411 | assert jfs.path.islink(filename3) is True 412 | 413 | 414 | def test_getatime(jfs, filename): 415 | start = int(time.time()) 416 | jfs.create(filename) 417 | stop = int(time.time()) 418 | assert start <= int(jfs.path.getatime(filename)) <= stop 419 | 420 | start = int(time.time()) 421 | jfs.close(jfs.open(filename, os.O_RDONLY)) 422 | stop = int(time.time()) 423 | assert start <= int(jfs.path.getatime(filename)) <= stop 424 | 425 | 426 | def test_getmtime(jfs, filename): 427 | start = int(time.time()) 428 | jfs.create(filename) 429 | stop = int(time.time()) 430 | assert start <= int(jfs.path.getmtime(filename)) <= stop 431 | 432 | start = int(time.time()) 433 | fd = jfs.open(filename, os.O_WRONLY) 434 | jfs.write(fd, CONTENT) 435 | jfs.close(fd) 436 | stop = int(time.time()) 437 | assert start <= int(jfs.path.getmtime(filename)) <= stop 438 | 439 | 440 | def test_getsize(jfs, filename): 441 | jfs.create(filename) 442 | fd = jfs.open(filename, os.O_WRONLY) 443 | jfs.write(fd, CONTENT) 444 | jfs.close(fd) 445 | assert jfs.path.getsize(filename) == len(CONTENT) 446 | 447 | fd = jfs.open(filename, os.O_WRONLY) 448 | jfs.write(fd, CONTENT2) 449 | jfs.close(fd) 450 | assert jfs.path.getsize(filename) == len(CONTENT2) 451 | 452 | 453 | def test_fdopen(jfs, filename): 454 | flags = os.O_CREAT | os.O_TRUNC | os.O_WRONLY 455 | fd = jfs.open(filename, flags) 456 | with jfs.fdopen(fd) as fp: 457 | fp.write(b"text") 458 | 459 | flags = os.O_RDONLY 460 | fd = jfs.open(filename, flags) 461 | with jfs.fdopen(fd) as fp: 462 | assert fp.read() == b"text" 463 | 464 | flags = os.O_RDWR 465 | fd = jfs.open(filename, flags) 466 | with jfs.fdopen(fd) as fp: 467 | assert fp.read() == b"text" 468 | fp.write(b"/text") 469 | 470 | flags = os.O_RDONLY 471 | fd = jfs.open(filename, flags) 472 | with jfs.fdopen(fd) as fp: 473 | assert fp.read() == b"text/text" 474 | 475 | 476 | def test_readlink(jfs, filename, filename2, dirname): 477 | jfs.mkdir(dirname) 478 | jfs.symlink("." + filename, filename2) 479 | 480 | assert jfs.readlink(filename2) == "." + filename 481 | 482 | 483 | def test_truncate(jfs, filename): 484 | jfs.create(filename) 485 | fd = jfs.open(filename, os.O_WRONLY) 486 | jfs.write(fd, CONTENT) 487 | jfs.close(fd) 488 | length = int(random.random() * len(CONTENT)) 489 | jfs.truncate(filename, length) 490 | assert jfs.path.getsize(filename) == length 491 | 492 | fd = jfs.open(filename, os.O_WRONLY) 493 | jfs.write(fd, CONTENT2) 494 | jfs.close(fd) 495 | length2 = int(random.random() * len(CONTENT2)) 496 | jfs.truncate(filename, length2) 497 | assert jfs.path.getsize(filename) == length2 498 | 499 | 500 | def test_ftruncate(jfs, filename): 501 | jfs.create(filename) 502 | fd = jfs.open(filename, os.O_WRONLY) 503 | jfs.write(fd, CONTENT) 504 | length = int(random.random() * len(CONTENT)) 505 | jfs.flush(fd) 506 | jfs.ftruncate(fd, length) 507 | jfs.close(fd) 508 | assert jfs.path.getsize(filename) == length 509 | 510 | fd = jfs.open(filename, os.O_WRONLY) 511 | jfs.write(fd, CONTENT2) 512 | length2 = int(random.random() * len(CONTENT2)) 513 | jfs.flush(fd) 514 | jfs.ftruncate(fd, length2) 515 | jfs.close(fd) 516 | assert jfs.path.getsize(filename) == length2 517 | 518 | 519 | @pytest.mark.skipif( 520 | sys.platform == "win32", 521 | reason="Windows can't use os.chmod() to change any file mode we want", 522 | ) 523 | def test_stat_file(jfs, filename, os_filename): 524 | jfs.create(filename, 0o644) 525 | 526 | with open(os_filename, "wb"): 527 | pass 528 | 529 | os.chmod(os_filename, 0o644) 530 | assert os.stat(os_filename).st_mode == jfs.stat(filename).st_mode 531 | 532 | os.chmod(os_filename, 0o755) 533 | jfs.chmod(filename, 0o755) 534 | assert os.stat(os_filename).st_mode == jfs.stat(filename).st_mode 535 | 536 | 537 | @pytest.mark.skipif( 538 | sys.platform == "win32", 539 | reason="Windows can't use os.chmod() to change any file mode we want", 540 | ) 541 | def test_stat_dir(jfs, dirname, os_dirname): 542 | jfs.mkdir(dirname) 543 | os.mkdir(os_dirname) 544 | os.chmod(os_dirname, 0o644) 545 | jfs.chmod(dirname, 0o644) 546 | assert os.stat(os_dirname).st_mode == jfs.stat(dirname).st_mode 547 | 548 | os.chmod(os_dirname, 0o755) 549 | jfs.chmod(dirname, 0o755) 550 | assert os.stat(os_dirname).st_mode == jfs.stat(dirname).st_mode 551 | 552 | 553 | def test_stat_link(jfs, filename, filename2, filename3, dirname): 554 | jfs.create(filename, 0o644) 555 | jfs.symlink(filename, filename2) 556 | 557 | assert jfs.stat(filename2) == jfs.stat(filename) 558 | 559 | jfs.chmod(filename, 0o755) 560 | assert jfs.stat(filename2) == jfs.stat(filename) 561 | 562 | jfs.mkdir(dirname) 563 | jfs.symlink(dirname, filename3) 564 | assert jfs.stat(filename3) == jfs.stat(dirname) 565 | 566 | jfs.chmod(dirname, 0o644) 567 | assert jfs.stat(filename3) == jfs.stat(dirname) 568 | 569 | jfs.chmod(dirname, 0o755) 570 | assert jfs.stat(filename3) == jfs.stat(dirname) 571 | 572 | 573 | def test_lstat_link(jfs, filename, filename2): 574 | jfs.symlink(filename, filename2) 575 | assert int(oct(jfs.lstat(filename2).st_mode)[-3:], 8) == 0o644 576 | 577 | 578 | @pytest.mark.skipif( 579 | sys.platform == "win32", 580 | reason="Windows can't use os.chmod() to change any file mode we want", 581 | ) 582 | def test_lstat_file(jfs, filename, os_filename): 583 | jfs.create(filename, 0o644) 584 | 585 | with open(os_filename, "wb"): 586 | pass 587 | 588 | os.chmod(os_filename, 0o644) 589 | assert os.lstat(os_filename).st_mode == jfs.lstat(filename).st_mode 590 | 591 | os.chmod(os_filename, 0o755) 592 | jfs.chmod(filename, 0o755) 593 | assert os.lstat(os_filename).st_mode == jfs.lstat(filename).st_mode 594 | 595 | 596 | @pytest.mark.skipif( 597 | sys.platform == "win32", 598 | reason="Windows can't use os.chmod() to change any file mode we want", 599 | ) 600 | def test_lstat_dir(jfs, dirname, os_dirname): 601 | jfs.mkdir(dirname) 602 | os.mkdir(os_dirname) 603 | os.chmod(os_dirname, 0o644) 604 | jfs.chmod(dirname, 0o644) 605 | assert os.lstat(os_dirname).st_mode == jfs.lstat(dirname).st_mode 606 | 607 | os.chmod(os_dirname, 0o755) 608 | jfs.chmod(dirname, 0o755) 609 | assert os.lstat(os_dirname).st_mode == jfs.lstat(dirname).st_mode 610 | 611 | 612 | def test_chmod(jfs, filename, dirname, filename2): 613 | jfs.create(filename) 614 | for mode in range(1, 0o777 + 1): 615 | jfs.chmod(filename, mode) 616 | assert int(oct(jfs.stat(filename).st_mode)[-3:], 8) == mode 617 | 618 | jfs.mkdir(dirname) 619 | for mode in range(1, 0o777 + 1): 620 | jfs.chmod(dirname, mode) 621 | assert int(oct(jfs.stat(dirname).st_mode)[-3:], 8) == mode 622 | 623 | jfs.symlink(filename, filename2) 624 | link_mode = int(oct(jfs.lstat(filename2).st_mode)[-3:], 8) 625 | for mode in range(1, 0o777 + 1): 626 | jfs.chmod(filename2, mode) 627 | assert int(oct(jfs.stat(filename2).st_mode)[-3:], 8) == mode 628 | assert int(oct(jfs.lstat(filename2).st_mode)[-3:], 8) == link_mode 629 | 630 | 631 | def test_lseek(jfs, filename): 632 | jfs.create(filename) 633 | fd = jfs.open(filename, os.O_WRONLY) 634 | l = jfs.write(fd, CONTENT) 635 | assert l == len(CONTENT) 636 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == len(CONTENT) 637 | 638 | l = jfs.write(fd, CONTENT) 639 | assert l == len(CONTENT) 640 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == len(2 * CONTENT) 641 | 642 | # test seek_set 643 | for i in range(len(3 * CONTENT)): 644 | pos = jfs.lseek(fd, i, os.SEEK_SET) 645 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == pos 646 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == i 647 | 648 | # test seek_cur 649 | jfs.lseek(fd, 0, os.SEEK_SET) 650 | pos = 0 651 | for i in range(10): 652 | offset = random.randint(-len(CONTENT), len(CONTENT)) 653 | if offset + pos < 0: 654 | jfs.lseek(fd, 0, os.SEEK_SET) 655 | pos = 0 656 | continue 657 | 658 | pos_now = jfs.lseek(fd, offset, os.SEEK_CUR) 659 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == pos_now 660 | assert pos + offset == pos_now 661 | pos = pos_now 662 | 663 | jfs.flush(fd) 664 | jfs.fsync(fd) 665 | 666 | l = jfs.lseek(fd, 0, os.SEEK_END) 667 | 668 | assert l == len(2 * CONTENT) 669 | assert jfs.lseek(fd, 0, os.SEEK_CUR) == len(2 * CONTENT) 670 | assert jfs.lseek(fd, 0, os.SEEK_END) == len(2 * CONTENT) 671 | 672 | 673 | def test_flush(jfs, filename): 674 | jfs.create(filename) 675 | fdw = jfs.open(filename, os.O_WRONLY) 676 | jfs.write(fdw, CONTENT) 677 | jfs.flush(fdw) 678 | 679 | fdr = jfs.open(filename, os.O_RDONLY) 680 | assert jfs.read(fdr, len(CONTENT)) == CONTENT 681 | jfs.close(fdr) 682 | 683 | jfs.write(fdw, CONTENT[::-1]) 684 | jfs.flush(fdw) 685 | 686 | fdr = jfs.open(filename, os.O_RDONLY) 687 | assert jfs.read(fdr, 2 * len(CONTENT)) == CONTENT + CONTENT[::-1] 688 | jfs.close(fdr) 689 | jfs.close(fdw) 690 | 691 | 692 | @pytest.mark.skipif( 693 | sys.platform == "win32", 694 | reason="jfs.concat() will raise FileNotFoundError on Windows", 695 | ) 696 | def test_concat(jfs, filename, filename2, filename3): 697 | jfs.create(filename) 698 | jfs.create(filename2) 699 | jfs.create(filename3) 700 | 701 | fdw = jfs.open(filename, os.O_WRONLY) 702 | jfs.write(fdw, CONTENT) 703 | jfs.close(fdw) 704 | 705 | fdw = jfs.open(filename2, os.O_WRONLY) 706 | jfs.write(fdw, CONTENT) 707 | jfs.close(fdw) 708 | 709 | jfs.concat(filename3, filename2, filename) 710 | 711 | fdr = jfs.open(filename3, os.O_RDONLY) 712 | size = jfs.path.getsize(filename3) 713 | assert jfs.read(fdr, size) == CONTENT * 2 714 | jfs.close(fdr) 715 | 716 | 717 | def test_summary(jfs, dirname): 718 | jfs.mkdir(dirname) 719 | summary = jfs.summary(dirname) 720 | 721 | assert summary.size == 0 722 | assert summary.files == 0 723 | assert summary.dirs == 1 724 | 725 | jfs.mkdir(Path(os.path.join(dirname, "dir")).as_posix()) 726 | jfs.create(Path(os.path.join(dirname, "dir", "file")).as_posix()) 727 | jfs.create(Path(os.path.join(dirname, "file")).as_posix()) 728 | 729 | fdw = jfs.open(Path(os.path.join(dirname, "file")).as_posix(), os.O_WRONLY) 730 | jfs.write(fdw, CONTENT) 731 | jfs.close(fdw) 732 | 733 | fdw = jfs.open(Path(os.path.join(dirname, "dir", "file")).as_posix(), os.O_WRONLY) 734 | jfs.write(fdw, CONTENT) 735 | jfs.close(fdw) 736 | 737 | summary = jfs.summary(dirname) 738 | 739 | assert summary.size == 8 740 | assert summary.files == 2 741 | assert summary.dirs == 2 742 | 743 | 744 | def test_pread(jfs, filename): 745 | jfs.create(filename) 746 | fdw = jfs.open(filename, os.O_WRONLY) 747 | jfs.write(fdw, CONTENT_WITH_ZERO) 748 | jfs.close(fdw) 749 | 750 | fdr = jfs.open(filename, os.O_RDONLY) 751 | for offset in range(5): 752 | for size in range(len(CONTENT_WITH_ZERO) + 5): 753 | start_pos = min(offset, len(CONTENT_WITH_ZERO)) 754 | stop_pos = min(offset + size, len(CONTENT_WITH_ZERO)) 755 | assert jfs.pread(fdr, size, offset) == CONTENT_WITH_ZERO[start_pos:stop_pos] 756 | jfs.close(fdr) 757 | 758 | 759 | def test_read_content_with_zero(jfs, filename): 760 | jfs.create(filename) 761 | fdw = jfs.open(filename, os.O_WRONLY) 762 | content_len = jfs.write(fdw, CONTENT_WITH_ZERO) 763 | assert content_len == len(CONTENT_WITH_ZERO) 764 | jfs.close(fdw) 765 | 766 | fdr = jfs.open(filename, os.O_RDONLY) 767 | assert jfs.read(fdr, 7) == CONTENT_WITH_ZERO[:7] 768 | assert jfs.read(fdr, len(CONTENT_WITH_ZERO) + 10) == CONTENT_WITH_ZERO[7:] 769 | jfs.close(fdr) 770 | 771 | from juicefs.io import open as _jfs_open 772 | 773 | with _jfs_open(jfs, filename, "rb") as f: 774 | f.read() == CONTENT_WITH_ZERO 775 | 776 | 777 | def test_xattr(jfs, filename): 778 | jfs.create(filename) 779 | 780 | attr = "com.megfile.color" 781 | assert jfs.listxattr(filename) == [] 782 | with pytest.raises(OSError): 783 | jfs.getxattr(filename, attr) 784 | jfs.setxattr(filename, attr, b"green") 785 | assert jfs.listxattr(filename) == [attr] 786 | assert jfs.getxattr(filename, attr) == b"green" 787 | jfs.removexattr(filename, attr) 788 | with pytest.raises(OSError): 789 | jfs.getxattr(filename, attr) 790 | 791 | 792 | # TODO: read-write mode not supported in this version 793 | # def test_read_write_wbp(jfs, filename): 794 | # TEXT = b"hello world" 795 | # jfs.create(filename) 796 | # fdw = jfs.open(filename, os.O_WRONLY) 797 | # jfs.write(fdw, TEXT) 798 | # jfs.close(fdw) 799 | 800 | # fd = jfs.open(filename, os.O_RDWR) 801 | # assert jfs.read(fd, 6) == TEXT[:6] 802 | # jfs.write(fd, TEXT[::-1]) 803 | # jfs.flush(fd) 804 | 805 | # jfs.lseek(fd, 0, os.SEEK_SET) 806 | 807 | # assert jfs.read(fd, 2 * len(TEXT)) == TEXT[:6] + TEXT[::-1] 808 | 809 | 810 | # TODO: chown() 目前无法使用，没搞懂它的行为，只会抛 Permission denied 811 | # def test_chown(jfs, filename, dirname): 812 | # jfs.create(filename) 813 | 814 | # new_user = "new_user" 815 | # new_group = "new_group" 816 | # origin_user = jfs.stat(filename).st_uid 817 | # origin_group = jfs.stat(filename).st_gid 818 | 819 | # assert origin_user != new_user 820 | # assert origin_group != new_group 821 | 822 | # jfs.chown(filename, origin_user, new_group) 823 | # assert jfs.stat(filename).st_gid == new_group 824 | # assert jfs.stat(filename).st_uid == origin_user 825 | 826 | # jfs.chown(filename, origin_user, origin_group) 827 | # assert jfs.stat(filename).st_gid == origin_group 828 | # assert jfs.stat(filename).st_uid == origin_user 829 | 830 | # jfs.chown(filename, new_user, origin_group) 831 | # assert jfs.stat(filename).st_gid == origin_group 832 | # assert jfs.stat(filename).st_uid == new_user 833 | 834 | # jfs.chown(filename, origin_user, origin_group) 835 | # assert jfs.stat(filename).st_gid == origin_group 836 | # assert jfs.stat(filename).st_uid == origin_user 837 | 838 | # jfs.chown(filename, new_user, origin_group) 839 | # assert jfs.stat(filename).st_gid == origin_group 840 | # assert jfs.stat(filename).st_uid == new_user 841 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU AFFERO GENERAL PUBLIC LICENSE 2 | Version 3, 19 November 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU Affero General Public License is a free, copyleft license for 11 | software and other kinds of works, specifically designed to ensure 12 | cooperation with the community in the case of network server software. 13 | 14 | The licenses for most software and other practical works are designed 15 | to take away your freedom to share and change the works. By contrast, 16 | our General Public Licenses are intended to guarantee your freedom to 17 | share and change all versions of a program--to make sure it remains free 18 | software for all its users. 19 | 20 | When we speak of free software, we are referring to freedom, not 21 | price. Our General Public Licenses are designed to make sure that you 22 | have the freedom to distribute copies of free software (and charge for 23 | them if you wish), that you receive source code or can get it if you 24 | want it, that you can change the software or use pieces of it in new 25 | free programs, and that you know you can do these things. 26 | 27 | Developers that use our General Public Licenses protect your rights 28 | with two steps: (1) assert copyright on the software, and (2) offer 29 | you this License which gives you legal permission to copy, distribute 30 | and/or modify the software. 31 | 32 | A secondary benefit of defending all users' freedom is that 33 | improvements made in alternate versions of the program, if they 34 | receive widespread use, become available for other developers to 35 | incorporate. Many developers of free software are heartened and 36 | encouraged by the resulting cooperation. However, in the case of 37 | software used on network servers, this result may fail to come about. 38 | The GNU General Public License permits making a modified version and 39 | letting the public access it on a server without ever releasing its 40 | source code to the public. 41 | 42 | The GNU Affero General Public License is designed specifically to 43 | ensure that, in such cases, the modified source code becomes available 44 | to the community. It requires the operator of a network server to 45 | provide the source code of the modified version running there to the 46 | users of that server. Therefore, public use of a modified version, on 47 | a publicly accessible server, gives the public access to the source 48 | code of the modified version. 49 | 50 | An older license, called the Affero General Public License and 51 | published by Affero, was designed to accomplish similar goals. This is 52 | a different license, not a version of the Affero GPL, but Affero has 53 | released a new version of the Affero GPL which permits relicensing under 54 | this license. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | TERMS AND CONDITIONS 60 | 61 | 0. Definitions. 62 | 63 | "This License" refers to version 3 of the GNU Affero General Public License. 64 | 65 | "Copyright" also means copyright-like laws that apply to other kinds of 66 | works, such as semiconductor masks. 67 | 68 | "The Program" refers to any copyrightable work licensed under this 69 | License. Each licensee is addressed as "you". "Licensees" and 70 | "recipients" may be individuals or organizations. 71 | 72 | To "modify" a work means to copy from or adapt all or part of the work 73 | in a fashion requiring copyright permission, other than the making of an 74 | exact copy. The resulting work is called a "modified version" of the 75 | earlier work or a work "based on" the earlier work. 76 | 77 | A "covered work" means either the unmodified Program or a work based 78 | on the Program. 79 | 80 | To "propagate" a work means to do anything with it that, without 81 | permission, would make you directly or secondarily liable for 82 | infringement under applicable copyright law, except executing it on a 83 | computer or modifying a private copy. Propagation includes copying, 84 | distribution (with or without modification), making available to the 85 | public, and in some countries other activities as well. 86 | 87 | To "convey" a work means any kind of propagation that enables other 88 | parties to make or receive copies. Mere interaction with a user through 89 | a computer network, with no transfer of a copy, is not conveying. 90 | 91 | An interactive user interface displays "Appropriate Legal Notices" 92 | to the extent that it includes a convenient and prominently visible 93 | feature that (1) displays an appropriate copyright notice, and (2) 94 | tells the user that there is no warranty for the work (except to the 95 | extent that warranties are provided), that licensees may convey the 96 | work under this License, and how to view a copy of this License. If 97 | the interface presents a list of user commands or options, such as a 98 | menu, a prominent item in the list meets this criterion. 99 | 100 | 1. Source Code. 101 | 102 | The "source code" for a work means the preferred form of the work 103 | for making modifications to it. "Object code" means any non-source 104 | form of a work. 105 | 106 | A "Standard Interface" means an interface that either is an official 107 | standard defined by a recognized standards body, or, in the case of 108 | interfaces specified for a particular programming language, one that 109 | is widely used among developers working in that language. 110 | 111 | The "System Libraries" of an executable work include anything, other 112 | than the work as a whole, that (a) is included in the normal form of 113 | packaging a Major Component, but which is not part of that Major 114 | Component, and (b) serves only to enable use of the work with that 115 | Major Component, or to implement a Standard Interface for which an 116 | implementation is available to the public in source code form. A 117 | "Major Component", in this context, means a major essential component 118 | (kernel, window system, and so on) of the specific operating system 119 | (if any) on which the executable work runs, or a compiler used to 120 | produce the work, or an object code interpreter used to run it. 121 | 122 | The "Corresponding Source" for a work in object code form means all 123 | the source code needed to generate, install, and (for an executable 124 | work) run the object code and to modify the work, including scripts to 125 | control those activities. However, it does not include the work's 126 | System Libraries, or general-purpose tools or generally available free 127 | programs which are used unmodified in performing those activities but 128 | which are not part of the work. For example, Corresponding Source 129 | includes interface definition files associated with source files for 130 | the work, and the source code for shared libraries and dynamically 131 | linked subprograms that the work is specifically designed to require, 132 | such as by intimate data communication or control flow between those 133 | subprograms and other parts of the work. 134 | 135 | The Corresponding Source need not include anything that users 136 | can regenerate automatically from other parts of the Corresponding 137 | Source. 138 | 139 | The Corresponding Source for a work in source code form is that 140 | same work. 141 | 142 | 2. Basic Permissions. 143 | 144 | All rights granted under this License are granted for the term of 145 | copyright on the Program, and are irrevocable provided the stated 146 | conditions are met. This License explicitly affirms your unlimited 147 | permission to run the unmodified Program. The output from running a 148 | covered work is covered by this License only if the output, given its 149 | content, constitutes a covered work. This License acknowledges your 150 | rights of fair use or other equivalent, as provided by copyright law. 151 | 152 | You may make, run and propagate covered works that you do not 153 | convey, without conditions so long as your license otherwise remains 154 | in force. You may convey covered works to others for the sole purpose 155 | of having them make modifications exclusively for you, or provide you 156 | with facilities for running those works, provided that you comply with 157 | the terms of this License in conveying all material for which you do 158 | not control copyright. Those thus making or running the covered works 159 | for you must do so exclusively on your behalf, under your direction 160 | and control, on terms that prohibit them from making any copies of 161 | your copyrighted material outside their relationship with you. 162 | 163 | Conveying under any other circumstances is permitted solely under 164 | the conditions stated below. Sublicensing is not allowed; section 10 165 | makes it unnecessary. 166 | 167 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 168 | 169 | No covered work shall be deemed part of an effective technological 170 | measure under any applicable law fulfilling obligations under article 171 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 172 | similar laws prohibiting or restricting circumvention of such 173 | measures. 174 | 175 | When you convey a covered work, you waive any legal power to forbid 176 | circumvention of technological measures to the extent such circumvention 177 | is effected by exercising rights under this License with respect to 178 | the covered work, and you disclaim any intention to limit operation or 179 | modification of the work as a means of enforcing, against the work's 180 | users, your or third parties' legal rights to forbid circumvention of 181 | technological measures. 182 | 183 | 4. Conveying Verbatim Copies. 184 | 185 | You may convey verbatim copies of the Program's source code as you 186 | receive it, in any medium, provided that you conspicuously and 187 | appropriately publish on each copy an appropriate copyright notice; 188 | keep intact all notices stating that this License and any 189 | non-permissive terms added in accord with section 7 apply to the code; 190 | keep intact all notices of the absence of any warranty; and give all 191 | recipients a copy of this License along with the Program. 192 | 193 | You may charge any price or no price for each copy that you convey, 194 | and you may offer support or warranty protection for a fee. 195 | 196 | 5. Conveying Modified Source Versions. 197 | 198 | You may convey a work based on the Program, or the modifications to 199 | produce it from the Program, in the form of source code under the 200 | terms of section 4, provided that you also meet all of these conditions: 201 | 202 | a) The work must carry prominent notices stating that you modified 203 | it, and giving a relevant date. 204 | 205 | b) The work must carry prominent notices stating that it is 206 | released under this License and any conditions added under section 207 | 7. This requirement modifies the requirement in section 4 to 208 | "keep intact all notices". 209 | 210 | c) You must license the entire work, as a whole, under this 211 | License to anyone who comes into possession of a copy. This 212 | License will therefore apply, along with any applicable section 7 213 | additional terms, to the whole of the work, and all its parts, 214 | regardless of how they are packaged. This License gives no 215 | permission to license the work in any other way, but it does not 216 | invalidate such permission if you have separately received it. 217 | 218 | d) If the work has interactive user interfaces, each must display 219 | Appropriate Legal Notices; however, if the Program has interactive 220 | interfaces that do not display Appropriate Legal Notices, your 221 | work need not make them do so. 222 | 223 | A compilation of a covered work with other separate and independent 224 | works, which are not by their nature extensions of the covered work, 225 | and which are not combined with it such as to form a larger program, 226 | in or on a volume of a storage or distribution medium, is called an 227 | "aggregate" if the compilation and its resulting copyright are not 228 | used to limit the access or legal rights of the compilation's users 229 | beyond what the individual works permit. Inclusion of a covered work 230 | in an aggregate does not cause this License to apply to the other 231 | parts of the aggregate. 232 | 233 | 6. Conveying Non-Source Forms. 234 | 235 | You may convey a covered work in object code form under the terms 236 | of sections 4 and 5, provided that you also convey the 237 | machine-readable Corresponding Source under the terms of this License, 238 | in one of these ways: 239 | 240 | a) Convey the object code in, or embodied in, a physical product 241 | (including a physical distribution medium), accompanied by the 242 | Corresponding Source fixed on a durable physical medium 243 | customarily used for software interchange. 244 | 245 | b) Convey the object code in, or embodied in, a physical product 246 | (including a physical distribution medium), accompanied by a 247 | written offer, valid for at least three years and valid for as 248 | long as you offer spare parts or customer support for that product 249 | model, to give anyone who possesses the object code either (1) a 250 | copy of the Corresponding Source for all the software in the 251 | product that is covered by this License, on a durable physical 252 | medium customarily used for software interchange, for a price no 253 | more than your reasonable cost of physically performing this 254 | conveying of source, or (2) access to copy the 255 | Corresponding Source from a network server at no charge. 256 | 257 | c) Convey individual copies of the object code with a copy of the 258 | written offer to provide the Corresponding Source. This 259 | alternative is allowed only occasionally and noncommercially, and 260 | only if you received the object code with such an offer, in accord 261 | with subsection 6b. 262 | 263 | d) Convey the object code by offering access from a designated 264 | place (gratis or for a charge), and offer equivalent access to the 265 | Corresponding Source in the same way through the same place at no 266 | further charge. You need not require recipients to copy the 267 | Corresponding Source along with the object code. If the place to 268 | copy the object code is a network server, the Corresponding Source 269 | may be on a different server (operated by you or a third party) 270 | that supports equivalent copying facilities, provided you maintain 271 | clear directions next to the object code saying where to find the 272 | Corresponding Source. Regardless of what server hosts the 273 | Corresponding Source, you remain obligated to ensure that it is 274 | available for as long as needed to satisfy these requirements. 275 | 276 | e) Convey the object code using peer-to-peer transmission, provided 277 | you inform other peers where the object code and Corresponding 278 | Source of the work are being offered to the general public at no 279 | charge under subsection 6d. 280 | 281 | A separable portion of the object code, whose source code is excluded 282 | from the Corresponding Source as a System Library, need not be 283 | included in conveying the object code work. 284 | 285 | A "User Product" is either (1) a "consumer product", which means any 286 | tangible personal property which is normally used for personal, family, 287 | or household purposes, or (2) anything designed or sold for incorporation 288 | into a dwelling. In determining whether a product is a consumer product, 289 | doubtful cases shall be resolved in favor of coverage. For a particular 290 | product received by a particular user, "normally used" refers to a 291 | typical or common use of that class of product, regardless of the status 292 | of the particular user or of the way in which the particular user 293 | actually uses, or expects or is expected to use, the product. A product 294 | is a consumer product regardless of whether the product has substantial 295 | commercial, industrial or non-consumer uses, unless such uses represent 296 | the only significant mode of use of the product. 297 | 298 | "Installation Information" for a User Product means any methods, 299 | procedures, authorization keys, or other information required to install 300 | and execute modified versions of a covered work in that User Product from 301 | a modified version of its Corresponding Source. The information must 302 | suffice to ensure that the continued functioning of the modified object 303 | code is in no case prevented or interfered with solely because 304 | modification has been made. 305 | 306 | If you convey an object code work under this section in, or with, or 307 | specifically for use in, a User Product, and the conveying occurs as 308 | part of a transaction in which the right of possession and use of the 309 | User Product is transferred to the recipient in perpetuity or for a 310 | fixed term (regardless of how the transaction is characterized), the 311 | Corresponding Source conveyed under this section must be accompanied 312 | by the Installation Information. But this requirement does not apply 313 | if neither you nor any third party retains the ability to install 314 | modified object code on the User Product (for example, the work has 315 | been installed in ROM). 316 | 317 | The requirement to provide Installation Information does not include a 318 | requirement to continue to provide support service, warranty, or updates 319 | for a work that has been modified or installed by the recipient, or for 320 | the User Product in which it has been modified or installed. Access to a 321 | network may be denied when the modification itself materially and 322 | adversely affects the operation of the network or violates the rules and 323 | protocols for communication across the network. 324 | 325 | Corresponding Source conveyed, and Installation Information provided, 326 | in accord with this section must be in a format that is publicly 327 | documented (and with an implementation available to the public in 328 | source code form), and must require no special password or key for 329 | unpacking, reading or copying. 330 | 331 | 7. Additional Terms. 332 | 333 | "Additional permissions" are terms that supplement the terms of this 334 | License by making exceptions from one or more of its conditions. 335 | Additional permissions that are applicable to the entire Program shall 336 | be treated as though they were included in this License, to the extent 337 | that they are valid under applicable law. If additional permissions 338 | apply only to part of the Program, that part may be used separately 339 | under those permissions, but the entire Program remains governed by 340 | this License without regard to the additional permissions. 341 | 342 | When you convey a copy of a covered work, you may at your option 343 | remove any additional permissions from that copy, or from any part of 344 | it. (Additional permissions may be written to require their own 345 | removal in certain cases when you modify the work.) You may place 346 | additional permissions on material, added by you to a covered work, 347 | for which you have or can give appropriate copyright permission. 348 | 349 | Notwithstanding any other provision of this License, for material you 350 | add to a covered work, you may (if authorized by the copyright holders of 351 | that material) supplement the terms of this License with terms: 352 | 353 | a) Disclaiming warranty or limiting liability differently from the 354 | terms of sections 15 and 16 of this License; or 355 | 356 | b) Requiring preservation of specified reasonable legal notices or 357 | author attributions in that material or in the Appropriate Legal 358 | Notices displayed by works containing it; or 359 | 360 | c) Prohibiting misrepresentation of the origin of that material, or 361 | requiring that modified versions of such material be marked in 362 | reasonable ways as different from the original version; or 363 | 364 | d) Limiting the use for publicity purposes of names of licensors or 365 | authors of the material; or 366 | 367 | e) Declining to grant rights under trademark law for use of some 368 | trade names, trademarks, or service marks; or 369 | 370 | f) Requiring indemnification of licensors and authors of that 371 | material by anyone who conveys the material (or modified versions of 372 | it) with contractual assumptions of liability to the recipient, for 373 | any liability that these contractual assumptions directly impose on 374 | those licensors and authors. 375 | 376 | All other non-permissive additional terms are considered "further 377 | restrictions" within the meaning of section 10. If the Program as you 378 | received it, or any part of it, contains a notice stating that it is 379 | governed by this License along with a term that is a further 380 | restriction, you may remove that term. If a license document contains 381 | a further restriction but permits relicensing or conveying under this 382 | License, you may add to a covered work material governed by the terms 383 | of that license document, provided that the further restriction does 384 | not survive such relicensing or conveying. 385 | 386 | If you add terms to a covered work in accord with this section, you 387 | must place, in the relevant source files, a statement of the 388 | additional terms that apply to those files, or a notice indicating 389 | where to find the applicable terms. 390 | 391 | Additional terms, permissive or non-permissive, may be stated in the 392 | form of a separately written license, or stated as exceptions; 393 | the above requirements apply either way. 394 | 395 | 8. Termination. 396 | 397 | You may not propagate or modify a covered work except as expressly 398 | provided under this License. Any attempt otherwise to propagate or 399 | modify it is void, and will automatically terminate your rights under 400 | this License (including any patent licenses granted under the third 401 | paragraph of section 11). 402 | 403 | However, if you cease all violation of this License, then your 404 | license from a particular copyright holder is reinstated (a) 405 | provisionally, unless and until the copyright holder explicitly and 406 | finally terminates your license, and (b) permanently, if the copyright 407 | holder fails to notify you of the violation by some reasonable means 408 | prior to 60 days after the cessation. 409 | 410 | Moreover, your license from a particular copyright holder is 411 | reinstated permanently if the copyright holder notifies you of the 412 | violation by some reasonable means, this is the first time you have 413 | received notice of violation of this License (for any work) from that 414 | copyright holder, and you cure the violation prior to 30 days after 415 | your receipt of the notice. 416 | 417 | Termination of your rights under this section does not terminate the 418 | licenses of parties who have received copies or rights from you under 419 | this License. If your rights have been terminated and not permanently 420 | reinstated, you do not qualify to receive new licenses for the same 421 | material under section 10. 422 | 423 | 9. Acceptance Not Required for Having Copies. 424 | 425 | You are not required to accept this License in order to receive or 426 | run a copy of the Program. Ancillary propagation of a covered work 427 | occurring solely as a consequence of using peer-to-peer transmission 428 | to receive a copy likewise does not require acceptance. However, 429 | nothing other than this License grants you permission to propagate or 430 | modify any covered work. These actions infringe copyright if you do 431 | not accept this License. Therefore, by modifying or propagating a 432 | covered work, you indicate your acceptance of this License to do so. 433 | 434 | 10. Automatic Licensing of Downstream Recipients. 435 | 436 | Each time you convey a covered work, the recipient automatically 437 | receives a license from the original licensors, to run, modify and 438 | propagate that work, subject to this License. You are not responsible 439 | for enforcing compliance by third parties with this License. 440 | 441 | An "entity transaction" is a transaction transferring control of an 442 | organization, or substantially all assets of one, or subdividing an 443 | organization, or merging organizations. If propagation of a covered 444 | work results from an entity transaction, each party to that 445 | transaction who receives a copy of the work also receives whatever 446 | licenses to the work the party's predecessor in interest had or could 447 | give under the previous paragraph, plus a right to possession of the 448 | Corresponding Source of the work from the predecessor in interest, if 449 | the predecessor has it or can get it with reasonable efforts. 450 | 451 | You may not impose any further restrictions on the exercise of the 452 | rights granted or affirmed under this License. For example, you may 453 | not impose a license fee, royalty, or other charge for exercise of 454 | rights granted under this License, and you may not initiate litigation 455 | (including a cross-claim or counterclaim in a lawsuit) alleging that 456 | any patent claim is infringed by making, using, selling, offering for 457 | sale, or importing the Program or any portion of it. 458 | 459 | 11. Patents. 460 | 461 | A "contributor" is a copyright holder who authorizes use under this 462 | License of the Program or a work on which the Program is based. The 463 | work thus licensed is called the contributor's "contributor version". 464 | 465 | A contributor's "essential patent claims" are all patent claims 466 | owned or controlled by the contributor, whether already acquired or 467 | hereafter acquired, that would be infringed by some manner, permitted 468 | by this License, of making, using, or selling its contributor version, 469 | but do not include claims that would be infringed only as a 470 | consequence of further modification of the contributor version. For 471 | purposes of this definition, "control" includes the right to grant 472 | patent sublicenses in a manner consistent with the requirements of 473 | this License. 474 | 475 | Each contributor grants you a non-exclusive, worldwide, royalty-free 476 | patent license under the contributor's essential patent claims, to 477 | make, use, sell, offer for sale, import and otherwise run, modify and 478 | propagate the contents of its contributor version. 479 | 480 | In the following three paragraphs, a "patent license" is any express 481 | agreement or commitment, however denominated, not to enforce a patent 482 | (such as an express permission to practice a patent or covenant not to 483 | sue for patent infringement). To "grant" such a patent license to a 484 | party means to make such an agreement or commitment not to enforce a 485 | patent against the party. 486 | 487 | If you convey a covered work, knowingly relying on a patent license, 488 | and the Corresponding Source of the work is not available for anyone 489 | to copy, free of charge and under the terms of this License, through a 490 | publicly available network server or other readily accessible means, 491 | then you must either (1) cause the Corresponding Source to be so 492 | available, or (2) arrange to deprive yourself of the benefit of the 493 | patent license for this particular work, or (3) arrange, in a manner 494 | consistent with the requirements of this License, to extend the patent 495 | license to downstream recipients. "Knowingly relying" means you have 496 | actual knowledge that, but for the patent license, your conveying the 497 | covered work in a country, or your recipient's use of the covered work 498 | in a country, would infringe one or more identifiable patents in that 499 | country that you have reason to believe are valid. 500 | 501 | If, pursuant to or in connection with a single transaction or 502 | arrangement, you convey, or propagate by procuring conveyance of, a 503 | covered work, and grant a patent license to some of the parties 504 | receiving the covered work authorizing them to use, propagate, modify 505 | or convey a specific copy of the covered work, then the patent license 506 | you grant is automatically extended to all recipients of the covered 507 | work and works based on it. 508 | 509 | A patent license is "discriminatory" if it does not include within 510 | the scope of its coverage, prohibits the exercise of, or is 511 | conditioned on the non-exercise of one or more of the rights that are 512 | specifically granted under this License. You may not convey a covered 513 | work if you are a party to an arrangement with a third party that is 514 | in the business of distributing software, under which you make payment 515 | to the third party based on the extent of your activity of conveying 516 | the work, and under which the third party grants, to any of the 517 | parties who would receive the covered work from you, a discriminatory 518 | patent license (a) in connection with copies of the covered work 519 | conveyed by you (or copies made from those copies), or (b) primarily 520 | for and in connection with specific products or compilations that 521 | contain the covered work, unless you entered into that arrangement, 522 | or that patent license was granted, prior to 28 March 2007. 523 | 524 | Nothing in this License shall be construed as excluding or limiting 525 | any implied license or other defenses to infringement that may 526 | otherwise be available to you under applicable patent law. 527 | 528 | 12. No Surrender of Others' Freedom. 529 | 530 | If conditions are imposed on you (whether by court order, agreement or 531 | otherwise) that contradict the conditions of this License, they do not 532 | excuse you from the conditions of this License. If you cannot convey a 533 | covered work so as to satisfy simultaneously your obligations under this 534 | License and any other pertinent obligations, then as a consequence you may 535 | not convey it at all. For example, if you agree to terms that obligate you 536 | to collect a royalty for further conveying from those to whom you convey 537 | the Program, the only way you could satisfy both those terms and this 538 | License would be to refrain entirely from conveying the Program. 539 | 540 | 13. Remote Network Interaction; Use with the GNU General Public License. 541 | 542 | Notwithstanding any other provision of this License, if you modify the 543 | Program, your modified version must prominently offer all users 544 | interacting with it remotely through a computer network (if your version 545 | supports such interaction) an opportunity to receive the Corresponding 546 | Source of your version by providing access to the Corresponding Source 547 | from a network server at no charge, through some standard or customary 548 | means of facilitating copying of software. This Corresponding Source 549 | shall include the Corresponding Source for any work covered by version 3 550 | of the GNU General Public License that is incorporated pursuant to the 551 | following paragraph. 552 | 553 | Notwithstanding any other provision of this License, you have 554 | permission to link or combine any covered work with a work licensed 555 | under version 3 of the GNU General Public License into a single 556 | combined work, and to convey the resulting work. The terms of this 557 | License will continue to apply to the part which is the covered work, 558 | but the work with which it is combined will remain governed by version 559 | 3 of the GNU General Public License. 560 | 561 | 14. Revised Versions of this License. 562 | 563 | The Free Software Foundation may publish revised and/or new versions of 564 | the GNU Affero General Public License from time to time. Such new versions 565 | will be similar in spirit to the present version, but may differ in detail to 566 | address new problems or concerns. 567 | 568 | Each version is given a distinguishing version number. If the 569 | Program specifies that a certain numbered version of the GNU Affero General 570 | Public License "or any later version" applies to it, you have the 571 | option of following the terms and conditions either of that numbered 572 | version or of any later version published by the Free Software 573 | Foundation. If the Program does not specify a version number of the 574 | GNU Affero General Public License, you may choose any version ever published 575 | by the Free Software Foundation. 576 | 577 | If the Program specifies that a proxy can decide which future 578 | versions of the GNU Affero General Public License can be used, that proxy's 579 | public statement of acceptance of a version permanently authorizes you 580 | to choose that version for the Program. 581 | 582 | Later license versions may give you additional or different 583 | permissions. However, no additional obligations are imposed on any 584 | author or copyright holder as a result of your choosing to follow a 585 | later version. 586 | 587 | 15. Disclaimer of Warranty. 588 | 589 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 590 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 591 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 592 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 593 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 594 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 595 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 596 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 597 | 598 | 16. Limitation of Liability. 599 | 600 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 601 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 602 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 603 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 604 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 605 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 606 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 607 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 608 | SUCH DAMAGES. 609 | 610 | 17. Interpretation of Sections 15 and 16. 611 | 612 | If the disclaimer of warranty and limitation of liability provided 613 | above cannot be given local legal effect according to their terms, 614 | reviewing courts shall apply local law that most closely approximates 615 | an absolute waiver of all civil liability in connection with the 616 | Program, unless a warranty or assumption of liability accompanies a 617 | copy of the Program in return for a fee. 618 | 619 | END OF TERMS AND CONDITIONS 620 | 621 | How to Apply These Terms to Your New Programs 622 | 623 | If you develop a new program, and you want it to be of the greatest 624 | possible use to the public, the best way to achieve this is to make it 625 | free software which everyone can redistribute and change under these terms. 626 | 627 | To do so, attach the following notices to the program. It is safest 628 | to attach them to the start of each source file to most effectively 629 | state the exclusion of warranty; and each file should have at least 630 | the "copyright" line and a pointer to where the full notice is found. 631 | 632 | 633 | Copyright (C) 634 | 635 | This program is free software: you can redistribute it and/or modify 636 | it under the terms of the GNU Affero General Public License as published 637 | by the Free Software Foundation, either version 3 of the License, or 638 | (at your option) any later version. 639 | 640 | This program is distributed in the hope that it will be useful, 641 | but WITHOUT ANY WARRANTY; without even the implied warranty of 642 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 643 | GNU Affero General Public License for more details. 644 | 645 | You should have received a copy of the GNU Affero General Public License 646 | along with this program. If not, see . 647 | 648 | Also add information on how to contact you by electronic and paper mail. 649 | 650 | If your software can interact with users remotely through a computer 651 | network, you should also make sure that it provides a way for users to 652 | get its source. For example, if your program is a web application, its 653 | interface could display a "Source" link that leads users to an archive 654 | of the code. There are many ways you could offer source, and different 655 | solutions will be better for different programs; see section 13 for the 656 | specific requirements. 657 | 658 | You should also get your employer (if you work as a programmer) or school, 659 | if any, to sign a "copyright disclaimer" for the program, if necessary. 660 | For more information on this, and how to apply and follow the GNU AGPL, see 661 | . --------------------------------------------------------------------------------