├── .github └── workflows │ └── test.yml ├── .gitignore ├── CHANGES.rst ├── LICENSE ├── MANIFEST.in ├── README.rst ├── pyproject.toml ├── requests_file.py └── tests └── test_requests_file.py /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | on: [push, pull_request] 3 | 4 | jobs: 5 | pytest: 6 | runs-on: ubuntu-latest 7 | strategy: 8 | matrix: 9 | python-version: ["3.x"] 10 | name: "pytest: Python ${{ matrix.python-version }}" 11 | steps: 12 | - uses: actions/checkout@v2 13 | - name: Setup python 14 | uses: actions/setup-python@v1 15 | with: 16 | python-version: ${{ matrix.python-version }} 17 | - name: Install build dependencies 18 | run: pip install --upgrade setuptools setuptools-scm wheel build 19 | - name: Install package 20 | run: pip install . 21 | - name: Install test dependencies 22 | run: pip install pytest pytest-cov 23 | 24 | - name: Test with pytest 25 | run: pytest --cov=. --cov-report=xml 26 | 27 | - name: Upload coverage to Codecov 28 | uses: codecov/codecov-action@v1 29 | with: 30 | file: ./coverage.xml 31 | fail_ci_if_error: false 32 | 33 | black: 34 | runs-on: ubuntu-latest 35 | steps: 36 | - uses: actions/checkout@v2 37 | - name: Setup python 38 | uses: actions/setup-python@v1 39 | with: 40 | python-version: "3.x" 41 | - name: Install black 42 | run: pip install black 43 | 44 | - name: Run black 45 | run: black --check . 46 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | __pycache__ 3 | build 4 | dist 5 | .eggs 6 | *.egg-info 7 | .*.swp 8 | .DS_Store 9 | -------------------------------------------------------------------------------- /CHANGES.rst: -------------------------------------------------------------------------------- 1 | 2.1.0 (21 May 2024) 2 | =================== 3 | - Set the request property in the returned Response object 4 | 5 | 2.0.0 (29 Jan 2024) 6 | =================== 7 | - Correct a typo in requests_file.py (github PR #21) 8 | - Remove dependency on six (github PR #23) 9 | - Move metadata to pyproject.toml (github PR #26) 10 | - Remove support for Python 2 11 | - Remove support for raw distutils 12 | - Correct homepage link in pyproject.toml (github PR #28) 13 | - Fix black formatting (github PR #27) 14 | 15 | 1.5.1 (25 Apr 2020) 16 | =================== 17 | - Fix python 2.7 compatibility 18 | - Rename test file for pytest 19 | - Add tests via github actions 20 | - Format code with black 21 | 22 | 1.5.0 (23 Apr 2020) 23 | ================== 24 | - Add set_content_length flag to disable on demand setting Content-Length 25 | 26 | 1.4.3 (2 Jan 2018) 27 | ================== 28 | - Skip the permissions test when running as root 29 | - Handle missing locale in tests 30 | 31 | 1.4.2 (28 Apr 2017) 32 | =================== 33 | - Set the response URL to the request URL 34 | 35 | 1.4.1 (13 Oct 2016) 36 | =================== 37 | - Add a wheel distribution 38 | 39 | 1.4 (24 Aug 2015) 40 | ================= 41 | 42 | - Use getprerredencoding instead of nl_langinfo (github issue #1) 43 | - Handle files with a drive component (github issue #2) 44 | - Fix some issues with running the tests on Windows 45 | 46 | 1.3.1 (18 May 2015) 47 | ================== 48 | 49 | - Add python version classifiers to the package info 50 | 51 | 1.3 (18 May 2015) 52 | ================= 53 | 54 | - Fix a crash when closing a file response. 55 | - Use named aliases instead of integers for status codes. 56 | 57 | 1.2 (8 May 2015) 58 | ================= 59 | 60 | - Added support for HEAD requests 61 | 62 | 1.1 (12 Mar 2015) 63 | ================= 64 | 65 | - Added handling for % escapes in URLs 66 | - Proofread the README 67 | 68 | 1.0 (10 Mar 2015) 69 | ================= 70 | 71 | - Initial release 72 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2015 Red Hat, Inc. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE 2 | include README.rst 3 | include requirements.txt 4 | include tests/*.py 5 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Requests-File 2 | ============= 3 | 4 | Requests-File is a transport adapter for use with the `Requests`_ Python 5 | library to allow local filesystem access via file:\/\/ URLs. 6 | 7 | To use: 8 | 9 | .. code-block:: python 10 | 11 | import requests 12 | from requests_file import FileAdapter 13 | 14 | s = requests.Session() 15 | s.mount('file://', FileAdapter()) 16 | 17 | resp = s.get('file:///path/to/file') 18 | 19 | Features 20 | -------- 21 | 22 | - Will open and read local files 23 | - Might set a Content-Length header 24 | - That's about it 25 | 26 | No encoding information is set in the response object, so be careful using 27 | Response.text: the chardet library will be used to convert the file to a 28 | unicode type and it may not detect what you actually want. 29 | 30 | EACCES is converted to a 403 status code, and ENOENT is converted to a 31 | 404. All other IOError types are converted to a 400. 32 | 33 | Contributing 34 | ------------ 35 | 36 | Contributions welcome! Feel free to open a pull request against 37 | https://github.com/dashea/requests-file 38 | 39 | License 40 | ------- 41 | 42 | To maximise compatibility with Requests, this code is licensed under the Apache 43 | license. See LICENSE for more details. 44 | 45 | .. _`Requests`: https://github.com/kennethreitz/requests 46 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools>=61.2", "setuptools_scm[toml]>=3.4.3"] 3 | build-backend = "setuptools.build_meta" 4 | 5 | [project] 6 | name = "requests-file" 7 | authors = [{name = "David Shea", email = "reallylongword@gmail.com"}] 8 | license = {text = "Apache 2.0"} 9 | description = "File transport adapter for Requests" 10 | readme = "README.rst" 11 | classifiers = [ 12 | "Development Status :: 3 - Alpha", 13 | "Environment :: Plugins", 14 | "Intended Audience :: Developers", 15 | "License :: OSI Approved :: Apache Software License", 16 | "Programming Language :: Python :: 3", 17 | ] 18 | urls = {Homepage = "https://github.com/dashea/requests-file"} 19 | dependencies = ["requests>=1.0.0"] 20 | dynamic = ["version"] 21 | 22 | [tool.distutils.bdist_wheel] 23 | universal = 1 24 | 25 | [tool.setuptools] 26 | py-modules = ["requests_file"] 27 | include-package-data = false 28 | 29 | [tool.setuptools_scm] 30 | -------------------------------------------------------------------------------- /requests_file.py: -------------------------------------------------------------------------------- 1 | from requests.adapters import BaseAdapter 2 | from requests.compat import urlparse, unquote 3 | from requests import Response, codes 4 | import errno 5 | import os 6 | import stat 7 | import locale 8 | import io 9 | 10 | try: 11 | from io import BytesIO 12 | except ImportError: 13 | from StringIO import StringIO as BytesIO 14 | 15 | 16 | class FileAdapter(BaseAdapter): 17 | def __init__(self, set_content_length=True): 18 | super(FileAdapter, self).__init__() 19 | self._set_content_length = set_content_length 20 | 21 | def send(self, request, **kwargs): 22 | """Wraps a file, described in request, in a Response object. 23 | 24 | :param request: The PreparedRequest` being "sent". 25 | :returns: a Response object containing the file 26 | """ 27 | 28 | # Check that the method makes sense. Only support GET 29 | if request.method not in ("GET", "HEAD"): 30 | raise ValueError("Invalid request method %s" % request.method) 31 | 32 | # Parse the URL 33 | url_parts = urlparse(request.url) 34 | 35 | # Reject URLs with a hostname component 36 | if url_parts.netloc and url_parts.netloc != "localhost": 37 | raise ValueError("file: URLs with hostname components are not permitted") 38 | 39 | resp = Response() 40 | resp.request = request 41 | 42 | # Open the file, translate certain errors into HTTP responses 43 | # Use urllib's unquote to translate percent escapes into whatever 44 | # they actually need to be 45 | try: 46 | # Split the path on / (the URL directory separator) and decode any 47 | # % escapes in the parts 48 | path_parts = [unquote(p) for p in url_parts.path.split("/")] 49 | 50 | # Strip out the leading empty parts created from the leading /'s 51 | while path_parts and not path_parts[0]: 52 | path_parts.pop(0) 53 | 54 | # If os.sep is in any of the parts, someone fed us some shenanigans. 55 | # Treat is like a missing file. 56 | if any(os.sep in p for p in path_parts): 57 | raise IOError(errno.ENOENT, os.strerror(errno.ENOENT)) 58 | 59 | # Look for a drive component. If one is present, store it separately 60 | # so that a directory separator can correctly be added to the real 61 | # path, and remove any empty path parts between the drive and the path. 62 | # Assume that a part ending with : or | (legacy) is a drive. 63 | if path_parts and ( 64 | path_parts[0].endswith("|") or path_parts[0].endswith(":") 65 | ): 66 | path_drive = path_parts.pop(0) 67 | if path_drive.endswith("|"): 68 | path_drive = path_drive[:-1] + ":" 69 | 70 | while path_parts and not path_parts[0]: 71 | path_parts.pop(0) 72 | else: 73 | path_drive = "" 74 | 75 | # Try to put the path back together 76 | # Join the drive back in, and stick os.sep in front of the path to 77 | # make it absolute. 78 | path = path_drive + os.sep + os.path.join(*path_parts) 79 | 80 | # Check if the drive assumptions above were correct. If path_drive 81 | # is set, and os.path.splitdrive does not return a drive, it wasn't 82 | # really a drive. Put the path together again treating path_drive 83 | # as a normal path component. 84 | if path_drive and not os.path.splitdrive(path): 85 | path = os.sep + os.path.join(path_drive, *path_parts) 86 | 87 | # Use io.open since we need to add a release_conn method, and 88 | # methods can't be added to file objects in python 2. 89 | resp.raw = io.open(path, "rb") 90 | resp.raw.release_conn = resp.raw.close 91 | except IOError as e: 92 | if e.errno == errno.EACCES: 93 | resp.status_code = codes.forbidden 94 | elif e.errno == errno.ENOENT: 95 | resp.status_code = codes.not_found 96 | else: 97 | resp.status_code = codes.bad_request 98 | 99 | # Wrap the error message in a file-like object 100 | # The error message will be localized, try to convert the string 101 | # representation of the exception into a byte stream 102 | resp_str = str(e).encode(locale.getpreferredencoding(False)) 103 | resp.raw = BytesIO(resp_str) 104 | if self._set_content_length: 105 | resp.headers["Content-Length"] = len(resp_str) 106 | 107 | # Add release_conn to the BytesIO object 108 | resp.raw.release_conn = resp.raw.close 109 | else: 110 | resp.status_code = codes.ok 111 | resp.url = request.url 112 | 113 | # If it's a regular file, set the Content-Length 114 | resp_stat = os.fstat(resp.raw.fileno()) 115 | if stat.S_ISREG(resp_stat.st_mode) and self._set_content_length: 116 | resp.headers["Content-Length"] = resp_stat.st_size 117 | 118 | return resp 119 | 120 | def close(self): 121 | pass 122 | -------------------------------------------------------------------------------- /tests/test_requests_file.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import requests 3 | from requests_file import FileAdapter 4 | 5 | import os, stat 6 | import tempfile 7 | import shutil 8 | import platform 9 | 10 | 11 | class FileRequestTestCase(unittest.TestCase): 12 | def setUp(self): 13 | self._session = requests.Session() 14 | self._session.mount("file://", FileAdapter()) 15 | 16 | def _pathToURL(self, path): 17 | """Convert a filesystem path to a URL path""" 18 | urldrive, urlpath = os.path.splitdrive(path) 19 | 20 | # Split the path on the os spearator and recombine it with / as the 21 | # separator. There probably aren't any OS's that allow / as a path 22 | # component, but just in case, encode any remaining /'s. 23 | urlsplit = (part.replace("/", "%2F") for part in urlpath.split(os.sep)) 24 | urlpath = "/".join(urlsplit) 25 | 26 | # Encode /'s in the drive for the imaginary case where that can be a thing 27 | urldrive = urldrive.replace("/", "%2F") 28 | 29 | # Add the leading /. If there is a drive component, this needs to be 30 | # placed before the drive. 31 | urldrive = "/" + urldrive 32 | 33 | return urldrive + urlpath 34 | 35 | def test_fetch_regular(self): 36 | # Fetch this file using requests 37 | with open(__file__, "rb") as f: 38 | testdata = f.read() 39 | response = self._session.get( 40 | "file://%s" % self._pathToURL(os.path.abspath(__file__)) 41 | ) 42 | 43 | self.assertEqual(response.status_code, requests.codes.ok) 44 | self.assertEqual(response.headers["Content-Length"], len(testdata)) 45 | self.assertEqual(response.content, testdata) 46 | 47 | response.close() 48 | 49 | def test_fetch_missing(self): 50 | # Fetch a file that (hopefully) doesn't exist, look for a 404 51 | response = self._session.get("file:///no/such/path") 52 | self.assertEqual(response.status_code, requests.codes.not_found) 53 | self.assertTrue(response.text) 54 | response.close() 55 | 56 | @unittest.skipIf( 57 | hasattr(os, "geteuid") and os.geteuid() == 0, 58 | "Skipping permissions test since running as root", 59 | ) 60 | def test_fetch_no_access(self): 61 | # Create a file and remove read permissions, try to get a 403 62 | # probably doesn't work on windows 63 | with tempfile.NamedTemporaryFile() as tmp: 64 | os.chmod(tmp.name, 0) 65 | response = self._session.get( 66 | "file://%s" % self._pathToURL(os.path.abspath(tmp.name)) 67 | ) 68 | 69 | self.assertEqual(response.status_code, requests.codes.forbidden) 70 | self.assertTrue(response.text) 71 | 72 | response.close() 73 | 74 | @unittest.skipIf(platform.system() == "Windows", "skipping locale test on windows") 75 | def test_fetch_missing_localized(self): 76 | # Make sure translated error messages don't cause any problems 77 | import locale 78 | 79 | saved_locale = locale.setlocale(locale.LC_MESSAGES, None) 80 | try: 81 | locale.setlocale(locale.LC_MESSAGES, "ru_RU.UTF-8") 82 | response = self._session.get("file:///no/such/path") 83 | self.assertEqual(response.status_code, requests.codes.not_found) 84 | self.assertTrue(response.text) 85 | response.close() 86 | except locale.Error: 87 | unittest.SkipTest("ru_RU.UTF-8 locale not available") 88 | finally: 89 | locale.setlocale(locale.LC_MESSAGES, saved_locale) 90 | 91 | def test_head(self): 92 | # Check that HEAD returns the content-length 93 | testlen = os.stat(__file__).st_size 94 | response = self._session.head( 95 | "file://%s" % self._pathToURL(os.path.abspath(__file__)) 96 | ) 97 | 98 | self.assertEqual(response.status_code, requests.codes.ok) 99 | self.assertEqual(response.headers["Content-Length"], testlen) 100 | 101 | response.close() 102 | 103 | def test_fetch_post(self): 104 | # Make sure that non-GET methods are rejected 105 | self.assertRaises( 106 | ValueError, 107 | self._session.post, 108 | ("file://%s" % self._pathToURL(os.path.abspath(__file__))), 109 | ) 110 | 111 | def test_fetch_nonlocal(self): 112 | # Make sure that network locations are rejected 113 | self.assertRaises( 114 | ValueError, 115 | self._session.get, 116 | ("file://example.com%s" % self._pathToURL(os.path.abspath(__file__))), 117 | ) 118 | self.assertRaises( 119 | ValueError, 120 | self._session.get, 121 | ("file://localhost:8080%s" % self._pathToURL(os.path.abspath(__file__))), 122 | ) 123 | 124 | # localhost is ok, though 125 | with open(__file__, "rb") as f: 126 | testdata = f.read() 127 | response = self._session.get( 128 | "file://localhost%s" % self._pathToURL(os.path.abspath(__file__)) 129 | ) 130 | self.assertEqual(response.status_code, requests.codes.ok) 131 | self.assertEqual(response.content, testdata) 132 | response.close() 133 | 134 | def test_funny_names(self): 135 | testdata = "yo wassup man\n".encode("ascii") 136 | tmpdir = tempfile.mkdtemp() 137 | 138 | try: 139 | with open(os.path.join(tmpdir, "spa ces"), "w+b") as space_file: 140 | space_file.write(testdata) 141 | space_file.flush() 142 | response = self._session.get( 143 | "file://%s/spa%%20ces" % self._pathToURL(tmpdir) 144 | ) 145 | self.assertEqual(response.status_code, requests.codes.ok) 146 | self.assertEqual(response.content, testdata) 147 | response.close() 148 | 149 | with open(os.path.join(tmpdir, "per%cent"), "w+b") as percent_file: 150 | percent_file.write(testdata) 151 | percent_file.flush() 152 | response = self._session.get( 153 | "file://%s/per%%25cent" % self._pathToURL(tmpdir) 154 | ) 155 | self.assertEqual(response.status_code, requests.codes.ok) 156 | self.assertEqual(response.content, testdata) 157 | response.close() 158 | 159 | # percent-encoded directory separators should be rejected 160 | with open(os.path.join(tmpdir, "badname"), "w+b") as bad_file: 161 | response = self._session.get( 162 | "file://%s%%%Xbadname" % (self._pathToURL(tmpdir), ord(os.sep)) 163 | ) 164 | self.assertEqual(response.status_code, requests.codes.not_found) 165 | response.close() 166 | 167 | finally: 168 | shutil.rmtree(tmpdir) 169 | 170 | def test_close(self): 171 | # Open a request for this file 172 | response = self._session.get( 173 | "file://%s" % self._pathToURL(os.path.abspath(__file__)) 174 | ) 175 | 176 | # Try closing it 177 | response.close() 178 | 179 | def test_missing_close(self): 180 | # Make sure non-200 responses can be closed 181 | response = self._session.get("file:///no/such/path") 182 | response.close() 183 | 184 | @unittest.skipIf(platform.system() != "Windows", "skipping windows URL test") 185 | def test_windows_legacy(self): 186 | """Test |-encoded drive characters on Windows""" 187 | with open(__file__, "rb") as f: 188 | testdata = f.read() 189 | 190 | drive, path = os.path.splitdrive(os.path.abspath(__file__)) 191 | response = self._session.get( 192 | "file:///%s|%s" % (drive[:-1], path.replace(os.sep, "/")) 193 | ) 194 | self.assertEqual(response.status_code, requests.codes.ok) 195 | self.assertEqual(response.headers["Content-Length"], len(testdata)) 196 | self.assertEqual(response.content, testdata) 197 | response.close() 198 | --------------------------------------------------------------------------------