├── pydrive2 ├── __init__.py ├── test │ ├── __init__.py │ ├── credentials │ │ └── .keepme │ ├── settings │ │ ├── local │ │ │ └── .keepme │ │ ├── test_oauth_test_05.yaml │ │ ├── test_oauth_test_03.yaml │ │ ├── test_oauth_test_08.yaml │ │ ├── test_oauth_default.yaml │ │ ├── test_oauth_test_01.yaml │ │ ├── test_oauth_test_04.yaml │ │ ├── default.yaml │ │ ├── test_oauth_test_07.yaml │ │ ├── test_oauth_test_09.yaml │ │ ├── test_oauth_test_06.yaml │ │ └── test_oauth_test_02.yaml │ ├── .gitignore │ ├── client_secrets.json │ ├── test_drive.py │ ├── test_apiattr.py │ ├── test_util.py │ ├── README.rst │ ├── test_filelist.py │ ├── test_oauth.py │ └── test_fs.py ├── fs │ ├── __init__.py │ ├── utils.py │ └── spec.py ├── __pyinstaller │ ├── __init__.py │ ├── hook-googleapiclient.py │ └── test_hook-googleapiclient.py ├── drive.py ├── apiattr.py ├── settings.py └── auth.py ├── docs ├── genindex.rst ├── requirements.txt ├── README.md ├── pydrive2.rst ├── index.rst ├── filelist.rst ├── fsspec.rst ├── quickstart.rst ├── oauth.rst ├── conf.py └── filemanagement.rst ├── CHANGES ├── .gitignore ├── MANIFEST.in ├── .flake8 ├── .pre-commit-config.yaml ├── .github ├── workflows │ ├── pre-commit.yml │ ├── docs.yml │ ├── publish.yml │ └── test.yml └── dependabot.yml ├── CONTRIBUTING.rst ├── examples ├── Upload-and-autoconvert-to-Google-Drive-Format-Example │ ├── README │ ├── settings.yaml │ └── upload.py ├── using_folders.py └── strip_bom_example.py ├── pyproject.toml ├── README.rst └── LICENSE /pydrive2/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /pydrive2/test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /pydrive2/test/credentials/.keepme: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/genindex.rst: -------------------------------------------------------------------------------- 1 | Index 2 | ===== 3 | -------------------------------------------------------------------------------- /pydrive2/test/settings/local/.keepme: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | sphinx 2 | furo 3 | -------------------------------------------------------------------------------- /CHANGES: -------------------------------------------------------------------------------- 1 | v1.0.0, Aug 16, 2013 -- Initial release. 2 | -------------------------------------------------------------------------------- /pydrive2/test/.gitignore: -------------------------------------------------------------------------------- 1 | credentials/* 2 | settings/local/* 3 | 4 | client_secrets.json 5 | *.p12 6 | -------------------------------------------------------------------------------- /pydrive2/fs/__init__.py: -------------------------------------------------------------------------------- 1 | from pydrive2.fs.spec import GDriveFileSystem 2 | 3 | __all__ = ["GDriveFileSystem"] 4 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_05.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: file 2 | client_config_file: client_secrets.json 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *~ 3 | 4 | *.egg-info 5 | dist 6 | .cache 7 | 8 | .env 9 | .idea 10 | pip-wheel-metadata 11 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include AUTHORS 2 | include CHANGES 3 | include LICENSE 4 | include MANIFEST.in 5 | include README.rst 6 | recursive-include docs * 7 | recursive-include pydrive2/test * 8 | recursive-exclude * *.py[co] 9 | -------------------------------------------------------------------------------- /pydrive2/__pyinstaller/__init__.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | 4 | def get_hook_dirs(): 5 | return [os.path.dirname(__file__)] 6 | 7 | 8 | def get_PyInstaller_tests(): 9 | return [os.path.dirname(__file__)] 10 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_03.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: file 2 | client_config_file: client_secrets.json 3 | 4 | save_credentials: False 5 | 6 | oauth_scope: 7 | - https://www.googleapis.com/auth/drive 8 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_08.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: service 2 | service_config: 3 | client_json_file_path: /tmp/pydrive2/credentials.json 4 | 5 | save_credentials: False 6 | 7 | oauth_scope: 8 | - https://www.googleapis.com/auth/drive 9 | -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = 3 | # Whitespace before ':' 4 | E203, 5 | # Too many leading '#' for block comment 6 | E266, 7 | # Line break occurred before a binary operator 8 | W503 9 | max-line-length = 89 10 | select = B,C,E,F,W,T4,B9 11 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - hooks: 3 | - id: black 4 | language_version: python3 5 | repo: https://github.com/psf/black 6 | rev: 23.12.1 7 | - hooks: 8 | - id: flake8 9 | language_version: python3 10 | repo: https://github.com/PyCQA/flake8 11 | rev: 6.0.0 12 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_default.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: file 2 | client_config_file: client_secrets.json 3 | 4 | save_credentials: True 5 | save_credentials_backend: file 6 | save_credentials_file: credentials/1.dat 7 | 8 | oauth_scope: 9 | - https://www.googleapis.com/auth/drive 10 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_01.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: file 2 | client_config_file: client_secrets.json 3 | 4 | save_credentials: True 5 | save_credentials_backend: file 6 | save_credentials_file: credentials/1.dat 7 | 8 | oauth_scope: 9 | - https://www.googleapis.com/auth/drive 10 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_04.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: file 2 | client_config_file: client_secrets.json 3 | 4 | save_credentials: True 5 | save_credentials_backend: file 6 | save_credentials_file: credentials/4.dat 7 | 8 | oauth_scope: 9 | - https://www.googleapis.com/auth/drive 10 | -------------------------------------------------------------------------------- /.github/workflows/pre-commit.yml: -------------------------------------------------------------------------------- 1 | name: pre-commit 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: [main] 7 | 8 | jobs: 9 | pre-commit: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: actions/checkout@v4 13 | - uses: actions/setup-python@v5 14 | - uses: pre-commit/action@v3.0.1 15 | -------------------------------------------------------------------------------- /pydrive2/test/settings/default.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: service 2 | service_config: 3 | client_json_file_path: /tmp/pydrive2/credentials.json 4 | 5 | save_credentials: True 6 | save_credentials_backend: file 7 | save_credentials_file: credentials/default.dat 8 | 9 | oauth_scope: 10 | - https://www.googleapis.com/auth/drive 11 | -------------------------------------------------------------------------------- /pydrive2/__pyinstaller/hook-googleapiclient.py: -------------------------------------------------------------------------------- 1 | from PyInstaller.utils.hooks import ( # pylint: disable=import-error 2 | copy_metadata, 3 | collect_data_files, 4 | ) 5 | 6 | datas = copy_metadata("google-api-python-client") 7 | datas += collect_data_files( 8 | "googleapiclient", excludes=["*.txt", "**/__pycache__"] 9 | ) 10 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_07.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: service 2 | service_config: 3 | client_json_file_path: /tmp/pydrive2/credentials.json 4 | 5 | save_credentials: True 6 | save_credentials_backend: file 7 | save_credentials_file: credentials/7.dat 8 | 9 | oauth_scope: 10 | - https://www.googleapis.com/auth/drive 11 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_09.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: service 2 | service_config: 3 | client_json_file_path: /tmp/pydrive2/credentials.json 4 | 5 | save_credentials: True 6 | save_credentials_backend: file 7 | save_credentials_file: credentials/9.dat 8 | 9 | oauth_scope: 10 | - https://www.googleapis.com/auth/drive 11 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | updates: 4 | - directory: "/" 5 | package-ecosystem: "pip" 6 | schedule: 7 | interval: "daily" 8 | labels: 9 | - "maintenance" 10 | 11 | - directory: "/" 12 | package-ecosystem: "github-actions" 13 | schedule: 14 | interval: "daily" 15 | labels: 16 | - "maintenance" 17 | -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_06.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: service 2 | service_config: 3 | client_service_email: your-service-account-email 4 | client_pkcs12_file_path: your-file-path.p12 5 | 6 | save_credentials: True 7 | save_credentials_backend: file 8 | save_credentials_file: credentials/6.dat 9 | 10 | oauth_scope: 11 | - https://www.googleapis.com/auth/drive 12 | -------------------------------------------------------------------------------- /pydrive2/test/client_secrets.json: -------------------------------------------------------------------------------- 1 | {"web":{"client_id":"47794215776-et2ir6ngpul4m4pn95tnfrtvuuahrvpt.apps.googleusercontent.com","project_id":"dvc-pydrive","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"FWSfDKs2i_0z_KQEoHAfqU2G","redirect_uris":["http://localhost:8080/"]}} -------------------------------------------------------------------------------- /pydrive2/test/settings/test_oauth_test_02.yaml: -------------------------------------------------------------------------------- 1 | client_config_backend: settings 2 | client_config: 3 | client_id: 47794215776-cd9ssb6a4vv5otkq6n0iadpgc4efgjb1.apps.googleusercontent.com 4 | client_secret: i2gerGA7uBjZbR08HqSOSt9Z 5 | 6 | save_credentials: True 7 | save_credentials_backend: file 8 | save_credentials_file: credentials/2.dat 9 | 10 | oauth_scope: 11 | - https://www.googleapis.com/auth/drive 12 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | Contributing guidelines 2 | ======================= 3 | 4 | How to become a contributor and submit your own code 5 | ---------------------------------------------------- 6 | 7 | TODO 8 | 9 | Contributing code 10 | ~~~~~~~~~~~~~~~~~ 11 | 12 | If you have improvements to PyDrive2, send us your pull requests! For those 13 | just getting started, Github has a `howto `_. 14 | -------------------------------------------------------------------------------- /examples/Upload-and-autoconvert-to-Google-Drive-Format-Example/README: -------------------------------------------------------------------------------- 1 | This script uploads a file to a folder on Google Drive. If the file can be 2 | converted into the Google format, such as a Google Sheet, it will be converted. 3 | 4 | To run the script you need to complete the following steps: 5 | 1. Update settings.yaml with your: 6 | - client ID, 7 | - client secret, and 8 | - credential storage path 9 | 2. Update upload.py with the location you save your settings.yaml file. This 10 | can be an absolute or relative path. 11 | 12 | 13 | This example is adapted from Evren Yurtesen (https://github.com/yurtesen/) 14 | with his consent. 15 | Originally posted here: https://github.com/prasmussen/gdrive/issues/154 16 | -------------------------------------------------------------------------------- /examples/Upload-and-autoconvert-to-Google-Drive-Format-Example/settings.yaml: -------------------------------------------------------------------------------- 1 | # Original author: Evren Yurtesen - https://github.com/yurtesen/ 2 | 3 | client_config_backend: settings 4 | client_config: 5 | client_id: 6 | client_secret: 7 | auth_uri: https://accounts.google.com/o/oauth2/auth 8 | token_uri: https://accounts.google.com/o/oauth2/token 9 | redirect_uri: urn:ietf:wg:oauth:2.0:oob 10 | revoke_uri: 11 | 12 | save_credentials: True 13 | save_credentials_backend: file 14 | save_credentials_file: 15 | 16 | get_refresh_token: True 17 | 18 | oauth_scope: 19 | - https://www.googleapis.com/auth/drive.file 20 | -------------------------------------------------------------------------------- /.github/workflows/docs.yml: -------------------------------------------------------------------------------- 1 | name: Build docs 2 | 3 | on: 4 | push: 5 | pull_request: 6 | 7 | jobs: 8 | build: 9 | runs-on: ubuntu-latest 10 | 11 | steps: 12 | - uses: actions/checkout@v4 13 | 14 | - uses: actions/setup-python@v5 15 | with: 16 | python-version: '3.8' 17 | 18 | - name: Build 19 | run: | 20 | pip install -U -r docs/requirements.txt 21 | pip install ".[fsspec]" 22 | sphinx-build docs dist/site -b dirhtml -a 23 | 24 | - name: Publish 25 | if: ${{ github.event_name == 'push' && github.ref_name == 'main' }} 26 | uses: peaceiris/actions-gh-pages@v4 27 | with: 28 | github_token: ${{ secrets.GITHUB_TOKEN }} 29 | publish_dir: dist/site 30 | -------------------------------------------------------------------------------- /.github/workflows/publish.yml: -------------------------------------------------------------------------------- 1 | on: 2 | push: 3 | branches: 4 | - main 5 | release: 6 | types: [published] 7 | workflow_dispatch: 8 | 9 | name: Publish 10 | 11 | jobs: 12 | publish: 13 | environment: pypi 14 | permissions: 15 | contents: read 16 | id-token: write 17 | runs-on: ubuntu-latest 18 | 19 | steps: 20 | - uses: actions/checkout@v4 21 | 22 | - uses: actions/setup-python@v5 23 | with: 24 | python-version: '3.8' 25 | 26 | - name: Install pypa/build 27 | run: python -m pip install build 28 | 29 | - name: Build the package 30 | run: | 31 | python -m build --sdist --wheel \ 32 | --outdir dist/ . 33 | 34 | - name: Publish 35 | if: github.event.action == 'published' 36 | uses: pypa/gh-action-pypi-publish@release/v1 37 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | This document outlines how to rebuild the documentation. 2 | 3 | ## Setup 4 | 5 | - Install Sphinx: `pip install sphinx` or `apt-get install python-sphinx` 6 | - Install theme: `pip install furo` 7 | - Build site: `sphinx-build docs dist/site -b dirhtml -a` 8 | 9 | Updating GitHub Pages: 10 | 11 | ```bash 12 | cd dist/site 13 | git init 14 | git add . 15 | git commit -m "update pages" 16 | git branch -M gh-pages 17 | git push -f git@github.com:iterative/PyDrive2 gh-pages 18 | ``` 19 | 20 | ## Contributing 21 | 22 | If code files were added, the easiest way to reflect code changes in the 23 | documentation by referencing the file from within `pydrive.rst`. 24 | 25 | If a non-code related file was added (it has to have the `.rst` ending), 26 | then add the file name to the list of names under "Table of Contents" 27 | in `index.rst`. Make sure to add the file name excluding the `.rst` file ending. 28 | -------------------------------------------------------------------------------- /pydrive2/test/test_drive.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from pydrive2.auth import GoogleAuth 4 | from pydrive2.drive import GoogleDrive 5 | from pydrive2.test.test_util import ( 6 | pydrive_retry, 7 | setup_credentials, 8 | settings_file_path, 9 | ) 10 | 11 | 12 | class GoogleDriveTest(unittest.TestCase): 13 | """Tests basic operations on meta-data information of the linked Google 14 | Drive account. 15 | """ 16 | 17 | @classmethod 18 | def setup_class(cls): 19 | setup_credentials() 20 | 21 | cls.ga = GoogleAuth(settings_file_path("default.yaml")) 22 | cls.ga.ServiceAuth() 23 | 24 | def test_01_About_Request(self): 25 | drive = GoogleDrive(self.ga) 26 | 27 | about_object = pydrive_retry(drive.GetAbout) 28 | self.assertTrue(about_object is not None, "About object not loading.") 29 | 30 | 31 | if __name__ == "__main__": 32 | unittest.main() 33 | -------------------------------------------------------------------------------- /examples/using_folders.py: -------------------------------------------------------------------------------- 1 | from pydrive2.auth import GoogleAuth 2 | from pydrive2.drive import GoogleDrive 3 | 4 | gauth = GoogleAuth() 5 | gauth.LocalWebserverAuth() 6 | 7 | drive = GoogleDrive(gauth) 8 | 9 | # Create folder. 10 | folder_metadata = { 11 | "title": "", 12 | # The mimetype defines this new file as a folder, so don't change this. 13 | "mimeType": "application/vnd.google-apps.folder", 14 | } 15 | folder = drive.CreateFile(folder_metadata) 16 | folder.Upload() 17 | 18 | # Get folder info and print to screen. 19 | folder_title = folder["title"] 20 | folder_id = folder["id"] 21 | print("title: %s, id: %s" % (folder_title, folder_id)) 22 | 23 | # Upload file to folder. 24 | f = drive.CreateFile( 25 | {"parents": [{"kind": "drive#fileLink", "id": folder_id}]} 26 | ) 27 | 28 | # Make sure to add the path to the file to upload below. 29 | f.SetContentFile("") 30 | f.Upload() 31 | -------------------------------------------------------------------------------- /docs/pydrive2.rst: -------------------------------------------------------------------------------- 1 | pydrive2 package 2 | ================ 3 | 4 | pydrive2.apiattr module 5 | ----------------------- 6 | 7 | .. automodule:: pydrive2.apiattr 8 | :members: 9 | :undoc-members: 10 | :show-inheritance: 11 | 12 | pydrive2.auth module 13 | -------------------- 14 | 15 | .. automodule:: pydrive2.auth 16 | :members: 17 | :undoc-members: 18 | :show-inheritance: 19 | 20 | pydrive2.drive module 21 | --------------------- 22 | 23 | .. automodule:: pydrive2.drive 24 | :members: 25 | :undoc-members: 26 | :show-inheritance: 27 | 28 | pydrive2.files module 29 | --------------------- 30 | 31 | .. automodule:: pydrive2.files 32 | :members: 33 | :undoc-members: 34 | :show-inheritance: 35 | 36 | pydrive2.settings module 37 | ------------------------ 38 | 39 | .. automodule:: pydrive2.settings 40 | :members: 41 | :undoc-members: 42 | :show-inheritance: 43 | 44 | pydrive2.fs module 45 | ------------------------ 46 | 47 | .. autoclass:: pydrive2.fs.GDriveFileSystem 48 | :show-inheritance: 49 | -------------------------------------------------------------------------------- /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | on: 2 | push: 3 | branches: 4 | - main 5 | pull_request_target: 6 | 7 | name: Tests 8 | 9 | jobs: 10 | authorize: 11 | environment: 12 | ${{ (github.event_name == 'pull_request_target' && 13 | github.event.pull_request.head.repo.full_name != github.repository) && 14 | 'external' || 'internal' }} 15 | runs-on: ubuntu-latest 16 | steps: 17 | - run: echo ✓ 18 | test: 19 | needs: authorize 20 | runs-on: ${{ matrix.os }} 21 | 22 | strategy: 23 | fail-fast: false 24 | max-parallel: 3 25 | matrix: 26 | os: [ubuntu-latest, macos-latest, windows-latest] 27 | pyv: ["3.9", "3.10", "3.11", "3.12"] 28 | 29 | steps: 30 | - uses: actions/checkout@v4 31 | with: 32 | # NOTE: needed for pull_request_target to use PR code 33 | ref: ${{ github.event.pull_request.head.sha || github.ref }} 34 | 35 | - uses: actions/setup-python@v5 36 | with: 37 | python-version: ${{ matrix.pyv }} 38 | 39 | - name: Install dependencies 40 | run: python -m pip install -e '.[fsspec, tests]' 41 | 42 | - name: Test 43 | run: python -m pytest -m "not manual" 44 | env: 45 | GDRIVE_USER_CREDENTIALS_DATA: ${{ secrets.GDRIVE_USER_CREDENTIALS_DATA }} 46 | -------------------------------------------------------------------------------- /pydrive2/test/test_apiattr.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from pydrive2.auth import GoogleAuth 4 | from pydrive2.drive import GoogleDrive 5 | from pydrive2.test.test_util import ( 6 | pydrive_retry, 7 | setup_credentials, 8 | settings_file_path, 9 | ) 10 | 11 | 12 | class ApiAttributeTest(unittest.TestCase): 13 | """Test ApiAttr functions.""" 14 | 15 | @classmethod 16 | def setup_class(cls): 17 | setup_credentials() 18 | 19 | def test_UpdateMetadataNotInfinitelyNesting(self): 20 | # Verify 'metadata' field present. 21 | self.assertTrue(self.file1.metadata is not None) 22 | pydrive_retry(self.file1.UpdateMetadata) 23 | 24 | # Verify 'metadata' field still present. 25 | self.assertTrue(self.file1.metadata is not None) 26 | # Ensure no 'metadata' field in 'metadata' (i.e. nested). 27 | self.assertTrue("metadata" not in self.file1.metadata) 28 | 29 | def setUp(self): 30 | ga = GoogleAuth(settings_file_path("default.yaml")) 31 | ga.ServiceAuth() 32 | self.drive = GoogleDrive(ga) 33 | self.file1 = self.drive.CreateFile() 34 | pydrive_retry(self.file1.Upload) 35 | 36 | def tearDown(self): 37 | pydrive_retry(self.file1.Delete) 38 | 39 | 40 | if __name__ == "__main__": 41 | unittest.main() 42 | -------------------------------------------------------------------------------- /pydrive2/__pyinstaller/test_hook-googleapiclient.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | 3 | from PyInstaller import __main__ as pyi_main 4 | 5 | 6 | # NOTE: importlib.resources.contents is available in py3.7+, but due to how 7 | # pyinstaller handles importlib, we need to use the importlib_resources 8 | # backport if there are any resources methods that are not available in a given 9 | # python version, which ends up being py<3.10 10 | _APP_SOURCE = """ 11 | import sys 12 | if sys.version_info >= (3, 10): 13 | from importlib.resources import contents 14 | else: 15 | from importlib_resources import contents 16 | 17 | import pydrive2.files 18 | 19 | 20 | cache_files = contents( 21 | "googleapiclient.discovery_cache.documents" 22 | ) 23 | assert len(cache_files) > 0 24 | """ 25 | 26 | 27 | def test_pyi_hook_google_api_client(tmp_path): 28 | app_name = "userapp" 29 | workpath = tmp_path / "build" 30 | distpath = tmp_path / "dist" 31 | app = tmp_path / f"{app_name}.py" 32 | app.write_text(_APP_SOURCE) 33 | pyi_main.run( 34 | [ 35 | "--workpath", 36 | str(workpath), 37 | "--distpath", 38 | str(distpath), 39 | "--specpath", 40 | str(tmp_path), 41 | str(app), 42 | ], 43 | ) 44 | subprocess.run([str(distpath / app_name / app_name)], check=True) 45 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. PyDrive2 documentation master file, created by 2 | sphinx-quickstart on Sun Jun 12 23:01:40 2016. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to PyDrive2's documentation! 7 | ==================================== 8 | 9 | PyDrive2 is a wrapper library of `google-api-python-client`_ that simplifies many common Google Drive API tasks. 10 | 11 | Project Info 12 | ============ 13 | 14 | - Package: `https://pypi.python.org/pypi/PyDrive2 `_ 15 | - Documentation: `https://docs.iterative.ai/PyDrive2 `_ 16 | - Source: `https://github.com/iterative/PyDrive2 `_ 17 | - Changelog: `https://github.com/iterative/PyDrive2/releases `_ 18 | 19 | How to install 20 | ============== 21 | 22 | You can install PyDrive2 with regular ``pip`` command. 23 | 24 | :: 25 | 26 | $ pip install PyDrive2 27 | 28 | To install the current development version from GitHub, use: 29 | 30 | :: 31 | 32 | $ pip install git+https://github.com/iterative/PyDrive2.git#egg=PyDrive2 33 | 34 | 35 | .. _`google-api-python-client`: https://github.com/google/google-api-python-client 36 | 37 | Table of Contents 38 | ================= 39 | 40 | .. toctree:: 41 | :maxdepth: 2 42 | 43 | quickstart 44 | oauth 45 | filemanagement 46 | filelist 47 | fsspec 48 | pydrive2 49 | genindex 50 | -------------------------------------------------------------------------------- /pydrive2/fs/utils.py: -------------------------------------------------------------------------------- 1 | import io 2 | 3 | 4 | class IterStream(io.RawIOBase): 5 | """Wraps an iterator yielding bytes as a file object""" 6 | 7 | def __init__(self, iterator): # pylint: disable=super-init-not-called 8 | self.iterator = iterator 9 | self.leftover = b"" 10 | 11 | def readable(self): 12 | return True 13 | 14 | def writable(self) -> bool: 15 | return False 16 | 17 | # Python 3 requires only .readinto() method, it still uses other ones 18 | # under some circumstances and falls back if those are absent. Since 19 | # iterator already constructs byte strings for us, .readinto() is not the 20 | # most optimal, so we provide .read1() too. 21 | 22 | def readinto(self, b): 23 | try: 24 | n = len(b) # We're supposed to return at most this much 25 | chunk = self.leftover or next(self.iterator) 26 | output, self.leftover = chunk[:n], chunk[n:] 27 | 28 | n_out = len(output) 29 | b[:n_out] = output 30 | return n_out 31 | except StopIteration: 32 | return 0 # indicate EOF 33 | 34 | readinto1 = readinto 35 | 36 | def read1(self, n=-1): 37 | try: 38 | chunk = self.leftover or next(self.iterator) 39 | except StopIteration: 40 | return b"" 41 | 42 | # Return an arbitrary number or bytes 43 | if n <= 0: 44 | self.leftover = b"" 45 | return chunk 46 | 47 | output, self.leftover = chunk[:n], chunk[n:] 48 | return output 49 | 50 | def peek(self, n): 51 | while len(self.leftover) < n: 52 | try: 53 | self.leftover += next(self.iterator) 54 | except StopIteration: 55 | break 56 | return self.leftover[:n] 57 | -------------------------------------------------------------------------------- /pydrive2/drive.py: -------------------------------------------------------------------------------- 1 | from .apiattr import ApiAttributeMixin 2 | from .files import GoogleDriveFile 3 | from .files import GoogleDriveFileList 4 | from .auth import LoadAuth 5 | 6 | 7 | class GoogleDrive(ApiAttributeMixin): 8 | """Main Google Drive class.""" 9 | 10 | def __init__(self, auth=None): 11 | """Create an instance of GoogleDrive. 12 | 13 | :param auth: authorized GoogleAuth instance. 14 | :type auth: pydrive2.auth.GoogleAuth. 15 | """ 16 | ApiAttributeMixin.__init__(self) 17 | self.auth = auth 18 | 19 | def CreateFile(self, metadata=None): 20 | """Create an instance of GoogleDriveFile with auth of this instance. 21 | 22 | This method would not upload a file to GoogleDrive. 23 | 24 | :param metadata: file resource to initialize GoogleDriveFile with. 25 | :type metadata: dict. 26 | :returns: pydrive2.files.GoogleDriveFile -- initialized with auth of this 27 | instance. 28 | """ 29 | return GoogleDriveFile(auth=self.auth, metadata=metadata) 30 | 31 | def ListFile(self, param=None): 32 | """Create an instance of GoogleDriveFileList with auth of this instance. 33 | 34 | This method will not fetch from Files.List(). 35 | 36 | :param param: parameter to be sent to Files.List(). 37 | :type param: dict. 38 | :returns: pydrive2.files.GoogleDriveFileList -- initialized with auth of 39 | this instance. 40 | """ 41 | return GoogleDriveFileList(auth=self.auth, param=param) 42 | 43 | @LoadAuth 44 | def GetAbout(self): 45 | """Return information about the Google Drive of the auth instance. 46 | 47 | :returns: A dictionary of Google Drive information like user, usage, quota etc. 48 | """ 49 | return self.auth.service.about().get().execute(http=self.http) 50 | -------------------------------------------------------------------------------- /examples/strip_bom_example.py: -------------------------------------------------------------------------------- 1 | from pydrive2.auth import GoogleAuth 2 | from pydrive2.drive import GoogleDrive 3 | 4 | # Authenticate the client. 5 | gauth = GoogleAuth() 6 | gauth.LocalWebserverAuth() 7 | drive = GoogleDrive(gauth) 8 | 9 | # Create a file, set content, and upload. 10 | file1 = drive.CreateFile() 11 | original_file_content = "Generic, non-exhaustive\n ASCII test string." 12 | file1.SetContentString(original_file_content) 13 | # {'convert': True} triggers conversion to a Google Drive document. 14 | file1.Upload({"convert": True}) 15 | 16 | 17 | # Download the file. 18 | file2 = drive.CreateFile({"id": file1["id"]}) 19 | 20 | # Print content before download. 21 | print("Original text:") 22 | print(bytes(original_file_content.encode("unicode-escape"))) 23 | print("Number of chars: %d" % len(original_file_content)) 24 | print("") 25 | # Original text: 26 | # Generic, non-exhaustive\n ASCII test string. 27 | # Number of chars: 43 28 | 29 | 30 | # Download document as text file WITH the BOM and print the contents. 31 | content_with_bom = file2.GetContentString(mimetype="text/plain") 32 | print("Content with BOM:") 33 | print(bytes(content_with_bom.encode("unicode-escape"))) 34 | print("Number of chars: %d" % len(content_with_bom)) 35 | print("") 36 | # Content with BOM: 37 | # \ufeffGeneric, non-exhaustive\r\n ASCII test string. 38 | # Number of chars: 45 39 | 40 | 41 | # Download document as text file WITHOUT the BOM and print the contents. 42 | content_without_bom = file2.GetContentString( 43 | mimetype="text/plain", remove_bom=True 44 | ) 45 | print("Content without BOM:") 46 | print(bytes(content_without_bom.encode("unicode-escape"))) 47 | print("Number of chars: %d" % len(content_without_bom)) 48 | print("") 49 | # Content without BOM: 50 | # Generic, non-exhaustive\r\n ASCII test string. 51 | # Number of chars: 44 52 | 53 | # *NOTE*: When downloading a Google Drive document as text file, line-endings 54 | # are converted to the Windows-style: \r\n. 55 | 56 | 57 | # Delete the file as necessary. 58 | file1.Delete() 59 | -------------------------------------------------------------------------------- /examples/Upload-and-autoconvert-to-Google-Drive-Format-Example/upload.py: -------------------------------------------------------------------------------- 1 | # Original author: Evren Yurtesen - https://github.com/yurtesen/ 2 | 3 | """ 4 | Uploads a file to a specific folder in Google Drive and converts it to a 5 | Google Doc/Sheet/etc. if possible. 6 | 7 | usage: upload.py 8 | example usage: upload.py 0B5XXXXY9KddXXXXXXXA2c3ZXXXX /path/to/my/file 9 | """ 10 | import sys 11 | from os import path 12 | from pydrive2.auth import GoogleAuth 13 | from pydrive2.drive import GoogleDrive 14 | from pydrive2.settings import LoadSettingsFile 15 | 16 | # Update this value to the correct location. 17 | # e.g. "/usr/local/scripts/pydrive/settings.yaml" 18 | PATH_TO_SETTINGS_FILE = None 19 | assert PATH_TO_SETTINGS_FILE is not None # Fail if path not specified. 20 | 21 | gauth = GoogleAuth() 22 | gauth.settings = LoadSettingsFile(filename=PATH_TO_SETTINGS_FILE) 23 | gauth.CommandLineAuth() 24 | drive = GoogleDrive(gauth) 25 | 26 | # If provided arguments incorrect, print usage instructions and exit. 27 | if len(sys.argv) < 2: 28 | print("usage: upload.py ") 29 | exit(1) # Exit program as incorrect parameters provided. 30 | 31 | parentId = sys.argv[1] 32 | myFilePath = sys.argv[2] 33 | myFileName = path.basename(sys.argv[2]) 34 | 35 | # Check if file name already exists in folder. 36 | file_list = drive.ListFile( 37 | { 38 | "q": '"{}" in parents and title="{}" and trashed=false'.format( 39 | parentId, myFileName 40 | ) 41 | } 42 | ).GetList() 43 | 44 | # If file is found, update it, otherwise create new file. 45 | if len(file_list) == 1: 46 | myFile = file_list[0] 47 | else: 48 | myFile = drive.CreateFile( 49 | {"parents": [{"kind": "drive#fileLink", "id": parentId}]} 50 | ) 51 | 52 | # Upload new file content. 53 | myFile.SetContentFile(myFilePath) 54 | myFile["title"] = myFileName 55 | # The `convert` flag indicates to Google Drive whether to convert the 56 | # uploaded file into a Google Drive native format, i.e. Google Sheet for 57 | # CSV or Google Document for DOCX. 58 | myFile.Upload({"convert": True}) 59 | -------------------------------------------------------------------------------- /docs/filelist.rst: -------------------------------------------------------------------------------- 1 | File listing made easy 2 | ============================= 3 | 4 | *PyDrive* handles paginations and parses response as list of `GoogleDriveFile`_. 5 | 6 | Get all files which matches the query 7 | ------------------------------------- 8 | 9 | Create `GoogleDriveFileList`_ instance with `parameters of Files.list()`_ as ``dict``. 10 | Call `GetList()`_ and you will get all files that matches your query as a list of `GoogleDriveFile`_. 11 | The syntax and possible option of the query ``q`` parameter can be found in `search for files` Google documentation. 12 | 13 | .. code-block:: python 14 | 15 | from pydrive2.drive import GoogleDrive 16 | 17 | drive = GoogleDrive(gauth) # Create GoogleDrive instance with authenticated GoogleAuth instance 18 | 19 | # Auto-iterate through all files in the root folder. 20 | file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList() 21 | for file1 in file_list: 22 | print('title: %s, id: %s' % (file1['title'], file1['id'])) 23 | 24 | You can update metadata or content of these `GoogleDriveFile`_ instances if you need it. 25 | 26 | Paginate and iterate through files 27 | ---------------------------------- 28 | 29 | *PyDrive* provides Pythonic way of paginating and iterating through list of files. 30 | Here is an example how to do this, ``maxResults`` below defines how many 31 | files it retrieves at once and we wrap it into a ``for`` loop to iterate: 32 | 33 | Sample code continues from above: 34 | 35 | .. code-block:: python 36 | 37 | # Paginate file lists by specifying number of max results 38 | for file_list in drive.ListFile({'q': 'trashed=true', 'maxResults': 10}): 39 | print('Received %s files from Files.list()' % len(file_list)) # <= 10 40 | for file1 in file_list: 41 | print('title: %s, id: %s' % (file1['title'], file1['id'])) 42 | 43 | 44 | .. _`GoogleDriveFile`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile 45 | .. _`GoogleDriveFileList`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFileList 46 | .. _`parameters of Files.list()`: https://developers.google.com/drive/v2/reference/files/list#request 47 | .. _`GetList()`: /PyDrive2/pydrive2/#pydrive2.apiattr.ApiResourceList.GetList 48 | .. _`search for files`: https://developers.google.com/drive/api/v2/search-files 49 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"] 3 | build-backend = "setuptools.build_meta" 4 | 5 | [project] 6 | name = "PyDrive2" 7 | description = "Google Drive API made easy. Maintained fork of PyDrive." 8 | readme = "README.rst" 9 | authors = [ 10 | {name = "JunYoung Gwak", email = "jgwak@dreamylab.com"}, 11 | ] 12 | maintainers = [ 13 | {name = "DVC team", email = "support@dvc.org"}, 14 | ] 15 | classifiers = [ 16 | "Development Status :: 4 - Beta", 17 | "Programming Language :: Python :: 3", 18 | "Programming Language :: Python :: 3.8", 19 | "Programming Language :: Python :: 3.9", 20 | "Programming Language :: Python :: 3.10", 21 | "Programming Language :: Python :: 3.11", 22 | ] 23 | requires-python = ">=3.8" 24 | dependencies = [ 25 | "google-api-python-client>=1.12.5", 26 | "oauth2client>=4.0.0", 27 | "PyYAML>=3.0", 28 | # https://github.com/iterative/PyDrive2/issues/361 29 | "cryptography<44", 30 | "pyOpenSSL>=19.1.0,<=24.2.1", 31 | ] 32 | license = {text = "Apache License 2.0"} 33 | dynamic = ["version"] 34 | 35 | [project.urls] 36 | Documentation = "https://docs.iterative.ai/PyDrive2" 37 | Source = "https://github.com/iterative/PyDrive2" 38 | Changelog = "https://github.com/iterative/PyDrive2/releases" 39 | 40 | [project.optional-dependencies] 41 | fsspec = [ 42 | "fsspec>=2021.07.0", 43 | "tqdm>=4.0.0", 44 | "funcy>=1.14", 45 | "appdirs>=1.4.3", 46 | ] 47 | tests = [ 48 | "pytest>=4.6.0", 49 | "timeout-decorator", 50 | "funcy>=1.14", 51 | "flake8", 52 | "flake8-docstrings", 53 | "pytest-mock", 54 | "pyinstaller", 55 | "importlib_resources<6; python_version < '3.10'", 56 | "black==24.10.0", 57 | ] 58 | 59 | [project.entry-points.pyinstaller40] 60 | hook-dirs = "pydrive2.__pyinstaller:get_hook_dirs" 61 | tests = "pydrive2.__pyinstaller:get_PyInstaller_tests" 62 | 63 | [tool.setuptools] 64 | packages=[ 65 | "pydrive2", 66 | "pydrive2.test", 67 | "pydrive2.fs", 68 | "pydrive2.__pyinstaller", 69 | ] 70 | 71 | [tool.setuptools_scm] 72 | 73 | [tool.black] 74 | line-length = 79 75 | include = '\.pyi?$' 76 | exclude = ''' 77 | /( 78 | \.eggs 79 | | \.git 80 | | \.hg 81 | | \.mypy_cache 82 | | \.tox 83 | | \.venv 84 | | _build 85 | | buck-out 86 | | build 87 | | dist 88 | )/ 89 | ''' 90 | 91 | [tool.pytest.ini_options] 92 | markers = [ 93 | "manual: mark tests to be runnable only in local environment and require user manual actions.", 94 | ] 95 | -------------------------------------------------------------------------------- /pydrive2/test/test_util.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | import random 3 | import re 4 | import os 5 | import posixpath 6 | 7 | from funcy import retry 8 | from funcy.py3 import cat 9 | from pydrive2.files import ApiRequestError 10 | from shutil import copyfile, rmtree 11 | 12 | newline_pattern = re.compile(r"[\r\n]") 13 | 14 | GDRIVE_USER_CREDENTIALS_DATA = "GDRIVE_USER_CREDENTIALS_DATA" 15 | DEFAULT_USER_CREDENTIALS_FILE = "/tmp/pydrive2/credentials.json" 16 | 17 | TESTS_ROOTDIR = os.path.dirname(__file__) 18 | SETTINGS_PATH = posixpath.join(TESTS_ROOTDIR, "settings/") 19 | LOCAL_PATH = posixpath.join(TESTS_ROOTDIR, "settings/local/") 20 | 21 | 22 | def setup_credentials(credentials_path=DEFAULT_USER_CREDENTIALS_FILE): 23 | os.chdir(TESTS_ROOTDIR) 24 | if os.getenv(GDRIVE_USER_CREDENTIALS_DATA): 25 | if not os.path.exists(os.path.dirname(credentials_path)): 26 | os.makedirs(os.path.dirname(credentials_path), exist_ok=True) 27 | with open(credentials_path, "w") as credentials_file: 28 | credentials_file.write(os.getenv(GDRIVE_USER_CREDENTIALS_DATA)) 29 | 30 | 31 | def settings_file_path(settings_file, wkdir=LOCAL_PATH): 32 | template_path = SETTINGS_PATH + settings_file 33 | wkdir = Path(wkdir) 34 | local_path = wkdir / settings_file 35 | assert os.path.exists(template_path) 36 | if not os.path.exists(wkdir): 37 | os.makedirs(wkdir, exist_ok=True) 38 | if not os.path.exists(local_path): 39 | copyfile(template_path, local_path) 40 | return local_path 41 | 42 | 43 | class PyDriveRetriableError(Exception): 44 | pass 45 | 46 | 47 | # 15 tries, start at 0.5s, multiply by golden ratio, cap at 20s 48 | @retry(15, PyDriveRetriableError, timeout=lambda a: min(0.5 * 1.618**a, 20)) 49 | def pydrive_retry(call, *args, **kwargs): 50 | try: 51 | result = call(*args, **kwargs) 52 | except ApiRequestError as exception: 53 | if exception.error["code"] in [403, 500, 502, 503, 504]: 54 | raise PyDriveRetriableError("Google API request failed") 55 | raise 56 | return result 57 | 58 | 59 | def pydrive_list_item(drive, query, max_results=1000): 60 | param = {"q": query, "maxResults": max_results} 61 | 62 | file_list = drive.ListFile(param) 63 | 64 | # Isolate and decorate fetching of remote drive items in pages 65 | get_list = lambda: pydrive_retry(next, file_list, None) # noqa: E731 66 | 67 | # Fetch pages until None is received, lazily flatten the thing 68 | return cat(iter(get_list, None)) 69 | 70 | 71 | def CreateRandomFileName(): 72 | hash = random.getrandbits(128) 73 | return "%032x" % hash 74 | 75 | 76 | def StripNewlines(string): 77 | return newline_pattern.sub("", string) 78 | 79 | 80 | def create_file(path, content): 81 | with open(path, "w") as f: 82 | f.write(content) 83 | 84 | 85 | def delete_file(path): 86 | if os.path.exists(path): 87 | os.remove(path) 88 | return True 89 | return False 90 | 91 | 92 | def delete_dir(path): 93 | rmtree(path, ignore_errors=True) 94 | -------------------------------------------------------------------------------- /pydrive2/test/README.rst: -------------------------------------------------------------------------------- 1 | Run tests locally 2 | ----------------- 3 | 4 | 1. Copy settings files to the :code:`pydrive2/test/settings/local` directory: 5 | 6 | :: 7 | 8 | cd pydrive2/test/settings && cp *.yaml local 9 | 10 | 2. Setup a Google service account for your Google Cloud Project: 11 | - Sign into the `Google API Console 12 | `_ 13 | - Select or `Create a new 14 | `_ 15 | project. 16 | - `Enable the Drive API 17 | `_ from the **APIs & 18 | Services** **Dashboard** (left sidebar), click on **+ ENABLE APIS AND 19 | SERVICES**. Find and select the "Google Drive API" in the API Library, and 20 | click on the **ENABLE** button. 21 | - Go back to **IAM & Admin** in the left 22 | sidebar, and select **Service Accounts**. Click **+ CREATE SERVICE 23 | ACCOUNT**, on the next screen, enter **Service account name** e.g. "PyDrive 24 | tests", and click **Create**. Select **Continue** at the next **Service 25 | account permissions** page, click at **+ CREATE KEY**, select **JSON** and 26 | **Create**. Save generated :code:`.json` key file at your local disk. 27 | - Copy downloaded :code:`json` file to :code:`/tmp/pydrive2/credentials.json` 28 | directory. 29 | 30 | 3. Optional. If you would like to use your own an OAuth client ID follow the steps: 31 | - Under `Google API Console `_ select 32 | **APIs & Services** from the left sidebar, and select **OAuth consent screen**. 33 | Chose a **User Type** and click **CREATE**. On the next screen, enter an 34 | **Application name** e.g. "PyDrive tests", and click the **Save** (scroll to 35 | bottom). 36 | - From the left sidebar, select **Credentials**, and click the 37 | **Create credentials** dropdown to select **OAuth client ID**. Chose **Other** 38 | and click **Create** to proceed with a default client name. At **Credentials** 39 | screen find a list of your **OAuth 2.0 Client IDs**, click download icon in 40 | front of your OAuth client id created previously. You should be prompted to 41 | download :code:`client_secret_xxx_.json` file. 42 | - Copy downloaded :code:`.json` file into :code:`pydrive2/test` directory 43 | and rename to :code:`client_secrets.json`. 44 | - Replace {{ }} sections 45 | in :code:`pydrive2/test/settings/local/test_oauth_test_02.yaml` with the 46 | relevant values of :code:`client_id` and :code:`client_secret` from your 47 | **client_secrets.json** file. 48 | 49 | 4. Setup virtual environment (recommended optional step): 50 | 51 | :: 52 | 53 | 54 | virtualenv -p python .env 55 | source .env/bin/activate 56 | 57 | 5. Install :code:`tests` deps from the root directory of the project: 58 | 59 | :: 60 | 61 | pip install -e .[tests,fsspec] 62 | 63 | 64 | 5. Run tests: 65 | 66 | :: 67 | 68 | py.test -v -s 69 | -------------------------------------------------------------------------------- /docs/fsspec.rst: -------------------------------------------------------------------------------- 1 | fsspec filesystem 2 | ================= 3 | 4 | *PyDrive2* provides easy way to work with your files through `fsspec`_ 5 | compatible `GDriveFileSystem`_. 6 | 7 | Installation 8 | ------------ 9 | 10 | .. code-block:: sh 11 | 12 | pip install 'pydrive2[fsspec]' 13 | 14 | Local webserver 15 | --------------- 16 | 17 | .. code-block:: python 18 | 19 | from pydrive2.fs import GDriveFileSystem 20 | 21 | fs = GDriveFileSystem( 22 | "root", 23 | client_id="my_client_id", 24 | client_secret="my_client_secret", 25 | ) 26 | 27 | By default, credentials will be cached per 'client_id', but if you are using 28 | multiple users you might want to use 'profile' to avoid accidentally using 29 | someone else's cached credentials: 30 | 31 | .. code-block:: python 32 | 33 | from pydrive2.fs import GDriveFileSystem 34 | 35 | fs = GDriveFileSystem( 36 | "root", 37 | client_id="my_client_id", 38 | client_secret="my_client_secret", 39 | profile="myprofile", 40 | ) 41 | 42 | Writing cached credentials to a file and using it if it already exists (which 43 | avoids interactive auth): 44 | 45 | .. code-block:: python 46 | 47 | from pydrive2.fs import GDriveFileSystem 48 | 49 | fs = GDriveFileSystem( 50 | "root", 51 | client_id="my_client_id", 52 | client_secret="my_client_secret", 53 | client_json_file_path="/path/to/keyfile.json", 54 | ) 55 | 56 | Using cached credentials from json string (avoids interactive auth): 57 | 58 | .. code-block:: python 59 | 60 | from pydrive2.fs import GDriveFileSystem 61 | 62 | fs = GDriveFileSystem( 63 | "root", 64 | client_id="my_client_id", 65 | client_secret="my_client_secret", 66 | client_json=json_string, 67 | ) 68 | 69 | Service account 70 | --------------- 71 | 72 | Using json keyfile path: 73 | 74 | .. code-block:: python 75 | 76 | from pydrive2.fs import GDriveFileSystem 77 | 78 | fs = GDriveFileSystem( 79 | # replace with ID of a drive or directory and give service account access to it 80 | "root", 81 | use_service_account=True, 82 | client_json_file_path="/path/to/keyfile.json", 83 | ) 84 | 85 | Using json keyfile string: 86 | 87 | .. code-block:: python 88 | 89 | from pydrive2.fs import GDriveFileSystem 90 | 91 | fs = GDriveFileSystem( 92 | # replace with ID of a drive or directory and give service account access to it 93 | "root", 94 | use_service_account=True, 95 | client_json=json_string, 96 | ) 97 | 98 | Use `client_user_email` if you are using `delegation of authority`_. 99 | 100 | Additional parameters 101 | --------------------- 102 | 103 | :trash_only (bool): Move files to trash instead of deleting. 104 | :acknowledge_abuse (bool): Acknowledging the risk and download file identified as abusive. See `Abusive files`_ for more info. 105 | 106 | Using filesystem 107 | ---------------- 108 | 109 | .. code-block:: python 110 | 111 | # replace `root` with ID of a drive or directory and give service account access to it 112 | for root, dnames, fnames in fs.walk("root"): 113 | for dname in dnames: 114 | print(f"dir: {root}/{dname}") 115 | 116 | for fname in fnames: 117 | print(f"file: {root}/{fname}") 118 | 119 | Filesystem instance offers a large number of methods for getting information 120 | about and manipulating files, refer to fsspec docs on 121 | `how to use a filesystem`_. 122 | 123 | .. _`fsspec`: https://filesystem-spec.readthedocs.io/en/latest/ 124 | .. _`GDriveFileSystem`: /PyDrive2/pydrive2/#pydrive2.fs.GDriveFileSystem 125 | .. _`delegation of authority`: https://developers.google.com/admin-sdk/directory/v1/guides/delegation 126 | .. _`Abusive files`: /PyDrive2/filemanagement/index.html#abusive-files 127 | .. _`how to use a filesystem`: https://filesystem-spec.readthedocs.io/en/latest/usage.html#use-a-file-system 128 | -------------------------------------------------------------------------------- /pydrive2/test/test_filelist.py: -------------------------------------------------------------------------------- 1 | import os 2 | import unittest 3 | 4 | from pydrive2.auth import GoogleAuth 5 | from pydrive2.drive import GoogleDrive 6 | from pydrive2.test import test_util 7 | from pydrive2.test.test_util import ( 8 | pydrive_retry, 9 | pydrive_list_item, 10 | setup_credentials, 11 | settings_file_path, 12 | ) 13 | 14 | 15 | class GoogleDriveFileListTest(unittest.TestCase): 16 | """Tests operations of files.GoogleDriveFileList class. 17 | Equivalent to Files.list in Google Drive API. 18 | """ 19 | 20 | @classmethod 21 | def setup_class(cls): 22 | setup_credentials() 23 | 24 | cls.ga = GoogleAuth(settings_file_path("default.yaml")) 25 | cls.ga.ServiceAuth() 26 | cls.drive = GoogleDrive(cls.ga) 27 | 28 | def test_01_Files_List_GetList(self): 29 | drive = GoogleDrive(self.ga) 30 | query = f"title = '{self.title}' and trashed = false" 31 | for file1 in pydrive_list_item(drive, query): 32 | found = False 33 | for file2 in pydrive_list_item(drive, query): 34 | if file1["id"] == file2["id"]: 35 | found = True 36 | self.assertEqual(found, True) 37 | 38 | def test_02_Files_List_ForLoop(self): 39 | drive = GoogleDrive(self.ga) 40 | query = f"title = '{self.title}' and trashed = false" 41 | files = [] 42 | for x in pydrive_list_item( 43 | drive, query, 2 44 | ): # Build iterator to access files simply with for loop 45 | files.append(x) 46 | for file1 in self.file_list: 47 | found = False 48 | for file2 in files: 49 | if file1["id"] == file2["id"]: 50 | found = True 51 | self.assertEqual(found, True) 52 | 53 | def test_03_Files_List_GetList_Iterate(self): 54 | drive = GoogleDrive(self.ga) 55 | flist = drive.ListFile( 56 | { 57 | "q": "title = '%s' and trashed = false" % self.title, 58 | "maxResults": 2, 59 | } 60 | ) 61 | files = [] 62 | while True: 63 | try: 64 | x = pydrive_retry(flist.GetList) 65 | self.assertTrue(len(x) <= 2) 66 | files.extend(x) 67 | except StopIteration: 68 | break 69 | for file1 in self.file_list: 70 | found = False 71 | for file2 in files: 72 | if file1["id"] == file2["id"]: 73 | found = True 74 | self.assertEqual(found, True) 75 | 76 | def test_File_List_Folders(self): 77 | drive = GoogleDrive(self.ga) 78 | folder1 = drive.CreateFile( 79 | { 80 | "mimeType": "application/vnd.google-apps.folder", 81 | "title": self.title, 82 | } 83 | ) 84 | pydrive_retry(folder1.Upload) 85 | self.file_list.append(folder1) 86 | query = f"title = '{self.title}' and trashed = false" 87 | count = 0 88 | for file1 in pydrive_list_item(drive, query): 89 | self.assertFileInFileList(file1) 90 | count += 1 91 | 92 | self.assertTrue(count == 11) 93 | 94 | # setUp and tearDown methods. 95 | # =========================== 96 | def setUp(self): 97 | title = test_util.CreateRandomFileName() 98 | file_list = [] 99 | for x in range(0, 10): 100 | file1 = self.drive.CreateFile() 101 | file1["title"] = title 102 | pydrive_retry(file1.Upload) 103 | file_list.append(file1) 104 | 105 | self.title = title 106 | self.file_list = file_list 107 | 108 | def tearDown(self): 109 | # Deleting uploaded files. 110 | for file1 in self.file_list: 111 | pydrive_retry(file1.Delete) 112 | 113 | def assertFileInFileList(self, file_object): 114 | found = False 115 | for file1 in self.file_list: 116 | if file_object["id"] == file1["id"]: 117 | found = True 118 | self.assertEqual(found, True) 119 | 120 | def DeleteOldFile(self, file_name): 121 | try: 122 | os.remove(file_name) 123 | except OSError: 124 | pass 125 | 126 | 127 | if __name__ == "__main__": 128 | unittest.main() 129 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | |CI| |Conda| |PyPI| 2 | 3 | .. |CI| image:: https://github.com/iterative/PyDrive2/workflows/Tests/badge.svg?branch=main 4 | :target: https://github.com/iterative/PyDrive2/actions 5 | :alt: GHA Tests 6 | 7 | .. |Conda| image:: https://img.shields.io/conda/v/conda-forge/PyDrive2.svg?label=conda&logo=conda-forge 8 | :target: https://anaconda.org/conda-forge/PyDrive2 9 | :alt: Conda-forge 10 | 11 | .. |PyPI| image:: https://img.shields.io/pypi/v/PyDrive2.svg?label=pip&logo=PyPI&logoColor=white 12 | :target: https://pypi.org/project/PyDrive2 13 | :alt: PyPI 14 | 15 | PyDrive2 16 | -------- 17 | 18 | *PyDrive2* is a wrapper library of 19 | `google-api-python-client `_ 20 | that simplifies many common Google Drive API V2 tasks. It is an actively 21 | maintained fork of `https://pypi.python.org/pypi/PyDrive `_. 22 | By the authors and maintainers of the `Git for Data `_ - DVC 23 | project. 24 | 25 | Project Info 26 | ------------ 27 | 28 | - Package: `https://pypi.python.org/pypi/PyDrive2 `_ 29 | - Documentation: `https://docs.iterative.ai/PyDrive2 `_ 30 | - Source: `https://github.com/iterative/PyDrive2 `_ 31 | - Changelog: `https://github.com/iterative/PyDrive2/releases `_ 32 | - `Running tests `_ 33 | 34 | Features of PyDrive2 35 | -------------------- 36 | 37 | - Simplifies OAuth2.0 into just few lines with flexible settings. 38 | - Wraps `Google Drive API V2 `_ into 39 | classes of each resource to make your program more object-oriented. 40 | - Helps common operations else than API calls, such as content fetching 41 | and pagination control. 42 | - Provides `fsspec`_ filesystem implementation. 43 | 44 | How to install 45 | -------------- 46 | 47 | You can install PyDrive2 with regular ``pip`` command. 48 | 49 | :: 50 | 51 | $ pip install PyDrive2 52 | 53 | To install the current development version from GitHub, use: 54 | 55 | :: 56 | 57 | $ pip install git+https://github.com/iterative/PyDrive2.git#egg=PyDrive2 58 | 59 | OAuth made easy 60 | --------------- 61 | 62 | Download *client\_secrets.json* from Google API Console and OAuth2.0 is 63 | done in two lines. You can customize behavior of OAuth2 in one settings 64 | file *settings.yaml*. 65 | 66 | .. code:: python 67 | 68 | 69 | from pydrive2.auth import GoogleAuth 70 | from pydrive2.drive import GoogleDrive 71 | 72 | gauth = GoogleAuth() 73 | gauth.LocalWebserverAuth() 74 | 75 | drive = GoogleDrive(gauth) 76 | 77 | File management made easy 78 | ------------------------- 79 | 80 | Upload/update the file with one method. PyDrive2 will do it in the most 81 | efficient way. 82 | 83 | .. code:: python 84 | 85 | file1 = drive.CreateFile({'title': 'Hello.txt'}) 86 | file1.SetContentString('Hello') 87 | file1.Upload() # Files.insert() 88 | 89 | file1['title'] = 'HelloWorld.txt' # Change title of the file 90 | file1.Upload() # Files.patch() 91 | 92 | content = file1.GetContentString() # 'Hello' 93 | file1.SetContentString(content+' World!') # 'Hello World!' 94 | file1.Upload() # Files.update() 95 | 96 | file2 = drive.CreateFile() 97 | file2.SetContentFile('hello.png') 98 | file2.Upload() 99 | print('Created file %s with mimeType %s' % (file2['title'], 100 | file2['mimeType'])) 101 | # Created file hello.png with mimeType image/png 102 | 103 | file3 = drive.CreateFile({'id': file2['id']}) 104 | print('Downloading file %s from Google Drive' % file3['title']) # 'hello.png' 105 | file3.GetContentFile('world.png') # Save Drive file as a local file 106 | 107 | # or download Google Docs files in an export format provided. 108 | # downloading a docs document as an html file: 109 | docsfile.GetContentFile('test.html', mimetype='text/html') 110 | 111 | File listing pagination made easy 112 | --------------------------------- 113 | 114 | *PyDrive2* handles file listing pagination for you. 115 | 116 | .. code:: python 117 | 118 | # Auto-iterate through all files that matches this query 119 | file_list = drive.ListFile({'q': "'root' in parents"}).GetList() 120 | for file1 in file_list: 121 | print('title: {}, id: {}'.format(file1['title'], file1['id'])) 122 | 123 | # Paginate file lists by specifying number of max results 124 | for file_list in drive.ListFile({'maxResults': 10}): 125 | print('Received {} files from Files.list()'.format(len(file_list))) # <= 10 126 | for file1 in file_list: 127 | print('title: {}, id: {}'.format(file1['title'], file1['id'])) 128 | 129 | Fsspec filesystem 130 | ----------------- 131 | 132 | *PyDrive2* provides easy way to work with your files through `fsspec`_ 133 | compatible `GDriveFileSystem`_. 134 | 135 | Install PyDrive2 with the required dependencies 136 | 137 | :: 138 | 139 | $ pip install PyDrive2[fsspec] 140 | 141 | .. code:: python 142 | 143 | from pydrive2.fs import GDriveFileSystem 144 | 145 | # replace `root` with ID of a drive or directory and give service account access to it 146 | fs = GDriveFileSystem("root", client_id=my_id, client_secret=my_secret) 147 | 148 | for root, dnames, fnames in fs.walk("root"): 149 | ... 150 | 151 | .. _`GDriveFileSystem`: https://docs.iterative.ai/PyDrive2/fsspec/ 152 | 153 | Concurrent access made easy 154 | --------------------------- 155 | 156 | All API functions made to be thread-safe. 157 | 158 | Contributors 159 | ------------ 160 | 161 | Thanks to all our contributors! 162 | 163 | .. image:: https://contrib.rocks/image?repo=iterative/PyDrive2 164 | :target: https://github.com/iterative/PyDrive2/graphs/contributors 165 | 166 | .. _`fsspec`: https://filesystem-spec.readthedocs.io/en/latest/ 167 | -------------------------------------------------------------------------------- /pydrive2/apiattr.py: -------------------------------------------------------------------------------- 1 | class ApiAttribute: 2 | """A data descriptor that sets and returns values.""" 3 | 4 | def __init__(self, name): 5 | """Create an instance of ApiAttribute. 6 | 7 | :param name: name of this attribute. 8 | :type name: str. 9 | """ 10 | self.name = name 11 | 12 | def __get__(self, obj, type=None): 13 | """Accesses value of this attribute.""" 14 | return obj.attr.get(self.name) 15 | 16 | def __set__(self, obj, value): 17 | """Write value of this attribute.""" 18 | obj.attr[self.name] = value 19 | if obj.dirty.get(self.name) is not None: 20 | obj.dirty[self.name] = True 21 | 22 | def __del__(self, obj=None): 23 | """Delete value of this attribute.""" 24 | if not obj: 25 | return 26 | 27 | del obj.attr[self.name] 28 | if obj.dirty.get(self.name) is not None: 29 | del obj.dirty[self.name] 30 | 31 | 32 | class ApiAttributeMixin: 33 | """Mixin to initialize required global variables to use ApiAttribute.""" 34 | 35 | def __init__(self): 36 | self.attr = {} 37 | self.dirty = {} 38 | self.http = None # Any element may make requests and will require this 39 | # field. 40 | 41 | 42 | class ApiResource(dict): 43 | """Super class of all api resources. 44 | 45 | Inherits and behaves as a python dictionary to handle api resources. 46 | Save clean copy of metadata in self.metadata as a dictionary. 47 | Provides changed metadata elements to efficiently update api resources. 48 | """ 49 | 50 | auth = ApiAttribute("auth") 51 | 52 | def __init__(self, *args, **kwargs): 53 | """Create an instance of ApiResource.""" 54 | super().__init__() 55 | self.update(*args, **kwargs) 56 | self.metadata = dict(self) 57 | 58 | def __getitem__(self, key): 59 | """Overwritten method of dictionary. 60 | 61 | :param key: key of the query. 62 | :type key: str. 63 | :returns: value of the query. 64 | """ 65 | return dict.__getitem__(self, key) 66 | 67 | def __setitem__(self, key, val): 68 | """Overwritten method of dictionary. 69 | 70 | :param key: key of the query. 71 | :type key: str. 72 | :param val: value of the query. 73 | """ 74 | dict.__setitem__(self, key, val) 75 | 76 | def __repr__(self): 77 | """Overwritten method of dictionary.""" 78 | dict_representation = dict.__repr__(self) 79 | return f"{type(self).__name__}({dict_representation})" 80 | 81 | def update(self, *args, **kwargs): 82 | """Overwritten method of dictionary.""" 83 | BAD_URL_PREFIX = "https://www.googleapis.comhttps:" 84 | for k, v in dict(*args, **kwargs).items(): 85 | if k == "downloadUrl" and v.startswith(BAD_URL_PREFIX): 86 | v = v.replace(BAD_URL_PREFIX, "https://www.googleapis.com", 1) 87 | self[k] = v 88 | 89 | def UpdateMetadata(self, metadata=None): 90 | """Update metadata and mark all of them to be clean.""" 91 | if metadata: 92 | self.update(metadata) 93 | self.metadata = dict(self) 94 | 95 | def GetChanges(self): 96 | """Returns changed metadata elements to update api resources efficiently. 97 | 98 | :returns: dict -- changed metadata elements. 99 | """ 100 | dirty = {} 101 | for key in self: 102 | if self.metadata.get(key) is None: 103 | dirty[key] = self[key] 104 | elif self.metadata[key] != self[key]: 105 | dirty[key] = self[key] 106 | return dirty 107 | 108 | 109 | class ApiResourceList(ApiAttributeMixin, ApiResource): 110 | """Abstract class of all api list resources. 111 | 112 | Inherits ApiResource and builds iterator to list any API resource. 113 | """ 114 | 115 | metadata = ApiAttribute("metadata") 116 | 117 | def __init__(self, auth=None, metadata=None): 118 | """Create an instance of ApiResourceList. 119 | 120 | :param auth: authorized GoogleAuth instance. 121 | :type auth: GoogleAuth. 122 | :param metadata: parameter to send to list command. 123 | :type metadata: dict. 124 | """ 125 | ApiAttributeMixin.__init__(self) 126 | ApiResource.__init__(self) 127 | self.auth = auth 128 | self.UpdateMetadata() 129 | if metadata: 130 | self.update(metadata) 131 | 132 | def __iter__(self): 133 | """Returns iterator object. 134 | 135 | :returns: ApiResourceList -- self 136 | """ 137 | return self 138 | 139 | def __next__(self): 140 | """Make API call to list resources and return them. 141 | 142 | Auto updates 'pageToken' every time it makes API call and 143 | raises StopIteration when it reached the end of iteration. 144 | 145 | :returns: list -- list of API resources. 146 | :raises: StopIteration 147 | """ 148 | if "pageToken" in self and self["pageToken"] is None: 149 | raise StopIteration 150 | result = self._GetList() 151 | self["pageToken"] = self.metadata.get("nextPageToken") 152 | return result 153 | 154 | def GetList(self): 155 | """Get list of API resources. 156 | 157 | If 'maxResults' is not specified, it will automatically iterate through 158 | every resources available. Otherwise, it will make API call once and 159 | update 'pageToken'. 160 | 161 | :returns: list -- list of API resources. 162 | """ 163 | if self.get("maxResults") is None: 164 | self["maxResults"] = 1000 165 | result = [] 166 | for x in self: 167 | result.extend(x) 168 | del self["maxResults"] 169 | return result 170 | else: 171 | return next(self) 172 | 173 | def _GetList(self): 174 | """Helper function which actually makes API call. 175 | 176 | Should be overwritten. 177 | 178 | :raises: NotImplementedError 179 | """ 180 | raise NotImplementedError 181 | 182 | def Reset(self): 183 | """Resets current iteration""" 184 | if "pageToken" in self: 185 | del self["pageToken"] 186 | -------------------------------------------------------------------------------- /docs/quickstart.rst: -------------------------------------------------------------------------------- 1 | Quickstart 2 | ============================= 3 | 4 | Authentication 5 | -------------- 6 | Drive API requires OAuth2.0 for authentication. *PyDrive2* makes your life much easier by handling complex authentication steps for you. 7 | 8 | 1. Go to `APIs Console`_ and make your own project. 9 | 2. Search for 'Google Drive API', select the entry, and click 'Enable'. 10 | 3. Select 'Credentials' from the left menu, click 'Create Credentials', select 'OAuth client ID'. 11 | 4. Now, the product name and consent screen need to be set -> click 'Configure consent screen' and follow the instructions. Once finished: 12 | 13 | a. Select 'Application type' to be *Web application*. 14 | b. Enter an appropriate name. 15 | c. Input *http://localhost:8080/* for 'Authorized redirect URIs'. 16 | d. Click 'Create'. 17 | 18 | 5. Click 'Download JSON' on the right side of Client ID to download **client_secret_.json**. 19 | 20 | The downloaded file has all authentication information of your application. 21 | **Rename the file to "client_secrets.json" and place it in your working directory.** 22 | 23 | Create *quickstart.py* file and copy and paste the following code. 24 | 25 | .. code-block:: python 26 | 27 | from pydrive2.auth import GoogleAuth 28 | 29 | gauth = GoogleAuth() 30 | gauth.LocalWebserverAuth() # Creates local webserver and auto handles authentication. 31 | 32 | Run this code with *python quickstart.py* and you will see a web browser asking you for authentication. Click *Accept* and you are done with authentication. For more details, take a look at documentation: `OAuth made easy`_ 33 | 34 | .. _`APIs Console`: https://console.developers.google.com/iam-admin/projects 35 | .. _`OAuth made easy`: /PyDrive2/oauth/ 36 | 37 | Creating and Updating Files 38 | --------------------------- 39 | 40 | There are many methods to create and update file metadata and contents. With *PyDrive2*, all you have to know is 41 | `Upload()`_ method which makes optimal API call for you. Add the following code to your *quickstart.py* and run it. 42 | 43 | .. code-block:: python 44 | 45 | from pydrive2.drive import GoogleDrive 46 | 47 | drive = GoogleDrive(gauth) 48 | 49 | file1 = drive.CreateFile({'title': 'Hello.txt'}) # Create GoogleDriveFile instance with title 'Hello.txt'. 50 | file1.SetContentString('Hello World!') # Set content of the file from given string. 51 | file1.Upload() 52 | 53 | This code will create a new file with title *Hello.txt* and its content *Hello World!*. You can see and open this 54 | file from `Google Drive`_ if you want. For more details, take a look at documentation: `File management made easy`_ 55 | 56 | .. _`Upload()`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.Upload 57 | .. _`Google Drive`: https://drive.google.com 58 | .. _`File management made easy`: /PyDrive2/filemanagement/ 59 | 60 | Listing Files 61 | ------------- 62 | 63 | *PyDrive2* handles paginations and parses response as list of `GoogleDriveFile`_. Let's get title and id of all the files in the root folder of Google Drive. Again, add the following code to *quickstart.py* and execute it. 64 | 65 | .. code-block:: python 66 | 67 | # Auto-iterate through all files that matches this query 68 | file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList() 69 | for file1 in file_list: 70 | print('title: %s, id: %s' % (file1['title'], file1['id'])) 71 | 72 | Creating a Folder 73 | ----------------- 74 | 75 | GoogleDrive treats everything as a file and assigns different mimetypes for different file formats. A folder is thus 76 | also a file with a special mimetype. The code below allows you to add a subfolder to an existing folder. 77 | 78 | .. code-block:: python 79 | 80 | def create_folder(parent_folder_id, subfolder_name): 81 | newFolder = drive.CreateFile({'title': subfolder_name, "parents": [{"kind": "drive#fileLink", "id": \ 82 | parent_folder_id}],"mimeType": "application/vnd.google-apps.folder"}) 83 | newFolder.Upload() 84 | return newFolder 85 | 86 | 87 | Return File ID via File Title 88 | ----------------------------- 89 | 90 | A common task is providing the Google Drive API with a file id. 91 | ``get_id_of_title`` demonstrates a simple workflow to return the id of a file handle by searching the file titles in a 92 | given directory. The function takes two arguments, ``title`` and ``parent_directory_id``. ``title`` is a string that 93 | will be compared against file titles included in a directory identified by the ``parent_directory_id``. 94 | 95 | .. code-block:: python 96 | 97 | def get_id_of_title(title,parent_directory_id): 98 | foldered_list=drive.ListFile({'q': "'"+parent_directory_id+"' in parents and trashed=false"}).GetList() 99 | for file in foldered_list: 100 | if(file['title']==title): 101 | return file['id'] 102 | return None 103 | 104 | Browse Folders 105 | -------------- 106 | This returns a json output of the data in a directory with some important attributes like size, title, parent_id. 107 | 108 | .. code-block:: python 109 | 110 | browsed=[] 111 | def folder_browser(folder_list,parent_id): 112 | for element in folder_list: 113 | if type(element) is dict: 114 | print (element['title']) 115 | else: 116 | print (element) 117 | print("Enter Name of Folder You Want to Use\nEnter '/' to use current folder\nEnter ':' to create New Folder and 118 | use that" ) 119 | inp=input() 120 | if inp=='/': 121 | return parent_id 122 | elif inp==':': 123 | print("Enter Name of Folder You Want to Create") 124 | inp=input() 125 | newfolder=create_folder(parent_id,inp) 126 | if not os.path.exists(HOME_DIRECTORY+ROOT_FOLDER_NAME+os.path.sep+USERNAME): 127 | os.makedirs(HOME_DIRECTORY+ROOT_FOLDER_NAME+os.path.sep+USERNAME) 128 | return newfolder['id'] 129 | 130 | else: 131 | folder_selected=inp 132 | for element in folder_list: 133 | if type(element) is dict: 134 | if element["title"]==folder_selected: 135 | struc=element["list"] 136 | browsed.append(folder_selected) 137 | print("Inside "+folder_selected) 138 | return folder_browser(struc,element['id']) 139 | 140 | here ``folder_list`` is the list of folders that is present 141 | 142 | You will see title and id of all the files and folders in root folder of your Google Drive. For more details, refer to the documentation: `File listing made easy`_ 143 | 144 | .. _`GoogleDriveFile`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile 145 | .. _`File listing made easy`: /PyDrive2/filelist/ 146 | -------------------------------------------------------------------------------- /pydrive2/settings.py: -------------------------------------------------------------------------------- 1 | from yaml import load 2 | from yaml import YAMLError 3 | 4 | try: 5 | from yaml import CSafeLoader as SafeLoader 6 | except ImportError: 7 | from yaml import SafeLoader 8 | 9 | SETTINGS_FILE = "settings.yaml" 10 | SETTINGS_STRUCT = { 11 | "client_config_backend": { 12 | "type": str, 13 | "required": True, 14 | "default": "file", 15 | "dependency": [ 16 | {"value": "file", "attribute": ["client_config_file"]}, 17 | {"value": "settings", "attribute": ["client_config"]}, 18 | {"value": "service", "attribute": ["service_config"]}, 19 | ], 20 | }, 21 | "save_credentials": { 22 | "type": bool, 23 | "required": True, 24 | "default": False, 25 | "dependency": [ 26 | {"value": True, "attribute": ["save_credentials_backend"]} 27 | ], 28 | }, 29 | "get_refresh_token": {"type": bool, "required": False, "default": False}, 30 | "client_config_file": { 31 | "type": str, 32 | "required": False, 33 | "default": "client_secrets.json", 34 | }, 35 | "save_credentials_backend": { 36 | "type": str, 37 | "required": False, 38 | "dependency": [ 39 | {"value": "file", "attribute": ["save_credentials_file"]}, 40 | {"value": "dictionary", "attribute": ["save_credentials_dict"]}, 41 | {"value": "dictionary", "attribute": ["save_credentials_key"]}, 42 | ], 43 | }, 44 | "client_config": { 45 | "type": dict, 46 | "required": False, 47 | "struct": { 48 | "client_id": {"type": str, "required": True}, 49 | "client_secret": {"type": str, "required": True}, 50 | "auth_uri": { 51 | "type": str, 52 | "required": True, 53 | "default": "https://accounts.google.com/o/oauth2/auth", 54 | }, 55 | "token_uri": { 56 | "type": str, 57 | "required": True, 58 | "default": "https://accounts.google.com/o/oauth2/token", 59 | }, 60 | "redirect_uri": { 61 | "type": str, 62 | "required": True, 63 | "default": "urn:ietf:wg:oauth:2.0:oob", 64 | }, 65 | "revoke_uri": {"type": str, "required": True, "default": None}, 66 | }, 67 | }, 68 | "service_config": { 69 | "type": dict, 70 | "required": False, 71 | "struct": { 72 | "client_user_email": { 73 | "type": str, 74 | "required": True, 75 | "default": None, 76 | }, 77 | "client_service_email": {"type": str, "required": False}, 78 | "client_pkcs12_file_path": {"type": str, "required": False}, 79 | "client_json_file_path": {"type": str, "required": False}, 80 | "client_json_dict": { 81 | "type": dict, 82 | "required": False, 83 | "struct": {}, 84 | }, 85 | "client_json": {"type": str, "required": False}, 86 | }, 87 | }, 88 | "oauth_scope": { 89 | "type": list, 90 | "required": True, 91 | "struct": str, 92 | "default": ["https://www.googleapis.com/auth/drive"], 93 | }, 94 | "save_credentials_file": {"type": str, "required": False}, 95 | "save_credentials_dict": {"type": dict, "required": False, "struct": {}}, 96 | "save_credentials_key": {"type": str, "required": False}, 97 | } 98 | 99 | 100 | class SettingsError(IOError): 101 | """Error while loading/saving settings""" 102 | 103 | 104 | class InvalidConfigError(IOError): 105 | """Error trying to read client configuration.""" 106 | 107 | 108 | def LoadSettingsFile(filename=SETTINGS_FILE): 109 | """Loads settings file in yaml format given file name. 110 | 111 | :param filename: path for settings file. 'settings.yaml' by default. 112 | :type filename: str. 113 | :raises: SettingsError 114 | """ 115 | try: 116 | with open(filename) as stream: 117 | data = load(stream, Loader=SafeLoader) 118 | except (YAMLError, OSError) as e: 119 | raise SettingsError(e) 120 | return data 121 | 122 | 123 | def ValidateSettings(data): 124 | """Validates if current settings is valid. 125 | 126 | :param data: dictionary containing all settings. 127 | :type data: dict. 128 | :raises: InvalidConfigError 129 | """ 130 | _ValidateSettingsStruct(data, SETTINGS_STRUCT) 131 | 132 | 133 | def _ValidateSettingsStruct(data, struct): 134 | """Validates if provided data fits provided structure. 135 | 136 | :param data: dictionary containing settings. 137 | :type data: dict. 138 | :param struct: dictionary containing structure information of settings. 139 | :type struct: dict. 140 | :raises: InvalidConfigError 141 | """ 142 | # Validate required elements of the setting. 143 | for key in struct: 144 | if struct[key]["required"]: 145 | _ValidateSettingsElement(data, struct, key) 146 | 147 | 148 | def _ValidateSettingsElement(data, struct, key): 149 | """Validates if provided element of settings data fits provided structure. 150 | 151 | :param data: dictionary containing settings. 152 | :type data: dict. 153 | :param struct: dictionary containing structure information of settings. 154 | :type struct: dict. 155 | :param key: key of the settings element to validate. 156 | :type key: str. 157 | :raises: InvalidConfigError 158 | """ 159 | # Check if data exists. If not, check if default value exists. 160 | value = data.get(key) 161 | data_type = struct[key]["type"] 162 | if value is None: 163 | try: 164 | default = struct[key]["default"] 165 | except KeyError: 166 | raise InvalidConfigError("Missing required setting %s" % key) 167 | else: 168 | data[key] = default 169 | # If data exists, Check type of the data 170 | elif not isinstance(value, data_type): 171 | raise InvalidConfigError(f"Setting {key} should be type {data_type}") 172 | # If type of this data is dict, check if structure of the data is valid. 173 | if data_type is dict: 174 | _ValidateSettingsStruct(data[key], struct[key]["struct"]) 175 | # If type of this data is list, check if all values in the list is valid. 176 | elif data_type is list: 177 | for element in data[key]: 178 | if not isinstance(element, struct[key]["struct"]): 179 | raise InvalidConfigError( 180 | "Setting %s should be list of %s" 181 | % (key, struct[key]["struct"]) 182 | ) 183 | # Check dependency of this attribute. 184 | dependencies = struct[key].get("dependency") 185 | if dependencies: 186 | for dependency in dependencies: 187 | if value == dependency["value"]: 188 | for reqkey in dependency["attribute"]: 189 | _ValidateSettingsElement(data, struct, reqkey) 190 | -------------------------------------------------------------------------------- /pydrive2/test/test_oauth.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import re 4 | import time 5 | import pytest 6 | 7 | from pydrive2.auth import AuthenticationError, GoogleAuth 8 | from pydrive2.test.test_util import ( 9 | setup_credentials, 10 | delete_file, 11 | settings_file_path, 12 | GDRIVE_USER_CREDENTIALS_DATA, 13 | ) 14 | from oauth2client.file import Storage 15 | 16 | 17 | def setup_module(module): 18 | setup_credentials() 19 | 20 | 21 | @pytest.mark.manual 22 | def test_01_LocalWebserverAuthWithClientConfigFromFile(): 23 | # Delete old credentials file 24 | delete_file("credentials/1.dat") 25 | # Test if authentication works with config read from file 26 | ga = GoogleAuth(settings_file_path("test_oauth_test_01.yaml")) 27 | ga.LocalWebserverAuth() 28 | assert not ga.access_token_expired 29 | # Test if correct credentials file is created 30 | CheckCredentialsFile("credentials/1.dat") 31 | time.sleep(1) 32 | 33 | 34 | @pytest.mark.manual 35 | def test_02_LocalWebserverAuthWithClientConfigFromSettings(): 36 | # Delete old credentials file 37 | delete_file("credentials/2.dat") 38 | # Test if authentication works with config read from settings 39 | ga = GoogleAuth(settings_file_path("test_oauth_test_02.yaml")) 40 | ga.LocalWebserverAuth() 41 | assert not ga.access_token_expired 42 | # Test if correct credentials file is created 43 | CheckCredentialsFile("credentials/2.dat") 44 | time.sleep(1) 45 | 46 | 47 | @pytest.mark.manual 48 | def test_03_LocalWebServerAuthWithNoCredentialsSaving(): 49 | # Delete old credentials file 50 | delete_file("credentials/3.dat") 51 | ga = GoogleAuth(settings_file_path("test_oauth_test_03.yaml")) 52 | assert not ga.settings["save_credentials"] 53 | ga.LocalWebserverAuth() 54 | assert not ga.access_token_expired 55 | time.sleep(1) 56 | 57 | 58 | @pytest.mark.manual 59 | def test_04_CommandLineAuthWithClientConfigFromFile(): 60 | # Delete old credentials file 61 | delete_file("credentials/4.dat") 62 | # Test if authentication works with config read from file 63 | ga = GoogleAuth(settings_file_path("test_oauth_test_04.yaml")) 64 | ga.CommandLineAuth() 65 | assert not ga.access_token_expired 66 | # Test if correct credentials file is created 67 | CheckCredentialsFile("credentials/4.dat") 68 | time.sleep(1) 69 | 70 | 71 | @pytest.mark.manual 72 | def test_05_ConfigFromSettingsWithoutOauthScope(): 73 | # Test if authentication works without oauth_scope 74 | ga = GoogleAuth(settings_file_path("test_oauth_test_05.yaml")) 75 | ga.LocalWebserverAuth() 76 | assert not ga.access_token_expired 77 | time.sleep(1) 78 | 79 | 80 | @pytest.mark.skip(reason="P12 authentication is deprecated") 81 | def test_06_ServiceAuthFromSavedCredentialsP12File(): 82 | setup_credentials("credentials/6.dat") 83 | ga = GoogleAuth(settings_file_path("test_oauth_test_06.yaml")) 84 | ga.ServiceAuth() 85 | assert not ga.access_token_expired 86 | time.sleep(1) 87 | 88 | 89 | def test_07_ServiceAuthFromSavedCredentialsJsonFile(): 90 | # Have an initial auth so that credentials/7.dat gets saved 91 | ga = GoogleAuth(settings_file_path("test_oauth_test_07.yaml")) 92 | credentials_file = ga.settings["save_credentials_file"] 93 | # Delete old credentials file 94 | delete_file(credentials_file) 95 | assert not os.path.exists(credentials_file) 96 | ga.ServiceAuth() 97 | assert os.path.exists(credentials_file) 98 | # Secondary auth should be made only using the previously saved 99 | # login info 100 | ga = GoogleAuth(settings_file_path("test_oauth_test_07.yaml")) 101 | ga.ServiceAuth() 102 | assert not ga.access_token_expired 103 | time.sleep(1) 104 | 105 | 106 | def test_08_ServiceAuthFromJsonFileNoCredentialsSaving(): 107 | # Test that no credentials are saved and API is still functional 108 | # We are testing that there are no exceptions at least 109 | ga = GoogleAuth(settings_file_path("test_oauth_test_08.yaml")) 110 | assert not ga.settings["save_credentials"] 111 | ga.ServiceAuth() 112 | time.sleep(1) 113 | 114 | 115 | def test_09_SaveLoadCredentialsUsesDefaultStorage(mocker): 116 | # Test fix for https://github.com/iterative/PyDrive2/issues/163 117 | # Make sure that Load and Save credentials by default reuse the 118 | # same Storage (since it defined lock which make it TS) 119 | ga = GoogleAuth(settings_file_path("test_oauth_test_09.yaml")) 120 | credentials_file = ga.settings["save_credentials_file"] 121 | # Delete old credentials file 122 | delete_file(credentials_file) 123 | assert not os.path.exists(credentials_file) 124 | spy = mocker.spy(Storage, "__init__") 125 | ga.ServiceAuth() 126 | ga.LoadCredentials() 127 | ga.SaveCredentials() 128 | assert spy.call_count == 0 129 | 130 | 131 | def test_10_ServiceAuthFromSavedCredentialsDictionary(): 132 | creds_dict = {} 133 | settings = { 134 | "client_config_backend": "service", 135 | "service_config": { 136 | "client_json_file_path": "/tmp/pydrive2/credentials.json", 137 | }, 138 | "oauth_scope": ["https://www.googleapis.com/auth/drive"], 139 | "save_credentials": True, 140 | "save_credentials_backend": "dictionary", 141 | "save_credentials_dict": creds_dict, 142 | "save_credentials_key": "creds", 143 | } 144 | ga = GoogleAuth(settings=settings) 145 | ga.ServiceAuth() 146 | assert not ga.access_token_expired 147 | assert creds_dict 148 | first_creds_dict = creds_dict.copy() 149 | # Secondary auth should be made only using the previously saved 150 | # login info 151 | ga = GoogleAuth(settings=settings) 152 | ga.ServiceAuth() 153 | assert not ga.access_token_expired 154 | assert creds_dict == first_creds_dict 155 | time.sleep(1) 156 | 157 | 158 | def test_11_ServiceAuthFromJsonNoCredentialsSaving(): 159 | client_json = os.environ[GDRIVE_USER_CREDENTIALS_DATA] 160 | settings = { 161 | "client_config_backend": "service", 162 | "service_config": { 163 | "client_json": client_json, 164 | }, 165 | "oauth_scope": ["https://www.googleapis.com/auth/drive"], 166 | } 167 | # Test that no credentials are saved and API is still functional 168 | # We are testing that there are no exceptions at least 169 | ga = GoogleAuth(settings=settings) 170 | assert not ga.settings["save_credentials"] 171 | ga.ServiceAuth() 172 | time.sleep(1) 173 | 174 | 175 | def test_12_ServiceAuthFromJsonDictNoCredentialsSaving(): 176 | client_json_dict = json.loads(os.environ[GDRIVE_USER_CREDENTIALS_DATA]) 177 | settings = { 178 | "client_config_backend": "service", 179 | "service_config": { 180 | "client_json_dict": client_json_dict, 181 | }, 182 | "oauth_scope": ["https://www.googleapis.com/auth/drive"], 183 | } 184 | # Test that no credentials are saved and API is still functional 185 | # We are testing that there are no exceptions at least 186 | ga = GoogleAuth(settings=settings) 187 | assert not ga.settings["save_credentials"] 188 | ga.ServiceAuth() 189 | time.sleep(1) 190 | 191 | 192 | def test_13_LocalWebServerAuthNonInterativeRaises(monkeypatch): 193 | settings = { 194 | "client_config_backend": "file", 195 | "client_config_file": "client_secrets.json", 196 | "oauth_scope": ["https://www.googleapis.com/auth/drive"], 197 | } 198 | ga = GoogleAuth(settings=settings) 199 | 200 | monkeypatch.setenv("GDRIVE_NON_INTERACTIVE", "true") 201 | # Test that exception is raised on trying to do browser auth if 202 | # we are running in a non interactive environment. 203 | with pytest.raises( 204 | AuthenticationError, 205 | match=re.escape( 206 | "Non interactive mode (GDRIVE_NON_INTERACTIVE env) is enabled" 207 | ), 208 | ): 209 | ga.LocalWebserverAuth() 210 | 211 | 212 | def CheckCredentialsFile(credentials, no_file=False): 213 | ga = GoogleAuth(settings_file_path("test_oauth_default.yaml")) 214 | ga.LoadCredentialsFile(credentials) 215 | assert ga.access_token_expired == no_file 216 | -------------------------------------------------------------------------------- /docs/oauth.rst: -------------------------------------------------------------------------------- 1 | OAuth made easy 2 | =============== 3 | 4 | Authentication in two lines 5 | --------------------------- 6 | 7 | OAuth2.0 is complex and difficult to start with. To make it more simple, 8 | *PyDrive2* makes all authentication into just two lines. 9 | 10 | .. code-block:: python 11 | 12 | from pydrive2.auth import GoogleAuth 13 | 14 | gauth = GoogleAuth() 15 | # Create local webserver and auto handles authentication. 16 | gauth.LocalWebserverAuth() 17 | 18 | # Or use the CommandLineAuth(), which provides you with a link to paste 19 | # into your browser. The site it leads to then provides you with an 20 | # authentication token which you paste into the command line. 21 | # Commented out as it is an alternative to the LocalWebserverAuth() above, 22 | # and someone will just copy-paste the entire thing into their editor. 23 | 24 | # gauth.CommandLineAuth() 25 | 26 | To make this code work, you need to download the application configurations file 27 | from APIs Console. Take a look at quickstart_ for detailed instructions. 28 | 29 | `LocalWebserverAuth()`_ is a built-in method of `GoogleAuth`_ which sets up 30 | local webserver to automatically receive authentication code from user and 31 | authorizes by itself. You can also use `CommandLineAuth()`_ which manually 32 | takes code from user at command line. 33 | 34 | .. _quickstart: /PyDrive2/quickstart/#authentication 35 | .. _`LocalWebserverAuth()`: /PyDrive2/pydrive2/#pydrive2.auth.GoogleAuth.LocalWebserverAuth 36 | .. _`GoogleAuth`: /PyDrive2/pydrive2/#pydrive2.auth.GoogleAuth 37 | .. _`CommandLineAuth()`: /PyDrive2/pydrive2/#pydrive.auth.GoogleAuth.CommandLineAuth 38 | 39 | Automatic and custom authentication with *settings.yaml* 40 | -------------------------------------------------------- 41 | 42 | Read this section if you need a custom authentication flow, **such as silent 43 | authentication on a remote machine**. For an example of such a setup have a look 44 | at `Sample settings.yaml`_. 45 | 46 | OAuth is complicated and it requires a lot of settings. By default, 47 | when you don't provide any settings, *PyDrive* will automatically set default 48 | values which works for most of the cases. Here are some default settings. 49 | 50 | - Read client configuration from file *client_secrets.json* 51 | - OAuth scope: :code:`https://www.googleapis.com/auth/drive` 52 | - Don't save credentials 53 | - Don't retrieve refresh token 54 | 55 | However, you might want to customize these settings while maintaining two lines 56 | of clean code. If that is the case, you can make *settings.yaml* file in your 57 | working directory and *PyDrive* will read it to customize authentication 58 | behavior. 59 | 60 | These are all the possible fields of a *settings.yaml* file: 61 | 62 | .. code-block:: python 63 | 64 | client_config_backend: {{str}} 65 | client_config_file: {{str}} 66 | client_config: 67 | client_id: {{str}} 68 | client_secret: {{str}} 69 | auth_uri: {{str}} 70 | token_uri: {{str}} 71 | redirect_uri: {{str}} 72 | revoke_uri: {{str}} 73 | 74 | service_config: 75 | client_user_email: {{str}} 76 | client_json_file_path: {{str}} 77 | client_json_dict: {{dict}} 78 | client_json: {{str}} 79 | 80 | save_credentials: {{bool}} 81 | save_credentials_backend: {{str}} 82 | save_credentials_file: {{str}} 83 | save_credentials_dict: {{dict}} 84 | save_credentials_key: {{str}} 85 | 86 | get_refresh_token: {{bool}} 87 | 88 | oauth_scope: {{list of str}} 89 | 90 | Fields explained: 91 | 92 | :client_config_backend (str): From where to read client configuration(API application settings such as client_id and client_secrets) from. Valid values are 'file', 'settings' and 'service'. **Default**: 'file'. **Required**: No. 93 | :client_config_file (str): When *client_config_backend* is 'file', path to the file containing client configuration. **Default**: 'client_secrets.json'. **Required**: No. 94 | :client_config (dict): Place holding dictionary for client configuration when *client_config_backend* is 'settings'. **Required**: Yes, only if *client_config_backend* is 'settings' and not using *service_config* 95 | :client_config['client_id'] (str): Client ID of the application. **Required**: Yes, only if *client_config_backend* is 'settings' 96 | :client_config['client_secret'] (str): Client secret of the application. **Required**: Yes, only if *client_config_backend* is 'settings' 97 | :client_config['auth_uri'] (str): The authorization server endpoint URI. **Default**: 'https://accounts.google.com/o/oauth2/auth'. **Required**: No. 98 | :client_config['token_uri'] (str): The token server endpoint URI. **Default**: 'https://accounts.google.com/o/oauth2/token'. **Required**: No. 99 | :client_config['redirect_uri'] (str): Redirection endpoint URI. **Default**: 'urn:ietf:wg:oauth:2.0:oob'. **Required**: No. 100 | :client_config['revoke_uri'] (str): Revoke endpoint URI. **Default**: None. **Required**: No. 101 | :service_config (dict): Place holding dictionary for client configuration when *client_config_backend* is 'service' or 'settings' and using service account. **Required**: Yes, only if *client_config_backend* is 'service' or 'settings' and not using *client_config* 102 | :service_config['client_user_email'] (str): User email that authority was delegated_ to. **Required**: No. 103 | :service_config['client_json_file_path'] (str): Path to service account `.json` key file. **Required**: No. 104 | :service_config['client_json_dict'] (dict): Service account `.json` key file loaded into a dictionary. **Required**: No. 105 | :service_config['client_json'] (str): Service account `.json` key file loaded into a string. **Required**: No. 106 | :save_credentials (bool): True if you want to save credentials. **Default**: False. **Required**: No. 107 | :save_credentials_backend (str): Backend to save credentials to. 'file' and 'dictionary' are the only valid values for now. **Default**: 'file'. **Required**: No. 108 | :save_credentials_file (str): Destination of credentials file. **Required**: Yes, only if *save_credentials_backend* is 'file'. 109 | :save_credentials_dict (dict): Dict to use for storing credentials. **Required**: Yes, only if *save_credentials_backend* is 'dictionary'. 110 | :save_credentials_key (str): Key within the *save_credentials_dict* to store the credentials in. **Required**: Yes, only if *save_credentials_backend* is 'dictionary'. 111 | :get_refresh_token (bool): True if you want to retrieve refresh token along with access token. **Default**: False. **Required**: No. 112 | :oauth_scope (list of str): OAuth scope to authenticate. **Default**: ['https://www.googleapis.com/auth/drive']. **Required**: No. 113 | 114 | .. _delegated: https://developers.google.com/admin-sdk/directory/v1/guides/delegation 115 | 116 | Sample *settings.yaml* 117 | ______________________ 118 | 119 | :: 120 | 121 | client_config_backend: settings 122 | client_config: 123 | client_id: 9637341109347.apps.googleusercontent.com 124 | client_secret: psDskOoWr1P602PXRTHi 125 | 126 | save_credentials: True 127 | save_credentials_backend: file 128 | save_credentials_file: credentials.json 129 | 130 | get_refresh_token: True 131 | 132 | oauth_scope: 133 | - https://www.googleapis.com/auth/drive.file 134 | - https://www.googleapis.com/auth/drive.install 135 | - https://www.googleapis.com/auth/drive.metadata 136 | 137 | Building your own authentication flow 138 | ------------------------------------- 139 | 140 | You might want to build your own authentication flow. For example, you might 141 | want to integrate your existing website with Drive API. In that case, you can 142 | customize an authentication flow as follows: 143 | 144 | 1. Get authentication Url from `GetAuthUrl()`_. 145 | 2. Ask users to visit the authentication Url and grant access to your application. Retrieve authentication code manually by user or automatically by building your own oauth2callback. 146 | 3. Call `Auth(code)`_ with the authentication code you retrieved from step 2. 147 | 148 | Your *settings.yaml* will work for your customized authentication flow, too. 149 | 150 | Here is a sample code for your customized authentication flow 151 | 152 | .. code-block:: python 153 | 154 | from pydrive2.auth import GoogleAuth 155 | 156 | gauth = GoogleAuth() 157 | auth_url = gauth.GetAuthUrl() # Create authentication url user needs to visit 158 | code = AskUserToVisitLinkAndGiveCode(auth_url) # Your customized authentication flow 159 | gauth.Auth(code) # Authorize and build service from the code 160 | 161 | .. _`GetAuthUrl()`: /PyDrive2/pydrive2/#pydrive2.auth.GoogleAuth.GetAuthUrl 162 | .. _`Auth(code)`: /PyDrive2/pydrive2/#pydrive2.auth.GoogleAuth.Auth 163 | 164 | 165 | Authentication with a service account 166 | -------------------------------------- 167 | 168 | A `Service account`_ is a special type of Google account intended to represent a 169 | non-human user that needs to authenticate and be authorized to access data in 170 | Google APIs. 171 | 172 | Typically, service accounts are used in scenarios such as: 173 | 174 | - Running workloads on virtual machines (VMs). 175 | - Running workloads on data centers that call Google APIs. 176 | - Running workloads which are not tied to the lifecycle of a human user. 177 | 178 | If we use OAuth client ID we need to do one manual login into the account with 179 | `LocalWebserverAuth()`_. If we use a service account the login is automatic. 180 | 181 | 182 | .. code-block:: python 183 | 184 | from pydrive2.auth import GoogleAuth 185 | from pydrive2.drive import GoogleDrive 186 | 187 | def login_with_service_account(): 188 | """ 189 | Google Drive service with a service account. 190 | note: for the service account to work, you need to share the folder or 191 | files with the service account email. 192 | 193 | :return: google auth 194 | """ 195 | # Define the settings dict to use a service account 196 | # We also can use all options available for the settings dict like 197 | # oauth_scope,save_credentials,etc. 198 | settings = { 199 | "client_config_backend": "service", 200 | "service_config": { 201 | "client_json_file_path": "service-secrets.json", 202 | } 203 | } 204 | # Create instance of GoogleAuth 205 | gauth = GoogleAuth(settings=settings) 206 | # Authenticate 207 | gauth.ServiceAuth() 208 | return gauth 209 | 210 | 211 | .. _`Service account`: https://developers.google.com/workspace/guides/create-credentials#service-account 212 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2013 Google Inc. All Rights Reserved. 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | 6 | Unless required by applicable law or agreed to in writing, software 7 | distributed under the License is distributed on an "AS IS" BASIS, 8 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 9 | See the License for the specific language governing permissions and 10 | limitations under the License. 11 | 12 | Apache License 13 | Version 2.0, January 2004 14 | http://www.apache.org/licenses/ 15 | 16 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 17 | 18 | 1. Definitions. 19 | 20 | "License" shall mean the terms and conditions for use, reproduction, 21 | and distribution as defined by Sections 1 through 9 of this document. 22 | 23 | "Licensor" shall mean the copyright owner or entity authorized by 24 | the copyright owner that is granting the License. 25 | 26 | "Legal Entity" shall mean the union of the acting entity and all 27 | other entities that control, are controlled by, or are under common 28 | control with that entity. For the purposes of this definition, 29 | "control" means (i) the power, direct or indirect, to cause the 30 | direction or management of such entity, whether by contract or 31 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 32 | outstanding shares, or (iii) beneficial ownership of such entity. 33 | 34 | "You" (or "Your") shall mean an individual or Legal Entity 35 | exercising permissions granted by this License. 36 | 37 | "Source" form shall mean the preferred form for making modifications, 38 | including but not limited to software source code, documentation 39 | source, and configuration files. 40 | 41 | "Object" form shall mean any form resulting from mechanical 42 | transformation or translation of a Source form, including but 43 | not limited to compiled object code, generated documentation, 44 | and conversions to other media types. 45 | 46 | "Work" shall mean the work of authorship, whether in Source or 47 | Object form, made available under the License, as indicated by a 48 | copyright notice that is included in or attached to the work 49 | (an example is provided in the Appendix below). 50 | 51 | "Derivative Works" shall mean any work, whether in Source or Object 52 | form, that is based on (or derived from) the Work and for which the 53 | editorial revisions, annotations, elaborations, or other modifications 54 | represent, as a whole, an original work of authorship. For the purposes 55 | of this License, Derivative Works shall not include works that remain 56 | separable from, or merely link (or bind by name) to the interfaces of, 57 | the Work and Derivative Works thereof. 58 | 59 | "Contribution" shall mean any work of authorship, including 60 | the original version of the Work and any modifications or additions 61 | to that Work or Derivative Works thereof, that is intentionally 62 | submitted to Licensor for inclusion in the Work by the copyright owner 63 | or by an individual or Legal Entity authorized to submit on behalf of 64 | the copyright owner. For the purposes of this definition, "submitted" 65 | means any form of electronic, verbal, or written communication sent 66 | to the Licensor or its representatives, including but not limited to 67 | communication on electronic mailing lists, source code control systems, 68 | and issue tracking systems that are managed by, or on behalf of, the 69 | Licensor for the purpose of discussing and improving the Work, but 70 | excluding communication that is conspicuously marked or otherwise 71 | designated in writing by the copyright owner as "Not a Contribution." 72 | 73 | "Contributor" shall mean Licensor and any individual or Legal Entity 74 | on behalf of whom a Contribution has been received by Licensor and 75 | subsequently incorporated within the Work. 76 | 77 | 2. Grant of Copyright License. Subject to the terms and conditions of 78 | this License, each Contributor hereby grants to You a perpetual, 79 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 80 | copyright license to reproduce, prepare Derivative Works of, 81 | publicly display, publicly perform, sublicense, and distribute the 82 | Work and such Derivative Works in Source or Object form. 83 | 84 | 3. Grant of Patent License. Subject to the terms and conditions of 85 | this License, each Contributor hereby grants to You a perpetual, 86 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 87 | (except as stated in this section) patent license to make, have made, 88 | use, offer to sell, sell, import, and otherwise transfer the Work, 89 | where such license applies only to those patent claims licensable 90 | by such Contributor that are necessarily infringed by their 91 | Contribution(s) alone or by combination of their Contribution(s) 92 | with the Work to which such Contribution(s) was submitted. If You 93 | institute patent litigation against any entity (including a 94 | cross-claim or counterclaim in a lawsuit) alleging that the Work 95 | or a Contribution incorporated within the Work constitutes direct 96 | or contributory patent infringement, then any patent licenses 97 | granted to You under this License for that Work shall terminate 98 | as of the date such litigation is filed. 99 | 100 | 4. Redistribution. You may reproduce and distribute copies of the 101 | Work or Derivative Works thereof in any medium, with or without 102 | modifications, and in Source or Object form, provided that You 103 | meet the following conditions: 104 | 105 | (a) You must give any other recipients of the Work or 106 | Derivative Works a copy of this License; and 107 | 108 | (b) You must cause any modified files to carry prominent notices 109 | stating that You changed the files; and 110 | 111 | (c) You must retain, in the Source form of any Derivative Works 112 | that You distribute, all copyright, patent, trademark, and 113 | attribution notices from the Source form of the Work, 114 | excluding those notices that do not pertain to any part of 115 | the Derivative Works; and 116 | 117 | (d) If the Work includes a "NOTICE" text file as part of its 118 | distribution, then any Derivative Works that You distribute must 119 | include a readable copy of the attribution notices contained 120 | within such NOTICE file, excluding those notices that do not 121 | pertain to any part of the Derivative Works, in at least one 122 | of the following places: within a NOTICE text file distributed 123 | as part of the Derivative Works; within the Source form or 124 | documentation, if provided along with the Derivative Works; or, 125 | within a display generated by the Derivative Works, if and 126 | wherever such third-party notices normally appear. The contents 127 | of the NOTICE file are for informational purposes only and 128 | do not modify the License. You may add Your own attribution 129 | notices within Derivative Works that You distribute, alongside 130 | or as an addendum to the NOTICE text from the Work, provided 131 | that such additional attribution notices cannot be construed 132 | as modifying the License. 133 | 134 | You may add Your own copyright statement to Your modifications and 135 | may provide additional or different license terms and conditions 136 | for use, reproduction, or distribution of Your modifications, or 137 | for any such Derivative Works as a whole, provided Your use, 138 | reproduction, and distribution of the Work otherwise complies with 139 | the conditions stated in this License. 140 | 141 | 5. Submission of Contributions. Unless You explicitly state otherwise, 142 | any Contribution intentionally submitted for inclusion in the Work 143 | by You to the Licensor shall be under the terms and conditions of 144 | this License, without any additional terms or conditions. 145 | Notwithstanding the above, nothing herein shall supersede or modify 146 | the terms of any separate license agreement you may have executed 147 | with Licensor regarding such Contributions. 148 | 149 | 6. Trademarks. This License does not grant permission to use the trade 150 | names, trademarks, service marks, or product names of the Licensor, 151 | except as required for reasonable and customary use in describing the 152 | origin of the Work and reproducing the content of the NOTICE file. 153 | 154 | 7. Disclaimer of Warranty. Unless required by applicable law or 155 | agreed to in writing, Licensor provides the Work (and each 156 | Contributor provides its Contributions) on an "AS IS" BASIS, 157 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 158 | implied, including, without limitation, any warranties or conditions 159 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 160 | PARTICULAR PURPOSE. You are solely responsible for determining the 161 | appropriateness of using or redistributing the Work and assume any 162 | risks associated with Your exercise of permissions under this License. 163 | 164 | 8. Limitation of Liability. In no event and under no legal theory, 165 | whether in tort (including negligence), contract, or otherwise, 166 | unless required by applicable law (such as deliberate and grossly 167 | negligent acts) or agreed to in writing, shall any Contributor be 168 | liable to You for damages, including any direct, indirect, special, 169 | incidental, or consequential damages of any character arising as a 170 | result of this License or out of the use or inability to use the 171 | Work (including but not limited to damages for loss of goodwill, 172 | work stoppage, computer failure or malfunction, or any and all 173 | other commercial damages or losses), even if such Contributor 174 | has been advised of the possibility of such damages. 175 | 176 | 9. Accepting Warranty or Additional Liability. While redistributing 177 | the Work or Derivative Works thereof, You may choose to offer, 178 | and charge a fee for, acceptance of support, warranty, indemnity, 179 | or other liability obligations and/or rights consistent with this 180 | License. However, in accepting such obligations, You may act only 181 | on Your own behalf and on Your sole responsibility, not on behalf 182 | of any other Contributor, and only if You agree to indemnify, 183 | defend, and hold each Contributor harmless for any liability 184 | incurred by, or claims asserted against, such Contributor by reason 185 | of your accepting any such warranty or additional liability. 186 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # PyDrive2 documentation build configuration file, created by 4 | # sphinx-quickstart on Sun Jun 12 23:01:40 2016. 5 | # 6 | # This file is execfile()d with the current directory set to its 7 | # containing dir. 8 | # 9 | # Note that not all possible configuration values are present in this 10 | # autogenerated file. 11 | # 12 | # All configuration values have a default; values that are commented out 13 | # serve to show the default. 14 | 15 | import sys 16 | import os 17 | 18 | # If extensions (or modules to document with autodoc) are in another directory, 19 | # add these directories to sys.path here. If the directory is relative to the 20 | # documentation root, use os.path.abspath to make it absolute, like shown here. 21 | sys.path.insert(0, os.path.abspath("../")) 22 | # exclude_patterns = ['_build', '**tests**', '**spi**'] 23 | exclude_dirnames = ["test"] 24 | # -- General configuration ------------------------------------------------ 25 | 26 | # If your documentation needs a minimal Sphinx version, state it here. 27 | needs_sphinx = "1.8" 28 | 29 | # Add any Sphinx extension module names here, as strings. They can be 30 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 31 | # ones. 32 | extensions = ["sphinx.ext.autodoc", "sphinx.ext.githubpages"] 33 | 34 | # Add any paths that contain templates here, relative to this directory. 35 | templates_path = ["_templates"] 36 | 37 | # The suffix(es) of source filenames. 38 | # You can specify multiple suffix as a list of string: 39 | # source_suffix = ['.rst', '.md'] 40 | source_suffix = ".rst" 41 | 42 | # The encoding of source files. 43 | # source_encoding = 'utf-8-sig' 44 | 45 | # The root toctree document. 46 | root_doc = "index" 47 | 48 | # General information about the project. 49 | project = "PyDrive2" 50 | copyright = ( 51 | "2024, JunYoung Gwak, Scott Blevins, Robin Nabel, Google Inc, " 52 | "Iterative Inc" 53 | ) 54 | author = "JunYoung Gwak, Scott Blevins, Robin Nabel, Iterative Inc" 55 | 56 | # The version info for the project you're documenting, acts as replacement for 57 | # |version| and |release|, also used in various other places throughout the 58 | # built documents. 59 | # 60 | # The short X.Y version. 61 | version = "1.19.0" 62 | # The full version, including alpha/beta/rc tags. 63 | release = "1.19.0" 64 | 65 | # The language for content autogenerated by Sphinx. Refer to documentation 66 | # for a list of supported languages. 67 | # 68 | # This is also used if you do content translation via gettext catalogs. 69 | # Usually you set "language" from the command line for these cases. 70 | language = "en" 71 | 72 | # There are two options for replacing |today|: either, you set today to some 73 | # non-false value, then it is used: 74 | # today = '' 75 | # Else, today_fmt is used as the format for a strftime call. 76 | # today_fmt = '%B %d, %Y' 77 | 78 | # List of patterns, relative to source directory, that match files and 79 | # directories to ignore when looking for source files. 80 | exclude_patterns = [ 81 | "_build", 82 | "pydrive2/test/*", 83 | "test/*", 84 | "pydrive2/test", 85 | "../pydrive2/test", 86 | ] 87 | 88 | # The reST default role (used for this markup: `text`) to use for all 89 | # documents. 90 | # default_role = None 91 | 92 | # If true, '()' will be appended to :func: etc. cross-reference text. 93 | # add_function_parentheses = True 94 | 95 | # If true, the current module name will be prepended to all description 96 | # unit titles (such as .. function::). 97 | # add_module_names = True 98 | 99 | # If true, sectionauthor and moduleauthor directives will be shown in the 100 | # output. They are ignored by default. 101 | # show_authors = False 102 | 103 | # The name of the Pygments (syntax highlighting) style to use. 104 | pygments_style = "sphinx" 105 | 106 | # A list of ignored prefixes for module index sorting. 107 | # modindex_common_prefix = [] 108 | 109 | # If true, keep warnings as "system message" paragraphs in the built documents. 110 | # keep_warnings = False 111 | 112 | # If true, `todo` and `todoList` produce output, else they produce nothing. 113 | todo_include_todos = False 114 | 115 | 116 | # -- Options for HTML output ---------------------------------------------- 117 | 118 | # The theme to use for HTML and HTML Help pages. See the documentation for 119 | # a list of builtin themes. 120 | html_theme = "furo" 121 | 122 | # Theme options are theme-specific and customize the look and feel of a theme 123 | # further. For a list of options available for each theme, see the 124 | # documentation. 125 | # html_theme_options = {} 126 | 127 | # Add any paths that contain custom themes here, relative to this directory. 128 | # html_theme_path = [] 129 | 130 | # The name for this set of Sphinx documents. If None, it defaults to 131 | # " v documentation". 132 | # html_title = None 133 | 134 | # A shorter title for the navigation bar. Default is the same as html_title. 135 | # html_short_title = None 136 | 137 | # The name of an image file (relative to this directory) to place at the top 138 | # of the sidebar. 139 | # html_logo = None 140 | 141 | # The name of an image file (relative to this directory) to use as a favicon of 142 | # the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 143 | # pixels large. 144 | # html_favicon = None 145 | 146 | # Add any paths that contain custom static files (such as style sheets) here, 147 | # relative to this directory. They are copied after the builtin static files, 148 | # so a file named "default.css" will overwrite the builtin "default.css". 149 | html_static_path = [] 150 | 151 | # Add any extra paths that contain custom files (such as robots.txt or 152 | # .htaccess) here, relative to this directory. These files are copied 153 | # directly to the root of the documentation. 154 | # html_extra_path = [] 155 | 156 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 157 | # using the given strftime format. 158 | # html_last_updated_fmt = '%b %d, %Y' 159 | 160 | # If true, SmartyPants will be used to convert quotes and dashes to 161 | # typographically correct entities. 162 | # html_use_smartypants = True 163 | 164 | # Custom sidebar templates, maps document names to template names. 165 | # html_sidebars = {} 166 | 167 | # Additional templates that should be rendered to pages, maps page names to 168 | # template names. 169 | # html_additional_pages = {} 170 | 171 | # If false, no module index is generated. 172 | # html_domain_indices = True 173 | 174 | # If false, no index is generated. 175 | # html_use_index = True 176 | 177 | # If true, the index is split into individual pages for each letter. 178 | # html_split_index = False 179 | 180 | # If true, links to the reST sources are added to the pages. 181 | # html_show_sourcelink = True 182 | 183 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 184 | # html_show_sphinx = True 185 | 186 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 187 | # html_show_copyright = True 188 | 189 | # If true, an OpenSearch description file will be output, and all pages will 190 | # contain a tag referring to it. The value of this option must be the 191 | # base URL from which the finished HTML is served. 192 | # html_use_opensearch = '' 193 | 194 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 195 | # html_file_suffix = None 196 | 197 | # Language to be used for generating the HTML full-text search index. 198 | # Sphinx supports the following languages: 199 | # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' 200 | # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' 201 | # html_search_language = 'en' 202 | 203 | # A dictionary with options for the search language support, empty by default. 204 | # Now only 'ja' uses this config value 205 | # html_search_options = {'type': 'default'} 206 | 207 | # The name of a javascript file (relative to the configuration directory) that 208 | # implements a search results scorer. If empty, the default will be used. 209 | # html_search_scorer = 'scorer.js' 210 | 211 | # Output file base name for HTML help builder. 212 | htmlhelp_basename = "PyDrive2doc" 213 | 214 | # -- Options for LaTeX output --------------------------------------------- 215 | 216 | latex_elements = { 217 | # The paper size ('letterpaper' or 'a4paper'). 218 | # 'papersize': 'letterpaper', 219 | # The font size ('10pt', '11pt' or '12pt'). 220 | # 'pointsize': '10pt', 221 | # Additional stuff for the LaTeX preamble. 222 | # 'preamble': '', 223 | # Latex figure (float) alignment 224 | # 'figure_align': 'htbp', 225 | } 226 | 227 | # Grouping the document tree into LaTeX files. List of tuples 228 | # (source start file, target name, title, 229 | # author, documentclass [howto, manual, or own class]). 230 | latex_documents = [ 231 | ( 232 | root_doc, 233 | "PyDrive2.tex", 234 | "PyDrive2 Documentation", 235 | "JunYoung Gwak, Scott Blevins, Robin Nabel, Iterative Inc", 236 | "manual", 237 | ) 238 | ] 239 | 240 | # The name of an image file (relative to this directory) to place at the top of 241 | # the title page. 242 | # latex_logo = None 243 | 244 | # For "manual" documents, if this is true, then toplevel headings are parts, 245 | # not chapters. 246 | # latex_use_parts = False 247 | 248 | # If true, show page references after internal links. 249 | # latex_show_pagerefs = False 250 | 251 | # If true, show URL addresses after external links. 252 | # latex_show_urls = False 253 | 254 | # Documents to append as an appendix to all manuals. 255 | # latex_appendices = [] 256 | 257 | # If false, no module index is generated. 258 | # latex_domain_indices = True 259 | 260 | 261 | # -- Options for manual page output --------------------------------------- 262 | 263 | # One entry per manual page. List of tuples 264 | # (source start file, name, description, authors, manual section). 265 | man_pages = [(root_doc, "pydrive2", "PyDrive2 Documentation", [author], 1)] 266 | 267 | # If true, show URL addresses after external links. 268 | # man_show_urls = False 269 | 270 | 271 | # -- Options for Texinfo output ------------------------------------------- 272 | 273 | # Grouping the document tree into Texinfo files. List of tuples 274 | # (source start file, target name, title, author, 275 | # dir menu entry, description, category) 276 | texinfo_documents = [ 277 | ( 278 | root_doc, 279 | "PyDrive2", 280 | "PyDrive2 Documentation", 281 | author, 282 | "PyDrive2", 283 | "One line description of project.", 284 | "Miscellaneous", 285 | ) 286 | ] 287 | 288 | # Documents to append as an appendix to all manuals. 289 | # texinfo_appendices = [] 290 | 291 | # If false, no module index is generated. 292 | # texinfo_domain_indices = True 293 | 294 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 295 | # texinfo_show_urls = 'footnote' 296 | 297 | # If true, do not generate a @detailmenu in the "Top" node's menu. 298 | # texinfo_no_detailmenu = False 299 | -------------------------------------------------------------------------------- /pydrive2/test/test_fs.py: -------------------------------------------------------------------------------- 1 | from io import StringIO 2 | import os 3 | import posixpath 4 | import secrets 5 | import uuid 6 | from concurrent import futures 7 | 8 | import pytest 9 | import fsspec 10 | from pydrive2.auth import GoogleAuth 11 | from pydrive2.fs import GDriveFileSystem 12 | from pydrive2.test.test_util import settings_file_path, setup_credentials 13 | from pydrive2.test.test_util import GDRIVE_USER_CREDENTIALS_DATA 14 | 15 | TEST_GDRIVE_REPO_BUCKET = "root" 16 | 17 | 18 | @pytest.fixture(scope="module") 19 | def base_remote_dir(): 20 | path = TEST_GDRIVE_REPO_BUCKET + "/" + str(uuid.uuid4()) 21 | return path 22 | 23 | 24 | @pytest.fixture 25 | def remote_dir(base_remote_dir): 26 | return base_remote_dir + "/" + str(uuid.uuid4()) 27 | 28 | 29 | @pytest.fixture(scope="module") 30 | def service_auth(tmp_path_factory): 31 | setup_credentials() 32 | tmpdir = tmp_path_factory.mktemp("settings") 33 | auth = GoogleAuth(settings_file_path("default.yaml", wkdir=tmpdir)) 34 | auth.ServiceAuth() 35 | return auth 36 | 37 | 38 | @pytest.fixture(scope="module") 39 | def fs_factory(base_remote_dir, service_auth): 40 | base_item = None 41 | GDriveFileSystem.cachable = False 42 | 43 | def _create_fs(): 44 | nonlocal base_item 45 | _, base = base_remote_dir.split("/", 1) 46 | fs = GDriveFileSystem(base_remote_dir, service_auth) 47 | if base_item is None: 48 | base_item = fs._gdrive_create_dir("root", base) 49 | 50 | return fs, base_item 51 | 52 | yield _create_fs 53 | 54 | GDriveFileSystem.cachable = True 55 | fs = GDriveFileSystem(base_remote_dir, service_auth) 56 | fs.rm_file(base_remote_dir) 57 | 58 | 59 | @pytest.fixture 60 | def fs(fs_factory): 61 | return fs_factory()[0] 62 | 63 | 64 | @pytest.mark.manual 65 | def test_fs_oauth(base_remote_dir): 66 | GDriveFileSystem( 67 | base_remote_dir, 68 | client_id="47794215776-cd9ssb6a4vv5otkq6n0iadpgc4efgjb1.apps.googleusercontent.com", # noqa: E501 69 | client_secret="i2gerGA7uBjZbR08HqSOSt9Z", 70 | ) 71 | 72 | 73 | def test_fs_service_json_file(base_remote_dir): 74 | creds = "credentials/fs.dat" 75 | setup_credentials(creds) 76 | GDriveFileSystem( 77 | base_remote_dir, 78 | use_service_account=True, 79 | client_json_file_path=creds, 80 | ) 81 | 82 | 83 | def test_fs_service_json(base_remote_dir): 84 | creds = os.environ[GDRIVE_USER_CREDENTIALS_DATA] 85 | GDriveFileSystem( 86 | base_remote_dir, 87 | use_service_account=True, 88 | client_json=creds, 89 | ) 90 | 91 | 92 | def test_info(fs, remote_dir): 93 | fs.touch(remote_dir + "/info/a.txt") 94 | fs.touch(remote_dir + "/info/b.txt") 95 | details = fs.info(remote_dir + "/info/a.txt") 96 | assert details["type"] == "file" 97 | assert details["name"] == remote_dir + "/info/a.txt" 98 | assert details["size"] == 0 99 | assert ( 100 | details["checksum"] == fs.info(remote_dir + "/info/b.txt")["checksum"] 101 | ) 102 | 103 | details = fs.info(remote_dir + "/info") 104 | assert details["type"] == "directory" 105 | assert details["name"] == remote_dir + "/info/" 106 | assert "checksum" not in details 107 | 108 | details = fs.info(remote_dir + "/info/") 109 | assert details["type"] == "directory" 110 | assert details["name"] == remote_dir + "/info/" 111 | 112 | 113 | def test_move(fs, remote_dir): 114 | fs.touch(remote_dir + "/a.txt") 115 | initial_info = fs.info(remote_dir + "/a.txt") 116 | 117 | fs.move(remote_dir + "/a.txt", remote_dir + "/b.txt") 118 | secondary_info = fs.info(remote_dir + "/b.txt") 119 | 120 | assert not fs.exists(remote_dir + "/a.txt") 121 | assert fs.exists(remote_dir + "/b.txt") 122 | 123 | initial_info.pop("name") 124 | secondary_info.pop("name") 125 | assert initial_info == secondary_info 126 | 127 | 128 | def test_rm(fs, remote_dir): 129 | fs.touch(remote_dir + "/a.txt") 130 | fs.rm(remote_dir + "/a.txt") 131 | assert not fs.exists(remote_dir + "/a.txt") 132 | 133 | fs.mkdir(remote_dir + "/dir") 134 | fs.touch(remote_dir + "/dir/a") 135 | fs.touch(remote_dir + "/dir/b") 136 | fs.mkdir(remote_dir + "/dir/c/") 137 | fs.touch(remote_dir + "/dir/c/a") 138 | fs.rm(remote_dir + "/dir", recursive=True) 139 | assert not fs.exists(remote_dir + "/dir/c/a") 140 | 141 | 142 | def test_ls(fs, remote_dir): 143 | _, base = fs.split_path(remote_dir + "/dir/") 144 | fs._path_to_item_ids(base, create=True) 145 | assert fs.ls(remote_dir + "/dir/") == [] 146 | 147 | files = set() 148 | for no in range(8): 149 | file = remote_dir + f"/dir/test_{no}" 150 | fs.touch(file) 151 | files.add(file) 152 | 153 | assert set(fs.ls(remote_dir + "/dir/")) == files 154 | 155 | dirs = fs.ls(remote_dir + "/dir/", detail=True) 156 | expected = [fs.info(file) for file in files] 157 | 158 | def by_name(details): 159 | return details["name"] 160 | 161 | dirs.sort(key=by_name) 162 | expected.sort(key=by_name) 163 | 164 | assert dirs == expected 165 | 166 | 167 | def test_basic_ops_caching(fs_factory, remote_dir, mocker): 168 | # Internally we have to derefence names into IDs to call GDrive APIs 169 | # we are trying hard to cache those and make sure that operations like 170 | # exists, ls, find, etc. don't hit the API more than once per path 171 | 172 | # ListFile (_gdrive_list) is the main operation that we use to retieve file 173 | # metadata in all operations like find/ls/exist - etc. It should be fine as 174 | # a basic benchmark to count those. 175 | # Note: we can't count direct API calls since we have retries, also can't 176 | # count even direct calls to the GDrive client - for the same reason 177 | fs, _ = fs_factory() 178 | spy = mocker.spy(fs, "_gdrive_list") 179 | 180 | dir_path = remote_dir + "/a/b/c/" 181 | file_path = dir_path + "test.txt" 182 | fs.touch(file_path) 183 | 184 | assert spy.call_count == 5 185 | spy.reset_mock() 186 | 187 | fs.exists(file_path) 188 | assert spy.call_count == 1 189 | spy.reset_mock() 190 | 191 | fs.ls(remote_dir) 192 | assert spy.call_count == 1 193 | spy.reset_mock() 194 | 195 | fs.ls(dir_path) 196 | assert spy.call_count == 1 197 | spy.reset_mock() 198 | 199 | fs.find(dir_path) 200 | assert spy.call_count == 1 201 | spy.reset_mock() 202 | 203 | fs.find(remote_dir) 204 | assert spy.call_count == 1 205 | spy.reset_mock() 206 | 207 | 208 | def test_ops_work_with_duplicate_names(fs_factory, remote_dir): 209 | fs, base_item = fs_factory() 210 | 211 | remote_dir_item = fs._gdrive_create_dir( 212 | base_item["id"], remote_dir.split("/")[-1] 213 | ) 214 | dir_name = str(uuid.uuid4()) 215 | dir1 = fs._gdrive_create_dir(remote_dir_item["id"], dir_name) 216 | dir2 = fs._gdrive_create_dir(remote_dir_item["id"], dir_name) 217 | 218 | # Two directories were created with the same name 219 | assert dir1["id"] != dir2["id"] 220 | 221 | dir_path = remote_dir + "/" + dir_name + "/" 222 | 223 | # ls returns both of them, even though the names are the same 224 | test_fs = fs 225 | result = test_fs.ls(remote_dir) 226 | assert len(result) == 2 227 | assert set(result) == {dir_path} 228 | 229 | # ls returns both of them, even though the names are the same 230 | test_fs, _ = fs_factory() 231 | result = test_fs.ls(remote_dir) 232 | assert len(result) == 2 233 | assert set(result) == {dir_path} 234 | 235 | for test_fs in [fs, fs_factory()[0]]: 236 | # find by default doesn't return dirs at all 237 | result = test_fs.find(remote_dir) 238 | assert len(result) == 0 239 | 240 | fs._gdrive_upload_fobj("a.txt", dir1["id"], StringIO("")) 241 | fs._gdrive_upload_fobj("b.txt", dir2["id"], StringIO("")) 242 | 243 | for test_fs in [fs, fs_factory()[0]]: 244 | # now we should have both files 245 | result = test_fs.find(remote_dir) 246 | assert len(result) == 2 247 | assert set(result) == {dir_path + file for file in ["a.txt", "b.txt"]} 248 | 249 | 250 | def test_ls_non_existing_dir(fs, remote_dir): 251 | with pytest.raises(FileNotFoundError): 252 | fs.ls(remote_dir + "dir/") 253 | 254 | 255 | def test_find(fs, fs_factory, remote_dir): 256 | fs.mkdir(remote_dir + "/dir") 257 | 258 | files = [ 259 | "a", 260 | "b", 261 | "c/a", 262 | "c/b", 263 | "c/d/a", 264 | "c/d/b", 265 | "c/d/c", 266 | "c/d/f/a", 267 | "c/d/f/b", 268 | ] 269 | files = [remote_dir + "/dir/" + file for file in files] 270 | dirnames = {posixpath.dirname(file) for file in files} 271 | 272 | for dirname in dirnames: 273 | fs.mkdir(dirname) 274 | 275 | for file in files: 276 | fs.touch(file) 277 | 278 | for test_fs in [fs, fs_factory()[0]]: 279 | # Test for https://github.com/iterative/PyDrive2/issues/229 280 | # It must go first, so that we test with a cache miss as well 281 | assert set(test_fs.find(remote_dir + "/dir/c/d/")) == set( 282 | [ 283 | file 284 | for file in files 285 | if file.startswith(remote_dir + "/dir/c/d/") 286 | ] 287 | ) 288 | 289 | # General find test 290 | assert set(test_fs.find(remote_dir)) == set(files) 291 | 292 | find_results = test_fs.find(remote_dir, detail=True) 293 | info_results = [test_fs.info(file) for file in files] 294 | info_results = {content["name"]: content for content in info_results} 295 | assert find_results == info_results 296 | 297 | 298 | def test_exceptions(fs, tmpdir, remote_dir): 299 | with pytest.raises(FileNotFoundError): 300 | with fs.open(remote_dir + "/a.txt"): 301 | ... 302 | 303 | with pytest.raises(FileNotFoundError): 304 | fs.copy(remote_dir + "/u.txt", remote_dir + "/y.txt") 305 | 306 | with pytest.raises(FileNotFoundError): 307 | fs.get_file(remote_dir + "/c.txt", tmpdir / "c.txt") 308 | 309 | 310 | def test_open_rw(fs, remote_dir): 311 | data = b"dvc.org" 312 | 313 | with fs.open(remote_dir + "/a.txt", "wb") as stream: 314 | stream.write(data) 315 | 316 | with fs.open(remote_dir + "/a.txt") as stream: 317 | assert stream.read() == data 318 | 319 | 320 | def test_concurrent_operations(fs, fs_factory, remote_dir): 321 | # Include an extra dir name to force upload operations creating it 322 | # this way we can also test that only a single directory is created 323 | # enven if multiple threads are uploading files into the same dir 324 | dir_name = secrets.token_hex(16) 325 | 326 | def create_random_file(): 327 | name = secrets.token_hex(16) 328 | with fs.open(remote_dir + f"/{dir_name}/" + name, "w") as stream: 329 | stream.write(name) 330 | return name 331 | 332 | def read_random_file(name): 333 | with fs.open(remote_dir + f"/{dir_name}/" + name, "r") as stream: 334 | return stream.read() 335 | 336 | with futures.ThreadPoolExecutor() as executor: 337 | write_futures, _ = futures.wait( 338 | [executor.submit(create_random_file) for _ in range(64)], 339 | return_when=futures.ALL_COMPLETED, 340 | ) 341 | write_names = {future.result() for future in write_futures} 342 | 343 | read_futures, _ = futures.wait( 344 | [executor.submit(read_random_file, name) for name in write_names], 345 | return_when=futures.ALL_COMPLETED, 346 | ) 347 | read_names = {future.result() for future in read_futures} 348 | 349 | assert write_names == read_names 350 | 351 | # Test that only a single dir is cretead 352 | for test_fs in [fs, fs_factory()[0]]: 353 | results = test_fs.ls(remote_dir) 354 | assert results == [remote_dir + f"/{dir_name}/"] 355 | 356 | 357 | def test_put_file(fs, tmpdir, remote_dir): 358 | src_file = tmpdir / "a.txt" 359 | with open(src_file, "wb") as file: 360 | file.write(b"data") 361 | 362 | fs.put_file(src_file, remote_dir + "/a.txt") 363 | 364 | with fs.open(remote_dir + "/a.txt") as stream: 365 | assert stream.read() == b"data" 366 | 367 | 368 | def test_get_file(fs, tmpdir, remote_dir): 369 | src_file = tmpdir / "a.txt" 370 | dest_file = tmpdir / "b.txt" 371 | 372 | with open(src_file, "wb") as file: 373 | file.write(b"data") 374 | 375 | fs.put_file(src_file, remote_dir + "/a.txt") 376 | fs.get_file(remote_dir + "/a.txt", dest_file) 377 | assert dest_file.read() == "data" 378 | 379 | 380 | def test_get_file_callback(fs, tmpdir, remote_dir): 381 | src_file = tmpdir / "a.txt" 382 | dest_file = tmpdir / "b.txt" 383 | 384 | with open(src_file, "wb") as file: 385 | file.write(b"data" * 10) 386 | 387 | fs.put_file(src_file, remote_dir + "/a.txt") 388 | callback = fsspec.Callback() 389 | fs.get_file( 390 | remote_dir + "/a.txt", dest_file, callback=callback, block_size=10 391 | ) 392 | assert dest_file.read() == "data" * 10 393 | 394 | assert callback.size == 40 395 | assert callback.value == 40 396 | -------------------------------------------------------------------------------- /docs/filemanagement.rst: -------------------------------------------------------------------------------- 1 | File management made easy 2 | ========================= 3 | 4 | There are many methods to create and update file metadata and contents. 5 | With *PyDrive*, you don't have to care about any of these different API methods. 6 | Manipulate file metadata and contents from `GoogleDriveFile`_ object and call 7 | `Upload()`_. *PyDrive* will make the optimal API call for you. 8 | 9 | Upload a new file 10 | ----------------- 11 | 12 | Here is a sample code to upload a file. ``gauth`` is an authenticated `GoogleAuth`_ object. 13 | 14 | .. code-block:: python 15 | 16 | from pydrive2.drive import GoogleDrive 17 | 18 | # Create GoogleDrive instance with authenticated GoogleAuth instance. 19 | drive = GoogleDrive(gauth) 20 | 21 | # Create GoogleDriveFile instance with title 'Hello.txt'. 22 | file1 = drive.CreateFile({'title': 'Hello.txt'}) 23 | file1.Upload() # Upload the file. 24 | print('title: %s, id: %s' % (file1['title'], file1['id'])) 25 | # title: Hello.txt, id: {{FILE_ID}} 26 | 27 | Now, you will have a file 'Hello.txt' uploaded to your Google Drive. You can open it from web interface to check its content, 'Hello World!'. 28 | 29 | Note that `CreateFile()`_ will create `GoogleDriveFile`_ instance but not actually upload a file to Google Drive. You can initialize `GoogleDriveFile`_ object by itself. However, it is not recommended to do so in order to keep authentication consistent. 30 | 31 | Delete, Trash and un-Trash files 32 | -------------------------------- 33 | You may want to delete, trash, or un-trash a file. To do this use ``Delete()``, 34 | ``Trash()`` or ``UnTrash()`` on a GoogleDriveFile object. 35 | 36 | *Note:* ``Trash()`` *moves a file into the trash and can be recovered,* 37 | ``Delete()`` *deletes the file permanently and immediately.* 38 | 39 | .. code-block:: python 40 | 41 | # Create GoogleDriveFile instance and upload it. 42 | file1 = drive.CreateFile() 43 | file1.Upload() 44 | 45 | file1.Trash() # Move file to trash. 46 | file1.UnTrash() # Move file out of trash. 47 | file1.Delete() # Permanently delete the file. 48 | 49 | Update file metadata 50 | -------------------- 51 | 52 | You can manipulate file metadata from a `GoogleDriveFile`_ object just as you manipulate a ``dict``. 53 | The format of file metadata can be found in the Google Drive API documentation: `Files resource`_. 54 | 55 | Sample code continues from `Upload a new file`_: 56 | 57 | .. code-block:: python 58 | 59 | file1['title'] = 'HelloWorld.txt' # Change title of the file. 60 | file1.Upload() # Update metadata. 61 | print('title: %s' % file1['title']) # title: HelloWorld.txt. 62 | 63 | Now, the title of your file has changed to 'HelloWorld.txt'. 64 | 65 | Download file metadata from file ID 66 | ----------------------------------- 67 | 68 | You might want to get file metadata from file ID. In that case, just initialize 69 | `GoogleDriveFile`_ with file ID and access metadata from `GoogleDriveFile`_ 70 | just as you access ``dict``. 71 | 72 | Sample code continues from above: 73 | 74 | .. code-block:: python 75 | 76 | # Create GoogleDriveFile instance with file id of file1. 77 | file2 = drive.CreateFile({'id': file1['id']}) 78 | print('title: %s, mimeType: %s' % (file2['title'], file2['mimeType'])) 79 | # title: HelloWorld.txt, mimeType: text/plain 80 | 81 | Handling special metadata 82 | ------------------------- 83 | 84 | Not all metadata can be set with the methods described above. 85 | PyDrive gives you access to the metadata of an object through 86 | ``file_object.FetchMetadata()``. This function has two optional parameters: 87 | ``fields`` and ``fetch_all``. 88 | 89 | .. code-block:: python 90 | 91 | file1 = drive.CreateFile({'id': ''}) 92 | 93 | # Fetches all basic metadata fields, including file size, last modified etc. 94 | file1.FetchMetadata() 95 | 96 | # Fetches all metadata available. 97 | file1.FetchMetadata(fetch_all=True) 98 | 99 | # Fetches the 'permissions' metadata field. 100 | file1.FetchMetadata(fields='permissions') 101 | # You can update a list of specific fields like this: 102 | file1.FetchMetadata(fields='permissions,labels,mimeType') 103 | 104 | For more information on available metadata fields have a look at the 105 | `official documentation`_. 106 | 107 | Insert permissions 108 | __________________ 109 | Insert, retrieving or deleting permissions is illustrated by making a file 110 | readable to all who have a link to the file. 111 | 112 | .. code-block:: python 113 | 114 | file1 = drive.CreateFile() 115 | file1.Upload() 116 | 117 | # Insert the permission. 118 | permission = file1.InsertPermission({ 119 | 'type': 'anyone', 120 | 'value': 'anyone', 121 | 'role': 'reader'}) 122 | 123 | print(file1['alternateLink']) # Display the sharable link. 124 | 125 | Note: ``InsertPermission()`` calls ``GetPermissions()`` after successfully 126 | inserting the permission. 127 | 128 | You can find more information on the permitted fields of a permission 129 | `here `_. 130 | This file is now shared and anyone with the link can view it. But what if you 131 | want to check whether a file is already shared? 132 | 133 | List permissions 134 | ________________ 135 | 136 | Permissions can be fetched using the ``GetPermissions()`` function of a 137 | ``GoogleDriveFile``, and can be used like so: 138 | 139 | .. code-block:: python 140 | 141 | # Create a new file 142 | file1 = drive.CreateFile() 143 | # Fetch permissions. 144 | permissions = file1.GetPermissions() 145 | print(permissions) 146 | 147 | # The permissions are also available as file1['permissions']: 148 | print(file1['permissions']) 149 | 150 | For the more advanced user: ``GetPermissions()`` is a shorthand for: 151 | 152 | .. code-block:: python 153 | 154 | # Fetch Metadata, including the permissions field. 155 | file1.FetchMetadata(fields='permissions') 156 | 157 | # The permissions array is now available for further use. 158 | print(file1['permissions']) 159 | 160 | Remove a Permission 161 | ___________________ 162 | *PyDrive* allows you to remove a specific permission using the 163 | ``DeletePermission(permission_id)`` function. This function allows you to delete 164 | one permission at a time by providing the permission's ID. 165 | 166 | .. code-block:: python 167 | 168 | file1 = drive.CreateFile({'id': ''}) 169 | permissions = file1.GetPermissions() # Download file permissions. 170 | 171 | permission_id = permissions[1]['id'] # Get a permission ID. 172 | 173 | file1.DeletePermission(permission_id) # Delete the permission. 174 | 175 | Get files by complex queries 176 | ---------------------------- 177 | 178 | We can get a file by name and by other constraints, usually a filename will be 179 | unique but we can have two equal names with different extensions, e.g., 180 | *123.jpeg and 123.mp3*. So if you expect only one file add more constraints to 181 | the query, see `Query string examples `_, as a result we get 182 | a list of `GoogleDriveFile`_ instances. 183 | 184 | .. code-block:: python 185 | 186 | from pydrive2.drive import GoogleDrive 187 | # Create GoogleDrive instance with authenticated GoogleAuth instance. 188 | drive = GoogleDrive(gauth) 189 | filename = 'file_test' 190 | # Query 191 | query = {'q': f"title = '{filename}' and mimeType='{mimetype}'"} 192 | # Get list of files that match against the query 193 | files = drive.ListFile(query).GetList() 194 | 195 | List revisions 196 | ________________ 197 | 198 | Revisions can be fetched using the ``GetRevisions()`` function of a 199 | ``GoogleDriveFile``, and can be used like so: 200 | 201 | .. code-block:: python 202 | 203 | # Create a new file 204 | file1 = drive.CreateFile() 205 | # Fetch revisions. 206 | revisions = file1.GetRevisions() 207 | print(revisions) 208 | 209 | Not all files objects have revisions. If GetRevisions is called on a 210 | file object that does not have revisions, an exception will be raised. 211 | 212 | Upload and update file content 213 | ------------------------------ 214 | 215 | Managing file content is as easy as managing file metadata. You can set file 216 | content with either `SetContentFile(filename)`_ or `SetContentString(content)`_ 217 | and call `Upload()`_ just as you did to upload or update file metadata. 218 | 219 | Sample code continues from `Download file metadata from file ID`_: 220 | 221 | .. code-block:: python 222 | 223 | file4 = drive.CreateFile({'title':'appdata.json', 'mimeType':'application/json'}) 224 | file4.SetContentString('{"firstname": "John", "lastname": "Smith"}') 225 | file4.Upload() # Upload file. 226 | file4.SetContentString('{"firstname": "Claudio", "lastname": "Afshar"}') 227 | file4.Upload() # Update content of the file. 228 | 229 | file5 = drive.CreateFile() 230 | # Read file and set it as a content of this instance. 231 | file5.SetContentFile('cat.png') 232 | file5.Upload() # Upload the file. 233 | print('title: %s, mimeType: %s' % (file5['title'], file5['mimeType'])) 234 | # title: cat.png, mimeType: image/png 235 | 236 | **Advanced Users:** If you call SetContentFile and GetContentFile you can can 237 | define which character encoding is to be used by using the optional 238 | parameter `encoding`. 239 | 240 | If you, for example, are retrieving a file which is stored on your Google 241 | Drive which is encoded with ISO-8859-1, then you can get the content string 242 | like so: 243 | 244 | .. code-block:: python 245 | 246 | content_string = file4.GetContentString(encoding='ISO-8859-1') 247 | 248 | Upload data as bytes in memory buffer 249 | -------------------------------------- 250 | 251 | Data can be kept as bytes in an in-memory buffer when we use the ``io`` module’s 252 | Byte IO operations, we can upload files that reside in memory, for 253 | example we have a base64 image, we can decode the string and upload it to drive 254 | without the need to save as a file and use `SetContentFile(filename)`_ 255 | 256 | .. code-block:: python 257 | 258 | import io 259 | from pydrive2.drive import GoogleDrive 260 | 261 | # Create GoogleDrive instance with authenticated GoogleAuth instance. 262 | drive = GoogleDrive(gauth) 263 | # Define file name and type 264 | metadata = { 265 | 'title': 'image_test', 266 | 'mimeType': 'image/jpeg' 267 | } 268 | # Create file 269 | file = drive.CreateFile(metadata=metadata) 270 | # Buffered I/O implementation using an in-memory bytes buffer. 271 | image_file = io.BytesIO(image_bytes) 272 | # Set the content of the file 273 | file.content = image_file 274 | # Upload the file to google drive 275 | file.Upload() 276 | 277 | Upload file to a specific folder 278 | -------------------------------- 279 | 280 | In order to upload a file into a specific drive folder we need to pass the 281 | ``id`` of the folder in the metadata ``param`` from `CreateFile()`_. 282 | Save the image from the previous example into a specific folder``:`` 283 | 284 | .. code-block:: python 285 | 286 | metadata = { 287 | 'parents': [ 288 | {"id": id_drive_folder} 289 | ], 290 | 'title': 'image_test', 291 | 'mimeType': 'image/jpeg' 292 | } 293 | # Create file 294 | file = drive.CreateFile(metadata=metadata) 295 | file.Upload() 296 | 297 | Download file content 298 | --------------------- 299 | 300 | Just as you uploaded file content, you can download it using 301 | `GetContentFile(filename)`_ or `GetContentString()`_. 302 | 303 | Sample code continues from above: 304 | 305 | .. code-block:: python 306 | 307 | # Initialize GoogleDriveFile instance with file id. 308 | file6 = drive.CreateFile({'id': file5['id']}) 309 | file6.GetContentFile('catlove.png') # Download file as 'catlove.png'. 310 | 311 | # Initialize GoogleDriveFile instance with file id. 312 | file7 = drive.CreateFile({'id': file4['id']}) 313 | content = file7.GetContentString() 314 | # content: '{"firstname": "Claudio", "lastname": "Afshar"}' 315 | 316 | file7.SetContentString(content.replace('lastname', 'familyname')) 317 | file7.Upload() 318 | # Uploaded content: '{"firstname": "Claudio", "familyname": "Afshar"}' 319 | 320 | **Advanced users**: Google Drive is `known`_ to add BOM (Byte Order Marks) to 321 | the beginning of some files, such as Google Documents downloaded as text files. 322 | In some cases confuses parsers and leads to corrupt files. 323 | PyDrive can remove the BOM from the beginning of a file when it 324 | is downloaded. Just set the `remove_bom` parameter in `GetContentString()` or 325 | `GetContentFile()` - see `examples/strip_bom_example.py` in the GitHub 326 | repository for an example. 327 | 328 | Abusive files 329 | ------------- 330 | 331 | Files identified as `abusive`_ (malware, etc.) are only downloadable by the owner. 332 | If you see a 333 | 'This file has been identified as malware or spam and cannot be downloaded' 334 | error, set 'acknowledge_abuse=True' parameter in `GetContentFile()`. By using 335 | it you indicate that you acknowledge the risks of downloading potential malware. 336 | 337 | .. _`GoogleDriveFile`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile 338 | .. _`Upload()`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.Upload 339 | .. _`GoogleAuth`: /PyDrive2/pydrive2/#pydrive2.auth.GoogleAuth 340 | .. _`CreateFile()`: /PyDrive2/pydrive2/#pydrive2.drive.GoogleDrive.CreateFile 341 | .. _`Files resource`: https://developers.google.com/drive/v2/reference/files#resource-representations 342 | .. _`SetContentFile(filename)`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.SetContentFile 343 | .. _`SetContentString(content)`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.SetContentString 344 | .. _`GetContentFile(filename)`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.GetContentFile 345 | .. _`GetContentString()`: /PyDrive2/pydrive2/#pydrive2.files.GoogleDriveFile.GetContentString 346 | .. _`official documentation`: https://developers.google.com/drive/v2/reference/files#resource-representations 347 | .. _`known`: https://productforums.google.com/forum/#!topic/docs/BJLimQDGtjQ 348 | .. _`abusive`: https://support.google.com/docs/answer/148505 349 | .. _`query_parameters`: https://developers.google.com/drive/api/guides/search-files#examples 350 | -------------------------------------------------------------------------------- /pydrive2/fs/spec.py: -------------------------------------------------------------------------------- 1 | import appdirs 2 | import errno 3 | import io 4 | import logging 5 | import os 6 | import posixpath 7 | import threading 8 | from collections import defaultdict 9 | from contextlib import contextmanager 10 | from itertools import chain 11 | 12 | from fsspec.spec import AbstractFileSystem 13 | from funcy import cached_property, retry, wrap_prop, wrap_with 14 | from tqdm.utils import CallbackIOWrapper 15 | 16 | from pydrive2.drive import GoogleDrive 17 | from pydrive2.fs.utils import IterStream 18 | from pydrive2.auth import GoogleAuth 19 | 20 | logger = logging.getLogger(__name__) 21 | 22 | FOLDER_MIME_TYPE = "application/vnd.google-apps.folder" 23 | 24 | COMMON_SETTINGS = { 25 | "get_refresh_token": True, 26 | "oauth_scope": [ 27 | "https://www.googleapis.com/auth/drive", 28 | "https://www.googleapis.com/auth/drive.appdata", 29 | ], 30 | } 31 | 32 | 33 | class GDriveAuthError(Exception): 34 | pass 35 | 36 | 37 | def _gdrive_retry(func): 38 | def should_retry(exc): 39 | from pydrive2.files import ApiRequestError 40 | 41 | if not isinstance(exc, ApiRequestError): 42 | return False 43 | 44 | error_code = exc.error.get("code", 0) 45 | result = False 46 | if 500 <= error_code < 600: 47 | result = True 48 | 49 | if error_code == 403: 50 | result = exc.GetField("reason") in [ 51 | "userRateLimitExceeded", 52 | "rateLimitExceeded", 53 | ] 54 | if result: 55 | logger.debug(f"Retrying GDrive API call, error: {exc}.") 56 | 57 | return result 58 | 59 | # 16 tries, start at 0.5s, multiply by golden ratio, cap at 20s 60 | return retry( 61 | 16, 62 | timeout=lambda a: min(0.5 * 1.618**a, 20), 63 | filter_errors=should_retry, 64 | )(func) 65 | 66 | 67 | @contextmanager 68 | def _wrap_errors(): 69 | try: 70 | yield 71 | except Exception as exc: 72 | # Handle AuthenticationError, RefreshError and other auth failures 73 | # It's hard to come up with a narrow exception, since PyDrive throws 74 | # a lot of different errors - broken credentials file, refresh token 75 | # expired, flow failed, etc. 76 | raise GDriveAuthError("Failed to authenticate GDrive") from exc 77 | 78 | 79 | def _client_auth( 80 | client_id=None, 81 | client_secret=None, 82 | client_json=None, 83 | client_json_file_path=None, 84 | profile=None, 85 | ): 86 | if client_json: 87 | save_settings = { 88 | "save_credentials_backend": "dictionary", 89 | "save_credentials_dict": {"creds": client_json}, 90 | "save_credentials_key": "creds", 91 | } 92 | else: 93 | creds_file = client_json_file_path 94 | if not creds_file: 95 | cache_dir = os.path.join( 96 | appdirs.user_cache_dir("pydrive2fs", appauthor=False), 97 | client_id, 98 | ) 99 | os.makedirs(cache_dir, exist_ok=True) 100 | 101 | profile = profile or "default" 102 | creds_file = os.path.join(cache_dir, f"{profile}.json") 103 | 104 | save_settings = { 105 | "save_credentials_backend": "file", 106 | "save_credentials_file": creds_file, 107 | } 108 | 109 | settings = { 110 | **COMMON_SETTINGS, 111 | "save_credentials": True, 112 | **save_settings, 113 | "client_config_backend": "settings", 114 | "client_config": { 115 | "client_id": client_id, 116 | "client_secret": client_secret, 117 | "auth_uri": "https://accounts.google.com/o/oauth2/auth", 118 | "token_uri": "https://oauth2.googleapis.com/token", 119 | "revoke_uri": "https://oauth2.googleapis.com/revoke", 120 | "redirect_uri": "", 121 | }, 122 | } 123 | 124 | auth = GoogleAuth(settings=settings) 125 | 126 | with _wrap_errors(): 127 | auth.LocalWebserverAuth() 128 | 129 | return auth 130 | 131 | 132 | def _service_auth( 133 | client_user_email=None, 134 | client_json=None, 135 | client_json_file_path=None, 136 | ): 137 | settings = { 138 | **COMMON_SETTINGS, 139 | "client_config_backend": "service", 140 | "service_config": { 141 | "client_user_email": client_user_email, 142 | "client_json": client_json, 143 | "client_json_file_path": client_json_file_path, 144 | }, 145 | } 146 | 147 | auth = GoogleAuth(settings=settings) 148 | 149 | with _wrap_errors(): 150 | auth.ServiceAuth() 151 | 152 | return auth 153 | 154 | 155 | class GDriveFileSystem(AbstractFileSystem): 156 | """Access to gdrive as an fsspec filesystem""" 157 | 158 | def __init__( 159 | self, 160 | path, 161 | google_auth=None, 162 | client_id=None, 163 | client_secret=None, 164 | client_user_email=None, 165 | client_json=None, 166 | client_json_file_path=None, 167 | use_service_account=False, 168 | profile=None, 169 | trash_only=True, 170 | acknowledge_abuse=False, 171 | **kwargs, 172 | ): 173 | """Create an instance of GDriveFileSystem. 174 | 175 | :param path: gdrive path. 176 | :type path: str. 177 | :param google_auth: Authenticated GoogleAuth instance. 178 | :type google_auth: GoogleAuth. 179 | :param client_id: Client ID of the application. 180 | :type client_id: str 181 | :param client_secret: Client secret of the application. 182 | :type client_secret: str. 183 | :param client_user_email: User email that authority was delegated to 184 | (only for service account). 185 | :type client_user_email: str. 186 | :param client_json: JSON keyfile loaded into a string. 187 | :type client_json: str. 188 | :param client_json_file_path: Path to JSON keyfile. 189 | :type client_json_file_path: str. 190 | :param use_service_account: Use service account. 191 | :type use_service_account: bool. 192 | :param profile: Profile name for caching credentials 193 | (ignored for service account). 194 | :param trash_only: Move files to trash instead of deleting. 195 | :type trash_only: bool. 196 | :param acknowledge_abuse: Acknowledging the risk and download file 197 | identified as abusive. 198 | :type acknowledge_abuse: bool 199 | :type profile: str. 200 | :raises: GDriveAuthError 201 | """ 202 | super().__init__(**kwargs) 203 | self.path = path 204 | self.root, self.base = self.split_path(self.path) 205 | 206 | if not google_auth: 207 | if ( 208 | not client_json 209 | and not client_json_file_path 210 | and not (client_id and client_secret) 211 | ): 212 | raise ValueError( 213 | "Specify credentials using one of these methods: " 214 | "client_id/client_secret or " 215 | "client_json or " 216 | "client_json_file_path" 217 | ) 218 | 219 | if use_service_account: 220 | google_auth = _service_auth( 221 | client_json=client_json, 222 | client_json_file_path=client_json_file_path, 223 | client_user_email=client_user_email, 224 | ) 225 | else: 226 | google_auth = _client_auth( 227 | client_id=client_id, 228 | client_secret=client_secret, 229 | client_json=client_json, 230 | client_json_file_path=client_json_file_path, 231 | profile=profile, 232 | ) 233 | 234 | self.client = GoogleDrive(google_auth) 235 | self._trash_only = trash_only 236 | self._acknowledge_abuse = acknowledge_abuse 237 | 238 | def split_path(self, path): 239 | parts = path.replace("//", "/").rstrip("/").split("/", 1) 240 | if len(parts) == 2: 241 | return parts 242 | else: 243 | return parts[0], "" 244 | 245 | @wrap_prop(threading.RLock()) 246 | @cached_property 247 | def _ids_cache(self): 248 | cache = {"dirs": defaultdict(list), "ids": {}} 249 | 250 | base_item_ids = self._path_to_item_ids(self.base, use_cache=False) 251 | if not base_item_ids: 252 | raise FileNotFoundError( 253 | errno.ENOENT, 254 | os.strerror(errno.ENOENT), 255 | f"Confirm {self.path} exists and you can access it", 256 | ) 257 | 258 | self._cache_path_id(self.base, *base_item_ids, cache=cache) 259 | 260 | return cache 261 | 262 | def _cache_path_id(self, path, *item_ids, cache=None): 263 | cache = cache or self._ids_cache 264 | for item_id in item_ids: 265 | cache["ids"][item_id] = path 266 | cache["dirs"][path].append(item_id) 267 | 268 | @cached_property 269 | def _list_params(self): 270 | params = {"corpora": "default"} 271 | if self.root != "root" and self.root != "appDataFolder": 272 | drive_id = self._gdrive_shared_drive_id(self.root) 273 | if drive_id: 274 | logger.debug( 275 | "GDrive remote '{}' is using shared drive id '{}'.".format( 276 | self.path, drive_id 277 | ) 278 | ) 279 | params["driveId"] = drive_id 280 | params["corpora"] = "drive" 281 | return params 282 | 283 | @_gdrive_retry 284 | def _gdrive_shared_drive_id(self, item_id): 285 | from pydrive2.files import ApiRequestError 286 | 287 | param = {"id": item_id} 288 | # it does not create a file on the remote 289 | item = self.client.CreateFile(param) 290 | # ID of the shared drive the item resides in. 291 | # Only populated for items in shared drives. 292 | try: 293 | item.FetchMetadata("driveId") 294 | except ApiRequestError as exc: 295 | error_code = exc.error.get("code", 0) 296 | if error_code == 404: 297 | raise PermissionError from exc 298 | raise 299 | 300 | return item.get("driveId", None) 301 | 302 | def _gdrive_list(self, query): 303 | param = {"q": query, "maxResults": 1000} 304 | param.update(self._list_params) 305 | file_list = self.client.ListFile(param) 306 | 307 | # Isolate and decorate fetching of remote drive items in pages. 308 | get_list = _gdrive_retry(lambda: next(file_list, None)) 309 | 310 | # Fetch pages until None is received, lazily flatten the thing. 311 | return chain.from_iterable(iter(get_list, None)) 312 | 313 | def _gdrive_list_ids(self, query_ids): 314 | query = " or ".join( 315 | f"'{query_id}' in parents" for query_id in query_ids 316 | ) 317 | query = f"({query}) and trashed=false" 318 | return self._gdrive_list(query) 319 | 320 | def _get_remote_item_ids( 321 | self, parent_ids, parent_path, title, use_cache=True 322 | ): 323 | if not parent_ids: 324 | return None 325 | query = "trashed=false and ({})".format( 326 | " or ".join( 327 | f"'{parent_id}' in parents" for parent_id in parent_ids 328 | ) 329 | ) 330 | query += " and title='{}'".format(title.replace("'", "\\'")) 331 | 332 | res = [] 333 | for item in self._gdrive_list(query): 334 | # GDrive list API is case insensitive, we need to compare 335 | # all results and pick the ones with the right title 336 | if item["title"] == title: 337 | res.append(item["id"]) 338 | 339 | if item["mimeType"] == FOLDER_MIME_TYPE and use_cache: 340 | self._cache_path_id( 341 | posixpath.join(parent_path, item["title"]), item["id"] 342 | ) 343 | 344 | return res 345 | 346 | def _get_cached_item_ids(self, path, use_cache): 347 | if not path: 348 | return [self.root] 349 | if use_cache: 350 | return self._ids_cache["dirs"].get(path, []) 351 | return [] 352 | 353 | def _path_to_item_ids(self, path, create=False, use_cache=True): 354 | item_ids = self._get_cached_item_ids(path, use_cache) 355 | if item_ids: 356 | return item_ids 357 | 358 | parent_path, title = posixpath.split(path) 359 | parent_ids = self._path_to_item_ids(parent_path, create, use_cache) 360 | item_ids = self._get_remote_item_ids( 361 | parent_ids, parent_path, title, use_cache 362 | ) 363 | if item_ids: 364 | return item_ids 365 | 366 | return ( 367 | [self._create_dir(min(parent_ids), title, path)] if create else [] 368 | ) 369 | 370 | def _get_item_id(self, path, create=False, use_cache=True): 371 | bucket, base = self.split_path(path) 372 | assert bucket == self.root 373 | 374 | item_ids = self._path_to_item_ids(base, create, use_cache) 375 | if item_ids: 376 | return min(item_ids) 377 | 378 | assert not create 379 | raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path) 380 | 381 | @_gdrive_retry 382 | def _gdrive_create_dir(self, parent_id, title): 383 | parent = {"id": parent_id} 384 | item = self.client.CreateFile( 385 | {"title": title, "parents": [parent], "mimeType": FOLDER_MIME_TYPE} 386 | ) 387 | item.Upload() 388 | return item 389 | 390 | @wrap_with(threading.RLock()) 391 | def _create_dir(self, parent_id, title, remote_path): 392 | cached = self._ids_cache["dirs"].get(remote_path) 393 | if cached: 394 | return cached[0] 395 | 396 | item = self._gdrive_create_dir(parent_id, title) 397 | self._cache_path_id(remote_path, item["id"]) 398 | 399 | return item["id"] 400 | 401 | def exists(self, path): 402 | try: 403 | self._get_item_id(path) 404 | except FileNotFoundError: 405 | return False 406 | else: 407 | return True 408 | 409 | @_gdrive_retry 410 | def info(self, path): 411 | bucket, base = self.split_path(path) 412 | item_id = self._get_item_id(path) 413 | gdrive_file = self.client.CreateFile({"id": item_id}) 414 | gdrive_file.FetchMetadata() 415 | 416 | metadata = {"name": posixpath.join(bucket, base.rstrip("/"))} 417 | if gdrive_file["mimeType"] == FOLDER_MIME_TYPE: 418 | metadata["type"] = "directory" 419 | metadata["size"] = 0 420 | metadata["name"] += "/" 421 | else: 422 | metadata["type"] = "file" 423 | metadata["size"] = int(gdrive_file.get("fileSize")) 424 | metadata["checksum"] = gdrive_file.get("md5Checksum") 425 | return metadata 426 | 427 | def ls(self, path, detail=False): 428 | bucket, base = self.split_path(path) 429 | assert bucket == self.root 430 | 431 | dir_ids = self._path_to_item_ids(base) 432 | if not dir_ids: 433 | raise FileNotFoundError( 434 | errno.ENOENT, os.strerror(errno.ENOENT), path 435 | ) 436 | 437 | root_path = posixpath.join(bucket, base) 438 | contents = [] 439 | for item in self._gdrive_list_ids(dir_ids): 440 | item_path = posixpath.join(root_path, item["title"]) 441 | if item["mimeType"] == FOLDER_MIME_TYPE: 442 | contents.append( 443 | { 444 | "type": "directory", 445 | "name": item_path.rstrip("/") + "/", 446 | "size": 0, 447 | } 448 | ) 449 | else: 450 | size = item.get("fileSize") 451 | contents.append( 452 | { 453 | "type": "file", 454 | "name": item_path, 455 | "size": int(size) if size is not None else size, 456 | "checksum": item.get("md5Checksum"), 457 | } 458 | ) 459 | 460 | if detail: 461 | return contents 462 | else: 463 | return [content["name"] for content in contents] 464 | 465 | def find(self, path, detail=False, **kwargs): 466 | bucket, base = self.split_path(path) 467 | assert bucket == self.root 468 | 469 | # Make sure the base path is cached and dir_ids below has some 470 | # dirs revelant to this call 471 | self._path_to_item_ids(base) 472 | 473 | dir_ids = [self._ids_cache["ids"].copy()] 474 | seen_paths = set() 475 | contents = [] 476 | while dir_ids: 477 | query_ids = { 478 | dir_id: dir_name 479 | for dir_id, dir_name in dir_ids.pop().items() 480 | if posixpath.commonpath([base, dir_name]) == base 481 | if dir_id not in seen_paths 482 | } 483 | if not query_ids: 484 | continue 485 | 486 | seen_paths |= query_ids.keys() 487 | 488 | new_query_ids = {} 489 | dir_ids.append(new_query_ids) 490 | for item in self._gdrive_list_ids(query_ids): 491 | parent_id = item["parents"][0]["id"] 492 | item_path = posixpath.join(query_ids[parent_id], item["title"]) 493 | if item["mimeType"] == FOLDER_MIME_TYPE: 494 | new_query_ids[item["id"]] = item_path 495 | self._cache_path_id(item_path, item["id"]) 496 | continue 497 | size = item.get("fileSize") 498 | contents.append( 499 | { 500 | "name": posixpath.join(bucket, item_path), 501 | "type": "file", 502 | "size": int(size) if size is not None else size, 503 | "checksum": item.get("md5Checksum"), 504 | } 505 | ) 506 | 507 | if detail: 508 | return {content["name"]: content for content in contents} 509 | else: 510 | return [content["name"] for content in contents] 511 | 512 | def upload_fobj(self, stream, rpath, callback=None, **kwargs): 513 | parent_id = self._get_item_id(self._parent(rpath), create=True) 514 | if callback: 515 | stream = CallbackIOWrapper( 516 | callback.relative_update, stream, "read" 517 | ) 518 | return self._gdrive_upload_fobj( 519 | posixpath.basename(rpath.rstrip("/")), parent_id, stream 520 | ) 521 | 522 | def put_file(self, lpath, rpath, callback=None, **kwargs): 523 | if callback: 524 | callback.set_size(os.path.getsize(lpath)) 525 | with open(lpath, "rb") as stream: 526 | self.upload_fobj(stream, rpath, callback=callback) 527 | 528 | @_gdrive_retry 529 | def _gdrive_upload_fobj(self, title, parent_id, stream, callback=None): 530 | item = self.client.CreateFile( 531 | {"title": title, "parents": [{"id": parent_id}]} 532 | ) 533 | item.content = stream 534 | item.Upload() 535 | return item 536 | 537 | def cp_file(self, lpath, rpath, **kwargs): 538 | """In-memory streamed copy""" 539 | with self.open(lpath) as stream: 540 | # IterStream objects doesn't support full-length 541 | # seek() calls, so we have to wrap the data with 542 | # an external buffer. 543 | buffer = io.BytesIO(stream.read()) 544 | self.upload_fobj(buffer, rpath) 545 | 546 | @_gdrive_retry 547 | def mv(self, path1, path2, maxdepth=None, **kwargs): 548 | if maxdepth is not None: 549 | raise NotImplementedError("Max depth move is not supported") 550 | 551 | src_name = posixpath.basename(path1) 552 | 553 | src_parent = self._parent(path1) 554 | 555 | if self.exists(path2): 556 | dst_name = src_name 557 | dst_parent = path2 558 | else: 559 | dst_name = posixpath.basename(path2) 560 | dst_parent = self._parent(path2) 561 | 562 | file1_id = self._get_item_id(path1) 563 | 564 | file1 = self.client.CreateFile({"id": file1_id}) 565 | 566 | if src_name != dst_name: 567 | file1["title"] = dst_name 568 | 569 | if src_parent != dst_parent: 570 | file2_parent_id = self._get_item_id(dst_parent) 571 | file1["parents"] = [{"id": file2_parent_id}] 572 | 573 | # TODO need to invalidate the cache for the old path, see #232 574 | file1.Upload() 575 | 576 | def get_file(self, rpath, lpath, callback=None, block_size=None, **kwargs): 577 | item_id = self._get_item_id(rpath) 578 | return self._gdrive_get_file( 579 | item_id, lpath, callback=callback, block_size=block_size 580 | ) 581 | 582 | @_gdrive_retry 583 | def _gdrive_get_file(self, item_id, rpath, callback=None, block_size=None): 584 | param = {"id": item_id} 585 | # it does not create a file on the remote 586 | gdrive_file = self.client.CreateFile(param) 587 | 588 | extra_args = {"acknowledge_abuse": self._acknowledge_abuse} 589 | if block_size: 590 | extra_args["chunksize"] = block_size 591 | 592 | if callback: 593 | 594 | def cb(value, _): 595 | callback.absolute_update(value) 596 | 597 | gdrive_file.FetchMetadata(fields="fileSize") 598 | callback.set_size(int(gdrive_file.get("fileSize"))) 599 | extra_args["callback"] = cb 600 | 601 | gdrive_file.GetContentFile(rpath, **extra_args) 602 | 603 | def _open(self, path, mode, **kwargs): 604 | assert mode in {"rb", "wb"} 605 | if mode == "wb": 606 | return GDriveBufferedWriter(self, path) 607 | else: 608 | item_id = self._get_item_id(path) 609 | return self._gdrive_open_file(item_id) 610 | 611 | @_gdrive_retry 612 | def _gdrive_open_file(self, item_id): 613 | param = {"id": item_id} 614 | # it does not create a file on the remote 615 | gdrive_file = self.client.CreateFile(param) 616 | fd = gdrive_file.GetContentIOBuffer( 617 | acknowledge_abuse=self._acknowledge_abuse 618 | ) 619 | return IterStream(iter(fd)) 620 | 621 | def rm_file(self, path): 622 | item_id = self._get_item_id(path) 623 | self._gdrive_delete_file(item_id) 624 | 625 | @_gdrive_retry 626 | def _gdrive_delete_file(self, item_id): 627 | from pydrive2.files import ApiRequestError 628 | 629 | param = {"id": item_id} 630 | # it does not create a file on the remote 631 | item = self.client.CreateFile(param) 632 | 633 | try: 634 | item.Trash() if self._trash_only else item.Delete() 635 | except ApiRequestError as exc: 636 | http_error_code = exc.error.get("code", 0) 637 | if ( 638 | http_error_code == 403 639 | and self._list_params["corpora"] == "drive" 640 | and exc.GetField("location") == "file.permissions" 641 | ): 642 | raise PermissionError( 643 | "Insufficient permissions to {}. You should have {} " 644 | "access level for the used shared drive. More details " 645 | "at {}.".format( 646 | "move the file into Trash" 647 | if self._trash_only 648 | else "permanently delete the file", 649 | "Manager or Content Manager" 650 | if self._trash_only 651 | else "Manager", 652 | "https://support.google.com/a/answer/7337554", 653 | ) 654 | ) from exc 655 | raise 656 | 657 | 658 | class GDriveBufferedWriter(io.IOBase): 659 | def __init__(self, fs, path): 660 | self.fs = fs 661 | self.path = path 662 | self.buffer = io.BytesIO() 663 | self._closed = False 664 | 665 | def write(self, *args, **kwargs): 666 | self.buffer.write(*args, **kwargs) 667 | 668 | def readable(self): 669 | return False 670 | 671 | def writable(self): 672 | return not self.readable() 673 | 674 | def flush(self): 675 | self.buffer.flush() 676 | try: 677 | self.fs.upload_fobj(self.buffer, self.path) 678 | finally: 679 | self._closed = True 680 | 681 | def close(self): 682 | if self._closed: 683 | return None 684 | 685 | self.flush() 686 | self.buffer.close() 687 | self._closed = True 688 | 689 | def __enter__(self): 690 | return self 691 | 692 | def __exit__(self, *exc_info): 693 | self.close() 694 | 695 | @property 696 | def closed(self): 697 | return self._closed 698 | -------------------------------------------------------------------------------- /pydrive2/auth.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import webbrowser 4 | import httplib2 5 | import oauth2client.clientsecrets as clientsecrets 6 | import threading 7 | 8 | from googleapiclient.discovery import build 9 | from functools import wraps 10 | from oauth2client.service_account import ServiceAccountCredentials 11 | from oauth2client.client import FlowExchangeError 12 | from oauth2client.client import AccessTokenRefreshError 13 | from oauth2client.client import OAuth2WebServerFlow 14 | from oauth2client.client import OOB_CALLBACK_URN 15 | from oauth2client.contrib.dictionary_storage import DictionaryStorage 16 | from oauth2client.file import Storage 17 | from oauth2client.tools import ClientRedirectHandler 18 | from oauth2client.tools import ClientRedirectServer 19 | from oauth2client._helpers import scopes_to_string 20 | from .apiattr import ApiAttribute 21 | from .apiattr import ApiAttributeMixin 22 | from .settings import LoadSettingsFile 23 | from .settings import ValidateSettings 24 | from .settings import SettingsError 25 | from .settings import InvalidConfigError 26 | 27 | 28 | class AuthError(Exception): 29 | """Base error for authentication/authorization errors.""" 30 | 31 | 32 | class InvalidCredentialsError(IOError): 33 | """Error trying to read credentials file.""" 34 | 35 | 36 | class AuthenticationRejected(AuthError): 37 | """User rejected authentication.""" 38 | 39 | 40 | class AuthenticationError(AuthError): 41 | """General authentication error.""" 42 | 43 | 44 | class RefreshError(AuthError): 45 | """Access token refresh error.""" 46 | 47 | 48 | def LoadAuth(decoratee): 49 | """Decorator to check if the auth is valid and loads auth if not.""" 50 | 51 | @wraps(decoratee) 52 | def _decorated(self, *args, **kwargs): 53 | # Initialize auth if needed. 54 | if self.auth is None: 55 | self.auth = GoogleAuth() 56 | # Re-create access token if it expired. 57 | if self.auth.access_token_expired: 58 | if getattr(self.auth, "auth_method", False) == "service": 59 | self.auth.ServiceAuth() 60 | else: 61 | self.auth.LocalWebserverAuth() 62 | 63 | # Initialise service if not built yet. 64 | if self.auth.service is None: 65 | self.auth.Authorize() 66 | 67 | # Ensure that a thread-safe HTTP object is provided. 68 | if ( 69 | kwargs is not None 70 | and "param" in kwargs 71 | and kwargs["param"] is not None 72 | and "http" in kwargs["param"] 73 | and kwargs["param"]["http"] is not None 74 | ): 75 | self.http = kwargs["param"]["http"] 76 | del kwargs["param"]["http"] 77 | 78 | else: 79 | # If HTTP object not specified, create or resuse an HTTP 80 | # object from the thread local storage. 81 | if not getattr(self.auth.thread_local, "http", None): 82 | self.auth.thread_local.http = self.auth.Get_Http_Object() 83 | self.http = self.auth.thread_local.http 84 | 85 | return decoratee(self, *args, **kwargs) 86 | 87 | return _decorated 88 | 89 | 90 | def CheckServiceAuth(decoratee): 91 | """Decorator to authorize service account.""" 92 | 93 | @wraps(decoratee) 94 | def _decorated(self, *args, **kwargs): 95 | self.auth_method = "service" 96 | dirty = False 97 | save_credentials = self.settings.get("save_credentials") 98 | if self.credentials is None and save_credentials: 99 | self.LoadCredentials() 100 | if self.credentials is None: 101 | decoratee(self, *args, **kwargs) 102 | self.Authorize() 103 | dirty = True 104 | elif self.access_token_expired: 105 | self.Refresh() 106 | dirty = True 107 | self.credentials.set_store(self._default_storage) 108 | if dirty and save_credentials: 109 | self.SaveCredentials() 110 | 111 | return _decorated 112 | 113 | 114 | def CheckAuth(decoratee): 115 | """Decorator to check if it requires OAuth2 flow request.""" 116 | 117 | @wraps(decoratee) 118 | def _decorated(self, *args, **kwargs): 119 | dirty = False 120 | code = None 121 | save_credentials = self.settings.get("save_credentials") 122 | if self.credentials is None and save_credentials: 123 | self.LoadCredentials() 124 | if self.flow is None: 125 | self.GetFlow() 126 | if self.credentials is None: 127 | code = decoratee(self, *args, **kwargs) 128 | dirty = True 129 | else: 130 | if self.access_token_expired: 131 | if self.credentials.refresh_token is not None: 132 | self.Refresh() 133 | else: 134 | code = decoratee(self, *args, **kwargs) 135 | dirty = True 136 | if code is not None: 137 | self.Auth(code) 138 | self.credentials.set_store(self._default_storage) 139 | if dirty and save_credentials: 140 | self.SaveCredentials() 141 | 142 | return _decorated 143 | 144 | 145 | class GoogleAuth(ApiAttributeMixin): 146 | """Wrapper class for oauth2client library in google-api-python-client. 147 | 148 | Loads all settings and credentials from one 'settings.yaml' file 149 | and performs common OAuth2.0 related functionality such as authentication 150 | and authorization. 151 | """ 152 | 153 | DEFAULT_SETTINGS = { 154 | "client_config_backend": "file", 155 | "client_config_file": "client_secrets.json", 156 | "save_credentials": False, 157 | "oauth_scope": ["https://www.googleapis.com/auth/drive"], 158 | } 159 | CLIENT_CONFIGS_LIST = [ 160 | "client_id", 161 | "client_secret", 162 | "auth_uri", 163 | "token_uri", 164 | "revoke_uri", 165 | "redirect_uri", 166 | ] 167 | SERVICE_CONFIGS_LIST = ["client_user_email"] 168 | settings = ApiAttribute("settings") 169 | client_config = ApiAttribute("client_config") 170 | flow = ApiAttribute("flow") 171 | credentials = ApiAttribute("credentials") 172 | http = ApiAttribute("http") 173 | service = ApiAttribute("service") 174 | auth_method = ApiAttribute("auth_method") 175 | 176 | def __init__( 177 | self, settings_file="settings.yaml", http_timeout=None, settings=None 178 | ): 179 | """Create an instance of GoogleAuth. 180 | 181 | :param settings_file: path of settings file. 'settings.yaml' by default. 182 | :type settings_file: str. 183 | :param settings: settings dict. 184 | :type settings: dict. 185 | """ 186 | self.http_timeout = http_timeout 187 | ApiAttributeMixin.__init__(self) 188 | self.thread_local = threading.local() 189 | self.client_config = {} 190 | 191 | if settings is None and settings_file: 192 | try: 193 | settings = LoadSettingsFile(settings_file) 194 | except SettingsError: 195 | pass 196 | 197 | self.settings = settings or self.DEFAULT_SETTINGS 198 | ValidateSettings(self.settings) 199 | 200 | storages, default = self._InitializeStoragesFromSettings() 201 | self._storages = storages 202 | self._default_storage = default 203 | 204 | @property 205 | def access_token_expired(self): 206 | """Checks if access token doesn't exist or is expired. 207 | 208 | :returns: bool -- True if access token doesn't exist or is expired. 209 | """ 210 | if self.credentials is None: 211 | return True 212 | return self.credentials.access_token_expired 213 | 214 | @CheckAuth 215 | def LocalWebserverAuth( 216 | self, 217 | host_name="localhost", 218 | port_numbers=None, 219 | launch_browser=True, 220 | bind_addr=None, 221 | ): 222 | """Authenticate and authorize from user by creating local web server and 223 | retrieving authentication code. 224 | 225 | This function is not for web server application. It creates local web 226 | server for user from standalone application. 227 | 228 | If GDRIVE_NON_INTERACTIVE environment variable is set, this function 229 | raises AuthenticationError. 230 | 231 | :param host_name: host name of the local web server. 232 | :type host_name: str. 233 | :param port_numbers: list of port numbers to be tried to used. 234 | :type port_numbers: list. 235 | :param launch_browser: should browser be launched automatically 236 | :type launch_browser: bool 237 | :param bind_addr: optional IP address for the local web server to listen on. 238 | If not specified, it will listen on the address specified in the 239 | host_name parameter. 240 | :type bind_addr: str. 241 | :returns: str -- code returned from local web server 242 | :raises: AuthenticationRejected, AuthenticationError 243 | """ 244 | if os.getenv("GDRIVE_NON_INTERACTIVE"): 245 | raise AuthenticationError( 246 | "Non interactive mode (GDRIVE_NON_INTERACTIVE env) is enabled" 247 | ) 248 | 249 | if port_numbers is None: 250 | port_numbers = [ 251 | 8080, 252 | 8090, 253 | ] # Mutable objects should not be default 254 | # values, as each call's changes are global. 255 | success = False 256 | port_number = 0 257 | for port in port_numbers: 258 | port_number = port 259 | try: 260 | httpd = ClientRedirectServer( 261 | (bind_addr or host_name, port), ClientRedirectHandler 262 | ) 263 | except OSError: 264 | pass 265 | else: 266 | success = True 267 | break 268 | if success: 269 | oauth_callback = f"http://{host_name}:{port_number}/" 270 | else: 271 | print( 272 | "Failed to start a local web server. Please check your firewall" 273 | ) 274 | print( 275 | "settings and locally running programs that may be blocking or" 276 | ) 277 | print("using configured ports. Default ports are 8080 and 8090.") 278 | raise AuthenticationError() 279 | self.flow.redirect_uri = oauth_callback 280 | authorize_url = self.GetAuthUrl() 281 | if launch_browser: 282 | webbrowser.open(authorize_url, new=1, autoraise=True) 283 | print("Your browser has been opened to visit:") 284 | else: 285 | print("Open your browser to visit:") 286 | print() 287 | print(" " + authorize_url) 288 | print() 289 | httpd.handle_request() 290 | if "error" in httpd.query_params: 291 | print("Authentication request was rejected") 292 | raise AuthenticationRejected("User rejected authentication") 293 | if "code" in httpd.query_params: 294 | return httpd.query_params["code"] 295 | else: 296 | print( 297 | 'Failed to find "code" in the query parameters of the redirect.' 298 | ) 299 | print("Try command-line authentication") 300 | raise AuthenticationError("No code found in redirect") 301 | 302 | @CheckAuth 303 | def CommandLineAuth(self): 304 | """Authenticate and authorize from user by printing authentication url 305 | retrieving authentication code from command-line. 306 | 307 | :returns: str -- code returned from commandline. 308 | """ 309 | self.flow.redirect_uri = OOB_CALLBACK_URN 310 | authorize_url = self.GetAuthUrl() 311 | print("Go to the following link in your browser:") 312 | print() 313 | print(" " + authorize_url) 314 | print() 315 | return input("Enter verification code: ").strip() 316 | 317 | @CheckServiceAuth 318 | def ServiceAuth(self): 319 | """Authenticate and authorize using P12 private key, client id 320 | and client email for a Service account. 321 | :raises: AuthError, InvalidConfigError 322 | """ 323 | if set(self.SERVICE_CONFIGS_LIST) - set(self.client_config): 324 | self.LoadServiceConfigSettings() 325 | scopes = scopes_to_string(self.settings["oauth_scope"]) 326 | keyfile_name = self.client_config.get("client_json_file_path") 327 | keyfile_dict = self.client_config.get("client_json_dict") 328 | keyfile_json = self.client_config.get("client_json") 329 | 330 | if not keyfile_dict and keyfile_json: 331 | # Compensating for missing ServiceAccountCredentials.from_json_keyfile 332 | keyfile_dict = json.loads(keyfile_json) 333 | 334 | if keyfile_dict: 335 | self.credentials = ( 336 | ServiceAccountCredentials.from_json_keyfile_dict( 337 | keyfile_dict=keyfile_dict, scopes=scopes 338 | ) 339 | ) 340 | elif keyfile_name: 341 | self.credentials = ( 342 | ServiceAccountCredentials.from_json_keyfile_name( 343 | filename=keyfile_name, scopes=scopes 344 | ) 345 | ) 346 | else: 347 | service_email = self.client_config["client_service_email"] 348 | file_path = self.client_config["client_pkcs12_file_path"] 349 | self.credentials = ServiceAccountCredentials.from_p12_keyfile( 350 | service_account_email=service_email, 351 | filename=file_path, 352 | scopes=scopes, 353 | ) 354 | 355 | user_email = self.client_config.get("client_user_email") 356 | if user_email: 357 | self.credentials = self.credentials.create_delegated( 358 | sub=user_email 359 | ) 360 | 361 | def _InitializeStoragesFromSettings(self): 362 | result = {"file": None, "dictionary": None} 363 | backend = self.settings.get("save_credentials_backend") 364 | save_credentials = self.settings.get("save_credentials") 365 | if backend == "file": 366 | credentials_file = self.settings.get("save_credentials_file") 367 | if credentials_file is None: 368 | raise InvalidConfigError( 369 | "Please specify credentials file to read" 370 | ) 371 | result[backend] = Storage(credentials_file) 372 | elif backend == "dictionary": 373 | creds_dict = self.settings.get("save_credentials_dict") 374 | if creds_dict is None: 375 | raise InvalidConfigError("Please specify credentials dict") 376 | 377 | creds_key = self.settings.get("save_credentials_key") 378 | if creds_key is None: 379 | raise InvalidConfigError("Please specify credentials key") 380 | 381 | result[backend] = DictionaryStorage(creds_dict, creds_key) 382 | elif save_credentials: 383 | raise InvalidConfigError( 384 | "Unknown save_credentials_backend: %s" % backend 385 | ) 386 | return result, result.get(backend) 387 | 388 | def LoadCredentials(self, backend=None): 389 | """Loads credentials or create empty credentials if it doesn't exist. 390 | 391 | :param backend: target backend to save credential to. 392 | :type backend: str. 393 | :raises: InvalidConfigError 394 | """ 395 | if backend is None: 396 | backend = self.settings.get("save_credentials_backend") 397 | if backend is None: 398 | raise InvalidConfigError("Please specify credential backend") 399 | if backend == "file": 400 | self.LoadCredentialsFile() 401 | elif backend == "dictionary": 402 | self._LoadCredentialsDictionary() 403 | else: 404 | raise InvalidConfigError("Unknown save_credentials_backend") 405 | 406 | def LoadCredentialsFile(self, credentials_file=None): 407 | """Loads credentials or create empty credentials if it doesn't exist. 408 | 409 | Loads credentials file from path in settings if not specified. 410 | 411 | :param credentials_file: path of credentials file to read. 412 | :type credentials_file: str. 413 | :raises: InvalidConfigError, InvalidCredentialsError 414 | """ 415 | if credentials_file is None: 416 | self._default_storage = self._storages["file"] 417 | if self._default_storage is None: 418 | raise InvalidConfigError( 419 | "Backend `file` is not configured, specify " 420 | "credentials file to read in the settings " 421 | "file or pass an explicit value" 422 | ) 423 | else: 424 | self._default_storage = Storage(credentials_file) 425 | 426 | try: 427 | self.credentials = self._default_storage.get() 428 | except OSError: 429 | raise InvalidCredentialsError( 430 | "Credentials file cannot be symbolic link" 431 | ) 432 | 433 | if self.credentials: 434 | self.credentials.set_store(self._default_storage) 435 | 436 | def _LoadCredentialsDictionary(self): 437 | self._default_storage = self._storages["dictionary"] 438 | if self._default_storage is None: 439 | raise InvalidConfigError( 440 | "Backend `dictionary` is not configured, specify " 441 | "credentials dict and key to read in the settings file" 442 | ) 443 | 444 | self.credentials = self._default_storage.get() 445 | 446 | if self.credentials: 447 | self.credentials.set_store(self._default_storage) 448 | 449 | def SaveCredentials(self, backend=None): 450 | """Saves credentials according to specified backend. 451 | 452 | If you have any specific credentials backend in mind, don't use this 453 | function and use the corresponding function you want. 454 | 455 | :param backend: backend to save credentials. 456 | :type backend: str. 457 | :raises: InvalidConfigError 458 | """ 459 | if backend is None: 460 | backend = self.settings.get("save_credentials_backend") 461 | if backend is None: 462 | raise InvalidConfigError("Please specify credential backend") 463 | if backend == "file": 464 | self.SaveCredentialsFile() 465 | elif backend == "dictionary": 466 | self._SaveCredentialsDictionary() 467 | else: 468 | raise InvalidConfigError("Unknown save_credentials_backend") 469 | 470 | def SaveCredentialsFile(self, credentials_file=None): 471 | """Saves credentials to the file in JSON format. 472 | 473 | :param credentials_file: destination to save file to. 474 | :type credentials_file: str. 475 | :raises: InvalidConfigError, InvalidCredentialsError 476 | """ 477 | if self.credentials is None: 478 | raise InvalidCredentialsError("No credentials to save") 479 | 480 | if credentials_file is None: 481 | storage = self._storages["file"] 482 | if storage is None: 483 | raise InvalidConfigError( 484 | "Backend `file` is not configured, specify " 485 | "credentials file to read in the settings " 486 | "file or pass an explicit value" 487 | ) 488 | else: 489 | storage = Storage(credentials_file) 490 | 491 | try: 492 | storage.put(self.credentials) 493 | except OSError: 494 | raise InvalidCredentialsError( 495 | "Credentials file cannot be symbolic link" 496 | ) 497 | 498 | def _SaveCredentialsDictionary(self): 499 | if self.credentials is None: 500 | raise InvalidCredentialsError("No credentials to save") 501 | 502 | storage = self._storages["dictionary"] 503 | if storage is None: 504 | raise InvalidConfigError( 505 | "Backend `dictionary` is not configured, specify " 506 | "credentials dict and key to write in the settings file" 507 | ) 508 | 509 | storage.put(self.credentials) 510 | 511 | def LoadClientConfig(self, backend=None): 512 | """Loads client configuration according to specified backend. 513 | 514 | If you have any specific backend to load client configuration from in mind, 515 | don't use this function and use the corresponding function you want. 516 | 517 | :param backend: backend to load client configuration from. 518 | :type backend: str. 519 | :raises: InvalidConfigError 520 | """ 521 | if backend is None: 522 | backend = self.settings.get("client_config_backend") 523 | if backend is None: 524 | raise InvalidConfigError( 525 | "Please specify client config backend" 526 | ) 527 | if backend == "file": 528 | self.LoadClientConfigFile() 529 | elif backend == "settings": 530 | self.LoadClientConfigSettings() 531 | elif backend == "service": 532 | self.LoadServiceConfigSettings() 533 | else: 534 | raise InvalidConfigError("Unknown client_config_backend") 535 | 536 | def LoadClientConfigFile(self, client_config_file=None): 537 | """Loads client configuration file downloaded from APIs console. 538 | 539 | Loads client config file from path in settings if not specified. 540 | 541 | :param client_config_file: path of client config file to read. 542 | :type client_config_file: str. 543 | :raises: InvalidConfigError 544 | """ 545 | if client_config_file is None: 546 | client_config_file = self.settings["client_config_file"] 547 | try: 548 | client_type, client_info = clientsecrets.loadfile( 549 | client_config_file 550 | ) 551 | except clientsecrets.InvalidClientSecretsError as error: 552 | raise InvalidConfigError("Invalid client secrets file %s" % error) 553 | if client_type not in ( 554 | clientsecrets.TYPE_WEB, 555 | clientsecrets.TYPE_INSTALLED, 556 | ): 557 | raise InvalidConfigError( 558 | "Unknown client_type of client config file" 559 | ) 560 | 561 | # General settings. 562 | try: 563 | config_index = [ 564 | "client_id", 565 | "client_secret", 566 | "auth_uri", 567 | "token_uri", 568 | ] 569 | for config in config_index: 570 | self.client_config[config] = client_info[config] 571 | 572 | self.client_config["revoke_uri"] = client_info.get("revoke_uri") 573 | self.client_config["redirect_uri"] = client_info["redirect_uris"][ 574 | 0 575 | ] 576 | except KeyError: 577 | raise InvalidConfigError("Insufficient client config in file") 578 | 579 | # Service auth related fields. 580 | service_auth_config = ["client_email"] 581 | try: 582 | for config in service_auth_config: 583 | self.client_config[config] = client_info[config] 584 | except KeyError: 585 | pass # The service auth fields are not present, handling code can go here. 586 | 587 | def LoadServiceConfigSettings(self): 588 | """Loads client configuration from settings. 589 | :raises: InvalidConfigError 590 | """ 591 | configs = [ 592 | "client_json_file_path", 593 | "client_json_dict", 594 | "client_json", 595 | "client_pkcs12_file_path", 596 | ] 597 | 598 | for config in configs: 599 | value = self.settings["service_config"].get(config) 600 | if value: 601 | self.client_config[config] = value 602 | break 603 | else: 604 | raise InvalidConfigError( 605 | f"One of {configs} is required for service authentication" 606 | ) 607 | 608 | if config == "client_pkcs12_file_path": 609 | self.SERVICE_CONFIGS_LIST.append("client_service_email") 610 | 611 | for config in self.SERVICE_CONFIGS_LIST: 612 | try: 613 | self.client_config[config] = self.settings["service_config"][ 614 | config 615 | ] 616 | except KeyError: 617 | err = "Insufficient service config in settings" 618 | err += f"\n\nMissing: {config} key." 619 | raise InvalidConfigError(err) 620 | 621 | def LoadClientConfigSettings(self): 622 | """Loads client configuration from settings file. 623 | 624 | :raises: InvalidConfigError 625 | """ 626 | for config in self.CLIENT_CONFIGS_LIST: 627 | try: 628 | self.client_config[config] = self.settings["client_config"][ 629 | config 630 | ] 631 | except KeyError: 632 | raise InvalidConfigError( 633 | "Insufficient client config in settings" 634 | ) 635 | 636 | def GetFlow(self): 637 | """Gets Flow object from client configuration. 638 | 639 | :raises: InvalidConfigError 640 | """ 641 | if not all( 642 | config in self.client_config for config in self.CLIENT_CONFIGS_LIST 643 | ): 644 | self.LoadClientConfig() 645 | constructor_kwargs = { 646 | "redirect_uri": self.client_config["redirect_uri"], 647 | "auth_uri": self.client_config["auth_uri"], 648 | "token_uri": self.client_config["token_uri"], 649 | "access_type": "online", 650 | } 651 | if self.client_config["revoke_uri"] is not None: 652 | constructor_kwargs["revoke_uri"] = self.client_config["revoke_uri"] 653 | self.flow = OAuth2WebServerFlow( 654 | self.client_config["client_id"], 655 | self.client_config["client_secret"], 656 | scopes_to_string(self.settings["oauth_scope"]), 657 | **constructor_kwargs, 658 | ) 659 | if self.settings.get("get_refresh_token"): 660 | self.flow.params.update( 661 | {"access_type": "offline", "approval_prompt": "force"} 662 | ) 663 | 664 | def Refresh(self): 665 | """Refreshes the access_token. 666 | 667 | :raises: RefreshError 668 | """ 669 | if self.credentials is None: 670 | raise RefreshError("No credential to refresh.") 671 | if ( 672 | self.credentials.refresh_token is None 673 | and self.auth_method != "service" 674 | ): 675 | raise RefreshError( 676 | "No refresh_token found." 677 | "Please set access_type of OAuth to offline." 678 | ) 679 | if self.http is None: 680 | self.http = self._build_http() 681 | try: 682 | self.credentials.refresh(self.http) 683 | except AccessTokenRefreshError as error: 684 | raise RefreshError("Access token refresh failed: %s" % error) 685 | 686 | def GetAuthUrl(self): 687 | """Creates authentication url where user visits to grant access. 688 | 689 | :returns: str -- Authentication url. 690 | """ 691 | if self.flow is None: 692 | self.GetFlow() 693 | return self.flow.step1_get_authorize_url() 694 | 695 | def Auth(self, code): 696 | """Authenticate, authorize, and build service. 697 | 698 | :param code: Code for authentication. 699 | :type code: str. 700 | :raises: AuthenticationError 701 | """ 702 | self.Authenticate(code) 703 | self.Authorize() 704 | 705 | def Authenticate(self, code): 706 | """Authenticates given authentication code back from user. 707 | 708 | :param code: Code for authentication. 709 | :type code: str. 710 | :raises: AuthenticationError 711 | """ 712 | if self.flow is None: 713 | self.GetFlow() 714 | try: 715 | self.credentials = self.flow.step2_exchange(code) 716 | except FlowExchangeError as e: 717 | raise AuthenticationError("OAuth2 code exchange failed: %s" % e) 718 | print("Authentication successful.") 719 | 720 | def _build_http(self): 721 | http = httplib2.Http(timeout=self.http_timeout) 722 | # 308's are used by several Google APIs (Drive, YouTube) 723 | # for Resumable Uploads rather than Permanent Redirects. 724 | # This asks httplib2 to exclude 308s from the status codes 725 | # it treats as redirects 726 | # See also: https://stackoverflow.com/a/59850170/298182 727 | try: 728 | http.redirect_codes = http.redirect_codes - {308} 729 | except AttributeError: 730 | # http.redirect_codes does not exist in previous versions 731 | # of httplib2, so pass 732 | pass 733 | return http 734 | 735 | def Authorize(self): 736 | """Authorizes and builds service. 737 | 738 | :raises: AuthenticationError 739 | """ 740 | if self.access_token_expired: 741 | raise AuthenticationError( 742 | "No valid credentials provided to authorize" 743 | ) 744 | 745 | if self.http is None: 746 | self.http = self._build_http() 747 | self.http = self.credentials.authorize(self.http) 748 | self.service = build( 749 | "drive", "v2", http=self.http, cache_discovery=False 750 | ) 751 | 752 | def Get_Http_Object(self): 753 | """Create and authorize an httplib2.Http object. Necessary for 754 | thread-safety. 755 | :return: The http object to be used in each call. 756 | :rtype: httplib2.Http 757 | """ 758 | http = self._build_http() 759 | http = self.credentials.authorize(http) 760 | return http 761 | --------------------------------------------------------------------------------