├── LICENSE ├── README.md ├── .gitignore ├── src └── apple_app_reviews_scraper.py └── notebooks └── scrape_app_store_reivews.ipynb /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Glenn Fang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![Python Version](https://img.shields.io/badge/Python-3.10-brightgreen) ![License](https://img.shields.io/pypi/l/app-store-scraper) 2 | 3 | # Apple App Store Reviews Scraper 4 | 5 | 6 | 7 | Apple App Store reviews scraper. Reviews are fetched using Apple's API, as the webpage of each app only displays a few reviews. 8 | 9 | It is adapted from 10 | [app-store-scraper](https://github.com/cowboy-bebug/app-store-scraper). I converted the classes to functions for ease of debugging and removed some redundant elements such as the date filter as Apple does not allow reviews to be sorted by date. Generally, the larger the `offset`, the older the reviews tend to be. 11 | 12 | This port is motivated by the rate limiting that is quickly encountered when using the original package. To address this, I added a default delay and backoff strategy. With the current configuration, it has been tested to scrape ~15,000 reviews with no rate limiting. 13 | 14 | Includes two simple functions `get_token` and `fetch_reviews`, which are used to retrieve an authentication token and fetch app reviews, respectively. 15 | 16 | 17 | ## Usage 18 | 19 | First, set some some `user_agents`. 20 | ```python 21 | user_agents = [ 22 | 'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15', 23 | 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36', 24 | ] 25 | ``` 26 | 27 | ### Obtaining a Bearer Token 28 | 29 | To fetch app reviews from the Apple App Store API, you need to obtain an authentication token. The `get_token` function retrieves this token. 30 | 31 | ```python 32 | from apple_app_reviews_scraper import get_token 33 | 34 | # Provide the necessary parameters 35 | country = 'sg' 36 | app_name = 'your-app-name' # can be named anything, really 37 | app_id = 'your-app-id' 38 | 39 | # Get token 40 | token = get_token(country, app_name, app_id, user_agents) 41 | 42 | print(f"Authentication Token: {token}") 43 | ``` 44 | 45 | ### Fetching App Reviews 46 | 47 | Once you have obtained the authentication token, use the `fetch_reviews` function to fetch app reviews. 48 | 49 | ```python 50 | from apple_app_reviews_scraper import fetch_reviews 51 | import pandas as pd 52 | 53 | country = 'sg' 54 | app_name = 'your-app-name' # can be named anything, really 55 | app_id = 'your-app-id' 56 | 57 | # Call the function 58 | reviews, offset, status_code = fetch_reviews(country, app_name, app_id, user_agents, token) 59 | 60 | # Preview as a DataFrame 61 | df = pd.json_normalize(reviews) 62 | ``` 63 | 64 | JSON structure of a single review: 65 | ``` 66 | {'id': '2801236969', 67 | 'type': 'user-reviews', 68 | 'attributes': {'date': '2022-06-30T08:36:15Z', 69 | 'review': "This is a great app!", 70 | 'rating': 5, 71 | 'isEdited': False, 72 | 'userName': 'updog', 73 | 'title': 'Nice'}, 74 | 'offset': '21', 75 | 'n_batch': 20, 76 | 'app_id': '324684580'} 77 | ``` 78 | If developer responded: 79 | ``` 80 | {'id': '9700279414', 81 | 'type': 'user-reviews', 82 | 'attributes': {'date': '2023-03-11T01:06:51Z', 83 | 'developerResponse': {'id': 35337720, 84 | 'body': "Thanks for your feedback!", 85 | 'modified': '2023-03-12T17:16:37Z'}, 86 | 'review': 'I love this app!', 87 | 'rating': 4, 88 | 'isEdited': False, 89 | 'title': 'Great', 90 | 'userName': 'Nice'}, 91 | 'offset': '21', 92 | 'n_batch': 20, 93 | 'app_id': '913943275'} 94 | ``` 95 | ## Authors 96 | 97 | - [@glennfang](https://www.github.com/glennfang) 98 | 99 | ## Credits 100 | - [@cowboy-bebug](https://www.github.com/cowboy-bebug/app-store-scraper) (original Python implementation) 101 | - [@facundoolano](https://github.com/facundoolano/app-store-scraper) (original Node.js implementation) 102 | 103 | 104 | 105 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | # .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | #Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | #.idea/ 161 | -------------------------------------------------------------------------------- /src/apple_app_reviews_scraper.py: -------------------------------------------------------------------------------- 1 | import random 2 | import requests 3 | import re 4 | import time 5 | from tqdm import tqdm 6 | 7 | def get_token(country:str , app_name:str , app_id: str, user_agents: dict): 8 | 9 | """ 10 | Retrieves the bearer token required for API requests 11 | Regex adapted from base.py of https://github.com/cowboy-bebug/app-store-scraper 12 | """ 13 | 14 | response = requests.get(f'https://apps.apple.com/{country}/app/{app_name}/id{app_id}', 15 | headers = {'User-Agent': random.choice(user_agents)}, 16 | ) 17 | 18 | if response.status_code != 200: 19 | print(f"GET request failed. Response: {response.status_code} {response.reason}") 20 | 21 | tags = response.text.splitlines() 22 | for tag in tags: 23 | if re.match(r"\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
idtypeoffsetn_batchapp_idattributes.dateattributes.developerResponse.idattributes.developerResponse.bodyattributes.developerResponse.modifiedattributes.reviewattributes.ratingattributes.isEditedattributes.titleattributes.userName
09700279414user-reviews21209139432752023-03-11T01:06:51Z35337720.0Thanks for your feedback. We're happy you're e...2023-03-12T17:16:37ZAfter updated to the latest version, TIDAL App...4False[Report] Search bug issueSoufiaj
13871570145user-reviews21209139432752019-03-12T03:27:40ZNaNNaNNaNI was slow coming to online music streaming. I...5FalseTidal is my new best friend.RobPhilp
27680736472user-reviews21209139432752021-08-11T13:45:01ZNaNNaNNaNDespite there were so many controversial debat...5FalseStill the best sound quality Music StreamingMlok99
35340990717user-reviews21209139432752019-12-31T02:45:44ZNaNNaNNaNI have been subscribing to Tidal for three yea...5FalseOutstanding audio qualityMKrishna
49237029356user-reviews21209139432752022-10-30T11:59:17ZNaNNaNNaNThe app is ok on iOS. I much prefer it on the ...3FalseTidal hifi is a great service, the app is okgamov SG
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n"," \n"," "]},"metadata":{},"execution_count":39}]},{"cell_type":"markdown","source":["**Scraping more reviews** \n","\n","You can write a loop to iterate through the different apps. \n","When `offset=None` and the response is `404`, it is likely that there are no more reviews. Note that the number of ratings does not equal the number of reviews, as not every user would have written a review."],"metadata":{"id":"JNRNOiLk7ga2"}},{"cell_type":"code","source":["for app in app_infos:\n"," country, app_name, app_id = [v for v in app_infos[app]]\n"," print(f\"{app:<10} {country:<2} | {app_name:<15} | {app_id:<10}\")\n","\n"," # Scraping loop ------------------------------------------------------------\n"," all_reviews = []\n"," offset = '1'\n"," MAX_REVIEWS = 100+21\n","\n"," start_time = time.time()\n"," while (offset != None) and (int(offset) <= MAX_REVIEWS):\n"," reviews, offset, response_code = fetch_reviews(country=country, \n"," app_name=app_name, \n"," user_agents=user_agents, \n"," app_id=app_id, \n"," token=token, \n"," offset=offset)\n"," print(f\"Current response code: {response_code} | Next offset: {offset}.\")\n"," all_reviews.extend(reviews)\n"," \n"," print(f\"Completed scraping {len(all_reviews)} reviews in {(time.time() - start_time)/60:.2f} minutes.\")\n"," # --------------------------------------------------------------------------\n","\n"," # Check max offset\n"," max_offset = np.nanmax([int(rev['offset']) for rev in all_reviews \n"," if rev['offset'] is not None and rev['n_batch'] == 20])\n"," print(f\"Backed up till offset {max_offset}.\")\n","\n"," # Store all reviews for the app\n"," #ios_reviews[app] = all_reviews\n"," ios_reviews[app].extend(all_reviews)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"rOIQDxvT7Y-u","executionInfo":{"status":"ok","timestamp":1684649505920,"user_tz":-480,"elapsed":7619,"user":{"displayName":"Glenn Fang","userId":"06064883442212685341"}},"outputId":"5b65aada-89c1-40c3-e7cc-d664b004bb7b"},"execution_count":24,"outputs":[{"output_type":"stream","name":"stdout","text":["Tidal sg | tidal-music-hifi-ad-free | 913943275 \n","Offset: 21\n","Current response code: 200 | Next offset: 21.\n","Offset: 41\n","Current response code: 200 | Next offset: 41.\n","Offset: 61\n","Current response code: 200 | Next offset: 61.\n","Offset: 81\n","Current response code: 200 | Next offset: 81.\n","Offset: 101\n","Current response code: 200 | Next offset: 101.\n","14 reviews scraped. This is fewer than the expected 20.\n","No offset found.\n","Current response code: 200 | Next offset: None.\n","Completed scraping 114 reviews in 0.05 minutes.\n","Backed up till offset 101.\n","Spotify sg | spotify-music-and-podcasts | 324684580 \n","Offset: 21\n","Current response code: 200 | Next offset: 21.\n","Offset: 41\n","Current response code: 200 | Next offset: 41.\n","Offset: 61\n","Current response code: 200 | Next offset: 61.\n","Offset: 81\n","Current response code: 200 | Next offset: 81.\n","Offset: 101\n","Current response code: 200 | Next offset: 101.\n","Offset: 121\n","Current response code: 200 | Next offset: 121.\n","Offset: 141\n","Current response code: 200 | Next offset: 141.\n","Completed scraping 140 reviews in 0.07 minutes.\n","Backed up till offset 141.\n"]}]},{"cell_type":"code","source":["ios_df = pd.DataFrame()\n","for key, reviews in ios_reviews.items():\n"," app_df = pd.json_normalize(reviews)\n"," app_df['app'] = key\n"," ios_df = pd.concat([ios_df, app_df], axis=0).reset_index(drop=True)\n","\n","ios_df.columns = [re.sub('^attributes\\.', '', x) for x in ios_df.columns]\n","ios_df.columns = [re.sub('\\.', '_', x).lower() for x in ios_df.columns]\n","ios_df['date'] = pd.to_datetime(ios_df['date'])\n","ios_df.head(3)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":433},"id":"xNjGgYGF7b-U","executionInfo":{"status":"ok","timestamp":1684649534296,"user_tz":-480,"elapsed":313,"user":{"displayName":"Glenn Fang","userId":"06064883442212685341"}},"outputId":"f47b629a-b4e5-483d-bb93-ab6ff5135f4c"},"execution_count":26,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" id type offset n_batch app_id \\\n","0 9700279414 user-reviews 21 20 913943275 \n","1 3871570145 user-reviews 21 20 913943275 \n","2 7680736472 user-reviews 21 20 913943275 \n","\n"," date developerresponse_id \\\n","0 2023-03-11 01:06:51+00:00 35337720.0 \n","1 2019-03-12 03:27:40+00:00 NaN \n","2 2021-08-11 13:45:01+00:00 NaN \n","\n"," developerresponse_body \\\n","0 Thanks for your feedback. We're happy you're e... \n","1 NaN \n","2 NaN \n","\n"," developerresponse_modified \\\n","0 2023-03-12T17:16:37Z \n","1 NaN \n","2 NaN \n","\n"," review rating isedited \\\n","0 After updated to the latest version, TIDAL App... 4 False \n","1 I was slow coming to online music streaming. I... 5 False \n","2 Despite there were so many controversial debat... 5 False \n","\n"," username title app \n","0 Soufiaj [Report] Search bug issue Tidal \n","1 RobPhilp Tidal is my new best friend. Tidal \n","2 Mlok99 Still the best sound quality Music Streaming Tidal "],"text/html":["\n","
\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
idtypeoffsetn_batchapp_iddatedeveloperresponse_iddeveloperresponse_bodydeveloperresponse_modifiedreviewratingiseditedusernametitleapp
09700279414user-reviews21209139432752023-03-11 01:06:51+00:0035337720.0Thanks for your feedback. We're happy you're e...2023-03-12T17:16:37ZAfter updated to the latest version, TIDAL App...4FalseSoufiaj[Report] Search bug issueTidal
13871570145user-reviews21209139432752019-03-12 03:27:40+00:00NaNNaNNaNI was slow coming to online music streaming. I...5FalseRobPhilpTidal is my new best friend.Tidal
27680736472user-reviews21209139432752021-08-11 13:45:01+00:00NaNNaNNaNDespite there were so many controversial debat...5FalseMlok99Still the best sound quality Music StreamingTidal
\n","
\n"," \n"," \n"," \n","\n"," \n","
\n","
\n"," "]},"metadata":{},"execution_count":26}]},{"cell_type":"code","source":["ios_df = ios_df.sort_values(['app', 'date']).reset_index(drop=True)\n","\n","# Week rollup\n","ios_df['week_start'] = ios_df['date'].dt.normalize() - pd.to_timedelta(ios_df['date'].dt.dayofweek, unit='D')\n","\n","# Calculate the moving average for each app\n","ios_df['mavg'] = ios_df.groupby('app')['rating'].rolling(window=7, min_periods=1).mean().reset_index(drop=True)"],"metadata":{"id":"hVyQG1PX80Mv","executionInfo":{"status":"ok","timestamp":1684649811409,"user_tz":-480,"elapsed":306,"user":{"displayName":"Glenn Fang","userId":"06064883442212685341"}}},"execution_count":30,"outputs":[]}]} --------------------------------------------------------------------------------