├── .gitignore ├── msdownloader.code-workspace ├── requirements.txt ├── .vscode └── settings.json ├── securities_example.yaml ├── CHANGELOG.md ├── README.md └── msdownloader.py /.gitignore: -------------------------------------------------------------------------------- 1 | call.rest 2 | securities.yaml -------------------------------------------------------------------------------- /msdownloader.code-workspace: -------------------------------------------------------------------------------- 1 | { 2 | "folders": [ 3 | { 4 | "path": "." 5 | } 6 | ] 7 | } -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests ~= 2.25.1 2 | PyYAML ~= 5.3.1 3 | python-dateutil ~= 2.8.1 4 | pydash ~= 4.0.0 -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "python.pythonPath": "/Users/stefano/.pyenv/versions/3.8.0/bin/python", 3 | "python.linting.enabled": true, 4 | "python.linting.pylintEnabled": false, 5 | "python.linting.flake8Enabled": true, 6 | "python.linting.flake8Args": ["--ignore", "E201,E202,E226,E231,E302,E501"] 7 | } -------------------------------------------------------------------------------- /securities_example.yaml: -------------------------------------------------------------------------------- 1 | funds: 2 | - code: GB0006010168 3 | name: Baillie Gifford Managed Fund B Acc 4 | - code: GB0003295010 5 | universe: FCGBR$$ALL 6 | name: "Baillie Gifford European Growth Trust Plc" 7 | - code: IE00BMC38736 8 | universe: ETEXG$XETR 9 | name: "VanEck Vectors Semiconductor UCITS ETF (EUR)" 10 | shares: 11 | - code: US0258161092 12 | name: American Express AXP 13 | currencies: 14 | - code: EUR 15 | base: GBP 16 | - code: GBP 17 | base: CHF 18 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 5 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 6 | ## [Unreleased] 7 | - custom name for each security (not it's the ISIN) 8 | ## [2.0.0] - 2021-01-18 9 | **BREAKING CHANGES** 10 | 11 | ### Added 12 | - Support for custom universe 13 | - Lookup function for a security 14 | ## [1.3.0] - 2021-01-14 15 | ### Added 16 | - Support for shares 17 | - Support for output to a file 18 | - Support for single call 19 | ## [1.2.0] - 2021-01-10 20 | ### Added 21 | - Support for dumping security data as JSON 22 | ### Chore 23 | - Code refactor 24 | ## [1.1.0] - 2021-01-10 25 | ### Added 26 | - Support for command line arguments 27 | - Conversion from GBX to GBP 28 | ## [1.0.0] - 2021-01-10 29 | ### Added 30 | - Initial release for funds only -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | Morningstar Downloader, aka `msdownloader`, is a small script for downloading securities prices from [Morningstar](https://www.morningstar.com) using official APIs rather than doing site scraping and returning the last price in a [hledger](https://hledger.org) suitable price format. 4 | 5 | ## How it works 6 | 7 | The script first places a call to [Morningstar](https://www.morningstar.co.uk) public international website to retrieve an authorization token which is dynamic (i.e. it changes with any call and does not last for more few minutes). The token is then used to call the official Morningstar API for prices and retrieve securitires' prices. 8 | For currencies, a different public endpoint is being used. This because Morningstar does not provide and easy way for retrieving currencies. I adopted an easier approach using [Foreign exchange rates API](https://exchangeratesapi.io). 9 | Finally, the script returns a list in the stdin of funds prices following the format requested by hleder with the date equal to the last price date (it is not the date of script execution). 10 | Example: 11 | 12 | ```bash 13 | P 2021-01-08 GB00B907VX32 214.18 GBX 14 | P 2021-01-08 GB0006010168 1625.0 GBX 15 | ``` 16 | ## Installation 17 | 18 | Download the script. To install the dependencies run: 19 | 20 | ```bash 21 | pip install -r requirements.txt 22 | ``` 23 | ## Configuration file 24 | 25 | Create a config file, in YAML format, in a location at your choice. The file must contain a key `funds` with a list of ISIN. You can check the included `securities_example.yaml` or the specimen below: 26 | 27 | ```yaml 28 | funds: 29 | - code: GB0006010168 30 | name: A fund 31 | - code: GB00B907VX32 32 | universe: FOGBR$$ALL 33 | name: Another fund 34 | shares: 35 | - code: US0258161092 36 | - code: IT0003497168 37 | currencis: 38 | - code: EUR 39 | base: GBP 40 | ``` 41 | 42 | ### Code 43 | The code **must** the ISIN for the security. Unfortunately we cannot accept [WKN](https://en.wikipedia.org/wiki/Wertpapierkennnummer), [VALOR](https://en.wikipedia.org/wiki/Valoren_number), [SEDOL](https://en.wikipedia.org/wiki/SEDOL) or the Morningstar very own identifier. The limitation is in API which cannot accept a list of different type of identifier at the same time. ISIN is definitely the most common. 44 | 45 | ### Universe 46 | The **universe** is not mandatory. If not added, the universe will be standard one from the UK website whichnormally includes most of securities. In case your security is listed on a *non-default* market, you might need to add the universe. The easiest way to identify the universe is looking at the URL when you are browsing your security. You can either see it as URL parameter, called `UniverseID` or at the end of a parameter called `SecurityToken`, example: 47 | ```url 48 | https://tools.morningstar.co.uk/uk/cefreport/default.aspx?SecurityToken=E0GBR00VWL]2]0]FCGBR$$ALL 49 | ``` 50 | 51 | The universe is the last 10 chars, i.e. **FCGBR$$ALL**. They follow a naming convention like the first five chars identify the type of investment and the country (Closed Fund Great Britain) and then a subset ($$ALL). Finding it should be straightforward, if not file a an issue and report the URL. 52 | Universes are not normally required, nor accepted now, for shares. 53 | 54 | ### Name 55 | The name is reserved for future use. The idea is to print a custom name in the output instead forcing the usage of ISIN which might not be the identifier you use today. 56 | 57 | ### Currencies 58 | Currencies are not optimized for bundle calls. Each currency requires and accepts a single pair of base and the code. 59 | 60 | ## Compatibility 61 | Morningstar provides a wide range of a financial instruments. Most of tests are realized on international markets (EMEA and APAC) The script has been found compabile with: 62 | - **funds**: Open Funds, Closed Funds, ETF 63 | - **shares**: stocks traded on most markets 64 | - **currencies**: currency conversions 65 | 66 | ## Usage 67 | 68 | Run the file `msdownloader.py` with the following arguments: 69 | 70 | |Argument|Accept|Description| 71 | |---|---|---| 72 | |-c|\|File name (with or without the full path) of the YAML config file containing all the securities| 73 | |-d|XH7946842KD| The ISIN code for which a full dump is requested. The script returns the whole JSON payload as returned by the API| 74 | |-l|XH7946842KD| A lookup function similar to the search box on the website. It might produce limited results| 75 | |-x||If specified, it forces the conversion from GBX to GBP| 76 | |-b||Return the beancount format instead of Ledger (Default)| 77 | |-o|output.txt|Save the output to a file instead of using the console| 78 | |-w||Force file overwrite instead of append (default)| 79 | 80 | ## Limits 81 | 82 | The currenct script is tested only against funds (ETC/ETF works as funds), Shares and Currencies. 83 | This script has been tested with Python >3.5 only 84 | 85 | ## Todo 86 | 87 | - [X] Accept arguments from command lines 88 | - [X] Provide conversion for GBX into GBP 89 | - [X] Save output directly into a file 90 | - [X] Save output into different format (ledger, beancount, etc) 91 | - [X] Test for currencies / shares / other type of securities 92 | - [X] Make a single call for multiple securities at the same time 93 | - [ ] Add the failback support to public API endpoint 94 | - [X] Improve code quality 95 | - [X] Utilize conversion per security 96 | - [X] Download currencies 97 | - [ ] Download Bond information -------------------------------------------------------------------------------- /msdownloader.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import re 3 | import uuid 4 | import json 5 | import yaml 6 | from dateutil.parser import parse 7 | import argparse 8 | import pydash 9 | 10 | 11 | # Function for retrieving the one-time bearer token from public website 12 | def get_ms_auth_token(): 13 | 14 | url = "https://www.morningstar.co.uk/Common/funds/snapshot/SustenabilitySAL.aspx" 15 | 16 | querystring = {"Site": "uk", "FC": "F00000NOM7", 17 | "IT": "FO", "LANG": "en-GB", "LITTLEMODE": "True"} 18 | 19 | headers = { 20 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 21 | 22 | response = requests.request( 23 | "GET", url, headers=headers, params=querystring) 24 | 25 | JWTregex = r"tokenMaaS\:\s\"(.+)\"" 26 | JWTtoken = re.findall(JWTregex, response.text) 27 | SALregex = r"salContentType\:\s\"(.+)\"" 28 | SALtoken = re.findall(SALregex, response.text) 29 | 30 | return SALtoken[0], JWTtoken[0] 31 | 32 | 33 | # Function for downloading fund data based on the API URL scrapped from the site morningstar.co.uk 34 | # This function is actually not used. 35 | def get_ms_fund_data_failback(SAL, JWT, SEC): 36 | url = "https://www.us-api.morningstar.com/sal/sal-service/fund/esg/v1/" + SEC + "/data" 37 | 38 | querystring = {"locale": "en-GB", "clientId": "MDC_intl", 39 | "benchmarkId": "category", "version": "3.36.1"} 40 | 41 | headers = {'x-sal-contenttype': SAL, 'Authorization': 'Bearer '+JWT, 'x-api-requestid': str(uuid.uuid4()), 42 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 43 | response = requests.request( 44 | "GET", url, headers=headers, params=querystring) 45 | snapshot = json.loads(response.text) 46 | return snapshot 47 | 48 | 49 | # Function for downloading fund data based on the official developer documentation 50 | def get_ms_security_snapshot(SAL, JWT, ISIN): 51 | url = " https://www.us-api.morningstar.com/ecint/v1/securities/" + ISIN 52 | querystring = {"viewid": "snapshot", 53 | "idtype": "isin", 54 | "responseViewFormat": "json"} 55 | headers = {'x-sal-contenttype': SAL, 'Authorization': 'Bearer '+JWT, 'x-api-requestid': str(uuid.uuid4()), 56 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 57 | response = requests.request( 58 | "GET", url, headers=headers, params=querystring) 59 | snapshot = json.loads(response.text) 60 | return snapshot 61 | 62 | 63 | def get_ms_security_data(SAL, JWT, term): 64 | url = "https://www.us-api.morningstar.com/ecint/v1/screener" 65 | querystring = { 66 | 'outputType': 'json', 67 | 'version': '1', 68 | 'languageId': 'en-GB', 69 | 'securityDataPoints': 'SecId,Name,HoldingTypeId,Universe,PriceCurrency,ISIN', 70 | 'term': term 71 | } 72 | headers = {'x-sal-contenttype': SAL, 'Authorization': 'Bearer '+JWT, 'x-api-requestid': str(uuid.uuid4()), 73 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 74 | response = requests.request( 75 | "GET", url, headers=headers, params=querystring) 76 | data = json.loads(response.text) 77 | return data 78 | 79 | # Function for downloading Fund/ETF closing prices from Screener API 80 | 81 | 82 | def get_ms_funds_prices(SAL, JWT, funds_list): 83 | output_dict = [] 84 | for d in funds_list: 85 | d.setdefault('universe', 'FOEUR$$ALL') 86 | output_dict.append(d['universe']) 87 | universe_list = '|'.join( 88 | [str(universe) for universe in pydash.arrays.sorted_uniq(output_dict)]) 89 | filters_list = ':'.join([str(item['code']) for item in funds_list]) 90 | 91 | url = "https://www.us-api.morningstar.com/ecint/v1/screener" 92 | querystring = { 93 | 'outputType': 'json', 94 | 'version': '1', 95 | 'languageId': 'en-GB', 96 | 'universeIds': universe_list, 97 | 'securityDataPoints': 'ISIN,ClosePriceDate,ClosePrice,PriceCurrency,Universe', 98 | 'filters': 'ISIN:IN:' + filters_list 99 | } 100 | headers = {'x-sal-contenttype': SAL, 'Authorization': 'Bearer '+JWT, 'x-api-requestid': str(uuid.uuid4()), 101 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 102 | response = requests.request( 103 | "GET", url, headers=headers, params=querystring) 104 | prices = json.loads(response.text) 105 | return prices['rows'] 106 | 107 | 108 | def get_ms_shares_prices(SAL, JWT, shares_list): 109 | output_dict = [] 110 | for d in shares_list: 111 | d.setdefault('universe', 'E0WWE$$ALL') 112 | output_dict.append(d['universe']) 113 | universe_list = '|'.join( 114 | [str(universe) for universe in pydash.arrays.sorted_uniq(output_dict)]) 115 | filters_list = ':'.join([str(item['code']) for item in shares_list]) 116 | url = "https://www.us-api.morningstar.com/ecint/v1/screener" 117 | querystring = { 118 | 'outputType': 'json', 119 | 'version': '1', 120 | 'languageId': 'en-GB', 121 | 'universeIds': universe_list, 122 | 'securityDataPoints': 'ISIN,ClosePriceDate,ClosePrice,PriceCurrency', 123 | 'filters': 'IsPrimary:EQ:True+ISIN:IN:' + filters_list 124 | } 125 | headers = {'x-sal-contenttype': SAL, 'Authorization': 'Bearer '+JWT, 'x-api-requestid': str(uuid.uuid4()), 126 | 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36'} 127 | response = requests.request( 128 | "GET", url, headers=headers, params=querystring) 129 | prices = json.loads(response.text) 130 | return prices['rows'] 131 | 132 | 133 | def get_currencies(currencies_list): 134 | output_dict = [] 135 | for d in currencies_list: 136 | item_dict = {} 137 | url = 'https://api.exchangeratesapi.io/latest' 138 | base_cur = d['base'] 139 | code_cur = d['code'] 140 | querystring = { 141 | 'base': base_cur, 142 | 'symbols': code_cur 143 | } 144 | response = requests.request( 145 | "GET", url, params=querystring) 146 | prices = json.loads(response.text) 147 | 148 | item_dict['ISIN'] = code_cur 149 | item_dict['ClosePriceDate'] = prices['date'] 150 | item_dict['ClosePrice'] = round(float(prices['rates'][code_cur]), 4) 151 | item_dict['PriceCurrency'] = base_cur 152 | output_dict.append(item_dict) 153 | 154 | return output_dict 155 | 156 | 157 | def extract_fund_value(snapshot, gbx_to_gbp): 158 | last_price_value = str(snapshot[0]['LastPrice']['Value']) 159 | last_price_date = parse( 160 | snapshot[0]['LastPrice']['Date']).strftime('%Y-%m-%d') 161 | last_price_currency = (snapshot[0]['LastPrice']['Currency']['Id']) 162 | 163 | if gbx_to_gbp and last_price_currency == "GBX": 164 | last_price_currency = 'GBP' 165 | last_price_value = round(float(last_price_value)/100, 4) 166 | 167 | return last_price_value, last_price_date, last_price_currency 168 | 169 | 170 | def read_securities(filename): 171 | with open(filename, 'r') as file: 172 | securities = yaml.full_load(file) 173 | return securities 174 | 175 | 176 | def print_prices(prices): 177 | for security_price in prices: 178 | close_price = str(round(float(security_price['ClosePrice'])/100, 4) 179 | if my_args.x is True and security_price['PriceCurrency'] == 'GBX' else security_price['ClosePrice']) 180 | price_currency = str( 181 | 'GBP' if my_args.x is True and security_price['PriceCurrency'] == 'GBX' else security_price['PriceCurrency']) 182 | if my_args.b is True: 183 | print('P {} {} {} {}'.format( 184 | security_price['ClosePriceDate'], 185 | security_price['ISIN'], 186 | close_price, 187 | price_currency)) 188 | else: 189 | print('{} {} {}'.format( 190 | security_price['ClosePriceDate'], 191 | security_price['ISIN'], 192 | close_price)) 193 | 194 | 195 | def save_prices(prices, filename, filemode): 196 | print(filemode) 197 | f = open(filename, ('a' if filemode is False else 'w')) 198 | for security_price in prices: 199 | close_price = str(round(float(security_price['ClosePrice'])/100, 4) 200 | if my_args.x is True and security_price['PriceCurrency'] == 'GBX' else security_price['ClosePrice']) 201 | price_currency = str( 202 | 'GBP' if my_args.x is True and security_price['PriceCurrency'] == 'GBX' else security_price['PriceCurrency']) 203 | if my_args.b is True: 204 | f.write('P {} {} {} {}\n'.format( 205 | security_price['ClosePriceDate'], 206 | security_price['ISIN'], 207 | close_price, 208 | price_currency)) 209 | else: 210 | f.write('{} {} {}\n'.format( 211 | security_price['ClosePriceDate'], 212 | security_price['ISIN'], 213 | close_price)) 214 | f.close() 215 | 216 | 217 | def read_args(): 218 | arg_parser = argparse.ArgumentParser( 219 | description='Download securities latest price from Morningstar') 220 | group_input = arg_parser.add_mutually_exclusive_group(required=True) 221 | group_input.add_argument('-c', action='store', metavar='securities.yaml', 222 | default='securities.yaml', type=str, help='The YAML file with the list of securities') 223 | group_input.add_argument('-d', action='store', metavar='XY03ID03131ID', 224 | help='Dump info for a single security in JSON') 225 | group_input.add_argument('-l', action='store', metavar='GB00B0XWNK36', 226 | help='Lookup for a security and print the simple data') 227 | arg_parser.add_argument('-x', action='store_true', 228 | help='Force conversion from pence sterling GBX into pound sterling GBP') 229 | arg_parser.add_argument('-b', action='store_false', 230 | help='Return beancount format instead of ledger/hledger') 231 | arg_parser.add_argument('-o', action='store', metavar='output.txt', 232 | type=str, help='The output file for the latest prices') 233 | arg_parser.add_argument('-w', action='store_true', 234 | help='Trucate the output file if exists and add new prices') 235 | return arg_parser.parse_args() 236 | 237 | 238 | my_args = read_args() 239 | 240 | auth = get_ms_auth_token() 241 | 242 | if my_args.d is not None: 243 | print(json.dumps(get_ms_security_snapshot( 244 | auth[0], auth[1], my_args.d), indent=4, sort_keys=True)) 245 | elif my_args.l is not None: 246 | print(json.dumps(get_ms_security_data( 247 | auth[0], auth[1], my_args.l), indent=4, sort_keys=True)) 248 | else: 249 | securities = read_securities(my_args.c) 250 | prices = [] 251 | for group in securities.keys(): 252 | if group == 'funds': 253 | funds_prices = get_ms_funds_prices( 254 | auth[0], auth[1], securities['funds']) 255 | prices = funds_prices 256 | elif group == 'shares': 257 | shares_prices = get_ms_shares_prices( 258 | auth[0], auth[1], securities['shares']) 259 | prices.extend(shares_prices) 260 | elif group == 'currencies': 261 | currencies_prices = get_currencies(securities['currencies']) 262 | prices.extend(currencies_prices) 263 | else: 264 | print('No match for ' + group) 265 | 266 | if my_args.o is not None: 267 | save_prices(prices, my_args.o, my_args.w) 268 | else: 269 | print_prices(prices) 270 | --------------------------------------------------------------------------------