├── requirements.txt
├── docs
    └── design-output.txt
├── CHANGELOG.md
├── README.md
├── winlatin1toascii.py
├── .gitignore
├── ofxpostern.py
├── LICENSE
└── testofx.py


/requirements.txt:
--------------------------------------------------------------------------------
1 | ### Runtime ###
2 | 
3 | requests
4 | xmltodict
5 | 
6 | ### Testing ###
7 | 
8 | 


--------------------------------------------------------------------------------
/docs/design-output.txt:
--------------------------------------------------------------------------------
 1 | ofx-postern: version 0.0.1
 2 | 
 3 | Start: 2018-07-28 14:12:43
 4 |   Sending GET /
 5 |   Sending GET OFX
 6 |   Analysing
 7 | End:   2018-07-28 14:13:05
 8 | 
 9 | Financial Institution
10 | =====================
11 | 
12 | Name:    ShadowBank
13 | Address: 123 Fake Street
14 |          Mountain, CO 12345
15 |          USA
16 | URL:     https://www.shadowbank.com***
17 | 
18 | OFX Server
19 | ==========
20 | 
21 | FID: 1001
22 | ORG: SB
23 | URL: http://ofx.shadowbank.com/ofx/ofx.dll
24 | TLS: Valid***
25 | 
26 | Capabilities
27 | ============
28 | 
29 | * Checking
30 | * Savings
31 | * Bill Pay
32 | 
33 | Fingerprint
34 | ===========
35 | 
36 |  Service Provider: Metavante
37 |  HTTP Server:      IIS 7.5
38 |  App Framework:    ASP.NET
39 | 
40 |  OFX Server
41 |   Company:         Metavante
42 |   Product:
43 |   Version:         7.2.0
44 | 
45 | Tests
46 | =====
47 | 
48 | Information Disclosure: FAIL
49 |   GET /: 
50 |      18: "Server: IIS 7.5"
51 |   GET OFX:
52 |     345: "Version: 7.2.0"
53 | 
54 | NULL Values Returned: FAIL
55 |   POST OFX: 
56 |      45: <FID>NULL<FID>
57 | 
58 | XSS: PASS
59 | 
60 | Weak Password Policy: FAIL
61 | 


--------------------------------------------------------------------------------
/CHANGELOG.md:
--------------------------------------------------------------------------------
 1 | # CHANGELOG
 2 | 
 3 | ## 0.2.0 - 2018-10-30
 4 | 
 5 | Added many new tests and capability parsing.
 6 | 
 7 | ### Features
 8 | 
 9 | * Parse OFX 2.0.x
10 | * Enumerate capabilities:
11 |   - Investments
12 |   - Bill Pay
13 |   - Credit Card
14 |   - 401(k)
15 |   - Tax
16 |   - Messaging
17 |   - Authentication
18 | * Fingerprint some Service Providers
19 | 
20 | ### Tests
21 | 
22 | * Check that TLS is required
23 | * Check for correct application/x-ofx content-type
24 | * Check for web server / framework version disclosure
25 | * Check for username disclosure
26 | * Check for NULL return values
27 | * Check for Internal Server Error 500
28 | * Check for internal IP address disclosure
29 | 
30 | ## 0.1.0 - 2018-08-16
31 | 
32 | All features (capabilities, fingerprinting, security scan) work in a limited capacity. The tool has been tested against multiple live OFX servers.
33 | 
34 | ### Features
35 | 
36 | * Parse OFX 1.0.x
37 | * Enumerate FI contact information
38 | * Fingerprint server stack based off HTTP headers
39 | * Fingerprint OFX software based off URL paths
40 | * Enumerate Banking capabilities
41 | * Run recon security tests with anonymous credentials
42 | 
43 | ### Tests
44 | 
45 | * Check for MFA support within the protocol
46 | * Check password policy
47 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # ofxpostern
 2 | Vulnerability scanner for OFX servers.
 3 | 
 4 | **ofxpostern** is a CLI tool which fingerprints an OFX service, describes its capabilities, and assesses its security.
 5 | 
 6 | ## Installation
 7 | 
 8 | **ofxpostern** is written in Python 3 with few external dependencies. It has only been tested on Linux.
 9 | 
10 | 1. `git clone git@github.com:sdann/ofxpostern.git`
11 | 2. `cd ofxpostern`
12 | 3. `pip install -r requirements.txt`
13 | 
14 | ## Usage
15 | 
16 | `./ofxpostern.py [-f FID] [-o ORG] url`
17 | 
18 | Example:
19 | 
20 | `./ofxpostern.py -o Cavion -f 11135 https://ofx.lanxtra.com/ofx/servlet/Teller`
21 | 
22 | The Financial Identifer (FID) and Organization (ORG) are sometimes optional, sometimes required depending on the Financial Institution.
23 | 
24 | A current list of public OFX servers is available at https://ofxhome.com/.
25 | 
26 | ## Security Scan
27 | 
28 | A small number of security tests are implemented. All are done with anonymous credentials.
29 | 
30 | * Check that TLS is required
31 | * Check for correct application/x-ofx content-type
32 | * Check for web server / framework version disclosure
33 | * Check for MFA support within the protocol
34 | * Check password policy
35 | * Check for username disclosure
36 | * Check for NULL return values
37 | * Check for Internal Server Error 500
38 | * Check for internal IP address disclosure
39 | 
40 | ## Advanced
41 | 
42 | Within the *ofxpostern.py* script the *cache* global variable can be enabled to store text copies of all OFX protocol responses to `$HOME/.ofxpostern/`.
43 | 


--------------------------------------------------------------------------------
/winlatin1toascii.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python3
 2 | 
 3 | '''
 4 | Script to convert Windows Latin 1 characters to ASCII equivalent
 5 | 
 6 | See: http://jkorpela.fi/www/windows-chars.html
 7 | '''
 8 | 
 9 | import os
10 | import sys
11 | 
12 | #
13 | # Defines
14 | #
15 | 
16 | #
17 | # Globals
18 | #
19 | 
20 | #
21 | # Helper Functions
22 | #
23 | 
24 | def usage():
25 |     '''
26 |     Print usage statement.
27 |     '''
28 |     cmd = os.path.basename(sys.argv[0])
29 |     indent = ' ' * 4
30 | 
31 |     print('Usage:')
32 |     print('{}{} <filename>'.format(indent * 1, cmd))
33 |     print('{}Write to stdout file with Windows Latin 1 characters converted'.format(indent * 2, cmd))
34 | 
35 | #
36 | # Core Logic
37 | #
38 | 
39 | def convert(buf):
40 | #    import ipdb; ipdb.set_trace()
41 |     out = bytearray()
42 |     for b in buf:
43 |         if b == 0x92: out.append(ord("'"))
44 |         elif b == 0x93: out.append(ord('"'))
45 |         elif b == 0x94: out.append(ord('"'))
46 |         elif b == 0x96: out.append(ord('-'))
47 |         elif b == 0x97: out.append(ord('-'))
48 |         elif b == 0xA0: continue
49 |         else: out.append(b)
50 | 
51 |     return out.decode('ascii')
52 | 
53 | 
54 | def main(args):
55 | 
56 |     if len(args) != 1:
57 |         usage()
58 |         sys.exit(1)
59 | 
60 |     with open(args[0], 'rb') as in_file:
61 |         buf = convert(in_file.read())
62 |         print(buf, end='')
63 | 
64 | 
65 | if __name__ == '__main__':
66 | 
67 |     try:
68 |         main(sys.argv[1:])
69 |     except KeyboardInterrupt:
70 |         pass
71 | 


--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
  1 | # Byte-compiled / optimized / DLL files
  2 | __pycache__/
  3 | *.py[cod]
  4 | *$py.class
  5 | 
  6 | # C extensions
  7 | *.so
  8 | 
  9 | # Distribution / packaging
 10 | .Python
 11 | build/
 12 | develop-eggs/
 13 | dist/
 14 | downloads/
 15 | eggs/
 16 | .eggs/
 17 | lib/
 18 | lib64/
 19 | parts/
 20 | sdist/
 21 | var/
 22 | wheels/
 23 | *.egg-info/
 24 | .installed.cfg
 25 | *.egg
 26 | MANIFEST
 27 | 
 28 | # PyInstaller
 29 | #  Usually these files are written by a python script from a template
 30 | #  before PyInstaller builds the exe, so as to inject date/other infos into it.
 31 | *.manifest
 32 | *.spec
 33 | 
 34 | # Installer logs
 35 | pip-log.txt
 36 | pip-delete-this-directory.txt
 37 | 
 38 | # Unit test / coverage reports
 39 | htmlcov/
 40 | .tox/
 41 | .coverage
 42 | .coverage.*
 43 | .cache
 44 | nosetests.xml
 45 | coverage.xml
 46 | *.cover
 47 | .hypothesis/
 48 | .pytest_cache/
 49 | 
 50 | # Translations
 51 | *.mo
 52 | *.pot
 53 | 
 54 | # Django stuff:
 55 | *.log
 56 | local_settings.py
 57 | db.sqlite3
 58 | 
 59 | # Flask stuff:
 60 | instance/
 61 | .webassets-cache
 62 | 
 63 | # Scrapy stuff:
 64 | .scrapy
 65 | 
 66 | # Sphinx documentation
 67 | docs/_build/
 68 | 
 69 | # PyBuilder
 70 | target/
 71 | 
 72 | # Jupyter Notebook
 73 | .ipynb_checkpoints
 74 | 
 75 | # pyenv
 76 | .python-version
 77 | 
 78 | # celery beat schedule file
 79 | celerybeat-schedule
 80 | 
 81 | # SageMath parsed files
 82 | *.sage.py
 83 | 
 84 | # Environments
 85 | .env
 86 | .venv
 87 | env/
 88 | venv/
 89 | ENV/
 90 | env.bak/
 91 | venv.bak/
 92 | 
 93 | # Spyder project settings
 94 | .spyderproject
 95 | .spyproject
 96 | 
 97 | # Rope project settings
 98 | .ropeproject
 99 | 
100 | # mkdocs documentation
101 | /site
102 | 
103 | # mypy
104 | .mypy_cache/
105 | 


--------------------------------------------------------------------------------
/ofxpostern.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python3
  2 | 
  3 | '''
  4 | Fingerprint an OFX server.
  5 | '''
  6 | 
  7 | import argparse
  8 | import json
  9 | import os
 10 | import pickle
 11 | import requests
 12 | import sys
 13 | import time
 14 | 
 15 | import testofx
 16 | 
 17 | #
 18 | # Defines
 19 | #
 20 | 
 21 | PROGRAM_DESCRIPTION = 'Fingerprint an OFX server.'
 22 | PROGRAM_NAME = 'ofxpostern'
 23 | VERSION = '0.2.0'
 24 | 
 25 | DATA_DIR = '{}/.{}'.format(os.environ['HOME'], PROGRAM_NAME)
 26 | FIS_DIR = '{}/{}'.format(DATA_DIR, 'fi')
 27 | FI_DIR_FMT = '{}/{}'.format(FIS_DIR, '{}-{}-{}')
 28 | 
 29 | STR_HEADERS = 'headers'
 30 | STR_BODY    = 'body'
 31 | STR_OBJ     = 'object'
 32 | 
 33 | #
 34 | # Globals
 35 | #
 36 | 
 37 | debug = False
 38 | cache = False
 39 | 
 40 | fi_dir = ''
 41 | 
 42 | req_results = dict()
 43 | 
 44 | #
 45 | # Helper Functions
 46 | #
 47 | 
 48 | def init(server):
 49 |     '''
 50 |     Initialize environment
 51 |     '''
 52 | 
 53 |     global fi_dir
 54 | 
 55 |     if cache:
 56 |         # Convert URL into usable filename
 57 |         url_fname = server.ofxurl.partition('/')[2][1:].replace('/','_').replace('&','+')
 58 |         fi_dir = FI_DIR_FMT.format(url_fname, server.fid, server.org)
 59 | 
 60 |         # Create directory to store cached data
 61 |         os.makedirs(DATA_DIR, mode=0o770, exist_ok=True)
 62 |         os.makedirs(FIS_DIR, mode=0o770, exist_ok=True)
 63 |         os.makedirs(fi_dir, mode=0o770, exist_ok=True)
 64 | 
 65 | 
 66 | def print_debug(msg):
 67 |     if debug: print('DEBUG: {}'.format(msg))
 68 | 
 69 | 
 70 | def print_header(msg, lvl):
 71 |     '''
 72 |     Print a header with underline on 2nd line
 73 | 
 74 |     Similar to <H1>, <H2>
 75 |     '''
 76 |     under_char = ''
 77 | 
 78 |     if lvl == 1: under_char = '#'
 79 |     elif lvl == 2: under_char = '='
 80 |     elif lvl == 3: under_char = '-'
 81 |     else: raise ValueError('Unknown lvl: {}'.format(lvl))
 82 | 
 83 |     print(msg)
 84 |     print(under_char * len(msg))
 85 | 
 86 | 
 87 | def print_kv_list(kv_list):
 88 |     '''
 89 |     Print key:value list with pretty formatting
 90 | 
 91 |     kv_list: list[tuples]
 92 |     '''
 93 | 
 94 |     k_width = 0
 95 |     for k, v in kv_list:
 96 |         if len(k) > k_width:
 97 |             k_width = len(k)
 98 | 
 99 |     for k, v in kv_list:
100 |         separator = ':' if len(k) > 0 else ' '
101 |         print('{:{}} {}'.format(k+separator, k_width+1, v))
102 | 
103 | 
104 | def print_tree(tree, lvl=1):
105 |     '''
106 |     Print embedded lists as an indented text tree
107 | 
108 |     Recursive to depth 3
109 | 
110 |     tree: list[val, list[]...]
111 |     '''
112 |     indent = 2
113 |     bullet = ''
114 | 
115 |     if lvl == 1: bullet = '*'
116 |     elif lvl == 2: bullet = '+'
117 |     elif lvl == 3: bullet = '-'
118 |     else: raise ValueError('Unknown lvl: {}'.format(lvl))
119 | 
120 |     for i in tree:
121 |         if type(i) is list:
122 |             print_tree(i, lvl+1)
123 |         else:
124 |             print('{}{} {}'.format(' '*(indent*(lvl-1)), bullet, i))
125 | 
126 | 
127 | def print_list(lst, indent=0):
128 |     '''
129 |     Print list with option intent
130 | 
131 |     lst: list[]
132 |     indent: int, number of spaces
133 |     '''
134 |     bullet = '*'
135 | 
136 |     for i in lst:
137 |         print('{}{} {}'.format(' '*indent, bullet, i))
138 | 
139 | #
140 | # Core Logic
141 | #
142 | 
143 | def send_req(server, req_name):
144 |     '''
145 |     Send request to the OFX server.
146 |     '''
147 | 
148 |     cached = True
149 |     res_name_base = req_name.replace('/', '+').replace(' ', '_')
150 |     res_obj_path = '{}/{}-{}'.format(fi_dir, res_name_base, STR_OBJ)
151 |     res_hdr_path = '{}/{}-{}'.format(fi_dir, res_name_base, STR_HEADERS)
152 |     res_body_path = '{}/{}-{}'.format(fi_dir, res_name_base, STR_BODY)
153 | 
154 |     # Pull results out of cache if they exist
155 |     if cache:
156 |         try:
157 |             with open(res_obj_path, 'rb') as fd:
158 |                 print_debug('Reading res from cache')
159 |                 req_results[req_name] = pickle.loads(fd.read())
160 |         except FileNotFoundError:
161 |             cached = False
162 | 
163 |     if not cache or not cached:
164 |         otc = testofx.OFXTestClient(output=debug, tls_verify=server.get_tls())
165 |         res = otc.send_req(req_name, server)
166 | 
167 |         # Store result for analysis
168 |         req_results[req_name] = res
169 | 
170 |         if cache:
171 |             # Store persistently for debugging
172 |             with open(res_obj_path, 'wb') as fd:
173 |                 fd.write(pickle.dumps(res))
174 |             with open(res_hdr_path, 'w') as fd:
175 |                 fd.write(json.dumps(dict(res.headers)))
176 |             with open(res_body_path, 'w') as fd:
177 |                 fd.write(res.text)
178 | 
179 | 
180 | def check_tls(server, tls_verify):
181 |     '''
182 |     Check server TLS settings.
183 |     '''
184 |     if cache:
185 |         return
186 | 
187 |     # Do a simple works/not works check for now
188 |     try:
189 |         r = requests.get(server.ofxurl)
190 |     except requests.exceptions.SSLError as ex:
191 |         server.set_tls(False)
192 |         print(ex)
193 |         if tls_verify:
194 |             sys.exit(-1)
195 |     else:
196 |         server.set_tls(True)
197 | 
198 | 
199 | def report_cli_fi(profrs):
200 |     '''
201 |     Print Financial Institution information
202 |     '''
203 | 
204 |     print_header('Financial Institution', 2)
205 |     print()
206 | 
207 |     if not profrs:
208 |         return
209 | 
210 |     fi_list = []
211 |     output = (
212 |             ('FINAME', 'Name'),
213 |             ('ADDR1', 'Address'),
214 |             ('ADDR2', ''),
215 |             ('ADDR3', ''),
216 |             )
217 | 
218 |     for tup in output:
219 |         try:
220 |             val = profrs.profile[tup[0]]
221 |             fi_list.append((tup[1], val))
222 |         except KeyError: pass
223 | 
224 |     city = ''
225 |     state = ''
226 |     postalcode = ''
227 | 
228 |     try:
229 |         city = profrs.profile['CITY']
230 |         state = profrs.profile['STATE']
231 |         postalcode = profrs.profile['POSTALCODE']
232 |     except KeyError: pass
233 | 
234 |     fi_list.append(('', '{}, {} {}'.format(city, state, postalcode)))
235 | 
236 |     country = ''
237 | 
238 |     try:
239 |         country = profrs.profile['COUNTRY']
240 |     except KeyError: pass
241 | 
242 |     fi_list.append(('', country))
243 | 
244 |     print_kv_list(fi_list)
245 | 
246 |     print()
247 | 
248 | 
249 | def report_cli_server(profrs):
250 |     '''
251 |     Print server information
252 |     '''
253 |     print_header('OFX Server', 2)
254 |     print()
255 | 
256 |     if not profrs:
257 |         return
258 | 
259 |     fi_list = []
260 | 
261 |     fi_list.append(('OFX Version', profrs.get_version()))
262 | 
263 |     try:
264 |         val = profrs.signon['FID']
265 |         fi_list.append(('FID', val))
266 |     except KeyError: pass
267 | 
268 |     try:
269 |         val = profrs.signon['ORG']
270 |         fi_list.append(('ORG', val))
271 |     except KeyError: pass
272 | 
273 |     try:
274 |         val = profrs.profile['OFXURL']
275 |         fi_list.append(('URL', val))
276 |     except KeyError: pass
277 | 
278 |     print_kv_list(fi_list)
279 | 
280 |     print()
281 | 
282 | 
283 | def report_cli_capabilities(profrs):
284 |     '''
285 |     Print server capabilities
286 |     '''
287 |     print_header('Capabilities', 2)
288 |     print()
289 | 
290 |     if not profrs:
291 |         return
292 | 
293 |     cap_tree = []
294 | 
295 |     try:
296 |         v1 = profrs.profile['BANKING']
297 |         cap_tree.append('Banking')
298 |         sub_tree = []
299 |         try:
300 |             if v1['INTRAXFR']:
301 |                 sub_tree.append('Intrabank Transfer')
302 |         except KeyError: pass
303 |         try:
304 |             v2 = v1['MESSAGES']
305 |             sub_sub_tree = []
306 |             try:
307 |                 if v2['EMAIL']:
308 |                     sub_sub_tree.append('Email')
309 |             except KeyError: pass
310 |             try:
311 |                 if v2['NOTIFY']:
312 |                     sub_sub_tree.append('Notifications')
313 |             except KeyError: pass
314 |             if len(sub_sub_tree) > 0:
315 |                 sub_tree.append('Messaging')
316 |                 sub_tree.append(sub_sub_tree)
317 |         except KeyError: pass
318 |         cap_tree.append(sub_tree)
319 |     except KeyError: pass
320 | 
321 |     try:
322 |         v1 = profrs.profile['INVESTMENT']
323 |         cap_tree.append('Investment')
324 |         sub_tree = []
325 |         try:
326 |             if v1['TRANSACTIONS']:
327 |                 sub_tree.append('Transactions')
328 |         except KeyError: pass
329 |         try:
330 |             if v1['OPENORDERS']:
331 |                 sub_tree.append('Open Orders')
332 |         except KeyError: pass
333 |         try:
334 |             if v1['POSITIONS']:
335 |                 sub_tree.append('Positions')
336 |         except KeyError: pass
337 |         try:
338 |             if v1['BALANCES']:
339 |                 sub_tree.append('Balances')
340 |         except KeyError: pass
341 |         try:
342 |             if v1['401K']:
343 |                 sub_tree.append('401(k)')
344 |         except KeyError: pass
345 |         try:
346 |             if v1['QUOTES']:
347 |                 sub_tree.append('Quotes')
348 |         except KeyError: pass
349 |         cap_tree.append(sub_tree)
350 |     except KeyError: pass
351 | 
352 |     try:
353 |         v1 = profrs.profile['CREDITCARD']
354 |         cap_tree.append('Credit Card')
355 |         sub_tree = []
356 |         try:
357 |             if v1['STATEMENT']:
358 |                 sub_tree.append('Closing Statement')
359 |         except KeyError: pass
360 |         cap_tree.append(sub_tree)
361 |     except KeyError: pass
362 | 
363 |     try:
364 |         v1 = profrs.profile['BILLPAY']
365 |         cap_tree.append('Bill Pay')
366 |         sub_tree = []
367 |         cap_tree.append(sub_tree)
368 |     except KeyError: pass
369 | 
370 |     try:
371 |         v1 = profrs.profile['TAXES']
372 |         cap_tree.append('Taxes')
373 |         sub_tree = []
374 |         try:
375 |             if v1['1099']:
376 |                 sub_tree.append('1099')
377 |         except KeyError: pass
378 |         try:
379 |             if v1['1099B']:
380 |                 sub_tree.append('Schedule D')
381 |         except KeyError: pass
382 |         try:
383 |             sub_sub_tree = []
384 |             v2 = v1['YEARS']
385 |             sub_tree.append('Years')
386 |             sub_sub_tree.append(v2)
387 |             sub_tree.append(sub_sub_tree)
388 |         except KeyError: pass
389 |         cap_tree.append(sub_tree)
390 |     except KeyError: pass
391 | 
392 |     try:
393 |         v1 = profrs.profile['MESSAGING']
394 |         cap_tree.append('Messaging')
395 |         sub_tree = []
396 |         try:
397 |             if v1['EMAIL']:
398 |                 sub_tree.append('Email')
399 |         except KeyError: pass
400 |         try:
401 |             if v1['MIME']:
402 |                 sub_tree.append('MIME')
403 |         except KeyError: pass
404 |         cap_tree.append(sub_tree)
405 |     except KeyError: pass
406 | 
407 |     try:
408 |         v1 = profrs.profile['AUTHENTICATION']
409 |         sub_tree = []
410 |         try:
411 |             v2 = v1['MFA']
412 |             sub_sub_tree = []
413 |             try:
414 |                 if v2['CLIENTUID']:
415 |                     sub_sub_tree.append('Require Client ID')
416 |             except KeyError: pass
417 |             if len(sub_sub_tree) > 0:
418 |                 sub_tree.append('MFA')
419 |                 sub_tree.append(sub_sub_tree)
420 |         except KeyError: pass
421 |         if len(sub_tree) > 0:
422 |             cap_tree.append('Authentication')
423 |             cap_tree.append(sub_tree)
424 |     except KeyError: pass
425 | 
426 |     print_tree(cap_tree)
427 | 
428 |     print()
429 | 
430 | 
431 | def report_cli_fingerprint(server):
432 |     '''
433 |     Print info about service framework and software
434 |     '''
435 |     print_header('Fingerprint', 2)
436 |     print()
437 | 
438 |     fng_list = [
439 |             ('HTTP Server', server.httpserver),
440 |             ('Web Framework', server.webframework)
441 |             ]
442 | 
443 |     print_kv_list(fng_list)
444 | 
445 |     print()
446 | 
447 |     print_header('OFX Software', 3)
448 |     print()
449 | 
450 |     svr_list = []
451 | 
452 |     if server.serviceprovider != '':
453 |         svr_list.append(('Service Provider', server.serviceprovider))
454 |         svr_list.append(('', ''))
455 | 
456 |     svr_list.extend([
457 |             ('Company', server.software['Company']),
458 |             ('Product', server.software['Product']),
459 |             ('Version', server.software['Version'])
460 |             ]
461 |     )
462 | 
463 |     print_kv_list(svr_list)
464 | 
465 |     print()
466 | 
467 | 
468 | def report_cli_tests(tests):
469 |     '''
470 |     Print info about security tests
471 |     '''
472 |     print_header('Tests', 2)
473 |     print()
474 | 
475 |     for tres in tests.results:
476 |         title = (tres['Title'], 'PASS' if tres['Passed'] else 'FAIL')
477 |         print_kv_list([title])
478 |         print_list(tres['Messages'], 2)
479 |         print ()
480 | 
481 | 
482 | def report_cli(server, profrs, tests):
483 |     '''
484 |     Print human readable report of all results to stdout
485 |     '''
486 |     report_cli_fi(profrs)
487 |     report_cli_server(profrs)
488 |     report_cli_capabilities(profrs)
489 |     report_cli_fingerprint(server)
490 |     report_cli_tests(tests)
491 | 
492 | 
493 | def main():
494 | 
495 |     parser = argparse.ArgumentParser(description=PROGRAM_DESCRIPTION)
496 |     parser.add_argument('url',
497 |             help='URL of OFX server to test')
498 |     parser.add_argument('-f', '--fid',
499 |             help='Financial ID of Institution',
500 |             required=False)
501 |     parser.add_argument('-o', '--org',
502 |             help='Organization within the Institution',
503 |             required=False)
504 |     parser.add_argument('--no-tls-verify',
505 |             dest='tls_verify',
506 |             action='store_false',
507 |             help='Skip TLS verification',
508 |             required=False)
509 |     parser.set_defaults(tls_verify=True)
510 |     args = parser.parse_args()
511 | 
512 |     print_debug(args)
513 | 
514 |     # TODO: validate input
515 |     server = testofx.OFXServerInstance(args.url, args.fid, args.org)
516 |     profrs = None
517 | 
518 |     # Initialize Persistent Cache
519 |     init(server)
520 | 
521 |     requests = [
522 |         testofx.REQ_NAME_GET_ROOT,
523 |         testofx.REQ_NAME_GET_OFX,
524 |         testofx.REQ_NAME_POST_OFX,
525 |         testofx.REQ_NAME_OFX_EMPTY,
526 |         testofx.REQ_NAME_OFX_PROFILE
527 |     ]
528 | 
529 |     # Display work in progress
530 |     print('{}: version {}'.format(parser.prog, VERSION))
531 |     print()
532 |     print('Start: {}'.format(time.asctime()))
533 |     print('  Checking TLS')
534 |     check_tls(server, args.tls_verify)
535 |     for req_name in requests:
536 |         print('  Sending {}'.format(req_name))
537 |         send_req(server, req_name)
538 |     print('  Analysing Server')
539 |     try:
540 |         profrs = testofx.OFXFile(req_results[testofx.REQ_NAME_OFX_PROFILE].text)
541 |     except ValueError as ex:
542 |         print('    {}'.format(ex))
543 |     print('  Fingerprinting')
544 |     try:
545 |         server.fingerprint(req_results)
546 |     except ValueError as ex:
547 |         print('    {}'.format(ex))
548 |     print('  Running Tests')
549 |     tests = testofx.OFXServerTests(server)
550 |     errors = tests.run_tests(req_results)
551 |     if len(errors) > 0:
552 |         for err in errors:
553 |             print('    {}'.format(err))
554 |     print('End:   {}'.format(time.asctime()))
555 |     time.sleep(1)
556 |     print()
557 | 
558 |     # Print Report
559 |     report_cli(server, profrs, tests)
560 | 
561 | if __name__ == '__main__':
562 |     main()
563 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
  1 |                     GNU GENERAL PUBLIC LICENSE
  2 |                        Version 3, 29 June 2007
  3 | 
  4 |  Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
  5 |  Everyone is permitted to copy and distribute verbatim copies
  6 |  of this license document, but changing it is not allowed.
  7 | 
  8 |                             Preamble
  9 | 
 10 |   The GNU General Public License is a free, copyleft license for
 11 | software and other kinds of works.
 12 | 
 13 |   The licenses for most software and other practical works are designed
 14 | to take away your freedom to share and change the works.  By contrast,
 15 | the GNU General Public License is intended to guarantee your freedom to
 16 | share and change all versions of a program--to make sure it remains free
 17 | software for all its users.  We, the Free Software Foundation, use the
 18 | GNU General Public License for most of our software; it applies also to
 19 | any other work released this way by its authors.  You can apply it to
 20 | your programs, too.
 21 | 
 22 |   When we speak of free software, we are referring to freedom, not
 23 | price.  Our General Public Licenses are designed to make sure that you
 24 | have the freedom to distribute copies of free software (and charge for
 25 | them if you wish), that you receive source code or can get it if you
 26 | want it, that you can change the software or use pieces of it in new
 27 | free programs, and that you know you can do these things.
 28 | 
 29 |   To protect your rights, we need to prevent others from denying you
 30 | these rights or asking you to surrender the rights.  Therefore, you have
 31 | certain responsibilities if you distribute copies of the software, or if
 32 | you modify it: responsibilities to respect the freedom of others.
 33 | 
 34 |   For example, if you distribute copies of such a program, whether
 35 | gratis or for a fee, you must pass on to the recipients the same
 36 | freedoms that you received.  You must make sure that they, too, receive
 37 | or can get the source code.  And you must show them these terms so they
 38 | know their rights.
 39 | 
 40 |   Developers that use the GNU GPL protect your rights with two steps:
 41 | (1) assert copyright on the software, and (2) offer you this License
 42 | giving you legal permission to copy, distribute and/or modify it.
 43 | 
 44 |   For the developers' and authors' protection, the GPL clearly explains
 45 | that there is no warranty for this free software.  For both users' and
 46 | authors' sake, the GPL requires that modified versions be marked as
 47 | changed, so that their problems will not be attributed erroneously to
 48 | authors of previous versions.
 49 | 
 50 |   Some devices are designed to deny users access to install or run
 51 | modified versions of the software inside them, although the manufacturer
 52 | can do so.  This is fundamentally incompatible with the aim of
 53 | protecting users' freedom to change the software.  The systematic
 54 | pattern of such abuse occurs in the area of products for individuals to
 55 | use, which is precisely where it is most unacceptable.  Therefore, we
 56 | have designed this version of the GPL to prohibit the practice for those
 57 | products.  If such problems arise substantially in other domains, we
 58 | stand ready to extend this provision to those domains in future versions
 59 | of the GPL, as needed to protect the freedom of users.
 60 | 
 61 |   Finally, every program is threatened constantly by software patents.
 62 | States should not allow patents to restrict development and use of
 63 | software on general-purpose computers, but in those that do, we wish to
 64 | avoid the special danger that patents applied to a free program could
 65 | make it effectively proprietary.  To prevent this, the GPL assures that
 66 | patents cannot be used to render the program non-free.
 67 | 
 68 |   The precise terms and conditions for copying, distribution and
 69 | modification follow.
 70 | 
 71 |                        TERMS AND CONDITIONS
 72 | 
 73 |   0. Definitions.
 74 | 
 75 |   "This License" refers to version 3 of the GNU General Public License.
 76 | 
 77 |   "Copyright" also means copyright-like laws that apply to other kinds of
 78 | works, such as semiconductor masks.
 79 | 
 80 |   "The Program" refers to any copyrightable work licensed under this
 81 | License.  Each licensee is addressed as "you".  "Licensees" and
 82 | "recipients" may be individuals or organizations.
 83 | 
 84 |   To "modify" a work means to copy from or adapt all or part of the work
 85 | in a fashion requiring copyright permission, other than the making of an
 86 | exact copy.  The resulting work is called a "modified version" of the
 87 | earlier work or a work "based on" the earlier work.
 88 | 
 89 |   A "covered work" means either the unmodified Program or a work based
 90 | on the Program.
 91 | 
 92 |   To "propagate" a work means to do anything with it that, without
 93 | permission, would make you directly or secondarily liable for
 94 | infringement under applicable copyright law, except executing it on a
 95 | computer or modifying a private copy.  Propagation includes copying,
 96 | distribution (with or without modification), making available to the
 97 | public, and in some countries other activities as well.
 98 | 
 99 |   To "convey" a work means any kind of propagation that enables other
100 | parties to make or receive copies.  Mere interaction with a user through
101 | a computer network, with no transfer of a copy, is not conveying.
102 | 
103 |   An interactive user interface displays "Appropriate Legal Notices"
104 | to the extent that it includes a convenient and prominently visible
105 | feature that (1) displays an appropriate copyright notice, and (2)
106 | tells the user that there is no warranty for the work (except to the
107 | extent that warranties are provided), that licensees may convey the
108 | work under this License, and how to view a copy of this License.  If
109 | the interface presents a list of user commands or options, such as a
110 | menu, a prominent item in the list meets this criterion.
111 | 
112 |   1. Source Code.
113 | 
114 |   The "source code" for a work means the preferred form of the work
115 | for making modifications to it.  "Object code" means any non-source
116 | form of a work.
117 | 
118 |   A "Standard Interface" means an interface that either is an official
119 | standard defined by a recognized standards body, or, in the case of
120 | interfaces specified for a particular programming language, one that
121 | is widely used among developers working in that language.
122 | 
123 |   The "System Libraries" of an executable work include anything, other
124 | than the work as a whole, that (a) is included in the normal form of
125 | packaging a Major Component, but which is not part of that Major
126 | Component, and (b) serves only to enable use of the work with that
127 | Major Component, or to implement a Standard Interface for which an
128 | implementation is available to the public in source code form.  A
129 | "Major Component", in this context, means a major essential component
130 | (kernel, window system, and so on) of the specific operating system
131 | (if any) on which the executable work runs, or a compiler used to
132 | produce the work, or an object code interpreter used to run it.
133 | 
134 |   The "Corresponding Source" for a work in object code form means all
135 | the source code needed to generate, install, and (for an executable
136 | work) run the object code and to modify the work, including scripts to
137 | control those activities.  However, it does not include the work's
138 | System Libraries, or general-purpose tools or generally available free
139 | programs which are used unmodified in performing those activities but
140 | which are not part of the work.  For example, Corresponding Source
141 | includes interface definition files associated with source files for
142 | the work, and the source code for shared libraries and dynamically
143 | linked subprograms that the work is specifically designed to require,
144 | such as by intimate data communication or control flow between those
145 | subprograms and other parts of the work.
146 | 
147 |   The Corresponding Source need not include anything that users
148 | can regenerate automatically from other parts of the Corresponding
149 | Source.
150 | 
151 |   The Corresponding Source for a work in source code form is that
152 | same work.
153 | 
154 |   2. Basic Permissions.
155 | 
156 |   All rights granted under this License are granted for the term of
157 | copyright on the Program, and are irrevocable provided the stated
158 | conditions are met.  This License explicitly affirms your unlimited
159 | permission to run the unmodified Program.  The output from running a
160 | covered work is covered by this License only if the output, given its
161 | content, constitutes a covered work.  This License acknowledges your
162 | rights of fair use or other equivalent, as provided by copyright law.
163 | 
164 |   You may make, run and propagate covered works that you do not
165 | convey, without conditions so long as your license otherwise remains
166 | in force.  You may convey covered works to others for the sole purpose
167 | of having them make modifications exclusively for you, or provide you
168 | with facilities for running those works, provided that you comply with
169 | the terms of this License in conveying all material for which you do
170 | not control copyright.  Those thus making or running the covered works
171 | for you must do so exclusively on your behalf, under your direction
172 | and control, on terms that prohibit them from making any copies of
173 | your copyrighted material outside their relationship with you.
174 | 
175 |   Conveying under any other circumstances is permitted solely under
176 | the conditions stated below.  Sublicensing is not allowed; section 10
177 | makes it unnecessary.
178 | 
179 |   3. Protecting Users' Legal Rights From Anti-Circumvention Law.
180 | 
181 |   No covered work shall be deemed part of an effective technological
182 | measure under any applicable law fulfilling obligations under article
183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or
184 | similar laws prohibiting or restricting circumvention of such
185 | measures.
186 | 
187 |   When you convey a covered work, you waive any legal power to forbid
188 | circumvention of technological measures to the extent such circumvention
189 | is effected by exercising rights under this License with respect to
190 | the covered work, and you disclaim any intention to limit operation or
191 | modification of the work as a means of enforcing, against the work's
192 | users, your or third parties' legal rights to forbid circumvention of
193 | technological measures.
194 | 
195 |   4. Conveying Verbatim Copies.
196 | 
197 |   You may convey verbatim copies of the Program's source code as you
198 | receive it, in any medium, provided that you conspicuously and
199 | appropriately publish on each copy an appropriate copyright notice;
200 | keep intact all notices stating that this License and any
201 | non-permissive terms added in accord with section 7 apply to the code;
202 | keep intact all notices of the absence of any warranty; and give all
203 | recipients a copy of this License along with the Program.
204 | 
205 |   You may charge any price or no price for each copy that you convey,
206 | and you may offer support or warranty protection for a fee.
207 | 
208 |   5. Conveying Modified Source Versions.
209 | 
210 |   You may convey a work based on the Program, or the modifications to
211 | produce it from the Program, in the form of source code under the
212 | terms of section 4, provided that you also meet all of these conditions:
213 | 
214 |     a) The work must carry prominent notices stating that you modified
215 |     it, and giving a relevant date.
216 | 
217 |     b) The work must carry prominent notices stating that it is
218 |     released under this License and any conditions added under section
219 |     7.  This requirement modifies the requirement in section 4 to
220 |     "keep intact all notices".
221 | 
222 |     c) You must license the entire work, as a whole, under this
223 |     License to anyone who comes into possession of a copy.  This
224 |     License will therefore apply, along with any applicable section 7
225 |     additional terms, to the whole of the work, and all its parts,
226 |     regardless of how they are packaged.  This License gives no
227 |     permission to license the work in any other way, but it does not
228 |     invalidate such permission if you have separately received it.
229 | 
230 |     d) If the work has interactive user interfaces, each must display
231 |     Appropriate Legal Notices; however, if the Program has interactive
232 |     interfaces that do not display Appropriate Legal Notices, your
233 |     work need not make them do so.
234 | 
235 |   A compilation of a covered work with other separate and independent
236 | works, which are not by their nature extensions of the covered work,
237 | and which are not combined with it such as to form a larger program,
238 | in or on a volume of a storage or distribution medium, is called an
239 | "aggregate" if the compilation and its resulting copyright are not
240 | used to limit the access or legal rights of the compilation's users
241 | beyond what the individual works permit.  Inclusion of a covered work
242 | in an aggregate does not cause this License to apply to the other
243 | parts of the aggregate.
244 | 
245 |   6. Conveying Non-Source Forms.
246 | 
247 |   You may convey a covered work in object code form under the terms
248 | of sections 4 and 5, provided that you also convey the
249 | machine-readable Corresponding Source under the terms of this License,
250 | in one of these ways:
251 | 
252 |     a) Convey the object code in, or embodied in, a physical product
253 |     (including a physical distribution medium), accompanied by the
254 |     Corresponding Source fixed on a durable physical medium
255 |     customarily used for software interchange.
256 | 
257 |     b) Convey the object code in, or embodied in, a physical product
258 |     (including a physical distribution medium), accompanied by a
259 |     written offer, valid for at least three years and valid for as
260 |     long as you offer spare parts or customer support for that product
261 |     model, to give anyone who possesses the object code either (1) a
262 |     copy of the Corresponding Source for all the software in the
263 |     product that is covered by this License, on a durable physical
264 |     medium customarily used for software interchange, for a price no
265 |     more than your reasonable cost of physically performing this
266 |     conveying of source, or (2) access to copy the
267 |     Corresponding Source from a network server at no charge.
268 | 
269 |     c) Convey individual copies of the object code with a copy of the
270 |     written offer to provide the Corresponding Source.  This
271 |     alternative is allowed only occasionally and noncommercially, and
272 |     only if you received the object code with such an offer, in accord
273 |     with subsection 6b.
274 | 
275 |     d) Convey the object code by offering access from a designated
276 |     place (gratis or for a charge), and offer equivalent access to the
277 |     Corresponding Source in the same way through the same place at no
278 |     further charge.  You need not require recipients to copy the
279 |     Corresponding Source along with the object code.  If the place to
280 |     copy the object code is a network server, the Corresponding Source
281 |     may be on a different server (operated by you or a third party)
282 |     that supports equivalent copying facilities, provided you maintain
283 |     clear directions next to the object code saying where to find the
284 |     Corresponding Source.  Regardless of what server hosts the
285 |     Corresponding Source, you remain obligated to ensure that it is
286 |     available for as long as needed to satisfy these requirements.
287 | 
288 |     e) Convey the object code using peer-to-peer transmission, provided
289 |     you inform other peers where the object code and Corresponding
290 |     Source of the work are being offered to the general public at no
291 |     charge under subsection 6d.
292 | 
293 |   A separable portion of the object code, whose source code is excluded
294 | from the Corresponding Source as a System Library, need not be
295 | included in conveying the object code work.
296 | 
297 |   A "User Product" is either (1) a "consumer product", which means any
298 | tangible personal property which is normally used for personal, family,
299 | or household purposes, or (2) anything designed or sold for incorporation
300 | into a dwelling.  In determining whether a product is a consumer product,
301 | doubtful cases shall be resolved in favor of coverage.  For a particular
302 | product received by a particular user, "normally used" refers to a
303 | typical or common use of that class of product, regardless of the status
304 | of the particular user or of the way in which the particular user
305 | actually uses, or expects or is expected to use, the product.  A product
306 | is a consumer product regardless of whether the product has substantial
307 | commercial, industrial or non-consumer uses, unless such uses represent
308 | the only significant mode of use of the product.
309 | 
310 |   "Installation Information" for a User Product means any methods,
311 | procedures, authorization keys, or other information required to install
312 | and execute modified versions of a covered work in that User Product from
313 | a modified version of its Corresponding Source.  The information must
314 | suffice to ensure that the continued functioning of the modified object
315 | code is in no case prevented or interfered with solely because
316 | modification has been made.
317 | 
318 |   If you convey an object code work under this section in, or with, or
319 | specifically for use in, a User Product, and the conveying occurs as
320 | part of a transaction in which the right of possession and use of the
321 | User Product is transferred to the recipient in perpetuity or for a
322 | fixed term (regardless of how the transaction is characterized), the
323 | Corresponding Source conveyed under this section must be accompanied
324 | by the Installation Information.  But this requirement does not apply
325 | if neither you nor any third party retains the ability to install
326 | modified object code on the User Product (for example, the work has
327 | been installed in ROM).
328 | 
329 |   The requirement to provide Installation Information does not include a
330 | requirement to continue to provide support service, warranty, or updates
331 | for a work that has been modified or installed by the recipient, or for
332 | the User Product in which it has been modified or installed.  Access to a
333 | network may be denied when the modification itself materially and
334 | adversely affects the operation of the network or violates the rules and
335 | protocols for communication across the network.
336 | 
337 |   Corresponding Source conveyed, and Installation Information provided,
338 | in accord with this section must be in a format that is publicly
339 | documented (and with an implementation available to the public in
340 | source code form), and must require no special password or key for
341 | unpacking, reading or copying.
342 | 
343 |   7. Additional Terms.
344 | 
345 |   "Additional permissions" are terms that supplement the terms of this
346 | License by making exceptions from one or more of its conditions.
347 | Additional permissions that are applicable to the entire Program shall
348 | be treated as though they were included in this License, to the extent
349 | that they are valid under applicable law.  If additional permissions
350 | apply only to part of the Program, that part may be used separately
351 | under those permissions, but the entire Program remains governed by
352 | this License without regard to the additional permissions.
353 | 
354 |   When you convey a copy of a covered work, you may at your option
355 | remove any additional permissions from that copy, or from any part of
356 | it.  (Additional permissions may be written to require their own
357 | removal in certain cases when you modify the work.)  You may place
358 | additional permissions on material, added by you to a covered work,
359 | for which you have or can give appropriate copyright permission.
360 | 
361 |   Notwithstanding any other provision of this License, for material you
362 | add to a covered work, you may (if authorized by the copyright holders of
363 | that material) supplement the terms of this License with terms:
364 | 
365 |     a) Disclaiming warranty or limiting liability differently from the
366 |     terms of sections 15 and 16 of this License; or
367 | 
368 |     b) Requiring preservation of specified reasonable legal notices or
369 |     author attributions in that material or in the Appropriate Legal
370 |     Notices displayed by works containing it; or
371 | 
372 |     c) Prohibiting misrepresentation of the origin of that material, or
373 |     requiring that modified versions of such material be marked in
374 |     reasonable ways as different from the original version; or
375 | 
376 |     d) Limiting the use for publicity purposes of names of licensors or
377 |     authors of the material; or
378 | 
379 |     e) Declining to grant rights under trademark law for use of some
380 |     trade names, trademarks, or service marks; or
381 | 
382 |     f) Requiring indemnification of licensors and authors of that
383 |     material by anyone who conveys the material (or modified versions of
384 |     it) with contractual assumptions of liability to the recipient, for
385 |     any liability that these contractual assumptions directly impose on
386 |     those licensors and authors.
387 | 
388 |   All other non-permissive additional terms are considered "further
389 | restrictions" within the meaning of section 10.  If the Program as you
390 | received it, or any part of it, contains a notice stating that it is
391 | governed by this License along with a term that is a further
392 | restriction, you may remove that term.  If a license document contains
393 | a further restriction but permits relicensing or conveying under this
394 | License, you may add to a covered work material governed by the terms
395 | of that license document, provided that the further restriction does
396 | not survive such relicensing or conveying.
397 | 
398 |   If you add terms to a covered work in accord with this section, you
399 | must place, in the relevant source files, a statement of the
400 | additional terms that apply to those files, or a notice indicating
401 | where to find the applicable terms.
402 | 
403 |   Additional terms, permissive or non-permissive, may be stated in the
404 | form of a separately written license, or stated as exceptions;
405 | the above requirements apply either way.
406 | 
407 |   8. Termination.
408 | 
409 |   You may not propagate or modify a covered work except as expressly
410 | provided under this License.  Any attempt otherwise to propagate or
411 | modify it is void, and will automatically terminate your rights under
412 | this License (including any patent licenses granted under the third
413 | paragraph of section 11).
414 | 
415 |   However, if you cease all violation of this License, then your
416 | license from a particular copyright holder is reinstated (a)
417 | provisionally, unless and until the copyright holder explicitly and
418 | finally terminates your license, and (b) permanently, if the copyright
419 | holder fails to notify you of the violation by some reasonable means
420 | prior to 60 days after the cessation.
421 | 
422 |   Moreover, your license from a particular copyright holder is
423 | reinstated permanently if the copyright holder notifies you of the
424 | violation by some reasonable means, this is the first time you have
425 | received notice of violation of this License (for any work) from that
426 | copyright holder, and you cure the violation prior to 30 days after
427 | your receipt of the notice.
428 | 
429 |   Termination of your rights under this section does not terminate the
430 | licenses of parties who have received copies or rights from you under
431 | this License.  If your rights have been terminated and not permanently
432 | reinstated, you do not qualify to receive new licenses for the same
433 | material under section 10.
434 | 
435 |   9. Acceptance Not Required for Having Copies.
436 | 
437 |   You are not required to accept this License in order to receive or
438 | run a copy of the Program.  Ancillary propagation of a covered work
439 | occurring solely as a consequence of using peer-to-peer transmission
440 | to receive a copy likewise does not require acceptance.  However,
441 | nothing other than this License grants you permission to propagate or
442 | modify any covered work.  These actions infringe copyright if you do
443 | not accept this License.  Therefore, by modifying or propagating a
444 | covered work, you indicate your acceptance of this License to do so.
445 | 
446 |   10. Automatic Licensing of Downstream Recipients.
447 | 
448 |   Each time you convey a covered work, the recipient automatically
449 | receives a license from the original licensors, to run, modify and
450 | propagate that work, subject to this License.  You are not responsible
451 | for enforcing compliance by third parties with this License.
452 | 
453 |   An "entity transaction" is a transaction transferring control of an
454 | organization, or substantially all assets of one, or subdividing an
455 | organization, or merging organizations.  If propagation of a covered
456 | work results from an entity transaction, each party to that
457 | transaction who receives a copy of the work also receives whatever
458 | licenses to the work the party's predecessor in interest had or could
459 | give under the previous paragraph, plus a right to possession of the
460 | Corresponding Source of the work from the predecessor in interest, if
461 | the predecessor has it or can get it with reasonable efforts.
462 | 
463 |   You may not impose any further restrictions on the exercise of the
464 | rights granted or affirmed under this License.  For example, you may
465 | not impose a license fee, royalty, or other charge for exercise of
466 | rights granted under this License, and you may not initiate litigation
467 | (including a cross-claim or counterclaim in a lawsuit) alleging that
468 | any patent claim is infringed by making, using, selling, offering for
469 | sale, or importing the Program or any portion of it.
470 | 
471 |   11. Patents.
472 | 
473 |   A "contributor" is a copyright holder who authorizes use under this
474 | License of the Program or a work on which the Program is based.  The
475 | work thus licensed is called the contributor's "contributor version".
476 | 
477 |   A contributor's "essential patent claims" are all patent claims
478 | owned or controlled by the contributor, whether already acquired or
479 | hereafter acquired, that would be infringed by some manner, permitted
480 | by this License, of making, using, or selling its contributor version,
481 | but do not include claims that would be infringed only as a
482 | consequence of further modification of the contributor version.  For
483 | purposes of this definition, "control" includes the right to grant
484 | patent sublicenses in a manner consistent with the requirements of
485 | this License.
486 | 
487 |   Each contributor grants you a non-exclusive, worldwide, royalty-free
488 | patent license under the contributor's essential patent claims, to
489 | make, use, sell, offer for sale, import and otherwise run, modify and
490 | propagate the contents of its contributor version.
491 | 
492 |   In the following three paragraphs, a "patent license" is any express
493 | agreement or commitment, however denominated, not to enforce a patent
494 | (such as an express permission to practice a patent or covenant not to
495 | sue for patent infringement).  To "grant" such a patent license to a
496 | party means to make such an agreement or commitment not to enforce a
497 | patent against the party.
498 | 
499 |   If you convey a covered work, knowingly relying on a patent license,
500 | and the Corresponding Source of the work is not available for anyone
501 | to copy, free of charge and under the terms of this License, through a
502 | publicly available network server or other readily accessible means,
503 | then you must either (1) cause the Corresponding Source to be so
504 | available, or (2) arrange to deprive yourself of the benefit of the
505 | patent license for this particular work, or (3) arrange, in a manner
506 | consistent with the requirements of this License, to extend the patent
507 | license to downstream recipients.  "Knowingly relying" means you have
508 | actual knowledge that, but for the patent license, your conveying the
509 | covered work in a country, or your recipient's use of the covered work
510 | in a country, would infringe one or more identifiable patents in that
511 | country that you have reason to believe are valid.
512 | 
513 |   If, pursuant to or in connection with a single transaction or
514 | arrangement, you convey, or propagate by procuring conveyance of, a
515 | covered work, and grant a patent license to some of the parties
516 | receiving the covered work authorizing them to use, propagate, modify
517 | or convey a specific copy of the covered work, then the patent license
518 | you grant is automatically extended to all recipients of the covered
519 | work and works based on it.
520 | 
521 |   A patent license is "discriminatory" if it does not include within
522 | the scope of its coverage, prohibits the exercise of, or is
523 | conditioned on the non-exercise of one or more of the rights that are
524 | specifically granted under this License.  You may not convey a covered
525 | work if you are a party to an arrangement with a third party that is
526 | in the business of distributing software, under which you make payment
527 | to the third party based on the extent of your activity of conveying
528 | the work, and under which the third party grants, to any of the
529 | parties who would receive the covered work from you, a discriminatory
530 | patent license (a) in connection with copies of the covered work
531 | conveyed by you (or copies made from those copies), or (b) primarily
532 | for and in connection with specific products or compilations that
533 | contain the covered work, unless you entered into that arrangement,
534 | or that patent license was granted, prior to 28 March 2007.
535 | 
536 |   Nothing in this License shall be construed as excluding or limiting
537 | any implied license or other defenses to infringement that may
538 | otherwise be available to you under applicable patent law.
539 | 
540 |   12. No Surrender of Others' Freedom.
541 | 
542 |   If conditions are imposed on you (whether by court order, agreement or
543 | otherwise) that contradict the conditions of this License, they do not
544 | excuse you from the conditions of this License.  If you cannot convey a
545 | covered work so as to satisfy simultaneously your obligations under this
546 | License and any other pertinent obligations, then as a consequence you may
547 | not convey it at all.  For example, if you agree to terms that obligate you
548 | to collect a royalty for further conveying from those to whom you convey
549 | the Program, the only way you could satisfy both those terms and this
550 | License would be to refrain entirely from conveying the Program.
551 | 
552 |   13. Use with the GNU Affero General Public License.
553 | 
554 |   Notwithstanding any other provision of this License, you have
555 | permission to link or combine any covered work with a work licensed
556 | under version 3 of the GNU Affero General Public License into a single
557 | combined work, and to convey the resulting work.  The terms of this
558 | License will continue to apply to the part which is the covered work,
559 | but the special requirements of the GNU Affero General Public License,
560 | section 13, concerning interaction through a network will apply to the
561 | combination as such.
562 | 
563 |   14. Revised Versions of this License.
564 | 
565 |   The Free Software Foundation may publish revised and/or new versions of
566 | the GNU General Public License from time to time.  Such new versions will
567 | be similar in spirit to the present version, but may differ in detail to
568 | address new problems or concerns.
569 | 
570 |   Each version is given a distinguishing version number.  If the
571 | Program specifies that a certain numbered version of the GNU General
572 | Public License "or any later version" applies to it, you have the
573 | option of following the terms and conditions either of that numbered
574 | version or of any later version published by the Free Software
575 | Foundation.  If the Program does not specify a version number of the
576 | GNU General Public License, you may choose any version ever published
577 | by the Free Software Foundation.
578 | 
579 |   If the Program specifies that a proxy can decide which future
580 | versions of the GNU General Public License can be used, that proxy's
581 | public statement of acceptance of a version permanently authorizes you
582 | to choose that version for the Program.
583 | 
584 |   Later license versions may give you additional or different
585 | permissions.  However, no additional obligations are imposed on any
586 | author or copyright holder as a result of your choosing to follow a
587 | later version.
588 | 
589 |   15. Disclaimer of Warranty.
590 | 
591 |   THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
592 | APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
596 | PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
597 | IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
599 | 
600 |   16. Limitation of Liability.
601 | 
602 |   IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
610 | SUCH DAMAGES.
611 | 
612 |   17. Interpretation of Sections 15 and 16.
613 | 
614 |   If the disclaimer of warranty and limitation of liability provided
615 | above cannot be given local legal effect according to their terms,
616 | reviewing courts shall apply local law that most closely approximates
617 | an absolute waiver of all civil liability in connection with the
618 | Program, unless a warranty or assumption of liability accompanies a
619 | copy of the Program in return for a fee.
620 | 
621 |                      END OF TERMS AND CONDITIONS
622 | 
623 |             How to Apply These Terms to Your New Programs
624 | 
625 |   If you develop a new program, and you want it to be of the greatest
626 | possible use to the public, the best way to achieve this is to make it
627 | free software which everyone can redistribute and change under these terms.
628 | 
629 |   To do so, attach the following notices to the program.  It is safest
630 | to attach them to the start of each source file to most effectively
631 | state the exclusion of warranty; and each file should have at least
632 | the "copyright" line and a pointer to where the full notice is found.
633 | 
634 |     <one line to give the program's name and a brief idea of what it does.>
635 |     Copyright (C) <year>  <name of author>
636 | 
637 |     This program is free software: you can redistribute it and/or modify
638 |     it under the terms of the GNU General Public License as published by
639 |     the Free Software Foundation, either version 3 of the License, or
640 |     (at your option) any later version.
641 | 
642 |     This program is distributed in the hope that it will be useful,
643 |     but WITHOUT ANY WARRANTY; without even the implied warranty of
644 |     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
645 |     GNU General Public License for more details.
646 | 
647 |     You should have received a copy of the GNU General Public License
648 |     along with this program.  If not, see <http://www.gnu.org/licenses/>.
649 | 
650 | Also add information on how to contact you by electronic and paper mail.
651 | 
652 |   If the program does terminal interaction, make it output a short
653 | notice like this when it starts in an interactive mode:
654 | 
655 |     <program>  Copyright (C) <year>  <name of author>
656 |     This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657 |     This is free software, and you are welcome to redistribute it
658 |     under certain conditions; type `show c' for details.
659 | 
660 | The hypothetical commands `show w' and `show c' should show the appropriate
661 | parts of the General Public License.  Of course, your program's commands
662 | might be different; for a GUI interface, you would use an "about box".
663 | 
664 |   You should also get your employer (if you work as a programmer) or school,
665 | if any, to sign a "copyright disclaimer" for the program, if necessary.
666 | For more information on this, and how to apply and follow the GNU GPL, see
667 | <http://www.gnu.org/licenses/>.
668 | 
669 |   The GNU General Public License does not permit incorporating your program
670 | into proprietary programs.  If your program is a subroutine library, you
671 | may consider it more useful to permit linking proprietary applications with
672 | the library.  If this is what you want to do, use the GNU Lesser General
673 | Public License instead of this License.  But first, please read
674 | <http://www.gnu.org/philosophy/why-not-lgpl.html>.
675 | 


--------------------------------------------------------------------------------
/testofx.py:
--------------------------------------------------------------------------------
   1 | #!/usr/bin/env python
   2 | 
   3 | """
   4 | OFX Test Client
   5 | """
   6 | 
   7 | import json
   8 | import re
   9 | import requests
  10 | import time
  11 | from urllib.parse import urlparse
  12 | from uuid import uuid4
  13 | from xmltodict import parse as xmlparse
  14 | 
  15 | #
  16 | # Defines
  17 | #
  18 | USER_AGENT   = 'InetClntApp/3.0'
  19 | CONTENT_TYPE = 'application/x-ofx'
  20 | 
  21 | HDR_OFXHEADER   = 'OFXHEADER'
  22 | HDR_DATA        = 'DATA'
  23 | HDR_VERSION     = 'VERSION'
  24 | HDR_SECURITY    = 'SECURITY'
  25 | HDR_ENCODING    = 'ENCODING'
  26 | HDR_CHARSET     = 'CHARSET'
  27 | HDR_COMPRESSION = 'COMPRESSION'
  28 | HDR_OLDFILEUID  = 'OLDFILEUID'
  29 | HDR_NEWFILEUID  = 'NEWFILEUID'
  30 | 
  31 | HDR_FIELDS_V1 = [HDR_OFXHEADER, HDR_DATA, HDR_VERSION, HDR_SECURITY,
  32 |         HDR_ENCODING, HDR_CHARSET, HDR_COMPRESSION, HDR_OLDFILEUID,
  33 |         HDR_NEWFILEUID]
  34 | 
  35 | HDR_FIELDS_V2 = [HDR_OFXHEADER, HDR_VERSION, HDR_SECURITY, HDR_OLDFILEUID,
  36 |         HDR_NEWFILEUID]
  37 | 
  38 | OFX_HEADER_100 = \
  39 | '''OFXHEADER:100
  40 | DATA:OFXSGML
  41 | VERSION:{version}
  42 | SECURITY:NONE
  43 | ENCODING:USASCII
  44 | CHARSET:1252
  45 | COMPRESSION:NONE
  46 | OLDFILEUID:NONE
  47 | NEWFILEUID:NONE
  48 | '''
  49 | 
  50 | OFX_HEADER_200 = \
  51 | '''<?xml version="1.0" encoding="UTF-8" standalone="no"?>
  52 | <?OFX OFXHEADER="200" VERSION="{version}" SECURITY="NONE" OLDFILEUID="NONE"
  53 | NEWFILEUID="NONE"?>
  54 | '''
  55 | 
  56 | REQ_NAME_GET_ROOT     = 'GET /'
  57 | REQ_NAME_GET_OFX      = 'GET OFX Path'
  58 | REQ_NAME_POST_OFX     = 'POST OFX Path'
  59 | REQ_NAME_OFX_EMPTY    = 'OFX Empty'
  60 | REQ_NAME_OFX_PROFILE  = 'OFX PROFILE'
  61 | REQ_NAME_OFX_ACCTINFO = 'OFX ACCTINFO'
  62 | 
  63 | REQ_NAMES = [
  64 |     REQ_NAME_GET_ROOT,
  65 |     REQ_NAME_GET_OFX,
  66 |     REQ_NAME_POST_OFX,
  67 |     REQ_NAME_OFX_EMPTY,
  68 |     REQ_NAME_OFX_PROFILE,
  69 |     REQ_NAME_OFX_ACCTINFO
  70 | ]
  71 | 
  72 | REQ_METHODS = {
  73 |     REQ_NAME_GET_ROOT:     'GET',
  74 |     REQ_NAME_GET_OFX:      'GET',
  75 |     REQ_NAME_POST_OFX:     'POST',
  76 |     REQ_NAME_OFX_EMPTY:    'POST',
  77 |     REQ_NAME_OFX_PROFILE:  'POST',
  78 |     REQ_NAME_OFX_ACCTINFO: 'POST'
  79 | }
  80 | 
  81 | #
  82 | # Helper Functions
  83 | #
  84 | 
  85 | def print_http_response(res):
  86 |     print("===Request Headers===")
  87 |     print(dict(res.request.headers))
  88 |     print("===Request Body===")
  89 |     print(res.request.body)
  90 |     print("=== Response Status ===")
  91 |     print(res.status_code)
  92 |     print("=== Response Headers ===")
  93 |     print(dict(res.headers))
  94 |     print("=== Response Body ===")
  95 |     print(res.text)
  96 | 
  97 | #
  98 | # Public Functions
  99 | #
 100 | 
 101 | def dt_now():
 102 |     # Example: 20170616141327.123[-7:MST]
 103 |     return time.strftime("%Y%m%d%H%M%S.123[-7:MST]", time.localtime())
 104 | 
 105 | 
 106 | def uid():
 107 |     # Example: C1B7C870-7CB2-1000-BD91-E1E23E560026
 108 |     return str(uuid4()).upper()
 109 | 
 110 | 
 111 | def is_ofx_response(resp_body):
 112 |     ret = False
 113 | 
 114 |     # Version 1 Header
 115 |     if resp_body.startswith('OFXHEADER'):
 116 |         ret = True
 117 | 
 118 |     # Version 2 Header
 119 |     if resp_body.find('<?OFX OFXHEADER') != -1:
 120 |         ret = True
 121 | 
 122 |     return ret
 123 | 
 124 | 
 125 | class OFXServerInstance():
 126 |     '''
 127 |     Representation of an OFX server
 128 |     '''
 129 | 
 130 |     httpserver = ''
 131 |     webframework = ''
 132 |     software = dict([
 133 |         ('Company', ''),
 134 |         ('Product', ''),
 135 |         ('Version', '')
 136 |         ])
 137 |     serviceprovider = ''
 138 |     tls = dict()
 139 | 
 140 |     def __init__(self, ofxurl, fid, org):
 141 |         self.ofxurl = ofxurl
 142 |         self.fid = fid if fid else ''
 143 |         self.org = org if org else ''
 144 | 
 145 |     def _extract_http_header(self, res, header, field, exclude, nooverwrite):
 146 |         '''
 147 |         Find and store header in HTTP response.
 148 | 
 149 |         nooverwrite: list[str] - set if nothing, but don't change exsiting
 150 |                                     entry with these values
 151 |         '''
 152 |         val = None
 153 |         try:
 154 |             val = res.headers[header]
 155 |         except KeyError:
 156 |             return
 157 | 
 158 |         # Skip if header value is in exclusion list
 159 |         if val in exclude:
 160 |             return
 161 | 
 162 |         cur_val = getattr(self, field)
 163 |         # Skip if we've already recorded the same header value
 164 |         if cur_val == val:
 165 |             return
 166 | 
 167 |         # Store the header value in the ServerInstance
 168 |         if cur_val == '':
 169 |             setattr(self, field, val)
 170 |         else:
 171 |             if val in nooverwrite:
 172 |                 return
 173 |             else:
 174 |                 setattr(self, field, val)
 175 | 
 176 |     def _fingerprint_httpserver(self, req_results):
 177 | 
 178 |         def _check_resp_body(res):
 179 |             html = res.text
 180 |             if html:
 181 |                 # Quick and dirty regex of <title> tag
 182 |                 prog = re.compile('<title>(.*)</title>')
 183 |                 match = prog.search(html)
 184 |                 if match:
 185 |                     title = match.group(1)
 186 |                     if title == 'IIS Windows Server':
 187 |                         if self.httpserver == '':
 188 |                             self.httpserver = 'Microsoft-IIS/8.5'
 189 |                     elif title == 'APACHE OFX APP':
 190 |                         if self.httpserver == '':
 191 |                             self.httpserver = 'Apache/2.2.23'
 192 |                     elif title == 'IBM HTTP Server 8.5':
 193 |                         if self.httpserver == '':
 194 |                             self.httpserver = 'IBM HTTP Server/8.5'
 195 |                     elif title.startswith('Apache Tomcat/'):
 196 |                         if self.httpserver in ['', 'Apache', 'Apache-Coyote/1.1']:
 197 |                             # Removing trailing " - Error Report"
 198 |                             self.httpserver = title[0:-15]
 199 |                     elif title.startswith('VMware vFabric tc Runtime'):
 200 |                         if self.httpserver == '':
 201 |                             # Removing trailing " - Error Report"
 202 |                             self.httpserver = title[0:-15]
 203 |                     elif title.startswith('JBoss'):
 204 |                         if self.httpserver in ['', 'Apache', 'Apache-Coyote/1.1']:
 205 |                             # Removing trailing " - Error Report"
 206 |                             self.httpserver = title[0:-15]
 207 |                     elif title.startswith('JBWEB'):
 208 |                         if self.httpserver in ['', 'Apache', 'Apache-Coyote/1.1']:
 209 |                             self.httpserver = 'JBoss'
 210 | 
 211 |         # Extract OFX "Server" header from OFX requests
 212 |         # The HTTP server on the root of the path can be different
 213 |         exclude = ['', 'not_available', 'Unspecified']
 214 |         nooverwrite = [
 215 |                 'Apache-Coyote/1.1',
 216 |                 'Apache',
 217 |                 'USAA-Service',
 218 |                 'USAA-Integrity'
 219 |                 ]
 220 | 
 221 |         for req_name in [
 222 |                 REQ_NAME_OFX_PROFILE,
 223 |                 REQ_NAME_OFX_EMPTY,
 224 |                 REQ_NAME_POST_OFX
 225 |                 ]:
 226 |             res = req_results[req_name]
 227 |             self._extract_http_header(
 228 |                     res,
 229 |                     'Server',
 230 |                     'httpserver',
 231 |                     exclude,
 232 |                     nooverwrite)
 233 | 
 234 |         # Extract Server out of error body
 235 |         for req_name in [
 236 |                 REQ_NAME_POST_OFX,
 237 |                 REQ_NAME_GET_OFX,
 238 |                 REQ_NAME_GET_ROOT
 239 |                 ]:
 240 | 
 241 |             res = req_results[req_name]
 242 |             _check_resp_body(res)
 243 | 
 244 |     def _fingerprint_webframework(self, req_results):
 245 | 
 246 |         def _check_resp_body(res):
 247 |             body = res.text
 248 |             if body:
 249 |                 first_line = body.splitlines()[0]
 250 |                 if first_line.startswith('Error 404: SRVE0190E:'):
 251 |                     if self.webframework == '':
 252 |                         self.webframework = 'WebSphere'
 253 | 
 254 |         # Extract Web Framework from successful OFX requests
 255 |         # The web framework on the root of the path can be different
 256 |         exclude = ['DI - An Intuit Company']
 257 |         for req_name in [
 258 |                 REQ_NAME_OFX_PROFILE
 259 |                 ]:
 260 |             res = req_results[req_name]
 261 |             self._extract_http_header(
 262 |                     res,
 263 |                     'X-Powered-By',
 264 |                     'webframework',
 265 |                     exclude,
 266 |                     [])
 267 | 
 268 |             if self.webframework == 'ASP.NET':
 269 |                 # Check for extra ASP.NET version headers
 270 |                 val = None
 271 |                 try:
 272 |                     val = res.headers['X-AspNet-Version']
 273 |                 except KeyError:
 274 |                     continue
 275 |                 self.webframework += '/{}'.format(val)
 276 | 
 277 |         # Extract framework out of error body
 278 |         for req_name in [
 279 |                 REQ_NAME_GET_ROOT
 280 |                 ]:
 281 | 
 282 |             res = req_results[req_name]
 283 |             _check_resp_body(res)
 284 | 
 285 |     def _fingerprint_software(self, req_results):
 286 | 
 287 |         # Determine software based off URL path
 288 |         # URL Path: Company, Product
 289 |         path_map = {
 290 |             '/cmr/cmr.ofx': ('Enterprise Engineering','EnterpriseFTX'),
 291 |             '/ofx/servlet/Teller': ('Finastra','Cavion'),
 292 |             '/ofx/OFXServlet': ('FIS','Metavante'),
 293 |             '/piles/ofx.pile/': ('First Data Corporation','FundsXPress'),
 294 |             '/scripts/serverext.dll': ('Fiserv', 'Corillian'),
 295 |             '/OROFX16Listener': ('Fiserv',''),
 296 |             '/ofx/process.ofx': ('Fiserv','Corillian'),
 297 |             '/eftxweb/access.ofx': ('Enterprise Engineering','EnterpriseFTX',),
 298 |             '/scripts/isaofx.dll': ('Fiserv', ''),
 299 |             '/scripts/serverext.dll': ('Fiserv','Corillian'),
 300 |             '/ofx/ofx.dll': ('ULTRADATA Corporation', ''),
 301 |             '/ofxserver/ofxsrvr.dll': ('Access Softek', 'OFXServer'),
 302 |             '/OFXServer/ofxsrvr.dll': ('Access Softek', 'OFXServer'),
 303 |         }
 304 | 
 305 |         parsed = urlparse(self.ofxurl)
 306 | 
 307 |         try:
 308 |             row = path_map[parsed.path]
 309 |             self.software['Company'] = row[0]
 310 |             self.software['Product'] = row[1]
 311 |         except KeyError:
 312 |             pass
 313 | 
 314 |         # Fingerprint version number
 315 |         if self.software['Company'] == 'Enterprise Engineering':
 316 |             # EFTX has a splash page with version number by default
 317 |             rpat = r'Servlet Version ([0-9.]+)'
 318 |             match = re.search(rpat, req_results[REQ_NAME_GET_OFX].text)
 319 |             if match:
 320 |                 self.software['Version'] = 'Servlet {}'.format(match.group(1))
 321 | 
 322 |     def _fingerprint_service_provider(self, req_results):
 323 | 
 324 |         # Determine which service provider (if any) is hosting this instance
 325 |         domain_map = {
 326 |             'ofx.netxclient.com': 'Pershing',
 327 |             'uat-ofx.netxclient.inautix.com': 'Pershing',
 328 |             'www.oasis.cfree.com': 'Fiserv - CheckFree - Oasis'
 329 |         }
 330 | 
 331 |         sp = ''
 332 | 
 333 |         parsed = urlparse(self.ofxurl)
 334 | 
 335 |         try:
 336 |             sp = domain_map[parsed.netloc]
 337 |         except KeyError:
 338 |             pass
 339 | 
 340 |         # Try the domain map first, fallback to SPNAME
 341 |         if sp == '':
 342 |             profrs = OFXFile(req_results[REQ_NAME_OFX_PROFILE].text)
 343 | 
 344 |             try:
 345 |                 sp = profrs.profile['SPNAME']
 346 |             except KeyError: pass
 347 | 
 348 |         self.serviceprovider = sp
 349 | 
 350 | 
 351 |     def fingerprint(self, req_results):
 352 |         '''
 353 |         Determine software and web frameworks running on instance.
 354 |         '''
 355 |         self._fingerprint_httpserver(req_results)
 356 |         self._fingerprint_webframework(req_results)
 357 |         self._fingerprint_software(req_results)
 358 |         self._fingerprint_service_provider(req_results)
 359 | 
 360 |     def get_tls(self):
 361 |         try:
 362 |             return self.tls['working']
 363 |         except KeyError:
 364 |             return True
 365 | 
 366 |     def set_tls(self, working):
 367 |         self.tls['working'] = working
 368 | 
 369 | 
 370 | class OFXTestClient():
 371 | 
 372 |     _payload_func = {}
 373 | 
 374 |     # Whether to print to stdout
 375 |     _output = True
 376 | 
 377 |     cache = {}
 378 | 
 379 |     def __init__(self,
 380 |             timeout=(3.2, 27),
 381 |             wait=0,
 382 |             use_cache=False,
 383 |             output=False,
 384 |             version='102',
 385 |             tls_verify=True
 386 |             ):
 387 |         self.timeout = timeout
 388 |         self.wait = wait
 389 |         self.use_cache = use_cache
 390 |         self._output=output
 391 |         self.version = version
 392 |         self.tls_verify = tls_verify
 393 | 
 394 |         if self.version[0] == '1':
 395 |             self.ofxheader = OFX_HEADER_100.format(version=self.version)
 396 |             self.content_type = 'text/sgml'
 397 |         elif self.version[0] == '2':
 398 |             self.ofxheader = OFX_HEADER_200.format(version=self.version)
 399 |             self.content_type = 'text/xml'
 400 |         else:
 401 |             raise ValueError(
 402 |                     'Unknown OFX version number {}'.format(self.version))
 403 | 
 404 |     def call_url_cached(self, url, tls_verify, body, method):
 405 |         '''
 406 |         return (request.response, boolean) - Response and whether it was
 407 |                 cached.
 408 |         '''
 409 | 
 410 |         if method not in ['GET', 'POST']:
 411 |             raise ValueError("Method must be 'GET' or 'POST'")
 412 | 
 413 |         # Impersonate PFM
 414 |         headers = {
 415 |                 'User-Agent': USER_AGENT,
 416 |                 }
 417 | 
 418 |         if method == 'POST':
 419 |             headers['Content-Type'] = CONTENT_TYPE
 420 | 
 421 |         # Simple in memory cache to avoid duplicate calls to the same URL.
 422 |         try:
 423 |             r = self.cache[url]
 424 |             return (r, True)
 425 |         except KeyError:
 426 |             pass
 427 | 
 428 |         if self._output: print("{}".format(url))
 429 |         try:
 430 |             if method == 'GET':
 431 |                 r = requests.get(
 432 |                         url,
 433 |                         headers=headers,
 434 |                         timeout=self.timeout,
 435 |                         verify=tls_verify
 436 |                         )
 437 |             elif method == 'POST':
 438 |                 r = requests.post(
 439 |                         url,
 440 |                         headers=headers,
 441 |                         timeout=self.timeout,
 442 |                         verify=tls_verify,
 443 |                         data=body
 444 |                         )
 445 |             if self.use_cache:
 446 |                 self.cache[url] = r
 447 |             return (r, False)
 448 |         except requests.ConnectionError as ex:
 449 |             if self._output: print('\tConnectionError: {}'.format(ex))
 450 |             # Set cache, but empty, to avoid further calls this run
 451 |             # Still cache connection errors even if use_cache == False
 452 |             self.cache[url] = None
 453 |         except requests.exceptions.ReadTimeout as ex:
 454 |             if self._output: print('\tConnectionError: {}'.format(ex))
 455 |             if wait > 0:
 456 |                 if self._output:
 457 |                     print('\tWaiting for {} seconds'.format(self.wait))
 458 |                 time.sleep(self.wait)
 459 | 
 460 |         return (None, False)
 461 | 
 462 |     def call_url_interactive(self, ofxurl, tls_verify, payload, method):
 463 |         res, was_cached = self.call_url_cached(
 464 |                 ofxurl,
 465 |                 tls_verify,
 466 |                 payload,
 467 |                 method
 468 |                 )
 469 | 
 470 |         # Connection was completed successfully
 471 |         if res is not None:
 472 |             print_http_response(res)
 473 | 
 474 |     def send_req(self, req_name, si):
 475 |         '''
 476 |         Send a pre-defined request to the OFX server.
 477 |         '''
 478 | 
 479 |         res = None
 480 | 
 481 |         if req_name == REQ_NAME_GET_ROOT:
 482 |             parsed = urlparse(si.ofxurl)
 483 |             url = parsed.scheme + '://' + parsed.netloc
 484 |             res, was_cached = self.call_url_cached(
 485 |                     url,
 486 |                     self.tls_verify,
 487 |                     self.get_empty_payload(si),
 488 |                     REQ_METHODS[req_name]
 489 |                     )
 490 |         elif req_name == REQ_NAME_GET_OFX:
 491 |             res, was_cached = self.call_url_cached(
 492 |                     si.ofxurl,
 493 |                     self.tls_verify,
 494 |                     self.get_empty_payload(si),
 495 |                     REQ_METHODS[req_name]
 496 |                     )
 497 |         elif req_name == REQ_NAME_POST_OFX:
 498 |             res, was_cached = self.call_url_cached(
 499 |                     si.ofxurl,
 500 |                     self.tls_verify,
 501 |                     self.get_empty_payload(si),
 502 |                     REQ_METHODS[req_name]
 503 |                     )
 504 |         elif req_name == REQ_NAME_OFX_EMPTY:
 505 |             res, was_cached = self.call_url_cached(
 506 |                     si.ofxurl,
 507 |                     self.tls_verify,
 508 |                     self.get_ofx_empty_payload(si),
 509 |                     REQ_METHODS[req_name]
 510 |                     )
 511 |         elif req_name == REQ_NAME_OFX_PROFILE:
 512 |             res, was_cached = self.call_url_cached(
 513 |                     si.ofxurl,
 514 |                     self.tls_verify,
 515 |                     self.get_profile_payload(si),
 516 |                     REQ_METHODS[req_name]
 517 |                     )
 518 |         else:
 519 |             raise ValueError('Unknown request name: {}'.format(req_name))
 520 | 
 521 |         return res
 522 | 
 523 |     def _get_signonmsg_anonymous_payload(self, si):
 524 | 
 525 |         if self.content_type == 'text/sgml':
 526 |             ofx_fi_fmt =  \
 527 | '''<FI>
 528 | <ORG>{org}
 529 | <FID>{fid}
 530 | </FI>
 531 | '''
 532 | 
 533 |             ofx_signon_fmt = \
 534 | '''<SIGNONMSGSRQV1>
 535 | <SONRQ>
 536 | <DTCLIENT>{dt}
 537 | <USERID>anonymous00000000000000000000000
 538 | <USERPASS>anonymous00000000000000000000000
 539 | <GENUSERKEY>N
 540 | <LANGUAGE>ENG
 541 | {fi}<APPID>QWIN
 542 | <APPVER>2700
 543 | </SONRQ>
 544 | </SIGNONMSGSRQV1>'''
 545 | 
 546 |         elif self.content_type == 'text/xml':
 547 |             ofx_fi_fmt =  \
 548 | '''<FI>
 549 | <ORG>{org}</ORG>
 550 | <FID>{fid}</FID>
 551 | </FI>
 552 | '''
 553 | 
 554 |             ofx_signon_fmt = \
 555 | '''<SIGNONMSGSRQV1>
 556 | <SONRQ>
 557 | <DTCLIENT>{dt}</DTCLIENT>
 558 | <USERID>anonymous00000000000000000000000</USERID>
 559 | <USERPASS>anonymous00000000000000000000000</USERPASS>
 560 | <GENUSERKEY>N</GENUSERKEY>
 561 | <LANGUAGE>ENG</LANGUAGE>
 562 | {fi}<APPID>QWIN</APPID>
 563 | <APPVER>2700</APPVER>
 564 | </SONRQ>
 565 | </SIGNONMSGSRQV1>'''
 566 | 
 567 |         if si is None:
 568 |             fi = ''
 569 |         else:
 570 |             fi = ofx_fi_fmt.format(
 571 |                 fid=si.fid,
 572 |                 org=si.org
 573 |                 )
 574 | 
 575 |         frag = ofx_signon_fmt.format(
 576 |                 dt=dt_now(),
 577 |                 fi=fi
 578 |                 )
 579 |         return frag
 580 | 
 581 |     def get_empty_payload(self, si):
 582 |         return ''
 583 | 
 584 |     def get_ofx_empty_payload(self, si):
 585 | 
 586 |         ofx_body = \
 587 | '''<OFX>
 588 | </OFX>
 589 | '''
 590 |         return "{}{}{}".format(self.ofxheader, '\n', ofx_body)
 591 | 
 592 |     def get_profile_payload(self, si):
 593 | 
 594 |         if self.content_type == 'text/sgml':
 595 |             ofx_body_fmt = \
 596 | '''<OFX>
 597 | {signonmsg}
 598 | <PROFMSGSRQV1>
 599 | <PROFTRNRQ>
 600 | <TRNUID>{uid}
 601 | <PROFRQ>
 602 | <CLIENTROUTING>MSGSET
 603 | <DTPROFUP>19900101
 604 | </PROFRQ>
 605 | </PROFTRNRQ>
 606 | </PROFMSGSRQV1>
 607 | </OFX>
 608 | '''
 609 | 
 610 |         elif self.content_type == 'text/xml':
 611 |             ofx_body_fmt = \
 612 | '''<OFX>
 613 | {signonmsg}
 614 | <PROFMSGSRQV1>
 615 | <PROFTRNRQ>
 616 | <TRNUID>{uid}</TRNUID>
 617 | <PROFRQ>
 618 | <CLIENTROUTING>MSGSET</CLIENTROUTING>
 619 | <DTPROFUP>19900101</DTPROFUP>
 620 | </PROFRQ>
 621 | </PROFTRNRQ>
 622 | </PROFMSGSRQV1>
 623 | </OFX>
 624 | '''
 625 | 
 626 |         body = ofx_body_fmt.format(
 627 |                 signonmsg=self._get_signonmsg_anonymous_payload(si),
 628 |                 uid=uid())
 629 |         return "{}{}{}".format(self.ofxheader, '\n', body)
 630 | 
 631 |     def get_acctinfo_payload(self, si):
 632 |         '''
 633 |         ACCTINFO Request payload
 634 |         '''
 635 | 
 636 |         ofx_body_fmt = \
 637 | '''<OFX>
 638 | {signonmsg}
 639 | <SIGNUPMSGSRQV1>
 640 | <ACCTINFOTRNRQ>
 641 | <TRNUID>{uid}
 642 | <ACCTINFORQ>
 643 | <DTACCTUP>19900101
 644 | </ACCTINFORQ>
 645 | </ACCTINFOTRNRQ>
 646 | </SIGNUPMSGSRQV1>
 647 | </OFX>
 648 | '''
 649 | 
 650 |         body = ofx_body_fmt.format(
 651 |                 signonmsg=self._get_signonmsg_anonymous_payload(si),
 652 |                 uid=uid())
 653 |         return "{}{}{}".format(self.ofxheader, '\n', body)
 654 | 
 655 |     def get_invstmtrn_payload(self, si, brokerid, acctid):
 656 |         '''
 657 |         INVSTMTTRRQ Request payload
 658 |         '''
 659 | 
 660 |         ofx_body_fmt = \
 661 | '''<OFX>
 662 | {signonmsg}
 663 | <INVSTMTMSGSRQV1>
 664 | <INVSTMTTRNRQ>
 665 | <TRNUID>{uid}
 666 | <INVSTMTRQ>
 667 | <INVACCTFROM>
 668 | <BROKERID>{broker_id}
 669 | <ACCTID>{acct_id}
 670 | </INVACCTFROM>
 671 | <INCTRAN>
 672 | <INCLUDE>Y
 673 | </INCTRAN>
 674 | <INCOO>Y
 675 | <INCPOS>
 676 | <INCLUDE>Y
 677 | </INCPOS>
 678 | <INCBAL>Y
 679 | </INVSTMTRQ>
 680 | </INVSTMTTRNRQ>
 681 | </INVSTMTMSGSRQV1>
 682 | </OFX>
 683 | '''
 684 |         body = ofx_body_fmt.format(
 685 |                 signonmsg=self._get_signonmsg_anonymous_payload(si),
 686 |                 uid=uid(),
 687 |                 broker_id=brokerid,
 688 |                 acct_id = acctid)
 689 | 
 690 |         return "{}{}{}".format(self.ofxheader, '\n', body)
 691 | 
 692 | 
 693 | class OFXFile():
 694 |     '''
 695 |     Read and parse an OFX file.
 696 | 
 697 |     This is simplistic parsing of specific fields, mainly the header and PROFRS
 698 |     '''
 699 | 
 700 |     _file_str = ''
 701 |     _v2_dict = {}
 702 | 
 703 |     headers = {}
 704 |     version = None
 705 |     signon = {}
 706 |     profile = {}
 707 | 
 708 |     def __init__(self, file_str):
 709 |         self._file_str = file_str
 710 | 
 711 |         self._convert_newlines()
 712 |         self._parse_header()
 713 | 
 714 |         if self.major_version() == 2:
 715 |             self._v2_dict = xmlparse(file_str)
 716 | 
 717 |         self._parse_signon()
 718 |         self._parse_profile()
 719 | 
 720 |     def _convert_newlines(self):
 721 |         '''
 722 |         Convert from network newlines to platform newlines.
 723 | 
 724 |         For now, just blindly Windows to Unix.
 725 |         '''
 726 | 
 727 |         self._file_str = self._file_str.replace('\r\n', '\n')
 728 | 
 729 |     def _parse_header(self):
 730 |         # Parse Version 1 Header
 731 | 
 732 |         # Example:
 733 |         #
 734 |         # OFXHEADER:100
 735 |         # DATA:OFXSGML
 736 |         # VERSION:102
 737 |         # SECURITY:NONE
 738 |         # ENCODING:USASCII
 739 |         # CHARSET:1252
 740 |         # COMPRESSION:NONE
 741 |         # OLDFILEUID:NONE
 742 |         # NEWFILEUID:NONE
 743 | 
 744 |         if self._file_str.startswith('OFXHEADER'):
 745 |             # Assume well formed and parse based on NEWLINES
 746 |             for line in self._file_str.splitlines():
 747 |                 # End of header
 748 |                 if line == '' or line.startswith('<OFX>') or len(line) > 13:
 749 |                     break
 750 |                 [k,v] = line.split(':')
 751 |                 self.headers[k] = v
 752 | 
 753 |             try:
 754 |                 self.version = int(self.headers[HDR_VERSION])
 755 |             except KeyError:
 756 |                 raise ValueError("Parse Error: No version")
 757 | 
 758 |         # Parse Version 2 Header
 759 | 
 760 |         # Example:
 761 |         # <?OFX OFXHEADER="200" VERSION="203" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?>
 762 | 
 763 |         elif self._file_str.find('<?OFX OFXHEADER') != -1:
 764 |             # Python (as of 3.7) has no way to read prolog declarations.
 765 |             # https://bugs.python.org/issue24287
 766 |             # So don't bother parsing as XML, just use a regex to read the
 767 |             # OFX header.
 768 | 
 769 |             # TODO: Pull ENCODING out of <?xml> declaration
 770 | 
 771 |             rpat = r'<\?OFX OFXHEADER="(?P<OFXHEADER>\d+)" VERSION="(?P<VERSION>\d+)" SECURITY="(?P<SECURITY>\w+)" OLDFILEUID="(?P<OLDFILEUID>\w+)" NEWFILEUID="(?P<NEWFILEUID>\w+)"\?>'
 772 | 
 773 |             match = re.search(rpat, self._file_str)
 774 |             if not match:
 775 |                 raise ValueError("Parse Error: Unable to parse V2 header with regex")
 776 |             for field in HDR_FIELDS_V2:
 777 |                 self.headers[field] = match.group(field)
 778 | 
 779 |             try:
 780 |                 self.version = int(self.headers[HDR_VERSION])
 781 |                 parsed = True
 782 |             except KeyError:
 783 |                 raise ValueError("Parse Error: No version")
 784 | 
 785 |         else:
 786 |             raise ValueError("Parse Error: Unable to parse header")
 787 | 
 788 |     def major_version(self):
 789 |         if str(self.version).startswith('1'):
 790 |             return 1
 791 |         elif str(self.version).startswith('2'):
 792 |             return 2
 793 | 
 794 |     def find_span_value(self, value, ofx_str=None, casesen=False):
 795 |         '''
 796 |         Find the span ELEMENT which contains the given value.
 797 |         '''
 798 | 
 799 |         if not ofx_str: ofx_str = self._file_str
 800 | 
 801 |         if self.major_version() == 1:
 802 |             rpat = r'<([\w.+]+)>'+value
 803 | 
 804 |         elif self.major_version() == 2:
 805 |             rpat = r'<(?P<elm>[\w.+]+)>'+value+r'</(?P=elm)>'
 806 | 
 807 |         flags = 0 if casesen else re.IGNORECASE
 808 |         matches = re.finditer(rpat, ofx_str, flags)
 809 | 
 810 |         if matches:
 811 |             return [m.group(0) for m in matches]
 812 |         else:
 813 |             return None
 814 | 
 815 |     def _parse_element_block(self, element, ofx_str=None):
 816 |         '''
 817 |         Read the internal values of a block element including other elements.
 818 |         '''
 819 |         if not ofx_str: ofx_str = self._file_str
 820 | 
 821 |         rpat = r'<'+element+r'>(.*)</'+element+r'>'
 822 |         match = re.search(rpat, ofx_str, re.DOTALL)
 823 |         if match:
 824 |             return match.group(1)
 825 |         else:
 826 |             return None
 827 | 
 828 |     def _parse_element_span(self, element, ofx_str=None):
 829 |         '''
 830 |         Read the value of a span <ELEMENT> tag
 831 |         '''
 832 |         if not ofx_str: ofx_str = self._file_str
 833 | 
 834 |         rpat = r'<'+element+r'>([^<\n]+)'
 835 |         match = re.search(rpat, ofx_str)
 836 |         if match:
 837 |             return match.group(1)
 838 |         else:
 839 |             return None
 840 | 
 841 |     def _v2_retrieve_element(self, elmpath, etype):
 842 |         '''
 843 |         Walk a dictionary tree and retrieve values from it.
 844 | 
 845 |         tagpath -  ex: 'ofx:signonmsgsrsv1:sonrs:fi:org'
 846 |         etype - ex: 'string'
 847 |         '''
 848 |         val = None
 849 |         node = self._v2_dict
 850 |         elmlist = elmpath.split(':')
 851 | 
 852 |         for elm in elmlist:
 853 |             try:
 854 |                 node = node[elm.upper()]
 855 |             except KeyError:
 856 |                 break
 857 |             if elm == elmlist[-1]:
 858 |                 if etype == 'string':
 859 |                     val = node
 860 |                 if etype == 'bool':
 861 |                     if node == 'Y':
 862 |                         val = True
 863 |                     elif node == 'N':
 864 |                         val = False
 865 |                     else:
 866 |                         raise ValueError('Unknown value for {}: {}'.format(
 867 |                             elmpath, val))
 868 |                 if etype == 'integer':
 869 |                     val = int(node)
 870 |                 elif etype == 'exist':
 871 |                     val = True
 872 |                 break
 873 | 
 874 |         return val
 875 | 
 876 | 
 877 |     def _path_to_dict(self, path, value):
 878 |         '''
 879 |         Given a string path, nest dictionaries and set a value.
 880 | 
 881 |         path - ex: 'investment:transactions'
 882 |         '''
 883 | 
 884 |         nodelist = path.split(':')
 885 |         node = self.profile
 886 | 
 887 |         for name in nodelist:
 888 |             try:
 889 |                 node = node[name]
 890 |             except KeyError:
 891 |                 if name == nodelist[-1] and value is not None:
 892 |                     node[name] = value
 893 |                 else:
 894 |                     node[name] = dict()
 895 | 
 896 | 
 897 |     def _parse_signon(self):
 898 |         '''
 899 |         Parse a SIGNON response if one exists
 900 |         '''
 901 |         if self.major_version() == 1:
 902 |             # Confirm a SIGNON response exists
 903 |             rpat = r'<SONRS>'
 904 | 
 905 |             match = re.search(rpat, self._file_str)
 906 |             if not match:
 907 |                 return
 908 | 
 909 |             # Get Server information
 910 |             elms = ('ORG', 'FID')
 911 |             for elm in elms:
 912 |                 val = self._parse_element_span(elm)
 913 |                 if val:
 914 |                     self.signon[elm] = val
 915 | 
 916 |         elif self.major_version() == 2:
 917 |             val = self._v2_retrieve_element(
 918 |                     'ofx:signonmsgsrsv1:sonrs:fi:org',
 919 |                     'string'
 920 |                     )
 921 |             if val:
 922 |                 self.signon['ORG'] = val
 923 | 
 924 |             val = self._v2_retrieve_element(
 925 |                     'ofx:signonmsgsrsv1:sonrs:fi:fid',
 926 |                     'string'
 927 |                     )
 928 |             if val:
 929 |                 self.signon['FID'] = val
 930 | 
 931 |     def _parse_profile(self):
 932 |         '''
 933 |         Parse a PROFILE response if one exists
 934 |         '''
 935 |         if self.major_version() == 1:
 936 |             # This is where we'd should start implementing a real SGML parser,
 937 |             # but I'm parsing with quick and loose regex as a first pass
 938 | 
 939 |             # Confirm that a PROFILE response exists
 940 |             profrs = self._parse_element_block('PROFRS')
 941 |             if not profrs:
 942 |                 return
 943 | 
 944 |             # Get FI contact information
 945 |             elms = ('FINAME', 'ADDR1', 'ADDR2', 'ADDR3', 'CITY',
 946 |                     'STATE', 'POSTALCODE', 'COUNTRY', 'EMAIL')
 947 | 
 948 |             for elm in elms:
 949 |                 val = self._parse_element_span(elm, profrs)
 950 |                 if val:
 951 |                     self.profile[elm] = val
 952 | 
 953 |             # Get OFX URL
 954 |             # Technically this can be different for every message set,
 955 |             # in practice it's usually identical to the PROFILE server.
 956 |             # However, if it is different, it's usually the same for every
 957 |             # other message set besides PROFMSGSET.
 958 | 
 959 |             block = self._parse_element_block('SIGNONMSGSET', profrs)
 960 |             if block:
 961 |                 val = self._parse_element_span('URL', block)
 962 |                 if val:
 963 |                     self.profile['OFXURL'] = val
 964 |                 val = self._parse_element_span('SPNAME', block)
 965 |                 if val:
 966 |                     self.profile['SPNAME'] = val
 967 | 
 968 |             # Get FI capabilities
 969 |             block = self._parse_element_block('BANKMSGSET', profrs)
 970 |             if block:
 971 |                 self.profile['BANKING'] = dict()
 972 |                 b2 = self._parse_element_block('XFERPROF', block)
 973 |                 if b2:
 974 |                     self.profile['BANKING']['INTRAXFR'] = True
 975 |                 b2 = self._parse_element_block('EMAILPROF', block)
 976 |                 if b2:
 977 |                     self.profile['BANKING']['MESSAGES'] = dict()
 978 |                     val = self._parse_element_span('CANEMAIL', b2)
 979 |                     if val == 'Y':
 980 |                         self.profile['BANKING']['MESSAGES']['EMAIL'] = True
 981 |                     val = self._parse_element_span('CANNOTIFY', b2)
 982 |                     if val == 'Y':
 983 |                         self.profile['BANKING']['MESSAGES']['NOTIFY'] = True
 984 | 
 985 |             block = self._parse_element_block('INVSTMTMSGSET', profrs)
 986 |             if block:
 987 |                 self.profile['INVESTMENT'] = dict()
 988 |                 val = self._parse_element_span('TRANDNLD', block)
 989 |                 if val:
 990 |                     self.profile['INVESTMENT']['TRANSACTIONS'] = True
 991 |                 val = self._parse_element_span('OODNLD', block)
 992 |                 if val:
 993 |                     self.profile['INVESTMENT']['OPENORDERS'] = True
 994 |                 val = self._parse_element_span('POSDNLD', block)
 995 |                 if val:
 996 |                     self.profile['INVESTMENT']['POSITIONS'] = True
 997 |                 val = self._parse_element_span('POSDNLD', block)
 998 |                 if val:
 999 |                     self.profile['INVESTMENT']['POSITIONS'] = True
1000 |                 val = self._parse_element_span('BALDNLD', block)
1001 |                 if val:
1002 |                     self.profile['INVESTMENT']['BALANCES'] = True
1003 | 
1004 |             block = self._parse_element_block('SECLISTMSGSET', profrs)
1005 |             if block:
1006 |                 val = self._parse_element_span('SECLISTRQDNLD', block)
1007 |                 if val:
1008 |                     self.profile['INVESTMENT']['QUOTES'] = True
1009 | 
1010 |             block = self._parse_element_block('CREDITCARDMSGSET', profrs)
1011 |             if block:
1012 |                 self.profile['CREDITCARD'] = dict()
1013 |                 val = self._parse_element_span('CLOSINGAVAIL', block)
1014 |                 if val == 'Y':
1015 |                     self.profile['CREDITCARD']['STATEMENT'] = True
1016 | 
1017 |             block = self._parse_element_block('BILLPAYMSGSET', profrs)
1018 |             if block:
1019 |                 self.profile['BILLPAY'] = dict()
1020 | 
1021 |             block = self._parse_element_block('EMAILMSGSET', profrs)
1022 |             if block:
1023 |                 self.profile['MESSAGING'] = dict()
1024 |                 val = self._parse_element_span('MAILSUP')
1025 |                 if val == 'Y':
1026 |                     self.profile['MESSAGING']['EMAIL'] = True
1027 |                 val = self._parse_element_span('GETMIMESUP')
1028 |                 if val == 'Y':
1029 |                     self.profile['MESSAGING']['MIME'] = True
1030 | 
1031 |             # Get Password Policy
1032 |             block = self._parse_element_block('SIGNONINFO', profrs)
1033 |             if block:
1034 |                 self.profile['AUTHENTICATION'] = dict()
1035 |                 val = self._parse_element_span('MIN', block)
1036 |                 if val:
1037 |                     self.profile['AUTHENTICATION']['MINPASS'] = int(val)
1038 |                 val = self._parse_element_span('MAX', block)
1039 |                 if val:
1040 |                     self.profile['AUTHENTICATION']['MAXPASS'] = int(val)
1041 |                 val = self._parse_element_span('CHARTYPE', block)
1042 |                 if val:
1043 |                     self.profile['AUTHENTICATION']['COMPLEXITY'] = val
1044 |                 val = self._parse_element_span('CASESEN', block)
1045 |                 if val:
1046 |                     if val == 'Y':
1047 |                         self.profile['AUTHENTICATION']['CASESEN'] = True
1048 |                     elif val == 'N':
1049 |                         self.profile['AUTHENTICATION']['CASESEN'] = False
1050 |                     else:
1051 |                         raise ValueError(
1052 |                             'Unknown value for CASESEN: {}'.format(va))
1053 |                 val = self._parse_element_span('SPECIAL', block)
1054 |                 if val:
1055 |                     if val == 'Y':
1056 |                         self.profile['AUTHENTICATION']['SPECIAL'] = True
1057 |                     elif val == 'N':
1058 |                         self.profile['AUTHENTICATION']['SPECIAL'] = False
1059 |                     else:
1060 |                         raise ValueError(
1061 |                             'Unknown value for SPECIAL: {}'.format(va))
1062 |                 val = self._parse_element_span('CLIENTUIDREQ')
1063 |                 if val == 'Y':
1064 |                     try:
1065 |                         tmp = self.profile['AUTHENTICATION']['MFA']
1066 |                     except KeyError:
1067 |                         self.profile['AUTHENTICATION']['MFA'] = dict()
1068 |                     self.profile['AUTHENTICATION']['MFA']['CLIENTUID'] = True
1069 | 
1070 |         elif self.major_version() == 2:
1071 |             # Confirm that a PROFILE response exists
1072 |             if not self._v2_retrieve_element(
1073 |                     'ofx:profmsgsrsv1:proftrnrs:profrs',
1074 |                     'exist'):
1075 |                 return
1076 | 
1077 |             # OFX Path, Retrieve Type, Profile Object Path
1078 |             profile_elms = [
1079 |                 # Get FI contact information
1080 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:finame',
1081 |                  'string',
1082 |                  'FINAME'),
1083 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:addr1',
1084 |                  'string',
1085 |                  'ADDR1'),
1086 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:addr2',
1087 |                  'string',
1088 |                  'ADDR2'),
1089 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:addr3',
1090 |                  'string',
1091 |                  'ADDR3'),
1092 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:city',
1093 |                  'string',
1094 |                  'CITY'),
1095 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:state',
1096 |                  'string',
1097 |                  'STATE'),
1098 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:postalcode',
1099 |                  'string',
1100 |                  'POSTALCODE'),
1101 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:country',
1102 |                  'string',
1103 |                  'COUNTRY'),
1104 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:email',
1105 |                  'string',
1106 |                  'EMAIL'),
1107 | 
1108 |                 # Get OFX URL
1109 |                 # Technically this can be different for every message set,
1110 |                 # in practice it's usually identical to the PROFILE server.
1111 |                 # However, if it is different, it's usually the same for every
1112 |                 # other message set besides PROFMSGSET.
1113 | 
1114 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:signonmsgset:signonmsgsetv1:msgsetcore:url',
1115 |                  'string',
1116 |                  'OFXURL'),
1117 | 
1118 |                 # Get FI capabilities
1119 | 
1120 |                 # Investments
1121 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset',
1122 |                  'exist',
1123 |                  'INVESTMENT'),
1124 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset:invstmtmsgsetv1:trandnld',
1125 |                  'bool',
1126 |                  'INVESTMENT:TRANSACTIONS'),
1127 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset:invstmtmsgsetv1:oodnld',
1128 |                  'bool',
1129 |                  'INVESTMENT:OPENORDERS'),
1130 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset:invstmtmsgsetv1:posdnld',
1131 |                  'bool',
1132 |                  'INVESTMENT:POSITIONS'),
1133 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset:invstmtmsgsetv1:baldnld',
1134 |                  'bool',
1135 |                  'INVESTMENT:BALANCES'),
1136 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:invstmtmsgset:invstmtmsgsetv1:inv401kdnld',
1137 |                  'bool',
1138 |                  'INVESTMENT:401K'),
1139 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:seclistmsgset:seclistmsgsetv1:seclistrqdnld',
1140 |                  'bool',
1141 |                  'INVESTMENT:QUOTES'),
1142 | 
1143 |                 # Taxes
1144 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:tax1099msgset:tax1099msgsetv1',
1145 |                  'exist',
1146 |                  'TAXES'),
1147 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:tax1099msgset:tax1099msgsetv1:tax1099dnld',
1148 |                  'bool',
1149 |                  'TAXES:1099'),
1150 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:tax1099msgset:tax1099msgsetv1:extd1099b',
1151 |                  'bool',
1152 |                  'TAXES:1099B'),
1153 |                 # TODO: There could be multiple years returned.
1154 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:msgsetlist:tax1099msgset:tax1099msgsetv1:taxyearsupported',
1155 |                  'string',
1156 |                  'TAXES:YEARS'),
1157 | 
1158 |                 # Get Password Policy
1159 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo',
1160 |                  'exist',
1161 |                  'AUTHENTICATION'),
1162 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:min',
1163 |                  'integer',
1164 |                  'AUTHENTICATION:MINPASS'),
1165 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:max',
1166 |                  'integer',
1167 |                  'AUTHENTICATION:MAXPASS'),
1168 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:chartype',
1169 |                  'string',
1170 |                  'AUTHENTICATION:COMPLEXITY'),
1171 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:casesen',
1172 |                  'bool',
1173 |                  'AUTHENTICATION:CASESEN'),
1174 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:special',
1175 |                  'bool',
1176 |                  'AUTHENTICATION:SPECIAL'),
1177 |                 ('ofx:profmsgsrsv1:proftrnrs:profrs:signoninfolist:signoninfo:clientuidreq',
1178 |                  'bool',
1179 |                  'AUTHENTICATION:MFA:CLIENTUID'),
1180 |             ]
1181 | 
1182 |             for elm in profile_elms:
1183 |                 val = self._v2_retrieve_element(elm[0], elm[1])
1184 |                 if val:
1185 |                     # We set None to mean create a sub-dictionary
1186 |                     val = val if elm[1] != 'exist' else None
1187 |                     self._path_to_dict(elm[2], val)
1188 | 
1189 |     def get_version(self):
1190 |         '''
1191 |         Return string representation of OFX document version number.
1192 |         '''
1193 |         if self.version:
1194 |             return '.'.join(list(str(self.version)))
1195 |         else:
1196 |             return ''
1197 | 
1198 | 
1199 | class OFXServerTests():
1200 |     '''
1201 |     Run collection of OFX tests
1202 |     '''
1203 | 
1204 |     results = []
1205 |     profrs = None
1206 | 
1207 |     def __init__(self, server):
1208 |         self.si = server
1209 | 
1210 |     def run_tests(self, req_results):
1211 |         messages = []
1212 | 
1213 |         try:
1214 |             self.profrs = OFXFile(req_results[REQ_NAME_OFX_PROFILE].text)
1215 |         except ValueError:
1216 |             msg = 'Cannot read OFX PROFILE, some tests will not be run.'
1217 |             messages.append(msg)
1218 | 
1219 |         self.test_tls(self.si)
1220 |         self.test_content_type(req_results)
1221 |         self.test_server_diclosure(self.si)
1222 | 
1223 |         if self.profrs:
1224 |             self.test_mfa(req_results)
1225 |             self.test_password_policy(req_results)
1226 |             self.test_user_disclosure(req_results)
1227 | 
1228 |         self.test_null_values(req_results)
1229 |         self.test_500_http_response(req_results)
1230 |         self.test_internal_ip(req_results)
1231 | 
1232 |         return messages
1233 | 
1234 |     def test_tls(self, server):
1235 |         title = 'Transport Layer Security (TLS)'
1236 |         passed = True
1237 |         messages = []
1238 | 
1239 |         if not server.get_tls():
1240 |             passed = False
1241 |             msg = 'Unable to securely connect to the server over TLS'
1242 |             messages.append(msg)
1243 | 
1244 |         self.results.append({
1245 |             'Title': title,
1246 |             'Passed': passed,
1247 |             'Messages': messages
1248 |             })
1249 | 
1250 |     def test_mfa(self, req_results):
1251 |         title = 'Multi-Factor Authentication'
1252 |         passed = True
1253 |         messages = []
1254 | 
1255 |         if self.profrs.major_version() == 1:
1256 |             requirement = 103
1257 |         elif self.profrs.major_version() == 2:
1258 |             requirement = 203
1259 | 
1260 |         if self.profrs.version and self.profrs.version < requirement:
1261 |             passed = False
1262 |             msg = 'OFX protocol version ({}) does not support MFA'.format(
1263 |                     self.profrs.get_version())
1264 |             messages.append(msg)
1265 | 
1266 |         self.results.append({
1267 |             'Title': title,
1268 |             'Passed': passed,
1269 |             'Messages': messages
1270 |             })
1271 | 
1272 |     def test_password_policy(self, req_results):
1273 |         title = 'Password Policy'
1274 |         passed = True
1275 |         messages = []
1276 | 
1277 |         minpass = None
1278 |         requirement = 8
1279 |         try:
1280 |             minpass = self.profrs.profile['AUTHENTICATION']['MINPASS']
1281 |         except KeyError:
1282 |             pass
1283 |         if minpass and minpass < requirement:
1284 |             passed = False
1285 |             msg = 'Minimum password length ({}) is less than recommended ({})'.format(
1286 |                     minpass, requirement)
1287 |             messages.append(msg)
1288 | 
1289 |         self.results.append({
1290 |             'Title': title,
1291 |             'Passed': passed,
1292 |             'Messages': messages
1293 |             })
1294 | 
1295 |     def test_user_disclosure(self, req_results):
1296 |         title = 'Username Disclosure'
1297 |         passed = True
1298 |         messages = []
1299 | 
1300 |         common_aliases_exact = ['test', 'members', 'it', 'email', 'assist']
1301 | 
1302 |         common_aliases_in = ['info', 'support', 'service', 'reply', 'online',
1303 |         'help', 'webmaster', 'quicken', 'question', 'postmaster', 'client',
1304 |         'internet', 'bank', 'commerce', 'business', 'deposit', 'customer',
1305 |         'feedback', 'central', 'center', 'bookkeep', 'ask', 'virtual',
1306 |         'management', 'operation', 'contact', 'inbox', 'staff', 'investor']
1307 | 
1308 |         try:
1309 |             email = self.profrs.profile['EMAIL']
1310 |         except:
1311 |             email = ''
1312 | 
1313 |         # 1) Check that it has an @ sign
1314 |         # 2) Lowercase
1315 |         # 3) Check if common_aliases_exact doesn't match
1316 |         # 4) Check common_aliases_in is not in it
1317 |         # 5) Check if name == domain
1318 |         # 6) Check if it has a '.' or '_'
1319 |         #     a) Likely a username
1320 |         #     b) Mark as Error
1321 |         # 7) Else:
1322 |         #     a) Likely a username
1323 |         #     b) Mark as Warning
1324 | 
1325 |         if '@' in email:
1326 |             email_lc = email.lower()
1327 |             name = email_lc.split('@')[0]
1328 |             domain = email_lc.split('@')[1].split('.')[-2]
1329 | 
1330 |             if name in common_aliases_exact:
1331 |                 pass
1332 |             elif any(substring in name for substring in common_aliases_in):
1333 |                 pass
1334 |             elif name.startswith(domain):
1335 |                 pass
1336 |             else:
1337 |                 passed = False
1338 |                 if '.' in name or '_' in name:
1339 |                     msg = 'Email address is likely a username: {}'.format(email)
1340 |                 else:
1341 |                     msg = 'Email address may be a username: {}'.format(
1342 |                             email)
1343 |                 messages.append(msg)
1344 | 
1345 |         self.results.append({
1346 |             'Title': title,
1347 |             'Passed': passed,
1348 |             'Messages': messages
1349 |             })
1350 | 
1351 |     def test_server_diclosure(self, server):
1352 |         title = 'Web Server Disclosure'
1353 |         passed = True
1354 |         messages = []
1355 | 
1356 |         # Because our current fingerprinting only uses server headers and
1357 |         # banners, we can assume if we've identified one, then it's a simple
1358 |         # information disclosure.
1359 | 
1360 |         # Only error on servers that diclose their version number
1361 | 
1362 |         s = server.httpserver
1363 |         if s != '' and any(str(num) in s for num in range(10)):
1364 |             passed = False
1365 |             msg = 'Web server discloses type and version: {}'.format(s)
1366 |             messages.append(msg)
1367 | 
1368 |         f = server.webframework
1369 |         if f != '' and any(str(num) in s for num in range(10)):
1370 |             passed = False
1371 |             msg = 'Web framework discloses type and version: {}'.format(f)
1372 |             messages.append(msg)
1373 | 
1374 |         self.results.append({
1375 |             'Title': title,
1376 |             'Passed': passed,
1377 |             'Messages': messages
1378 |             })
1379 | 
1380 |     def test_content_type(self, req_results):
1381 |         title = 'Incorrect Content-Type'
1382 |         passed = True
1383 |         messages = []
1384 | 
1385 |         for req_name in [
1386 |             REQ_NAME_POST_OFX,
1387 |             REQ_NAME_OFX_EMPTY,
1388 |             REQ_NAME_OFX_PROFILE]:
1389 | 
1390 |             # TODO: Check for OFX header
1391 |             if req_results[req_name].status_code == 400:
1392 |                 continue
1393 | 
1394 |             try:
1395 |                 ctype = req_results[req_name].headers['Content-Type'].split(';')[0]
1396 |             except KeyError:
1397 |                 # TODO: test if content-length is 0, if not raise an error
1398 |                 continue
1399 | 
1400 |             if ctype != CONTENT_TYPE:
1401 |                 msg = '{}: Incorrect Content-Type in OFX response: {}'.format(
1402 |                         req_name, ctype)
1403 |                 messages.append(msg)
1404 |                 passed = False
1405 | 
1406 |         self.results.append({
1407 |             'Title': title,
1408 |             'Passed': passed, 'Messages': messages
1409 |             })
1410 | 
1411 |     def test_null_values(self, req_results):
1412 |         title = 'Null Values Returned'
1413 |         passed = True
1414 |         messages = []
1415 | 
1416 |         for req_name in [
1417 |             REQ_NAME_POST_OFX,
1418 |             REQ_NAME_OFX_EMPTY,
1419 |             REQ_NAME_OFX_PROFILE]:
1420 | 
1421 |             try:
1422 |                 resp = OFXFile(req_results[req_name].text)
1423 |                 matches = resp.find_span_value('null')
1424 |                 for m in matches:
1425 |                     msg = '{}: null value returned: {}'.format(req_name, m)
1426 |                     messages.append(msg)
1427 |                     passed = False
1428 |             except ValueError:
1429 |                 # Ignore if we didn't get an OFX reponse back
1430 |                 pass
1431 | 
1432 |         self.results.append({
1433 |             'Title': title,
1434 |             'Passed': passed,
1435 |             'Messages': messages
1436 |             })
1437 | 
1438 |     def test_500_http_response(self, req_results):
1439 |         title = 'Internal Server Errors'
1440 |         passed = True
1441 |         messages = []
1442 | 
1443 |         # We'll ignore most 500 errors. Only complain about information
1444 |         # disclosure.
1445 | 
1446 |         for req_name in [
1447 |             REQ_NAME_POST_OFX,
1448 |             REQ_NAME_OFX_EMPTY,
1449 |             REQ_NAME_OFX_PROFILE]:
1450 | 
1451 |             if req_results[req_name].status_code != 500:
1452 |                 continue
1453 | 
1454 |             body = req_results[req_name].text
1455 | 
1456 |             # Check for known bad return messages
1457 |             if body.startswith('Error 500:'):
1458 |                 msg = '{}: Verbose error message: {}'.format(req_name,
1459 |                         body.splitlines()[0])
1460 |                 messages.append(msg)
1461 |             elif req_name in [
1462 |                 REQ_NAME_POST_OFX,
1463 |                 REQ_NAME_OFX_EMPTY
1464 |                 ]:
1465 |                 msg = '{}: Unhandled exception parsing invalid input'.format(
1466 |                         req_name)
1467 |                 messages.append(msg)
1468 |             passed = False
1469 | 
1470 |         self.results.append({
1471 |             'Title': title,
1472 |             'Passed': passed,
1473 |             'Messages': messages
1474 |             })
1475 | 
1476 |     def test_internal_ip(self, req_results):
1477 |         title = 'Internal IP Address Disclosure'
1478 |         passed = True
1479 |         messages = []
1480 | 
1481 |         for req_name in [
1482 |             REQ_NAME_GET_ROOT,
1483 |             REQ_NAME_GET_OFX,
1484 |             REQ_NAME_POST_OFX,
1485 |             REQ_NAME_OFX_EMPTY,
1486 |             REQ_NAME_OFX_PROFILE]:
1487 | 
1488 |             body = req_results[req_name].text
1489 | 
1490 |             if len(body) == 0:
1491 |                 continue
1492 | 
1493 |             # regex for known internal IP addresses
1494 |             rpat = r'http[s]?://((localhost)|(127\.0\.0\.[0-9]{1,3})|(10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})):?[0-9]{0,5}/?[\w\-/]*'
1495 | 
1496 |             matches = re.finditer(rpat, body)
1497 | 
1498 |             if matches:
1499 |                 for m in [m.group(0) for m in matches]:
1500 |                     msg = '{}: Internal IP returned: {}'.format(req_name, m)
1501 |                     messages.append(msg)
1502 |                     passed = False
1503 | 
1504 |         self.results.append({
1505 |             'Title': title,
1506 |             'Passed': passed,
1507 |             'Messages': messages
1508 |             })
1509 | 
1510 | 


--------------------------------------------------------------------------------