├── .gitattributes
├── README.md
├── valSECfilings.py
├── loadSECfilings.py
├── extractRatios.py
└── LICENSE


/.gitattributes:
--------------------------------------------------------------------------------
 1 | # Auto detect text files and perform LF normalization
 2 | * text=auto
 3 | * -whitespace
 4 | 
 5 | # Custom for Visual Studio
 6 | *.cs     diff=csharp
 7 | *.sln    merge=union
 8 | *.csproj merge=union
 9 | *.vbproj merge=union
10 | *.fsproj merge=union
11 | *.dbproj merge=union
12 | 
13 | # Standard to msysgit
14 | *.doc	 diff=astextplain
15 | *.DOC	 diff=astextplain
16 | *.docx diff=astextplain
17 | *.DOCX diff=astextplain
18 | *.dot  diff=astextplain
19 | *.DOT  diff=astextplain
20 | *.pdf  diff=astextplain
21 | *.PDF	 diff=astextplain
22 | *.rtf	 diff=astextplain
23 | *.RTF	 diff=astextplain
24 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | sec-xbrl
  2 | ========
  3 | 
  4 | Copyright 2014 Altova GmbH
  5 | 
  6 | Licensed under the Apache License, Version 2.0 (the "License");
  7 | you may not use this file except in compliance with the License.
  8 | You may obtain a copy of the License at
  9 | 
 10 |   http://www.apache.org/licenses/LICENSE-2.0
 11 | 
 12 | Unless required by applicable law or agreed to in writing, software
 13 | distributed under the License is distributed on an "AS IS" BASIS,
 14 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 15 | See the License for the specific language governing permissions and
 16 | limitations under the License.
 17 | 
 18 | -------------------------------------------------------------------------
 19 | 
 20 | <h2>XBRL.US Webinar: How to download and process SEC XBRL Data Directly from EDGAR</h2>
 21 | 
 22 | These are the supporting Python files for the XBRL.US Webinar that is availble
 23 | on YouTube: https://www.youtube.com/watch?v=2Oe9ZqXVGME as well as the slides
 24 | available here on SlideShare: http://www.slideshare.net/afalk42/xbrl-us-altova-webinar
 25 | 
 26 | See also my recent blog post: http://www.xmlaficionado.com/2014/09/how-to-download-and-process-sec-xbrl.html
 27 | 
 28 | Please watch the YouTube video and review the slides to see how these Python
 29 | scripts are intended to be used. Also note that these scripts were written with
 30 | Python 3.3.3 so they may require modifications if you use them with a different
 31 | version of Python.
 32 | 
 33 | To use this approach you will need to download and install RaptorXML+XBRL Server from
 34 | the Altova website: http://www.altova.com/download-trial-server.html and then 
 35 | request a 30-day free evaluation license key.
 36 | 
 37 | These sample Python scripts available here on GitHub were tested with a the MacOS
 38 | version of RaptorXML+XBRL Server, but should function with the Windows or Linux version
 39 | as well. You may need to change the file-paths pointing to the RaptorXML+XBRL Server
 40 | executable in the Python script, though, or add them to the global PATH environment
 41 | variable on your system.
 42 | 
 43 | In addition to the standard Python libraries, you also need to install the Python
 44 | feedparser module/library available here: https://pypi.python.org/pypi/feedparser
 45 | 
 46 | For more information on RaptorXML, please see here: http://www.altova.com/raptorxml.html
 47 | 
 48 | -------------------------------------------------------------------------
 49 | 
 50 | <h3>Usage Information</h3>
 51 | 
 52 | These scripts now require RaptorXML+XBRL v2015r3 or newer.
 53 | 
 54 | <h4>loadSECfilings</h4>
 55 | 
 56 |     loadSECfilings.py -y <year> -m <month> | -f <from_year> -t <to_year>
 57 | 
 58 | These creates a subdirectory sec/ and then subsequent year-based directories and months
 59 | underneath and downloads all SES XBRL filings from the EDGAR system to your local hard
 60 | disk for further processing. Please use only during off-peak hours in order to not
 61 | overload the SEC servers. This downloads the ZIPped XBRL filings, so you'll have one
 62 | ZIP file per filing submitted to the SEC on your drive. If you call this script
 63 | again for the current or any previous month at a later day, it will only download
 64 | any files that are new and have not yet been downloaded before.
 65 | 
 66 | <h5>Examples</h5>
 67 | 
 68 |     python3 loadSECfilings.py -y 2014 -m 9
 69 | 
 70 | This will load all SEC filing for September 2014.
 71 | 
 72 |     python3 loadSECfilings.py -f 2005 -t 2014
 73 | 
 74 | This will load all SEC filing for the start of the XBRL pilot program in 2005 until 2014.
 75 | WARNING: If you download all years available (2005-2014) this will be about 127,000 files
 76 | and take about 18GB of data on your hard disk, so please use with caution, especially 
 77 | when you are on a slow Internet connection.
 78 | 
 79 | 
 80 | <h4>valSECfilings</h4>
 81 | 
 82 |     valSECfilings ( -y <year> | -f <from_year> -t <to_year> ) -m <month> 
 83 |               -c <cik> -k <ticker> -s <script>
 84 | 
 85 | This will call RaptorXML+XBRL Server to validate the SEC filings for a specified year
 86 | and month or for a range of years. It assumes that the files have been downloaded by
 87 | the script above into a local sub-directory sec/. You can restrict the filings to just
 88 | those for a particular company or for a list of companies by providing their respective 
 89 | CIKs or ticker symbols. Optionally you can pass a Python script to RaptorXML+XBRL Server
 90 | with the -s parameter, which will then be executed by the built-in Python interpreter
 91 | inside of RaptorXML+XBRL Server to perform additional post-validation processing of
 92 | the XBRL files. As an example, there is a Python script extractRatios.py in this project
 93 | that demonstrates how to extract common financial ratios (quick ratio, cash ratio) from
 94 | the XBRL filings.
 95 | 
 96 | <h5>Examples</h5>
 97 | 
 98 |     python3 valSECfilings.py -y 2014 -m 9
 99 | 
100 | This will validate all downloaded SEC filings for the month of September 2014. If a large
101 | number of files is passed to the Python script, it will create batches of about 20 jobs
102 | each and pass those to RaptorXML+XBRL Server in sequential batches.
103 | 
104 |     python3 valSECfilings.py -f 2013 -t 2014 -k AAPL,MSFT,ORCL
105 | 
106 | This will validate all SEC filings submitted by Apple, Microsoft, and Oracle for the
107 | years 2013 and 2014. Positive validation messages as well as any errors or warnings
108 | are output to the console window.
109 | 
110 |     python3 valSECfilings.py -f 2013 -t 2014 -k ORCL -s extractRatios.py
111 | 
112 | This will validate all Oracle XBRL filings for the years 2013-2014 and then perform
113 | post-validation analysis of the filings using the supplied Python script extractRatios.py
114 | that gets passed to RaptorXML+XBRL Server and executed by its built-in Python interpreter.
115 | This particular example script prints document and entity information and then extracts
116 | various balance sheet facts to calculate current ratio, quick ratio, and cash ratio as
117 | and example of how to do post-validation XBRL processing. Furthermore, it appends those
118 | ratios to an output file ratios.csv in the same directory.
119 | 
120 | -------------------------------------------------------------------------
121 | 
122 | <h3>Reminder</h3>
123 | 
124 | To see these scripts and a lot more in-depth explanation, please watch the
125 | YouTube video of the webinar here: https://www.youtube.com/watch?v=2Oe9ZqXVGME
126 | 


--------------------------------------------------------------------------------
/valSECfilings.py:
--------------------------------------------------------------------------------
  1 | # Copyright 2014 Altova GmbH
  2 | #
  3 | # Licensed under the Apache License, Version 2.0 (the "License");
  4 | # you may not use this file except in compliance with the License.
  5 | # You may obtain a copy of the License at
  6 | #
  7 | #     http://www.apache.org/licenses/LICENSE-2.0
  8 | #
  9 | # Unless required by applicable law or agreed to in writing, software
 10 | # distributed under the License is distributed on an "AS IS" BASIS,
 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | # See the License for the specific language governing permissions and
 13 | # limitations under the License.
 14 | 
 15 | import glob
 16 | import os.path
 17 | import sys, getopt
 18 | import time
 19 | from datetime import datetime
 20 | from subprocess import call
 21 | import zipfile
 22 | from multiprocessing import Pool
 23 | from urllib.request import urlopen
 24 | from urllib.error import URLError, HTTPError
 25 | import xml.etree.ElementTree as ET
 26 | 
 27 | joblist = []
 28 | job_limit = 20						# number of simultaneous jobs passed to Raptor per batch
 29 | usage_string = 'valSECfilings ( -y <year> | -f <from_year> -t <to_year> ) -m <month> -c <cik> -k <ticker> -s <script>'
 30 | 
 31 | def lookup_cik(ticker, name=None):
 32 | 	# Given a ticker symbol, retrieves the CIK.
 33 | 	good_read = False
 34 | 	ticker = ticker.strip().upper()
 35 | 	url = 'http://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK={cik}&count=10&output=xml'.format(cik=ticker)
 36 | 
 37 | 	try:
 38 | 		xmlFile = urlopen( url )
 39 | 		try:
 40 | 			xmlData = xmlFile.read()
 41 | 			good_read = True
 42 | 		finally:
 43 | 			xmlFile.close()
 44 | 	except HTTPError as e:
 45 | 		print( "HTTP Error:", e.code )
 46 | 	except URLError as e:
 47 | 		print( "URL Error:", e.reason )
 48 | 	except TimeoutError as e:
 49 | 		print( "Timeout Error:", e.reason )
 50 | 	except socket.timeout:
 51 | 		print( "Socket Timeout Error" )
 52 | 	if not good_read:
 53 | 		print( "Unable to lookup CIK for ticker:", ticker )
 54 | 		return
 55 | 	try:
 56 | 		root = ET.fromstring(xmlData)
 57 | 	except ET.ParseError as perr:
 58 | 		print( "XML Parser Error:", perr )
 59 | 
 60 | 	try:
 61 | 		cikElement = list(root.iter( "CIK" ))[0]
 62 | 		return int(cikElement.text)
 63 | 	except StopIteration:
 64 | 		pass
 65 | 
 66 | def chunks(l, n):
 67 |     # Yield successive n-sized chunks from l
 68 |     for i in range(0, len(l), n):
 69 |         yield l[i:i+n]
 70 | 
 71 | def appendjoblist( year, month, cik=None ):
 72 | 	target_dir = "sec/" + str(year) + '/' + str(month).zfill(2) + '/'
 73 | 	cikPattern = None
 74 | 	if not cik==None:
 75 | 		cikStr = list(map( str, cik ))
 76 | 		cikPattern = tuple(cs.zfill(10) for cs in cikStr)
 77 | 	try:
 78 | 		for filename in os.listdir( target_dir ):
 79 | 			add_file = False
 80 | 			if os.path.splitext(filename)[1] == ".zip":
 81 | 				if cik == None:
 82 | 					add_file = True
 83 | 				else:
 84 | 					if filename.startswith( cikPattern ):
 85 | 						add_file = True
 86 | 				if add_file:
 87 | 					zipname = target_dir+filename
 88 | 					joblist.append( zipname )
 89 | 	except FileNotFoundError as fe:
 90 | 		print( 'Error: no SEC filings found in directory', target_dir )
 91 | 
 92 | 
 93 | def validatejob( joblist, script=None ):
 94 | 	result = None
 95 | 	if script==None:
 96 | 		result = call(["raptorxmlxbrl", "xbrl", "--listfile", joblist] )	#  "--verbose=true",
 97 | 	else:
 98 | 		result = call(["raptorxmlxbrl", "xbrl", "--script-api-version=1", "--script="+script,
 99 | 						"--listfile", joblist] )
100 | 	return result
101 | 
102 | def xbrlname( zipname ):
103 | 	result = None
104 | 	z = zipfile.ZipFile( zipname )
105 | 	xmllist = [ f for f in z.namelist() if f.endswith('.xml') ]
106 | 	filtered = [ v for v in xmllist if not v.startswith('defnref') ]			# shortest xml filename may not always work
107 | 	xbrlname = min( filtered, key=len )
108 | 	return zipname+"|zip/"+xbrlname
109 | 
110 | def partitionjoblist(joblist, script=None):
111 | 	partition_count = 0
112 | 	for c in chunks( joblist, job_limit ):
113 | 		partition_count += 1
114 | 		print( "Partition:", partition_count )
115 | 		runjoblist( c, script )
116 | 
117 | def runjoblist(joblist, script=None):
118 | 	jobfile = open( "valSECfilings.jobs", "w" )
119 | 	try:
120 | 		for job in joblist:
121 | 			jobfile.write( xbrlname( job ) )
122 | 			jobfile.write( "\n" )
123 | 	finally:
124 | 		jobfile.close()
125 | 	print( "Running batch of", len(joblist), "validation jobs..." )
126 | 	validatejob( "valSECfilings.jobs", script )
127 | 
128 | def main(argv):
129 | 	year = 1999
130 | 	month = 0
131 | 	from_year = 1999
132 | 	to_year = 1999
133 | 	year_range = False
134 | 	one_cik = False
135 | 	script = None
136 | 	cik = None
137 | 	ticker = None
138 | 	if not os.path.exists( "sec" ):
139 | 		os.makedirs( "sec" )
140 | 	start_time  = time.time()
141 | 	try:
142 | 		opts, args = getopt.getopt(argv,"hy:m:f:t:c:k:s:",["year=","month=","from=","to=","cik=","ticker=","script="])
143 | 	except getopt.GetoptError:
144 | 		print( usage_string )
145 | 		sys.exit(2)
146 | 	for opt, arg in opts:
147 | 		if opt == '-h':
148 | 			print( usage_string )
149 | 			sys.exit()
150 | 		elif opt in ("-y", "--year"):
151 | 			year = int(arg)
152 | 		elif opt in ("-m", "--month"):
153 | 			month = int(arg)
154 | 		elif opt in ("-f", "--from"):
155 | 			from_year = int(arg)
156 | 			year_range = True
157 | 		elif opt in ("-t", "--to"):
158 | 			to_year = int(arg)
159 | 			year_range = True
160 | 		elif opt in ("-c", "--cik"):
161 | 			cik = list(map(int, arg.split(',')))
162 | 		elif opt in ("-k", "--ticker"):
163 | 			ticker = arg.split(',')
164 | 			cik = list(map( lookup_cik, ticker ))
165 | 		elif opt in ("-s", "--script"):
166 | 			script = arg
167 | 	if not ticker==None:
168 | 		print( "Tickers: ", ticker )
169 | 	if not cik==None:
170 | 		print( "CIKs: ", cik )
171 | 	if not year_range and month==0 and not year==1999:
172 | 		from_year=year
173 | 		to_year=year
174 | 		year_range=True
175 | 	if year_range:
176 | 		if from_year == 1999:
177 | 			from_year = to_year
178 | 		if to_year == 1999:
179 | 			to_year = from_year
180 | 		if from_year < 2005:
181 | 			print( 'Error: SEC filings only started in 2005' )
182 | 			print( 'Please specify a year > 2005 for -y or -f argument' )
183 | 			print( usage_string )
184 | 			sys.exit(2)
185 | 		for year in range( from_year, to_year+1 ):
186 | 			last_month = 12
187 | 			if year==datetime.today().year:
188 | 				last_month=datetime.today().month
189 | 			for month in range( 1, last_month+1 ):
190 | 				appendjoblist( year, month, cik )
191 | 	else:
192 | 		if year < 2005:
193 | 			print( 'Error: SEC filings only started in 2005' )
194 | 			print( 'Please specify a year > 2005 for -y or -f argument' )
195 | 			print( usage_string )
196 | 			sys.exit(2)
197 | 		appendjoblist( year, month, cik )
198 | 	print( len(joblist), "validation jobs queued up for RaptorXML+XBRL" )
199 | 	if len(joblist) > job_limit:
200 | 		partitionjoblist(joblist, script)
201 | 	else:
202 | 		runjoblist(joblist, script)
203 | 	end_time = time.time()
204 | 	print( "Elapsed real time:", end_time - start_time, "seconds" )
205 | 
206 | if __name__ == "__main__":
207 | 	main(sys.argv[1:])
208 | 


--------------------------------------------------------------------------------
/loadSECfilings.py:
--------------------------------------------------------------------------------
  1 | # Copyright 2014 Altova GmbH
  2 | # 
  3 | # Licensed under the Apache License, Version 2.0 (the "License");
  4 | # you may not use this file except in compliance with the License.
  5 | # You may obtain a copy of the License at
  6 | # 
  7 | #     http://www.apache.org/licenses/LICENSE-2.0
  8 | # 
  9 | # Unless required by applicable law or agreed to in writing, software
 10 | # distributed under the License is distributed on an "AS IS" BASIS,
 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | # See the License for the specific language governing permissions and
 13 | # limitations under the License.
 14 | 
 15 | import feedparser
 16 | import os.path
 17 | import sys, getopt
 18 | import time
 19 | import socket
 20 | from urllib.request import urlopen
 21 | from urllib.error import URLError, HTTPError
 22 | import xml.etree.ElementTree as ET
 23 | import zipfile
 24 | import zlib
 25 | 
 26 | def downloadfile( sourceurl, targetfname ):
 27 | 	mem_file = ""
 28 | 	good_read = False
 29 | 	xbrlfile = None
 30 | 	if os.path.isfile( targetfname ):
 31 | 		print( "Local copy already exists" )
 32 | 		return True
 33 | 	else:
 34 | 		print( "Downloading:", sourceurl )
 35 | 		try:
 36 | 			xbrlfile = urlopen( sourceurl )
 37 | 			try:
 38 | 				mem_file = xbrlfile.read()
 39 | 				good_read = True
 40 | 			finally:
 41 | 				xbrlfile.close()
 42 | 		except HTTPError as e:
 43 | 			print( "HTTP Error:", e.code )
 44 | 		except URLError as e:
 45 | 			print( "URL Error:", e.reason )
 46 | 		except TimeoutError as e:
 47 | 			print( "Timeout Error:", e.reason )
 48 | 		except socket.timeout:
 49 | 			print( "Socket Timeout Error" )
 50 | 		if good_read:
 51 | 			output = open( targetfname, 'wb' )
 52 | 			output.write( mem_file )
 53 | 			output.close()
 54 | 		return good_read
 55 | 
 56 | def SECdownload(year, month):
 57 | 	root = None
 58 | 	feedFile = None
 59 | 	feedData = None
 60 | 	good_read = False
 61 | 	itemIndex = 0
 62 | 	edgarFilingsFeed = 'http://www.sec.gov/Archives/edgar/monthly/xbrlrss-' + str(year) + '-' + str(month).zfill(2) + '.xml'
 63 | 	print( edgarFilingsFeed )
 64 | 	if not os.path.exists( "sec/" + str(year) ):
 65 | 		os.makedirs( "sec/" + str(year) )
 66 | 	if not os.path.exists( "sec/" + str(year) + '/' + str(month).zfill(2) ):
 67 | 		os.makedirs( "sec/" + str(year) + '/' + str(month).zfill(2) )
 68 | 	target_dir = "sec/" + str(year) + '/' + str(month).zfill(2) + '/'
 69 | 	try:
 70 | 		feedFile = urlopen( edgarFilingsFeed )
 71 | 		try:
 72 | 			feedData = feedFile.read()
 73 | 			good_read = True
 74 | 		finally:
 75 | 			feedFile.close()
 76 | 	except HTTPError as e:
 77 | 		print( "HTTP Error:", e.code )
 78 | 	except URLError as e:
 79 | 		print( "URL Error:", e.reason )
 80 | 	except TimeoutError as e:
 81 | 		print( "Timeout Error:", e.reason )
 82 | 	except socket.timeout:
 83 | 		print( "Socket Timeout Error" )
 84 | 	if not good_read:
 85 | 		print( "Unable to download RSS feed document for the month:", year, month )
 86 | 		return
 87 | 	# we have to unfortunately use both feedparser (for normal cases) and ET for old-style RSS feeds,
 88 | 	# because feedparser cannot handle the case where multiple xbrlFiles are referenced without enclosure
 89 | 	try:
 90 | 		root = ET.fromstring(feedData)
 91 | 	except ET.ParseError as perr:
 92 | 		print( "XML Parser Error:", perr )
 93 | 	feed = feedparser.parse( feedData )
 94 | 	try:
 95 | 		print( feed[ "channel" ][ "title" ] )
 96 | 	except KeyError as e:
 97 | 		print( "Key Error:", e )
 98 | 	# Process RSS feed and walk through all items contained
 99 | 	for item in feed.entries:
100 | 		print( item[ "summary" ], item[ "title" ], item[ "published" ] )
101 | 		try:
102 | 			# Identify ZIP file enclosure, if available
103 | 			enclosures = [ l for l in item[ "links" ] if l[ "rel" ] == "enclosure" ]
104 | 			if ( len( enclosures ) > 0 ):
105 | 				# ZIP file enclosure exists, so we can just download the ZIP file
106 | 				enclosure = enclosures[0]
107 | 				sourceurl = enclosure[ "href" ]
108 | 				cik = item[ "edgar_ciknumber" ]
109 | 				targetfname = target_dir+cik+'-'+sourceurl.split('/')[-1]
110 | 				retry_counter = 3
111 | 				while retry_counter > 0:
112 | 					good_read = downloadfile( sourceurl, targetfname )
113 | 					if good_read:
114 | 						break
115 | 					else:
116 | 						print( "Retrying:", retry_counter )
117 | 						retry_counter -= 1
118 | 			else:
119 | 				# We need to manually download all XBRL files here and ZIP them ourselves...
120 | 				linkname = item[ "link" ].split('/')[-1]
121 | 				linkbase = os.path.splitext(linkname)[0]
122 | 				cik = item[ "edgar_ciknumber" ]
123 | 				zipfname = target_dir+cik+'-'+linkbase+"-xbrl.zip"
124 | 				if os.path.isfile( zipfname ):
125 | 					print( "Local copy already exists" )
126 | 				else:
127 | 					edgarNamespace = {'edgar': 'http://www.sec.gov/Archives/edgar'}
128 | 					currentItem = list(root.iter( "item" ))[itemIndex]
129 | 					xbrlFiling = currentItem.find( "edgar:xbrlFiling", edgarNamespace )
130 | 					xbrlFilesItem = xbrlFiling.find( "edgar:xbrlFiles", edgarNamespace )
131 | 					xbrlFiles = xbrlFilesItem.findall( "edgar:xbrlFile", edgarNamespace )
132 | 					if not os.path.exists(  target_dir+"temp" ):
133 | 						os.makedirs( target_dir+"temp" )
134 | 					zf = zipfile.ZipFile( zipfname, "w" )
135 | 					try:
136 | 						for xf in xbrlFiles:
137 | 							xfurl = xf.get( "{http://www.sec.gov/Archives/edgar}url" )
138 | 							if xfurl.endswith( (".xml",".xsd") ):
139 | 								targetfname = target_dir+"temp/"+xfurl.split('/')[-1]
140 | 								retry_counter = 3
141 | 								while retry_counter > 0:
142 | 									good_read = downloadfile( xfurl, targetfname )
143 | 									if good_read:
144 | 										break
145 | 									else:
146 | 										print( "Retrying:", retry_counter )
147 | 										retry_counter -= 1
148 | 								zf.write( targetfname, xfurl.split('/')[-1], zipfile.ZIP_DEFLATED )
149 | 								os.remove( targetfname )
150 | 					finally:
151 | 						zf.close()
152 | 						os.rmdir( target_dir+"temp" )
153 | 		except KeyError as e:
154 | 			print( "Key Error:", e )
155 | 		finally:
156 | 			print( "----------" )
157 | 		itemIndex += 1
158 | 
159 | def main(argv):
160 | 	year = 2013
161 | 	month = 1
162 | 	from_year = 1999
163 | 	to_year = 1999
164 | 	year_range = False
165 | 	if not os.path.exists( "sec" ):
166 | 		os.makedirs( "sec" )
167 | 	socket.setdefaulttimeout(10)
168 | 	start_time  = time.time()
169 | 	try:
170 | 		opts, args = getopt.getopt(argv,"hy:m:f:t:",["year=","month=","from=","to="])
171 | 	except getopt.GetoptError:
172 | 		print( 'loadSECfilings -y <year> -m <month> | -f <from_year> -t <to_year>' )
173 | 		sys.exit(2)
174 | 	for opt, arg in opts:
175 | 		if opt == '-h':
176 | 			print( 'loadSECfilings -y <year> -m <month> | -f <from_year> -t <to_year>' )
177 | 			sys.exit()
178 | 		elif opt in ("-y", "--year"):
179 | 			year = int(arg)
180 | 		elif opt in ("-m", "--month"):
181 | 			month = int(arg)
182 | 		elif opt in ("-f", "--from"):
183 | 			from_year = int(arg)
184 | 			year_range = True
185 | 		elif opt in ("-t", "--to"):
186 | 			to_year = int(arg)
187 | 			year_range = True
188 | 	if year_range:
189 | 		if from_year == 1999:
190 | 			from_year = to_year
191 | 		if to_year == 1999:
192 | 			to_year = from_year
193 | 		for year in range( from_year, to_year+1 ):
194 | 			for month in range( 1, 12+1 ):
195 | 				SECdownload( year, month )
196 | 	else:
197 | 		SECdownload( year, month )
198 | 	end_time = time.time()
199 | 	print( "Elapsed time:", end_time - start_time, "seconds" )
200 | 
201 | if __name__ == "__main__":
202 | 	main(sys.argv[1:])
203 | 


--------------------------------------------------------------------------------
/extractRatios.py:
--------------------------------------------------------------------------------
  1 | # Copyright 2014 Altova GmbH
  2 | # 
  3 | # Licensed under the Apache License, Version 2.0 (the "License");
  4 | # you may not use this file except in compliance with the License.
  5 | # You may obtain a copy of the License at
  6 | # 
  7 | #	  http://www.apache.org/licenses/LICENSE-2.0
  8 | # 
  9 | # Unless required by applicable law or agreed to in writing, software
 10 | # distributed under the License is distributed on an "AS IS" BASIS,
 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | # See the License for the specific language governing permissions and
 13 | # limitations under the License.
 14 | 
 15 | import os, re, sys
 16 | from altova import xml, xsd, xbrl
 17 | 
 18 | sec_ns =  '/dei/'		# was: 'http://xbrl.sec.gov/dei/'
 19 | fasb_ns = '/us-gaap/'	# was: 'http://fasb.org/us-gaap/'
 20 | 
 21 | # We need to implement a simple locking class that allows us to avoid having 
 22 | # on_xbrl_valid called from multiple threads in RaptorXML at the same time.
 23 | 
 24 | if sys.platform == 'win32':
 25 | 	import msvcrt
 26 | 
 27 | 	# fcntl is not available under Windows, so we'll use a 
 28 | 	# file locking function from the msvcrt library instead...
 29 | 
 30 | 	class Lock:
 31 | 
 32 | 		def __init__(self, filename):
 33 | 			self.filename = filename
 34 | 			# This will create it if it does not exist already
 35 | 			self.handle = open(filename, 'w')
 36 | 
 37 | 		# Bitwise OR fcntl.LOCK_NB if you need a non-blocking lock
 38 | 		def acquire(self):
 39 | 			msvcrt.locking(self.handle.fileno(), msvcrt.LK_LOCK, 1)
 40 | 
 41 | 		def release(self):
 42 | 			msvcrt.locking(self.handle.fileno(), msvcrt.LK_UNLCK, 1)
 43 | 
 44 | 		def __del__(self):
 45 | 			self.handle.close()
 46 | 
 47 | else:
 48 | 	import fcntl
 49 | 
 50 | 	# Under Linux and MacOS we can use the fcntl library to implement
 51 | 	# the simple file locking mechanism...
 52 | 
 53 | 	class Lock:
 54 | 
 55 | 		def __init__(self, filename):
 56 | 			self.filename = filename
 57 | 			# This will create it if it does not exist already
 58 | 			self.handle = open(filename, 'w')
 59 | 
 60 | 		# Bitwise OR fcntl.LOCK_NB if you need a non-blocking lock
 61 | 		def acquire(self):
 62 | 			fcntl.flock(self.handle, fcntl.LOCK_EX)
 63 | 
 64 | 		def release(self):
 65 | 			fcntl.flock(self.handle, fcntl.LOCK_UN)
 66 | 
 67 | 		def __del__(self):
 68 | 			self.handle.close()
 69 | 
 70 | def camelToSpaces( label ):
 71 | 	# Utility for pretty-printing the labels
 72 | 	s1 = re.sub('(.)([A-Z][a-z]+)', r'\1 \2', label)
 73 | 	return re.sub('([a-z0-9])([A-Z])', r'\1 \2', s1)
 74 | 
 75 | def factFinder( instance, namespace, label ):
 76 | 	# Locate facts in the instance document by namespace and label, ignoring facts that have a context with a segment_element
 77 | 	l = []
 78 | 	for f in instance.items:
 79 | 		if f.qname.namespace_name.find( namespace ) and f.qname.local_name == label:
 80 | 			segment = None
 81 | 			try:
 82 | 				entElement = f.context.entity.element
 83 | 				for childElement in entElement.children:
 84 | 					if childElement.local_name=="segment":
 85 | 						segment = childElement
 86 | 			except:
 87 | 				pass
 88 | 			if segment==None:
 89 | 				l.append( f )
 90 | 	return l
 91 | 
 92 | def printFacts( facts, indent=1, targetDate=None ):
 93 | 	# Find the fact for the relevant target date and print it
 94 | 	factValue = 0
 95 | 	for fact in facts:
 96 | 		if targetDate==None or fact.context.period.instant == targetDate:
 97 | 			if fact.concept.item_type==fact.concept.MONETARY_ITEM_TYPE:
 98 | 				factValue = fact.effective_numeric_value
 99 | 				print( indent * "\t", camelToSpaces( fact.qname.local_name ).ljust(100-indent*8), "$", '{0:>16,}'.format( factValue ) )
100 | 			else:
101 | 				factValue = fact.normalized_value
102 | 				print( indent * "\t", camelToSpaces( fact.qname.local_name ).ljust(100-indent*8), factValue )
103 | 	return factValue
104 | 
105 | def on_xbrl_valid( job, instance ):
106 | 
107 | 	try:
108 | 		# a portable solution to get the tmp dir
109 | 		import tempfile
110 | 		tmp = tempfile.gettempdir()
111 | 		tmplk = os.path.join( tmp, "extract_ratios_lock.tmp" )
112 | 		lock = Lock(tmplk)
113 | 		lock.acquire()
114 | 
115 | 		# Create output CSV file if it doesn't exist yet
116 | 		if not os.path.isfile( "ratios.csv" ):
117 | 			with open("ratios.csv", "a") as ratiofile:
118 | 				ratiofile.write( "DocumentType,EntityName,CIK,PeriodEndDate,CurrentRatio,QuickRatio,CashRatio\n" )
119 | 				ratiofile.close()
120 | 
121 | 		# Extract some basic facts from the filing, such as the effective end-date for balance sheet etc.
122 | 		docEndDate = "2013-12-31"
123 | 		documentType = factFinder( instance, sec_ns, "DocumentType" )
124 | 		documentFiscalYearFocus = factFinder( instance, sec_ns, "DocumentFiscalYearFocus" )
125 | 		documentFiscalPeriodFocus = factFinder( instance, sec_ns, "DocumentFiscalPeriodFocus" )
126 | 		documentPeriodEndDate = factFinder( instance, sec_ns, "DocumentPeriodEndDate" )
127 | 		if len(documentPeriodEndDate) > 0:
128 | 			docEndDate = documentPeriodEndDate[0].normalized_value
129 | 
130 | 		# Extract Filer Name and other key data
131 | 		entityRegistrantName = factFinder( instance, sec_ns, "EntityRegistrantName" )
132 | 		entityCentralIndexKey = factFinder( instance, sec_ns, "EntityCentralIndexKey" )
133 | 		entityCommonStockSharesOutstanding = factFinder( instance, sec_ns, "EntityCommonStockSharesOutstanding" )
134 | 
135 | 		# Print information about filing and entity
136 | 		print( "Document and Entity Information:" )
137 | 		docType = printFacts( documentType )
138 | 		entityName = printFacts( entityRegistrantName )
139 | 		entityCIK = printFacts( entityCentralIndexKey )
140 | 		printFacts( documentPeriodEndDate )
141 | 		printFacts( documentFiscalPeriodFocus )
142 | 		printFacts( documentFiscalYearFocus )
143 | 
144 | 		if docType=="10-K" or docType=="10-Q":
145 | 			# Now let's calculate some useful ratios from the balance sheet
146 | 			print( "Analytical Ratios:" )
147 | 			print( "\tBalance Sheet:" )
148 | 
149 | 			# Current Ratio
150 | 			currentRatio = 0
151 | 			print( "\t\tCurrent Ratio = Current Assets / Current Liabilities:" )
152 | 			currentAssetsFacts = factFinder( instance, fasb_ns, "AssetsCurrent" )
153 | 			currentLiabilitiesFacts = factFinder( instance, fasb_ns, "LiabilitiesCurrent" )
154 | 			currentAssets = printFacts( currentAssetsFacts, 3, docEndDate )
155 | 			currentLiabilities = printFacts( currentLiabilitiesFacts, 3, docEndDate )
156 | 			if not currentLiabilities==0:
157 | 				currentRatio = currentAssets / currentLiabilities
158 | 			print( 3 * "\t", "Current Ratio = ".ljust(100-3*8), '{0:.2f}'.format( currentRatio ) )
159 | 
160 | 			# Quick Ratio
161 | 			quickRatio = 0
162 | 			print( "\t\tQuick Ratio = ( Cash + Short-Term Marketable Securities + Accounts Receivable ) / Current Liabilities:" )
163 | 			cashFacts = factFinder( instance, fasb_ns, "Cash" )
164 | 			if len(cashFacts)==0:
165 | 				cashFacts = factFinder( instance, fasb_ns, "CashAndCashEquivalentsAtCarryingValue" )
166 | 			if len(cashFacts)==0:
167 | 				cashFacts = factFinder( instance, fasb_ns, "CashCashEquivalentsAndShortTermInvestments" )
168 | 			marketableSecuritiesFacts = factFinder( instance, fasb_ns, "MarketableSecuritiesCurrent" )
169 | 			if len(marketableSecuritiesFacts)==0:
170 | 				marketableSecuritiesFacts = factFinder( instance, fasb_ns, "AvailableForSaleSecuritiesCurrent" )
171 | 			if len(marketableSecuritiesFacts)==0:
172 | 				marketableSecuritiesFacts = factFinder( instance, fasb_ns, "ShortTermInvestments" )
173 | 			if len(marketableSecuritiesFacts)==0:
174 | 				marketableSecuritiesFacts = factFinder( instance, fasb_ns, "OtherShortTermInvestments" )
175 | 			accountsReceivableFacts = factFinder( instance, fasb_ns, "AccountsReceivableNetCurrent" )
176 | 			currentLiabilitiesFacts = factFinder( instance, fasb_ns, "LiabilitiesCurrent" )
177 | 			cash = printFacts( cashFacts, 3, docEndDate )
178 | 			marketableSecurities = printFacts( marketableSecuritiesFacts, 3, docEndDate )
179 | 			accountsReceivable = printFacts( accountsReceivableFacts, 3, docEndDate )
180 | 			currentLiabilities = printFacts( currentLiabilitiesFacts, 3, docEndDate )
181 | 			if not currentLiabilities==0:
182 | 				quickRatio = ( cash + marketableSecurities + accountsReceivable ) / currentLiabilities
183 | 			print( 3 * "\t", "Quick Ratio = ".ljust(100-3*8), '{0:.2f}'.format( quickRatio ) )
184 | 
185 | 			# Cash Ratio
186 | 			cashRatio = 0
187 | 			print( "\t\tCash Ratio = ( Cash + Short-Term Marketable Securities ) / Current Liabilities:" )
188 | 			cash = printFacts( cashFacts, 3, docEndDate )
189 | 			marketableSecurities = printFacts( marketableSecuritiesFacts, 3, docEndDate )
190 | 			currentLiabilities = printFacts( currentLiabilitiesFacts, 3, docEndDate )
191 | 			if not currentLiabilities==0:
192 | 				cashRatio = ( cash + marketableSecurities ) / currentLiabilities
193 | 			print( 3 * "\t", "Cash Ratio = ".ljust(100-3*8), '{0:.2f}'.format( cashRatio ) )
194 | 
195 | 
196 | 			# Append ratios to a CSV file for further analysis
197 | 			with open("ratios.csv", "a") as ratiofile:
198 | 				ratiofile.write( docType + ',"' + entityName + '",' + entityCIK + "," + docEndDate + "," + '{0:.2f}'.format( currentRatio ) + "," + '{0:.2f}'.format( quickRatio ) + "," + '{0:.2f}'.format( cashRatio ) + "\n" )
199 | 				ratiofile.close()
200 | 
201 | 	finally:
202 | 		lock.release()
203 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
  1 | Apache License
  2 |                            Version 2.0, January 2004
  3 |                         http://www.apache.org/licenses/
  4 | 
  5 |    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
  6 | 
  7 |    1. Definitions.
  8 | 
  9 |       "License" shall mean the terms and conditions for use, reproduction,
 10 |       and distribution as defined by Sections 1 through 9 of this document.
 11 | 
 12 |       "Licensor" shall mean the copyright owner or entity authorized by
 13 |       the copyright owner that is granting the License.
 14 | 
 15 |       "Legal Entity" shall mean the union of the acting entity and all
 16 |       other entities that control, are controlled by, or are under common
 17 |       control with that entity. For the purposes of this definition,
 18 |       "control" means (i) the power, direct or indirect, to cause the
 19 |       direction or management of such entity, whether by contract or
 20 |       otherwise, or (ii) ownership of fifty percent (50%) or more of the
 21 |       outstanding shares, or (iii) beneficial ownership of such entity.
 22 | 
 23 |       "You" (or "Your") shall mean an individual or Legal Entity
 24 |       exercising permissions granted by this License.
 25 | 
 26 |       "Source" form shall mean the preferred form for making modifications,
 27 |       including but not limited to software source code, documentation
 28 |       source, and configuration files.
 29 | 
 30 |       "Object" form shall mean any form resulting from mechanical
 31 |       transformation or translation of a Source form, including but
 32 |       not limited to compiled object code, generated documentation,
 33 |       and conversions to other media types.
 34 | 
 35 |       "Work" shall mean the work of authorship, whether in Source or
 36 |       Object form, made available under the License, as indicated by a
 37 |       copyright notice that is included in or attached to the work
 38 |       (an example is provided in the Appendix below).
 39 | 
 40 |       "Derivative Works" shall mean any work, whether in Source or Object
 41 |       form, that is based on (or derived from) the Work and for which the
 42 |       editorial revisions, annotations, elaborations, or other modifications
 43 |       represent, as a whole, an original work of authorship. For the purposes
 44 |       of this License, Derivative Works shall not include works that remain
 45 |       separable from, or merely link (or bind by name) to the interfaces of,
 46 |       the Work and Derivative Works thereof.
 47 | 
 48 |       "Contribution" shall mean any work of authorship, including
 49 |       the original version of the Work and any modifications or additions
 50 |       to that Work or Derivative Works thereof, that is intentionally
 51 |       submitted to Licensor for inclusion in the Work by the copyright owner
 52 |       or by an individual or Legal Entity authorized to submit on behalf of
 53 |       the copyright owner. For the purposes of this definition, "submitted"
 54 |       means any form of electronic, verbal, or written communication sent
 55 |       to the Licensor or its representatives, including but not limited to
 56 |       communication on electronic mailing lists, source code control systems,
 57 |       and issue tracking systems that are managed by, or on behalf of, the
 58 |       Licensor for the purpose of discussing and improving the Work, but
 59 |       excluding communication that is conspicuously marked or otherwise
 60 |       designated in writing by the copyright owner as "Not a Contribution."
 61 | 
 62 |       "Contributor" shall mean Licensor and any individual or Legal Entity
 63 |       on behalf of whom a Contribution has been received by Licensor and
 64 |       subsequently incorporated within the Work.
 65 | 
 66 |    2. Grant of Copyright License. Subject to the terms and conditions of
 67 |       this License, each Contributor hereby grants to You a perpetual,
 68 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 69 |       copyright license to reproduce, prepare Derivative Works of,
 70 |       publicly display, publicly perform, sublicense, and distribute the
 71 |       Work and such Derivative Works in Source or Object form.
 72 | 
 73 |    3. Grant of Patent License. Subject to the terms and conditions of
 74 |       this License, each Contributor hereby grants to You a perpetual,
 75 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 76 |       (except as stated in this section) patent license to make, have made,
 77 |       use, offer to sell, sell, import, and otherwise transfer the Work,
 78 |       where such license applies only to those patent claims licensable
 79 |       by such Contributor that are necessarily infringed by their
 80 |       Contribution(s) alone or by combination of their Contribution(s)
 81 |       with the Work to which such Contribution(s) was submitted. If You
 82 |       institute patent litigation against any entity (including a
 83 |       cross-claim or counterclaim in a lawsuit) alleging that the Work
 84 |       or a Contribution incorporated within the Work constitutes direct
 85 |       or contributory patent infringement, then any patent licenses
 86 |       granted to You under this License for that Work shall terminate
 87 |       as of the date such litigation is filed.
 88 | 
 89 |    4. Redistribution. You may reproduce and distribute copies of the
 90 |       Work or Derivative Works thereof in any medium, with or without
 91 |       modifications, and in Source or Object form, provided that You
 92 |       meet the following conditions:
 93 | 
 94 |       (a) You must give any other recipients of the Work or
 95 |           Derivative Works a copy of this License; and
 96 | 
 97 |       (b) You must cause any modified files to carry prominent notices
 98 |           stating that You changed the files; and
 99 | 
100 |       (c) You must retain, in the Source form of any Derivative Works
101 |           that You distribute, all copyright, patent, trademark, and
102 |           attribution notices from the Source form of the Work,
103 |           excluding those notices that do not pertain to any part of
104 |           the Derivative Works; and
105 | 
106 |       (d) If the Work includes a "NOTICE" text file as part of its
107 |           distribution, then any Derivative Works that You distribute must
108 |           include a readable copy of the attribution notices contained
109 |           within such NOTICE file, excluding those notices that do not
110 |           pertain to any part of the Derivative Works, in at least one
111 |           of the following places: within a NOTICE text file distributed
112 |           as part of the Derivative Works; within the Source form or
113 |           documentation, if provided along with the Derivative Works; or,
114 |           within a display generated by the Derivative Works, if and
115 |           wherever such third-party notices normally appear. The contents
116 |           of the NOTICE file are for informational purposes only and
117 |           do not modify the License. You may add Your own attribution
118 |           notices within Derivative Works that You distribute, alongside
119 |           or as an addendum to the NOTICE text from the Work, provided
120 |           that such additional attribution notices cannot be construed
121 |           as modifying the License.
122 | 
123 |       You may add Your own copyright statement to Your modifications and
124 |       may provide additional or different license terms and conditions
125 |       for use, reproduction, or distribution of Your modifications, or
126 |       for any such Derivative Works as a whole, provided Your use,
127 |       reproduction, and distribution of the Work otherwise complies with
128 |       the conditions stated in this License.
129 | 
130 |    5. Submission of Contributions. Unless You explicitly state otherwise,
131 |       any Contribution intentionally submitted for inclusion in the Work
132 |       by You to the Licensor shall be under the terms and conditions of
133 |       this License, without any additional terms or conditions.
134 |       Notwithstanding the above, nothing herein shall supersede or modify
135 |       the terms of any separate license agreement you may have executed
136 |       with Licensor regarding such Contributions.
137 | 
138 |    6. Trademarks. This License does not grant permission to use the trade
139 |       names, trademarks, service marks, or product names of the Licensor,
140 |       except as required for reasonable and customary use in describing the
141 |       origin of the Work and reproducing the content of the NOTICE file.
142 | 
143 |    7. Disclaimer of Warranty. Unless required by applicable law or
144 |       agreed to in writing, Licensor provides the Work (and each
145 |       Contributor provides its Contributions) on an "AS IS" BASIS,
146 |       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 |       implied, including, without limitation, any warranties or conditions
148 |       of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 |       PARTICULAR PURPOSE. You are solely responsible for determining the
150 |       appropriateness of using or redistributing the Work and assume any
151 |       risks associated with Your exercise of permissions under this License.
152 | 
153 |    8. Limitation of Liability. In no event and under no legal theory,
154 |       whether in tort (including negligence), contract, or otherwise,
155 |       unless required by applicable law (such as deliberate and grossly
156 |       negligent acts) or agreed to in writing, shall any Contributor be
157 |       liable to You for damages, including any direct, indirect, special,
158 |       incidental, or consequential damages of any character arising as a
159 |       result of this License or out of the use or inability to use the
160 |       Work (including but not limited to damages for loss of goodwill,
161 |       work stoppage, computer failure or malfunction, or any and all
162 |       other commercial damages or losses), even if such Contributor
163 |       has been advised of the possibility of such damages.
164 | 
165 |    9. Accepting Warranty or Additional Liability. While redistributing
166 |       the Work or Derivative Works thereof, You may choose to offer,
167 |       and charge a fee for, acceptance of support, warranty, indemnity,
168 |       or other liability obligations and/or rights consistent with this
169 |       License. However, in accepting such obligations, You may act only
170 |       on Your own behalf and on Your sole responsibility, not on behalf
171 |       of any other Contributor, and only if You agree to indemnify,
172 |       defend, and hold each Contributor harmless for any liability
173 |       incurred by, or claims asserted against, such Contributor by reason
174 |       of your accepting any such warranty or additional liability.
175 | 
176 |    END OF TERMS AND CONDITIONS
177 | 
178 |    APPENDIX: How to apply the Apache License to your work.
179 | 
180 |       To apply the Apache License to your work, attach the following
181 |       boilerplate notice, with the fields enclosed by brackets "{}"
182 |       replaced with your own identifying information. (Don't include
183 |       the brackets!)  The text should be enclosed in the appropriate
184 |       comment syntax for the file format. We also recommend that a
185 |       file or class name and description of purpose be included on the
186 |       same "printed page" as the copyright notice for easier
187 |       identification within third-party archives.
188 | 
189 |    Copyright {yyyy} {name of copyright owner}
190 | 
191 |    Licensed under the Apache License, Version 2.0 (the "License");
192 |    you may not use this file except in compliance with the License.
193 |    You may obtain a copy of the License at
194 | 
195 |        http://www.apache.org/licenses/LICENSE-2.0
196 | 
197 |    Unless required by applicable law or agreed to in writing, software
198 |    distributed under the License is distributed on an "AS IS" BASIS,
199 |    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 |    See the License for the specific language governing permissions and
201 |    limitations under the License.
202 | 
203 | 


--------------------------------------------------------------------------------