├── .gitignore
├── README.md
├── bin
    ├── index_lookup_local
    ├── index_lookup_remote
    └── remote_copy
├── docs
    ├── data_block.png
    ├── file_format.graffle
    │   ├── data.plist
    │   └── image1.pdf
    ├── file_overview.png
    ├── header.png
    ├── index_block.png
    ├── project_header.png
    └── tree.png
└── lib
    ├── __init__.py
    ├── adaptor.py
    ├── pbtree.py
    ├── prefix.py
    ├── sorted_urls
    ├── test.py
    ├── test_map.py
    └── test_pbtree.py


/.gitignore:
--------------------------------------------------------------------------------
1 | *.py[co]
2 | .DS_Store
3 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | ![Trivio Logo](/docs/project_header.png?raw=true)
  2 | 
  3 | ==================
  4 | 
  5 | The Common Crawl data  is composed of billions of  pages randomly crawled from the internet. Each page is archived with thousands of other pages into an archive file, commonly known as an ARC file. There are hundreds of thousands of these archives and the pages are stored essentially randomly distributed and unordered within these archives. 
  6 | 
  7 | 
  8 | Without an index, even if you know a pages URL, you're forced to download, uncompress and each of the archives until you locate the pages you're are interested in.
  9 | 
 10 | Using the index described here, you can find all the archive files that contain pages for a give URL prefix, subdomain or top level domain with no more than 3 small network requests. There is also a utility script bin/remote_copy 
 11 | (documented at the bottom of this readme) which uses the index to copy all the webpages from a specified list of domains 
 12 | to an S3 location of your choosing.
 13 | 
 14 | 
 15 | Challenges
 16 | ---------
 17 | To understand the design of the index let's first look at the challenges unique to the Common Crawl Project. 
 18 | 
 19 | Specifically:
 20 | 
 21 | 
 22 | 1. It's huge
 23 |   * 5 Billion unique URLs
 24 |   * Average URL is 66 bytes
 25 |   * A pointer to an individual page requires an additional 28 bytes 
 26 |    (see file format for details)
 27 |    
 28 |   Just storing this information  uncompressed, requires a file greater
 29 |   than 437 GB in size. (5x 10^9 x (66+28)).
 30 |   
 31 | 2. The size alone prevents constructing the entire index in memory
 32 | 
 33 | 3. Or.. even on one modest machine
 34 | 
 35 | 4. There's a large community with a variety of tools interested in accessing the index.
 36 | 
 37 | 5. Common Crawl is a non profit, it's especially important to keep processing and hosting costs down while ensuring the data is highly available.
 38 | 
 39 | Since 2012, the Common Crawl data has been hosted for free by Amazon on the Amazon Public Data Sets. The Amazon Public Data Set program is a boon to everyone who uses data - it lowers the overhead costs for organizations that want to share data, lowers the cost for users to download open data, and makes valuable data sets more discoverable.
 40 | 
 41 | Putting this altogether leads to the following goals.
 42 | 
 43 | 
 44 | Goals:
 45 | ------
 46 | 
 47 |   * Store and share the entire index from S3 without any other services
 48 |   * The entire index should be utilizable without downloading the entire thing.
 49 |   * The number of network calls needed should be minimized.
 50 |   * You can search for any URL, URL prefix, subdomain or top-level domain.
 51 |   * Once you've determined the approximate location of the URL you're
 52 |   interested in you can sequentially read other similar urls.
 53 |   * It should be easy to access the index from any programming language *This the main reason we opted to roll our own format rather than rely on a third party library
 54 | 
 55 | 
 56 | File Format
 57 | ------------
 58 | 
 59 | The file format is based on a [Prefixed B-tree](http://ict.pue.udlap.mx/people/carlos/is215/papers/p11-bayer.pdf). The proceeding link will take you to a paper that gives you an in depth overview of this data structure, however, hopefully, the information we provide here will be enough for your to utilize the index without requiring you to read the whole paper. 
 60 | 
 61 | Conceptually the index is organized into a b+ tree like this.
 62 | 
 63 | ![Tree](/docs/tree.png?raw=true)
 64 | 
 65 | To access any given URL in the index, you  start by reading the root block in the tree and then follow the pointers to zero or more other index blocks and finally to a data block. The urls is the data block are stored in lexicographic order. So for  a url of `http://example.com/page1.html` will come before `http://example.com/page2.html`. Because of this property you
 66 | can find all the pages that share a common prefix by subsequently reading
 67 | each url in the data portion of the file.
 68 | 
 69 | 
 70 | ###File overview
 71 | The entire index plus data are stored in one file that has 3 major parts. **The header**, **index blocks** and **data blocks** as depicted below.
 72 | 
 73 | ![File Overview](/docs/file_overview.png?raw=true)
 74 | 
 75 | 
 76 | ###Header
 77 | The header is exactly 8 bytes long. The first 4 bytes represents the **block size** used in the file and the second 4 bytes represent the number of index blocks or **block count** contained in the file.
 78 | 
 79 | All numbers are encoded in little-endian order
 80 | 
 81 | ![Header](/docs/header.png?raw=true)
 82 | 
 83 | Once you have the block size and block count you can randomly access any block by following the instructions in the Operations section of this guide.
 84 | 
 85 | Interpreting a block depends on whether it is a **Index block** or a **Data block**.
 86 | 
 87 | 
 88 | ###Index block
 89 | Any block number that is less than the block count in the header is considered an index block. It is interpreted as follows:
 90 | 
 91 | ![Index Block](/docs/index_block.png?raw=true)
 92 | 
 93 | An index block always starts with 4 byte <block number> then one or more <prefix> <null> <block number> triplets until the next triplet can not fit within the block, at which point it is padded with additional <null> bytes.
 94 |   
 95 | As a result should you ever encounter a <null> immediately after reading a <block number> you know you've reached the end of the block.
 96 | 
 97 | See the section on searching the index to see how to use the index blocks to find the first data block appropriate for your search.
 98 | 
 99 | ###Data block
100 | 
101 | Data blocks consist of oner or more <url> <null> <location pointer> collectively known as an **item**. There are a variable number of items in a block based on the length of the url. 
102 | 
103 | Just like index blocks, data blocks are padded with <null> bytes, when during construction, the next item  can not fit within the given block size.
104 | 
105 | ![Data Block](/docs/data_block.png?raw=true)
106 | 
107 | 
108 | A <url> consists of one or more characters terminated by the null byte. 
109 |   
110 | The location pointer is 32 bytes long and can be interpreted as follows. The first 8 bytes represents the segment id, the next 8 bytes represents the ARC file creation date, followed by 4 bytes that represent the ARC file partition, followed by 8 bytes that represent the offset within the ARC file and then
111 | finally the last 4 bytes represent the size of compressed data stored inside the ARC file.
112 | 
113 | See the the section on retrieving a page, to put all this information together to access a page.
114 | 
115 | 
116 | Operations
117 | ----------
118 | 
119 | ###Read a block
120 | 
121 | Once you've read the header you can use this information to randomly read any block given it's block number. Keep in mind the size of the header is exactly 8 bytes long.
122 | 
123 | First determine the offset of the block in the file
124 | 
125 | ```
126 | block offset = (block number * block size) + header size
127 | ```
128 | 
129 | Then read the range of bytes starting from the *block offset* plus the size of a single block.
130 | 
131 | For example imagine the block size is 65536 bytes long and you wanted to read  block number 2. You'd first calculate tho block offset which is (2 x 65536) + 8 = 131080. Then you would read the block. 
132 | 
133 | If you were making an http request to S3 this would be done using the HTTP Range header as follows
134 | 
135 | ```
136 | Range: bytes=131080-196616
137 | ```
138 | 
139 | If you have download the index you can simply seek to position 131080 in the file and then read the next 65536 bytes.
140 | 
141 | ###Jumping directly to the start of the data block
142 | 
143 | Any block number that is larger than the index block count in the header, is  a data block. You can skip over all the index blocks to the very first data block by following the instructions in the Reading a Block  and using the block count as the block number.
144 | 
145 | ###Searching the Index
146 | 
147 | To find a URL in the index or find all the URLs that start with a common prefix, we'll call this a "target", start by reading block 0 also known as the **root block**
148 | 
149 | 1. Read through each prefix found in the index block until you find the first prefix that is lexicographically greater than or equal to the given target.
150 | 
151 | 2. The next block number you need to read is the number that was found immediately to the left of the prefix that was greater than or equal to your target. If your target is bigger than every prefix in the block use the last block number in the block.
152 | 
153 | 3. Repeat steps 1 through 3 until the block number is greater than the block count that is stored in the header. You've now found the first data block that could possibly contain your target.
154 | 
155 | 
156 | Once you've found the first data block.
157 | 
158 | 1. Retrieve the given block number
159 | 2. Read all characters up to the first <null> byte
160 | 3. Read the  32 byte location pointer
161 | 4. Repeat steps 1-3; as long as the URL is greater than the target or you reach the end of the block. If you reach the end of the block without finding the target, it is not in the index.
162 | 5. Now read each item out of the data block, until each item url no longer starts with the target string.
163 | 
164 | 
165 | ###Retrieving a page
166 | 
167 | The location pointer represents 5 numbers:
168 | 
169 | * segment id
170 | * file date
171 | * partition
172 | * file offset
173 | * compressed sized
174 | 
175 | Using the first 3 numbers you can construct the URL of the arc file that contains the page you are interested in.
176 | 
177 | ```
178 |  s3://aws-publicdatasets/common-crawl/parse-output/segment/[segment id]/[file date]_[partition].arc.gz
179 | ```
180 | 
181 | The file offset and compressed size can be used to fetch the compressed chunk from the arc file, without
182 | downloading the entire arc file.
183 | 
184 | Here is an example using the boto library in python to retreive and uncompress the chunk.
185 | 
186 | 
187 | 
188 | ```python
189 | def arc_file(s3, bucket, info):
190 | 
191 |   bucket = s3.lookup(bucket)
192 |   keyname = "/common-crawl/parse-output/segment/{arcSourceSegmentId}/{arcFileDate}_{arcFileParition}.
193 | arc.gz".format(**info)
194 |   key = bucket.lookup(keyname)
195 |   
196 |   start = info['arcFileOffset']
197 |   end = start + info['compressedSize'] - 1
198 |   
199 |   headers={'Range' : 'bytes={}-{}'.format(start, end)}
200 |   
201 |   chunk = StringIO(
202 |     key.get_contents_as_string(headers=headers)
203 |   )
204 |   
205 |   return GzipFile(fileobj=chunk).read()
206 | ```
207 |  
208 | ### Using the remote_copy utility script
209 | 
210 | bin/remote_copy is a utility script which takes a domain or comma-separated list of domains, 
211 | uses the index to find the webpages in the crawl belonging to the given domains, 
212 | downloads them (to get just the bytes corresponding to the webpages from the 100 MB segment files), 
213 | and reuploads them to a specified S3 location. It uses python multiprocessing to make several 
214 | requests in parallel. As this still takes a lot of bandwidth, it is recommended that you run it 
215 | on an m1.xlarge EC2 instance for a fast connection to S3.
216 | 
217 | The "check" command will give stats such as number of webpages and total file size for a list of domains. 
218 | The "copy" command will download the webpages from aws-publicdatasets and reupload to a specified S3 location.
219 | 
220 | Example usage:
221 | 
222 |     chmod +x bin/remote_copy
223 |     export AWS_ACCESS_KEY=<your aws access key>
224 |     export AWS_SECRET_KEY=<your aws secret key>
225 |     bin/remote_copy check "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus"
226 |     bin/remote_copy copy "com.nytimes.blogs.fivethirtyeight, com.nytimes.blogs.thecaucus" --bucket your-output-bucket --key common_crawl/blogs_crawl --parallel 4 
227 | 
228 | 


--------------------------------------------------------------------------------
/bin/index_lookup_local:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | # Copyright [2012] [Triv.io, Scott Robertson]
 4 | # 
 5 | #   Licensed under the Apache License, Version 2.0 (the "License");
 6 | #   you may not use this file except in compliance with the License.
 7 | #   You may obtain a copy of the License at
 8 | # 
 9 | #       http://www.apache.org/licenses/LICENSE-2.0
10 | # 
11 | #   Unless required by applicable law or agreed to in writing, software
12 | #   distributed under the License is distributed on an "AS IS" BASIS,
13 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | #   See the License for the specific language governing permissions and
15 | #   limitations under the License.
16 | # 
17 | 
18 | import sys
19 | import mmap
20 | 
21 | from  os.path import join, dirname
22 | sys.path.append(join(dirname(__file__), '..'))
23 | 
24 | from lib.pbtree import PBTreeDictReader
25 | 
26 | 
27 | def main():
28 | 
29 |   stream = open(sys.argv[1], 'r+')
30 |   stream = mmap.mmap(stream.fileno(),0)
31 |   reader = PBTreeDictReader(
32 |     stream,
33 |     value_format="<QQIQ", 
34 |     item_keys=(
35 |       'arcSourceSegmentId',
36 |       'arcFileDate',
37 |       'arcFilePartition',
38 |       'arcFileOffset',
39 |       'compressedSize'
40 |     )
41 |   )
42 | 
43 |   print reader.count_levels()
44 | 
45 |   for url, d in reader.itemsiter(sys.argv[2]):
46 |     print url,d
47 |     
48 | if __name__ == '__main__':
49 |   main()
50 | 


--------------------------------------------------------------------------------
/bin/index_lookup_remote:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | # Copyright [2012] [Triv.io, Scott Robertson]
 4 | # 
 5 | #   Licensed under the Apache License, Version 2.0 (the "License");
 6 | #   you may not use this file except in compliance with the License.
 7 | #   You may obtain a copy of the License at
 8 | # 
 9 | #       http://www.apache.org/licenses/LICENSE-2.0
10 | # 
11 | #   Unless required by applicable law or agreed to in writing, software
12 | #   distributed under the License is distributed on an "AS IS" BASIS,
13 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | #   See the License for the specific language governing permissions and
15 | #   limitations under the License.
16 | # 
17 | 
18 | 
19 | import sys
20 | import struct
21 | from  os.path import join, dirname
22 | sys.path.append(join(dirname(__file__), '..'))
23 | 
24 | from lib.pbtree import IndexBlockReader, PBTreeDictReader
25 | 
26 | import boto
27 | 
28 | class BotoMap(object):
29 |   def __init__(self, bucket, key_name, access_key=None, access_secret=None ):
30 |     if access_key and access_secret:
31 |       self.conn  = boto.connect_s3(access_key,access_secret)
32 |     else:
33 |       self.conn  = boto.connect_s3(anon=True)
34 |     bucket = self.conn.lookup(bucket)
35 |     self.key = bucket.lookup(key_name)
36 |     self.block_size = 2**16
37 |     self.cached_block = -1
38 |   
39 |   def __getitem__(self, i):
40 |     if isinstance(i, slice):
41 |       start = i.start
42 |       end = i.stop - 1
43 |     else:
44 |       start = i
45 |       end = start + 1
46 |       
47 |     return self.fetch(start,end)
48 | 
49 |   def fetch(self, start, end):
50 |     try:
51 |       return self.key.get_contents_as_string(
52 |         headers={'Range' : 'bytes={0}-{1}'.format(start, end)}
53 |       )
54 |     except boto.exception.S3ResponseError, e:
55 |       # invalid range, we've reached the end of the file
56 |       if e.status == 416:
57 |         return ''
58 | 
59 | 
60 |     
61 | if __name__ == "__main__":
62 |   mmap = BotoMap(
63 |     'aws-publicdatasets',
64 |     '/common-crawl/projects/url-index/url-index.1356128792'
65 | #    '<YOUR AWS KEY>',
66 | #    '<YOUR AWS SECRET>',
67 |   )
68 |   
69 |   reader = PBTreeDictReader(
70 |     mmap,
71 |     value_format="<QQIQI", 
72 |     item_keys=(
73 |       'arcSourceSegmentId',
74 |       'arcFileDate',
75 |       'arcFilePartition',
76 |       'arcFileOffset',
77 |       'compressedSize'
78 |     )
79 |   )
80 |   
81 |   try:
82 |     for url, d in reader.itemsiter(sys.argv[1]):
83 |       print url,d
84 |   except KeyboardInterrupt:
85 |     pass
86 | 


--------------------------------------------------------------------------------
/bin/remote_copy:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python
  2 | #
  3 | #   Licensed under the Apache License, Version 2.0 (the "License");
  4 | #   you may not use this file except in compliance with the License.
  5 | #   You may obtain a copy of the License at
  6 | # 
  7 | #       http://www.apache.org/licenses/LICENSE-2.0
  8 | # 
  9 | #   Unless required by applicable law or agreed to in writing, software
 10 | #   distributed under the License is distributed on an "AS IS" BASIS,
 11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | #   See the License for the specific language governing permissions and
 13 | #   limitations under the License.
 14 | # 
 15 | 
 16 | import argparse
 17 | import boto
 18 | import os
 19 | import sys
 20 | import struct
 21 | import tempfile
 22 | import time
 23 | import traceback
 24 | 
 25 | from datetime import timedelta
 26 | from itertools import chain
 27 | from multiprocessing import Pool, Queue
 28 | 
 29 | sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
 30 | 
 31 | from lib.pbtree import IndexBlockReader, PBTreeDictReader
 32 | 
 33 | class BotoMap(object):
 34 |     def __init__(self, s3, bucket, key_name):
 35 |         self.key = bucket.get_key(key_name)
 36 |         self.block_size = 2**16
 37 |         self.cached_block = -1
 38 | 
 39 |     def __getitem__(self, i):
 40 |         if isinstance(i, slice):
 41 |             start = i.start
 42 |             end = i.stop - 1
 43 |         else:
 44 |             start = i
 45 |             end = start + 1
 46 |         
 47 |         return self.fetch(start,end)
 48 | 
 49 |     def fetch(self, start, end):
 50 |         try:
 51 |             return self.key.get_contents_as_string(
 52 |               headers={'Range' : 'bytes={}-{}'.format(start, end)}
 53 |             )
 54 |         except boto.exception.S3ResponseError, e:
 55 |             # invalid range, we've reached the end of the file
 56 |             if e.status == 416:
 57 |                 return ''
 58 | 
 59 | # See copy_arc_files_init below
 60 | # Arguments except for index_results are passed using copy_arc_files_init
 61 | # so that copy_arc_files can be used with multiprocessing pool.map()
 62 | def copy_arc_files(index_results):
 63 |     try:
 64 |         s3_anon = boto.connect_s3(anon=True)
 65 |         s3_user = boto.connect_s3(copy_arc_files.access_key, copy_arc_files.secret_key)
 66 | 
 67 |         src_bucket_name = 'aws-publicdatasets'
 68 |         src_bucket = s3_anon.lookup(src_bucket_name)
 69 | 
 70 |         dest_bucket = s3_user.lookup(copy_arc_files.dest_bucket_name)
 71 | 
 72 |         src_keystem = '/common-crawl/parse-output/segment/{arcSourceSegmentId}/{arcFileDate}_{arcFilePartition}.arc.gz'
 73 |         
 74 |         chunk = tempfile.NamedTemporaryFile('ab+')
 75 |         src_key_cache = {}
 76 | 
 77 |         for i, key_info in enumerate(index_results):
 78 |             src_keyname = src_keystem.format(**key_info)
 79 | 
 80 |             src_key = None
 81 |             if src_keyname in src_key_cache:
 82 |                 src_key = src_key_cache[src_keyname]
 83 |             else:
 84 |                 src_key = src_bucket.lookup(src_keyname)
 85 |                 src_key_cache[src_keyname] = src_key
 86 | 
 87 |             start = key_info['arcFileOffset']
 88 |             end = start + key_info['compressedSize'] - 1
 89 |             headers={'Range' : 'bytes={}-{}'.format(start, end)}
 90 | 
 91 |             if src_key:
 92 |                 src_key.get_contents_to_file(chunk, headers=headers)
 93 |             else:
 94 |                 copy_arc_files.progress_queue.put(("warning", "WARNING: could not find key " + src_keyname))
 95 |             copy_arc_files.progress_queue.put(("download", key_info['compressedSize']))
 96 | 
 97 |         dest_keyname = '/' + copy_arc_files.dest_keystem + '/' + str(os.getpid()) + ".gz"
 98 |         dest_key = dest_bucket.new_key(dest_keyname)
 99 | 
100 |         upload_bytes = chunk.tell()
101 |         chunk.seek(0, 0)
102 |         dest_key.set_contents_from_file(chunk, replace=True)
103 |         chunk.close()
104 |         copy_arc_files.progress_queue.put(("upload", upload_bytes))
105 |     except:
106 |         print ""
107 |         print "ERROR, pid=" + str(os.getpid())
108 |         print traceback.format_exc()
109 |         copy_arc_files.progress_queue.put(("error", None))
110 | 
111 | def copy_arc_files_init(access_key, secret_key, bucket_name, keystem, queue):
112 |     copy_arc_files.access_key = access_key
113 |     copy_arc_files.secret_key = secret_key
114 |     copy_arc_files.dest_bucket_name = bucket_name
115 |     copy_arc_files.dest_keystem = keystem
116 |     copy_arc_files.progress_queue = queue
117 | 
118 | def partition(li, n):
119 |     division = len(li) / float(n)
120 |     for i in xrange(n):
121 |         yield li[int(round(division * i)): int(round(division * (i + 1)))]
122 | 
123 | if __name__ == "__main__":
124 |     argparser = argparse.ArgumentParser(description="Copy common-crawl webpages from a given domain or comma-delimited list of domains to an s3 location specified by the user.")
125 |     argparser.add_argument('command', choices=['check', 'copy'], help='check to list # of webpages in index for given domain; copy to copy those webpages to an s3 location')
126 |     argparser.add_argument('domains', help='domain or comma-delimited list of domains to check/copy from the index')
127 |     argparser.add_argument('-b', '--bucket', help='webpages stored in s3://<bucket>/<key>/<process-id>.gz (multiple files if parallel > 1)')
128 |     argparser.add_argument('-k', '--key', help='webpages stored in s3://<bucket>/<key>/<process-id>.gz (multiple files if parallel > 1)')
129 |     argparser.add_argument('-p', '--parallel', type=int, default=4, help='how many parallel processes to run (default = 4)')
130 |     argparser.add_argument('-O', '--aws-access-key', default=os.environ.get('AWS_ACCESS_KEY', None), help='AWS Access Key ID. Defaults to the value of the AWS_ACCESS_KEY environment variable (if set).')
131 |     argparser.add_argument('-W', '--aws-secret-key', default=os.environ.get('AWS_SECRET_KEY', None), help='AWS Secret Access Key. Defaults to the value of the AWS_SECRET_KEY environment variable (if set).')
132 |     args = argparser.parse_args()
133 | 
134 |     if not args.aws_access_key:
135 |         argparser.error("Error: neither --aws-access-key option nor AWS_ACCESS_KEY environment variable set")
136 |     if not args.aws_secret_key:
137 |         argparser.error("Error: neither --aws-secret-key option nor AWS_SECRET_KEY environment variable set")
138 | 
139 |     if args.command == "copy":
140 |         if not args.bucket:
141 |             argparser.error("Error: --bucket option is required for copy operation")
142 |         if not args.key:
143 |             argparser.error("Error: --key option is required for copy operation")
144 | 
145 |     s3_anon = boto.connect_s3(anon=True)
146 | 
147 |     src_bucket_name = 'aws-publicdatasets'
148 |     src_bucket = s3_anon.lookup(src_bucket_name)
149 | 
150 |     src_keystem = '/common-crawl/parse-output/segment/{arcSourceSegmentId}/{arcFileDate}_{arcFilePartition}.arc.gz'
151 | 
152 |     mmap = BotoMap(s3_anon, src_bucket, '/common-crawl/projects/url-index/url-index.1356128792')
153 | 
154 |     reader = PBTreeDictReader(
155 |         mmap,
156 |         value_format="<QQIQI", 
157 |         item_keys=(
158 |             'arcSourceSegmentId',
159 |             'arcFileDate',
160 |             'arcFilePartition',
161 |             'arcFileOffset',
162 |             'compressedSize'
163 |         )
164 |     )
165 | 
166 |     src_keys = set()
167 |     index_results = []
168 | 
169 |     domains = [reader.itemsiter(s.strip()) for s in args.domains.split(',')]
170 |     for url, index_data in chain.from_iterable(domains):
171 |         src_keys.add(src_keystem.format(**index_data))
172 |         index_results.append(index_data)
173 | 
174 |     num_files = len(src_keys)
175 |     num_webpages = len(index_results)
176 |     dest_compressed_size = sum([data['compressedSize'] for data in index_results])
177 |     dest_compressed_size_mb = dest_compressed_size / 1000000
178 | 
179 |     print ""
180 |     print "# files: " + str(num_files)
181 |     print "# webpages: " + str(num_webpages)
182 |     print ""
183 |     print "Source compressed file size (MB): " + str(num_files * 100)
184 |     print "Destination compressed file size (MB): " + str(dest_compressed_size_mb)
185 |     print ""
186 | 
187 |     if args.command == "copy":
188 |         if len(index_results) == 0:
189 |             print "No webpages found for domains \"" + str(args.domains) + "\""
190 |             print ""
191 |             exit()
192 | 
193 |         print "Starting copy..."
194 |         copy_start_time = time.time()
195 |             
196 |         progress_queue = Queue()
197 |         pool = Pool(args.parallel, copy_arc_files_init, [args.aws_access_key, args.aws_secret_key, args.bucket, args.key, progress_queue])
198 |         index_results = partition(index_results, args.parallel)
199 | 
200 |         result = pool.map_async(copy_arc_files, index_results)
201 |         pool.close()
202 | 
203 |         bytes_downloaded = bytes_uploaded = 0
204 |         while(True):
205 |             if (result.ready()): 
206 |                 sys.stdout.write("\rDownload: 100%\tUpload: 100%")
207 |                 sys.stdout.flush()
208 |                 break
209 | 
210 |             while not progress_queue.empty():
211 |                 obj = progress_queue.get()
212 |                 if obj[0] == "download":
213 |                     bytes_downloaded += obj[1]
214 |                 elif obj[0] == "upload":
215 |                     bytes_uploaded += obj[1]
216 |                 elif obj[0] == "warning":
217 |                     print "\n" + obj[1]
218 |                 elif obj[0] == "error":
219 |                     # error logged in child process
220 |                     pool.terminate()
221 |                     pool.join()
222 |                     exit()
223 | 
224 |             sys.stdout.write("\rDownload: %d%%\tUpload: %d%%" % (100 * bytes_downloaded / dest_compressed_size, 100 * bytes_uploaded / dest_compressed_size))
225 |             sys.stdout.flush()
226 |             time.sleep(1) 
227 | 
228 |         pool.join()
229 | 
230 |         copy_elapsed_time = int(round(time.time() - copy_start_time))
231 |         copy_timedelta = timedelta(seconds=copy_elapsed_time)
232 | 
233 |         print ""
234 |         print "Copy complete!"
235 |         print "Took " + str(copy_timedelta)
236 |         print ""
237 | 


--------------------------------------------------------------------------------
/docs/data_block.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/data_block.png


--------------------------------------------------------------------------------
/docs/file_format.graffle/data.plist:
--------------------------------------------------------------------------------
   1 | <?xml version="1.0" encoding="UTF-8"?>
   2 | <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   3 | <plist version="1.0">
   4 | <dict>
   5 | 	<key>ActiveLayerIndex</key>
   6 | 	<integer>0</integer>
   7 | 	<key>ApplicationVersion</key>
   8 | 	<array>
   9 | 		<string>com.omnigroup.OmniGrafflePro</string>
  10 | 		<string>139.16.0.171715</string>
  11 | 	</array>
  12 | 	<key>AutoAdjust</key>
  13 | 	<true/>
  14 | 	<key>BackgroundGraphic</key>
  15 | 	<dict>
  16 | 		<key>Bounds</key>
  17 | 		<string>{{0, 0}, {576, 1466}}</string>
  18 | 		<key>Class</key>
  19 | 		<string>SolidGraphic</string>
  20 | 		<key>FontInfo</key>
  21 | 		<dict>
  22 | 			<key>Font</key>
  23 | 			<string>LucidaGrande-Bold</string>
  24 | 			<key>Size</key>
  25 | 			<real>18</real>
  26 | 		</dict>
  27 | 		<key>ID</key>
  28 | 		<integer>2</integer>
  29 | 		<key>Style</key>
  30 | 		<dict>
  31 | 			<key>shadow</key>
  32 | 			<dict>
  33 | 				<key>Draws</key>
  34 | 				<string>NO</string>
  35 | 			</dict>
  36 | 			<key>stroke</key>
  37 | 			<dict>
  38 | 				<key>Draws</key>
  39 | 				<string>NO</string>
  40 | 			</dict>
  41 | 		</dict>
  42 | 	</dict>
  43 | 	<key>BaseZoom</key>
  44 | 	<integer>0</integer>
  45 | 	<key>CanvasOrigin</key>
  46 | 	<string>{0, 0}</string>
  47 | 	<key>ColumnAlign</key>
  48 | 	<integer>1</integer>
  49 | 	<key>ColumnSpacing</key>
  50 | 	<real>36</real>
  51 | 	<key>CreationDate</key>
  52 | 	<string>2012-11-13 17:07:40 +0000</string>
  53 | 	<key>Creator</key>
  54 | 	<string>Scott Robertson</string>
  55 | 	<key>DisplayScale</key>
  56 | 	<string>1 0/72 in = 1 0/72 in</string>
  57 | 	<key>GraphDocumentVersion</key>
  58 | 	<integer>8</integer>
  59 | 	<key>GraphicsList</key>
  60 | 	<array>
  61 | 		<dict>
  62 | 			<key>Class</key>
  63 | 			<string>LineGraphic</string>
  64 | 			<key>Head</key>
  65 | 			<dict>
  66 | 				<key>ID</key>
  67 | 				<integer>37901</integer>
  68 | 			</dict>
  69 | 			<key>ID</key>
  70 | 			<integer>37902</integer>
  71 | 			<key>Points</key>
  72 | 			<array>
  73 | 				<string>{325.33805135644178, 900.31115892819366}</string>
  74 | 				<string>{379.89529297952606, 959.76688858859177}</string>
  75 | 			</array>
  76 | 			<key>Style</key>
  77 | 			<dict>
  78 | 				<key>stroke</key>
  79 | 				<dict>
  80 | 					<key>HeadArrow</key>
  81 | 					<string>FilledArrow</string>
  82 | 					<key>HeadScale</key>
  83 | 					<real>0.92857086658477783</real>
  84 | 					<key>Legacy</key>
  85 | 					<true/>
  86 | 					<key>Pattern</key>
  87 | 					<integer>1</integer>
  88 | 					<key>TailArrow</key>
  89 | 					<string>0</string>
  90 | 					<key>TailScale</key>
  91 | 					<real>0.5</real>
  92 | 				</dict>
  93 | 			</dict>
  94 | 			<key>Tail</key>
  95 | 			<dict>
  96 | 				<key>ID</key>
  97 | 				<integer>37853</integer>
  98 | 			</dict>
  99 | 		</dict>
 100 | 		<dict>
 101 | 			<key>Class</key>
 102 | 			<string>LineGraphic</string>
 103 | 			<key>Head</key>
 104 | 			<dict>
 105 | 				<key>ID</key>
 106 | 				<integer>37887</integer>
 107 | 			</dict>
 108 | 			<key>ID</key>
 109 | 			<integer>37899</integer>
 110 | 			<key>Points</key>
 111 | 			<array>
 112 | 				<string>{204.32895315715842, 906.49991653860252}</string>
 113 | 				<string>{205.21530664678642, 955.00010110840731}</string>
 114 | 			</array>
 115 | 			<key>Style</key>
 116 | 			<dict>
 117 | 				<key>stroke</key>
 118 | 				<dict>
 119 | 					<key>HeadArrow</key>
 120 | 					<string>FilledArrow</string>
 121 | 					<key>HeadScale</key>
 122 | 					<real>0.92857086658477783</real>
 123 | 					<key>Legacy</key>
 124 | 					<true/>
 125 | 					<key>Pattern</key>
 126 | 					<integer>1</integer>
 127 | 					<key>TailArrow</key>
 128 | 					<string>0</string>
 129 | 					<key>TailScale</key>
 130 | 					<real>0.5</real>
 131 | 				</dict>
 132 | 			</dict>
 133 | 			<key>Tail</key>
 134 | 			<dict>
 135 | 				<key>ID</key>
 136 | 				<integer>37847</integer>
 137 | 			</dict>
 138 | 		</dict>
 139 | 		<dict>
 140 | 			<key>Class</key>
 141 | 			<string>LineGraphic</string>
 142 | 			<key>Head</key>
 143 | 			<dict>
 144 | 				<key>ID</key>
 145 | 				<integer>37888</integer>
 146 | 			</dict>
 147 | 			<key>ID</key>
 148 | 			<integer>37898</integer>
 149 | 			<key>Points</key>
 150 | 			<array>
 151 | 				<string>{90.672534737826467, 893.22399837943101}</string>
 152 | 				<string>{36.266666666666609, 956}</string>
 153 | 			</array>
 154 | 			<key>Style</key>
 155 | 			<dict>
 156 | 				<key>stroke</key>
 157 | 				<dict>
 158 | 					<key>HeadArrow</key>
 159 | 					<string>FilledArrow</string>
 160 | 					<key>HeadScale</key>
 161 | 					<real>0.92857086658477783</real>
 162 | 					<key>Legacy</key>
 163 | 					<true/>
 164 | 					<key>Pattern</key>
 165 | 					<integer>1</integer>
 166 | 					<key>TailArrow</key>
 167 | 					<string>0</string>
 168 | 					<key>TailScale</key>
 169 | 					<real>0.5</real>
 170 | 				</dict>
 171 | 			</dict>
 172 | 			<key>Tail</key>
 173 | 			<dict>
 174 | 				<key>ID</key>
 175 | 				<integer>37845</integer>
 176 | 			</dict>
 177 | 		</dict>
 178 | 		<dict>
 179 | 			<key>Class</key>
 180 | 			<string>LineGraphic</string>
 181 | 			<key>Head</key>
 182 | 			<dict>
 183 | 				<key>ID</key>
 184 | 				<integer>37853</integer>
 185 | 			</dict>
 186 | 			<key>ID</key>
 187 | 			<integer>37897</integer>
 188 | 			<key>Points</key>
 189 | 			<array>
 190 | 				<string>{296.17720420364861, 821.48222527485575}</string>
 191 | 				<string>{309.61307252812549, 870.51749984194748}</string>
 192 | 			</array>
 193 | 			<key>Style</key>
 194 | 			<dict>
 195 | 				<key>stroke</key>
 196 | 				<dict>
 197 | 					<key>HeadArrow</key>
 198 | 					<string>FilledArrow</string>
 199 | 					<key>HeadScale</key>
 200 | 					<real>0.92857086658477783</real>
 201 | 					<key>Legacy</key>
 202 | 					<true/>
 203 | 					<key>Pattern</key>
 204 | 					<integer>1</integer>
 205 | 					<key>TailArrow</key>
 206 | 					<string>0</string>
 207 | 					<key>TailScale</key>
 208 | 					<real>0.5</real>
 209 | 				</dict>
 210 | 			</dict>
 211 | 			<key>Tail</key>
 212 | 			<dict>
 213 | 				<key>ID</key>
 214 | 				<integer>37825</integer>
 215 | 			</dict>
 216 | 		</dict>
 217 | 		<dict>
 218 | 			<key>Class</key>
 219 | 			<string>LineGraphic</string>
 220 | 			<key>Head</key>
 221 | 			<dict>
 222 | 				<key>ID</key>
 223 | 				<integer>37845</integer>
 224 | 			</dict>
 225 | 			<key>ID</key>
 226 | 			<integer>37896</integer>
 227 | 			<key>Points</key>
 228 | 			<array>
 229 | 				<string>{177.87504060493688, 813.09318665694229}</string>
 230 | 				<string>{112.37780562847492, 870.87061623778675}</string>
 231 | 			</array>
 232 | 			<key>Style</key>
 233 | 			<dict>
 234 | 				<key>stroke</key>
 235 | 				<dict>
 236 | 					<key>HeadArrow</key>
 237 | 					<string>FilledArrow</string>
 238 | 					<key>HeadScale</key>
 239 | 					<real>0.92857086658477783</real>
 240 | 					<key>Legacy</key>
 241 | 					<true/>
 242 | 					<key>Pattern</key>
 243 | 					<integer>1</integer>
 244 | 					<key>TailArrow</key>
 245 | 					<string>0</string>
 246 | 					<key>TailScale</key>
 247 | 					<real>0.5</real>
 248 | 				</dict>
 249 | 			</dict>
 250 | 			<key>Tail</key>
 251 | 			<dict>
 252 | 				<key>ID</key>
 253 | 				<integer>37823</integer>
 254 | 			</dict>
 255 | 		</dict>
 256 | 		<dict>
 257 | 			<key>Bounds</key>
 258 | 			<string>{{380.5714111328125, 956}, {185.4285888671875, 30}}</string>
 259 | 			<key>Class</key>
 260 | 			<string>ShapedGraphic</string>
 261 | 			<key>ID</key>
 262 | 			<integer>37901</integer>
 263 | 			<key>Magnets</key>
 264 | 			<array>
 265 | 				<string>{-0.45300457989336662, -0.033333333333331439}</string>
 266 | 			</array>
 267 | 			<key>Shape</key>
 268 | 			<string>Rectangle</string>
 269 | 			<key>Style</key>
 270 | 			<dict>
 271 | 				<key>fill</key>
 272 | 				<dict>
 273 | 					<key>Draws</key>
 274 | 					<string>NO</string>
 275 | 				</dict>
 276 | 				<key>shadow</key>
 277 | 				<dict>
 278 | 					<key>Draws</key>
 279 | 					<string>NO</string>
 280 | 				</dict>
 281 | 				<key>stroke</key>
 282 | 				<dict>
 283 | 					<key>Width</key>
 284 | 					<real>2</real>
 285 | 				</dict>
 286 | 			</dict>
 287 | 		</dict>
 288 | 		<dict>
 289 | 			<key>Bounds</key>
 290 | 			<string>{{195.71429443359375, 956}, {185.4285888671875, 30}}</string>
 291 | 			<key>Class</key>
 292 | 			<string>ShapedGraphic</string>
 293 | 			<key>ID</key>
 294 | 			<integer>37887</integer>
 295 | 			<key>Magnets</key>
 296 | 			<array>
 297 | 				<string>{-0.44761166822796894, -0.033333333333331439}</string>
 298 | 			</array>
 299 | 			<key>Shape</key>
 300 | 			<string>Rectangle</string>
 301 | 			<key>Style</key>
 302 | 			<dict>
 303 | 				<key>fill</key>
 304 | 				<dict>
 305 | 					<key>Draws</key>
 306 | 					<string>NO</string>
 307 | 				</dict>
 308 | 				<key>shadow</key>
 309 | 				<dict>
 310 | 					<key>Draws</key>
 311 | 					<string>NO</string>
 312 | 				</dict>
 313 | 				<key>stroke</key>
 314 | 				<dict>
 315 | 					<key>Width</key>
 316 | 					<real>2</real>
 317 | 				</dict>
 318 | 			</dict>
 319 | 		</dict>
 320 | 		<dict>
 321 | 			<key>Bounds</key>
 322 | 			<string>{{10.28570556640625, 956}, {185.4285888671875, 30}}</string>
 323 | 			<key>Class</key>
 324 | 			<string>ShapedGraphic</string>
 325 | 			<key>ID</key>
 326 | 			<integer>37888</integer>
 327 | 			<key>Magnets</key>
 328 | 			<array>
 329 | 				<string>{-0.42064710990098297, -0.066666666666669983}</string>
 330 | 			</array>
 331 | 			<key>Shape</key>
 332 | 			<string>Rectangle</string>
 333 | 			<key>Style</key>
 334 | 			<dict>
 335 | 				<key>fill</key>
 336 | 				<dict>
 337 | 					<key>Draws</key>
 338 | 					<string>NO</string>
 339 | 				</dict>
 340 | 				<key>shadow</key>
 341 | 				<dict>
 342 | 					<key>Draws</key>
 343 | 					<string>NO</string>
 344 | 				</dict>
 345 | 				<key>stroke</key>
 346 | 				<dict>
 347 | 					<key>Width</key>
 348 | 					<real>2</real>
 349 | 				</dict>
 350 | 			</dict>
 351 | 		</dict>
 352 | 		<dict>
 353 | 			<key>Class</key>
 354 | 			<string>TableGroup</string>
 355 | 			<key>Graphics</key>
 356 | 			<array>
 357 | 				<dict>
 358 | 					<key>Bounds</key>
 359 | 					<string>{{10.28570556640625, 956.00003051757812}, {92.571426391601562, 30}}</string>
 360 | 					<key>Class</key>
 361 | 					<string>ShapedGraphic</string>
 362 | 					<key>ID</key>
 363 | 					<integer>37890</integer>
 364 | 					<key>Shape</key>
 365 | 					<string>Rectangle</string>
 366 | 					<key>Style</key>
 367 | 					<dict>
 368 | 						<key>shadow</key>
 369 | 						<dict>
 370 | 							<key>Draws</key>
 371 | 							<string>NO</string>
 372 | 						</dict>
 373 | 						<key>stroke</key>
 374 | 						<dict>
 375 | 							<key>Pattern</key>
 376 | 							<integer>2</integer>
 377 | 						</dict>
 378 | 					</dict>
 379 | 					<key>Text</key>
 380 | 					<dict>
 381 | 						<key>Text</key>
 382 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 383 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 384 | {\colortbl;\red255\green255\blue255;}
 385 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 386 | 
 387 | \f0\b\fs24 \cf0 item 1}</string>
 388 | 					</dict>
 389 | 				</dict>
 390 | 				<dict>
 391 | 					<key>Bounds</key>
 392 | 					<string>{{102.85713958740234, 956.00003051757812}, {92.571426391601562, 30}}</string>
 393 | 					<key>Class</key>
 394 | 					<string>ShapedGraphic</string>
 395 | 					<key>ID</key>
 396 | 					<integer>37891</integer>
 397 | 					<key>Shape</key>
 398 | 					<string>Rectangle</string>
 399 | 					<key>Style</key>
 400 | 					<dict>
 401 | 						<key>shadow</key>
 402 | 						<dict>
 403 | 							<key>Draws</key>
 404 | 							<string>NO</string>
 405 | 						</dict>
 406 | 						<key>stroke</key>
 407 | 						<dict>
 408 | 							<key>Pattern</key>
 409 | 							<integer>2</integer>
 410 | 						</dict>
 411 | 					</dict>
 412 | 					<key>Text</key>
 413 | 					<dict>
 414 | 						<key>Text</key>
 415 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 416 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 417 | {\colortbl;\red255\green255\blue255;}
 418 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 419 | 
 420 | \f0\b\fs24 \cf0 item 2}</string>
 421 | 					</dict>
 422 | 				</dict>
 423 | 				<dict>
 424 | 					<key>Bounds</key>
 425 | 					<string>{{195.42855834960938, 956.00003051757812}, {92.571441650390625, 30}}</string>
 426 | 					<key>Class</key>
 427 | 					<string>ShapedGraphic</string>
 428 | 					<key>ID</key>
 429 | 					<integer>37892</integer>
 430 | 					<key>Shape</key>
 431 | 					<string>Rectangle</string>
 432 | 					<key>Style</key>
 433 | 					<dict>
 434 | 						<key>shadow</key>
 435 | 						<dict>
 436 | 							<key>Draws</key>
 437 | 							<string>NO</string>
 438 | 						</dict>
 439 | 						<key>stroke</key>
 440 | 						<dict>
 441 | 							<key>Pattern</key>
 442 | 							<integer>2</integer>
 443 | 						</dict>
 444 | 					</dict>
 445 | 					<key>Text</key>
 446 | 					<dict>
 447 | 						<key>Text</key>
 448 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 449 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 450 | {\colortbl;\red255\green255\blue255;}
 451 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 452 | 
 453 | \f0\b\fs24 \cf0 item 3}</string>
 454 | 					</dict>
 455 | 				</dict>
 456 | 				<dict>
 457 | 					<key>Bounds</key>
 458 | 					<string>{{287.99998474121094, 956.00003051757812}, {92.571441650390625, 30}}</string>
 459 | 					<key>Class</key>
 460 | 					<string>ShapedGraphic</string>
 461 | 					<key>ID</key>
 462 | 					<integer>37893</integer>
 463 | 					<key>Shape</key>
 464 | 					<string>Rectangle</string>
 465 | 					<key>Style</key>
 466 | 					<dict>
 467 | 						<key>shadow</key>
 468 | 						<dict>
 469 | 							<key>Draws</key>
 470 | 							<string>NO</string>
 471 | 						</dict>
 472 | 						<key>stroke</key>
 473 | 						<dict>
 474 | 							<key>Pattern</key>
 475 | 							<integer>2</integer>
 476 | 						</dict>
 477 | 					</dict>
 478 | 					<key>Text</key>
 479 | 					<dict>
 480 | 						<key>Text</key>
 481 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 482 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 483 | {\colortbl;\red255\green255\blue255;}
 484 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 485 | 
 486 | \f0\b\fs24 \cf0 item 4}</string>
 487 | 					</dict>
 488 | 				</dict>
 489 | 				<dict>
 490 | 					<key>Bounds</key>
 491 | 					<string>{{380.57145690917969, 956.00003051757812}, {92.571441650390625, 30}}</string>
 492 | 					<key>Class</key>
 493 | 					<string>ShapedGraphic</string>
 494 | 					<key>ID</key>
 495 | 					<integer>37894</integer>
 496 | 					<key>Shape</key>
 497 | 					<string>Rectangle</string>
 498 | 					<key>Style</key>
 499 | 					<dict>
 500 | 						<key>shadow</key>
 501 | 						<dict>
 502 | 							<key>Draws</key>
 503 | 							<string>NO</string>
 504 | 						</dict>
 505 | 						<key>stroke</key>
 506 | 						<dict>
 507 | 							<key>Pattern</key>
 508 | 							<integer>2</integer>
 509 | 						</dict>
 510 | 					</dict>
 511 | 					<key>Text</key>
 512 | 					<dict>
 513 | 						<key>Text</key>
 514 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 515 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 516 | {\colortbl;\red255\green255\blue255;}
 517 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 518 | 
 519 | \f0\b\fs24 \cf0 item 5}</string>
 520 | 					</dict>
 521 | 				</dict>
 522 | 				<dict>
 523 | 					<key>Bounds</key>
 524 | 					<string>{{473.14288330078125, 956.00003051757812}, {92.5714111328125, 30}}</string>
 525 | 					<key>Class</key>
 526 | 					<string>ShapedGraphic</string>
 527 | 					<key>ID</key>
 528 | 					<integer>37895</integer>
 529 | 					<key>Shape</key>
 530 | 					<string>Rectangle</string>
 531 | 					<key>Style</key>
 532 | 					<dict>
 533 | 						<key>shadow</key>
 534 | 						<dict>
 535 | 							<key>Draws</key>
 536 | 							<string>NO</string>
 537 | 						</dict>
 538 | 						<key>stroke</key>
 539 | 						<dict>
 540 | 							<key>Pattern</key>
 541 | 							<integer>2</integer>
 542 | 						</dict>
 543 | 					</dict>
 544 | 					<key>Text</key>
 545 | 					<dict>
 546 | 						<key>Text</key>
 547 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 548 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 549 | {\colortbl;\red255\green255\blue255;}
 550 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 551 | 
 552 | \f0\b\fs24 \cf0 item 6}</string>
 553 | 					</dict>
 554 | 				</dict>
 555 | 			</array>
 556 | 			<key>GridV</key>
 557 | 			<array>
 558 | 				<integer>37890</integer>
 559 | 				<integer>37891</integer>
 560 | 				<integer>37892</integer>
 561 | 				<integer>37893</integer>
 562 | 				<integer>37894</integer>
 563 | 				<integer>37895</integer>
 564 | 				<array/>
 565 | 			</array>
 566 | 			<key>ID</key>
 567 | 			<integer>37889</integer>
 568 | 		</dict>
 569 | 		<dict>
 570 | 			<key>Class</key>
 571 | 			<string>TableGroup</string>
 572 | 			<key>Graphics</key>
 573 | 			<array>
 574 | 				<dict>
 575 | 					<key>Bounds</key>
 576 | 					<string>{{325, 871}, {79, 35}}</string>
 577 | 					<key>Class</key>
 578 | 					<string>ShapedGraphic</string>
 579 | 					<key>ID</key>
 580 | 					<integer>37852</integer>
 581 | 					<key>Shape</key>
 582 | 					<string>Rectangle</string>
 583 | 					<key>Text</key>
 584 | 					<dict>
 585 | 						<key>Text</key>
 586 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 587 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 588 | {\colortbl;\red255\green255\blue255;}
 589 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 590 | 
 591 | \f0\b\fs24 \cf0 prefix N}</string>
 592 | 					</dict>
 593 | 				</dict>
 594 | 				<dict>
 595 | 					<key>Bounds</key>
 596 | 					<string>{{304, 871}, {21, 35}}</string>
 597 | 					<key>Class</key>
 598 | 					<string>ShapedGraphic</string>
 599 | 					<key>ID</key>
 600 | 					<integer>37853</integer>
 601 | 					<key>Shape</key>
 602 | 					<string>Rectangle</string>
 603 | 					<key>Text</key>
 604 | 					<dict>
 605 | 						<key>Text</key>
 606 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 607 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 608 | {\colortbl;\red255\green255\blue255;}
 609 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 610 | 
 611 | \f0\b\fs24 \cf0 5}</string>
 612 | 					</dict>
 613 | 				</dict>
 614 | 				<dict>
 615 | 					<key>Bounds</key>
 616 | 					<string>{{404, 871}, {26, 35}}</string>
 617 | 					<key>Class</key>
 618 | 					<string>ShapedGraphic</string>
 619 | 					<key>ID</key>
 620 | 					<integer>37855</integer>
 621 | 					<key>Shape</key>
 622 | 					<string>Rectangle</string>
 623 | 					<key>Text</key>
 624 | 					<dict>
 625 | 						<key>Text</key>
 626 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 627 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 628 | {\colortbl;\red255\green255\blue255;}
 629 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 630 | 
 631 | \f0\b\fs24 \cf0 6}</string>
 632 | 					</dict>
 633 | 				</dict>
 634 | 				<dict>
 635 | 					<key>Bounds</key>
 636 | 					<string>{{430, 871}, {26, 35}}</string>
 637 | 					<key>Class</key>
 638 | 					<string>ShapedGraphic</string>
 639 | 					<key>ID</key>
 640 | 					<integer>37856</integer>
 641 | 					<key>Shape</key>
 642 | 					<string>Rectangle</string>
 643 | 					<key>Text</key>
 644 | 					<dict>
 645 | 						<key>Text</key>
 646 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 647 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 648 | {\colortbl;\red255\green255\blue255;}
 649 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 650 | 
 651 | \f0\b\fs24 \cf0 ...}</string>
 652 | 					</dict>
 653 | 				</dict>
 654 | 			</array>
 655 | 			<key>GridV</key>
 656 | 			<array>
 657 | 				<integer>37853</integer>
 658 | 				<integer>37852</integer>
 659 | 				<integer>37855</integer>
 660 | 				<integer>37856</integer>
 661 | 				<array/>
 662 | 			</array>
 663 | 			<key>ID</key>
 664 | 			<integer>37851</integer>
 665 | 		</dict>
 666 | 		<dict>
 667 | 			<key>Bounds</key>
 668 | 			<string>{{172, 658.5}, {144, 30}}</string>
 669 | 			<key>Class</key>
 670 | 			<string>ShapedGraphic</string>
 671 | 			<key>FitText</key>
 672 | 			<string>YES</string>
 673 | 			<key>Flow</key>
 674 | 			<string>Resize</string>
 675 | 			<key>FontInfo</key>
 676 | 			<dict>
 677 | 				<key>Color</key>
 678 | 				<dict>
 679 | 					<key>w</key>
 680 | 					<string>0</string>
 681 | 				</dict>
 682 | 				<key>Font</key>
 683 | 				<string>LucidaGrande-Bold</string>
 684 | 				<key>Size</key>
 685 | 				<real>12</real>
 686 | 			</dict>
 687 | 			<key>ID</key>
 688 | 			<integer>37850</integer>
 689 | 			<key>Shape</key>
 690 | 			<string>Rectangle</string>
 691 | 			<key>Style</key>
 692 | 			<dict>
 693 | 				<key>fill</key>
 694 | 				<dict>
 695 | 					<key>Draws</key>
 696 | 					<string>NO</string>
 697 | 				</dict>
 698 | 				<key>shadow</key>
 699 | 				<dict>
 700 | 					<key>Draws</key>
 701 | 					<string>NO</string>
 702 | 				</dict>
 703 | 				<key>stroke</key>
 704 | 				<dict>
 705 | 					<key>Draws</key>
 706 | 					<string>NO</string>
 707 | 				</dict>
 708 | 			</dict>
 709 | 			<key>Text</key>
 710 | 			<dict>
 711 | 				<key>Pad</key>
 712 | 				<integer>0</integer>
 713 | 				<key>Text</key>
 714 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 715 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 716 | {\colortbl;\red255\green255\blue255;}
 717 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 718 | 
 719 | \f0\b\fs24 \cf0 item \
 720 | (
 721 | \b0 url and arc file location
 722 | \b )}</string>
 723 | 				<key>VerticalPad</key>
 724 | 				<integer>0</integer>
 725 | 			</dict>
 726 | 			<key>Wrap</key>
 727 | 			<string>NO</string>
 728 | 		</dict>
 729 | 		<dict>
 730 | 			<key>Bounds</key>
 731 | 			<string>{{37, 609.05108642578125}, {428, 41.948900000000002}}</string>
 732 | 			<key>Class</key>
 733 | 			<string>ShapedGraphic</string>
 734 | 			<key>ID</key>
 735 | 			<integer>37849</integer>
 736 | 			<key>ImageID</key>
 737 | 			<integer>1</integer>
 738 | 			<key>Shape</key>
 739 | 			<string>Rectangle</string>
 740 | 			<key>Style</key>
 741 | 			<dict>
 742 | 				<key>fill</key>
 743 | 				<dict>
 744 | 					<key>Draws</key>
 745 | 					<string>NO</string>
 746 | 				</dict>
 747 | 				<key>shadow</key>
 748 | 				<dict>
 749 | 					<key>Draws</key>
 750 | 					<string>NO</string>
 751 | 				</dict>
 752 | 				<key>stroke</key>
 753 | 				<dict>
 754 | 					<key>Draws</key>
 755 | 					<string>NO</string>
 756 | 				</dict>
 757 | 			</dict>
 758 | 		</dict>
 759 | 		<dict>
 760 | 			<key>Class</key>
 761 | 			<string>TableGroup</string>
 762 | 			<key>Graphics</key>
 763 | 			<array>
 764 | 				<dict>
 765 | 					<key>Bounds</key>
 766 | 					<string>{{112, 871}, {79, 35}}</string>
 767 | 					<key>Class</key>
 768 | 					<string>ShapedGraphic</string>
 769 | 					<key>ID</key>
 770 | 					<integer>37844</integer>
 771 | 					<key>Shape</key>
 772 | 					<string>Rectangle</string>
 773 | 					<key>Text</key>
 774 | 					<dict>
 775 | 						<key>Text</key>
 776 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 777 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 778 | {\colortbl;\red255\green255\blue255;}
 779 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 780 | 
 781 | \f0\b\fs24 \cf0 prefix 1}</string>
 782 | 					</dict>
 783 | 				</dict>
 784 | 				<dict>
 785 | 					<key>Bounds</key>
 786 | 					<string>{{91, 871}, {21, 35}}</string>
 787 | 					<key>Class</key>
 788 | 					<string>ShapedGraphic</string>
 789 | 					<key>ID</key>
 790 | 					<integer>37845</integer>
 791 | 					<key>Magnets</key>
 792 | 					<array>
 793 | 						<string>{0.071428571428571175, -0.27142857142857224}</string>
 794 | 					</array>
 795 | 					<key>Shape</key>
 796 | 					<string>Rectangle</string>
 797 | 					<key>Text</key>
 798 | 					<dict>
 799 | 						<key>Text</key>
 800 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 801 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 802 | {\colortbl;\red255\green255\blue255;}
 803 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 804 | 
 805 | \f0\b\fs24 \cf0 3}</string>
 806 | 					</dict>
 807 | 				</dict>
 808 | 				<dict>
 809 | 					<key>Bounds</key>
 810 | 					<string>{{191, 871}, {26, 35}}</string>
 811 | 					<key>Class</key>
 812 | 					<string>ShapedGraphic</string>
 813 | 					<key>ID</key>
 814 | 					<integer>37847</integer>
 815 | 					<key>Shape</key>
 816 | 					<string>Rectangle</string>
 817 | 					<key>Text</key>
 818 | 					<dict>
 819 | 						<key>Text</key>
 820 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 821 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 822 | {\colortbl;\red255\green255\blue255;}
 823 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 824 | 
 825 | \f0\b\fs24 \cf0 4}</string>
 826 | 					</dict>
 827 | 				</dict>
 828 | 				<dict>
 829 | 					<key>Bounds</key>
 830 | 					<string>{{217, 871}, {37.5, 35}}</string>
 831 | 					<key>Class</key>
 832 | 					<string>ShapedGraphic</string>
 833 | 					<key>ID</key>
 834 | 					<integer>37848</integer>
 835 | 					<key>Shape</key>
 836 | 					<string>Rectangle</string>
 837 | 					<key>Text</key>
 838 | 					<dict>
 839 | 						<key>Text</key>
 840 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 841 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 842 | {\colortbl;\red255\green255\blue255;}
 843 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 844 | 
 845 | \f0\b\fs24 \cf0 ...}</string>
 846 | 					</dict>
 847 | 				</dict>
 848 | 			</array>
 849 | 			<key>GridV</key>
 850 | 			<array>
 851 | 				<integer>37845</integer>
 852 | 				<integer>37844</integer>
 853 | 				<integer>37847</integer>
 854 | 				<integer>37848</integer>
 855 | 				<array/>
 856 | 			</array>
 857 | 			<key>ID</key>
 858 | 			<integer>37843</integer>
 859 | 		</dict>
 860 | 		<dict>
 861 | 			<key>Class</key>
 862 | 			<string>TableGroup</string>
 863 | 			<key>Graphics</key>
 864 | 			<array>
 865 | 				<dict>
 866 | 					<key>Bounds</key>
 867 | 					<string>{{199.25, 786}, {79, 35}}</string>
 868 | 					<key>Class</key>
 869 | 					<string>ShapedGraphic</string>
 870 | 					<key>ID</key>
 871 | 					<integer>37822</integer>
 872 | 					<key>Shape</key>
 873 | 					<string>Rectangle</string>
 874 | 					<key>Text</key>
 875 | 					<dict>
 876 | 						<key>Text</key>
 877 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 878 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 879 | {\colortbl;\red255\green255\blue255;}
 880 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 881 | 
 882 | \f0\b\fs24 \cf0 prefix 1}</string>
 883 | 					</dict>
 884 | 				</dict>
 885 | 				<dict>
 886 | 					<key>Bounds</key>
 887 | 					<string>{{178.25, 786}, {21, 35}}</string>
 888 | 					<key>Class</key>
 889 | 					<string>ShapedGraphic</string>
 890 | 					<key>ID</key>
 891 | 					<integer>37823</integer>
 892 | 					<key>Shape</key>
 893 | 					<string>Rectangle</string>
 894 | 					<key>Text</key>
 895 | 					<dict>
 896 | 						<key>Text</key>
 897 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 898 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 899 | {\colortbl;\red255\green255\blue255;}
 900 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 901 | 
 902 | \f0\b\fs24 \cf0 1}</string>
 903 | 					</dict>
 904 | 				</dict>
 905 | 				<dict>
 906 | 					<key>Bounds</key>
 907 | 					<string>{{278.25, 786}, {26, 35}}</string>
 908 | 					<key>Class</key>
 909 | 					<string>ShapedGraphic</string>
 910 | 					<key>ID</key>
 911 | 					<integer>37825</integer>
 912 | 					<key>Shape</key>
 913 | 					<string>Rectangle</string>
 914 | 					<key>Text</key>
 915 | 					<dict>
 916 | 						<key>Text</key>
 917 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 918 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 919 | {\colortbl;\red255\green255\blue255;}
 920 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 921 | 
 922 | \f0\b\fs24 \cf0 2}</string>
 923 | 					</dict>
 924 | 				</dict>
 925 | 				<dict>
 926 | 					<key>Bounds</key>
 927 | 					<string>{{304.25, 786}, {64.5, 35}}</string>
 928 | 					<key>Class</key>
 929 | 					<string>ShapedGraphic</string>
 930 | 					<key>ID</key>
 931 | 					<integer>37826</integer>
 932 | 					<key>Shape</key>
 933 | 					<string>Rectangle</string>
 934 | 					<key>Text</key>
 935 | 					<dict>
 936 | 						<key>Text</key>
 937 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 938 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 939 | {\colortbl;\red255\green255\blue255;}
 940 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 941 | 
 942 | \f0\b\fs24 \cf0 ...}</string>
 943 | 					</dict>
 944 | 				</dict>
 945 | 			</array>
 946 | 			<key>GridV</key>
 947 | 			<array>
 948 | 				<integer>37823</integer>
 949 | 				<integer>37822</integer>
 950 | 				<integer>37825</integer>
 951 | 				<integer>37826</integer>
 952 | 				<array/>
 953 | 			</array>
 954 | 			<key>ID</key>
 955 | 			<integer>37821</integer>
 956 | 		</dict>
 957 | 		<dict>
 958 | 			<key>Class</key>
 959 | 			<string>TableGroup</string>
 960 | 			<key>Graphics</key>
 961 | 			<array>
 962 | 				<dict>
 963 | 					<key>Bounds</key>
 964 | 					<string>{{471, 566.99993896484375}, {84, 35}}</string>
 965 | 					<key>Class</key>
 966 | 					<string>ShapedGraphic</string>
 967 | 					<key>ID</key>
 968 | 					<integer>37818</integer>
 969 | 					<key>Shape</key>
 970 | 					<string>Rectangle</string>
 971 | 					<key>Text</key>
 972 | 					<dict>
 973 | 						<key>Text</key>
 974 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 975 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 976 | {\colortbl;\red255\green255\blue255;}
 977 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 978 | 
 979 | \f0\b\fs24 \cf0 url 2}</string>
 980 | 					</dict>
 981 | 				</dict>
 982 | 				<dict>
 983 | 					<key>Bounds</key>
 984 | 					<string>{{30, 566.99993896484375}, {108, 35}}</string>
 985 | 					<key>Class</key>
 986 | 					<string>ShapedGraphic</string>
 987 | 					<key>ID</key>
 988 | 					<integer>37809</integer>
 989 | 					<key>Shape</key>
 990 | 					<string>Rectangle</string>
 991 | 					<key>Text</key>
 992 | 					<dict>
 993 | 						<key>Text</key>
 994 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
 995 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
 996 | {\colortbl;\red255\green255\blue255;}
 997 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
 998 | 
 999 | \f0\b\fs24 \cf0 url 1}</string>
1000 | 					</dict>
1001 | 				</dict>
1002 | 				<dict>
1003 | 					<key>Bounds</key>
1004 | 					<string>{{243, 566.99993896484375}, {76, 35}}</string>
1005 | 					<key>Class</key>
1006 | 					<string>ShapedGraphic</string>
1007 | 					<key>ID</key>
1008 | 					<integer>37813</integer>
1009 | 					<key>Shape</key>
1010 | 					<string>Rectangle</string>
1011 | 					<key>Text</key>
1012 | 					<dict>
1013 | 						<key>Text</key>
1014 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1015 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1016 | {\colortbl;\red255\green255\blue255;}
1017 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1018 | 
1019 | \f0\b\fs24 \cf0 date}</string>
1020 | 					</dict>
1021 | 				</dict>
1022 | 				<dict>
1023 | 					<key>Bounds</key>
1024 | 					<string>{{167, 566.99993896484375}, {76, 35}}</string>
1025 | 					<key>Class</key>
1026 | 					<string>ShapedGraphic</string>
1027 | 					<key>ID</key>
1028 | 					<integer>37812</integer>
1029 | 					<key>Shape</key>
1030 | 					<string>Rectangle</string>
1031 | 					<key>Text</key>
1032 | 					<dict>
1033 | 						<key>Text</key>
1034 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1035 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1036 | {\colortbl;\red255\green255\blue255;}
1037 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1038 | 
1039 | \f0\b\fs24 \cf0 segment}</string>
1040 | 					</dict>
1041 | 				</dict>
1042 | 				<dict>
1043 | 					<key>Bounds</key>
1044 | 					<string>{{138, 566.99993896484375}, {29, 35}}</string>
1045 | 					<key>Class</key>
1046 | 					<string>ShapedGraphic</string>
1047 | 					<key>ID</key>
1048 | 					<integer>37811</integer>
1049 | 					<key>Shape</key>
1050 | 					<string>Rectangle</string>
1051 | 					<key>Text</key>
1052 | 					<dict>
1053 | 						<key>Text</key>
1054 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1055 | {\fonttbl\f0\fnil\fcharset222 Ayuthaya;}
1056 | {\colortbl;\red255\green255\blue255;}
1057 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1058 | 
1059 | \f0\fs28 \cf0 0}</string>
1060 | 					</dict>
1061 | 				</dict>
1062 | 				<dict>
1063 | 					<key>Bounds</key>
1064 | 					<string>{{319, 566.99993896484375}, {76, 35}}</string>
1065 | 					<key>Class</key>
1066 | 					<string>ShapedGraphic</string>
1067 | 					<key>ID</key>
1068 | 					<integer>37815</integer>
1069 | 					<key>Shape</key>
1070 | 					<string>Rectangle</string>
1071 | 					<key>Text</key>
1072 | 					<dict>
1073 | 						<key>Text</key>
1074 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1075 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1076 | {\colortbl;\red255\green255\blue255;}
1077 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1078 | 
1079 | \f0\b\fs24 \cf0 partition}</string>
1080 | 					</dict>
1081 | 				</dict>
1082 | 				<dict>
1083 | 					<key>Bounds</key>
1084 | 					<string>{{395, 566.99993896484375}, {76, 35}}</string>
1085 | 					<key>Class</key>
1086 | 					<string>ShapedGraphic</string>
1087 | 					<key>ID</key>
1088 | 					<integer>37816</integer>
1089 | 					<key>Shape</key>
1090 | 					<string>Rectangle</string>
1091 | 					<key>Text</key>
1092 | 					<dict>
1093 | 						<key>Text</key>
1094 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1095 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1096 | {\colortbl;\red255\green255\blue255;}
1097 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1098 | 
1099 | \f0\b\fs24 \cf0 file offset}</string>
1100 | 					</dict>
1101 | 				</dict>
1102 | 			</array>
1103 | 			<key>GridV</key>
1104 | 			<array>
1105 | 				<integer>37809</integer>
1106 | 				<integer>37811</integer>
1107 | 				<integer>37812</integer>
1108 | 				<integer>37813</integer>
1109 | 				<integer>37815</integer>
1110 | 				<integer>37816</integer>
1111 | 				<integer>37818</integer>
1112 | 				<array/>
1113 | 			</array>
1114 | 			<key>ID</key>
1115 | 			<integer>37808</integer>
1116 | 		</dict>
1117 | 		<dict>
1118 | 			<key>Bounds</key>
1119 | 			<string>{{25, 561}, {543, 68}}</string>
1120 | 			<key>Class</key>
1121 | 			<string>ShapedGraphic</string>
1122 | 			<key>ID</key>
1123 | 			<integer>37807</integer>
1124 | 			<key>Shape</key>
1125 | 			<string>Rectangle</string>
1126 | 			<key>Style</key>
1127 | 			<dict>
1128 | 				<key>stroke</key>
1129 | 				<dict>
1130 | 					<key>Pattern</key>
1131 | 					<integer>4</integer>
1132 | 				</dict>
1133 | 			</dict>
1134 | 			<key>Text</key>
1135 | 			<dict>
1136 | 				<key>Align</key>
1137 | 				<integer>2</integer>
1138 | 				<key>Text</key>
1139 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1140 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1141 | {\colortbl;\red255\green255\blue255;}
1142 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qr
1143 | 
1144 | \f0\b\fs24 \cf0 data block}</string>
1145 | 			</dict>
1146 | 			<key>TextPlacement</key>
1147 | 			<integer>2</integer>
1148 | 		</dict>
1149 | 		<dict>
1150 | 			<key>Bounds</key>
1151 | 			<string>{{456, 470.57662963867188}, {14, 22}}</string>
1152 | 			<key>Class</key>
1153 | 			<string>ShapedGraphic</string>
1154 | 			<key>FitText</key>
1155 | 			<string>YES</string>
1156 | 			<key>Flow</key>
1157 | 			<string>Resize</string>
1158 | 			<key>FontInfo</key>
1159 | 			<dict>
1160 | 				<key>Color</key>
1161 | 				<dict>
1162 | 					<key>w</key>
1163 | 					<string>0</string>
1164 | 				</dict>
1165 | 				<key>Font</key>
1166 | 				<string>LucidaGrande-Bold</string>
1167 | 				<key>Size</key>
1168 | 				<real>18</real>
1169 | 			</dict>
1170 | 			<key>ID</key>
1171 | 			<integer>37806</integer>
1172 | 			<key>Shape</key>
1173 | 			<string>Rectangle</string>
1174 | 			<key>Style</key>
1175 | 			<dict>
1176 | 				<key>fill</key>
1177 | 				<dict>
1178 | 					<key>Draws</key>
1179 | 					<string>NO</string>
1180 | 				</dict>
1181 | 				<key>shadow</key>
1182 | 				<dict>
1183 | 					<key>Draws</key>
1184 | 					<string>NO</string>
1185 | 				</dict>
1186 | 				<key>stroke</key>
1187 | 				<dict>
1188 | 					<key>Draws</key>
1189 | 					<string>NO</string>
1190 | 				</dict>
1191 | 			</dict>
1192 | 			<key>Text</key>
1193 | 			<dict>
1194 | 				<key>Pad</key>
1195 | 				<integer>0</integer>
1196 | 				<key>Text</key>
1197 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1198 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1199 | {\colortbl;\red255\green255\blue255;}
1200 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1201 | 
1202 | \f0\b\fs36 \cf0 ...}</string>
1203 | 				<key>VerticalPad</key>
1204 | 				<integer>0</integer>
1205 | 			</dict>
1206 | 			<key>Wrap</key>
1207 | 			<string>NO</string>
1208 | 		</dict>
1209 | 		<dict>
1210 | 			<key>Class</key>
1211 | 			<string>TableGroup</string>
1212 | 			<key>Graphics</key>
1213 | 			<array>
1214 | 				<dict>
1215 | 					<key>Bounds</key>
1216 | 					<string>{{51, 457.57662963867188}, {79, 35}}</string>
1217 | 					<key>Class</key>
1218 | 					<string>ShapedGraphic</string>
1219 | 					<key>ID</key>
1220 | 					<integer>37795</integer>
1221 | 					<key>Shape</key>
1222 | 					<string>Rectangle</string>
1223 | 					<key>Text</key>
1224 | 					<dict>
1225 | 						<key>Text</key>
1226 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1227 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1228 | {\colortbl;\red255\green255\blue255;}
1229 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1230 | 
1231 | \f0\b\fs24 \cf0 prefix 1}</string>
1232 | 					</dict>
1233 | 				</dict>
1234 | 				<dict>
1235 | 					<key>Bounds</key>
1236 | 					<string>{{30, 457.57662963867188}, {21, 35}}</string>
1237 | 					<key>Class</key>
1238 | 					<string>ShapedGraphic</string>
1239 | 					<key>ID</key>
1240 | 					<integer>37796</integer>
1241 | 					<key>Shape</key>
1242 | 					<string>Rectangle</string>
1243 | 					<key>Text</key>
1244 | 					<dict>
1245 | 						<key>Text</key>
1246 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1247 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1248 | {\colortbl;\red255\green255\blue255;}
1249 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1250 | 
1251 | \f0\b\fs24 \cf0 1}</string>
1252 | 					</dict>
1253 | 				</dict>
1254 | 				<dict>
1255 | 					<key>Bounds</key>
1256 | 					<string>{{130, 457.57662963867188}, {29, 35}}</string>
1257 | 					<key>Class</key>
1258 | 					<string>ShapedGraphic</string>
1259 | 					<key>ID</key>
1260 | 					<integer>37797</integer>
1261 | 					<key>Shape</key>
1262 | 					<string>Rectangle</string>
1263 | 					<key>Text</key>
1264 | 					<dict>
1265 | 						<key>Text</key>
1266 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1267 | {\fonttbl\f0\fnil\fcharset222 Ayuthaya;}
1268 | {\colortbl;\red255\green255\blue255;}
1269 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1270 | 
1271 | \f0\fs28 \cf0 0}</string>
1272 | 					</dict>
1273 | 				</dict>
1274 | 				<dict>
1275 | 					<key>Bounds</key>
1276 | 					<string>{{159, 457.57662963867188}, {26, 35}}</string>
1277 | 					<key>Class</key>
1278 | 					<string>ShapedGraphic</string>
1279 | 					<key>ID</key>
1280 | 					<integer>37799</integer>
1281 | 					<key>Shape</key>
1282 | 					<string>Rectangle</string>
1283 | 					<key>Text</key>
1284 | 					<dict>
1285 | 						<key>Text</key>
1286 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1287 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1288 | {\colortbl;\red255\green255\blue255;}
1289 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1290 | 
1291 | \f0\b\fs24 \cf0 2}</string>
1292 | 					</dict>
1293 | 				</dict>
1294 | 				<dict>
1295 | 					<key>Bounds</key>
1296 | 					<string>{{185, 457.57662963867188}, {64.5, 35}}</string>
1297 | 					<key>Class</key>
1298 | 					<string>ShapedGraphic</string>
1299 | 					<key>ID</key>
1300 | 					<integer>37800</integer>
1301 | 					<key>Shape</key>
1302 | 					<string>Rectangle</string>
1303 | 					<key>Text</key>
1304 | 					<dict>
1305 | 						<key>Text</key>
1306 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1307 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1308 | {\colortbl;\red255\green255\blue255;}
1309 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1310 | 
1311 | \f0\b\fs24 \cf0 prefix 2}</string>
1312 | 					</dict>
1313 | 				</dict>
1314 | 				<dict>
1315 | 					<key>Bounds</key>
1316 | 					<string>{{249.5, 457.57662963867188}, {29, 35}}</string>
1317 | 					<key>Class</key>
1318 | 					<string>ShapedGraphic</string>
1319 | 					<key>ID</key>
1320 | 					<integer>37801</integer>
1321 | 					<key>Shape</key>
1322 | 					<string>Rectangle</string>
1323 | 					<key>Text</key>
1324 | 					<dict>
1325 | 						<key>Text</key>
1326 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1327 | {\fonttbl\f0\fnil\fcharset222 Ayuthaya;}
1328 | {\colortbl;\red255\green255\blue255;}
1329 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1330 | 
1331 | \f0\fs28 \cf0 0}</string>
1332 | 					</dict>
1333 | 				</dict>
1334 | 				<dict>
1335 | 					<key>Bounds</key>
1336 | 					<string>{{278.5, 457.57662963867188}, {29, 35}}</string>
1337 | 					<key>Class</key>
1338 | 					<string>ShapedGraphic</string>
1339 | 					<key>ID</key>
1340 | 					<integer>37802</integer>
1341 | 					<key>Shape</key>
1342 | 					<string>Rectangle</string>
1343 | 					<key>Text</key>
1344 | 					<dict>
1345 | 						<key>Text</key>
1346 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1347 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1348 | {\colortbl;\red255\green255\blue255;}
1349 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1350 | 
1351 | \f0\b\fs24 \cf0 3}</string>
1352 | 					</dict>
1353 | 				</dict>
1354 | 				<dict>
1355 | 					<key>Bounds</key>
1356 | 					<string>{{307.5, 457.57662963867188}, {72, 35}}</string>
1357 | 					<key>Class</key>
1358 | 					<string>ShapedGraphic</string>
1359 | 					<key>ID</key>
1360 | 					<integer>37803</integer>
1361 | 					<key>Shape</key>
1362 | 					<string>Rectangle</string>
1363 | 					<key>Text</key>
1364 | 					<dict>
1365 | 						<key>Text</key>
1366 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1367 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1368 | {\colortbl;\red255\green255\blue255;}
1369 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1370 | 
1371 | \f0\b\fs24 \cf0 prefix 3}</string>
1372 | 					</dict>
1373 | 				</dict>
1374 | 				<dict>
1375 | 					<key>Bounds</key>
1376 | 					<string>{{379.5, 457.57662963867188}, {29, 35}}</string>
1377 | 					<key>Class</key>
1378 | 					<string>ShapedGraphic</string>
1379 | 					<key>ID</key>
1380 | 					<integer>37804</integer>
1381 | 					<key>Shape</key>
1382 | 					<string>Rectangle</string>
1383 | 					<key>Text</key>
1384 | 					<dict>
1385 | 						<key>Text</key>
1386 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1387 | {\fonttbl\f0\fnil\fcharset222 Ayuthaya;}
1388 | {\colortbl;\red255\green255\blue255;}
1389 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1390 | 
1391 | \f0\fs28 \cf0 0}</string>
1392 | 					</dict>
1393 | 				</dict>
1394 | 				<dict>
1395 | 					<key>Bounds</key>
1396 | 					<string>{{408.5, 457.57662963867188}, {29, 35}}</string>
1397 | 					<key>Class</key>
1398 | 					<string>ShapedGraphic</string>
1399 | 					<key>ID</key>
1400 | 					<integer>37805</integer>
1401 | 					<key>Shape</key>
1402 | 					<string>Rectangle</string>
1403 | 					<key>Text</key>
1404 | 					<dict>
1405 | 						<key>Text</key>
1406 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1407 | {\fonttbl\f0\fnil\fcharset222 Ayuthaya;}
1408 | {\colortbl;\red255\green255\blue255;}
1409 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1410 | 
1411 | \f0\fs28 \cf0 4}</string>
1412 | 					</dict>
1413 | 				</dict>
1414 | 			</array>
1415 | 			<key>GridV</key>
1416 | 			<array>
1417 | 				<integer>37796</integer>
1418 | 				<integer>37795</integer>
1419 | 				<integer>37797</integer>
1420 | 				<integer>37799</integer>
1421 | 				<integer>37800</integer>
1422 | 				<integer>37801</integer>
1423 | 				<integer>37802</integer>
1424 | 				<integer>37803</integer>
1425 | 				<integer>37804</integer>
1426 | 				<integer>37805</integer>
1427 | 				<array/>
1428 | 			</array>
1429 | 			<key>ID</key>
1430 | 			<integer>37794</integer>
1431 | 		</dict>
1432 | 		<dict>
1433 | 			<key>Bounds</key>
1434 | 			<string>{{19, 447}, {543, 58}}</string>
1435 | 			<key>Class</key>
1436 | 			<string>ShapedGraphic</string>
1437 | 			<key>ID</key>
1438 | 			<integer>37798</integer>
1439 | 			<key>Shape</key>
1440 | 			<string>Rectangle</string>
1441 | 			<key>Style</key>
1442 | 			<dict>
1443 | 				<key>stroke</key>
1444 | 				<dict>
1445 | 					<key>Pattern</key>
1446 | 					<integer>4</integer>
1447 | 				</dict>
1448 | 			</dict>
1449 | 			<key>Text</key>
1450 | 			<dict>
1451 | 				<key>Align</key>
1452 | 				<integer>2</integer>
1453 | 				<key>Text</key>
1454 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1455 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1456 | {\colortbl;\red255\green255\blue255;}
1457 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qr
1458 | 
1459 | \f0\b\fs24 \cf0 index block}</string>
1460 | 			</dict>
1461 | 			<key>TextPlacement</key>
1462 | 			<integer>2</integer>
1463 | 		</dict>
1464 | 		<dict>
1465 | 			<key>Bounds</key>
1466 | 			<string>{{339, 215.94891357421875}, {72, 15}}</string>
1467 | 			<key>Class</key>
1468 | 			<string>ShapedGraphic</string>
1469 | 			<key>FitText</key>
1470 | 			<string>YES</string>
1471 | 			<key>Flow</key>
1472 | 			<string>Resize</string>
1473 | 			<key>FontInfo</key>
1474 | 			<dict>
1475 | 				<key>Color</key>
1476 | 				<dict>
1477 | 					<key>w</key>
1478 | 					<string>0</string>
1479 | 				</dict>
1480 | 				<key>Font</key>
1481 | 				<string>LucidaGrande-Bold</string>
1482 | 				<key>Size</key>
1483 | 				<real>12</real>
1484 | 			</dict>
1485 | 			<key>ID</key>
1486 | 			<integer>37791</integer>
1487 | 			<key>Shape</key>
1488 | 			<string>Rectangle</string>
1489 | 			<key>Style</key>
1490 | 			<dict>
1491 | 				<key>fill</key>
1492 | 				<dict>
1493 | 					<key>Draws</key>
1494 | 					<string>NO</string>
1495 | 				</dict>
1496 | 				<key>shadow</key>
1497 | 				<dict>
1498 | 					<key>Draws</key>
1499 | 					<string>NO</string>
1500 | 				</dict>
1501 | 				<key>stroke</key>
1502 | 				<dict>
1503 | 					<key>Draws</key>
1504 | 					<string>NO</string>
1505 | 				</dict>
1506 | 			</dict>
1507 | 			<key>Text</key>
1508 | 			<dict>
1509 | 				<key>Align</key>
1510 | 				<integer>2</integer>
1511 | 				<key>Pad</key>
1512 | 				<integer>0</integer>
1513 | 				<key>Text</key>
1514 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1515 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1516 | {\colortbl;\red255\green255\blue255;}
1517 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qr
1518 | 
1519 | \f0\b\fs24 \cf0 data blocks}</string>
1520 | 				<key>VerticalPad</key>
1521 | 				<integer>0</integer>
1522 | 			</dict>
1523 | 			<key>Wrap</key>
1524 | 			<string>NO</string>
1525 | 		</dict>
1526 | 		<dict>
1527 | 			<key>Bounds</key>
1528 | 			<string>{{133.25, 76.423347473144531}, {79, 15}}</string>
1529 | 			<key>Class</key>
1530 | 			<string>ShapedGraphic</string>
1531 | 			<key>FitText</key>
1532 | 			<string>YES</string>
1533 | 			<key>Flow</key>
1534 | 			<string>Resize</string>
1535 | 			<key>FontInfo</key>
1536 | 			<dict>
1537 | 				<key>Color</key>
1538 | 				<dict>
1539 | 					<key>w</key>
1540 | 					<string>0</string>
1541 | 				</dict>
1542 | 				<key>Font</key>
1543 | 				<string>LucidaGrande-Bold</string>
1544 | 				<key>Size</key>
1545 | 				<real>12</real>
1546 | 			</dict>
1547 | 			<key>ID</key>
1548 | 			<integer>146</integer>
1549 | 			<key>Shape</key>
1550 | 			<string>Rectangle</string>
1551 | 			<key>Style</key>
1552 | 			<dict>
1553 | 				<key>fill</key>
1554 | 				<dict>
1555 | 					<key>Draws</key>
1556 | 					<string>NO</string>
1557 | 				</dict>
1558 | 				<key>shadow</key>
1559 | 				<dict>
1560 | 					<key>Draws</key>
1561 | 					<string>NO</string>
1562 | 				</dict>
1563 | 				<key>stroke</key>
1564 | 				<dict>
1565 | 					<key>Draws</key>
1566 | 					<string>NO</string>
1567 | 				</dict>
1568 | 			</dict>
1569 | 			<key>Text</key>
1570 | 			<dict>
1571 | 				<key>Align</key>
1572 | 				<integer>2</integer>
1573 | 				<key>Pad</key>
1574 | 				<integer>0</integer>
1575 | 				<key>Text</key>
1576 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1577 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1578 | {\colortbl;\red255\green255\blue255;}
1579 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qr
1580 | 
1581 | \f0\b\fs24 \cf0 index blocks}</string>
1582 | 				<key>VerticalPad</key>
1583 | 				<integer>0</integer>
1584 | 			</dict>
1585 | 			<key>Wrap</key>
1586 | 			<string>NO</string>
1587 | 		</dict>
1588 | 		<dict>
1589 | 			<key>Bounds</key>
1590 | 			<string>{{106.5, 99.948900000000009}, {132.5, 22}}</string>
1591 | 			<key>Class</key>
1592 | 			<string>ShapedGraphic</string>
1593 | 			<key>ID</key>
1594 | 			<integer>37789</integer>
1595 | 			<key>ImageID</key>
1596 | 			<integer>1</integer>
1597 | 			<key>Shape</key>
1598 | 			<string>Rectangle</string>
1599 | 			<key>Style</key>
1600 | 			<dict>
1601 | 				<key>fill</key>
1602 | 				<dict>
1603 | 					<key>Draws</key>
1604 | 					<string>NO</string>
1605 | 				</dict>
1606 | 				<key>shadow</key>
1607 | 				<dict>
1608 | 					<key>Draws</key>
1609 | 					<string>NO</string>
1610 | 				</dict>
1611 | 				<key>stroke</key>
1612 | 				<dict>
1613 | 					<key>Draws</key>
1614 | 					<string>NO</string>
1615 | 				</dict>
1616 | 			</dict>
1617 | 			<key>VFlip</key>
1618 | 			<string>YES</string>
1619 | 		</dict>
1620 | 		<dict>
1621 | 			<key>Class</key>
1622 | 			<string>TableGroup</string>
1623 | 			<key>Graphics</key>
1624 | 			<array>
1625 | 				<dict>
1626 | 					<key>Bounds</key>
1627 | 					<string>{{30, 130.47445678710938}, {58, 35}}</string>
1628 | 					<key>Class</key>
1629 | 					<string>ShapedGraphic</string>
1630 | 					<key>ID</key>
1631 | 					<integer>37783</integer>
1632 | 					<key>Shape</key>
1633 | 					<string>Rectangle</string>
1634 | 					<key>Text</key>
1635 | 					<dict>
1636 | 						<key>Text</key>
1637 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1638 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1639 | {\colortbl;\red255\green255\blue255;}
1640 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1641 | 
1642 | \f0\b\fs24 \cf0 header}</string>
1643 | 					</dict>
1644 | 				</dict>
1645 | 				<dict>
1646 | 					<key>Bounds</key>
1647 | 					<string>{{88, 130.47445678710938}, {79, 35}}</string>
1648 | 					<key>Class</key>
1649 | 					<string>ShapedGraphic</string>
1650 | 					<key>ID</key>
1651 | 					<integer>37784</integer>
1652 | 					<key>Shape</key>
1653 | 					<string>Rectangle</string>
1654 | 					<key>Text</key>
1655 | 					<dict>
1656 | 						<key>Text</key>
1657 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1658 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1659 | {\colortbl;\red255\green255\blue255;}
1660 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1661 | 
1662 | \f0\b\fs24 \cf0 block 1}</string>
1663 | 					</dict>
1664 | 				</dict>
1665 | 				<dict>
1666 | 					<key>Bounds</key>
1667 | 					<string>{{167, 130.47445678710938}, {79, 35}}</string>
1668 | 					<key>Class</key>
1669 | 					<string>ShapedGraphic</string>
1670 | 					<key>ID</key>
1671 | 					<integer>37785</integer>
1672 | 					<key>Shape</key>
1673 | 					<string>Rectangle</string>
1674 | 					<key>Text</key>
1675 | 					<dict>
1676 | 						<key>Text</key>
1677 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1678 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1679 | {\colortbl;\red255\green255\blue255;}
1680 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1681 | 
1682 | \f0\b\fs24 \cf0 block 2}</string>
1683 | 					</dict>
1684 | 				</dict>
1685 | 				<dict>
1686 | 					<key>Bounds</key>
1687 | 					<string>{{246, 130.47445678710938}, {79, 35}}</string>
1688 | 					<key>Class</key>
1689 | 					<string>ShapedGraphic</string>
1690 | 					<key>ID</key>
1691 | 					<integer>37786</integer>
1692 | 					<key>Shape</key>
1693 | 					<string>Rectangle</string>
1694 | 					<key>Text</key>
1695 | 					<dict>
1696 | 						<key>Text</key>
1697 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1698 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1699 | {\colortbl;\red255\green255\blue255;}
1700 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1701 | 
1702 | \f0\b\fs24 \cf0 block 3}</string>
1703 | 					</dict>
1704 | 				</dict>
1705 | 				<dict>
1706 | 					<key>Bounds</key>
1707 | 					<string>{{325, 130.47445678710938}, {79, 35}}</string>
1708 | 					<key>Class</key>
1709 | 					<string>ShapedGraphic</string>
1710 | 					<key>ID</key>
1711 | 					<integer>37787</integer>
1712 | 					<key>Shape</key>
1713 | 					<string>Rectangle</string>
1714 | 					<key>Text</key>
1715 | 					<dict>
1716 | 						<key>Text</key>
1717 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1718 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1719 | {\colortbl;\red255\green255\blue255;}
1720 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1721 | 
1722 | \f0\b\fs24 \cf0 ...}</string>
1723 | 					</dict>
1724 | 				</dict>
1725 | 				<dict>
1726 | 					<key>Bounds</key>
1727 | 					<string>{{404, 130.47445678710938}, {79, 35}}</string>
1728 | 					<key>Class</key>
1729 | 					<string>ShapedGraphic</string>
1730 | 					<key>ID</key>
1731 | 					<integer>37788</integer>
1732 | 					<key>Shape</key>
1733 | 					<string>Rectangle</string>
1734 | 					<key>Text</key>
1735 | 					<dict>
1736 | 						<key>Text</key>
1737 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1738 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1739 | {\colortbl;\red255\green255\blue255;}
1740 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1741 | 
1742 | \f0\b\fs24 \cf0 block n}</string>
1743 | 					</dict>
1744 | 				</dict>
1745 | 			</array>
1746 | 			<key>GridV</key>
1747 | 			<array>
1748 | 				<integer>37783</integer>
1749 | 				<integer>37784</integer>
1750 | 				<integer>37785</integer>
1751 | 				<integer>37786</integer>
1752 | 				<integer>37787</integer>
1753 | 				<integer>37788</integer>
1754 | 				<array/>
1755 | 			</array>
1756 | 			<key>ID</key>
1757 | 			<integer>37781</integer>
1758 | 		</dict>
1759 | 		<dict>
1760 | 			<key>Bounds</key>
1761 | 			<string>{{262, 174.00001335144043}, {216, 41.948900000000002}}</string>
1762 | 			<key>Class</key>
1763 | 			<string>ShapedGraphic</string>
1764 | 			<key>ID</key>
1765 | 			<integer>37780</integer>
1766 | 			<key>ImageID</key>
1767 | 			<integer>1</integer>
1768 | 			<key>Shape</key>
1769 | 			<string>Rectangle</string>
1770 | 			<key>Style</key>
1771 | 			<dict>
1772 | 				<key>fill</key>
1773 | 				<dict>
1774 | 					<key>Draws</key>
1775 | 					<string>NO</string>
1776 | 				</dict>
1777 | 				<key>shadow</key>
1778 | 				<dict>
1779 | 					<key>Draws</key>
1780 | 					<string>NO</string>
1781 | 				</dict>
1782 | 				<key>stroke</key>
1783 | 				<dict>
1784 | 					<key>Draws</key>
1785 | 					<string>NO</string>
1786 | 				</dict>
1787 | 			</dict>
1788 | 		</dict>
1789 | 		<dict>
1790 | 			<key>Class</key>
1791 | 			<string>TableGroup</string>
1792 | 			<key>Graphics</key>
1793 | 			<array>
1794 | 				<dict>
1795 | 					<key>Bounds</key>
1796 | 					<string>{{31.5, 314}, {109, 35}}</string>
1797 | 					<key>Class</key>
1798 | 					<string>ShapedGraphic</string>
1799 | 					<key>ID</key>
1800 | 					<integer>211</integer>
1801 | 					<key>Shape</key>
1802 | 					<string>Rectangle</string>
1803 | 					<key>Text</key>
1804 | 					<dict>
1805 | 						<key>Text</key>
1806 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1807 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1808 | {\colortbl;\red255\green255\blue255;}
1809 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1810 | 
1811 | \f0\b\fs24 \cf0 block size}</string>
1812 | 					</dict>
1813 | 				</dict>
1814 | 				<dict>
1815 | 					<key>Bounds</key>
1816 | 					<string>{{140.5, 314}, {109, 35}}</string>
1817 | 					<key>Class</key>
1818 | 					<string>ShapedGraphic</string>
1819 | 					<key>ID</key>
1820 | 					<integer>222</integer>
1821 | 					<key>Shape</key>
1822 | 					<string>Rectangle</string>
1823 | 					<key>Text</key>
1824 | 					<dict>
1825 | 						<key>Text</key>
1826 | 						<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1827 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1828 | {\colortbl;\red255\green255\blue255;}
1829 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qc
1830 | 
1831 | \f0\b\fs24 \cf0 block count}</string>
1832 | 					</dict>
1833 | 				</dict>
1834 | 			</array>
1835 | 			<key>GridV</key>
1836 | 			<array>
1837 | 				<integer>211</integer>
1838 | 				<integer>222</integer>
1839 | 				<array/>
1840 | 			</array>
1841 | 			<key>ID</key>
1842 | 			<integer>209</integer>
1843 | 		</dict>
1844 | 		<dict>
1845 | 			<key>Bounds</key>
1846 | 			<string>{{25, 309}, {231, 68}}</string>
1847 | 			<key>Class</key>
1848 | 			<string>ShapedGraphic</string>
1849 | 			<key>ID</key>
1850 | 			<integer>37790</integer>
1851 | 			<key>Shape</key>
1852 | 			<string>Rectangle</string>
1853 | 			<key>Style</key>
1854 | 			<dict>
1855 | 				<key>stroke</key>
1856 | 				<dict>
1857 | 					<key>Pattern</key>
1858 | 					<integer>4</integer>
1859 | 				</dict>
1860 | 			</dict>
1861 | 			<key>Text</key>
1862 | 			<dict>
1863 | 				<key>Align</key>
1864 | 				<integer>2</integer>
1865 | 				<key>Text</key>
1866 | 				<string>{\rtf1\ansi\ansicpg1252\cocoartf1138\cocoasubrtf510
1867 | {\fonttbl\f0\fnil\fcharset0 LucidaGrande;}
1868 | {\colortbl;\red255\green255\blue255;}
1869 | \pard\tx560\tx1120\tx1680\tx2240\tx2800\tx3360\tx3920\tx4480\tx5040\tx5600\tx6160\tx6720\pardirnatural\qr
1870 | 
1871 | \f0\b\fs24 \cf0 header}</string>
1872 | 			</dict>
1873 | 			<key>TextPlacement</key>
1874 | 			<integer>2</integer>
1875 | 		</dict>
1876 | 	</array>
1877 | 	<key>GridInfo</key>
1878 | 	<dict/>
1879 | 	<key>GuidesLocked</key>
1880 | 	<string>NO</string>
1881 | 	<key>GuidesVisible</key>
1882 | 	<string>YES</string>
1883 | 	<key>HPages</key>
1884 | 	<integer>1</integer>
1885 | 	<key>ImageCounter</key>
1886 | 	<integer>2</integer>
1887 | 	<key>ImageLinkBack</key>
1888 | 	<array>
1889 | 		<dict>
1890 | 			<key>ApplicationURL</key>
1891 | 			<string>http://www.omnigroup.com/applications/OmniGraffle</string>
1892 | 			<key>appData</key>
1893 | 			<dict>
1894 | 				<key>Color</key>
1895 | 				<dict>
1896 | 					<key>w</key>
1897 | 					<string>1</string>
1898 | 				</dict>
1899 | 				<key>DocumentSettings</key>
1900 | 				<dict>
1901 | 					<key>ApplicationVersion</key>
1902 | 					<array>
1903 | 						<string>com.omnigroup.OmniGrafflePro</string>
1904 | 						<string>137.11.0.108132</string>
1905 | 					</array>
1906 | 					<key>FileName</key>
1907 | 					<string>Base Wireframe Kit.gstencil</string>
1908 | 					<key>GraphDocumentVersion</key>
1909 | 					<integer>6</integer>
1910 | 					<key>ModelCount</key>
1911 | 					<integer>1</integer>
1912 | 					<key>ModelIndex</key>
1913 | 					<integer>0</integer>
1914 | 					<key>ModificationDate</key>
1915 | 					<string>2009-03-06 10:18:30 -0500</string>
1916 | 					<key>Modifier</key>
1917 | 					<string>Michael Angeles</string>
1918 | 					<key>SheetTitle</key>
1919 | 					<string>Canvas 1</string>
1920 | 				</dict>
1921 | 				<key>GraphicsList</key>
1922 | 				<array>
1923 | 					<dict>
1924 | 						<key>Class</key>
1925 | 						<string>LineGraphic</string>
1926 | 						<key>ControlPoints</key>
1927 | 						<array>
1928 | 							<string>{0.331787, 0.0552979}</string>
1929 | 							<string>{-80, -1}</string>
1930 | 							<string>{31, 0}</string>
1931 | 							<string>{0, 0}</string>
1932 | 							<string>{0, 0}</string>
1933 | 							<string>{-30, 1}</string>
1934 | 							<string>{79, 0}</string>
1935 | 							<string>{-0.231934, 0.712708}</string>
1936 | 						</array>
1937 | 						<key>ID</key>
1938 | 						<integer>37779</integer>
1939 | 						<key>LayerIndex</key>
1940 | 						<integer>0</integer>
1941 | 						<key>Points</key>
1942 | 						<array>
1943 | 							<string>{301.717, 414.589}</string>
1944 | 							<string>{381.717, 435.589}</string>
1945 | 							<string>{421.717, 453.589}</string>
1946 | 							<string>{463.717, 434.589}</string>
1947 | 							<string>{542.717, 414.589}</string>
1948 | 						</array>
1949 | 						<key>Style</key>
1950 | 						<dict>
1951 | 							<key>stroke</key>
1952 | 							<dict>
1953 | 								<key>Bezier</key>
1954 | 								<true/>
1955 | 								<key>Color</key>
1956 | 								<dict>
1957 | 									<key>b</key>
1958 | 									<string>0.998718</string>
1959 | 									<key>g</key>
1960 | 									<string>0.781033</string>
1961 | 									<key>r</key>
1962 | 									<string>0.424323</string>
1963 | 								</dict>
1964 | 								<key>HeadArrow</key>
1965 | 								<string>0</string>
1966 | 								<key>LineType</key>
1967 | 								<integer>1</integer>
1968 | 								<key>TailArrow</key>
1969 | 								<string>0</string>
1970 | 								<key>Width</key>
1971 | 								<real>2</real>
1972 | 							</dict>
1973 | 						</dict>
1974 | 					</dict>
1975 | 				</array>
1976 | 				<key>Layers</key>
1977 | 				<array>
1978 | 					<dict>
1979 | 						<key>Lock</key>
1980 | 						<string>NO</string>
1981 | 						<key>Name</key>
1982 | 						<string>Layer 1</string>
1983 | 						<key>Print</key>
1984 | 						<string>YES</string>
1985 | 						<key>View</key>
1986 | 						<string>YES</string>
1987 | 					</dict>
1988 | 				</array>
1989 | 				<key>ZoomLevel</key>
1990 | 				<real>1</real>
1991 | 			</dict>
1992 | 			<key>bundleId</key>
1993 | 			<string>com.omnigroup.OmniGrafflePro</string>
1994 | 			<key>refresh</key>
1995 | 			<real>0.0</real>
1996 | 			<key>serverAppName</key>
1997 | 			<string>OmniGraffle</string>
1998 | 			<key>serverName</key>
1999 | 			<string>OmniGraffle</string>
2000 | 			<key>version</key>
2001 | 			<string>A</string>
2002 | 		</dict>
2003 | 	</array>
2004 | 	<key>ImageList</key>
2005 | 	<array>
2006 | 		<string>image1.pdf</string>
2007 | 	</array>
2008 | 	<key>KeepToScale</key>
2009 | 	<false/>
2010 | 	<key>Layers</key>
2011 | 	<array>
2012 | 		<dict>
2013 | 			<key>Lock</key>
2014 | 			<string>NO</string>
2015 | 			<key>Name</key>
2016 | 			<string>Layer 1</string>
2017 | 			<key>Print</key>
2018 | 			<string>YES</string>
2019 | 			<key>View</key>
2020 | 			<string>YES</string>
2021 | 		</dict>
2022 | 	</array>
2023 | 	<key>LayoutInfo</key>
2024 | 	<dict>
2025 | 		<key>Animate</key>
2026 | 		<string>NO</string>
2027 | 		<key>circoMinDist</key>
2028 | 		<real>18</real>
2029 | 		<key>circoSeparation</key>
2030 | 		<real>0.0</real>
2031 | 		<key>layoutEngine</key>
2032 | 		<string>dot</string>
2033 | 		<key>neatoSeparation</key>
2034 | 		<real>0.0</real>
2035 | 		<key>twopiSeparation</key>
2036 | 		<real>0.0</real>
2037 | 	</dict>
2038 | 	<key>LinksVisible</key>
2039 | 	<string>NO</string>
2040 | 	<key>MagnetsVisible</key>
2041 | 	<string>NO</string>
2042 | 	<key>MasterSheets</key>
2043 | 	<array/>
2044 | 	<key>ModificationDate</key>
2045 | 	<string>2012-11-13 18:45:51 +0000</string>
2046 | 	<key>Modifier</key>
2047 | 	<string>Scott Robertson</string>
2048 | 	<key>NotesVisible</key>
2049 | 	<string>NO</string>
2050 | 	<key>Orientation</key>
2051 | 	<integer>2</integer>
2052 | 	<key>OriginVisible</key>
2053 | 	<string>NO</string>
2054 | 	<key>PageBreaks</key>
2055 | 	<string>YES</string>
2056 | 	<key>PrintInfo</key>
2057 | 	<dict>
2058 | 		<key>NSBottomMargin</key>
2059 | 		<array>
2060 | 			<string>float</string>
2061 | 			<string>41</string>
2062 | 		</array>
2063 | 		<key>NSHorizonalPagination</key>
2064 | 		<array>
2065 | 			<string>coded</string>
2066 | 			<string>BAtzdHJlYW10eXBlZIHoA4QBQISEhAhOU051bWJlcgCEhAdOU1ZhbHVlAISECE5TT2JqZWN0AIWEASqEhAFxlwCG</string>
2067 | 		</array>
2068 | 		<key>NSLeftMargin</key>
2069 | 		<array>
2070 | 			<string>float</string>
2071 | 			<string>18</string>
2072 | 		</array>
2073 | 		<key>NSPaperSize</key>
2074 | 		<array>
2075 | 			<string>size</string>
2076 | 			<string>{612, 792}</string>
2077 | 		</array>
2078 | 		<key>NSPrintReverseOrientation</key>
2079 | 		<array>
2080 | 			<string>int</string>
2081 | 			<string>0</string>
2082 | 		</array>
2083 | 		<key>NSRightMargin</key>
2084 | 		<array>
2085 | 			<string>float</string>
2086 | 			<string>18</string>
2087 | 		</array>
2088 | 		<key>NSTopMargin</key>
2089 | 		<array>
2090 | 			<string>float</string>
2091 | 			<string>18</string>
2092 | 		</array>
2093 | 	</dict>
2094 | 	<key>PrintOnePage</key>
2095 | 	<false/>
2096 | 	<key>ReadOnly</key>
2097 | 	<string>NO</string>
2098 | 	<key>RowAlign</key>
2099 | 	<integer>1</integer>
2100 | 	<key>RowSpacing</key>
2101 | 	<real>36</real>
2102 | 	<key>SheetTitle</key>
2103 | 	<string>Canvas 1</string>
2104 | 	<key>SmartAlignmentGuidesActive</key>
2105 | 	<string>YES</string>
2106 | 	<key>SmartDistanceGuidesActive</key>
2107 | 	<string>YES</string>
2108 | 	<key>UniqueID</key>
2109 | 	<integer>1</integer>
2110 | 	<key>UseEntirePage</key>
2111 | 	<false/>
2112 | 	<key>VPages</key>
2113 | 	<integer>2</integer>
2114 | 	<key>WindowInfo</key>
2115 | 	<dict>
2116 | 		<key>CurrentSheet</key>
2117 | 		<integer>0</integer>
2118 | 		<key>ExpandedCanvases</key>
2119 | 		<array>
2120 | 			<dict>
2121 | 				<key>name</key>
2122 | 				<string>Canvas 1</string>
2123 | 			</dict>
2124 | 		</array>
2125 | 		<key>Frame</key>
2126 | 		<string>{{241, 0}, {919, 874}}</string>
2127 | 		<key>ListView</key>
2128 | 		<true/>
2129 | 		<key>OutlineWidth</key>
2130 | 		<integer>142</integer>
2131 | 		<key>RightSidebar</key>
2132 | 		<false/>
2133 | 		<key>ShowRuler</key>
2134 | 		<true/>
2135 | 		<key>Sidebar</key>
2136 | 		<true/>
2137 | 		<key>SidebarWidth</key>
2138 | 		<integer>120</integer>
2139 | 		<key>VisibleRegion</key>
2140 | 		<string>{{-104, 421}, {784, 719}}</string>
2141 | 		<key>Zoom</key>
2142 | 		<real>1</real>
2143 | 		<key>ZoomValues</key>
2144 | 		<array>
2145 | 			<array>
2146 | 				<string>Canvas 1</string>
2147 | 				<real>1</real>
2148 | 				<real>1</real>
2149 | 			</array>
2150 | 		</array>
2151 | 	</dict>
2152 | </dict>
2153 | </plist>
2154 | 


--------------------------------------------------------------------------------
/docs/file_format.graffle/image1.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/file_format.graffle/image1.pdf


--------------------------------------------------------------------------------
/docs/file_overview.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/file_overview.png


--------------------------------------------------------------------------------
/docs/header.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/header.png


--------------------------------------------------------------------------------
/docs/index_block.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/index_block.png


--------------------------------------------------------------------------------
/docs/project_header.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/project_header.png


--------------------------------------------------------------------------------
/docs/tree.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/trivio/common_crawl_index/235ba4dd81474c741908e8a0cc29604cbcada545/docs/tree.png


--------------------------------------------------------------------------------
/lib/__init__.py:
--------------------------------------------------------------------------------
 1 | # Copyright [2012] [Triv.io, Scott Robertson]
 2 | # 
 3 | #   Licensed under the Apache License, Version 2.0 (the "License");
 4 | #   you may not use this file except in compliance with the License.
 5 | #   You may obtain a copy of the License at
 6 | # 
 7 | #       http://www.apache.org/licenses/LICENSE-2.0
 8 | # 
 9 | #   Unless required by applicable law or agreed to in writing, software
10 | #   distributed under the License is distributed on an "AS IS" BASIS,
11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | #   See the License for the specific language governing permissions and
13 | #   limitations under the License.
14 | # 
15 | 
16 | import urlparse
17 | 
18 | def reversehost(url):
19 | # reverse netlocation http://www.example.com/foo -> com.example.www/foo:http
20 |   url = urlparse.urlsplit(str(url))
21 |   
22 |   netloc = url.netloc.split(':')
23 |   host = netloc[0]
24 |   if len(netloc) == 2:
25 |     port = ':' + netloc[1]
26 |   else:
27 |     port = ''
28 |   
29 | 
30 |   # reverse the host
31 |   host = '.'.join(reversed(host.split('.')))
32 | 
33 |   return (
34 |     host + 
35 |     url.path + 
36 |     (('?' + url.query) if url.query else '' ) +
37 |     port +
38 |     ':' + url.scheme
39 |   )
40 | 
41 | 


--------------------------------------------------------------------------------
/lib/adaptor.py:
--------------------------------------------------------------------------------
 1 | # Copyright [2012] [Triv.io, Scott Robertson]
 2 | # 
 3 | #   Licensed under the Apache License, Version 2.0 (the "License");
 4 | #   you may not use this file except in compliance with the License.
 5 | #   You may obtain a copy of the License at
 6 | # 
 7 | #       http://www.apache.org/licenses/LICENSE-2.0
 8 | # 
 9 | #   Unless required by applicable law or agreed to in writing, software
10 | #   distributed under the License is distributed on an "AS IS" BASIS,
11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | #   See the License for the specific language governing permissions and
13 | #   limitations under the License.
14 | # 
15 | 
16 | import re
17 | import struct
18 | 
19 | 
20 | from triv.io import datasources
21 | from .pbtree import PBTreeDictWriter, DataWriter, IndexWriter, DataBlockReader
22 | 
23 | BLOCK_SIZE   = 2**16
24 | VALUE_FORMAT = "<QQIQ"
25 | VALUE_SIZE   = struct.calcsize(VALUE_FORMAT)
26 | 
27 | ITEM_KEYS=(
28 |   'arcSourceSegmentId',
29 |   'arcFileDate',
30 |   'arcFileParition',
31 |   'arcFileOffset'
32 | )
33 | 
34 | 
35 | @datasources.write_mimetype('application/vnd.commoncrawl.org.url_index')
36 | def output_stream(stream):
37 |   return PBTreeDictWriter(stream, 
38 |     value_format=VALUE_FORMAT,
39 |     block_size =BLOCK_SIZE,
40 |     item_keys=ITEM_KEYS
41 |   )
42 |   
43 |   
44 | @datasources.write_mimetype('application/vnd.commoncrawl.org.data_segment')
45 | def output_stream(stream):
46 |   class DictPacker(object):
47 |     def __init__(self, item_keys, value_format):
48 |       self.item_keys = item_keys
49 |       self.value_format = value_format
50 |       self.block_count = 0
51 | 
52 |     def pack_value(self, dict):
53 |       value = [dict[k] for k in self.item_keys]      
54 |       return struct.pack(self.value_format, *value)
55 |     
56 |     def on_new_block(self, key):
57 |       self.block_count += 1
58 |         
59 |     def on_item_exceeds_block_size(self, key,value):
60 |       print ""
61 |   
62 |   return DataWriter(stream, BLOCK_SIZE, '\0', DictPacker(ITEM_KEYS, VALUE_FORMAT))
63 |   
64 | @datasources.write_mimetype('application/vnd.commoncrawl.org.index_segment')
65 | def output_stream(stream):
66 |   return IndexWriter(stream, block_size=BLOCK_SIZE, terminator='\0')
67 | 
68 | 
69 | 
70 | @datasources.read_mimetype('application/vnd.commoncrawl.org.data_segment')
71 | def data_block_reader(fd, size, url, params):
72 |   part_number = "%04d" % int(re.search(':reduce:(\d+)\-',url).group(1))
73 | 
74 |   # read each block in the part, yielding the first key and part number
75 |   
76 |   params.last_key = None
77 |   
78 |   while True:
79 |     block_buffer = fd.read(BLOCK_SIZE)
80 |     if block_buffer == '':
81 |       break
82 |     block = iter(DataBlockReader(block_buffer, VALUE_SIZE))
83 |     yield part_number, block
84 |   


--------------------------------------------------------------------------------
/lib/pbtree.py:
--------------------------------------------------------------------------------
  1 | # Copyright [2012] triv.io, Scott Robertson
  2 | # 
  3 | #   Licensed under the Apache License, Version 2.0 (the "License");
  4 | #   you may not use this file except in compliance with the License.
  5 | #   You may obtain a copy of the License at
  6 | # 
  7 | #       http://www.apache.org/licenses/LICENSE-2.0
  8 | # 
  9 | #   Unless required by applicable law or agreed to in writing, software
 10 | #   distributed under the License is distributed on an "AS IS" BASIS,
 11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | #   See the License for the specific language governing permissions and
 13 | #   limitations under the License.
 14 | # 
 15 | 
 16 | import bisect
 17 | import os
 18 | import mmap
 19 | import struct
 20 | import sys
 21 | from tempfile import TemporaryFile, SpooledTemporaryFile
 22 | from cStringIO import StringIO
 23 | import itertools
 24 | 
 25 | from .prefix import signifigant
 26 | 
 27 | MB = 1024**2
 28 | DISK_BLOCK_SIZE=1024 * 4
 29 | 
 30 | OFFSET_FMT = '<I'
 31 | OFFSET_SIZE = struct.calcsize(OFFSET_FMT)
 32 | 
 33 | 
 34 | 
 35 | class PBTreeWriter(object):
 36 |   """
 37 |   Constructs a disk based prefixed btreefor a sacalr value.
 38 |   """
 39 |     
 40 |   def __init__(self, stream, block_size=MB, terminator='\0', value_format="<Q"):
 41 |     self.stream = stream
 42 |     
 43 |     assert len(terminator) == 1, "terminator must be of legth 1"
 44 |     
 45 |     self.value_format = value_format
 46 |     self.value_size = struct.calcsize(self.value_format)
 47 |       
 48 |     self.block_size = block_size
 49 | 
 50 |     self.last_key = ''
 51 |     
 52 |     self.data_segment  = DataWriter(TemporaryFile(), block_size, terminator, self)
 53 |     self.index_segment = IndexWriter(stream, block_size, terminator)
 54 | 
 55 |   
 56 |   def pack_value(self, value):
 57 |     return struct.pack(self.value_format, value)
 58 | 
 59 | 
 60 |   def add(self, key, value):
 61 |     self.data_segment.add(key,value)
 62 |     self.last_key = key
 63 |     
 64 | 
 65 |   def on_new_block(self, key):
 66 |     prefix_key = signifigant(self.last_key, key)
 67 |     self.index_segment.add(0, prefix_key)
 68 | 
 69 | 
 70 |   def on_item_exceeds_block_size(self,key,value):
 71 |     raise ValueError("key '%s'  exceeds block size" % key)
 72 | 
 73 | 
 74 |   def commit(self):
 75 |     out = self.stream
 76 |     
 77 |     self.index_segment.finish()    
 78 |     self.data_segment.finish()
 79 |     
 80 |     while True:
 81 |       bytes = self.data_segment.read(DISK_BLOCK_SIZE)
 82 |       if bytes == '':
 83 |         break
 84 |       else:
 85 |         out.write(bytes)
 86 | 
 87 |     
 88 |   def close(self):
 89 |     self.commit()
 90 |     self.stream.close()
 91 |   
 92 | class PBTreeSequenceWriter(PBTreeWriter):
 93 |   """
 94 |   Writes a sequence to a pbtree, it assumes that an appropriate value_format
 95 |   is specified and that every sequence passed to add() is the same length.
 96 |   
 97 |   """
 98 | 
 99 |   def  pack_value(self, value):
100 |     # this assumes an appropriate value_format was specified
101 |     return struct.pack(self.value_format, *value)
102 |   
103 | class PBTreeDictWriter(PBTreeWriter):
104 |   """
105 |   Writes a dictionary to a pbtree 
106 |   """
107 |   
108 |   def __init__(self, stream, item_keys, **options):
109 |     self.item_keys = item_keys
110 |     super(PBTreeDictWriter, self).__init__(stream, **options)
111 | 
112 |   def pack_value(self, dict):
113 |     value = [dict[k] for k in self.item_keys]      
114 |     return struct.pack(self.value_format, *value)
115 | 
116 | 
117 | 
118 | class PBTreeReader(object):
119 |   """
120 |   Reads a prefixed btree
121 |   """
122 |   
123 |   @classmethod
124 |   def parse(cls, stream, block_size):
125 |     while True:
126 |       block = stream.read(block_size)
127 |       if block == '':
128 |         break
129 |       for offset, key in IndexBlockReader(block):
130 |         yield offset, key
131 |     
132 |   def __init__(self, mmap, terminator='\0', value_format="<Q"):
133 |     self.mmap = mmap
134 |     #self.mmap.seek(0)
135 |     self.terminator = terminator
136 |     self.value_format = value_format
137 |     self.value_size = struct.calcsize(self.value_format)
138 |     
139 | 
140 |     self.header_fmt = "<II"
141 |     self.header_size = struct.calcsize(self.header_fmt)
142 |     
143 |     self.block_size, self.index_block_size = self.fetch_header()
144 |   
145 |   
146 |   def fetch(self, start, end):
147 |     return self.mmap[start:end]
148 |     
149 |   def fetch_header(self):
150 |     return struct.unpack(self.header_fmt, self.fetch(0,self.header_size))
151 |           
152 |   def block(self, block_number):
153 |     """
154 |     Returns the block for given block number 
155 |     """
156 |     
157 |     offset = self.block_offset(block_number)
158 |     block = self.fetch(offset, offset+self.block_size)    
159 |     return IndexBlockReader(block)
160 |     
161 |   def block_offset(self, block_number):
162 |     return self.header_size + (self.block_size*block_number)
163 |     
164 |   def count_levels(self):
165 |     """Return the number of 'levels' in the index. This number represents
166 |     how many seeks have to be preformed before finding
167 |     the starting data block"""
168 |     
169 |     
170 |     block_number = 0
171 |     block = self.block(block_number)
172 |     levels = 1
173 |     
174 |     while True:
175 |       buffer = StringIO(block.data)
176 |       next_block_number = block.read_offset(buffer)
177 |       if next_block_number < self.index_block_size:
178 |         block_number = next_block_number
179 |         block = self.block(block_number)
180 |         levels += 1
181 |       else:
182 |         # it's the start of the data segments,
183 |         return levels
184 | 
185 |     
186 |   def expected_location(self, key):
187 |     """
188 |     Given a key, return the expected starting location for the key in the data
189 |     segment. The return value is only the location  where the key "should" be
190 |     not neccesarily where it is. This enables range queries
191 |      
192 |     """
193 |     if key == '':
194 |       start_block = self.index_block_size
195 |       return self.block_offset(start_block)
196 |     
197 |     starting_block = self.find_starting_data_block(key)
198 |     offset = self.block_offset(starting_block)
199 |     data = self.fetch(offset,offset+self.block_size)
200 | 
201 |     
202 |     # linear scan through the block, looking for the position of the stored key
203 |     # that is greater than the given key
204 |     
205 |     start = 0
206 |     loc = 0
207 |     while True:
208 |       pos = data.find(self.terminator, start)
209 |       if pos == -1:
210 |         # can this really happen?
211 |         return len(data)
212 |       stored = data[start:pos]
213 |       if key <= stored:
214 |         return start + offset
215 |       else:
216 |         start = pos + 1 + self.value_size
217 |         
218 | 
219 | 
220 |   def find_starting_data_block(self,key):
221 |     """
222 |     Return the number of the first data block where the key should be found.
223 |     """
224 | 
225 |     block_number = 0
226 |     block = self.block(block_number)
227 |     
228 |     while True:
229 |       next_block_number = block.find(key)
230 |       if next_block_number < self.index_block_size:
231 |         block_number = next_block_number
232 |         block = self.block(block_number)
233 |       else:
234 |         # it's the start of the data segments,
235 |         return next_block_number
236 |     
237 |   def keys(self, prefix=''):
238 |     return list(self.keyiter(prefix))
239 | 
240 |   def keyiter(self, prefix=''):
241 |     for key, value in self.itemsiter(prefix):
242 |       if key.startswith(prefix):
243 |         yield key
244 | 
245 |   def values(self, prefix=''):
246 |     return list(self.valueiter(prefix))
247 |     
248 |   def valueiter(self, prefix=''):
249 |     for key, value in self.itemsiter(prefix):
250 |       if key.startswith(prefix):
251 |         yield value
252 |       
253 | 
254 |   def items(self, prefix=''):
255 |     return list(self.itemsiter(prefix))
256 | 
257 |   def itemsiter(self, prefix):
258 |     starting_block = self.find_starting_data_block(prefix)
259 |     blocks = self.blockiter(starting_block)
260 | 
261 |     # skip over keys in the first block
262 |     first_block = blocks.next()
263 |     for key,value in itertools.dropwhile(lambda (k,v): not k.startswith(prefix), self.dataiter(first_block)):
264 |       if not key.startswith(prefix):
265 |         return
266 |       yield key,value
267 | 
268 |     for block in blocks:
269 |       for key,value in self.dataiter(block):
270 |         if not key.startswith(prefix):
271 |           return
272 |         else:
273 |           yield key,value
274 |     
275 |     
276 |   def parse_value(self, bytes):
277 |     """
278 |     Returns a scalar
279 |     """
280 |     return struct.unpack(
281 |       self.value_format,
282 |       bytes
283 |     )[0]
284 |     
285 | 
286 |   def blockiter(self, block_number):
287 |     """
288 |     Iterate over blocks starting with the given block_number
289 |     """
290 | 
291 |     while True:
292 |       offset = self.block_offset(block_number)
293 |       block = self.mmap[offset:offset+self.block_size]
294 |       if block:
295 |         yield block
296 |         block_number += 1
297 |       else:
298 |         break
299 |         
300 |   def dataiter(self, block):    
301 |     for key, value in DataBlockReader(block, self.value_size, self.terminator):
302 |       yield key, self.parse_value(value)
303 | 
304 | 
305 | class PBTreeDictReader(PBTreeReader):
306 |   def __init__(self, stream, item_keys, **options):
307 |     self.item_keys = item_keys
308 |     super(PBTreeDictReader, self).__init__(stream, **options)
309 |     
310 |   def parse_value(self, bytes):
311 |     """
312 |     Returns a dictionary
313 |     """
314 |     
315 |     return dict(zip(
316 |       self.item_keys,
317 |       struct.unpack(self.value_format,bytes)
318 |     ))
319 |     
320 | ####
321 | # Lower level constructs whose functionality has been seperated out to make
322 | # usabale within map/reduce constructs
323 | ###
324 | 
325 | class DataWriter(object):
326 |   def __init__(self, stream, block_size, terminator, delegate):
327 |     self.finalized = False
328 |     self.delegate = delegate
329 |     
330 |     self.block_size = block_size
331 |     self.remaining = self.block_size
332 |         
333 |     self.terminator = terminator
334 |     self.term_length = len(self.terminator)
335 |     
336 |     self.stream = stream
337 |     self.write_buffer = bytearray()
338 |         
339 |     
340 |   def add(self, key, value):
341 | 
342 |     packet = self.delegate.pack_value(value)
343 |     size = len(key) + self.term_length + len(packet)
344 |     
345 |     if size > self.block_size:
346 |       self.delegate.on_item_exceeds_block_size(key,value)
347 |       return
348 |     
349 |     if size > self.remaining:
350 |       self.write_buffer.extend(self.terminator * self.remaining)
351 |       self.stream.write(self.write_buffer)
352 |       del self.write_buffer[:]
353 |       self.remaining = self.block_size
354 |       self.delegate.on_new_block(key)
355 |       
356 |     self.write_buffer.extend(key)
357 |     self.write_buffer.extend(self.terminator)
358 |     self.write_buffer.extend(packet)
359 | 
360 |     self.remaining -= size
361 |  
362 |   def close(self):
363 |     if not self.finalized:
364 |       self.finish()
365 |     self.stream.close()
366 |     
367 |   def finish(self):
368 |     if self.write_buffer:
369 |       self.write_buffer.extend(self.terminator * self.remaining)
370 |       self.stream.write(self.write_buffer)
371 |     
372 |     del self.write_buffer
373 |     self.delegate = None
374 |     self.stream.seek(0)
375 |     self.finalized = True
376 |     
377 |   def read(self, bytes = None):
378 |     """
379 |     Returns bytes written to the stream.
380 |     
381 |     It is an error to call this method prior to calling DataWriter.finish()
382 |     """
383 |     return self.stream.read(bytes)
384 |   
385 | class IndexWriter(object):
386 |   def __init__(self, stream,  block_size, terminator, pointer_format='<I'):
387 |     self.stream = stream
388 |     self.block_size = block_size
389 |     self.terminator = terminator
390 |     self.term_size = len(terminator)
391 |     
392 |     self.pointer_format = pointer_format
393 |     self.pointer_size = struct.calcsize(pointer_format)
394 |     
395 |     self.indexes = []
396 |     self.push_index()
397 |   
398 |   def add(self, level, key):
399 |     """
400 |     Add's the key to the given index level
401 |     """
402 |     size = len(key) + self.term_size + self.pointer_size
403 |     stream, pointers, remaining = self.indexes[level]
404 |     
405 |     
406 |     if size > remaining:
407 |       # pad the rest with null bytes
408 |       stream.write(self.terminator * remaining)
409 |       assert stream.tell() % self.block_size == 0
410 |             
411 |       # start the block off with the offset
412 |       stream.write(struct.pack(self.pointer_format, pointers))
413 |       
414 |       next_level = level + 1
415 |       if next_level > len(self.indexes)-1:
416 |         self.push_index()
417 |       self.add(next_level, key)
418 |       remaining = self.block_size - self.pointer_size
419 |     
420 |     pointers += 1  
421 |     stream.write(key)
422 |     stream.write(self.terminator)
423 |     stream.write(struct.pack(self.pointer_format, pointers))
424 | 
425 |     remaining = remaining - size 
426 |     self.indexes[level] = stream, pointers, remaining
427 | 
428 |   def push_index(self):
429 |     stream = SpooledTemporaryFile(max_size = 20*MB)
430 |   
431 |     pointers = 0
432 |     stream.write(struct.pack(OFFSET_FMT,pointers))
433 |   
434 |     self.indexes.append([
435 |       stream, pointers, self.block_size-self.pointer_size
436 |     ])
437 |     
438 |   def finish(self):
439 |     out = self.stream
440 |     blocks_written = 0
441 |     
442 |     out.write(struct.pack(OFFSET_FMT, self.block_size))
443 |     
444 |     # blocks in the index
445 |     out.write(struct.pack(OFFSET_FMT, 0))
446 |         
447 |     
448 |     for stream, pointers, remaining in reversed(self.indexes):
449 | 
450 |       # pad the stream    
451 |       stream.write(self.terminator * remaining)
452 |       level_length = stream.tell()
453 |       
454 |       assert level_length % self.block_size == 0
455 |       
456 |       blocks_to_write = (level_length / self.block_size)
457 |       
458 |       stream.seek(0)
459 | 
460 |       # loop through each pointer and key writing
461 |       for o, key in PBTreeReader.parse(stream, self.block_size):
462 |         out.write(struct.pack(self.pointer_format, o+blocks_written+blocks_to_write))
463 |         # note: the last key in the block returned by the reader will be all null bytes
464 |         # pads
465 |         out.write(key)
466 | 
467 |       blocks_written += blocks_to_write
468 |       stream.close()
469 |     
470 |     out.seek(OFFSET_SIZE)
471 |     out.write(struct.pack(OFFSET_FMT, blocks_written))
472 |     out.seek(0,2) # move to the end of the file  
473 | 
474 |     self.blocks_written = blocks_written
475 |   
476 | 
477 |   def close(self):
478 |     self.finish()
479 | 
480 | class DataBlockReader(object):
481 |   def __init__(self, bytes, value_size, terminator='\0'):
482 |     self.bytes        = bytes
483 |     self.terminator   = terminator
484 |     self.value_size   = value_size
485 |     
486 |     
487 |   def __iter__(self):
488 |     block = self.bytes
489 |     start = 0
490 |     while True:
491 |       pos = block.find(self.terminator, start)
492 |       if pos == -1:
493 |         break
494 |       key = block[start:pos]
495 |       start=pos+1
496 | 
497 |       if key == '':
498 |         return
499 |       else:
500 |         value = block[start:start+self.value_size]
501 |         start+=self.value_size
502 | 
503 |         yield key, value
504 | 
505 |   
506 | 
507 | class IndexBlockReader(object):
508 |   def __init__(self, data):
509 |     self.data = data
510 | 
511 | 
512 |   def __iter__(self):
513 |     end_of_block = False
514 |     buffer = StringIO(self.data)
515 |   
516 |     while not end_of_block:
517 |       offset = self.read_offset(buffer)
518 | 
519 |       key = self.read_key(buffer)
520 |       # near the end of the block, eveything from here on should be null bytes
521 |       if key == '':
522 |         # ended on block boundry
523 |         end_of_block = True
524 |       elif key == '\0':
525 |         # nothing but pad bytes left
526 |         end_of_block = True
527 |         key += self.read_rest_of_block(buffer)
528 | 
529 |       yield offset, key
530 |   
531 |   def find(self, key):
532 |     pointers = []
533 |     prefixes = []
534 |     for pointer, prefix in self:
535 |       pointers.append(pointer)
536 |       prefixes.append(prefix)
537 |       
538 |     # discard padding
539 |     prefixes.pop()
540 |     
541 |     index = bisect.bisect(prefixes, key)
542 |     return pointers[index]
543 |         
544 | 
545 |   def read_offset(self, buffer):
546 | 
547 |     bytes = buffer.read(OFFSET_SIZE)
548 |     # bytes should never be empty when reading an offset if so the file
549 |     # corrupt
550 |     return struct.unpack(OFFSET_FMT, bytes)[0]
551 | 
552 | 
553 |   def read_key(self, buffer):
554 |     buff = bytearray()
555 |     while True:
556 |       c = buffer.read(1)
557 |       if c == '':
558 |         if len(buff):
559 |           raise RuntimeError('EOF found when string was expected')
560 |         else:
561 |           break
562 |       elif c == '\0':
563 |         buff.append(c)
564 |         break
565 |       else:
566 |         buff.append(c)
567 |     return str(buff)
568 |     
569 |   def read_rest_of_block(self, buffer):
570 |     return buffer.read()
571 | 
572 | 
573 | 
574 |     
575 | 
576 | 
577 | 


--------------------------------------------------------------------------------
/lib/prefix.py:
--------------------------------------------------------------------------------
 1 | # Copyright 2012 Triv.io, Scott Robertson
 2 | # 
 3 | #   Licensed under the Apache License, Version 2.0 (the "License");
 4 | #   you may not use this file except in compliance with the License.
 5 | #   You may obtain a copy of the License at
 6 | # 
 7 | #       http://www.apache.org/licenses/LICENSE-2.0
 8 | # 
 9 | #   Unless required by applicable law or agreed to in writing, software
10 | #   distributed under the License is distributed on an "AS IS" BASIS,
11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | #   See the License for the specific language governing permissions and
13 | #   limitations under the License.
14 | # 
15 | 
16 | from  itertools import izip_longest, dropwhile
17 | 
18 | def commonlen(s1,s2):
19 |   """
20 |   Returns the length of the common prefix
21 |   """
22 |       
23 |   # given "hi", "hip"
24 |   # izip_longest("hi", "hip") -> ('h','h'), ('i','i'), (None, 'p')
25 |   # enumerate -> (0,('h','h')), (1,('i','i')), (2,(None, 'p'))
26 |   # dropwhile(lambda (i,(x,y)): x == 5 -> (2,(None,'p')) ...
27 |   
28 |   try:  
29 |     return dropwhile(lambda (i,(x,y)): x == y,enumerate(zip(s1, s2))).next()[0]
30 |   except StopIteration:
31 |     # strings are identical return the len of one of them
32 |     return len(s1)
33 | 
34 | def common(s1,s2):
35 |   """
36 |   Returns the  common prefix
37 |   """
38 |   cl = commonlen(s1,s2)
39 |   return s2[:cl]
40 | 
41 | def signifigant(s1,s2):
42 |   """
43 |   Given two strings s1 and s2, and assuming s2 > s1 returns the character
44 |   that make s2 gerater.
45 |   """  
46 |   cl = commonlen(s1,s2)
47 |   return s2[:cl+1]
48 | 


--------------------------------------------------------------------------------
/lib/test.py:
--------------------------------------------------------------------------------
  1 | # Copyright [2012] [Triv.io, Scott Robertson]
  2 | # 
  3 | #   Licensed under the Apache License, Version 2.0 (the "License");
  4 | #   you may not use this file except in compliance with the License.
  5 | #   You may obtain a copy of the License at
  6 | # 
  7 | #       http://www.apache.org/licenses/LICENSE-2.0
  8 | # 
  9 | #   Unless required by applicable law or agreed to in writing, software
 10 | #   distributed under the License is distributed on an "AS IS" BASIS,
 11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12 | #   See the License for the specific language governing permissions and
 13 | #   limitations under the License.
 14 | # 
 15 | 
 16 | from unittest import TestCase
 17 | from tempfile import NamedTemporaryFile
 18 | from functools import partial
 19 | from nose.tools import eq_
 20 | import mmap
 21 | 
 22 | 
 23 | class TestIndex(TestCase):
 24 |   def test_btree_index(self):
 25 |     from .pbtree import PBTreeWriter, PBTreeReader
 26 |     
 27 |     def data():
 28 |       """Returns an iterator of (url, linepos)"""
 29 |       return ((url.strip(), pos) for pos, url in enumerate(open('sorted_urls')))
 30 | 
 31 |     self.validate(
 32 |       PBTreeWriter,
 33 |       PBTreeReader,
 34 |       data(),
 35 |       prefix = 'http://natebeaty.com/',
 36 |       known_keys = [
 37 |         'http://natebeaty.com/illustration/4452349850',
 38 |         'http://natebeaty.com/illustration/4573016166',
 39 |         'http://natebeaty.com/illustration/4747271212',
 40 |         'http://natebeaty.com/illustration/4752986875',
 41 |       ],
 42 |       known_values =[
 43 |         1891,
 44 |         1892,
 45 |         1893,
 46 |         1894
 47 |       ]
 48 |     )
 49 |     
 50 |   def test_btree_dict_index(self):
 51 |     from .pbtree import PBTreeDictWriter, PBTreeDictReader
 52 |     
 53 |     writer = partial(PBTreeDictWriter, item_keys=("key1", "key2"), value_format="<QI")
 54 |     reader = partial(PBTreeDictReader, item_keys=("key1", "key2"), value_format="<QI")
 55 | 
 56 |     
 57 |     def data():
 58 |       for pos, url in enumerate(open('sorted_urls')):
 59 |         yield url.strip(), {'key1': pos, 'key2': pos}
 60 | 
 61 | 
 62 |     self.validate(
 63 |       writer,
 64 |       reader,
 65 |       data(),
 66 |       prefix = 'http://natebeaty.com/',
 67 |       known_keys = [
 68 |         'http://natebeaty.com/illustration/4452349850',
 69 |         'http://natebeaty.com/illustration/4573016166',
 70 |         'http://natebeaty.com/illustration/4747271212',
 71 |         'http://natebeaty.com/illustration/4752986875',
 72 |       ],
 73 |       known_values =[
 74 |         {'key1': 1891, 'key2': 1891},
 75 |         {'key1': 1892, 'key2': 1892},
 76 |         {'key1': 1893, 'key2': 1893},
 77 |         {'key1': 1894, 'key2': 1894}
 78 |       ]
 79 |     )
 80 |   
 81 |   
 82 |   def validate(self, writer, reader, data, prefix, known_keys, known_values):
 83 |     """
 84 |     Verify  given writer produces content that can be read by the given
 85 |     reader and  returns known keys and values for the given prefix
 86 |     
 87 |     """
 88 |   
 89 |     temp = NamedTemporaryFile(delete=False)
 90 |     index = writer(temp)
 91 | 
 92 |     keys_written = []
 93 |     for key, value in data:
 94 |       index.add(key, value)
 95 |       keys_written.append(key)
 96 |     index.close()
 97 | 
 98 |     ii = open(temp.name, 'r+')
 99 |     map = mmap.mmap(ii.fileno(),0)
100 |     index = reader(map)
101 |     keys = index.keys(prefix)
102 | 
103 |     self.assertListEqual(keys,known_keys)
104 |     
105 |     values = index.values(prefix)
106 |     self.assertListEqual(values,known_values)
107 |     
108 |     items = index.items(prefix)
109 | 
110 |     self.assertListEqual(items, zip(known_keys, known_values))
111 | 
112 |     self.assertListEqual(index.keys(), keys_written)
113 |   


--------------------------------------------------------------------------------
/lib/test_map.py:
--------------------------------------------------------------------------------
  1 | # <data segment> :: = <item> (<item> | ( {<null>} <null>))
  2 | # <item> ::= <key> <null> <location pointer>
  3 | 
  4 | # 
  5 | #
  6 | 
  7 | from unittest import TestCase
  8 | from .prefix import signifigant
  9 | 
 10 | 
 11 | class TestIndexMapCase(TestCase):
 12 |   def test(self):
 13 |     class P():
 14 |       pass
 15 |     params = P()
 16 |   
 17 |     results =[]
 18 |     for partition_number, input in enumerate(inputs):
 19 |       params.last_key = None
 20 |       params.partition_number = partition_number
 21 |       for block in input:
 22 |         for item in map_block(iter(block), params):
 23 |           results.append(item)
 24 | 
 25 |     self.assertSequenceEqual(results, final)  
 26 | 
 27 | 
 28 | 
 29 | file1 = [
 30 |   [
 31 |     'key01',
 32 |     'key02',
 33 |     'key03a',
 34 |     'key03ac',
 35 |   ],
 36 |   [
 37 |     'key03bc'
 38 |     'key06',
 39 |     'key07',
 40 |     'key08z',
 41 |   ],
 42 |   [
 43 |     'key08zafz'
 44 |     'key10',
 45 |     'key11',
 46 |     'key12',
 47 |   ],
 48 | ]
 49 | 
 50 | file2= [
 51 |   [
 52 |     'key13feee',
 53 |     'key14',
 54 |     'key16',
 55 |     'key16a',
 56 |   ],
 57 |   [
 58 |     'key16b'
 59 |     'key18',
 60 |     'key19',
 61 |     'key20',
 62 |   ]
 63 | ]
 64 | 
 65 | 
 66 | final=(
 67 |   (0,"key01"),
 68 |   (0,"key03b"),
 69 |   (0,"key08za"),
 70 |   (1,"key13feee"),
 71 |   (1,"key16b")
 72 | )
 73 | 
 74 | inputs = [file1, file2]
 75 | 
 76 | def map_block(block, params):
 77 |   # yield first item and last
 78 | 
 79 |   first_key = block.next()
 80 |   assert first_key.find('\0') == -1
 81 |   
 82 |   if params.last_key is None:
 83 |     yield params.partition_number, first_key
 84 |   else:
 85 |     yield params.partition_number, signifigant(params.last_key, first_key)
 86 | 
 87 |   second_to_last = None
 88 |   for key in block:
 89 |     if not key.startswith('\0'):
 90 |       second_to_last = key
 91 | 
 92 |   if second_to_last is not None:
 93 |     params.last_key = second_to_last
 94 |   else:
 95 |     params.last_key = first_key
 96 |   
 97 |   
 98 |   
 99 |   
100 |   


--------------------------------------------------------------------------------
/lib/test_pbtree.py:
--------------------------------------------------------------------------------
 1 | # Copyright 2012 Triv.io, Scott Robertson]
 2 | # 
 3 | #   Licensed under the Apache License, Version 2.0 (the "License");
 4 | #   you may not use this file except in compliance with the License.
 5 | #   You may obtain a copy of the License at
 6 | # 
 7 | #       http://www.apache.org/licenses/LICENSE-2.0
 8 | # 
 9 | #   Unless required by applicable law or agreed to in writing, software
10 | #   distributed under the License is distributed on an "AS IS" BASIS,
11 | #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | #   See the License for the specific language governing permissions and
13 | #   limitations under the License.
14 | # 
15 | 
16 | from unittest import TestCase
17 | 
18 | from nose.tools import eq_
19 | 
20 | from .pbtree import PBTreeWriter, PBTreeReader, IndexWriter
21 | from tempfile import TemporaryFile
22 | 
23 | class TestPBTree(TestCase):
24 |   def test_btree_index(self):
25 |     t = TemporaryFile()
26 |     pbtree = PBTreeWriter(t)
27 |     pbtree.add("blah", 1)
28 |     #pbtree.commit()
29 |     
30 |     #t.seek(0)
31 |     packet = pbtree.data_segment.write_buffer #t.read()
32 |     
33 |     eq_(packet, 'blah\x00\x01\x00\x00\x00\x00\x00\x00\x00')
34 |     
35 |   def test_one_key_per_block_writer(self):
36 |     # 2 pointers and a 1 byte string null terminated string = 10 bytes
37 |     stream = TemporaryFile()
38 |     
39 |     i = IndexWriter(stream, block_size=10, terminator='\0')
40 |     i.add(0, 'b')
41 |     eq_(len(i.indexes), 1)
42 |     
43 |     i.add(0, 'c')
44 |     eq_(len(i.indexes), 2)
45 |     i.finish()
46 | 
47 | 
48 |     stream.seek(0)
49 |     packet = stream.read()
50 |     eq_(len(packet), 30)
51 |     
52 | 
53 |     root_block = packet[:10]
54 |     eq_(root_block, '\x01\x00\x00\x00c\x00\x02\x00\x00\x00')
55 |     
56 |     block_1 = packet[10:20]
57 |     eq_(block_1, '\x03\x00\x00\x00b\x00\x04\x00\x00\x00')
58 |     
59 |     block_2 = packet[20:]
60 |     eq_(block_2, '\x04\x00\x00\x00c\x00\x05\x00\x00\x00')


--------------------------------------------------------------------------------