├── LICENSE.txt ├── README.md ├── bextest.py ├── bextree.py ├── hasbeen.py ├── mapper.py └── spatialindex.py /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (C) 2013 Marc Pfister 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a 4 | copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included 12 | in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 15 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ### FYI this is reaaallly old 2 | 3 | 4 | 5 | HASBEEN 6 | ========== 7 | 8 | Tools to create ESRI's .sbn spatial indexing format 9 | 10 | Background 11 | ------------ 12 | 13 | In October 2011 Joel Lawhead figured out [how to decode ESRI's SBN spatial indexing format](http://geospatialpython.com/2011/10/your-chance-to-make-gis-history.html). He didn't recognize the indexing algorithm, so in the best Open Source spirit he released his code and some sample data with the challenge to figure out what was going on. 14 | 15 | I took his data and started trying to visualize it differently, looking for patterns that might give away the algorithm. First I noticed [Z-shaped curves](http://flic.kr/p/atrEim), which suggest quadtrees, or a z-order curve. I then noticed that the many of features were actually on seams between possible divisions of the space, and by tracking them it became clear that the spatial index was a binary division. The index space (a 256 x 256 grid) initally split in half horizontally. Then each rectangular half would split vertical into two squares, and so on. 16 | 17 | To further test how the algorithm worked, I created a series of incremental shapefiles that would add one feature to the previous one. I then wrote some scripts that would track how the index tree grew and also tracked features as they were sorted in the tree. Tracking through thousands of shapefiles, I noticed is that no matter the shapefiles, the size of the tree [jumped at fixed feature counts](http://flic.kr/p/cnSSTC). Looking at the numbers, it became clear that the size of tree was designed to keep an average of 8 features or less per node. By tracking individual nodes, I also noticed that a node would fill up until it had 8 features in it, and after that it would send the feature down to its child nodes. 18 | 19 | With thousands of test shapefiles to work with, I also tested the algorithm that maps the features to index space and found that it does some interesting rounding. At this point it seems like we had the algorithm figured out. But in some of the test shapefiles some features would be sorted one level lower than the ESRI algorithm places them. After checking the individual features, we couldn't find any reason why the features did not sort all the way to the bin that should contain them. And there didn't seem to be any obvious explanation why. 20 | 21 | After looking at many shapefiles with errors, something popped out. If you look at a bin, you'd usually see something like this: 22 | 23 | BIN 12 : 5, 67, 122, 156 24 | 25 | Where the numbers are the Feature ID's of the features in that bin. They are almost always in ascending order. But in the cases where features were sorting too low, I'd see: 26 | 27 | BIN 15: 13, 23, 77, 145, 20, 92 28 | 29 | Where features 20 and 92 were two features that should not be in this node, but actually in its child nodes. Being out of order, it seemed like they had been sorted down, then pulled back up. Looking at a lot of shapefiles with the pattern revealed another level to the algorithm: after sorting all the features into the tree, another process would check nodes to see if their grandchildren were empty - if they were, then it would pull its child features back up. Once this was implemented in the code I was able to correctly reproduce the ESRI SBN sorting in my test shapefiles. 30 | 31 | There was only one problem - the test shapefiles were randomly generated and spatially very uniformly distributed. Testing against real world data, like the world cities shapefile, still showed some features that were sorted further into the tree than the ESRI algorithm. I have some ideas why, but haven't cracked this final step. 32 | 33 | About The Code 34 | -------------- 35 | 36 | hasbeen.py gives you the spatial index tree and a way to add features to it, and get the whole tree as a flat array. It does not write the SBN files. SBN read/write is provided by Joel Lawhead's spatialindex.py, of which I include an older version for reading SBNs for testing. Joel is integrating hasbeen into spatialindex. 37 | 38 | Also 39 | ---------- 40 | 41 | Since the division is binary, it looks like you can do the sorting quickly with a little [bit twiddling](https://gist.github.com/drwelby/5977601). 42 | -------------------------------------------------------------------------------- /bextest.py: -------------------------------------------------------------------------------- 1 | # Bextest.py tests binning algoritms against an existing .sbn file 2 | # 3 | import shapefile 4 | from hasbeen import HasBeen 5 | from bextree import Tree 6 | from mapper import mapshapefile 7 | from spatialindex import SBN 8 | import sys 9 | 10 | if len(sys.argv) > 1: 11 | shpfile = sys.argv[1] 12 | else: 13 | print "Usage: bextest.py [shapefile]" 14 | print "Bins the features in a shapefile and compares it to an existing sbn" 15 | sys.exit() 16 | 17 | 18 | h = HasBeen(shpfile) 19 | 20 | # open up the existing sbn to compare against 21 | sbn = SBN(shpfile + ".sbn") 22 | 23 | # compare the generated sbn versus ESRI's 24 | for id, bin in enumerate(sbn.bins): 25 | if id ==0 : continue 26 | a = len(bin.features) == len(h.bins[id].features) 27 | if not a: 28 | print "Bin %s -" % id, 29 | print "E: %s, L:%s" % (len(bin.features), len(h.bins[id].features)) 30 | if len(bin.features) > 0: 31 | tb = bin.features 32 | else: 33 | tb = h.bins[id].features 34 | xmin = 255 35 | ymin = 255 36 | xmax = 0 37 | ymax = 0 38 | for f in tb: 39 | print "%s," % f.id, 40 | xmin = min(xmin,f.xmin) 41 | ymin = min(ymin,f.ymin) 42 | xmax = max(xmax,f.xmax) 43 | ymax = max(ymax,f.ymax) 44 | print "\n f_bbox %s-%s,%s-%s" % (xmin,xmax,ymin,ymax) 45 | node = t.nodes[id] 46 | print " node %s-%s,%s-%s/%s\n" % (node.xmin,node.xmax,node.ymin,node.ymax,node.splitcoord) 47 | 48 | 49 | 50 | -------------------------------------------------------------------------------- /bextree.py: -------------------------------------------------------------------------------- 1 | # Binary EXpanding TREE 2 | # (for those not familiar with ESRI trivia) 3 | # The spatial index tree behind the SBN file format 4 | 5 | 6 | from spatialindex import Bin, Feature 7 | from math import log, ceil 8 | 9 | class Node: 10 | id = None 11 | tree = None 12 | split = None # "x" or "y" based on axis of cutting coord 13 | righttop = None # child node, either right or top 14 | parent = None # parent node 15 | leftbottom = None # child node, either left or bottom 16 | sibling = None # sibling node 17 | splitcoord = 0 18 | features = [] 19 | holdfeatures = [] 20 | full = False 21 | 22 | def __init__(self, id): 23 | self.id = id 24 | self.count = 0 25 | self.features = [] 26 | self.holdfeatures = [] 27 | self.full = False 28 | self.sibling = None 29 | 30 | def __repr__(self): 31 | return 'Node %s: (%s-%s,%s-%s)/%s' % (self.id, self.xmin, self.xmax, self.ymin, self.ymax, self.splitcoord) 32 | 33 | def addsplitcoord(self): 34 | if self.split == "x": 35 | mid = int((self.xmin + self.xmax) / 2.0) + 1 36 | else: 37 | mid = int((self.ymin + self.ymax) / 2.0) + 1 38 | self.splitcoord = mid - mid%2 39 | 40 | 41 | def addchildren(self): 42 | #first child node 43 | rt = self.tree.nodes[self.id*2] 44 | rt.tree = self.tree 45 | if self.split == "x": 46 | rt.xmin = int(self.splitcoord) + 1 47 | rt.xmax = self.xmax 48 | rt.ymin = self.ymin 49 | rt.ymax = self.ymax 50 | rt.split = "y" 51 | rt.addsplitcoord() 52 | else: 53 | rt.xmin = self.xmin 54 | rt.xmax = self.xmax 55 | rt.ymin = int(self.splitcoord) + 1 56 | rt.ymax = self.ymax 57 | rt.split = "x" 58 | rt.addsplitcoord() 59 | rt.parent = self 60 | self.righttop = rt 61 | #second child node 62 | lb = self.tree.nodes[self.id * 2 + 1] 63 | lb.tree = self.tree 64 | if self.split == "x": 65 | lb.xmax = int(self.splitcoord) 66 | lb.xmin = self.xmin 67 | lb.ymin = self.ymin 68 | lb.ymax = self.ymax 69 | lb.split = "y" 70 | lb.addsplitcoord() 71 | else: 72 | lb.xmin = self.xmin 73 | lb.xmax = self.xmax 74 | lb.ymax = int(self.splitcoord) 75 | lb.ymin = self.ymin 76 | lb.split = "x" 77 | lb.addsplitcoord() 78 | lb.parent = self 79 | self.leftbottom = lb 80 | lb.sibling = rt 81 | rt.sibling = lb 82 | 83 | def grow(self): 84 | #recursively grow the tree 85 | if self.id >= self.tree.firstleafid: 86 | return 87 | self.addchildren() 88 | self.righttop.grow() 89 | self.leftbottom.grow() 90 | 91 | def insert(self,feature): 92 | # if this is leaf, just take the feature 93 | if self.id >= self.tree.firstleafid: 94 | self.features.append(feature) 95 | return 96 | # it takes 8 features to split a node 97 | # so we'll hold 8 features first 98 | if self.id > 1: 99 | if not self.full: 100 | if len(self.holdfeatures) < 8 : 101 | self.holdfeatures.append(feature) 102 | return 103 | if len(self.holdfeatures) == 8 : 104 | self.full = True 105 | self.holdfeatures.append(feature) 106 | for f in self.holdfeatures: 107 | self.insert(f) 108 | return 109 | # The node is split so we can sort features 110 | if self.split == "x": 111 | (min,max) = (feature.xmin, feature.xmax) 112 | (smin,smax) = (feature.ymin, feature.ymax) 113 | else: 114 | (min,max) = (feature.ymin, feature.ymax) 115 | (smin,smax) = (feature.xmin, feature.xmax) 116 | # Grab features on the seam we can't split 117 | if min <= self.splitcoord and max > self.splitcoord: 118 | self.features.append(feature) 119 | return 120 | else: 121 | self.passfeature(feature) 122 | 123 | def passfeature(self, feature): 124 | # pass the feature to a child node 125 | if self.split == "x": 126 | (min,max) = (feature.xmin, feature.xmax) 127 | else: 128 | (min,max) = (feature.ymin, feature.ymax) 129 | if min < self.splitcoord: 130 | self.leftbottom.insert(feature) 131 | else: 132 | self.righttop.insert(feature) 133 | 134 | def allfeatures(self): 135 | # return all the features in the node 136 | if self.id >= self.tree.firstleafid: 137 | return self.features 138 | if self.id == 1: 139 | return self.features 140 | if len(self.holdfeatures) <= 8: 141 | return self.holdfeatures 142 | else: 143 | return self.features 144 | 145 | def siblingfeaturecount(self): 146 | # return the number of features of a node and its sibling 147 | return len(self.allfeatures()) + len(self.sibling.allfeatures()) 148 | 149 | class Tree: 150 | 151 | nodes = [] 152 | levels = 0 153 | firstleafid = 0 154 | root = None 155 | 156 | def __init__(self, featurecount): 157 | self.nodes = [] 158 | self.levels = int(log(((featurecount -1 )/8.0 + 1),2) + 1) 159 | if self.levels < 2 : 160 | self.levels = 2 161 | if self.levels > 15: 162 | self.levels = 15 163 | self.firstleafid = 2**(self.levels-1) 164 | for i in xrange(2**self.levels): 165 | n = Node(i) 166 | self.nodes.append(n) 167 | 168 | self.root = self.nodes[1] 169 | self.root.id = 1 170 | self.root.tree = self 171 | self.root.split = "x" 172 | self.root.xmin = 0 173 | self.root.ymin = 0 174 | self.root.xmax = 255 175 | self.root.ymax = 255 176 | self.root.addsplitcoord() 177 | self.root.grow() 178 | 179 | def insert(self,feature): 180 | #insert a feature into the tree 181 | self.root.insert(feature) 182 | 183 | def tobins(self): 184 | #convert a tree structure to a SpatialIndex bin array 185 | bins = [] 186 | b = Bin() 187 | bins.append(b) 188 | for node in self.nodes[1:]: 189 | b = Bin() 190 | b.features = node.allfeatures() 191 | bins.append(b) 192 | return bins 193 | 194 | def featuresinlevel(self, level): 195 | # return the number of features in a level 196 | start = int(2**(level-1)) 197 | end = int((2 * start) - 1) 198 | featurecount = 0 199 | for node in self.nodes[start:end+1]: 200 | featurecount += len(node.features) 201 | return featurecount 202 | 203 | 204 | def compactseamfeatures(self): 205 | # the mystery algorithm - compaction? optimization? obfuscation? 206 | if self.levels < 4: 207 | return 208 | if self.levels > 4: 209 | start = self.firstleafid/2 - 1 210 | end = start/8 211 | if start < 3: start = 3 212 | if end < 1 : end = 1 213 | for node in self.nodes[start:end:-1]: 214 | #if len(node.features) > 0: 215 | # continue 216 | id = node.id 217 | children = self.nodes[id*2:id*2+2] 218 | for child in children: 219 | cid = child.id 220 | grandchildren = self.nodes[cid*2:cid*2 + 2] 221 | gccount = 0 222 | for gcnode in grandchildren: 223 | gccount += len(gcnode.allfeatures()) 224 | #print "Node %s has %s GC" % (id,gccount) 225 | if gccount == 0: 226 | #print "Slurping %s features from node %s" % (len(child.allfeatures()),child.id) 227 | #node.features.extend(child.features) 228 | if len(child.allfeatures()) < 4: # this is weird but it works 229 | node.features.extend(child.allfeatures()) 230 | child.features = [] 231 | child.holdfeatures = [] 232 | 233 | def compactseamfeatures2(self): 234 | # another run at the mystery algorithm 235 | 236 | #for node in self.nodes[1:self.firstleafid/4]: 237 | start = self.firstleafid/2 - 1 238 | end = start/8 239 | if start < 3: start = 3 240 | if end < 1: end =1 241 | for node in self.nodes[start:end:-1]: 242 | #if len(node.features) > 0 and self.levels < 6: 243 | # continue 244 | id = node.id 245 | children = self.nodes[id*2:id*2+2] 246 | grandchildren = self.nodes[id*4:id*4+4] 247 | gccount = 0 248 | for gcnode in grandchildren: 249 | gccount += len(gcnode.allfeatures()) 250 | #print "Node %s has %s GC" % (id,gccount) 251 | if gccount == 0: 252 | for cnode in children: 253 | if len(cnode.allfeatures()) + len(node.features) > 8: 254 | continue 255 | #print "Slurping %s features from node %s" % (len(cnode.features),cnode.id) 256 | node.features.extend(cnode.allfeatures()) 257 | #node.features.extend(cnode.features) 258 | cnode.features = [] 259 | cnode.holdfeatures = [] 260 | # compact unsplit nodes see cities/248 261 | return 262 | for node in self.nodes[start:end:-1]: 263 | level = ceil(log(node.id,2)) 264 | id = node.id 265 | children = self.nodes[id*2+1:id*2-1:-1] 266 | empty = False 267 | childrenfeatures = 0 268 | for child in children: 269 | #if not child.full: 270 | # held = True 271 | cid = child.id 272 | childrenfeatures += len(child.features) 273 | grandchildren = self.nodes[cid*2:cid*2 + 2] 274 | for gcnode in grandchildren: 275 | if len(gcnode.features) == 0: 276 | empty = True 277 | #print "Node %s childless: %s" % (cid,empty) 278 | print empty, childrenfeatures 279 | if empty and childrenfeatures > 0: 280 | #node.features.extend(child.features) 281 | for child in children: 282 | if child.siblingfeaturecount() < 4 and child.siblingfeaturecount() > 0 : 283 | continue 284 | #if self.featuresinlevel(level) >= 8: 285 | # return 286 | #print "Slurping %s features from node %s" % (len(child.allfeatures()),child.id) 287 | node.features.extend(child.allfeatures()) 288 | #node.full = True 289 | child.features = [] 290 | child.holdfeatures = [] 291 | return 292 | 293 | 294 | 295 | if __name__ == "__main__": 296 | 297 | t = Tree(4) 298 | 299 | for i in range(8): 300 | f = Feature() 301 | f.xmin = 120 302 | f.xmax = 130 303 | f.ymin = 120 304 | f.ymax = 130 305 | 306 | t.insert(f) 307 | 308 | f = Feature() 309 | f.xmin = 130 310 | f.xmax = 140 311 | f.ymin = 130 312 | f.ymax = 140 313 | 314 | t.insert(f) 315 | 316 | for node in t.nodes[1:]: 317 | hfl = len(node.holdfeatures) 318 | nfl = len(node.nodefeatures) 319 | print "%s: %s %s" % (node.id, nfl, hfl) 320 | 321 | -------------------------------------------------------------------------------- /hasbeen.py: -------------------------------------------------------------------------------- 1 | import shapefile 2 | from bextree import Tree 3 | from mapper import mapshapefile 4 | 5 | class HasBeen: 6 | 7 | def __init__(self,shpfile): 8 | 9 | # open the shapefile 10 | self.shp = shapefile.Reader(shpfile) 11 | # map the shapfile features to index space 12 | self.map = mapshapefile(self.shp) 13 | # build an empty tree of the right size 14 | self.tree = Tree(len(self.shp.shapes())) 15 | # insert the mapped features into the tree 16 | for feature in self.map: 17 | self.tree.insert(feature) 18 | # do the magic compaction algorithm 19 | self.tree.compactseamfeatures() 20 | #convert the tree structure to a flat array 21 | self.bins = self.tree.tobins() 22 | -------------------------------------------------------------------------------- /mapper.py: -------------------------------------------------------------------------------- 1 | import shapefile 2 | from spatialindex import SBN, Bin, Feature 3 | from math import floor, ceil 4 | 5 | def mapshapefile(sf): 6 | # map features in a shapefile to index space 7 | shapes = sf.shapes() 8 | features = [] 9 | for index, shape in enumerate(shapes): 10 | ft = mapfeature(index,shape,sf) 11 | features.append(ft) 12 | return features 13 | 14 | 15 | def mapfeature(index,shape,sf): 16 | # map individual features, returns Features 17 | sf_xrange = sf.bbox[2]-sf.bbox[0] 18 | sf_yrange = sf.bbox[3]-sf.bbox[1] 19 | 20 | ft = Feature() 21 | ft.id = index + 1 22 | if sf.shapeType == 1: 23 | (ptx,pty) = shape.points[0] 24 | sh_bbox = (ptx, pty, ptx, pty) 25 | else: 26 | sh_bbox = shape.bbox 27 | 28 | ft_xmin = ((sh_bbox[0]-sf.bbox[0])/sf_xrange*255.0) 29 | # not sure why this rounding is needed, but it is 30 | mod_xmin = (ft_xmin%1 - .005)%1 + int(ft_xmin) 31 | ft.xmin = int(floor(mod_xmin)) 32 | if ft.xmin < 0 : ft.xmin = 0 33 | 34 | ft_ymin = ((sh_bbox[1]-sf.bbox[1])/sf_yrange*255.0) 35 | mod_ymin = (ft_ymin%1 - .005)%1 + int(ft_ymin) 36 | ft.ymin = int(floor(mod_ymin)) 37 | if ft.ymin < 0 : ft.ymin = 0 38 | 39 | ft_xmax = ((sh_bbox[2]-sf.bbox[0])/sf_xrange*255.0) 40 | mod_xmax = (ft_xmax%1 + .005)%1 + int(ft_xmax) 41 | ft.xmax = int(ceil(mod_xmax)) 42 | if ft.xmax > 255: ft.xmax = 255 43 | 44 | ft_ymax = ((sh_bbox[3]-sf.bbox[1])/sf_yrange*255.0) 45 | mod_ymax = (ft_ymax%1 + .005)%1 + int(ft_ymax) 46 | ft.ymax = int(ceil(mod_ymax)) 47 | if ft.ymax > 255: ft.ymax = 255 48 | 49 | return ft 50 | 51 | if __name__ == "__main__": 52 | file = "../cities/132" 53 | s = shapefile.Reader(file) 54 | sf = mapshapefile(s) 55 | sbn = SBN(file + ".sbn") 56 | for id, bin in enumerate(sbn.bins): 57 | for f in bin.features: 58 | m = sf[f.id-1] 59 | if (f.xmin,f.xmax,f.ymin,f.ymax) != (m.xmin,m.xmax,m.ymin,m.ymax): 60 | print "Bin %s (%s)" % (id, bin.id) 61 | print " SBN Feature %s (%s,%s,%s,%s)" % (f.id, f.xmin, f.ymin, f.xmax, f.ymax) 62 | print " Mapper Feature %s (%s,%s,%s,%s)\n" %(m.id, m.xmin, m.ymin, m.xmax, m.ymax) 63 | -------------------------------------------------------------------------------- /spatialindex.py: -------------------------------------------------------------------------------- 1 | import struct 2 | import os 3 | 4 | class Bin: 5 | def __init__(self): 6 | self.id = 1 7 | self.features = [] 8 | 9 | class Feature: 10 | def __init__(self): 11 | self.id = None 12 | self.xmin = None 13 | self.ymin = None 14 | self.xmax = None 15 | self.ymax = None 16 | 17 | class SBN: 18 | def __init__(self, sbn=None): 19 | self.bins = [] 20 | self.numFeat = 0 21 | self.sbnName = sbn 22 | self.load(self.sbnName) 23 | 24 | def load(self, sbnName=None): 25 | self.sbn = None 26 | self.fileCode = 9994 27 | self.unknownByte4 = -400 28 | self.unused1 = None 29 | self.unused2 = None 30 | self.unused3 = None 31 | self.unused4 = None 32 | self.fileLen = 0 33 | self.numRecords = 0 34 | self.xmin = None 35 | self.ymin = None 36 | self.xmax = None 37 | self.ymax = None 38 | self.zmin = None 39 | self.zmax = None 40 | self.zmax = None 41 | self.mmin = None 42 | self.mmax = None 43 | self.unknownByte96 = 0 44 | if sbnName: 45 | try: 46 | self.sbn = open(sbnName, "rb") 47 | except IOError: 48 | raise Exception("Unable to open sbn file or sbn file not specified") 49 | # Read sbn header 50 | self.fileCode = struct.unpack(">i", self.sbn.read(4))[0] 51 | self.unknownByte4 = struct.unpack(">i", self.sbn.read(4))[0] 52 | self.unused1 = struct.unpack(">i", self.sbn.read(4))[0] 53 | self.unused2 = struct.unpack(">i", self.sbn.read(4))[0] 54 | self.unused3 = struct.unpack(">i", self.sbn.read(4))[0] 55 | self.unused4 = struct.unpack(">i", self.sbn.read(4))[0] 56 | self.fileLen = struct.unpack(">i", self.sbn.read(4))[0] * 2 57 | # NOTE: numRecords is records in shp/dbf file not # of bins! 58 | self.numRecords = struct.unpack(">i", self.sbn.read(4))[0] 59 | self.xmin = struct.unpack(">d", self.sbn.read(8))[0] 60 | self.ymin = struct.unpack(">d", self.sbn.read(8))[0] 61 | self.xmax = struct.unpack(">d", self.sbn.read(8))[0] 62 | self.ymax = struct.unpack(">d", self.sbn.read(8))[0] 63 | self.zmin = struct.unpack(">d", self.sbn.read(8))[0] 64 | self.zmax = struct.unpack(">d", self.sbn.read(8))[0] 65 | self.mmin = struct.unpack(">d", self.sbn.read(8))[0] 66 | self.mmax = struct.unpack(">d", self.sbn.read(8))[0] 67 | self.unknownByte96 = struct.unpack(">i", self.sbn.read(4))[0] 68 | # Read BIN descriptors 69 | recNum = struct.unpack(">i", self.sbn.read(4))[0] 70 | recLen = struct.unpack(">i", self.sbn.read(4))[0] * 2 71 | # Append a blank convienience bin as a placeholder for bin 1 72 | b = Bin() 73 | b.id = 1 74 | self.bins.append(b) 75 | for i in range(recLen/8): 76 | b = Bin() 77 | b.id = struct.unpack(">i", self.sbn.read(4))[0] 78 | b.numFeat = struct.unpack(">i", self.sbn.read(4))[0] 79 | self.bins.append(b) 80 | # Read Bin contents 81 | while self.sbn.tell() < self.fileLen: 82 | binId = struct.unpack(">i", self.sbn.read(4))[0] 83 | recLen = struct.unpack(">i", self.sbn.read(4))[0] * 2 84 | 85 | for i in range(recLen/8): 86 | f = Feature() 87 | f.xmin = struct.unpack(">B",self.sbn.read(1))[0] 88 | f.ymin = struct.unpack(">B",self.sbn.read(1))[0] 89 | f.xmax = struct.unpack(">B",self.sbn.read(1))[0] 90 | f.ymax = struct.unpack(">B",self.sbn.read(1))[0] 91 | f.id = struct.unpack(">i", self.sbn.read(4))[0] 92 | b = self.bin(binId) 93 | b.features.append(f) 94 | self.numFeat += 1 95 | self.sbn.close() 96 | 97 | def bin(self, bid): 98 | for b in self.bins: 99 | if b.id == bid: 100 | return b 101 | 102 | def save(self, sbnName=None): 103 | sbnName = sbnName or self.sbnName 104 | sbn = open(sbnName, "wb") 105 | sbxName = os.path.splitext(sbnName)[0] + ".sbx" 106 | sbx = open(sbxName, "wb") 107 | # Write headers 108 | sbn.write(struct.pack(">i", self.fileCode)) 109 | sbx.write(struct.pack(">i", self.fileCode)) 110 | sbn.write(struct.pack(">i", self.unknownByte4)) 111 | sbx.write(struct.pack(">i", self.unknownByte4)) 112 | sbn.write(struct.pack(">i", self.unused1)) 113 | sbx.write(struct.pack(">i", self.unused1)) 114 | sbn.write(struct.pack(">i", self.unused2)) 115 | sbx.write(struct.pack(">i", self.unused2)) 116 | sbn.write(struct.pack(">i", self.unused3)) 117 | sbx.write(struct.pack(">i", self.unused3)) 118 | sbn.write(struct.pack(">i", self.unused4)) 119 | sbx.write(struct.pack(">i", self.unused4)) 120 | 121 | # Calculate File Length fields 122 | # first bin descriptors 123 | totalBinSize = len(self.bins[1:]) * 8 124 | # then bins with features 125 | usedBinSize = len([b for b in self.bins if b.id > 0]) * 8 126 | sbxSize = 100 + usedBinSize 127 | sbnSize = 100 + totalBinSize + usedBinSize 128 | sbxSize //= 2 129 | sbnSize += self.numFeat * 8 130 | sbnSize //= 2 131 | sbn.write(struct.pack(">i", sbnSize)) 132 | sbx.write(struct.pack(">i", sbxSize)) 133 | sbn.write(struct.pack(">i", self.numRecords)) 134 | sbx.write(struct.pack(">i", self.numRecords)) 135 | sbn.write(struct.pack(">d", self.xmin)) 136 | sbx.write(struct.pack(">d", self.xmin)) 137 | sbn.write(struct.pack(">d", self.ymin)) 138 | sbx.write(struct.pack(">d", self.ymin)) 139 | sbn.write(struct.pack(">d", self.xmax)) 140 | sbx.write(struct.pack(">d", self.xmax)) 141 | sbn.write(struct.pack(">d", self.ymax)) 142 | sbx.write(struct.pack(">d", self.ymax)) 143 | sbn.write(struct.pack(">d", self.zmin)) 144 | sbx.write(struct.pack(">d", self.zmin)) 145 | sbn.write(struct.pack(">d", self.zmax)) 146 | sbx.write(struct.pack(">d", self.zmax)) 147 | sbn.write(struct.pack(">d", self.mmin)) 148 | sbx.write(struct.pack(">d", self.mmin)) 149 | sbn.write(struct.pack(">d", self.mmax)) 150 | sbx.write(struct.pack(">d", self.mmax)) 151 | sbn.write(struct.pack(">i", self.unknownByte96)) 152 | sbx.write(struct.pack(">i", self.unknownByte96)) 153 | # sbn and sbx records 154 | # first create bin descriptors record 155 | recLen = (len(self.bins[1:]) * 4) 156 | sbn.write(struct.pack(">i", 1)) 157 | sbn.write(struct.pack(">i", recLen)) 158 | sbx.write(struct.pack(">i", 100)) 159 | sbx.write(struct.pack(">i", recLen + 2)) 160 | for b in self.bins[1:]: 161 | sbn.write(struct.pack(">i", b.id)) 162 | sbn.write(struct.pack(">i", len(b.features))) 163 | # write actual bins 164 | for b in self.bins[1:]: 165 | if b.id <= 0: 166 | continue 167 | sbx.write(struct.pack(">i", sbn.tell())) 168 | sbx.write(struct.pack(">i", (len(b.features) * 2))) 169 | sbn.write(struct.pack(">i", b.id)) 170 | sbn.write(struct.pack(">i", len(b.features) * 4)) 171 | for f in b.features: 172 | sbn.write(struct.pack(">B", f.xmin)) 173 | sbn.write(struct.pack(">B", f.ymin)) 174 | sbn.write(struct.pack(">B", f.xmax)) 175 | sbn.write(struct.pack(">B", f.ymax)) 176 | sbn.write(struct.pack(">i", f.id)) 177 | sbn.close() 178 | sbx.close() 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | --------------------------------------------------------------------------------