├── .gitignore ├── LICENSE ├── README.md ├── cb └── __init__.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | 29 | # Translations 30 | *.mo 31 | 32 | # Mr Developer 33 | .mr.developer.cfg 34 | .project 35 | .pydevproject 36 | 37 | # Pycharm project 38 | .idea 39 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License - Complicated Build (cb) 2 | 3 | Copyright (c) 2013 Joe Jordan 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | 7 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | complicated_build 2 | ================= 3 | 4 | (released under the MIT license, see the LICENSE file.) 5 | 6 | a python extension build system (with Cython support) which allows automated compilation a mixture of C, C++, FORTRAN and Cython sources without F2PY interference. 7 | 8 | *NEW IN v1.4:* support for specifying library locations (only globally.) 9 | 10 | *NEW IN v1.3:* support for linking against static libs on a per-extension basis. see new 'link_to' key on extension dicts. 11 | 12 | inspired by the problem in [this stackoverflow question](http://stackoverflow.com/questions/12696520/cython-and-fortran-how-to-compile-together-without-f2py). 13 | 14 | Note, this library also provides a significant improvement on the default python build system for native extensions with many source files: by default it *caches all temporary build objects*, and only recompiles the particular source files that have changed. The distutils default (designed for single file extensions, no doubt) recompiles *all* sources if *any* have changed. When debugging extensions, e.g. making small changes to one or two source files in a long list, this can present a significant time saving in each build/run cycle. 15 | 16 | *SUBNOTE:* There is also a minor 'bug' in that, since you don't specify header files like in a `Makefile`, it doesn't detect changes in header files. If this is annoying enough please post a bug, and I'll work out how to fix it without breaking the existing interface (which will be a minor pain). 17 | 18 | This module uses the *default python flags* for building sources, which includes all kinds of cruft that were generated in the makefile that compiled python itself. Also, the `arch` argument is no longer passed to the compiler, as this is not compatible with some compilers, and is now only used in temp directory names. (if such a flag is required, it will be present in the `distutils.sysconfig` vars without further intervention.) 19 | 20 | note, finally, that this has been modified to act as a decorator for `distutils.core.setup`, see the examples of the new invocation style below. 21 | 22 | example usage in a setup.py: 23 | 24 | ```python 25 | import cb 26 | 27 | global_includes = ['/opt/local/include'] 28 | global_lib_dirs = ['/opt/local/lib'] 29 | global_macros = [("__SOME_MACRO__",), ("__MACRO_WITH_VALUE__", "5")] 30 | 31 | extensions = [{'name' : 'mylib.impl', 32 | 'sources' : ['mylib/impl.pyx', 'clibs/some_other_code.cpp'], 33 | 'link_to' : ["pthread"] # will include the -l when added to the linker line. 34 | }] 35 | 36 | import datetime 37 | cb.setup(extensions, global_macros = global_macros, global_includes = global_includes, 38 | global_lib_dirs = global_lib_dirs)( 39 | name="mylib", 40 | version=datetime.date.today().strftime("%d%b%Y"), 41 | author="A N Other", 42 | author_email="a.other@domain.lol", 43 | url="http://", 44 | packages=["mylib"] 45 | ) 46 | ``` 47 | 48 | some default values to watch for: 49 | 50 | ```python 51 | compiler = { 52 | 'cpp' : " ".join(distutils.sysconfig.get_config_vars('CXX', 'CPPFLAGS')), # normally something like g++ 53 | 'c' : " ".join(distutils.sysconfig.get_config_vars('CC', 'CFLAGS')), # normally something like gcc 54 | 'f90' : 'gfortran' 55 | } 56 | ``` 57 | 58 | *(sysconfig functions return what was in the Makefile that built python.)* 59 | 60 | if you want to support, for example, `F77` files you can do: 61 | 62 | ```python 63 | cb.compiler['f'] = 'gfortran' 64 | ``` 65 | 66 | or, if you want to use the NAG fortran compilers: 67 | 68 | ```python 69 | cb.compiler['f90'] = 'nagfor' 70 | ``` 71 | 72 | or if you're strange (cough cough) and use `.cxx` or `.cc` instead of `.cpp` you can do: 73 | 74 | ```python 75 | cb.compiler['cxx'] = cb.compiler['cpp'] 76 | ``` 77 | 78 | however, you may need to modify the function `cb._linker_vars` to better reflect what runtimes you need to link against (especially if you are mixing fortran and C++ sources.) 79 | 80 | additionally, if you don't have Cython installed on your system, the program shouldn't crash (unless you're trying to compile `.pyx` files, of course) -- it only gets around to `import`ing Cython when it needs to `cythonize` the sources. 81 | 82 | Finally, a more involved example: 83 | 84 | ```python 85 | import cb 86 | import numpy as np 87 | 88 | global_includes = [np.get_include()] 89 | global_macros = [("__FORCE_CPP__",)] 90 | 91 | extensions = [ 92 | {'name' : 'pywat.watershed', 93 | 'sources' : [ 94 | 'pywat/watershed.pyx', 95 | 'clibs/watershed.cpp', 96 | 'clibs/stripack.f90', 97 | 'clibs/tensors/D3Vector.cpp' 98 | ]}, 99 | {'name' : 'pywat.shapes', 100 | 'sources' : [ 101 | 'pywat/shapes.pyx', 102 | 'clibs/custom_types/d3shape.cpp', 103 | 'clibs/custom_types/sphere.cpp', 104 | 'clibs/custom_types/polyhedron.cpp', 105 | 'clibs/custom_types/cylinder.cpp', 106 | 'clibs/tensors/D3Vector.cpp' 107 | ]} 108 | ] 109 | 110 | import datetime 111 | cb.setup(extensions, global_macros = global_macros, global_includes = global_includes)( 112 | name="pywat", 113 | version=datetime.date.today().strftime("%d%b%Y"), 114 | author="Joe Jordan", 115 | author_email="joe.jordan@imperial.ac.uk", 116 | url="TBA", 117 | packages=["pywat"] 118 | ) 119 | ``` 120 | 121 | **INDEPENDENCE** 122 | 123 | If you don't want your project to require the user to come and find my library and install it, you can bundle it with your software as follows: 124 | 125 | ```python 126 | try: 127 | import cb 128 | except ImportError: 129 | print "downloading complicated build..." 130 | import urllib2 131 | response = urllib2.urlopen('https://raw.github.com/joe-jordan/complicated_build/master/cb/__init__.py') 132 | content = response.read() 133 | f = open('cb.py', 'w') 134 | f.write(content) 135 | f.close() 136 | import cb 137 | print "done!" 138 | ``` 139 | 140 | You can see this setup in action in one of my other projects, [pyvoro](https://github.com/joe-jordan/pyvoro), the snippet is simply included in the top of [setup.py](https://github.com/joe-jordan/pyvoro/blob/master/setup.py) 141 | -------------------------------------------------------------------------------- /cb/__init__.py: -------------------------------------------------------------------------------- 1 | # Complicated Build (cb) Copyright (c) 2013 Joe Jordan 2 | # This code is licensed under MIT license (see LICENSE for details) 3 | # 4 | 5 | from distutils.sysconfig import get_python_inc, get_config_vars 6 | import sys, os, os.path, shutil, re, glob 7 | 8 | final_prefix = None 9 | temp_prefix = 'build' + os.sep + 'cb_temp' + os.sep 10 | 11 | from distutils.core import setup as distsetup 12 | 13 | # add more file extensions, with appropriate compiler command, here: 14 | # be warned that you may need to customise _linker_vars below too. 15 | compiler = { 16 | 'cpp' : " ".join(get_config_vars('CXX', 'BASECFLAGS', 'OPT', 'CPPFLAGS', 'CFLAGSFORSHARED')), 17 | 'c' : " ".join(get_config_vars('CC', 'BASECFLAGS', 'OPT', 'CFLAGSFORSHARED')), 18 | 'f90' : 'gfortran ' + " ".join(get_config_vars('BASECFLAGS', 'CFLAGSFORSHARED')) 19 | } 20 | 21 | cythonize = None 22 | 23 | def _custom_cythonise(files): 24 | global cythonize 25 | for i, f in enumerate(files): 26 | if os.path.split(f)[1].split('.')[1] == "pyx": 27 | if cythonize == None: 28 | from Cython.Build import cythonize 29 | e = cythonize(f)[0] 30 | files[i] = e.sources[0] 31 | 32 | def _ensure_dirs(*args): 33 | for d in args: 34 | try: 35 | os.makedirs(d) 36 | except OSError: 37 | # we don't care if the containing folder already exists. 38 | pass 39 | 40 | def _final_target(name): 41 | return final_prefix + name.replace('.', os.sep) + '.so' 42 | 43 | def _temp_dir_for_seperate_module(name, arch): 44 | return temp_prefix + name.replace('.', '') + arch + os.sep 45 | 46 | def _macro_string(macros): 47 | if macros == None or len(macros) == 0: 48 | return "" 49 | outs = [] 50 | for m in macros: 51 | try: 52 | outs.append("-D" + m[0] + '=' + m[1]) 53 | except: 54 | outs.append("-D" + m[0]) 55 | return ' '.join(outs) 56 | 57 | def _include_string(includes): 58 | if includes == None or len(includes) == 0: 59 | return "" 60 | outs = [] 61 | for i in includes: 62 | outs.append("-I" + i) 63 | return ' '.join(outs) 64 | 65 | source_to_object_re = re.compile('[.' + os.sep + ']') 66 | def _source_to_object(f): 67 | return source_to_object_re.sub('', f) + '.o' 68 | 69 | def _linker_vars(file_exts, link_to=None): 70 | linking_compiler = get_config_vars("LDSHARED")[0] 71 | file_exts = set(file_exts) 72 | runtime_libs = "" 73 | cxx = False 74 | if 'cpp' in file_exts: 75 | tmp = get_config_vars("LDCXXSHARED")[0] 76 | if tmp: 77 | linking_compiler = tmp 78 | else: 79 | # if linking using the C compiler, make sure we also link against the C++ runtime. 80 | runtime_libs = "-lstdc++" 81 | cxx = True 82 | if 'f90' in file_exts: 83 | if cxx: 84 | runtime_libs = "-lc -lstdc++" 85 | linking_compiler = " ".join([compiler['f90']] + linking_compiler.split()[1:]) 86 | 87 | if link_to: 88 | runtime_libs = runtime_libs + " " + " ".join(["-l" + i for i in link_to]) 89 | 90 | return (linking_compiler, runtime_libs) 91 | 92 | def _exists_and_newer(target, source): 93 | if os.path.exists(target) and min([target, source], key=os.path.getmtime) == source: 94 | return True 95 | return False 96 | 97 | def _run_command(c, err="compiler error detected!"): 98 | print c 99 | if os.system(c) != 0: 100 | print err 101 | exit() 102 | 103 | def _seperate_build(extension, global_macros, global_includes, global_lib_dirs): 104 | target = _final_target(extension['name']) 105 | temp = _temp_dir_for_seperate_module(extension['name'], extension['arch']) 106 | _ensure_dirs(os.path.split(target)[0], temp) 107 | _custom_cythonise(extension['sources']) 108 | compile_commands = [] 109 | file_exts = [] 110 | object_files = [] 111 | global_macros = _macro_string(global_macros) 112 | global_includes = _include_string(global_includes) 113 | 114 | try: 115 | link_to = extension['link_to'] 116 | except KeyError: 117 | link_to = False 118 | 119 | # compute the compile line for each file: 120 | for f in extension['sources']: 121 | file_exts.append(os.path.split(f)[1].split('.')[1]) 122 | object_files.append(temp + _source_to_object(f)) 123 | compile_commands.append( 124 | ' '.join([ 125 | compiler[file_exts[-1]], 126 | global_macros, 127 | _macro_string(extension['define_macros']) if 'define_macros' in extension else "", 128 | global_includes, 129 | _include_string(extension['include_dirs']) if 'include_dirs' in extension else "", 130 | '-o', object_files[-1], 131 | '-c', f 132 | ]) 133 | ) 134 | 135 | # penultimately, we compute the linking line: 136 | linking_compiler, runtime_libs = _linker_vars(file_exts, link_to) 137 | link_command = ' '.join([ 138 | linking_compiler] + 139 | ["-L" + dir for dir in global_lib_dirs] + [ 140 | ' '.join(object_files), 141 | '-o', target, runtime_libs 142 | ]) 143 | 144 | # now actually run the commands, if the object files need refreshing: 145 | compiled_something = False 146 | for i, cc in enumerate(compile_commands): 147 | if _exists_and_newer(object_files[i], extension['sources'][i]): 148 | print "skipping ", extension['sources'][i], "is already up to date." 149 | continue 150 | compiled_something = True 151 | _run_command(cc) 152 | 153 | if os.path.exists(target) and not compiled_something: 154 | print "module", extension['name'], "was all up to date." 155 | return 156 | _run_command(link_command, "linker error detected!") 157 | 158 | def _common_build(extensions, global_macros, global_includes, global_lib_dirs, arch): 159 | targets = [_final_target(e['name']) for e in extensions] 160 | temp = temp_prefix + "common_build" + arch + os.sep 161 | 162 | _ensure_dirs(*([os.path.split(t)[0] for t in targets] + [temp])) 163 | 164 | # build a common pool of sources: 165 | sources = [] 166 | for e in extensions: 167 | # may result in cythonizing the same file mulitple times, but we need to keep track of which 168 | # extension contains which file types, so this is fiddly to avoid. 169 | _custom_cythonise(e['sources']) 170 | sources += e['sources'] 171 | sources = list(set(sources)) 172 | 173 | compile_commands = [] 174 | file_exts = [] 175 | object_files = [] 176 | global_macros = _macro_string(global_macros) 177 | global_includes = _include_string(global_includes) 178 | 179 | # generate the compile lines for them: 180 | for f in sources: 181 | file_exts.append(os.path.split(f)[1].split('.')[1]) 182 | object_files.append(temp + _source_to_object(f)) 183 | compile_commands.append( 184 | ' '.join([ 185 | compiler[file_exts[-1]], 186 | global_macros, 187 | global_includes, 188 | '-o', object_files[-1], 189 | '-c', f 190 | ]) 191 | ) 192 | 193 | # generate a list of linker lines: 194 | linker_lines = [] 195 | for i, e in enumerate(extensions): 196 | # for each linker line, we have to choose the correct set of file extensions. 197 | linking_compiler, runtime_libs = _linker_vars([os.path.split(f)[1].split('.')[1] for f in e['sources']]) 198 | linker_lines.append(' '.join([ 199 | linking_compiler] + 200 | ["-L" + dir for dir in global_lib_dirs] + 201 | [temp + _source_to_object(f) for f in e['sources']] + [ 202 | '-o', targets[i], runtime_libs 203 | ])) 204 | 205 | # run everything. 206 | compiled_something = False 207 | for i, cc in enumerate(compile_commands): 208 | if _exists_and_newer(object_files[i], sources[i]): 209 | print "skipping ", sources[i], "is already up to date." 210 | continue 211 | compiled_something = True 212 | _run_command(cc) 213 | 214 | # if ANY source files have changed, run all linker lines. 215 | if all([os.path.exists(t) for t in targets]) and not compiled_something: 216 | print "all modules already up to date." 217 | return 218 | 219 | for cc in linker_lines: 220 | _run_command(cc, "linker error detected!") 221 | 222 | def build(extensions, arch='x86_64', global_macros=None, global_includes=None, global_lib_dirs=None): 223 | """extensions should be an array of dicts containing: 224 | { 225 | 'name' : 'mylib.mymodule', 226 | 'sources' : [ 227 | 'path/to/source1.cpp', 228 | 'path/to/source2.f90', 229 | 'path/to/source3.pyx', 230 | 'path/to/source4.c' 231 | ], 232 | # optional: 233 | 'include_dirs' : ['paths'], 234 | 'define_macros' : [("MACRO_NAME", "VALUE")], # or just ("MACRO_NAME",) but remember the ,! 235 | 'link_to' : ['gmp'] # passes linker the -lgmp argument. 236 | } 237 | if global_macros is provided, and 'define_macros' and 'include_dirs' is missing 238 | for all extensions, common sources will only be built once, and linked multiple times. 239 | note, you may still declare global_macros and global_includes. 240 | note also, that the arch argument is now only used to determine temp directory names. 241 | """ 242 | if global_includes == None: 243 | global_includes = [get_python_inc()] 244 | else: 245 | global_includes = [get_python_inc()] + global_includes 246 | if global_lib_dirs == None: 247 | global_lib_dirs = [] 248 | 249 | if (len(extensions) > 1 and 250 | all(['define_macros' not in e and 'include_dirs' not in e and 'link_to' not in e for e in extensions])): 251 | _common_build(extensions, global_macros, global_includes, global_lib_dirs, arch) 252 | else: 253 | for e in extensions: 254 | e['arch'] = arch 255 | _seperate_build(e, global_macros, global_includes, global_lib_dirs) 256 | 257 | def BuildError(Exception): pass 258 | 259 | def setup(*args, **kwargs): 260 | """decorator for distutils.core.setup - use as: 261 | args_to_cb_dot_build = [] 262 | mysetup = cb.setup(*args_to_cb_build) 263 | mysetup(args_to_distutils_setup)""" 264 | def wrapped(*setupargs, **setupkwargs): 265 | global final_prefix 266 | 267 | # check if we're trying to install and haven't built yet? 268 | libs = glob.glob('build/lib*') 269 | if ('install' in sys.argv and len(libs) == 0): 270 | raise BuildError('please run build before trying to install.') 271 | 272 | distsetup(*setupargs, **setupkwargs) 273 | 274 | if 'build' in sys.argv: 275 | # check that lib directory exists: 276 | libs = glob.glob('build/lib*') 277 | if len(libs) != 1: 278 | raise BuildError("please run clean before build, cb can't handle multiple arches!") 279 | final_prefix = libs[0] + os.sep 280 | build(*args, **kwargs) 281 | 282 | return wrapped 283 | 284 | 285 | 286 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | 3 | setup(name="Complicated build", 4 | version="1.4.2", 5 | description="library which provides a simple interface to perform complex compilation tasks with multi-ligual sources.", 6 | author="Joe Jordan", 7 | author_email="tehwalrus@h2j9k.org", 8 | url="https://github.com/joe-jordan/complicated_build", 9 | packages=['cb'] 10 | ) 11 | --------------------------------------------------------------------------------