├── testing.pdf ├── packaging.pdf ├── LSSTC-DSFP_docs_slides.pdf ├── LSSTC-DSFP_repos_slides.pdf ├── LSSTC-DSFP_testing_slides.pdf ├── LICENSE ├── TestingandCI.ipynb ├── TestingandCI_solutions.ipynb ├── Documentation.ipynb └── SoftwareRepositories.ipynb /testing.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eteq/pyastro17-tutorials/HEAD/testing.pdf -------------------------------------------------------------------------------- /packaging.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eteq/pyastro17-tutorials/HEAD/packaging.pdf -------------------------------------------------------------------------------- /LSSTC-DSFP_docs_slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eteq/pyastro17-tutorials/HEAD/LSSTC-DSFP_docs_slides.pdf -------------------------------------------------------------------------------- /LSSTC-DSFP_repos_slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eteq/pyastro17-tutorials/HEAD/LSSTC-DSFP_repos_slides.pdf -------------------------------------------------------------------------------- /LSSTC-DSFP_testing_slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/eteq/pyastro17-tutorials/HEAD/LSSTC-DSFP_testing_slides.pdf -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Erik Tollerud 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /TestingandCI.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from __future__ import print_function, division, absolute_import" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "# Code Testing and CI\n", 19 | "\n", 20 | "\n", 21 | "The notebook contains problems about code testing and continuous integration.\n", 22 | "\n", 23 | "* * *\n", 24 | "\n", 25 | "E Tollerud (STScI)" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "## Problem 0: create the function" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": { 39 | "collapsed": true 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "!mkdir fizz_buzz\n", 44 | "!touch fizz_buzz/__init__.py" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "%%file fizz_buzz/fizz_buzz.py\n", 56 | "\n", 57 | "def fizz_buzz(number):\n", 58 | " if number == 15:\n", 59 | " return 'FizzBuzz'\n", 60 | " if number == 5:\n", 61 | " return 'Buzz'\n", 62 | " if number % 3 == 0:\n", 63 | " return 'Fizz'\n", 64 | " return number" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "## Problem 1: Set up py.test in you repo" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "In this problem we'll aim to get the [py.test](https://docs.pytest.org/en/latest/) testing framework up and running in the code repository you set up in the last set of problems. We can then use it to collect and run tests of the code." 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "### 1a: Ensure py.test is installed\n", 86 | "\n", 87 | "Of course ``py.test`` must actually be installed before you can use it. The commands below should work for the Anaconda Python Distribution, but if you have some other Python installation you'll want to install `pytest` (and its coverage plugin) as directed in the install instructions for ``py.test``." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "collapsed": true 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "!conda install pytest pytest-cov" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "### 1b: Ensure your repo has code suitable for unit tests\n", 106 | "\n", 107 | "Depending on what your code actually does, you might need to modify it to actually perform something testable. For example, if all it does is print something, you might find it difficult to write an effective unit test. Try adding a function that actually performs some operation and returns something different depending on various inputs. That tends to be the easiest function to unit-test: one with a clear \"right\" answer in certain situations." 108 | ] 109 | }, 110 | { 111 | "cell_type": "markdown", 112 | "metadata": {}, 113 | "source": [ 114 | "Also be sure you have `cd`ed to the *root* of the repo for `pytest` to operate correctly." 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "### 1c: Add a test file with a test function\n", 122 | "\n", 123 | "The test must be part of the package and follow the convention that the file and the function begin with ``test`` to get picked up by the test collection machinery. Inside the test function, you'll need some code that fails if the test condition fails. The easiest way to do this is with an ``assert`` statement, which raises an error if its first argument is False.\n", 124 | "\n", 125 | "*Hint: remember that to be a valid python package, a directory must have an ``__init__.py``*" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": { 132 | "collapsed": true 133 | }, 134 | "outputs": [], 135 | "source": [ 136 | "!mkdir #complete\n", 137 | "!touch #complete" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": { 144 | "collapsed": true 145 | }, 146 | "outputs": [], 147 | "source": [ 148 | "%%file tests/test_something.py\n", 149 | "\n", 150 | "def test_something_func():\n", 151 | " assert #complete" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "### 1d: Run the test directly\n", 159 | "\n", 160 | "While this is not how you'd ordinarily run the tests, it's instructive to first try to execute the test *directly*, without using any fancy test framework. If your test function just runs, all is good. If you get an exception, the test failed (which in this case might be *good*).\n", 161 | "\n", 162 | "*Hint: you may need to use `reload` or just re-start your notebook kernel to get the cell below to recognize the changes.*" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": { 169 | "collapsed": true 170 | }, 171 | "outputs": [], 172 | "source": [ 173 | "from .tests import test_something\n", 174 | "\n", 175 | "test_something.test_something_func()" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "### 1e: Run the tests with ``py.test``\n", 183 | "\n", 184 | "Once you have an example test, you can try invoking ``py.test``, which is how you should run the tests in the future. This should yield a report that shows a dot for each test. If all you see are dots, the tests ran sucessfully. But if there's a failure, you'll see the error, and the traceback showing where the error happened." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "metadata": { 191 | "collapsed": true 192 | }, 193 | "outputs": [], 194 | "source": [ 195 | "!py.test" 196 | ] 197 | }, 198 | { 199 | "cell_type": "markdown", 200 | "metadata": {}, 201 | "source": [ 202 | "### 1f: Make the test fail (or succeed...)\n", 203 | "\n", 204 | "If your test failed when you ran it, you should now try to fix the test (or the code...) to make it work. Try running" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "(Modify your test to fail if it succeeded before, or vice versa)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": null, 217 | "metadata": { 218 | "collapsed": true 219 | }, 220 | "outputs": [], 221 | "source": [ 222 | "!py.test" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "### 1g: Check coverage\n", 230 | "\n", 231 | "The coverage plugin we installed will let you check which lines of your code are actually run by the testing suite." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": { 238 | "collapsed": true 239 | }, 240 | "outputs": [], 241 | "source": [ 242 | "!py.test --cov= tests/ #complete" 243 | ] 244 | }, 245 | { 246 | "cell_type": "markdown", 247 | "metadata": {}, 248 | "source": [ 249 | "This should yield a report, which you can use to decide if you need to add more tests to acheive complete coverage. Check out the command line arguments to see if you can get a more detailed line-by-line report." 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "metadata": {}, 255 | "source": [ 256 | "## Problem 2: Implement some unit tests" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "The sub-problems below each contain different unit testing complications. Place the code from the snippets in your repository (either using an editor or the ``%%file`` trick), and write tests to ensure the correctness of the functions. Try to achieve 100% coverage for all of them (especially to catch some hidden bugs!).\n", 264 | "\n", 265 | "Also, note that some of these examples are not really practical - that is, you wouldn't want to do this in *real* code because there's better ways to do it. But because of that, they are good examples of where something can go subtly wrong... and therefore where you want to make tests!" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "metadata": {}, 271 | "source": [ 272 | "### 2a\n", 273 | "\n", 274 | "When you have a function with a default, it's wise to test both the with-default call (``function_b()``), and when you give a value (``function_b(1.2)``)\n", 275 | "\n", 276 | "*Hint: Beware of numbers that come close to 0... write your tests to accomodate floating-point errors!*" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "metadata": { 283 | "collapsed": false 284 | }, 285 | "outputs": [], 286 | "source": [ 287 | "#%%file /.py #complete, or just use your editor\n", 288 | "\n", 289 | "# `math` here is for *scalar* math... normally you'd use numpy but this makes it a bit simpler to debug\n", 290 | "\n", 291 | "import math \n", 292 | "\n", 293 | "inf = float('inf') # this is a quick-and-easy way to get the \"infinity\" value \n", 294 | "\n", 295 | "def function_a(angle=180):\n", 296 | " anglerad = math.radians(angle)\n", 297 | " return math.sin(anglerad/2)/math.sin(anglerad)" 298 | ] 299 | }, 300 | { 301 | "cell_type": "markdown", 302 | "metadata": {}, 303 | "source": [ 304 | "### 2b\n", 305 | "\n", 306 | "This test has an intentional bug... but depending how you right the test you *might* not catch it... Use unit tests to find it! (and then fix it...)\n" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": null, 312 | "metadata": { 313 | "collapsed": true 314 | }, 315 | "outputs": [], 316 | "source": [ 317 | "#%%file /.py #complete, or just use your editor\n", 318 | "\n", 319 | "def function_b(value):\n", 320 | " if value < 0:\n", 321 | " return value - 1\n", 322 | " else:\n", 323 | " value2 = subfunction_b(value + 1)\n", 324 | " return value + value2\n", 325 | " \n", 326 | "def subfunction_b(inp):\n", 327 | " vals_to_accum = []\n", 328 | " for i in range(10):\n", 329 | " vals_to_accum.append(inp ** (i/10))\n", 330 | " if vals_to_accum[-1] > 2:\n", 331 | " vals.append(100)\n", 332 | " # really you would use numpy to do this kind of number-crunching... but we're doing this for the sake of example right now\n", 333 | " return sum(vals_to_accum) " 334 | ] 335 | }, 336 | { 337 | "cell_type": "markdown", 338 | "metadata": {}, 339 | "source": [ 340 | "### 2c\n", 341 | "\n", 342 | "There are (at least) *two* significant bugs in this code (one fairly apparent, one much more subtle). Try to catch them both, and write a regression test that covers those cases once you've found them.\n", 343 | "\n", 344 | "One note about this function: in real code you're probably better off just using [the Angle object from `astropy.coordinates`](http://docs.astropy.org/en/stable/coordinates/angles.html). But this example demonstrates one of the reasons *why* that was created, as it's very easy to write a buggy version of this code.\n", 345 | "\n", 346 | "*Hint: you might find it useful to use `astropy.coordinates.Angle` to create test cases...*" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": null, 352 | "metadata": { 353 | "collapsed": false 354 | }, 355 | "outputs": [], 356 | "source": [ 357 | "#%%file /.py #complete, or just use your editor\n", 358 | "\n", 359 | "import math\n", 360 | "\n", 361 | "# know that to not have to worry about this, you should just use `astropy.coordinates`.\n", 362 | "def angle_to_sexigesimal(angle_in_degrees, decimals=3):\n", 363 | " \"\"\"\n", 364 | " Convert the given angle to a sexigesimal string of hours of RA.\n", 365 | " \n", 366 | " Parameters\n", 367 | " ----------\n", 368 | " angle_in_degrees : float\n", 369 | " A scalar angle, expressed in degrees\n", 370 | " \n", 371 | " Returns\n", 372 | " -------\n", 373 | " hms_str : str\n", 374 | " The sexigesimal string giving the hours, minutes, and seconds of RA for the given `angle_in_degrees`\n", 375 | " \n", 376 | " \"\"\"\n", 377 | " if math.floor(decimals) != decimals:\n", 378 | " raise ValueError('decimals should be an integer!')\n", 379 | " \n", 380 | " hours_num = angle_in_degrees*24/180\n", 381 | " hours = math.floor(hours_num)\n", 382 | " \n", 383 | " min_num = (hours_num - hours)*60\n", 384 | " minutes = math.floor(min_num)\n", 385 | " \n", 386 | " seconds = (min_num - minutes)*60\n", 387 | "\n", 388 | " format_string = '{}:{}:{:.' + str(decimals) + 'f}'\n", 389 | " return format_string.format(hours, minutes, seconds)" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "### 2d\n", 397 | "\n", 398 | "*Hint: numpy has some useful functions in [numpy.testing](https://docs.scipy.org/doc/numpy/reference/routines.testing.html) for comparing arrays.*" 399 | ] 400 | }, 401 | { 402 | "cell_type": "code", 403 | "execution_count": null, 404 | "metadata": { 405 | "collapsed": false 406 | }, 407 | "outputs": [], 408 | "source": [ 409 | "#%%file /.py #complete, or just use your editor\n", 410 | "\n", 411 | "import numpy as np\n", 412 | "\n", 413 | "def function_d(array1=np.arange(10)*2, array2=np.arange(10), operation='-'):\n", 414 | " \"\"\"\n", 415 | " Makes a matrix where the [i,j]th element is array1[i] array2[j]\n", 416 | " \"\"\"\n", 417 | " if operation == '+':\n", 418 | " return array1[:, np.newaxis] + array2\n", 419 | " elif operation == '-':\n", 420 | " return array1[:, np.newaxis] - array2\n", 421 | " elif operation == '*':\n", 422 | " return array1[:, np.newaxis] * array2\n", 423 | " elif operation == '/':\n", 424 | " return array1[:, np.newaxis] / array2\n", 425 | " else:\n", 426 | " raise ValueError('Unrecognized operation \"{}\"'.format(operation))" 427 | ] 428 | }, 429 | { 430 | "cell_type": "markdown", 431 | "metadata": {}, 432 | "source": [ 433 | "## Problem 3: Set up travis to run your tests whenever a change is made" 434 | ] 435 | }, 436 | { 437 | "cell_type": "markdown", 438 | "metadata": {}, 439 | "source": [ 440 | "Now that you have a testing suite set up, you can try to turn on a continuous integration service to constantly check that any update you might send doesn't create a bug. We will the [Travis-CI](https://travis-ci.org/) service for this purpose, as it has one of the lowest barriers to entry from Github." 441 | ] 442 | }, 443 | { 444 | "cell_type": "markdown", 445 | "metadata": {}, 446 | "source": [ 447 | "### 3a: Ensure the test suite is passing locally\n", 448 | "\n", 449 | "Seems obvious, but it's easy to forget to check this and only later realize that all the trouble you thought you had setting up the CI service was because the tests were actually broken..." 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": null, 455 | "metadata": { 456 | "collapsed": true 457 | }, 458 | "outputs": [], 459 | "source": [ 460 | "!py.test" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "### 3b: Set up an account on travis\n", 468 | "\n", 469 | "This turns out to be quite convenient. If you go to the [Travis web site](https://travis-ci.org/), you'll see a \"Sign in with GitHub\" button. You'll need to authorize Travis, but once you've done so it will automatically log you in and know which repositories are yours." 470 | ] 471 | }, 472 | { 473 | "cell_type": "markdown", 474 | "metadata": {}, 475 | "source": [ 476 | "### 3c: Create a minimal ``.travis.yml`` file.\n", 477 | "\n", 478 | "Before we can activate travis on our repo, we need to tell travis a variety of metadata about what's in the repository and how to run it. The template below should be sufficient for the simplest needs." 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": null, 484 | "metadata": { 485 | "collapsed": true 486 | }, 487 | "outputs": [], 488 | "source": [ 489 | "%%file .travis.yml\n", 490 | "\n", 491 | "language: python\n", 492 | "python:\n", 493 | " - \"3.6\"\n", 494 | "# command to install dependencies\n", 495 | "#install: \"pip install numpy\" #uncomment this if your code depends on numpy or similar\n", 496 | "# command to run tests\n", 497 | "script: pytest" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "metadata": {}, 503 | "source": [ 504 | "Be sure to commit and push this to github before proceeding:" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": null, 510 | "metadata": { 511 | "collapsed": true 512 | }, 513 | "outputs": [], 514 | "source": [ 515 | "!git #complete" 516 | ] 517 | }, 518 | { 519 | "cell_type": "markdown", 520 | "metadata": {}, 521 | "source": [ 522 | "### 3d: activate Travis\n", 523 | "\n", 524 | "You can now click on your profile picture in the upper-right and choose \"accounts\". You should see your repo listed there, presumably with a grey X next to it. Click on the X, which should slide the button over and therefore activate travis on that repository. Once you've done this, you should be able to click on the name of the reposository in the travis accounts dashboard, popping up a window showing the build already in progress (if not, just be a bit patient)." 525 | ] 526 | }, 527 | { 528 | "cell_type": "markdown", 529 | "metadata": { 530 | "collapsed": true 531 | }, 532 | "source": [ 533 | "Wait for the tests to complete. If all is good you should see a green check next to the repository name. Otherwise you'll need to go in and fix it and the tests will automatically trigger when you send a new update." 534 | ] 535 | }, 536 | { 537 | "cell_type": "markdown", 538 | "metadata": {}, 539 | "source": [ 540 | "### 3e: Break the build\n", 541 | "\n", 542 | "Make a small change to the repository to break a test. If all else fails simply add the following test:\n", 543 | "\n", 544 | "```\n", 545 | "def test_fail():\n", 546 | " assert False\n", 547 | "```" 548 | ] 549 | }, 550 | { 551 | "cell_type": "markdown", 552 | "metadata": { 553 | "collapsed": true 554 | }, 555 | "source": [ 556 | "Push that change up and go look at travis. It should automatically run the tests and result in them failing." 557 | ] 558 | }, 559 | { 560 | "cell_type": "markdown", 561 | "metadata": {}, 562 | "source": [ 563 | "### 3f: Have your neighbor fix your repo\n", 564 | "\n", 565 | "Challenge your nieghbor to find the bug and fix it. Have them follow the Pull Request workflow, but do *not* merge the PR until Travis' tests have finished (they *should* run automatically, and leave note in the github PR page to that effect). Once the tests have finished, they will tell you if the fix really does cure the bug. If it does, merge it and say thank you. If it doesn't, ask your neighbor to try updating their fix with the feedback from Travis...\n", 566 | "\n", 567 | "*Hint: it may be late in the day, but keep being nice!*" 568 | ] 569 | }, 570 | { 571 | "cell_type": "markdown", 572 | "metadata": {}, 573 | "source": [ 574 | "## Challenge Problem 1: Use py.test \"parametrization\"" 575 | ] 576 | }, 577 | { 578 | "cell_type": "markdown", 579 | "metadata": {}, 580 | "source": [ 581 | "``py.test`` has a feature called test parametrization that can be extremely useful for writing easier-to-understand tests. The key idea is that you can use one simple test *function*, but with multiple inputs, and break that out into separate tests. At first glance this might appear similar to just one test where you interate over lots of inputs, but it's actually much more useful because it doesn't stop at the *first* failure. Rather it will run all the inputs ever time, helpinf you debug subtle problems where only certain inputs fail.\n", 582 | "\n", 583 | "For more info and how to actually *use* the feature, see [the py.test docs on the subject](https://docs.pytest.org/en/latest/parametrize.html). In this challenge problem, try adapting the Problem 2 cases to use this feature. 2c and 2d are particularly amenable to this approach." 584 | ] 585 | }, 586 | { 587 | "cell_type": "markdown", 588 | "metadata": {}, 589 | "source": [ 590 | "## Challenge Problem 2: Test-driven development" 591 | ] 592 | }, 593 | { 594 | "cell_type": "markdown", 595 | "metadata": {}, 596 | "source": [ 597 | "Test-driven development is a radically different approach to designing code from what we're generally used to. In test-driven design, you write the tests *first*. That is, you write how you expect your code to behave before writing the code.\n", 598 | "\n", 599 | "For this problem, try experimenting with test-driven desgin. Choose a problem (ideally from your science interests) where you know some clear cases that you could write tests for. Write the full testing suite (using the techniques you developed above). Then run the tests to ensure all the new ones are failing due to lack of implementation, and then write the new code. A few ideas are given below, but, again, for a real challenge try to come up with your own problem." 600 | ] 601 | }, 602 | { 603 | "cell_type": "markdown", 604 | "metadata": {}, 605 | "source": [ 606 | "* Compute the location of Lagrange points for two arbitrary mass bodies. (Good test cases are the Earth-Moon or Earth-Sun system, which you can probably find on wikipedia.) Consider solving the problem numerically instead of with formulae you can look up, but use the formulae to concoct the test cases.\n", 607 | "* Write a function that uses one of the a clustering algorithm in [scikit-learn](http://scikit-learn.org/stable/modules/clustering.html) to identify the centers of two 2D gaussian point-clouds. The tests are particularly easy to formulate before-hand because you know the right answer at the outset if you generate the point-clouds yourself." 608 | ] 609 | } 610 | ], 611 | "metadata": { 612 | "anaconda-cloud": {}, 613 | "kernelspec": { 614 | "display_name": "Python [default]", 615 | "language": "python", 616 | "name": "python3" 617 | }, 618 | "language_info": { 619 | "codemirror_mode": { 620 | "name": "ipython", 621 | "version": 3 622 | }, 623 | "file_extension": ".py", 624 | "mimetype": "text/x-python", 625 | "name": "python", 626 | "nbconvert_exporter": "python", 627 | "pygments_lexer": "ipython3", 628 | "version": "3.5.2" 629 | } 630 | }, 631 | "nbformat": 4, 632 | "nbformat_minor": 2 633 | } 634 | -------------------------------------------------------------------------------- /TestingandCI_solutions.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from __future__ import print_function, division, absolute_import" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "# Code Testing and CI\n", 19 | "\n", 20 | "**Version 0.1**\n", 21 | "\n", 22 | "The notebook contains problems about code testing and continuous integration.\n", 23 | "\n", 24 | "* * *\n", 25 | "\n", 26 | "E Tollerud (STScI)" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": {}, 32 | "source": [ 33 | "## Problem 1: Set up py.test in you repo" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "In this problem we'll aim to get the [py.test](https://docs.pytest.org/en/latest/) testing framework up and running in the code repository you set up in the last set of problems. We can then use it to collect and run tests of the code." 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "### 1a: Ensure py.test is installed\n", 48 | "\n", 49 | "Of course ``py.test`` must actually be installed before you can use it. The commands below should work for the Anaconda Python Distribution, but if you have some other Python installation you'll want to install `pytest` (and its coverage plugin) as directed in the install instructions for ``py.test``." 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": { 56 | "collapsed": true 57 | }, 58 | "outputs": [], 59 | "source": [ 60 | "!conda install pytest pytest-cov" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "### 1b: Ensure your repo has code suitable for unit tests\n", 68 | "\n", 69 | "Depending on what your code actually does, you might need to modify it to actually perform something testable. For example, if all it does is print something, you might find it difficult to write an effective unit test. Try adding a function that actually performs some operation and returns something different depending on various inputs. That tends to be the easiest function to unit-test: one with a clear \"right\" answer in certain situations." 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "Also be sure you have `cd`ed to the *root* of the repo for `pytest` to operate correctly." 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "### 1c: Add a test file with a test function\n", 84 | "\n", 85 | "The test must be part of the package and follow the convention that the file and the function begin with ``test`` to get picked up by the test collection machinery. Inside the test function, you'll need some code that fails if the test condition fails. The easiest way to do this is with an ``assert`` statement, which raises an error if its first argument is False.\n", 86 | "\n", 87 | "*Hint: remember that to be a valid python package, a directory must have an ``__init__.py``*" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "collapsed": true 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "!mkdir #complete\n", 99 | "!touch #complete" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": { 106 | "collapsed": true 107 | }, 108 | "outputs": [], 109 | "source": [ 110 | "%%file /tests/test_something.py\n", 111 | "\n", 112 | "def test_something_func():\n", 113 | " assert #complete" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "### 1d: Run the test directly\n", 121 | "\n", 122 | "While this is not how you'd ordinarily run the tests, it's instructive to first try to execute the test *directly*, without using any fancy test framework. If your test function just runs, all is good. If you get an exception, the test failed (which in this case might be *good*).\n", 123 | "\n", 124 | "*Hint: you may need to use `reload` or just re-start your notebook kernel to get the cell below to recognize the changes.*" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": { 131 | "collapsed": true 132 | }, 133 | "outputs": [], 134 | "source": [ 135 | "from .tests import test_something\n", 136 | "\n", 137 | "test_something.test_something_func()" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "### 1e: Run the tests with ``py.test``\n", 145 | "\n", 146 | "Once you have an example test, you can try invoking ``py.test``, which is how you should run the tests in the future. This should yield a report that shows a dot for each test. If all you see are dots, the tests ran sucessfully. But if there's a failure, you'll see the error, and the traceback showing where the error happened." 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": { 153 | "collapsed": true 154 | }, 155 | "outputs": [], 156 | "source": [ 157 | "!py.test" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": {}, 163 | "source": [ 164 | "### 1f: Make the test fail (or succeed...)\n", 165 | "\n", 166 | "If your test failed when you ran it, you should now try to fix the test (or the code...) to make it work. Try running" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "(Modify your test to fail if it succeeded before, or vice versa)" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "!py.test" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "### 1g: Check coverage\n", 192 | "\n", 193 | "The coverage plugin we installed will let you check which lines of your code are actually run by the testing suite." 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": { 200 | "collapsed": true 201 | }, 202 | "outputs": [], 203 | "source": [ 204 | "!py.test --cov= tests/ #complete" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "This should yield a report, which you can use to decide if you need to add more tests to acheive complete coverage. Check out the command line arguments to see if you can get a more detailed line-by-line report." 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "## Problem 2: Implement some unit tests" 219 | ] 220 | }, 221 | { 222 | "cell_type": "markdown", 223 | "metadata": {}, 224 | "source": [ 225 | "The sub-problems below each contain different unit testing complications. Place the code from the snippets in your repository (either using an editor or the ``%%file`` trick), and write tests to ensure the correctness of the functions. Try to achieve 100% coverage for all of them (especially to catch some hidden bugs!).\n", 226 | "\n", 227 | "Also, note that some of these examples are not really practical - that is, you wouldn't want to do this in *real* code because there's better ways to do it. But because of that, they are good examples of where something can go subtly wrong... and therefore where you want to make tests!" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "### 2a\n", 235 | "\n", 236 | "When you have a function with a default, it's wise to test both the with-default call (``function_b()``), and when you give a value (``function_b(1.2)``)\n", 237 | "\n", 238 | "*Hint: Beware of numbers that come close to 0... write your tests to accomodate floating-point errors!*" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": null, 244 | "metadata": { 245 | "collapsed": false 246 | }, 247 | "outputs": [], 248 | "source": [ 249 | "#%%file /.py #complete, or just use your editor\n", 250 | "\n", 251 | "# `math` here is for *scalar* math... normally you'd use numpy but this makes it a bit simpler to debug\n", 252 | "\n", 253 | "import math \n", 254 | "\n", 255 | "inf = float('inf') # this is a quick-and-easy way to get the \"infinity\" value \n", 256 | "\n", 257 | "def function_a(angle=180):\n", 258 | " anglerad = math.radians(angle)\n", 259 | " return math.sin(anglerad/2)/math.sin(anglerad)" 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "### Solution (one of many...)" 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": null, 272 | "metadata": { 273 | "collapsed": false 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "def test_default_bad():\n", 278 | " # this will fail, although it *seems* like it should succeed... the sin function has rounding errors\n", 279 | " assert function_a() == inf \n", 280 | "\n", 281 | "def test_default_good():\n", 282 | " assert function_a() > 1e10\n", 283 | " \n", 284 | "def test_otherval_bad():\n", 285 | " # again it seems like it should succed, but rounding errors make it fail\n", 286 | " assert function_a(90) == math.sqrt(2)/2\n", 287 | " \n", 288 | "def test_otherval_good():\n", 289 | " assert abs(function_a(90) - math.sqrt(2)/2) < 1e-10" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "### 2b\n", 297 | "\n", 298 | "This test has an intentional bug... but depending how you right the test you *might* not catch it... Use unit tests to find it! (and then fix it...)\n" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": null, 304 | "metadata": { 305 | "collapsed": true 306 | }, 307 | "outputs": [], 308 | "source": [ 309 | "#%%file /.py #complete, or just use your editor\n", 310 | "\n", 311 | "def function_b(value):\n", 312 | " if value < 0:\n", 313 | " return value - 1\n", 314 | " else:\n", 315 | " value2 = subfunction_b(value + 1)\n", 316 | " return value + value2\n", 317 | " \n", 318 | "def subfunction_b(inp):\n", 319 | " vals_to_accum = []\n", 320 | " for i in range(10):\n", 321 | " vals_to_accum.append(inp ** (i/10))\n", 322 | " if vals_to_accum[-1] > 2:\n", 323 | " vals.append(100)\n", 324 | " # really you would use numpy to do this kind of number-crunching... but we're doing this for the sake of example right now\n", 325 | " return sum(vals_to_accum) " 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "metadata": {}, 331 | "source": [ 332 | "### Solution (one of many...)" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": null, 338 | "metadata": { 339 | "collapsed": false 340 | }, 341 | "outputs": [], 342 | "source": [ 343 | "def test_neg():\n", 344 | " assert function_b(-10) == -11\n", 345 | " \n", 346 | "def test_zero():\n", 347 | " assert function_b(0) == 10\n", 348 | " \n", 349 | "def test_pos_lt1():\n", 350 | " res = function_b(.5) \n", 351 | " assert res > 10\n", 352 | " assert res < 100\n", 353 | " \n", 354 | "def test_pos_gt1():\n", 355 | " res = function_b(1.5) \n", 356 | " assert res > 100\n", 357 | " \n", 358 | "# this test reveals that `subfunction_b()` has a ``vals`` where it should have a ``vals_to_accum``" 359 | ] 360 | }, 361 | { 362 | "cell_type": "markdown", 363 | "metadata": {}, 364 | "source": [ 365 | "### 2c\n", 366 | "\n", 367 | "There are (at least) *two* significant bugs in this code (one fairly apparent, one much more subtle). Try to catch them both, and write a regression test that covers those cases once you've found them.\n", 368 | "\n", 369 | "One note about this function: in real code you're probably better off just using [the Angle object from `astropy.coordinates`](http://docs.astropy.org/en/stable/coordinates/angles.html). But this example demonstrates one of the reasons *why* that was created, as it's very easy to write a buggy version of this code.\n", 370 | "\n", 371 | "*Hint: you might find it useful to use `astropy.coordinates.Angle` to create test cases...*" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "metadata": { 378 | "collapsed": false 379 | }, 380 | "outputs": [], 381 | "source": [ 382 | "#%%file /.py #complete, or just use your editor\n", 383 | "\n", 384 | "import math\n", 385 | "\n", 386 | "# know that to not have to worry about this, you should just use `astropy.coordinates`.\n", 387 | "def angle_to_sexigesimal(angle_in_degrees, decimals=3):\n", 388 | " \"\"\"\n", 389 | " Convert the given angle to a sexigesimal string of hours of RA.\n", 390 | " \n", 391 | " Parameters\n", 392 | " ----------\n", 393 | " angle_in_degrees : float\n", 394 | " A scalar angle, expressed in degrees\n", 395 | " \n", 396 | " Returns\n", 397 | " -------\n", 398 | " hms_str : str\n", 399 | " The sexigesimal string giving the hours, minutes, and seconds of RA for the given `angle_in_degrees`\n", 400 | " \n", 401 | " \"\"\"\n", 402 | " if math.floor(decimals) != decimals:\n", 403 | " raise ValueError('decimals should be an integer!')\n", 404 | " \n", 405 | " hours_num = angle_in_degrees*24/180\n", 406 | " hours = math.floor(hours_num)\n", 407 | " \n", 408 | " min_num = (hours_num - hours)*60\n", 409 | " minutes = math.floor(min_num)\n", 410 | " \n", 411 | " seconds = (min_num - minutes)*60\n", 412 | "\n", 413 | " format_string = '{}:{}:{:.' + str(decimals) + 'f}'\n", 414 | " return format_string.format(hours, minutes, seconds)" 415 | ] 416 | }, 417 | { 418 | "cell_type": "markdown", 419 | "metadata": {}, 420 | "source": [ 421 | "### Solution (one of many...)" 422 | ] 423 | }, 424 | { 425 | "cell_type": "code", 426 | "execution_count": null, 427 | "metadata": { 428 | "collapsed": false 429 | }, 430 | "outputs": [], 431 | "source": [ 432 | "def test_decimals():\n", 433 | " assert angle_to_sexigesimal(0) == '0:0:0.000'\n", 434 | " assert angle_to_sexigesimal(0, decimals=5) == '0:0:0.00000'\n", 435 | " \n", 436 | "\n", 437 | "def test_qtrs():\n", 438 | " assert angle_to_sexigesimal(90, decimals=0) == '6:0:0'\n", 439 | " assert angle_to_sexigesimal(180, decimals=0) == '12:0:0'\n", 440 | " assert angle_to_sexigesimal(270, decimals=0) == '18:0:0'\n", 441 | " assert angle_to_sexigesimal(360, decimals=0) == '24:0:0'\n", 442 | " # this reveals the major bug that the 180 at the top should be 360\n", 443 | " \n", 444 | "def test_350():\n", 445 | " assert angle_to_sexigesimal(350, decimals=0) == '23:20:00'\n", 446 | " # this fails, revealing that sometimes the values round \n", 447 | "\n", 448 | "def test_neg():\n", 449 | " assert angle_to_sexigesimal(-7.5, decimals=0) == '-0:30:0'\n", 450 | " assert angle_to_sexigesimal(-20, decimals=0) == angle_to_sexigesimal(340, decimals=0)\n", 451 | " # these fail, revealing a \"debatable\" bug: that negative degrees give negative RAs that are \n", 452 | " # nonsense. You could always tell users not to give negative values... but users, particularly \n", 453 | " # future you, probably won't listen.\n", 454 | " \n", 455 | "def test_neg_decimals():\n", 456 | " import pytest\n", 457 | " \n", 458 | " with pytest.raises(ValueError):\n", 459 | " angle_to_sexigesimal(10, decimals=-2)" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "### 2d\n", 467 | "\n", 468 | "*Hint: numpy has some useful functions in [numpy.testing](https://docs.scipy.org/doc/numpy/reference/routines.testing.html) for comparing arrays.*" 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": null, 474 | "metadata": { 475 | "collapsed": false 476 | }, 477 | "outputs": [], 478 | "source": [ 479 | "#%%file /.py #complete, or just use your editor\n", 480 | "\n", 481 | "import numpy as np\n", 482 | "\n", 483 | "def function_d(array1=np.arange(10)*2, array2=np.arange(10), operation='-'):\n", 484 | " \"\"\"\n", 485 | " Makes a matrix where the [i,j]th element is array1[i] array2[j]\n", 486 | " \"\"\"\n", 487 | " if operation == '+':\n", 488 | " return array1[:, np.newaxis] + array2\n", 489 | " elif operation == '-':\n", 490 | " return array1[:, np.newaxis] - array2\n", 491 | " elif operation == '*':\n", 492 | " return array1[:, np.newaxis] * array2\n", 493 | " elif operation == '/':\n", 494 | " return array1[:, np.newaxis] / array2\n", 495 | " else:\n", 496 | " raise ValueError('Unrecognized operation \"{}\"'.format(operation))" 497 | ] 498 | }, 499 | { 500 | "cell_type": "markdown", 501 | "metadata": {}, 502 | "source": [ 503 | "### Solution (one of many...)" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "execution_count": null, 509 | "metadata": { 510 | "collapsed": false 511 | }, 512 | "outputs": [], 513 | "source": [ 514 | "def test_minus():\n", 515 | " array1 = np.arange(10)*2\n", 516 | " array2 = np.arange(10)\n", 517 | " \n", 518 | " func_mat = function_d(array1, array2, operation='-')\n", 519 | " \n", 520 | " for i, val1 in enumerate(array1):\n", 521 | " for j, val2 in enumerate(array2):\n", 522 | " assert func_mat[i, j] == val1 - val2\n", 523 | " \n", 524 | "def test_plus():\n", 525 | " array1 = np.arange(10)*2\n", 526 | " array2 = np.arange(10)\n", 527 | " \n", 528 | " func_mat = function_d(array1, array2, operation='+')\n", 529 | " \n", 530 | " for i, val1 in enumerate(array1):\n", 531 | " for j, val2 in enumerate(array2):\n", 532 | " assert func_mat[i, j] == val1 + val2\n", 533 | " \n", 534 | "def test_times():\n", 535 | " array1 = np.arange(10)*2\n", 536 | " array2 = np.arange(10)\n", 537 | " \n", 538 | " func_mat = function_d(array1, array2, operation='*')\n", 539 | " \n", 540 | " for i, val1 in enumerate(array1):\n", 541 | " for j, val2 in enumerate(array2):\n", 542 | " assert func_mat[i, j] == val1 * val2\n", 543 | " \n", 544 | "def test_div():\n", 545 | " array1 = np.arange(10)*2\n", 546 | " array2 = np.arange(10)\n", 547 | " \n", 548 | " func_mat = function_d(array1, array2, operation='/')\n", 549 | " \n", 550 | " for i, val1 in enumerate(array1):\n", 551 | " for j, val2 in enumerate(array2):\n", 552 | " assert func_mat[i, j] == val1 / val2\n", 553 | " #GOTCHA! This doesn't work because of floating point differences between numpy and python scalars \n", 554 | " # This is where that numpy stuff is handy - see the next function\n", 555 | "\n", 556 | "def test_div_npt():\n", 557 | " from numpy import testing as npt\n", 558 | " \n", 559 | " array1 = np.arange(10)*2\n", 560 | " array2 = np.arange(10)\n", 561 | " \n", 562 | " func_mat = function_d(array1, array2, operation='/')\n", 563 | " \n", 564 | " test_mat = np.empty(((len(array1), len(array2))))\n", 565 | " for i, val1 in enumerate(array1):\n", 566 | " for j, val2 in enumerate(array2):\n", 567 | " test_mat[i, j] = val1 / val2\n", 568 | " \n", 569 | " npt.assert_array_almost_equal(func_mat, test_mat)" 570 | ] 571 | }, 572 | { 573 | "cell_type": "markdown", 574 | "metadata": {}, 575 | "source": [ 576 | "## Problem 3: Set up travis to run your tests whenever a change is made" 577 | ] 578 | }, 579 | { 580 | "cell_type": "markdown", 581 | "metadata": {}, 582 | "source": [ 583 | "Now that you have a testing suite set up, you can try to turn on a continuous integration service to constantly check that any update you might send doesn't create a bug. We will the [Travis-CI](https://travis-ci.org/) service for this purpose, as it has one of the lowest barriers to entry from Github." 584 | ] 585 | }, 586 | { 587 | "cell_type": "markdown", 588 | "metadata": {}, 589 | "source": [ 590 | "### 3a: Ensure the test suite is passing locally\n", 591 | "\n", 592 | "Seems obvious, but it's easy to forget to check this and only later realize that all the trouble you thought you had setting up the CI service was because the tests were actually broken..." 593 | ] 594 | }, 595 | { 596 | "cell_type": "code", 597 | "execution_count": null, 598 | "metadata": { 599 | "collapsed": true 600 | }, 601 | "outputs": [], 602 | "source": [ 603 | "!py.test" 604 | ] 605 | }, 606 | { 607 | "cell_type": "markdown", 608 | "metadata": {}, 609 | "source": [ 610 | "### 3b: Set up an account on travis\n", 611 | "\n", 612 | "This turns out to be quite convenient. If you go to the [Travis web site](https://travis-ci.org/), you'll see a \"Sign in with GitHub\" button. You'll need to authorize Travis, but once you've done so it will automatically log you in and know which repositories are yours." 613 | ] 614 | }, 615 | { 616 | "cell_type": "markdown", 617 | "metadata": {}, 618 | "source": [ 619 | "### 3c: Create a minimal ``.travis.yml`` file.\n", 620 | "\n", 621 | "Before we can activate travis on our repo, we need to tell travis a variety of metadata about what's in the repository and how to run it. The template below should be sufficient for the simplest needs." 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "metadata": { 628 | "collapsed": true 629 | }, 630 | "outputs": [], 631 | "source": [ 632 | "%%file .travis.yml\n", 633 | "\n", 634 | "language: python\n", 635 | "python:\n", 636 | " - \"3.6\"\n", 637 | "# command to install dependencies\n", 638 | "#install: \"pip install numpy\" #uncomment this if your code depends on numpy or similar\n", 639 | "# command to run tests\n", 640 | "script: pytest" 641 | ] 642 | }, 643 | { 644 | "cell_type": "markdown", 645 | "metadata": {}, 646 | "source": [ 647 | "Be sure to commit and push this to github before proceeding:" 648 | ] 649 | }, 650 | { 651 | "cell_type": "code", 652 | "execution_count": null, 653 | "metadata": { 654 | "collapsed": true 655 | }, 656 | "outputs": [], 657 | "source": [ 658 | "!git #complete" 659 | ] 660 | }, 661 | { 662 | "cell_type": "markdown", 663 | "metadata": {}, 664 | "source": [ 665 | "### 3d: activate Travis\n", 666 | "\n", 667 | "You can now click on your profile picture in the upper-right and choose \"accounts\". You should see your repo listed there, presumably with a grey X next to it. Click on the X, which should slide the button over and therefore activate travis on that repository. Once you've done this, you should be able to click on the name of the reposository in the travis accounts dashboard, popping up a window showing the build already in progress (if not, just be a bit patient)." 668 | ] 669 | }, 670 | { 671 | "cell_type": "markdown", 672 | "metadata": { 673 | "collapsed": true 674 | }, 675 | "source": [ 676 | "Wait for the tests to complete. If all is good you should see a green check next to the repository name. Otherwise you'll need to go in and fix it and the tests will automatically trigger when you send a new update." 677 | ] 678 | }, 679 | { 680 | "cell_type": "markdown", 681 | "metadata": {}, 682 | "source": [ 683 | "### 3e: Break the build\n", 684 | "\n", 685 | "Make a small change to the repository to break a test. If all else fails simply add the following test:\n", 686 | "\n", 687 | "```\n", 688 | "def test_fail():\n", 689 | " assert False\n", 690 | "```" 691 | ] 692 | }, 693 | { 694 | "cell_type": "markdown", 695 | "metadata": { 696 | "collapsed": true 697 | }, 698 | "source": [ 699 | "Push that change up and go look at travis. It should automatically run the tests and result in them failing." 700 | ] 701 | }, 702 | { 703 | "cell_type": "markdown", 704 | "metadata": {}, 705 | "source": [ 706 | "### 3f: Have your neighbor fix your repo\n", 707 | "\n", 708 | "Challenge your nieghbor to find the bug and fix it. Have them follow the Pull Request workflow, but do *not* merge the PR until Travis' tests have finished (they *should* run automatically, and leave note in the github PR page to that effect). Once the tests have finished, they will tell you if the fix really does cure the bug. If it does, merge it and say thank you. If it doesn't, ask your neighbor to try updating their fix with the feedback from Travis...\n", 709 | "\n", 710 | "*Hint: it may be late in the day, but keep being nice!*" 711 | ] 712 | }, 713 | { 714 | "cell_type": "markdown", 715 | "metadata": {}, 716 | "source": [ 717 | "## Challenge Problem 1: Use py.test \"parametrization\"" 718 | ] 719 | }, 720 | { 721 | "cell_type": "markdown", 722 | "metadata": {}, 723 | "source": [ 724 | "``py.test`` has a feature called test parametrization that can be extremely useful for writing easier-to-understand tests. The key idea is that you can use one simple test *function*, but with multiple inputs, and break that out into separate tests. At first glance this might appear similar to just one test where you interate over lots of inputs, but it's actually much more useful because it doesn't stop at the *first* failure. Rather it will run all the inputs ever time, helpinf you debug subtle problems where only certain inputs fail.\n", 725 | "\n", 726 | "For more info and how to actually *use* the feature, see [the py.test docs on the subject](https://docs.pytest.org/en/latest/parametrize.html). In this challenge problem, try adapting the Problem 2 cases to use this feature. 2c and 2d are particularly amenable to this approach." 727 | ] 728 | }, 729 | { 730 | "cell_type": "markdown", 731 | "metadata": {}, 732 | "source": [ 733 | "## Challenge Problem 2: Test-driven development" 734 | ] 735 | }, 736 | { 737 | "cell_type": "markdown", 738 | "metadata": {}, 739 | "source": [ 740 | "Test-driven development is a radically different approach to designing code from what we're generally used to. In test-driven design, you write the tests *first*. That is, you write how you expect your code to behave before writing the code.\n", 741 | "\n", 742 | "For this problem, try experimenting with test-driven desgin. Choose a problem (ideally from your science interests) where you know some clear cases that you could write tests for. Write the full testing suite (using the techniques you developed above). Then run the tests to ensure all the new ones are failing due to lack of implementation, and then write the new code. A few ideas are given below, but, again, for a real challenge try to come up with your own problem." 743 | ] 744 | }, 745 | { 746 | "cell_type": "markdown", 747 | "metadata": {}, 748 | "source": [ 749 | "* Compute the location of Lagrange points for two arbitrary mass bodies. (Good test cases are the Earth-Moon or Earth-Sun system, which you can probably find on wikipedia.) Consider solving the problem numerically instead of with formulae you can look up, but use the formulae to concoct the test cases.\n", 750 | "* Write a function that uses one of the a clustering algorithm in [scikit-learn](http://scikit-learn.org/stable/modules/clustering.html) to identify the centers of two 2D gaussian point-clouds. The tests are particularly easy to formulate before-hand because you know the right answer at the outset if you generate the point-clouds yourself." 751 | ] 752 | } 753 | ], 754 | "metadata": { 755 | "anaconda-cloud": {}, 756 | "kernelspec": { 757 | "display_name": "Python [default]", 758 | "language": "python", 759 | "name": "python3" 760 | }, 761 | "language_info": { 762 | "codemirror_mode": { 763 | "name": "ipython", 764 | "version": 3 765 | }, 766 | "file_extension": ".py", 767 | "mimetype": "text/x-python", 768 | "name": "python", 769 | "nbconvert_exporter": "python", 770 | "pygments_lexer": "ipython3", 771 | "version": "3.5.2" 772 | } 773 | }, 774 | "nbformat": 4, 775 | "nbformat_minor": 2 776 | } 777 | -------------------------------------------------------------------------------- /Documentation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from __future__ import print_function, division, absolute_import" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "# Documenting Code\n", 19 | "\n", 20 | "**Version 0.1**\n", 21 | "\n", 22 | "The notebook contains problems for documenting code, particularly in Python.\n", 23 | "\n", 24 | "These probablems assume you've done the \"code repo\" problem set, or otherwise are familiar with code repositories and packaging enough to have a functioning test repository with a Python package.\n", 25 | "\n", 26 | "* * *\n", 27 | "\n", 28 | "E Tollerud (STScI)" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "# Problem 1: Making and using docstrings" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "One of Python's most powerful documentation tools are docstrings. These are basically just little strings you put at the top of a class, function, or similar, which then gets bound as sort of a glorified comment. But with internal consistency and tenacity, these can do most of the work you need to document your code.\n", 43 | "\n", 44 | "Note that all of this problem *can* be done in the notebook, and it is shown that way to make the notebook internally consistent. But you might find it more useful to use a function from your code repository, as that makes it clearer why docstrings are useful (i.e., the code is not immediately visible, as it is in the notebook)." 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### 1a: Create a function with a docstring" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "Make a function (or just use one from your pre-existing repo) and give it a docstring. The docstring can be anything in principal, but the example case below shows one of the most standard conventions used for scientific Python code. (The format originated in [numpy](https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt), although with some \"flavors\" like that in [astropy](http://docs.astropy.org/en/stable/development/docguide.html).)\n", 59 | "\n", 60 | "*Hint: you might want to keep the code part of your function secret from your neighbor, as it will reduce the work for 1c if you do so*" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": { 67 | "collapsed": true 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "def do_something(arg1, arg2):\n", 72 | " \"\"\"\n", 73 | " A short sentence describing what this function does.\n", 74 | " \n", 75 | " More description\n", 76 | " \n", 77 | " Parameters\n", 78 | " ----------\n", 79 | " arg1 : type1\n", 80 | " Description of the parameter ``arg1``\n", 81 | " arg2 : type2\n", 82 | " Description of the parameter ``arg2``\n", 83 | " \n", 84 | " Returns\n", 85 | " -------\n", 86 | " type of return value (e.g. int, float, string, etc.)\n", 87 | " A description of the thing the function returns (if anything)\n", 88 | " \"\"\" # complete\n", 89 | " # complete" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "### 1b: Access the docstring from within Python" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "It turns out that docstrings are more than just strings sitting un-used inside a function: they are associated with and carried around as a part of the function. This means you can access them in useful ways even if you don't have the code right in front of you. Explore some of the ways available to look at your function's docstring. Try the various ways below and consider which seems most useful in various contexts." 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "collapsed": false 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "do_something.__doc__" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": { 121 | "collapsed": false 122 | }, 123 | "outputs": [], 124 | "source": [ 125 | "help(do_something)" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": { 132 | "collapsed": true 133 | }, 134 | "outputs": [], 135 | "source": [ 136 | "do_something?" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": { 143 | "collapsed": true 144 | }, 145 | "outputs": [], 146 | "source": [ 147 | "# this one does more than just show the docstring, but is useful to know nonetheless\n", 148 | "do_something??" 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "metadata": {}, 154 | "source": [ 155 | "### 1c: Share your docstring" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "Using either the function you just wrote or a new one (if you've already shown the function to your neighbor), try to communicate the essentials of your function enough for your neighbor to use the function. That is, have your neighbor run your function (and do something with the output) using *only* the information in the docstring.\n", 163 | "\n", 164 | "No peeking at the code! One way to ensure this is to put the function in your github repo and have your neighbor pull down the updates and import the code directly. Or you can just type your function higher in the notebook and scroll down without your neighbor looking.\n", 165 | "\n", 166 | "Rinse and repeat for *you* looking at your *neighbor*'s code." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": { 173 | "collapsed": true 174 | }, 175 | "outputs": [], 176 | "source": [ 177 | "from your_neighbors_package import your_neighbors_code # complete\n", 178 | "\n", 179 | "your_neighbors_code? # complete" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": { 186 | "collapsed": true 187 | }, 188 | "outputs": [], 189 | "source": [ 190 | "... = your_neighbors_code(...)\n", 191 | "... # complete" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "### 1d: Write a class docstring" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "Try making a docstring for a *class*. This is quite similar to a function, but with some subtle difference (as detailed in the template below)." 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": { 212 | "collapsed": true 213 | }, 214 | "outputs": [], 215 | "source": [ 216 | "class MyClass: # if you're using Py2, you'll want to do \"MyClass(object)\"\n", 217 | " \"\"\"\n", 218 | " A short description of the class.\n", 219 | " \n", 220 | " Possibly some extended description, notes on how to sub-class, etc.\n", 221 | " \n", 222 | " Parameters\n", 223 | " ----------\n", 224 | " arg1 : type\n", 225 | " Describe the first argument of the initializer\n", 226 | " arg2 : type\n", 227 | " Describe the second argument of the initializer\n", 228 | " \"\"\"\n", 229 | " def __init__(self, arg1 arg2):\n", 230 | " # note that the initializer gets *no* docstring, because it's in the class docs\n", 231 | " #complete\n", 232 | " \n", 233 | " def some_method(self, method_arg):\n", 234 | " \"\"\"\n", 235 | " A short description of the method.\n", 236 | " \n", 237 | " Possibly extended description.\n", 238 | " \n", 239 | " Parameters\n", 240 | " ----------\n", 241 | " method_arg : type\n", 242 | " A description of the method's first (non-self) argument.\n", 243 | " \n", 244 | " Returns\n", 245 | " -------\n", 246 | " return type\n", 247 | " Description of the return value (if any)\n", 248 | " \"\"\"\n", 249 | " #complete" 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "metadata": {}, 255 | "source": [ 256 | "### 1e: Write a docstring for your modules (and package)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "Add a docstring to the module and packages in your repository. This is usually just free-form text (not as much structure as a function or class), although you might include some structure like section headings or bullet-pointed lists. Once you've done that (and reloaded or restarted the kernel), the commands below should pop up your documentation.\n", 264 | "\n", 265 | "*Hint: remember that a package's ``__init__.py`` file acts sort of like the \"package.py\" file for the package*" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": { 272 | "collapsed": true 273 | }, 274 | "outputs": [], 275 | "source": [ 276 | "import #complete\n", 277 | "\n", 278 | "? #complete" 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": null, 284 | "metadata": { 285 | "collapsed": true 286 | }, 287 | "outputs": [], 288 | "source": [ 289 | "from import #complete\n", 290 | "\n", 291 | "? #complete" 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": {}, 297 | "source": [ 298 | "# Problem 2: Building your Docs with Sphinx" 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": {}, 304 | "source": [ 305 | "Intro" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "### 2a: Make sure Sphinx is installed" 313 | ] 314 | }, 315 | { 316 | "cell_type": "markdown", 317 | "metadata": {}, 318 | "source": [ 319 | "You may have sphinx installed already, but if not, you'll want to install it. The invocation below is appropriate for the Anaconda Python Distribution (but change it from `conda` to `pip` to install from pip)." 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": null, 325 | "metadata": { 326 | "collapsed": true 327 | }, 328 | "outputs": [], 329 | "source": [ 330 | "!conda install sphinx " 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "### 2b: Create a directory for the docs" 338 | ] 339 | }, 340 | { 341 | "cell_type": "markdown", 342 | "metadata": {}, 343 | "source": [ 344 | "It's good practice to keep the *narrative* documentation (i.e., the non-docstring part) and the code in separate places. To do that you'll need to create a new directory for the docs." 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "metadata": { 351 | "collapsed": true 352 | }, 353 | "outputs": [], 354 | "source": [ 355 | "%cd #complete" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "metadata": { 362 | "collapsed": false 363 | }, 364 | "outputs": [], 365 | "source": [ 366 | "!mkdir docs\n", 367 | "%cd docs" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": {}, 373 | "source": [ 374 | "### 2c: Set up the standard sphinx documentation layout" 375 | ] 376 | }, 377 | { 378 | "cell_type": "markdown", 379 | "metadata": {}, 380 | "source": [ 381 | "Sphinx has a standard layout of files that it uses, and even provides a tool for doing this called ``sphinx-quickstart``. Use that command to create a sphinx repo.\n", 382 | "\n", 383 | "The invocation of ``sphinx-quickstart`` below just gives you the defaults for everything without prompting. If you want to see *all* the options, you can run this tool in a terminal inside your ``docs`` directory - by default it prompts you for lots of information, though, and we can't respond to those in the notebook. If you do that, be sure to answer \"yes\" to the question about \"autodoc\" (more on that in 2g)." 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "metadata": { 390 | "collapsed": true 391 | }, 392 | "outputs": [], 393 | "source": [ 394 | "!ls\n", 395 | "# should be empty..." 396 | ] 397 | }, 398 | { 399 | "cell_type": "code", 400 | "execution_count": null, 401 | "metadata": { 402 | "collapsed": false 403 | }, 404 | "outputs": [], 405 | "source": [ 406 | "!sphinx-quickstart -a \"\" -p -v --ext-autodoc -q #complete" 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": { 413 | "collapsed": false 414 | }, 415 | "outputs": [], 416 | "source": [ 417 | "!ls" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "You should see various files have appeared in your ``docs`` directory, most critically a ``conf.py`` and ``index.rst``" 425 | ] 426 | }, 427 | { 428 | "cell_type": "markdown", 429 | "metadata": {}, 430 | "source": [ 431 | "### 2d: Add some content to the ``index.rst`` file" 432 | ] 433 | }, 434 | { 435 | "cell_type": "markdown", 436 | "metadata": {}, 437 | "source": [ 438 | "The ``index.rst`` file is the root for all of your documentation. You'll see it's already been pre-populated with some boilerplate structure. None of this is specific to your package, however. Open this file in an editor, and add some documentation for your package. You'll likely want to put it after the \"Welcome to 's documentation!\" heading, but before the \"Contents:\" (which would have a table of contents if you had any other files).\n", 439 | "\n", 440 | "It's up to you to decide what should go in this front page for your package, although a simple idea is given below.\n", 441 | "\n", 442 | "Another important thing to keep in mind is the format for these packes. The markup language is Restructured Text (reST), which is roughly driven by the philosophy of \"readable as plain text, but with extra bits to make it prettier in doc pages\". Sphinx provides a great [reST primer](http://www.sphinx-doc.org/en/stable/rest.html), which you can reference to build your docs." 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": { 448 | "collapsed": true 449 | }, 450 | "source": [ 451 | "```\n", 452 | "This package does some neat stuff! It's features include:\n", 453 | "\n", 454 | "* A cool thing\n", 455 | "* Another thing\n", 456 | "* Something else that's not quite so useful but I like.\n", 457 | "\n", 458 | "Citing this code\n", 459 | "----------------\n", 460 | "There's no way to cite this code right now. But it would be great if you acknowledge if you use it. Someday I hope to put it up on `Zenodo `_, though...\n", 461 | "```" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "### 2e: Build the docs" 469 | ] 470 | }, 471 | { 472 | "cell_type": "markdown", 473 | "metadata": {}, 474 | "source": [ 475 | "Now try actually *building* the docs with sphinx. The command below should be all that's required. Once it's finished, have a look at the page it generates (start from the ``index.html``)." 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": null, 481 | "metadata": { 482 | "collapsed": false 483 | }, 484 | "outputs": [], 485 | "source": [ 486 | "!make html" 487 | ] 488 | }, 489 | { 490 | "cell_type": "markdown", 491 | "metadata": {}, 492 | "source": [ 493 | "### 2f: Commit the doc files and push them up to github" 494 | ] 495 | }, 496 | { 497 | "cell_type": "markdown", 498 | "metadata": {}, 499 | "source": [ 500 | "If you haven't been doing so, now's a good time to add all the new content of the ``docs`` directory to git and push it up to github. \n", 501 | "\n", 502 | "Important gotcha to watch out for: if you just ``git add docs``, you'll probably get both your docs *and* the stuff in the ``_build`` directory. In general you never want to include generated files in your github repository, because it confuses users (and is nearly-impossible to keep up-to-date, anyway). To keep yourself from getting confused, you can create a ``.gitignore`` file that is aware of all the generated-file directories. That prevents them from getting added by ``git`` accidentally. An example is shown below that should be appropriate for your repo if you've followed the other notebooks. " 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": null, 508 | "metadata": { 509 | "collapsed": true 510 | }, 511 | "outputs": [], 512 | "source": [ 513 | "%cd .. #or whatever you need to do to get back to the base of your repository" 514 | ] 515 | }, 516 | { 517 | "cell_type": "code", 518 | "execution_count": null, 519 | "metadata": { 520 | "collapsed": true 521 | }, 522 | "outputs": [], 523 | "source": [ 524 | "%%file .gitignore\n", 525 | "\n", 526 | "docs/_build/*\n", 527 | "build\n", 528 | "dist" 529 | ] 530 | }, 531 | { 532 | "cell_type": "code", 533 | "execution_count": null, 534 | "metadata": { 535 | "collapsed": true 536 | }, 537 | "outputs": [], 538 | "source": [ 539 | "!git add .gitignore docs\n", 540 | "!git commit -m #complete" 541 | ] 542 | }, 543 | { 544 | "cell_type": "markdown", 545 | "metadata": {}, 546 | "source": [ 547 | "# Problem 3: Building your docs on Read the Docs" 548 | ] 549 | }, 550 | { 551 | "cell_type": "markdown", 552 | "metadata": {}, 553 | "source": [ 554 | "[Read the Docs](http://readthedocs.org/) is an online service that automatically builds documentation for public projects. In this problem, we will set up the repo that you just got sphinx working in to generate its documentation on RTD." 555 | ] 556 | }, 557 | { 558 | "cell_type": "markdown", 559 | "metadata": {}, 560 | "source": [ 561 | "### 3a: Register for a Read the Docs account" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "metadata": {}, 567 | "source": [ 568 | "You'll need an RTD account to be able to do anything with it, of course. Go to the [Read the Docs front page](http://readthedocs.org/), and you should see a \"sign up\" button on the upper right. Use that to create an account." 569 | ] 570 | }, 571 | { 572 | "cell_type": "markdown", 573 | "metadata": {}, 574 | "source": [ 575 | "### 3b: Add your github repo to your RTD account" 576 | ] 577 | }, 578 | { 579 | "cell_type": "markdown", 580 | "metadata": {}, 581 | "source": [ 582 | "RTD can automatically read your github repos if you authorize it to connect to Github, and is often smart enough to do the right thing in one click. You may need to go to your account settings->connected services page and \"Connect to Github\" to get this to work. Once you've done that, go to your RTD dashboard (click on your username in the upper-right) and \"Import a Project\". If the github sync worked, you should see your project.\n", 583 | "\n", 584 | "Alternatively, you can choose the \"Import Manually\" option, where you have to manually provide the name and github repo URL." 585 | ] 586 | }, 587 | { 588 | "cell_type": "markdown", 589 | "metadata": {}, 590 | "source": [ 591 | "### 3c: wait for your docs to build" 592 | ] 593 | }, 594 | { 595 | "cell_type": "markdown", 596 | "metadata": {}, 597 | "source": [ 598 | "RTD takes a bit of time to build. You can watch the progress by going to the project you just created in your dashboard and hitting the \"builds\" button. But all you really can do about it is wait until it finishes.\n", 599 | "\n", 600 | "*Hint: Maybe you'd like some refreshing tea in the meantime? I believe there's some in the back.*" 601 | ] 602 | }, 603 | { 604 | "cell_type": "markdown", 605 | "metadata": {}, 606 | "source": [ 607 | "### 3d: Check that the docs look right" 608 | ] 609 | }, 610 | { 611 | "cell_type": "markdown", 612 | "metadata": {}, 613 | "source": [ 614 | "Once the build finishes, have a look at your doc page and see if it looks like you think it should. RTD will sometimes succeed in building even though something went wrong on a page, so it's usually worth a look at any significant changes from the last build." 615 | ] 616 | }, 617 | { 618 | "cell_type": "markdown", 619 | "metadata": {}, 620 | "source": [ 621 | "### 3e: Set up the web hook to recognize when github changes come in" 622 | ] 623 | }, 624 | { 625 | "cell_type": "markdown", 626 | "metadata": {}, 627 | "source": [ 628 | "RTD is at its most powerful when you have it run automatically. To do this, you need to set up a \"web hook\" that tells RTD when a new commit is sent up to github. You may already be seeing a message in your dashboard warning you that \"This repository doesn't have a valid webhook set up\". If so, try clicking the link there that is supposed to set the hook up automatically. Sometimes this doesn't work, though. In that case you can [follow RTD's instructions for doing it manually](http://docs.readthedocs.io/en/latest/webhooks.html)." 629 | ] 630 | }, 631 | { 632 | "cell_type": "markdown", 633 | "metadata": {}, 634 | "source": [ 635 | "### 3f: Send a commit to github and watch the gears turn" 636 | ] 637 | }, 638 | { 639 | "cell_type": "markdown", 640 | "metadata": {}, 641 | "source": [ 642 | "Now try making a change to your documentation and pushing it up to github. You should see RTD spring into action, and after a bit, your is automatically appears on your RTD docs site!" 643 | ] 644 | }, 645 | { 646 | "cell_type": "markdown", 647 | "metadata": {}, 648 | "source": [ 649 | "# Problem 4: Use some of the features that makes the trouble of Sphinx worthwhile " 650 | ] 651 | }, 652 | { 653 | "cell_type": "markdown", 654 | "metadata": {}, 655 | "source": [ 656 | "This problem extends your documentation to use some of the features that Sphinx offers that makes it powerful for documenting code.\n", 657 | "\n", 658 | "Note that many of these are in the form of [extensions](http://www.sphinx-doc.org/en/stable/extensions.html) - pieces of Sphinx that are not part of the core functionality. Some of these are built-in (and used below) and others that are downloaded separately. Here we won't be using any third-party extensions, but you can [check some out](http://www.sphinx-doc.org/en/stable/ext/thirdparty.html) if you're interested." 659 | ] 660 | }, 661 | { 662 | "cell_type": "markdown", 663 | "metadata": {}, 664 | "source": [ 665 | "### 4a: Add a second document" 666 | ] 667 | }, 668 | { 669 | "cell_type": "markdown", 670 | "metadata": {}, 671 | "source": [ 672 | "One of Sphinx's most important features is that it understands how to link things *across documents*. So lets try creating another page in your docs. The example below will do the trick. After you've made that file (or something similar), you'll need to add the text ``second_doc`` into the ``..toctree::`` section of the ``index.rst``. That will add the new document to your table of contents. Once you've done all this, build the doc again and have a look at your handiwork.\n", 673 | "\n", 674 | "*Hint: like Python, sphinx uses indentation for contextual meaning. So be sure you have consistent indentation in an .rst file, just like in a .py file*" 675 | ] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "execution_count": null, 680 | "metadata": { 681 | "collapsed": false 682 | }, 683 | "outputs": [], 684 | "source": [ 685 | "%%file second_doc.rst\n", 686 | "\n", 687 | "A Document title goes here\n", 688 | "--------------------------\n", 689 | "\n", 690 | "More information. Here's a link back to the index page: :doc:`index`.\n" 691 | ] 692 | }, 693 | { 694 | "cell_type": "markdown", 695 | "metadata": {}, 696 | "source": [ 697 | "### 4b: Render a docstring in sphinx" 698 | ] 699 | }, 700 | { 701 | "cell_type": "markdown", 702 | "metadata": {}, 703 | "source": [ 704 | "Another important feature of Sphinx is the ability to include docstrings in the documentation. The example below shows some of that functionality. Add a file like that, add it to your table of contents (just like for ``second_doc``), and then re-build and examine your docs.\n", 705 | "\n", 706 | "*Hint: you need to have your package accessible from python for this to work. So if you haven't done a ``python setup.py install`` on your package yet, you'll need to do that. If this bothers you, the challenge problem shows how to set up machinery that does not require installing the package to generate the docs.*" 707 | ] 708 | }, 709 | { 710 | "cell_type": "code", 711 | "execution_count": null, 712 | "metadata": { 713 | "collapsed": false 714 | }, 715 | "outputs": [], 716 | "source": [ 717 | "%%file api_docs.rst\n", 718 | "\n", 719 | "API Documentation\n", 720 | "=================\n", 721 | "\n", 722 | "This package has two modules, detailed below. \n", 723 | "\n", 724 | "Also, after doing this, you can mention some of the functions from *anywhere* in the docs by doing :func:`..`.\n", 725 | "\n", 726 | "\n", 727 | "-----------------\n", 728 | "\n", 729 | ".. automodule:: \n", 730 | "\n", 731 | " \n", 732 | "\n", 733 | "-----------------\n", 734 | " \n", 735 | ".. automodule:: .\n", 736 | "\n", 737 | ".. autofunction:: ..\n", 738 | ".. autofunction:: .." 739 | ] 740 | }, 741 | { 742 | "cell_type": "markdown", 743 | "metadata": {}, 744 | "source": [ 745 | "You may not need multiple ``automodule`` calls if uo on how you structured things. Also you *might* be able to avoid the ``autofunction`` directives depending on how you laid things out, by adding ``:members:`` undert he automodule directive (properly indented. The above explicit approach is what Sphinx recommends, but see the challenge problem for a tool that makes this all quite a bit simpler." 746 | ] 747 | }, 748 | { 749 | "cell_type": "markdown", 750 | "metadata": {}, 751 | "source": [ 752 | "### 4c: Make a class inheritance diagram" 753 | ] 754 | }, 755 | { 756 | "cell_type": "markdown", 757 | "metadata": {}, 758 | "source": [ 759 | "As an example of some of the extensions that Sphinx provides, there's a neat tool to generate class \"inheritance diagrams\" - basically these are diagrams that show which classes are subclass of what other classes. For a complex example of a diagram that justifies this feature, see [here](http://docs.astropy.org/en/stable/modeling/index.html#id2).\n", 760 | "\n", 761 | "For this problem you should add the set of (do-nothing) classes shown below. Then, you'll need to add the string ``'sphinx.ext.inheritance_diagram'`` to the ``extensions`` list in your ``conf.py``. Then somewhere in your ``.rst`` file, add ``.. inheritance-diagram:: yourmodule.class_heirarchy``. Then once you build it with sphinx, you should see the diagram at that point in the docs." 762 | ] 763 | }, 764 | { 765 | "cell_type": "code", 766 | "execution_count": null, 767 | "metadata": { 768 | "collapsed": true 769 | }, 770 | "outputs": [], 771 | "source": [ 772 | "%%file /class_heirarchy.py #complete\n", 773 | "\n", 774 | "class A(): # this needs to be \"A(object)\" in py 2.x\n", 775 | " pass\n", 776 | "\n", 777 | "class B(A):\n", 778 | " pass\n", 779 | "\n", 780 | "class C(A):\n", 781 | " pass\n", 782 | "\n", 783 | "class D(B,C):\n", 784 | " pass" 785 | ] 786 | }, 787 | { 788 | "cell_type": "markdown", 789 | "metadata": {}, 790 | "source": [ 791 | "## Challenge Problem: Use the doc tools built into the Astropy Affiliated Package Template" 792 | ] 793 | }, 794 | { 795 | "cell_type": "markdown", 796 | "metadata": {}, 797 | "source": [ 798 | "The Astropy [package template](https://github.com/astropy/package-template) (discussed in a previous challenge problem) contains all the machinery necessary to build documentation just like Astropy. One particularly useful tool is the `automodapi` machinery, which lets you do:\n", 799 | "```\n", 800 | ".. automodapi:: mypackage\n", 801 | "```\n", 802 | "in the sphinx, and if you have docstrings (in the format expected), it will *automatically* generated pages like [this one](http://docs.astropy.org/en/stable/units/index.html#reference-api). \n", 803 | "\n", 804 | "For this problem try to adapt the affiliated package template to your package, and use it to generate an API section. Note that you can also try using `automodapi` on its own (it can be used independently by installing the [stand-alone version](https://github.com/astropy/sphinx-automodapi)), but it's probably easier to just use it from the template." 805 | ] 806 | } 807 | ], 808 | "metadata": { 809 | "anaconda-cloud": {}, 810 | "kernelspec": { 811 | "display_name": "Python [default]", 812 | "language": "python", 813 | "name": "python3" 814 | }, 815 | "language_info": { 816 | "codemirror_mode": { 817 | "name": "ipython", 818 | "version": 3 819 | }, 820 | "file_extension": ".py", 821 | "mimetype": "text/x-python", 822 | "name": "python", 823 | "nbconvert_exporter": "python", 824 | "pygments_lexer": "ipython3", 825 | "version": "3.5.2" 826 | } 827 | }, 828 | "nbformat": 4, 829 | "nbformat_minor": 2 830 | } 831 | -------------------------------------------------------------------------------- /SoftwareRepositories.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "from __future__ import print_function, division, absolute_import" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "# Code Repositories\n", 19 | "\n", 20 | "**Version 0.1**\n", 21 | "\n", 22 | "The notebook contains problems oriented around building a basic Python code repository and making it public via [Github](http://www.github.com). Of course there are other places to put code repositories, with complexity ranging from services comparable to github to simple hosting a git server on your local machine. But this focuses on git and github as a ready-to-use example with plenty of additional resources to be found online. \n", 23 | "\n", 24 | "Note that these problems assum you are using the Anaconda Python distribution. This is particular useful for these problems because it makes it very easy to install testing packages in virtual environments quickly and with little wasted disk space. If you are not using anaconda, you can either use an alternative virtual environment scheme (e.g. in Py 3, the built-in `venv`), or just install pacakges directly into your default python (and hope for the best...).\n", 25 | "\n", 26 | "For `git` interaction, this notebook also uses the `git` command line tools directly. There are a variety of GUI tools that make working with `git` more visually intuitive (e.g. [SourceTree](http://www.sourcetreeapp.com), [gitkraken](http://www.gitkraken.com), or the [github desktop client](https://desktop.github.com)), but this notebook uses the command line tools as the lowest common denominator. You are welcome to try to reproduce the steps with your client, however - feel free to ask your neighbors or instructors if you run into trouble there.\n", 27 | "\n", 28 | "As a final note, this notebook's examples assume you are using a system with a unix-like shell (e.g. macOS, Linux, or Windows with git-bash or the Linux subsystem shell). \n", 29 | "\n", 30 | "* * *\n", 31 | "\n", 32 | "E Tollerud (STScI)" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "# Problem 0: Using Jupyter as a shell " 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "As an initial step before diving into code repositories, it's important to understand how you can use Jupyter as a shell. Most of the steps in this notebook require interaction with the system that's easier done with a shell or editor rather than using Python code in a notebook. While this could be done by opening up a terminal beside this notebook, to keep most of your work in the notebook itself, you can use the capabilities Jupyter + IPython offer for shell interaction." 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "### 0a: Figure out your base shell path and what's in it \n", 54 | "\n", 55 | "The critical trick here is the ``!`` magic in IPython. Anything after a leading ``!`` in IPython gets run by the shell instead of as python code. Run the shell command ``pwd`` and ``ls`` to see where IPython thinks you are on your system, and the contents of the directory.\n", 56 | "\n", 57 | "*hint: Be sure to remove the \"#complete\"s below when you've done so. IPython will interpret that as part of the shell command if you don't*" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": null, 63 | "metadata": { 64 | "collapsed": true 65 | }, 66 | "outputs": [], 67 | "source": [ 68 | "! #complete\n", 69 | "! #complete" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "### 0b: Try a multi-line shell command \n", 77 | "\n", 78 | "IPython magics often support \"cell\" magics by having ``%%`` at the top of a cell. Use that to cd into the directory below this one (\"..\") and then ``ls`` inside that directory.\n", 79 | "\n", 80 | "*Hint: if you need syntax tips, run the ``magic()`` function and look for the `!` or `!!` commands*" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": true 88 | }, 89 | "outputs": [], 90 | "source": [ 91 | "%%sh\n", 92 | "\n", 93 | "#complete" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "### 0c: Create a new directory from Jupyter\n", 101 | "\n", 102 | "While you can do this almost as easily with `os.mkdir` in Python, for this case try to do it using shell magics instead. Make a new directory in the directory you are currently in. Use your system file browser to ensure you were sucessful." 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": { 109 | "collapsed": true 110 | }, 111 | "outputs": [], 112 | "source": [ 113 | "! #complete" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "### 0d: Change directory to your new directory" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": {}, 126 | "source": [ 127 | "One thing about shell commands is that they always start wherever you started your IPython instance. So doing ``cd`` as a shell command only changes things temporarily (i.e. within that shell command). IPython provides a `%cd` magic that makes this change last, though. Use this to `%cd` into the directory you just created, and then use the `pwd` shell command to ensure this cd \"stuck\" (You can also try doing `cd` as a **shell** command to prove to yourself that it's different from the `%cd` magic.)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": { 134 | "collapsed": true 135 | }, 136 | "outputs": [], 137 | "source": [ 138 | "%cd #complete" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "Final note: ``%cd -0`` is a convenient shorthand to switch back to the initial directory." 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "## Problem 1: Creating a bare-bones repo and getting it on Github" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "Here we'll create a simple (public) code repository with a minimal set of content, and publish it in github." 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "### 1a: Create a basic repository locally" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "Start by creating the simplest possible code repository, composed of a single code file. Create a directory (or use the one from *0c*), and place a ``code.py`` file in it, with a bit of Python code of your choosing. (Bonus points for witty or sarcastic code...) You could even use non-Python code if you desired, although Problems 3 & 4 feature Python-specific bits so I wouldn't recommend it.\n", 174 | "\n", 175 | "To make the file from the notebook, the ``%%file `` magic is a convenient way to write the contents of a notebook cell to a file." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": { 182 | "collapsed": true 183 | }, 184 | "outputs": [], 185 | "source": [ 186 | "!mkdir #complete only if you didn't do 0c, or want a different name for your code directory" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "metadata": { 193 | "collapsed": true 194 | }, 195 | "outputs": [], 196 | "source": [ 197 | "%%file /code.py\n", 198 | "\n", 199 | "def do_something():\n", 200 | " # complete\n", 201 | " print(something)# this will make it much easier in future problems to see that something is actually happening" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "If you want to test-run your code:" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": true 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "%run /code.py # complete\n", 220 | "do_something()" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "### 1b: Convert the directory into a git repo" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "Make that code into a git repository by doing ``git init`` in the directory you created, then ``git add`` and ``git commit``." 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": { 241 | "collapsed": true 242 | }, 243 | "outputs": [], 244 | "source": [ 245 | "%cd # complete" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": { 252 | "collapsed": true 253 | }, 254 | "outputs": [], 255 | "source": [ 256 | "!git init" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": { 263 | "collapsed": true 264 | }, 265 | "outputs": [], 266 | "source": [ 267 | "!git add code.py\n", 268 | "!git commit -m #complete" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "metadata": {}, 274 | "source": [ 275 | "### 1c: Create a repository for your code in Github" 276 | ] 277 | }, 278 | { 279 | "cell_type": "markdown", 280 | "metadata": {}, 281 | "source": [ 282 | "Go to [github's web site](http://www.github.com) in your web browser. If you do not have a github account, you'll need to create one (follow the prompts on the github site)." 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": {}, 288 | "source": [ 289 | "Once you've got an account, you'll need to make sure your git client can [authenticate with github](https://help.github.com/categories/authenticating-to-github/). If you're using a GUI, you'll have to figure it out (usually it's pretty easy). On the command line you have two options: \n", 290 | "* The simplest way is to connect to github using HTTPS. This requires no initial setup, but `git` will prompt you for your github username and password every so often.\n", 291 | "* If you find that annoying (I do...), you can set up your system to use SSH to talk to github. Look for the \"SSH and GPG keys\" section of your settings on github's site, or if you're not sure how to work with SSH keys, check out [github's help on the subject](https://help.github.com/articles/connecting-to-github-with-ssh/)." 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": {}, 297 | "source": [ 298 | "Once you've got github set up to talk to your computer, you'll need to create a new repository for the code you created. Hit the \"+\" in the upper-right, create a \"new repository\" and fill out the appropriate details (don't create a README just yet). \n", 299 | "\n", 300 | "To stay sane, I recommend using the same name for your repository as the local directory name you used... But that is *not* a requirement, just a recommendation." 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "metadata": {}, 306 | "source": [ 307 | "Once you've created the repository, connect your local repository to github and push your changes up to github." 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": null, 313 | "metadata": { 314 | "collapsed": true 315 | }, 316 | "outputs": [], 317 | "source": [ 318 | "!git remote add #complete" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": { 325 | "collapsed": true 326 | }, 327 | "outputs": [], 328 | "source": [ 329 | "!git push master -u" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": {}, 335 | "source": [ 336 | "The ``-u`` is a convenience that means from then on you can use just ``git push`` and ``git pull`` to send your code to and from github." 337 | ] 338 | }, 339 | { 340 | "cell_type": "markdown", 341 | "metadata": {}, 342 | "source": [ 343 | "### 1e: Modify the code and send it back up to github" 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "metadata": {}, 349 | "source": [ 350 | "We'll discuss proper documentation later. But for now make sure to add a README to your code repository. Always add a README with basic documentation. Always. Even if only you are going to use this code, trust me, future you will be very happy you did it. \n", 351 | "\n", 352 | "You can just call it `README`, but to get it to get rendered nicely on the github repository, you can call it ``README.md`` and write it using markdown syntax, ``REAMDE.rst`` in ReST (if you know what that is) or various other similar markup languages github understands. If you don't know/care, just use ``README.md``, as that's pretty standard at this point." 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": null, 358 | "metadata": { 359 | "collapsed": true 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "%%file README.md\n", 364 | "\n", 365 | "# complete" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "Don't forget to add and commit via ``git`` and push up to github..." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": { 379 | "collapsed": true 380 | }, 381 | "outputs": [], 382 | "source": [ 383 | "!git #complete" 384 | ] 385 | }, 386 | { 387 | "cell_type": "markdown", 388 | "metadata": {}, 389 | "source": [ 390 | "### 1f: Choose a License" 391 | ] 392 | }, 393 | { 394 | "cell_type": "markdown", 395 | "metadata": {}, 396 | "source": [ 397 | "A bet you didn't expect to be reading legalese today... but it turns out this is important. If you do not explicitly license your code, in most countries (including the US and EU) it is technically **illegal** for anyone to use your code for any purpose other than just looking at it.\n", 398 | "\n", 399 | "(Un?)Fortunately, there are a lot of possible open source licenses out there. Assuming you want an open license, the best resources is to use the [\"Choose a License\" website](http://choosealicense.org). Have a look over the options there and decide which you think is appropriate for your code." 400 | ] 401 | }, 402 | { 403 | "cell_type": "markdown", 404 | "metadata": {}, 405 | "source": [ 406 | "Once you've chosen a License, grab a copy of the license text, and place it in your repository as a file called ``LICENSE`` (or ``LICENSE.md`` or the like). Some licenses might also suggest you place the license text or just a copyright notice in the source code as well, but that's up to you." 407 | ] 408 | }, 409 | { 410 | "cell_type": "markdown", 411 | "metadata": {}, 412 | "source": [ 413 | "Once you've done that, do as we've done before: push all your additions up to github. If you've done it right, github will automatically figure out your license and show it in the upper-right corner of your repo's github page." 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": null, 419 | "metadata": { 420 | "collapsed": true 421 | }, 422 | "outputs": [], 423 | "source": [ 424 | "!git #complete" 425 | ] 426 | }, 427 | { 428 | "cell_type": "markdown", 429 | "metadata": {}, 430 | "source": [ 431 | "## Problem 2: Collaborating with others' repos" 432 | ] 433 | }, 434 | { 435 | "cell_type": "markdown", 436 | "metadata": {}, 437 | "source": [ 438 | "There's not much point in having open source code if no one else can look at it or use it. So now we'll have you try modify your neighbors' project using github's Pull Request feature." 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "### 2a: Get (git?) your neighbor's code repo" 446 | ] 447 | }, 448 | { 449 | "cell_type": "markdown", 450 | "metadata": {}, 451 | "source": [ 452 | "Find someone sitting near you who has gotten through Problem 1. Ask them their github user name and the name of their repository. " 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "Once you've got the name of their repo, navigate to it on github. The URL pattern is always \"https://www.github.com/theirusername/reponame\". Use the github interface to \"fork\" that repo, yielding a \"yourusername/reponame\" repository. Go to that one, take note of the URL needed to clone it (you'll need to grab it from the repo web page, either in \"HTTPS\" or \"SSH\" form, depending on your choice in 1a). Then clone that onto your local machine." 460 | ] 461 | }, 462 | { 463 | "cell_type": "code", 464 | "execution_count": null, 465 | "metadata": { 466 | "collapsed": true 467 | }, 468 | "outputs": [], 469 | "source": [ 470 | "# Don't forget to do this cd or something like it... otherwise you'll clone *inside* your repo\n", 471 | "%cd -0\n", 472 | "\n", 473 | "!git clone #complete \n", 474 | "%cd #complete " 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": {}, 480 | "source": [ 481 | "### 2c: create a branch for your change" 482 | ] 483 | }, 484 | { 485 | "cell_type": "markdown", 486 | "metadata": {}, 487 | "source": [ 488 | "You're going to make some changes to their code, but who knows... maybe they'll spend so long reviewing it that you want to do another. So it's always best to make changes in a specific \"branch\" for that change. So to do this we need to make a github branch." 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": null, 494 | "metadata": { 495 | "collapsed": true 496 | }, 497 | "outputs": [], 498 | "source": [ 499 | "!git branch #complete" 500 | ] 501 | }, 502 | { 503 | "cell_type": "markdown", 504 | "metadata": {}, 505 | "source": [ 506 | "### 2c: modify the code" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": {}, 512 | "source": [ 513 | "Make some change to their code repo. Usually this would be a new feature or a bug fix or documentation clarification or the like... But it's up to you. \n", 514 | "\n", 515 | "Once you've done that, be sure to commit the change locally." 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": { 522 | "collapsed": true 523 | }, 524 | "outputs": [], 525 | "source": [ 526 | "!git add #complete\n", 527 | "!git commit -m \"\"#complete" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "and push it up (to a branch on *your* github fork)." 535 | ] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "execution_count": null, 540 | "metadata": { 541 | "collapsed": true 542 | }, 543 | "outputs": [], 544 | "source": [ 545 | "!git push origin #complete" 546 | ] 547 | }, 548 | { 549 | "cell_type": "markdown", 550 | "metadata": {}, 551 | "source": [ 552 | "### 2d: Issue a pull request" 553 | ] 554 | }, 555 | { 556 | "cell_type": "markdown", 557 | "metadata": {}, 558 | "source": [ 559 | "Now use the github interface to create a new \"pull request\". If you time it right, once you've pushed your new branch up, you'll see a prompt to do this automatically appear on your fork's web page. But if you don't, use the \"branches\" drop-down to navigate to the new branch, and then hit the \"pull request\" button. That should show you an interface that you can use to leave a title and description (in github markdown), and then submit the PR. Go ahead and do this." 560 | ] 561 | }, 562 | { 563 | "cell_type": "markdown", 564 | "metadata": {}, 565 | "source": [ 566 | "### 2e: Have them review the PR" 567 | ] 568 | }, 569 | { 570 | "cell_type": "markdown", 571 | "metadata": {}, 572 | "source": [ 573 | "Tell your neighbor that you've issued the PR. They should be able to go to *their* repo, and see that a new pull request has been created. There they'll review the PR, possibly leaving comments for you to change. If so, go to 2f, but if not, they should hit the \"Merge\" button, and you can jump to 2g." 574 | ] 575 | }, 576 | { 577 | "cell_type": "markdown", 578 | "metadata": {}, 579 | "source": [ 580 | "### 2f: (If necessary) make changes and update the code" 581 | ] 582 | }, 583 | { 584 | "cell_type": "markdown", 585 | "metadata": {}, 586 | "source": [ 587 | "If they left you some comments that require changing prior to merging, you'll need to make those changes in your local copy, commit those changes, and then push them up to your branch on your fork." 588 | ] 589 | }, 590 | { 591 | "cell_type": "code", 592 | "execution_count": null, 593 | "metadata": { 594 | "collapsed": true 595 | }, 596 | "outputs": [], 597 | "source": [ 598 | "!git #complete" 599 | ] 600 | }, 601 | { 602 | "cell_type": "markdown", 603 | "metadata": {}, 604 | "source": [ 605 | "Hopefully they are now satisfied and are willing to hit the merge button." 606 | ] 607 | }, 608 | { 609 | "cell_type": "markdown", 610 | "metadata": {}, 611 | "source": [ 612 | "### 2g: Get the updated version" 613 | ] 614 | }, 615 | { 616 | "cell_type": "markdown", 617 | "metadata": {}, 618 | "source": [ 619 | "Now you should get the up-to-date version from the original owner of the repo, because that way you'll have both your changes and any other changes they might have made in the meantime. To do this you'll need to connect your local copy to your *nieghbor*'s github repo (**not** your fork)." 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": null, 625 | "metadata": { 626 | "collapsed": true 627 | }, 628 | "outputs": [], 629 | "source": [ 630 | "!git remote add #complete\n", 631 | "!git fetch #complete\n", 632 | "!git branch --set-upstream-to=/master master\n", 633 | "!git checkout master\n", 634 | "!git pull" 635 | ] 636 | }, 637 | { 638 | "cell_type": "markdown", 639 | "metadata": {}, 640 | "source": [ 641 | "Now if you look at the local repo, it should include your changes." 642 | ] 643 | }, 644 | { 645 | "cell_type": "markdown", 646 | "metadata": {}, 647 | "source": [ 648 | "*Suggestion* To stay sane, you might change the \"origin\" remote to your username. E.g. ``git remote rename origin ``. To go further, you might even *delete* your fork's `master` branch, so that only your neighbor's `master` exists. That might save you headaches in the long run if you were to ever access this repo again in the future. " 649 | ] 650 | }, 651 | { 652 | "cell_type": "markdown", 653 | "metadata": {}, 654 | "source": [ 655 | "### 2h: Have them reciprocate" 656 | ] 657 | }, 658 | { 659 | "cell_type": "markdown", 660 | "metadata": {}, 661 | "source": [ 662 | "Science (Data or otherwise) and open source code is a social enterprise built on shared effort, mutual respect, and trust. So ask them to issue a PR aginst *your* code, too. The more we can stand on each others' shoulders, the farther we will all see.\n", 663 | "\n", 664 | "*Hint: Ask them nicely. Maybe offer a cookie or something?*" 665 | ] 666 | }, 667 | { 668 | "cell_type": "markdown", 669 | "metadata": {}, 670 | "source": [ 671 | "## Problem 3: Setting up a bare-bones Python Package" 672 | ] 673 | }, 674 | { 675 | "cell_type": "markdown", 676 | "metadata": {}, 677 | "source": [ 678 | "Up to this point we've been working on the simplest possible shared code: a single file with all the content. But for most substantial use cases this isn't going to cut it. After all, Python was designed around the idea of namespaces that let you hide away or show code to make writing, maintaining, and versioning code much easier. But to make use of these, we need to deploy the installational tools that Python provides. This is typically called \"packaging\". In this problem we will take the code you just made it and build it into a proper python package that can be installed and then used anywhere.\n", 679 | "\n", 680 | "For more background and detail (and the most up-to-date recommendations) see the [Python Packaging Guide](https://packaging.python.org/current/)." 681 | ] 682 | }, 683 | { 684 | "cell_type": "markdown", 685 | "metadata": {}, 686 | "source": [ 687 | "### 3a: Set up a Python package structure for your code" 688 | ] 689 | }, 690 | { 691 | "cell_type": "markdown", 692 | "metadata": {}, 693 | "source": [ 694 | "First we adjust the structure of your code from Problem 1 to allow it to live in a package structure rather than as a stand-alone ``.py`` file. All you need to do is create a directory, move the ``code.py`` file into that directory, and add a file (can be empty) called ``__init__.py`` into the directory.\n", 695 | "\n", 696 | "You'll have to pick a name for the package, which is usually the same as the repo name (although that's not strictly required).\n", 697 | "\n", 698 | "*Hint: don't forget to switch back to *your* code repo directory, if you are doing this immediately after Problem 2.*" 699 | ] 700 | }, 701 | { 702 | "cell_type": "code", 703 | "execution_count": null, 704 | "metadata": { 705 | "collapsed": true 706 | }, 707 | "outputs": [], 708 | "source": [ 709 | "!mkdir #complete\n", 710 | "!git mv code.py #complete" 711 | ] 712 | }, 713 | { 714 | "cell_type": "code", 715 | "execution_count": null, 716 | "metadata": { 717 | "collapsed": true 718 | }, 719 | "outputs": [], 720 | "source": [ 721 | "#The \"touch\" unix command simply creates an empty file if there isn't one already. \n", 722 | "#You could also use an editor to create an empty file if you prefer.\n", 723 | "\n", 724 | "!touch /__init__.py#complete" 725 | ] 726 | }, 727 | { 728 | "cell_type": "markdown", 729 | "metadata": {}, 730 | "source": [ 731 | "### 3b: Test your package" 732 | ] 733 | }, 734 | { 735 | "cell_type": "markdown", 736 | "metadata": {}, 737 | "source": [ 738 | "You should now be able to import your package and the code inside it as though it were some installed package like `numpy`, `astropy`, `pandas`, etc." 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": null, 744 | "metadata": { 745 | "collapsed": true 746 | }, 747 | "outputs": [], 748 | "source": [ 749 | "from import code#complete\n", 750 | "\n", 751 | "#if your code.py has a function called `do_something` as in the example above, you can now run it like:\n", 752 | "code.do_something()" 753 | ] 754 | }, 755 | { 756 | "cell_type": "markdown", 757 | "metadata": {}, 758 | "source": [ 759 | "### 3c: Apply packaging tricks" 760 | ] 761 | }, 762 | { 763 | "cell_type": "markdown", 764 | "metadata": {}, 765 | "source": [ 766 | "One of the nice things about packages is that they let you hide the implementation of some part of your code in one place while exposing a \"cleaner\" namespace to the users of your package. To see a (trivial) example, of this, lets pull a function from your ``code.py`` into the base namespace of the package. In the below make the ``__init__.py`` have one line: ``from .code import do_something``. That places the ``do_something()`` function into the package's root namespace." 767 | ] 768 | }, 769 | { 770 | "cell_type": "code", 771 | "execution_count": null, 772 | "metadata": { 773 | "collapsed": true 774 | }, 775 | "outputs": [], 776 | "source": [ 777 | "%%file /__init__.py\n", 778 | "\n", 779 | "#complete" 780 | ] 781 | }, 782 | { 783 | "cell_type": "markdown", 784 | "metadata": {}, 785 | "source": [ 786 | "Now the following should work." 787 | ] 788 | }, 789 | { 790 | "cell_type": "code", 791 | "execution_count": null, 792 | "metadata": { 793 | "collapsed": true 794 | }, 795 | "outputs": [], 796 | "source": [ 797 | "import #complete\n", 798 | ".do_something()#complete" 799 | ] 800 | }, 801 | { 802 | "cell_type": "markdown", 803 | "metadata": {}, 804 | "source": [ 805 | "*BUT* you will probably get an error here. That's because Python is smart about imports: once it's imported a package once it won't re-import it later. Usually that saves time, but here it's a hassle. Fortunately, we can use the ``reload`` function to get around this:" 806 | ] 807 | }, 808 | { 809 | "cell_type": "code", 810 | "execution_count": null, 811 | "metadata": { 812 | "collapsed": false 813 | }, 814 | "outputs": [], 815 | "source": [ 816 | "from importlib import reload #not necessary on Py 2.x, where reload() is built-in\n", 817 | "\n", 818 | "reload()#complete\n", 819 | ".do_something()#complete" 820 | ] 821 | }, 822 | { 823 | "cell_type": "markdown", 824 | "metadata": {}, 825 | "source": [ 826 | "### 3d: Create a setup.py file" 827 | ] 828 | }, 829 | { 830 | "cell_type": "markdown", 831 | "metadata": {}, 832 | "source": [ 833 | "Ok, that's great in a pinch, but what if you want your package to be available from *other* directories? If you open a new terminal somewhere else and try to ``import `` you'll see that it will fail, because Python doesn't know where to find your package. Fortunately, Python (both the language and the larger ecosystem) provide built-in tools to install packages. These are built around creating a ``setup.py`` script that controls installation of a python packages into a shared location on your machine. Essentially all Python packages are installed this way, even if it happens silently behind-the-scenes." 834 | ] 835 | }, 836 | { 837 | "cell_type": "markdown", 838 | "metadata": {}, 839 | "source": [ 840 | "Below is a template bare-bones setup.py file. Fill it in with the relevant details for your package." 841 | ] 842 | }, 843 | { 844 | "cell_type": "code", 845 | "execution_count": null, 846 | "metadata": { 847 | "collapsed": false 848 | }, 849 | "outputs": [], 850 | "source": [ 851 | "%%file /Users/erik/tmp/lsst-test/setup.py\n", 852 | "#!/usr/bin/env python\n", 853 | "\n", 854 | "from distutils.core import setup\n", 855 | "\n", 856 | "setup(name='',\n", 857 | " version='0.1dev',\n", 858 | " description='',\n", 859 | " author='',\n", 860 | " author_email='',\n", 861 | " packages=[''],\n", 862 | " ) #complete" 863 | ] 864 | }, 865 | { 866 | "cell_type": "markdown", 867 | "metadata": {}, 868 | "source": [ 869 | "### 3e: Build the package" 870 | ] 871 | }, 872 | { 873 | "cell_type": "markdown", 874 | "metadata": {}, 875 | "source": [ 876 | "Now you should be able to \"build\" the package. In complex packages this will involve more involved steps like linking against C or FORTRAN code, but for pure-python packages like yours, it simply involves filtering out some extraneous files and copying the essential pieces into a build directory. " 877 | ] 878 | }, 879 | { 880 | "cell_type": "code", 881 | "execution_count": null, 882 | "metadata": { 883 | "collapsed": true 884 | }, 885 | "outputs": [], 886 | "source": [ 887 | "!python setup.py build" 888 | ] 889 | }, 890 | { 891 | "cell_type": "markdown", 892 | "metadata": {}, 893 | "source": [ 894 | "To test that it built sucessfully, the easiest thing to do is cd into the `build/lib.X-Y-Z` directory (\"X-Y-Z\" here is OS and machine-specific). Then you should be able to ``import ``. It's usually best to do this as a completely independent process in python. That way you can be sure you aren't accidentally using an old import as we saw above." 895 | ] 896 | }, 897 | { 898 | "cell_type": "code", 899 | "execution_count": null, 900 | "metadata": { 901 | "collapsed": false, 902 | "scrolled": true 903 | }, 904 | "outputs": [], 905 | "source": [ 906 | "%%sh\n", 907 | "\n", 908 | "cd build/lib.X-Y-Z #complete\n", 909 | "python -c \"import ;.do_something()\" #complete" 910 | ] 911 | }, 912 | { 913 | "cell_type": "markdown", 914 | "metadata": {}, 915 | "source": [ 916 | "### 3f: Install the package" 917 | ] 918 | }, 919 | { 920 | "cell_type": "markdown", 921 | "metadata": {}, 922 | "source": [ 923 | "Alright, now that it looks like it's all working as expected, we can install the package. Note that if we do this willy-nilly, we'll end up with lots of packages, perhaps with the wrong versions, and it's easy to get confused about what's installed (there's no reliable ``uninstall`` command...) So before installing we first create a virtual environment using Anaconda, and install into that. If you don't have anaconda or a similar virtual environment scheme, you can just do ``python setup.py install``. But just remember that this will be difficult to back out (hence the reason for Python environments in the first place!)" 924 | ] 925 | }, 926 | { 927 | "cell_type": "code", 928 | "execution_count": null, 929 | "metadata": { 930 | "collapsed": true 931 | }, 932 | "outputs": [], 933 | "source": [ 934 | "%%sh\n", 935 | "\n", 936 | "conda create -n test_ anaconda #complete\n", 937 | "source activate test_ #complete\n", 938 | "python setup.py install" 939 | ] 940 | }, 941 | { 942 | "cell_type": "markdown", 943 | "metadata": {}, 944 | "source": [ 945 | "Now we can try running the package from *anywhere* (not just the source code directory), as long as we're in the same environment that we installed the package in." 946 | ] 947 | }, 948 | { 949 | "cell_type": "code", 950 | "execution_count": null, 951 | "metadata": { 952 | "collapsed": true 953 | }, 954 | "outputs": [], 955 | "source": [ 956 | "%%sh\n", 957 | "\n", 958 | "cd $HOME\n", 959 | "source activate test_ #complete\n", 960 | "python -c \"import ;.do_something()\" #complete" 961 | ] 962 | }, 963 | { 964 | "cell_type": "markdown", 965 | "metadata": {}, 966 | "source": [ 967 | "### 3g: Update the package on github" 968 | ] 969 | }, 970 | { 971 | "cell_type": "markdown", 972 | "metadata": {}, 973 | "source": [ 974 | "OK, it's now installable. You'll now want to make sure to update the github version to reflect these improvements. You'll need to add and commit all the files. You'll also want to update the README to instruct users that they should use ``python setup.py install`` to install the package." 975 | ] 976 | }, 977 | { 978 | "cell_type": "code", 979 | "execution_count": null, 980 | "metadata": { 981 | "collapsed": true 982 | }, 983 | "outputs": [], 984 | "source": [ 985 | "!git #complete" 986 | ] 987 | }, 988 | { 989 | "cell_type": "markdown", 990 | "metadata": {}, 991 | "source": [ 992 | "## Problem 4: Publishing your package on (fake) PyPI" 993 | ] 994 | }, 995 | { 996 | "cell_type": "markdown", 997 | "metadata": {}, 998 | "source": [ 999 | "Now that your package can be installed by anyone who comes across it on github. But it tends to scare some people that they need to download the source code and know ``git`` to use your code. The Python Package Index (PyPI), combined with the ``pip`` tool (now standard in Python) provides a much simpler way to distribute code. Here we will publish your code to a **testing** version of PyPI." 1000 | ] 1001 | }, 1002 | { 1003 | "cell_type": "markdown", 1004 | "metadata": {}, 1005 | "source": [ 1006 | "### 4a: Create a PyPI account" 1007 | ] 1008 | }, 1009 | { 1010 | "cell_type": "markdown", 1011 | "metadata": {}, 1012 | "source": [ 1013 | "First you'll need an account on PyPI to register new packages. Go to the [testing PyPI](https://testpypi.python.org/pypi), and register. You'll also need to supply your login details in the ``.pypirc`` directory in your home directory as shown below. (If it were the real PyPI you'd want to be more secure and not have your password in plain text. But for the testing server that's not really an issue.)" 1014 | ] 1015 | }, 1016 | { 1017 | "cell_type": "code", 1018 | "execution_count": null, 1019 | "metadata": { 1020 | "collapsed": false 1021 | }, 1022 | "outputs": [], 1023 | "source": [ 1024 | "%%file ~/.pypirc\n", 1025 | "\n", 1026 | "[distutils]\n", 1027 | "index-servers=\n", 1028 | " testpypi\n", 1029 | "\n", 1030 | "[testpypi]\n", 1031 | "repository = https://testpypi.python.org/pypi\n", 1032 | "username = \n", 1033 | "password = " 1034 | ] 1035 | }, 1036 | { 1037 | "cell_type": "markdown", 1038 | "metadata": {}, 1039 | "source": [ 1040 | "### 4b: Register your package on PyPI" 1041 | ] 1042 | }, 1043 | { 1044 | "cell_type": "markdown", 1045 | "metadata": {}, 1046 | "source": [ 1047 | "``distutils`` has built-in functionality for interacting with PyPI. This includes the ability to register your package directly from the command line, automatically filling out the details you provided in your ``setup.py``.\n", 1048 | "\n", 1049 | "*Hint: You'll want to make sure your package version is something you want to release before executing the register command. Released versions can't be duplicates of existing versions, and shouldn't end in \"dev\" or \"b\" or the like.*" 1050 | ] 1051 | }, 1052 | { 1053 | "cell_type": "code", 1054 | "execution_count": null, 1055 | "metadata": { 1056 | "collapsed": true 1057 | }, 1058 | "outputs": [], 1059 | "source": [ 1060 | "!python setup.py register -r https://testpypi.python.org/pypi" 1061 | ] 1062 | }, 1063 | { 1064 | "cell_type": "markdown", 1065 | "metadata": {}, 1066 | "source": [ 1067 | "(The ``-r`` is normally unnecessary, but we need it here because we're using the \"testing\" PyPI)" 1068 | ] 1069 | }, 1070 | { 1071 | "cell_type": "markdown", 1072 | "metadata": {}, 1073 | "source": [ 1074 | "### 4c: Build a \"source\" version of your package" 1075 | ] 1076 | }, 1077 | { 1078 | "cell_type": "markdown", 1079 | "metadata": {}, 1080 | "source": [ 1081 | "Check out the PyPI page for your package. You'll see it now has the info from your setup.py *but* there's no package. Again, `distutils` provides a tool to do this automatically - you take the source distribution that was created, and upload it:" 1082 | ] 1083 | }, 1084 | { 1085 | "cell_type": "code", 1086 | "execution_count": null, 1087 | "metadata": { 1088 | "collapsed": true 1089 | }, 1090 | "outputs": [], 1091 | "source": [ 1092 | "!python setup.py sdist" 1093 | ] 1094 | }, 1095 | { 1096 | "cell_type": "markdown", 1097 | "metadata": {}, 1098 | "source": [ 1099 | "Verify that there is a ``-.tar.gz`` file in the ``dist`` directory. It should have all of the source code necessary for your package." 1100 | ] 1101 | }, 1102 | { 1103 | "cell_type": "markdown", 1104 | "metadata": {}, 1105 | "source": [ 1106 | "### 4d: Upload your package to PyPI" 1107 | ] 1108 | }, 1109 | { 1110 | "cell_type": "markdown", 1111 | "metadata": {}, 1112 | "source": [ 1113 | "Check out the PyPI page for your package. You'll see it now has the info from your setup.py *but* there's no package. Again, `distutils` provides a tool to do this automatically - you take the source distribution that was created, and upload it:" 1114 | ] 1115 | }, 1116 | { 1117 | "cell_type": "code", 1118 | "execution_count": null, 1119 | "metadata": { 1120 | "collapsed": true 1121 | }, 1122 | "outputs": [], 1123 | "source": [ 1124 | "!python setup.py sdist upload -r https://testpypi.python.org/pypi" 1125 | ] 1126 | }, 1127 | { 1128 | "cell_type": "markdown", 1129 | "metadata": {}, 1130 | "source": [ 1131 | "If for some reason this fails (which does happen for unclear reasons on occasion), you can usually just directly upload the ``.tar.gz`` file from the web interface without too much trouble." 1132 | ] 1133 | }, 1134 | { 1135 | "cell_type": "markdown", 1136 | "metadata": {}, 1137 | "source": [ 1138 | "### 4e: Install your package with ``pip``" 1139 | ] 1140 | }, 1141 | { 1142 | "cell_type": "markdown", 1143 | "metadata": {}, 1144 | "source": [ 1145 | "The ``pip`` tool is a convenient way to install packages on PyPI. Again, we use Anaconda to create a testing environment to make sure everything worked correctly.\n", 1146 | "\n", 1147 | "(Normally the ``-i`` wouldn't be necessary - we're using it here only because we're using the \"testing\" PyPI)" 1148 | ] 1149 | }, 1150 | { 1151 | "cell_type": "code", 1152 | "execution_count": null, 1153 | "metadata": { 1154 | "collapsed": true 1155 | }, 1156 | "outputs": [], 1157 | "source": [ 1158 | "%%sh\n", 1159 | "\n", 1160 | "conda create -n test_pypi_ anaconda #complete\n", 1161 | "source activate test_pypi_ #complete\n", 1162 | "pip install -i https://testpypi.python.org/pypi " 1163 | ] 1164 | }, 1165 | { 1166 | "cell_type": "code", 1167 | "execution_count": null, 1168 | "metadata": { 1169 | "collapsed": true 1170 | }, 1171 | "outputs": [], 1172 | "source": [ 1173 | "%%sh\n", 1174 | "\n", 1175 | "cd $HOME\n", 1176 | "source activate test_pypi_ #complete\n", 1177 | "python -c \"import ;.do_something()\" #complete" 1178 | ] 1179 | }, 1180 | { 1181 | "cell_type": "markdown", 1182 | "metadata": {}, 1183 | "source": [ 1184 | "### 4f: have your neighbor try to install your package" 1185 | ] 1186 | }, 1187 | { 1188 | "cell_type": "markdown", 1189 | "metadata": {}, 1190 | "source": [ 1191 | "Ask your neighbor to try to install your package just like you did above. Hopefully they'll get it to work right out of the box.\n", 1192 | "\n", 1193 | "*Hint: Don't forget to be nice to them! Always be nice to your users - it makes them want to be nice to your by contributing improvements or citations... Also, it's just good to be nice, period, dontcha think?*" 1194 | ] 1195 | }, 1196 | { 1197 | "cell_type": "markdown", 1198 | "metadata": {}, 1199 | "source": [ 1200 | "## Challenge Problem: Use the Astropy package template" 1201 | ] 1202 | }, 1203 | { 1204 | "cell_type": "markdown", 1205 | "metadata": {}, 1206 | "source": [ 1207 | "The above is all based on the assumption of a bare-bones package. In practice there's a lot of stuff you can add to a package that's convenient, but requires a variety of boilerplate setup or knowledge about tricks to get it all to work together. Astropy has created a \"package template\" that's meant to reduce the burden of this by providing a package without actual code but lots of \"batteries included\". Then you simply need to fill in your code and *use* the tools, instead of needing to set them up." 1208 | ] 1209 | }, 1210 | { 1211 | "cell_type": "markdown", 1212 | "metadata": {}, 1213 | "source": [ 1214 | "### C1: Use the package template to package your already-made code" 1215 | ] 1216 | }, 1217 | { 1218 | "cell_type": "markdown", 1219 | "metadata": {}, 1220 | "source": [ 1221 | "Try setting up a package using the astropy package template. Go to the [Astropy affiliated package site](http://affiliated.astropy.org), and follow the instructions at the bottom leading you to the package template and how to use it. Populate it with your code from above." 1222 | ] 1223 | }, 1224 | { 1225 | "cell_type": "markdown", 1226 | "metadata": {}, 1227 | "source": [ 1228 | "### C2: Use the package template to compile the built-in Cython examples" 1229 | ] 1230 | }, 1231 | { 1232 | "cell_type": "markdown", 1233 | "metadata": {}, 1234 | "source": [ 1235 | "For an extra challenge (and to see one of the reasons why it's useful), see if you can make the Cython example code work. Cython is a tool that lets you compile Python-like code into C, which can be orders-of-magnitude faster depending on how you design the code. It get be tricky to package correctly, though, and the affiliated package template gets rid of a lot of that pain.\n", 1236 | "\n", 1237 | "The template comes with a simple example of a Cython code. Try to get it compiled and running." 1238 | ] 1239 | }, 1240 | { 1241 | "cell_type": "markdown", 1242 | "metadata": {}, 1243 | "source": [ 1244 | "### C3: Use the package template to write your own Cython code" 1245 | ] 1246 | }, 1247 | { 1248 | "cell_type": "markdown", 1249 | "metadata": {}, 1250 | "source": [ 1251 | "The sky's the limit here. Can you make some Cython code go faster than the `numpy` equivalent? Experiment as you wish, but revel in *not* needing to understand how to invoke Cython. " 1252 | ] 1253 | } 1254 | ], 1255 | "metadata": { 1256 | "anaconda-cloud": {}, 1257 | "kernelspec": { 1258 | "display_name": "Python 3", 1259 | "language": "python", 1260 | "name": "python3" 1261 | }, 1262 | "language_info": { 1263 | "codemirror_mode": { 1264 | "name": "ipython", 1265 | "version": 3 1266 | }, 1267 | "file_extension": ".py", 1268 | "mimetype": "text/x-python", 1269 | "name": "python", 1270 | "nbconvert_exporter": "python", 1271 | "pygments_lexer": "ipython3", 1272 | "version": "3.5.2" 1273 | } 1274 | }, 1275 | "nbformat": 4, 1276 | "nbformat_minor": 2 1277 | } 1278 | --------------------------------------------------------------------------------