├── .gitignore ├── 1-files.ipynb ├── 2-CLI.ipynb ├── 3-filesystem.ipynb ├── 4-processes.ipynb ├── 5-regex.ipynb ├── README.rst ├── base.rst └── build.sh /.gitignore: -------------------------------------------------------------------------------- 1 | *.html 2 | .ipynb_checkpoints 3 | -------------------------------------------------------------------------------- /1-files.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Reading and Writing Files\n", 8 | "I like to start going over administrative scripting with the topic of files because files are fundamental to the way a Unix system thinks about data. If the filesystem were a relational database, files would be the tables, and each line would be like a record. This is obviously not true of every file, but it is a pervasive pattern. To the system, files are not only data stored on disk. They can be anything that can do IO streaming. Devices attached to the computer show up as files, sockets can show up as files and many other things as well.\n", 9 | "\n", 10 | "## Opening Files" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "data": { 20 | "text/plain": [ 21 | "\"Some text concerning foo.\\n\"" 22 | ] 23 | }, 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "output_type": "execute_result" 27 | } 28 | ], 29 | "source": [ 30 | "# write some text into a file\n", 31 | "io = open(\"foo.txt\", \"w\")\n", 32 | "println(io, \"Some text concerning foo.\")\n", 33 | "close(io)\n", 34 | "\n", 35 | "# read the text from a file\n", 36 | "io = open(\"foo.txt\")\n", 37 | "read(io, String)" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "The basics of working with files in Julia are not much different from other programming languages. There is an `open` method which takes then name of the file as a string and a mode argument, and returns an `IO` instance. The modes you'll most often be using are `\"r\"`, `\"w\"` and `\"a\"`, for _read_, _write_ and _append_. These correspond to `<`, `>` and `>>` in the shell. `\"r\"` is the default. There are more mode arguments, and you can read about them in the [documentation for open](https://docs.julialang.org/en/v1/base/io-network/#Base.open). There is a `write` function for writing to files, but `print` and `println` work just as well, and they will convert any non-string arguments to a string representation before sending it to the file. The `write` function, however, can also take an array of bytes (`UInt8`, in Julia parlance) and send those to the specified stream as well.\n", 45 | "\n", 46 | "Likewise, `read` can also return an array of bytes. In fact, this is the default behavior. This is why, in our first example, the second argument, `String` is used. Here is the result if it is omitted:" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 2, 52 | "metadata": {}, 53 | "outputs": [ 54 | { 55 | "name": "stdout", 56 | "output_type": "stream", 57 | "text": [ 58 | "UInt8[0x53, 0x6f, 0x6d, 0x65, 0x20, 0x74, 0x65, 0x78, 0x74, 0x20, 0x63, 0x6f, 0x6e, 0x63, 0x65, 0x72, 0x6e, 0x69, 0x6e, 0x67, 0x20, 0x66, 0x6f, 0x6f, 0x2e, 0x0a]" 59 | ] 60 | } 61 | ], 62 | "source": [ 63 | "# return to beginning of file\n", 64 | "seek(io, 0)\n", 65 | "\n", 66 | "show(read(io))\n", 67 | "close(io)" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "We've also seen the `close` function so far. This cleans up the file descriptor for the system and flushes any data remaining in buffers. However, you normally won't call it yourself.\n", 75 | "\n", 76 | "For one thing, if you want to be lazy, the file descriptor will be cleaned up when the IO object is garbage-collected, so you _can_ ignore it, espeically if you're not opening many files. However, if you are opening a lot of files and you aren't sure when the garbage collector runs, There are other ways to do it. The first one is functionally similar to a context manager in Python, if you're familiar with that, but it looks a little different.\n", 77 | "\n", 78 | "In Python, you'd write:\n", 79 | "\n", 80 | "```python\n", 81 | "with open(\"foo.txt\") as io:\n", 82 | " print(io.read())\n", 83 | "```\n", 84 | "In Julia, it's a [do block](https://docs.julialang.org/en/v1/manual/functions/#Do-Block-Syntax-for-Function-Arguments-1):" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 3, 90 | "metadata": {}, 91 | "outputs": [ 92 | { 93 | "name": "stdout", 94 | "output_type": "stream", 95 | "text": [ 96 | "Some text concerning foo.\n" 97 | ] 98 | } 99 | ], 100 | "source": [ 101 | "open(\"foo.txt\") do io\n", 102 | " print(read(io, String))\n", 103 | "end" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "Do blocks with `open` are useful because they still do the cleanup step even if an exception is thrown inside the block. However, Julia has better shortcuts than that. Many functions that would take a readable `IO` instance as their argument can take the name of the file directly instead." 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": 4, 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "data": { 120 | "text/plain": [ 121 | "\"Some text concerning foo.\\n\"" 122 | ] 123 | }, 124 | "execution_count": 4, 125 | "metadata": {}, 126 | "output_type": "execute_result" 127 | } 128 | ], 129 | "source": [ 130 | "read(\"foo.txt\", String)" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "The do-block version is always the safest if you're doing anything inside the block besides just calling the \"read\" function, but it doesn't make a big difference if you're not planning on using up all your file descriptors. Now let's get rid of that file and get to the next section." 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 5, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "rm(\"foo.txt\")\n", 147 | "# yes, that's really how you remove a file in Julia." 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": {}, 153 | "source": [ 154 | "## Iterating on Files" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 6, 160 | "metadata": { 161 | "scrolled": false 162 | }, 163 | "outputs": [ 164 | { 165 | "name": "stdout", 166 | "output_type": "stream", 167 | "text": [ 168 | "The first line\n", 169 | "Another line\n", 170 | "The last line\n" 171 | ] 172 | } 173 | ], 174 | "source": [ 175 | "# setup a dummy file for this section\n", 176 | "open(\"dummy.txt\", \"w\") do io\n", 177 | " print(io,\n", 178 | " \"\"\"\n", 179 | " The first line\n", 180 | " Another line\n", 181 | " The last line\n", 182 | " \"\"\")\n", 183 | "end\n", 184 | "print(read(\"dummy.txt\", String))\n", 185 | "\n", 186 | "# Note that Julia truncates lines in triple-quote strings so you can still\n", 187 | "# use pretty indentation." 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "Reading a file as a chunck of text is fine, but Unix tools typically need to break files into lines and deal with them one line at a time. In Julia, there are a couple ways to do this. The first is using `readlines` to read the lines in the file into an array. Like `read`, `readlines` can take an IO object or a filename as the first argument." 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 7, 200 | "metadata": {}, 201 | "outputs": [ 202 | { 203 | "name": "stdout", 204 | "output_type": "stream", 205 | "text": [ 206 | "[\"The first line\", \"Another line\", \"The last line\"]" 207 | ] 208 | } 209 | ], 210 | "source": [ 211 | "show(readlines(\"dummy.txt\"))" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "Notice that Julia has very shell-like instincts about this. Trailing newlines are skipped\n", 219 | "automatically, whereas this takes an extra step in any other language, even Perl, whose syntax and semantics are largely based on the shell. If you want to `keep` the trailing newlines, that's also possible, just not default." 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 8, 225 | "metadata": {}, 226 | "outputs": [ 227 | { 228 | "name": "stdout", 229 | "output_type": "stream", 230 | "text": [ 231 | "[\"The first line\\n\", \"Another line\\n\", \"The last line\\n\"]" 232 | ] 233 | } 234 | ], 235 | "source": [ 236 | "show(readlines(\"dummy.txt\", keep=true))" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | "`readline` will be fine for most files, but it's not good if you have to read a large file that can't fit in memory. A more robust way to deal with lines is lazily. That's what `eachline` is for. It takes the same kind arguments as `readlines`, but doesn't load everything into memory at once. You just loop over it and get your lines." 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 9, 249 | "metadata": {}, 250 | "outputs": [ 251 | { 252 | "name": "stdout", 253 | "output_type": "stream", 254 | "text": [ 255 | "\"The first line\"\n", 256 | "\"Another line\"\n", 257 | "\"The last line\"\n" 258 | ] 259 | } 260 | ], 261 | "source": [ 262 | "for line in eachline(\"dummy.txt\")\n", 263 | " println(repr(line))\n", 264 | "end" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "metadata": {}, 270 | "source": [ 271 | "`eachline` will close the file when it reaches the end, but not if iteration is interupted. Therefore, if the loop could be broken and you're worried about running out of file descriptors, it's safer to use a do block." 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 10, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "open(\"dummy.txt\") do io\n", 281 | " for line in eachline(io)\n", 282 | " # do something\n", 283 | " end\n", 284 | "end" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "There are many more functions you can use with `IO` objects, but this covers the common case for administrative scripting. You can read the [documentation](https://docs.julialang.org/en/v1/base/io-network/) if you want more info. We're moving on to [command-line interfaces](2-CLI.ipynb).\n", 292 | "\n", 293 | "_Note: the Julia standard library only deals with ASCII/UTF8 strings. You may wish to check out [JuliaStrings](https://github.com/JuliaStrings) for support for other encodings._" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "rm(\"dummy.txt\")" 303 | ] 304 | } 305 | ], 306 | "metadata": { 307 | "kernelspec": { 308 | "display_name": "Julia 1.1.1", 309 | "language": "julia", 310 | "name": "julia-1.1" 311 | }, 312 | "language_info": { 313 | "file_extension": ".jl", 314 | "mimetype": "application/julia", 315 | "name": "julia", 316 | "version": "1.1.1" 317 | } 318 | }, 319 | "nbformat": 4, 320 | "nbformat_minor": 2 321 | } 322 | -------------------------------------------------------------------------------- /2-CLI.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Command-Line Interfaces\n", 8 | "In order to write flexible, reusable scripts, one must get information from the user and also send it back to them. Hard codeing a bunch of global constants is no way to live!\n", 9 | "\n", 10 | "## `stdin`, `stdout` and `stderr`\n", 11 | "These are your standard streams. The reason I started with a section on files was so I could get to these babies. They are `IO` objects that every script starts with open, and they automatically close at the end. They aren't \"real\" files, but they give the same interfaces as files (besides `seek`). `stdin` is open for reading and both `stdout` and `stderr` are open for writing.\n", 12 | "\n", 13 | "As you probably know, you can send data to the `stdin` of a program by piping the output of another program to it.\n", 14 | "\n", 15 | "```bash\n", 16 | "$ ls / | grep \"b\"\n", 17 | "bin\n", 18 | "boot\n", 19 | "lib\n", 20 | "lib64\n", 21 | "sbin\n", 22 | "```\n", 23 | "\n", 24 | "You can also send data to `stdin` by using file redirection.\n", 25 | "\n", 26 | "```bash\n", 27 | "$ grep \"b\" < some_file\n", 28 | "...\n", 29 | "```\n", 30 | "\n", 31 | "From inside the script, this looks like any other IO object, and you can do whatever you need with the lines.\n", 32 | "\n", 33 | "```julia\n", 34 | "for line in eachline(stdin)\n", 35 | " # do something\n", 36 | "end\n", 37 | "```\n", 38 | "\n", 39 | "However, the creators of Julia know that this is such a common case that both `readlines` and `eachline` default to using stdin. `eachline()` is identical to `eachline(stdin)`\n", 40 | "\n", 41 | "`stdout` is the easy one. You already know how to write to it: the `print` and `println` functions. You can also use `write`, of course, if you need to write binary data.\n", 42 | "\n", 43 | "`stderr` is exactly the same as stdout, but you'd explicitely state that you wanted things to go there: " 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 1, 49 | "metadata": {}, 50 | "outputs": [ 51 | { 52 | "name": "stderr", 53 | "output_type": "stream", 54 | "text": [ 55 | "things are real bad in this script\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "println(stderr, \"things are real bad in this script\")" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "Normally, you want to send data to stdout that is suitable to be used by the stdin of another program (maybe `grep` or `sed`?), and `stderr` is for messages for the user about what's happening in the script (error messages, logging, debugging info). For more advanced logging, Julia provides a [Logging module](https://docs.julialang.org/en/v1/stdlib/Logging/) in the standard library." 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "## CLI Arguments\n", 75 | "\n", 76 | "Another important way to get information from your users is through command line arguments. As in most languages, you get an array of strings. Unlike many languages, the first item in this array is _not_ the name of the program. That's in a global variable called `PROGRAM_FILE`. That can also be useful, but we're just talking about the `ARGS` array for now.\n", 77 | "\n", 78 | "Here is a simple clone of `cp`:\n", 79 | "\n", 80 | "```julia\n", 81 | "# cp.jl\n", 82 | "\n", 83 | "function main()\n", 84 | " dest = ARGS[end]\n", 85 | " srcfiles = ARGS[1:end-1]\n", 86 | " \n", 87 | " if isdir(dest)\n", 88 | " dest = joinpath.(dest, basename.(srcfiles))\n", 89 | " end\n", 90 | "\n", 91 | " cp.(srcfiles, dest)\n", 92 | "end\n", 93 | "\n", 94 | "main()\n", 95 | "```\n", 96 | "\n", 97 | "Which you would use like this :\n", 98 | "\n", 99 | "```bash\n", 100 | "$ julia cp.jl afile otherfile targetdir\n", 101 | "```\n", 102 | "\n", 103 | "We don't really need the main function here, it's just best practice to put everything besides constants inside of a function in Julia for performance reasons (globals are slow unless they are constants), and because it leads to more modular, reusable code.\n", 104 | "\n", 105 | "For more sophisticated argument parsing, two popular third-party modules are [ArgParse.jl](https://juliaobserver.com/packages/ArgParse) and [DocOpt.jl](https://juliaobserver.com/packages/DocOpt), which provide similar interfaces to the Python modules of the same names.\n", 106 | "\n", 107 | "> _Note on vectorizing functions_:\n", 108 | ">\n", 109 | "> If you're new to Julia, you might have trouble understanding a couple of lines:\n", 110 | ">\n", 111 | "> dest = joinpath.(dest, basename.(srcfiles))\n", 112 | ">\n", 113 | "> and\n", 114 | ">\n", 115 | "> cp.(srcfiles, dest)\n", 116 | ">\n", 117 | "> These lines make use of the Julia's [dot syntax for vectorizing functions](https://docs.julialang.org/en/v1/manual/functions/#man-vectorized-1) as an alternative to loops. In the first case, `srcfiles` is a vector of strings. `basename.(srcfiles)` returns an array of the basename of each path in `srcfiles`. It's the same as `[basename(s) for s in srcfiles]`. Each element in this array is then joined with the original `dest` directory for the full path. Because this operation contains nested dot operations, they are all _fused_ into a single loop for greater efficiency.\n", 118 | ">\n", 119 | "> Because `dest` can now either be a vector or a string, `cp.(srcfiles, dest)` can mean two different things: If `dest` is still a string, something like this happens:\n", 120 | ">\n", 121 | "> ```julia\n", 122 | "> for file in srcfiles\n", 123 | "> cp(file, dest)\n", 124 | "> end\n", 125 | "> ```\n", 126 | ">\n", 127 | "> If dest has become a vector, however, it means this:\n", 128 | "> ```julia\n", 129 | "> for (file, target) in zip(srcfiles, dest)\n", 130 | "> cp(file, target)\n", 131 | "> end\n", 132 | "> ```\n", 133 | ">\n", 134 | "> This is handy for our case because, no matter which type `dest` has in the end, the vectorized version will do the right thing!\n", 135 | ">\n", 136 | "> For more on the nuances of vectorizing functions, check out the documentation on [broadcasting](https://docs.julialang.org/en/v1/manual/arrays/#Broadcasting-1)" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "## Environment Variables and Config Files\n", 144 | "\n", 145 | "Another way to get info from your user is from configuration settings. Though it is not the approach I prefer, one popular way to do this is using environment to variables store settings, which are exported in `~/.profile` or some other shell configuration file. In Julia, environment variables are stored in the `ENV` dictionary." 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 2, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "var = \"SHELL\"\n", 158 | "ENV[var] = \"/usr/bin/zsh\"\n", 159 | "var = \"EDITOR\"\n", 160 | "ENV[var] = \"nvim\"\n", 161 | "var = \"USER\"\n", 162 | "ENV[var] = \"ninjaaron\"\n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "for var in (\"SHELL\", \"EDITOR\", \"USER\")\n", 168 | " @show var ENV[var]\n", 169 | "end" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "I personally prefer to use config files over environment variables. [TOML](https://github.com/toml-lang/toml) seems to be what all the cool kids are using these days, and it's also used by Julia's built-in package manager, so that's probably not a bad choice. There is a \"secret\" TOML module in the standard library which is vendor by `Pkg`.\n", 177 | "\n", 178 | "You can get at it this way:" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 2, 184 | "metadata": { 185 | "scrolled": true 186 | }, 187 | "outputs": [ 188 | { 189 | "data": { 190 | "text/plain": [ 191 | "parse (generic function with 3 methods)" 192 | ] 193 | }, 194 | "execution_count": 2, 195 | "metadata": {}, 196 | "output_type": "execute_result" 197 | } 198 | ], 199 | "source": [ 200 | "import Pkg: TOML\n", 201 | "TOML.parse" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "Because it's vendored*, it's probably considered an implementation detail and subject to disappear without notice. I don't know what the deal is. Anyway, the library they vendor can be found [here](https://github.com/wildart/TOML.jl). There are a couple other TOML libraries on juliaobserver.com. There are also a semi-official looking packages under the JuliaIO org on github called [ConfigParser.jl](https://github.com/JuliaIO/ConfParser.jl) That can deal with ini files a few other types. There is also a [JSON.jl](https://github.com/JuliaIO/JSON.jl). I'm pretty against using JSON for config files, but there it is.\n", 209 | "\n", 210 | "Next, were going to look at working with the [filesystem](3-filesystem.ipynb).", 211 | "\n", 212 | "* Note: in Julia 1.6, `TOML` will be part of the standard library, so this will no lonter apply." 213 | ] 214 | } 215 | ], 216 | "metadata": { 217 | "kernelspec": { 218 | "display_name": "Julia 1.1.1", 219 | "language": "julia", 220 | "name": "julia-1.1" 221 | }, 222 | "language_info": { 223 | "file_extension": ".jl", 224 | "mimetype": "application/julia", 225 | "name": "julia", 226 | "version": "1.1.1" 227 | } 228 | }, 229 | "nbformat": 4, 230 | "nbformat_minor": 2 231 | } 232 | -------------------------------------------------------------------------------- /3-filesystem.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "ein.tags": "worksheet-0", 7 | "slideshow": { 8 | "slide_type": "-" 9 | } 10 | }, 11 | "source": [ 12 | "# Filesystem Stuff\n", 13 | "\n", 14 | "## Paths\n", 15 | "\n", 16 | "Julia provides a lot of built-ins for working with paths in a\n", 17 | "cross-platformy way." 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": { 24 | "autoscroll": false, 25 | "ein.hycell": false, 26 | "ein.tags": "worksheet-0", 27 | "slideshow": { 28 | "slide_type": "-" 29 | } 30 | }, 31 | "outputs": [ 32 | { 33 | "name": "stdout", 34 | "output_type": "stream", 35 | "text": [ 36 | "basename(currdir) = \"administrative-scripting-with-julia\"\n", 37 | "dirname(currdir) = \"/home/ninjaaron/doc\"\n" 38 | ] 39 | }, 40 | { 41 | "data": { 42 | "text/plain": [ 43 | "\"/home/ninjaaron/doc/administrative-scripting-with-julia/README.rst\"" 44 | ] 45 | }, 46 | "execution_count": 1, 47 | "metadata": {}, 48 | "output_type": "execute_result" 49 | } 50 | ], 51 | "source": [ 52 | "currdir = pwd()\n", 53 | "@show basename(currdir)\n", 54 | "@show dirname(currdir)\n", 55 | "readme = joinpath(currdir, \"README.rst\")" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": { 61 | "ein.tags": "worksheet-0", 62 | "slideshow": { 63 | "slide_type": "-" 64 | } 65 | }, 66 | "source": [ 67 | "`joinpath` can join an arbitrary number of path elements. For splitting directory componets, there are two options. There is a `splitdir` function, which returns a 2-tuple of the basename and the parent directory:" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 2, 73 | "metadata": { 74 | "autoscroll": false, 75 | "ein.hycell": false, 76 | "ein.tags": "worksheet-0", 77 | "slideshow": { 78 | "slide_type": "-" 79 | } 80 | }, 81 | "outputs": [ 82 | { 83 | "data": { 84 | "text/plain": [ 85 | "(\"/home/ninjaaron/doc\", \"administrative-scripting-with-julia\")" 86 | ] 87 | }, 88 | "execution_count": 2, 89 | "metadata": {}, 90 | "output_type": "execute_result" 91 | } 92 | ], 93 | "source": [ 94 | "splitdir(currdir)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": { 100 | "ein.tags": "worksheet-0", 101 | "slideshow": { 102 | "slide_type": "-" 103 | } 104 | }, 105 | "source": [ 106 | "And in Julia 1.1+, there is a `splitpath` function, which breaks up all the path components:" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 3, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "5-element Array{String,1}:\n", 118 | " \"/\" \n", 119 | " \"home\" \n", 120 | " \"ninjaaron\" \n", 121 | " \"doc\" \n", 122 | " \"administrative-scripting-with-julia\"" 123 | ] 124 | }, 125 | "execution_count": 3, 126 | "metadata": {}, 127 | "output_type": "execute_result" 128 | } 129 | ], 130 | "source": [ 131 | "splitpath(currdir)" 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": {}, 137 | "source": [ 138 | "Here just a few other functions that work on paths:" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 4, 144 | "metadata": { 145 | "autoscroll": false, 146 | "ein.hycell": false, 147 | "ein.tags": "worksheet-0", 148 | "slideshow": { 149 | "slide_type": "-" 150 | } 151 | }, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "splitext(\"README.rst\") = (\"README\", \".rst\")\n", 158 | "isdir(readme) = false\n", 159 | "isfile(readme) = true\n" 160 | ] 161 | }, 162 | { 163 | "data": { 164 | "text/plain": [ 165 | "StatStruct(mode=0o100644, size=42607)" 166 | ] 167 | }, 168 | "execution_count": 4, 169 | "metadata": {}, 170 | "output_type": "execute_result" 171 | } 172 | ], 173 | "source": [ 174 | "@show splitext(\"README.rst\")\n", 175 | "@show isdir(readme)\n", 176 | "@show isfile(readme)\n", 177 | "st = stat(readme)" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": { 183 | "ein.tags": "worksheet-0", 184 | "slideshow": { 185 | "slide_type": "-" 186 | } 187 | }, 188 | "source": [ 189 | "`StatStruct` instances have a lot more attributes than this. They have [all these\n", 190 | "attributes](https://docs.julialang.org/en/v1/base/file/#Base.stat) as\n", 191 | "well. A couple of these attributes, like `mtime` and `ctime` are in\n", 192 | "Unix time, so it might be good mention that you can convert them to a\n", 193 | "human readable representation with the Dates module, which is in the\n", 194 | "standard library. It will be covered more in a later section. (Note\n", 195 | "that this pretty-printed date is just the way it prints. It is a data\n", 196 | "structure, not a string.)" 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": 5, 202 | "metadata": { 203 | "autoscroll": false, 204 | "ein.hycell": false, 205 | "ein.tags": "worksheet-0", 206 | "slideshow": { 207 | "slide_type": "-" 208 | } 209 | }, 210 | "outputs": [ 211 | { 212 | "data": { 213 | "text/plain": [ 214 | "2019-06-18T08:09:37.918" 215 | ] 216 | }, 217 | "execution_count": 5, 218 | "metadata": {}, 219 | "output_type": "execute_result" 220 | } 221 | ], 222 | "source": [ 223 | "import Dates\n", 224 | "Dates.unix2datetime(st.mtime)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": { 230 | "ein.tags": "worksheet-0", 231 | "slideshow": { 232 | "slide_type": "-" 233 | } 234 | }, 235 | "source": [ 236 | "There are many other methods available in Base which have names you should already recognize, which I won't demonstrate now. Names include: `cd`, `rm`, `mkdir`, `mkpath` (like `mkdir -p` in the shell), `symlink`, `chown`, `chmod` (careful to make sure youre mode argument is in octal, `0o644` or whatever), `cp`, `mv`, `touch`, as well as a lot of tests like `isfile`, `isdir`, `islink`, `isfifo`, etc. If you've done shell scripting, you know what they do, and you can [read the docs] if you need more. The one thing function that's missing the familiar shell name is `ls`. That's called `readdir`, after the libc function." 237 | ] 238 | }, 239 | { 240 | "cell_type": "code", 241 | "execution_count": 6, 242 | "metadata": { 243 | "autoscroll": false, 244 | "ein.hycell": false, 245 | "ein.tags": "worksheet-0", 246 | "slideshow": { 247 | "slide_type": "-" 248 | } 249 | }, 250 | "outputs": [ 251 | { 252 | "data": { 253 | "text/plain": [ 254 | "11-element Array{String,1}:\n", 255 | " \".git\" \n", 256 | " \".gitignore\" \n", 257 | " \".ipynb_checkpoints\"\n", 258 | " \"1-files.ipynb\" \n", 259 | " \"2-CLI.ipynb\" \n", 260 | " \"3-filesystem.ipynb\"\n", 261 | " \"4-processes.ipynb\" \n", 262 | " \"5-regex.ipynb\" \n", 263 | " \"README.rst\" \n", 264 | " \"base.rst\" \n", 265 | " \"build.sh\" " 266 | ] 267 | }, 268 | "execution_count": 6, 269 | "metadata": {}, 270 | "output_type": "execute_result" 271 | } 272 | ], 273 | "source": [ 274 | "readdir()" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": { 280 | "ein.tags": "worksheet-0", 281 | "slideshow": { 282 | "slide_type": "-" 283 | } 284 | }, 285 | "source": [ 286 | "There's also a\n", 287 | "[`walkdir`](https://docs.julialang.org/en/v1/base/file/#Base.Filesystem.walkdir)\n", 288 | "which recursively walks the directory and returns tuples of\n", 289 | "`(rootpath, dirs, files)` which is rather handy.\n", 290 | "\n", 291 | "There are a few things Julia still lacks in the filesystem\n", 292 | "department. It doesn't support any kind of file globbing, but that's\n", 293 | "easy enough to handle with regex or plain substring matching." 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 7, 299 | "metadata": { 300 | "autoscroll": false, 301 | "ein.hycell": false, 302 | "ein.tags": "worksheet-0", 303 | "slideshow": { 304 | "slide_type": "-" 305 | } 306 | }, 307 | "outputs": [ 308 | { 309 | "data": { 310 | "text/plain": [ 311 | "6-element Array{String,1}:\n", 312 | " \".ipynb_checkpoints\"\n", 313 | " \"1-files.ipynb\" \n", 314 | " \"2-CLI.ipynb\" \n", 315 | " \"3-filesystem.ipynb\"\n", 316 | " \"4-processes.ipynb\" \n", 317 | " \"5-regex.ipynb\" " 318 | ] 319 | }, 320 | "execution_count": 7, 321 | "metadata": {}, 322 | "output_type": "execute_result" 323 | } 324 | ], 325 | "source": [ 326 | "[path for path in readdir() if occursin(\"ipynb\", path)]" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": 8, 332 | "metadata": { 333 | "autoscroll": false, 334 | "ein.hycell": false, 335 | "ein.tags": "worksheet-0", 336 | "slideshow": { 337 | "slide_type": "-" 338 | } 339 | }, 340 | "outputs": [ 341 | { 342 | "data": { 343 | "text/plain": [ 344 | "8-element Array{String,1}:\n", 345 | " \"1-files.ipynb\" \n", 346 | " \"2-CLI.ipynb\" \n", 347 | " \"3-filesystem.ipynb\"\n", 348 | " \"4-processes.ipynb\" \n", 349 | " \"5-regex.ipynb\" \n", 350 | " \"README.rst\" \n", 351 | " \"base.rst\" \n", 352 | " \"build.sh\" " 353 | ] 354 | }, 355 | "execution_count": 8, 356 | "metadata": {}, 357 | "output_type": "execute_result" 358 | } 359 | ], 360 | "source": [ 361 | "# or\n", 362 | "filter(p -> !startswith(p, \".\"), readdir())" 363 | ] 364 | }, 365 | { 366 | "cell_type": "markdown", 367 | "metadata": { 368 | "ein.tags": "worksheet-0", 369 | "slideshow": { 370 | "slide_type": "-" 371 | } 372 | }, 373 | "source": [ 374 | "It also weirdly lacks a function for making hard links. Bah. I guess\n", 375 | "that's what the [C interface](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/)\n", 376 | "is for. (I'm both thumping my chest and groaning inside as I say that,\n", 377 | "but at least it is crazy easy to call C from Julia and is as efficient\n", 378 | "as native calls)" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": 9, 384 | "metadata": { 385 | "autoscroll": false, 386 | "ein.hycell": false, 387 | "ein.tags": "worksheet-0", 388 | "slideshow": { 389 | "slide_type": "-" 390 | } 391 | }, 392 | "outputs": [ 393 | { 394 | "name": "stdout", 395 | "output_type": "stream", 396 | "text": [ 397 | "st.nlink = 2\n" 398 | ] 399 | } 400 | ], 401 | "source": [ 402 | "function hardlink(oldpath, newpath)\n", 403 | " # calling: int link(char *oldpath, char *newpath)\n", 404 | " ret_code = ccall(:link, Int32, (Cstring, Cstring), oldpath, newpath)\n", 405 | " # handle system errors from libc:\n", 406 | " systemerror(\"linking $oldpath -> $newpath\", ret_code != 0)\n", 407 | "end\n", 408 | "\n", 409 | "hardlink(\"README.rst\", \"foo.txt\")\n", 410 | "st = stat(\"foo.txt\")\n", 411 | "@show st.nlink\n", 412 | "rm(\"foo.txt\")" 413 | ] 414 | }, 415 | { 416 | "cell_type": "markdown", 417 | "metadata": { 418 | "ein.tags": "worksheet-0", 419 | "slideshow": { 420 | "slide_type": "-" 421 | } 422 | }, 423 | "source": [ 424 | "Course, using `ccall` sort of depends on, you know, knowing enough C\n", 425 | "to read and understand C function declarations for simple things, and\n", 426 | "it involves pointers and memory allocation crap if you want to do\n", 427 | "something more serious. It's C. What did you expect?\n", 428 | "\n", 429 | "Julia also lacks Python's easy, built-in support for compression and\n", 430 | "archive formats, though third-party packages do exist for GZip and Zip\n", 431 | "archives. Maybe I should work on an archiving library. Hm. Maybe not.\n", 432 | "\n", 433 | "Anyhow, there's more than one way to skin that cat. One distinctive\n", 434 | "feature of Julia is that is very clear after you use it a little, but\n", 435 | "it's hard to point to any one thing, is that it wants to make it easy\n", 436 | "to bootstrap whatever functionality you need into the language. The\n", 437 | "`ccall` API is part of that. It is used liberally in the\n", 438 | "implementation of OS interfaces in the standard library, as well as some of the mathematical\n", 439 | "libraries (`ccall` works on any dynamic libraries, not just those written in C). Though they aren't shipped with Julia, the community also maintains PyCall.jl and RCall.jl, which\n", 440 | "allow \"native\" calls into those runtimes for wrapping their\n", 441 | "libraries. Macros are different example of the same thing. Language\n", 442 | "missing a feature? Alter the semantics with a macro. Yet another\n", 443 | "example of this \"bootstrap-ability\" of Julia is the ease with which it\n", 444 | "allows the programmer to orchestrate the use of external processes.\n", 445 | "\n", 446 | "To take the example of the above `hardlink` function, If programming\n", 447 | "in C ain't your bag, Julia has really great support for running\n", 448 | "external processes, so it is also possible (but rather slower) to\n", 449 | "simply do:" 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": 10, 455 | "metadata": { 456 | "autoscroll": false, 457 | "ein.hycell": false, 458 | "ein.tags": "worksheet-1", 459 | "slideshow": { 460 | "slide_type": "-" 461 | } 462 | }, 463 | "outputs": [ 464 | { 465 | "data": { 466 | "text/plain": [ 467 | "hardlink (generic function with 1 method)" 468 | ] 469 | }, 470 | "execution_count": 10, 471 | "metadata": {}, 472 | "output_type": "execute_result" 473 | } 474 | ], 475 | "source": [ 476 | "hardlink(oldpath, newpath) = run(`link $oldpath $newpath`)" 477 | ] 478 | }, 479 | { 480 | "cell_type": "markdown", 481 | "metadata": {}, 482 | "source": [ 483 | "So, let's transition to the section on [commands and processes](4-processes.ipynb)." 484 | ] 485 | } 486 | ], 487 | "metadata": { 488 | "kernelspec": { 489 | "display_name": "Julia 1.0.2", 490 | "language": "julia", 491 | "name": "julia-1.0" 492 | }, 493 | "language_info": { 494 | "file_extension": ".jl", 495 | "mimetype": "application/julia", 496 | "name": "julia", 497 | "version": "1.1.1" 498 | }, 499 | "name": "filesystem.ipynb" 500 | }, 501 | "nbformat": 4, 502 | "nbformat_minor": 2 503 | } 504 | -------------------------------------------------------------------------------- /4-processes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Running Processes" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "data": { 17 | "text/plain": [ 18 | "`\u001b[4mlink\u001b[24m \u001b[4mREADME.rst\u001b[24m \u001b[4mfoo.txt\u001b[24m`" 19 | ] 20 | }, 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "output_type": "execute_result" 24 | } 25 | ], 26 | "source": [ 27 | "command = `link README.rst foo.txt`" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "What is it? It's glorious!" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 2, 40 | "metadata": {}, 41 | "outputs": [ 42 | { 43 | "data": { 44 | "text/plain": [ 45 | "Cmd" 46 | ] 47 | }, 48 | "execution_count": 2, 49 | "metadata": {}, 50 | "output_type": "execute_result" 51 | } 52 | ], 53 | "source": [ 54 | "typeof(command)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "It's Julia's `Cmd` literal, and it's a thing of beauty. What has it done? Nothing.\n", 62 | "\n", 63 | "Command literals, though they look the same, are not like process substitution in Perl, Ruby or Bash in that they execute a command and return the output as a string. They are something so much better. They create a `Cmd` instance which contains the arguments and some other information, and that object can be sent to various different functions to be executed in different ways. [The documentation](https://docs.julialang.org/en/v1/manual/running-external-programs/) gives a good description of how to use these little marvels, so I'll just cover a few simple cases here and explain what makes these so great.\n", 64 | "\n", 65 | "The simplest thing you can do, and the thing you need most often, is simply to run the command." 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 3, 71 | "metadata": {}, 72 | "outputs": [ 73 | { 74 | "data": { 75 | "text/plain": [ 76 | "Process(`\u001b[4mlink\u001b[24m \u001b[4mREADME.rst\u001b[24m \u001b[4mfoo.txt\u001b[24m`, ProcessExited(0))" 77 | ] 78 | }, 79 | "execution_count": 3, 80 | "metadata": {}, 81 | "output_type": "execute_result" 82 | } 83 | ], 84 | "source": [ 85 | "filename = \"foo.txt\"\n", 86 | "run(`link README.rst $filename`)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 4, 92 | "metadata": {}, 93 | "outputs": [ 94 | { 95 | "name": "stdout", 96 | "output_type": "stream", 97 | "text": [ 98 | "-rw-r--r-- 2 ninjaaron users 42K Jun 18 10:09 foo.txt\n" 99 | ] 100 | }, 101 | { 102 | "data": { 103 | "text/plain": [ 104 | "Process(`\u001b[4mls\u001b[24m \u001b[4m-lh\u001b[24m \u001b[4mfoo.txt\u001b[24m`, ProcessExited(0))" 105 | ] 106 | }, 107 | "execution_count": 4, 108 | "metadata": {}, 109 | "output_type": "execute_result" 110 | } 111 | ], 112 | "source": [ 113 | "run(`ls -lh $filename`)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "What actually happened here? Obviously we ran the `link` executable and the `ls` executable on the local system, but maybe not in the way you'd expect if comming from other languages, where the default methods for running commands _generally_ create a subshell and execute your input there. In Julia, commands never get a shell. As far as I know, the only way to give a command a shell would be to do so explicitely, something like `bash -c echo \"my injection vulnerability\"`, but you really don't need a shell, so that's fine. What Julia's command literals do is pass the string to a parser for a shell-like mini-language, which converts the command into a vector of strings--which will ultimately be handed to one of the OS's `exec` familiy of functions--on \\*nix. I don't know how these things happen on Windows.\n", 121 | "\n", 122 | "The result is that running commands in Julia is safe and secure by default because the shell never has the chance to do horrible things with user input.\n", 123 | "\n", 124 | "What's more, while Julia's shell mini-language resembles POSIX syntax on a surface level, it is actually much saner and safer. It's very easy to convert a working Bash script to Julia, but the result will usually be safer in the end, which you can't say in most languages! For example, in a Bash script, you should not really do this:\n", 125 | "\n", 126 | "```bash\n", 127 | "link README.rst $filename\n", 128 | "```\n", 129 | "\n", 130 | "You should always put double quotes around the variable, because otherwise it will be expanded into multiple arguments on whitespace. However, in Julia, interpolated strings are never expanded in this way. Some things are expanded, however: iterables" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 5, 136 | "metadata": {}, 137 | "outputs": [ 138 | { 139 | "data": { 140 | "text/plain": [ 141 | "`\u001b[4mecho\u001b[24m \u001b[4m1\u001b[24m \u001b[4m2\u001b[24m \u001b[4m3\u001b[24m \u001b[4m4\u001b[24m \u001b[4m5\u001b[24m \u001b[4m6\u001b[24m \u001b[4m7\u001b[24m \u001b[4m8\u001b[24m \u001b[4m9\u001b[24m \u001b[4m10\u001b[24m`" 142 | ] 143 | }, 144 | "execution_count": 5, 145 | "metadata": {}, 146 | "output_type": "execute_result" 147 | } 148 | ], 149 | "source": [ 150 | "`echo $(1:10)`" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "As you can see, this is expanded by Julia before the command is even run. These can also combine with other elements to make Cartesian products in a way similar to how brace expansion works in the shell:" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 6, 163 | "metadata": {}, 164 | "outputs": [ 165 | { 166 | "data": { 167 | "text/plain": [ 168 | "`\u001b[4m./file1\u001b[24m \u001b[4m./file2\u001b[24m \u001b[4m./file3\u001b[24m \u001b[4m./file4\u001b[24m \u001b[4m./file5\u001b[24m \u001b[4m./file6\u001b[24m \u001b[4m./file7\u001b[24m \u001b[4m./file8\u001b[24m \u001b[4m./file9\u001b[24m \u001b[4m./file10\u001b[24m`" 169 | ] 170 | }, 171 | "execution_count": 6, 172 | "metadata": {}, 173 | "output_type": "execute_result" 174 | } 175 | ], 176 | "source": [ 177 | "`./file$(1:10)`" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": 7, 183 | "metadata": {}, 184 | "outputs": [ 185 | { 186 | "data": { 187 | "text/plain": [ 188 | "`\u001b[4mfoo1\u001b[24m \u001b[4mfoo2\u001b[24m \u001b[4mfoo3\u001b[24m \u001b[4mbar1\u001b[24m \u001b[4mbar2\u001b[24m \u001b[4mbar3\u001b[24m \u001b[4mbaz1\u001b[24m \u001b[4mbaz2\u001b[24m \u001b[4mbaz3\u001b[24m`" 189 | ] 190 | }, 191 | "execution_count": 7, 192 | "metadata": {}, 193 | "output_type": "execute_result" 194 | } 195 | ], 196 | "source": [ 197 | "words = [\"foo\", \"bar\", \"baz\"]\n", 198 | "numbers = 1:3\n", 199 | "`$words$numbers`" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "As seen in some of these examples, using a `$()` inside of a command doesn't do process substitution as in the shell, it does, uh, \"Julia substitution,\" as it would in a Julia string--aside from the expansion of iterables.\n", 207 | "\n", 208 | "Julia has some other nice, logical features around commands. For example, when a process exits with a non-zero exit code in Bash, the script just tries to keep going and do who-know's-what. Same goes for starting processes in most other languages. That's just silly, and Julia knows it." 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 8, 214 | "metadata": {}, 215 | "outputs": [ 216 | { 217 | "name": "stderr", 218 | "output_type": "stream", 219 | "text": [ 220 | "link: cannot create link 'foo.txt' to 'README.rst': File exists\n" 221 | ] 222 | }, 223 | { 224 | "ename": "ErrorException", 225 | "evalue": "failed process: Process(`link README.rst foo.txt`, ProcessExited(1)) [1]", 226 | "output_type": "error", 227 | "traceback": [ 228 | "failed process: Process(`link README.rst foo.txt`, ProcessExited(1)) [1]", 229 | "", 230 | "Stacktrace:", 231 | " [1] run(::Cmd) at ./process.jl:724", 232 | " [2] top-level scope at In[8]:1" 233 | ] 234 | } 235 | ], 236 | "source": [ 237 | "run(`link README.rst $filename`)" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "That's right: Finished processes raise an error when there is a non-zero exit status in the general case. Why doesn't every other language do this by default? No idea. There are cases where you don't want this, like if you're using `grep`, for example. `grep` exits 1 if no matches were found, which isn't exactly an error.\n", 245 | "\n", 246 | "You can avoid it by passing additional arguments to the `Cmd` constructor." 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 9, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "name": "stderr", 256 | "output_type": "stream", 257 | "text": [ 258 | "link: cannot create link 'foo.txt' to 'README.rst': File exists\n" 259 | ] 260 | }, 261 | { 262 | "data": { 263 | "text/plain": [ 264 | "Process(`\u001b[4mlink\u001b[24m \u001b[4mREADME.rst\u001b[24m \u001b[4mfoo.txt\u001b[24m`, ProcessExited(1))" 265 | ] 266 | }, 267 | "execution_count": 9, 268 | "metadata": {}, 269 | "output_type": "execute_result" 270 | } 271 | ], 272 | "source": [ 273 | "run(Cmd(`link README.rst $filename`, ignorestatus=true))" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "So the error message still goes to stderr, because it's from the process itself, but it prevents a non-zero exit status from throwing an error.\n", 281 | "\n", 282 | "Another nice feature which shows that the Julia developers \"get it\" when it comes to processes, is that basically any function that can be applied to a file can be applied to a command literal." 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": 10, 288 | "metadata": {}, 289 | "outputs": [ 290 | { 291 | "data": { 292 | "text/plain": [ 293 | "9-element Array{String,1}:\n", 294 | " \"1-files.ipynb\" \n", 295 | " \"2-CLI.ipynb\" \n", 296 | " \"3-filesystem.ipynb\"\n", 297 | " \"4-processes.ipynb\" \n", 298 | " \"5-regex.ipynb\" \n", 299 | " \"base.rst\" \n", 300 | " \"build.sh\" \n", 301 | " \"foo.txt\" \n", 302 | " \"README.rst\" " 303 | ] 304 | }, 305 | "execution_count": 10, 306 | "metadata": {}, 307 | "output_type": "execute_result" 308 | } 309 | ], 310 | "source": [ 311 | "readlines(`ls`)" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 11, 317 | "metadata": {}, 318 | "outputs": [ 319 | { 320 | "name": "stdout", 321 | "output_type": "stream", 322 | "text": [ 323 | "FOO\n" 324 | ] 325 | } 326 | ], 327 | "source": [ 328 | "open(`tr a-z A-Z`, \"w\", stdout) do io\n", 329 | " println(io, \"foo\")\n", 330 | "end" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": {}, 336 | "source": [ 337 | "Julia also supports pipelines, of course, but not with the pipe operator, `|`. Instead, one uses the `pipeline` function, which is also useful if you want to do more complex IO things. Rather than cover all this here, I will once again direct the reader to the [documentation](https://docs.julialang.org/en/v1/manual/running-external-programs/#Pipelines-1), where it is all laied out very clearly.\n", 338 | "\n", 339 | "Word of warning to the reader: while it's wonderful that it's so easy and safe to work with processes in Julia, keep in mind that starting a process is very expensive for the OS relative to executing code in the current process. Particularly inside of hot loops, You should look for a way to do what you need directly in Julia first, and only resort to calling process when there is no apparent way to do the needful natively. It is so much slower.\n", 340 | "\n", 341 | "One place where someone with a background writing shell scripts in other languages, but not as much experience in other languages might be tempted to use for string filtering utilities in coreutils--sed, grep, awk, etc. This would usually be a no-no, so the next section will provide a quick introduction about how to do the kinds of things you frequently do with those tools using Julia's [regular expressions](5-regex.ipynb)." 342 | ] 343 | }, 344 | { 345 | "cell_type": "code", 346 | "execution_count": 12, 347 | "metadata": {}, 348 | "outputs": [], 349 | "source": [ 350 | "rm(\"foo.txt\")" 351 | ] 352 | } 353 | ], 354 | "metadata": { 355 | "kernelspec": { 356 | "display_name": "Julia 1.1.1", 357 | "language": "julia", 358 | "name": "julia-1.1" 359 | }, 360 | "language_info": { 361 | "file_extension": ".jl", 362 | "mimetype": "application/julia", 363 | "name": "julia", 364 | "version": "1.1.1" 365 | } 366 | }, 367 | "nbformat": 4, 368 | "nbformat_minor": 2 369 | } 370 | -------------------------------------------------------------------------------- /5-regex.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# String Filtering and Manipulation (with regex and otherwise)\n", 8 | "\n", 9 | "This section is primarily for those used to writing shell scripts who want to do similar kinds of string jobs as one does with coreutils. If you're used to string manipulation in other programming languages, Julia will not be dramatically different, but you may still want to read a little just to see how the basics look.\n", 10 | "\n", 11 | "Note on regex dialects that I originally wrote for the the [Python tutorial](https://github.com/ninjaaron/replacing-bash-scripting-with-python):\n", 12 | "\n", 13 | "> One thing to be aware of is that Python's regex is more like PCRE (Perl-style -- also similar to Ruby, JavaScript, etc.) than BRE or ERE that most shell utilities support. If you mostly do sed or grep without the -E option, you may want to look at the rules for Python regex (BRE is the regex dialect you know). If you're used to writing regex for awk or egrep (ERE), Python regex is more or less a superset of what you know. You still may want to look at the documentation for some of the more advanced things you can do. If you know regex from either vi/Vim or Emacs, they each use their own dialect of regex, but they are supersets of BRE, and Python's regex will have some major differences.\n", 14 | "\n", 15 | "This is also true for Julia, except that Julia's regex isn't \"like\" PCRE, it uses the actual PCRE library. The canonical resource on this dialect of regex is the [Perl regex manpage](http://perldoc.perl.org/perlre.html), but note that, while Perl generally places regexes between slashes (`/a regex/`), Julia regex literals look like this: `r\"a regex\"`. Also be aware that julia doesn't have the same kinds of operators for dealing with regexes, like =~, s, m, etc. Instead, normal functions are used with regex literals, as in JavaScript and Ruby.\n", 16 | "\n", 17 | "## how to `grep`\n", 18 | "\n", 19 | "If you want to check if a substring occurs in a string, julia has a function called `occursin` for that." 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "data": { 29 | "text/plain": [ 30 | "true" 31 | ] 32 | }, 33 | "execution_count": 1, 34 | "metadata": {}, 35 | "output_type": "execute_result" 36 | } 37 | ], 38 | "source": [ 39 | "occursin(\"substring\", \"string containing substring\")" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "As with most functions dealing with substrings in Julia, `occursin` can also be used with regular expressions." 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 2, 52 | "metadata": {}, 53 | "outputs": [ 54 | { 55 | "data": { 56 | "text/plain": [ 57 | "true" 58 | ] 59 | }, 60 | "execution_count": 2, 61 | "metadata": {}, 62 | "output_type": "execute_result" 63 | } 64 | ], 65 | "source": [ 66 | "occursin(r\"\\w the pattern\", \"string containing the pattern\")" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "So let's get a long array of strings to grep." 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 3, 79 | "metadata": {}, 80 | "outputs": [ 81 | { 82 | "data": { 83 | "text/plain": [ 84 | "226-element Array{SubString{String},1}:\n", 85 | " \".\" \n", 86 | " \"./3-filesystem.ipynb\" \n", 87 | " \"./base.rst\" \n", 88 | " \"./.git\" \n", 89 | " \"./.git/config\" \n", 90 | " \"./.git/packed-refs\" \n", 91 | " \"./.git/index\" \n", 92 | " \"./.git/logs\" \n", 93 | " \"./.git/logs/HEAD\" \n", 94 | " \"./.git/logs/refs\" \n", 95 | " \"./.git/logs/refs/remotes\" \n", 96 | " \"./.git/logs/refs/remotes/origin\" \n", 97 | " \"./.git/logs/refs/remotes/origin/HEAD\" \n", 98 | " ⋮ \n", 99 | " \"./.ipynb_checkpoints/5-regex-checkpoint.ipynb\" \n", 100 | " \"./.ipynb_checkpoints/2-CLI-checkpoint.ipynb\" \n", 101 | " \"./.ipynb_checkpoints/4-processes-checkpoint.ipynb\" \n", 102 | " \"./.ipynb_checkpoints/3-filesystem-checkpoint.ipynb\"\n", 103 | " \"./.ipynb_checkpoints/1-files-checkpoint.ipynb\" \n", 104 | " \"./5-regex.ipynb\" \n", 105 | " \"./1-files.ipynb\" \n", 106 | " \"./README.rst\" \n", 107 | " \"./2-CLI.ipynb\" \n", 108 | " \"./build.sh\" \n", 109 | " \"./4-processes.ipynb\" \n", 110 | " \"\" " 111 | ] 112 | }, 113 | "execution_count": 3, 114 | "metadata": {}, 115 | "output_type": "execute_result" 116 | } 117 | ], 118 | "source": [ 119 | "filenames = split(read(`find -print0`, String), '\\0')" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "> Note 1: You wouldn't normally use `find` in a Julia script. You'd be more likely to use the `walkdir` function, documented [here](https://docs.julialang.org/en/v1/base/file/#Base.Filesystem.walkdir).\n", 127 | ">\n", 128 | "> Note 2: the reason this is isn't just ```readlines(`find`)``` is that POSIX filenames can contain newlines. Isn't that horrible? `-print0` uses the null byte to separate characters, rather than a newline to avoid exactly this problem, since it's the only byte that is forbidden in a filename.\n", 129 | "\n", 130 | "So, let's try to match some git hashes that have four adjecent letters." 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 4, 136 | "metadata": {}, 137 | "outputs": [ 138 | { 139 | "data": { 140 | "text/plain": [ 141 | "10-element Array{SubString{String},1}:\n", 142 | " \"./.git/objects/68/0c692e7095ecab805f649885ccc0e32c63ae1b\"\n", 143 | " \"./.git/objects/9c/f63bd3bbeea6c067d1e08f762acce5ac8adfe0\"\n", 144 | " \"./.git/objects/1c/3c450edb480db60f6c949adf0b5dccdaebfc64\"\n", 145 | " \"./.git/objects/92/1cab47e3aafe6adab84ffdd9b06a16c34fa2e0\"\n", 146 | " \"./.git/objects/5e/8eeb92ced0763ccaa1a094c2c75e812ba090b9\"\n", 147 | " \"./.git/objects/b8/2403b2c7d4f507c4debdb47b46fb3754a3085c\"\n", 148 | " \"./.git/objects/b8/54273918b2f809ceb8a2d567665b8bdabb7d9d\"\n", 149 | " \"./.git/objects/bf/2627a295497343ecb3dbec23a853b0ebaa8c4f\"\n", 150 | " \"./.git/objects/33/c9b993c55a75a2424acae6f1bcc5dcbf1f1ef7\"\n", 151 | " \"./.git/objects/d0/0db2ebda0b296f6f08e54ad06f3102e7abdec6\"" 152 | ] 153 | }, 154 | "execution_count": 4, 155 | "metadata": {}, 156 | "output_type": "execute_result" 157 | } 158 | ], 159 | "source": [ 160 | "filter(s->occursin(r\".git/objects/.*[abcde]{4}\", s), filenames)" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 5, 166 | "metadata": {}, 167 | "outputs": [ 168 | { 169 | "data": { 170 | "text/plain": [ 171 | "10-element Array{SubString{String},1}:\n", 172 | " \"./.git/objects/68/0c692e7095ecab805f649885ccc0e32c63ae1b\"\n", 173 | " \"./.git/objects/9c/f63bd3bbeea6c067d1e08f762acce5ac8adfe0\"\n", 174 | " \"./.git/objects/1c/3c450edb480db60f6c949adf0b5dccdaebfc64\"\n", 175 | " \"./.git/objects/92/1cab47e3aafe6adab84ffdd9b06a16c34fa2e0\"\n", 176 | " \"./.git/objects/5e/8eeb92ced0763ccaa1a094c2c75e812ba090b9\"\n", 177 | " \"./.git/objects/b8/2403b2c7d4f507c4debdb47b46fb3754a3085c\"\n", 178 | " \"./.git/objects/b8/54273918b2f809ceb8a2d567665b8bdabb7d9d\"\n", 179 | " \"./.git/objects/bf/2627a295497343ecb3dbec23a853b0ebaa8c4f\"\n", 180 | " \"./.git/objects/33/c9b993c55a75a2424acae6f1bcc5dcbf1f1ef7\"\n", 181 | " \"./.git/objects/d0/0db2ebda0b296f6f08e54ad06f3102e7abdec6\"" 182 | ] 183 | }, 184 | "execution_count": 5, 185 | "metadata": {}, 186 | "output_type": "execute_result" 187 | } 188 | ], 189 | "source": [ 190 | "# this can also be done with comprehension syntax, of course\n", 191 | "\n", 192 | "[fn for fn in filenames if occursin(r\".git/objects/.*[abcde]{4}\", fn)]" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "Notes about performance:\n", 200 | "\n", 201 | "these examples are given for the sake of sympicity and nice print-outs, but, in cases where you don't know the size of the input data in advance, you will want to use generators rather than arrays. Generators expressions look like list comprehensions, but are in parentheses rather than brackets. For a streaming version of the filter function, use `Iterators.filter`." 202 | ] 203 | } 204 | ], 205 | "metadata": { 206 | "kernelspec": { 207 | "display_name": "Julia 1.1.1", 208 | "language": "julia", 209 | "name": "julia-1.1" 210 | }, 211 | "language_info": { 212 | "file_extension": ".jl", 213 | "mimetype": "application/julia", 214 | "name": "julia", 215 | "version": "1.1.1" 216 | } 217 | }, 218 | "nbformat": 4, 219 | "nbformat_minor": 2 220 | } 221 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Administrative Scripting with Julia 2 | =================================== 3 | 4 | Note that this tutorial is still in process. The intended 5 | order, after the introduction is: 6 | 7 | - files_ 8 | - CLI_ 9 | - filesystem_ 10 | - processes_ 11 | - regex_ (still being written) 12 | 13 | However, each part should theoretically stand on its own to some extent. 14 | 15 | Those parts which have been written should nonetheless be considered 16 | drafts for the moment, but you may still find them useful. 17 | 18 | .. _files: 1-files.ipynb 19 | .. _CLI: 2-CLI.ipynb 20 | .. _filesystem: 3-filesystem.ipynb 21 | .. _processes: 4-processes.ipynb 22 | .. _regex: 5-regex.ipynb 23 | 24 | Introduction 25 | ------------ 26 | If you know anything about Julia_, you probably know it's an interactive 27 | language which is gaining popularity for numeric computing that competes 28 | with the likes of R, Matlab, NumPy and others. You've probably also 29 | heard that it compiles to machine code at runtime [#]_ with an LLVM JIT and 30 | can often be optimized to within a factor of two of C or Fortran. 31 | 32 | Given these promises, it's not surprising that it's attracted some very 33 | high-profile users_. 34 | 35 | I don't do any of that kind of programming. I like Julia for other 36 | reasons altogether. For me, the appeal is that it feels good to write. 37 | It's like all the things I like from Python, Perl and Scheme all rolled 38 | into one. The abstractions, both in the language and the standard 39 | library, just feel like they are always hitting the right points. 40 | 41 | Semantically and syntactically, it feels similar to Python and Ruby, 42 | though it promotes some design patterns more common in functional 43 | languages and doesn't support classical OO patterns in the same way. 44 | Instead, it relies on structs, abstract types, and multiple dispatch for 45 | its type system. Julia's metaprogramming story is simple yet deep. It 46 | allows operator overloading and other kinds of magic methods for 47 | applying built-in syntax features to your own types. If that isn't 48 | enough, it has Lisp-like AST macros. 49 | 50 | Finally, reading the standard library (which is implemented mostly in 51 | very readable Julia), you see just how pragmatic it is. It is happy to 52 | call into libc for the things libc is good at. It's equally happy to 53 | shell out if that's the most practical thing. Check out the code for 54 | the download_ function for an instructive example. Julia is very happy 55 | to rely on PCRE_ for regular expressions. On the other hand, Julia is 56 | fast enough that many of the bundled data structures and primitives 57 | are implemented directly in Julia. 58 | 59 | While keeping the syntax fairly clean and straightforward, the Julia 60 | ethos is ultimately about getting things done and empowering the 61 | programmer. If that means performance, you can optimize to your heart's 62 | content. If it means downloading files with ``curl``, it will do that, 63 | too! 64 | 65 | This ethos fits very well with system automation. The classic languages 66 | in this domain are Perl and Bash. Perl has the reputation of being 67 | "write only," and Bash is much worse than that! However, both are 68 | languages that emphasize pragmatism over purity, and that seems to be a 69 | win for short scripts. Julia is more readable than either of these, but 70 | it is not less pragmatic. [#]_ 71 | 72 | This tutorial follows roughly the approach of my `Python tutorial`_ on 73 | administrative scripting and may refer to it at various points. Note 74 | that I've been using Linux exclusively for more than a decade and I'm 75 | not very knowledgable about Windows or OS X. However, if people wish to 76 | contribute content necessary to make this tutorial more compatible with 77 | those platforms, I would be very happy to learn. 78 | 79 | .. _Julia: https://julialang.org/ 80 | .. _users: https://juliacomputing.com/case-studies/ 81 | .. _download: 82 | https://github.com/JuliaLang/julia/blob/e7d15d4a013a43442b75ba4e477382804fa4ac49/base/download.jl 83 | .. _PCRE: https://pcre.org/ 84 | .. _Python tutorial: 85 | https://github.com/ninjaaron/replacing-bash-scripting-with-python 86 | 87 | .. [#] recent versions of Julia do more caching of generated machine code. 88 | 89 | .. [#] This is not to fault the creators of Perl or Bourne Shell. They 90 | are much older langauges, and all dynamic languages to come after, 91 | including Julia, have followed in their footsteps. Later 92 | languages learned from their problems, but they also learned from 93 | what they did right, which was a lot! 94 | 95 | .. contents:: 96 | 97 | Why You Shouldn't Use Julia for Administrative Scripts 98 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 99 | It's just a bad idea! 100 | 101 | - Julia has a fat runtime and it has a human-perceptible load time on a 102 | slower system. For the kinds of intensive problems it's targeted at, 103 | this is nothing. On a constrained server or an embedded system, it's 104 | bad. 105 | - Julia's highly optimizing JIT compiler also takes a little time to 106 | warm up. There are ways to precompile some things, but who wants to 107 | bother for little scripts? The speed of the compiler is impressive for 108 | how good it actually is, but it's not instant. 109 | 110 | Note: in recent versions of Julia, there is a ``--compile=min`` option 111 | which can be used to force Julia to do less code specialization. 112 | Combining this with ``-O0`` will reduce JIT warmup time, though method 113 | resolution will still take time in some instances. 114 | 115 | The above are reasonable arguments against using Julia on a certain 116 | class of servers. However, none of this stuff really matters on a 117 | PC/workstation running an OS with current packages. If your system can 118 | run a modern web browser, Julia's runtime is a pittance. 119 | 120 | If you already want to learn Julia, which there are many good reasons to 121 | do, writing small automation scripts is a gentle way to become 122 | acquainted with the basic features of the language. 123 | 124 | The other reason you might want to try administrative scripting in Julia 125 | is because the abstractions it provides are surprisingly well suited to 126 | the task. Translating a Bash script to Julia is very easy but will 127 | automatically make your script safer and easier to debug. 128 | 129 | One final reason to use Julia for administrative is that it means you're 130 | not using Bash! I've made a `case against Bash`_ for anything but 131 | running and connecting other processes in Bash in my Python tutorial. In 132 | short, Bash is great for interactive use, but it's difficult to do 133 | things in a safe and correct way in scripts, and dealing with data is an 134 | exercise in suffering. Handle data and complexity in programs in other 135 | languages. 136 | 137 | .. _case against bash: 138 | https://github.com/ninjaaron/replacing-bash-scripting-with-python#if-the-shell-is-so-great-what-s-the-problem 139 | 140 | 141 | Learning Julia 142 | ~~~~~~~~~~~~~~ 143 | This tutorial isn't going to show you how to do control flow in Julia 144 | itself, and it certainly isn't going to cover all the ways of dealing 145 | with the rich data structures that Julia provides. To be honest, I'm 146 | still in the process of learning Julia myself, and I'm relying heavily 147 | on the `official docs`_ for that, especially the "Manual" section. As an 148 | experienced Python programmer, the interfaces provided by Julia feel 149 | very familiar, and I suspect the feeling will be similar for Ruby 150 | programmers. For us, becoming productive in Julia should only take a few 151 | hours, though there are rather major differences as one progresses in 152 | the language. 153 | 154 | For a quick introduction to the language, the `learning`_ page has some 155 | good links. The `Intro to Julia`_ with Jane Herriman goes over 156 | everything you'll need to know to understand this tutorial. If you 157 | choose to follow this tutorial, you will be guided to log into 158 | juliabox.com, but you don't need to unless you want to. You can 159 | download and run the `Jupyter Notebooks`_ locally if you wish, and you 160 | can also simply follow along in the Julia REPL in a terminal. 161 | 162 | The `Fast Track to Julia`_ is a handy cheatsheet if you're learning 163 | the language 164 | 165 | .. _official docs: https://docs.julialang.org 166 | .. _learning: https://julialang.org/learning/ 167 | .. _Intro to Julia: https://www.youtube.com/watch?v=8h8rQyEpiZA&t= 168 | .. _Jupyter Notebooks: https://github.com/JuliaComputing/JuliaBoxTutorials 169 | .. _Fast Track to Julia: https://juliadocs.github.io/Julia-Cheat-Sheet/ 170 | 171 | Anway, let's get straight on to files_. 172 | -------------------------------------------------------------------------------- /base.rst: -------------------------------------------------------------------------------- 1 | Administrative Scripting with Julia 2 | =================================== 3 | 4 | Note that this tutorial is still in process. The intended 5 | order, after the introduction is: 6 | 7 | - files_ 8 | - CLI_ 9 | - filesystem_ 10 | - processes_ 11 | - regex_ (still being written) 12 | 13 | However, each part should theoretically stand on its own to some extent. 14 | 15 | Those parts which have been written should nonetheless be considered 16 | drafts for the moment, but you may still find them useful. 17 | 18 | .. _files: 1-files.ipynb 19 | .. _CLI: 2-CLI.ipynb 20 | .. _filesystem: 3-filesystem.ipynb 21 | .. _processes: 4-processes.ipynb 22 | .. _regex: 5-regex.ipynb 23 | 24 | Introduction 25 | ------------ 26 | If you know anything about Julia_, you probably know it's an interpreted 27 | language which is gaining popularity for numeric computing that competes 28 | with the likes of R, Matlab, NumPy and others. You've probably also 29 | heard that it compiles to LLVM bytecode at runtime and can often be 30 | optimized to within a factor of two of C or Fortran. 31 | 32 | Given these promises, it's not surprising that it's attracted some very 33 | high-profile users_. 34 | 35 | I don't do any of that kind of programming. I like Julia for other 36 | reasons altogether. For me, the appeal is that it feels good to write. 37 | It's like all the things I like from Python, Perl and Scheme all rolled 38 | into one. The abstractions, both in the language and the standard 39 | library, just feel like they are always hitting the right points. 40 | 41 | Semantically and syntactically, it feels similar to Python and Ruby, 42 | though it promotes some design patterns more common in functional 43 | languages and doesn't support classical OO patterns in the same way. 44 | Instead, it relies on structs, abstract types, and multiple dispatch for 45 | its type system. Julia's metaprogramming story is simple yet deep. It 46 | allows operator overloading and other kinds of magic methods for 47 | applying built-in syntax features to your own types. If that isn't 48 | enough, it has Lips-like AST macros. 49 | 50 | Finally, reading the standard library (which is implemented mostly in 51 | very readable Julia), you see just how pragmatic it is. It is happy to 52 | call into libc for the things libc is good at. It's equally happy to 53 | shell out if that's the most practical thing. Check out the code for 54 | the download_ function for an instructive example. Julia is very happy 55 | to rely on PCRE_ for regular expressions. On the other hand, Julia is 56 | fast enough that many of the bundled data structures and primitives 57 | are implemented directly in Julia. 58 | 59 | While keeping the syntax fairly clean and straightforward, the Julia 60 | ethos is ultimately about getting things done and empowering the 61 | programmer. If that means performance, you can optimize to your heart's 62 | content. If it means downloading files with ``curl``, it will do that, 63 | too! 64 | 65 | This ethos fits very well with system automation. The classic languages 66 | in this domain are Perl and Bash. Perl has the reputation of being 67 | "write only," and Bash is much worse than that! However, both are 68 | languages that emphasize pragmatism over purity, and that seems to be a 69 | win for short scripts. Julia is more readable than either of these, but 70 | it is not less pragmatic. [#]_ 71 | 72 | This tutorial follows roughly the approach of my `Python tutorial`_ on 73 | administrative scripting and may refer to it at various points. Note 74 | that I've been using Linux exclusively for more than a decade and I'm 75 | not very knowledgable about Windows or OS X. However, if people wish to 76 | contribute content necessary to make this tutorial more compatible with 77 | those platforms, I would be very happy to learn. 78 | 79 | .. _Julia: https://julialang.org/ 80 | .. _users: https://juliacomputing.com/case-studies/ 81 | .. _download: 82 | https://github.com/JuliaLang/julia/blob/e7d15d4a013a43442b75ba4e477382804fa4ac49/base/download.jl 83 | .. _PCRE: https://pcre.org/ 84 | .. _Python tutorial: 85 | https://github.com/ninjaaron/replacing-bash-scripting-with-python 86 | 87 | .. [#] This is not to fault the creators of Perl or Bourne Shell. They 88 | are much older langauges, and all interpreted languages, 89 | including Julia, have followed in their footsteps. Later 90 | languages learned from their problems, but they also learned from 91 | what they did right, which was a lot! 92 | 93 | .. contents:: 94 | 95 | Why You Shouldn't Use Julia for Administrative Scripts 96 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 97 | It's just a bad idea! 98 | 99 | - Julia is not the most portable. It's a relatively new language and has 100 | only had it's 1.1 release this year (2019). Many Linux distros don't 101 | have a package available. Ubuntu 18.04 (the latest one as I write 102 | this) doesn't have a package in the repository, though it is available 103 | as a snap. 104 | - Julia has a fat runtime and it has a human-perceptible load time on a 105 | slower system. For the kinds of intensive problems it's targeted at, 106 | this is nothing. On a constrained server or an embedded system, it's 107 | bad. 108 | - Julia's highly optimizing JIT compiler also takes a little time to 109 | warm up. There are ways to precompile some things, but who wants to 110 | bother for little scripts? The speed of the compiler is impressive for 111 | how good it actually is, but it's not instant. 112 | 113 | The above are reasonable arguments against using Julia on a certain 114 | class of servers. However, none of this stuff really matters on a 115 | PC/workstation running an OS with current packages. If your system can 116 | run a modern web browser, Julia's runtime is a pittance. 117 | 118 | If you already want to learn Julia, which there are many good reasons to 119 | do, writing small automation scripts is a gentle way to become 120 | acquainted with the basic features of the language. 121 | 122 | The other reason you might want to try administrative scripting in Julia 123 | is because the abstractions it provides are surprisingly well suited to 124 | the task. Translating a Bash script to Julia is very easy but will 125 | automatically make your script safer and easier to debug. 126 | 127 | One final reason to use Julia for administrative is that it means you're 128 | not using Bash! I've made a `case against Bash`_ for anything but 129 | running and connecting other processes in Bash in my Python tutorial. In 130 | short, Bash is great for interactive use, but it's difficult to do 131 | things in a safe and correct way in scripts, and dealing with data is an 132 | exercise in suffering. Handle data and complexity in programs in other 133 | languages. 134 | 135 | .. _case against bash: 136 | https://github.com/ninjaaron/replacing-bash-scripting-with-python#if-the-shell-is-so-great-what-s-the-problem 137 | 138 | 139 | Learning Julia 140 | ~~~~~~~~~~~~~~ 141 | This tutorial isn't going to show you how to do control flow in Julia 142 | itself, and it certainly isn't going to cover all the ways of dealing 143 | with the rich data structures that Julia provides. To be honest, I'm 144 | still in the process of learning Julia myself, and I'm relying heavily 145 | on the `official docs`_ for that, especially the "Manual" section. As an 146 | experienced Python programmer, the interfaces provided by Julia feel 147 | very familiar, and I suspect the feeling will be similar for Ruby 148 | programmers. For us, becoming productive in Julia should only take a few 149 | hours, though there are rather major differences as one progresses in 150 | the language. 151 | 152 | For a quick introduction to the language, the `learning`_ page has some 153 | good links. The `Intro to Julia`_ with Jane Herriman goes over 154 | everything you'll need to know to understand this tutorial. If you 155 | choose to follow this tutorial, you will be guided to log into 156 | juliabox.com, but you don't need to unless you want to. You can 157 | download and run the `Jupyter Notebooks`_ locally if you wish, and you 158 | can also simply follow along in the Julia REPL in a terminal. 159 | 160 | The `Fast Track to Julia`_ is a handy cheatsheet if you're learning 161 | the language 162 | 163 | .. _official docs: https://docs.julialang.org 164 | .. _learning: https://julialang.org/learning/ 165 | .. _Intro to Julia: https://www.youtube.com/watch?v=8h8rQyEpiZA&t= 166 | .. _Jupyter Notebooks: https://github.com/JuliaComputing/JuliaBoxTutorials 167 | .. _Fast Track to Julia: https://juliadocs.github.io/Julia-Cheat-Sheet/ 168 | 169 | Anway, let's get straight on to files_. 170 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | cp ./base.rst README.rst 3 | # echo "" >> README.rst 4 | # for file in *.ipynb; do 5 | # jupyter nbconvert --stdout --to rst "$file" >> README.rst 6 | # done 7 | --------------------------------------------------------------------------------