├── LICENSE ├── README.md └── shelm /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2019, Robert 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # shelm 2 | _subverting Elm packaging since 2019_ 3 | 4 | __status: works for me, might eat your files__ 5 | 6 | `shelm` is a bash wrapper around the `elm` tool that allows bypassing 7 | the Elm packaging infrastructure. 8 | 9 | It allows you to 10 | - depend on Elm packages that have not been published to 11 | [package.elm-lang.org](https://package.elm-lang.org), such as 12 | versioned Github releases, local paths or arbitrary git urls. 13 | - work with native Javascript code: Depend on modified versions of core 14 | Elm modules or write your own kernel modules by placing a package within 15 | the `elm/` or `elm-explorations/` package namespace. 16 | 17 | __Note: Best not talk about this on the offical Elm channels unless you're 18 | trolling.__ 19 | 20 | 21 | ## How to use 22 | 23 | `shelm` expects to be called from the top level directory of an Elm project, 24 | which contains an `elm.json` file. The basic workflow is: First call 25 | `shelm fetch` to download dependencies into the local package cache. Then 26 | call `shelm make` as you would usually call `elm make`. (Other `elm` subcommands 27 | are passed through, too, but might not work.) 28 | 29 | With a regular `elm.json`, there should be no functional difference to normal 30 | `elm` tool use. To make use of `shelm`'s extra features, you might: 31 | 32 | - Add dependencies for a packages that have not been released to the Elm 33 | package repository. Suppose you have an Elm package in a GitHub repository 34 | `me/elm-experiment` that you haven't published using `elm publish`. Then if 35 | you tag a commit as "1.0.0", depending on 36 | ``` 37 | "dependencies": { 38 | "direct": { 39 | ... 40 | "me/elm-experiment": "1.0.0", 41 | ... 42 | ``` 43 | in `elm.json` as usual will make `shelm` download the 1.0.0 release. 44 | 45 | - Specify alternative locations for dependencies. E.g., to use a patched 46 | version of the `elm/time` package that's published at `github.com/me/elm-time`, 47 | you would change the dependency section of `elm.json` to read 48 | ``` 49 | "dependencies": { 50 | "direct": { 51 | ... 52 | "elm/time": "1.0.0" 53 | }, 54 | "indirect": { 55 | ... 56 | }, 57 | "locations": { 58 | "elm/time": { 59 | "method": "github", 60 | "name": "me/elm-time" 61 | } 62 | } 63 | } 64 | ``` 65 | See below for full documentation of the `"locations"` field. 66 | 67 | 68 | ## Locations 69 | 70 | `shelm` currently supports the following location types: 71 | 72 | __github__ 73 | 74 | Parameters: `name` as "author/project". 75 | 76 | Tagged GitHub releases, as used by regular `elm` packages. Downloads the 77 | source archive for the release specified by the dependency version. 78 | 79 | See above for an example. 80 | 81 | __file__ 82 | 83 | Parameters: `path` as a local directory. 84 | 85 | Copies over the given path as the source. 86 | 87 | E.g., if you're developing `my-fancy-gui-toolkit` and want to test 88 | it in `my-flashy-app` before publishing, you might tweak `my-flashy-app`'s 89 | `elm.json` to include 90 | ``` 91 | "locations": { 92 | "me/my-fancy-gui-toolkit": { 93 | "method": "file", 94 | "path": "../my-fancy-gui-toolkit" 95 | } 96 | } 97 | ``` 98 | 99 | __git__ 100 | 101 | Parameters: `url` as a git url, `ref` as a branch/tag/commit (defaults to version). 102 | 103 | Checks out the given reference from the given git repository. 104 | 105 | E.g., to build your application against the upstream development version 106 | of `elm/time`, 107 | ``` 108 | "locations": { 109 | "elm/time": { 110 | "method": "git", 111 | "url": "https://github.com/elm/time.git", 112 | "ref": "master" 113 | } 114 | } 115 | ``` 116 | 117 | 118 | ## Troubleshooting 119 | 120 | The `elm` tool error messages are explicitly unhelpful if something goes wrong 121 | compiling dependencies. If you get some complaints about corruption or version 122 | conflicts, chances are that `shelm` or your `elm.json` setup is at fault. Wiping 123 | `elm-stuff` and calling `shelm fetch` again is a good first try. 124 | 125 | It's important that both `elm make` and `elm make --docs=docs.json` works for 126 | each dependency. 127 | 128 | It's your responsibility to make sure that the source archive pointed at by 129 | a location actually has the same version as listed in the dependencies. Things 130 | are likely to go wrong if those don't align. 131 | 132 | 133 | ## Dependencies 134 | 135 | __Elm compiler__ 136 | 137 | The `elm` tool should be in the path. Versions 0.19.0 and 0.19.1 are supported. 138 | 139 | __jq__ 140 | 141 | The [jq](https://stedolan.github.io/jq/) JSON processor is used to read `elm.json`. 142 | 143 | __curl, git, tar__ 144 | 145 | `git` is required for the "git" location method, `curl` and `tar` for the default 146 | "github" archive method. 147 | 148 | 149 | ## How it works 150 | 151 | shelm manages its own local copy of the Elm package database in 152 | `elm-stuff/home/.elm`, by downloading source archives using a variety 153 | of measures, and generating a matching package registry. 154 | It calls `elm` with `$HOME` pointed at this directory and networking 155 | disabled by setting an invalid `$HTTP_PROXY` variable. 156 | 157 | The following two articles go into this with a bit more detail: 158 | 159 | - https://vllmrt.net/spam/guix-elm-2.html describes a Guix build system 160 | for Elm applications that uses the same techniques. 161 | - https://vllmrt.net/spam/subverting-elm.html shows how shelm came about 162 | when figuring out how to build an Elm app with native code. 163 | -------------------------------------------------------------------------------- /shelm: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -eu -o pipefail 4 | 5 | usage() { 6 | cat < ... Pass through to elm 10 | EOF 11 | } 12 | 13 | fail() { 14 | echo error: "$@" 1>&2 15 | exit 1 16 | } 17 | 18 | # Set up some global variables and the package cache directories. 19 | init() { 20 | [ ! -f elm.json ] && fail "elm.json not found in current directory" 21 | 22 | elmhome=$(pwd)/elm-stuff/home 23 | mkdir -p "$elmhome" 24 | 25 | elmversion=$(json_elm_version < elm.json) 26 | 27 | case $elmversion in 28 | 0.19.0) 29 | pkgdir="$elmhome"/.elm/0.19.0/package 30 | registry="$pkgdir"/versions.dat 31 | ;; 32 | 0.19.1) 33 | pkgdir="$elmhome"/.elm/0.19.1/packages 34 | registry="$pkgdir"/registry.dat 35 | ;; 36 | *) 37 | fail "unsupported elm version: $elmversion" 38 | ;; 39 | esac 40 | mkdir -p "$pkgdir" 41 | } 42 | 43 | # query the elm version from elm.json 44 | json_elm_version() { 45 | jq -r '."elm-version"' 46 | } 47 | 48 | # collect dependencies from elm.json 49 | json_dependencies() { 50 | jq -r '.dependencies.direct+.dependencies.indirect 51 | | to_entries[] 52 | | [.key, .value] 53 | | @tsv' 54 | } 55 | 56 | # find a dependency by name 57 | json_dependency_version() { 58 | jq -r '.dependencies.direct+.dependencies.indirect | ."'"$1"'"' 59 | } 60 | 61 | # read location from elm.json, defaulting to github location 62 | json_location() { 63 | pkgname="$1" 64 | jq '.dependencies.locations."'"$pkgname"'" // 65 | { "method": "github", "name": "'"$pkgname"'" }' 66 | } 67 | 68 | fetch_github_release() { 69 | pkgname="$1" 70 | version="$2" 71 | curlargs=-sSLf # no progress, show errors, follow redirects, exit with error on HTTP status >=400 72 | url="https://github.com/$pkgname/archive/$version.tar.gz" 73 | echo "fetching $url" 74 | curl "$curlargs" "$url" | tar -xz 75 | } 76 | 77 | fetch_git_ref() { 78 | url="$1" 79 | ref="$2" 80 | dest="$3" 81 | mkdir "$dest" 82 | cd "$dest" || return 83 | git init -q 84 | git remote add origin "$url" 85 | echo "fetching $ref from $url" 86 | git fetch -q origin 87 | git checkout -q "$ref" 88 | rm -rf .git 89 | } 90 | 91 | # Fetch a single version of a dependency. Expects to be passed an 92 | # empty working directory. Archives are usually fetched like elm 93 | # does from github releases, but are fetched using other methods 94 | # if specified elm.json:dependencies.locations. 95 | fetch() { 96 | tmpdir="$1" 97 | pkgname="$2" 98 | version="$3" 99 | 100 | location="$(json_location "$pkgname" < elm.json)" 101 | method="$(jq -r '.method' <<< "$location")" 102 | case "$method" in 103 | github) 104 | name="$(jq -r '.name' <<< "$location")" 105 | (cd "$tmpdir" && fetch_github_release "$name" "$version") 106 | ;; 107 | file) 108 | path="$(jq -r '.path' <<< "$location")" 109 | echo "copying $path" 110 | cp -r "$path" "$tmpdir"/"$version" 111 | ;; 112 | git) 113 | url="$(jq -r '.url' <<< "$location")" 114 | ref="$(jq -r '.ref // "'"$version"'"' <<< "$location")" 115 | (cd "$tmpdir" && fetch_git_ref "$url" "$ref" "$version") 116 | ;; 117 | *) 118 | fail "unknown location method: $method" 119 | ;; 120 | esac 121 | } 122 | 123 | # List dependencies in the local package cache, in the form $author/$project/$version. 124 | list_dependencies() { 125 | cd "$pkgdir" && find . -type d -depth 3 | sed 's|^./||' 126 | } 127 | 128 | # Prune dependencies from the local package cache that don't match the required 129 | # dependency versions. This is necessary due to the limitation to one version 130 | # per package of the registry generators. 131 | prune_dependencies() { 132 | list_dependencies | while IFS=/ read -r author project version 133 | do 134 | dep="$(json_dependency_version "$author/$project" < elm.json)" 135 | if [ "$version" != "$dep" ]; then 136 | echo "pruning stale dependency $author/$project-$version" 137 | rm -r "${pkgdir:?}/$author/$project/$version" 138 | fi 139 | done 140 | } 141 | 142 | # Fetch Elm dependency source archives into local package cache. 143 | fetch_dependencies() { 144 | unpack=$(mktemp -d) 145 | json_dependencies < elm.json | while read -r name version; do 146 | mkdir -p "$pkgdir"/"$name" 147 | dest="$pkgdir"/"$name"/"$version" 148 | 149 | [ -f "$dest"/elm.json ] && continue 150 | 151 | [ -d "$dest" ] && rm -r "$dest" 152 | fetch "$unpack" "$name" "$version" 153 | mv "$unpack"/* "$dest" 154 | done 155 | rm -r "$unpack" 156 | } 157 | 158 | # Haskell binary encoding helpers 159 | 160 | # Haskell binary encoding of integers as 8 bytes big-endian 161 | encode_int64() { 162 | hex=$(printf "%016x" "$1") 163 | printf "%b%b%b%b%b%b%b%b" \ 164 | "\\x${hex:0:2}" "\\x${hex:2:2}" "\\x${hex:4:2}" "\\x${hex:6:2}" \ 165 | "\\x${hex:8:2}" "\\x${hex:10:2}" "\\x${hex:12:2}" "\\x${hex:14:2}" 166 | } 167 | 168 | # Haskell binary encoding of UTF8 strings 169 | encode_string() { 170 | encode_int64 ${#1} 171 | printf "%s" "$1" 172 | } 173 | 174 | # Haskell binary encoding of bytes 175 | encode_byte() { 176 | hex=$(printf "%02x" "$1") 177 | printf "%b" "\\x${hex}" 178 | } 179 | 180 | # Elm 0.19.1 compact encoding of short UTF8 strings 181 | encode_string_short() { 182 | encode_byte ${#1} 183 | printf "%s" "$1" 184 | } 185 | 186 | # Build Elm 0.19.0 versions.dat 187 | build_versions_dat() { 188 | cd "$pkgdir" || return 189 | count=$(list_dependencies | wc -l) 190 | ( 191 | # total number of versions 192 | encode_int64 "$count" 193 | # number of packages 194 | encode_int64 "$count" 195 | 196 | list_dependencies \ 197 | | sort -t / -k 1,1 -k 2,2 \ 198 | | while IFS=/ read -r author project version 199 | do 200 | pkg=$author/$project 201 | [ "$pkg" = "${prevpkg:-}" ] && fail "multiple versions of package $pkg" 202 | 203 | # $pkg and $prevpkg are local to the subshell that runs this while loop, 204 | # so the warnings about interference with build_registry_dat below are 205 | # irrelevant. 206 | # shellcheck disable=SC2030 207 | prevpkg="$author"/"$project" 208 | 209 | encode_string "$author" 210 | encode_string "$project" 211 | 212 | # number of versions 213 | encode_int64 1 214 | 215 | # only version 216 | IFS=. read -ra vparts <<< "$version" 217 | encode_byte "${vparts[0]}" 218 | encode_byte "${vparts[1]}" 219 | encode_byte "${vparts[2]}" 220 | done 221 | ) 222 | } 223 | 224 | # Build Elm 0.19.1 registry.dat 225 | build_registry_dat() { 226 | cd "$pkgdir" || return 227 | count=$(list_dependencies | wc -l) 228 | ( 229 | # total number of versions 230 | encode_int64 "$count" 231 | # number of packages 232 | encode_int64 "$count" 233 | 234 | list_dependencies \ 235 | | sort -t / -k 1,1 -k 2,2 \ 236 | | while IFS=/ read -r author project version 237 | do 238 | pkg=$author/$project 239 | # shellcheck disable=SC2031 240 | [ "$pkg" = "${prevpkg:-}" ] && fail "multiple versions of package $pkg" 241 | prevpkg="$author"/"$project" 242 | 243 | encode_string_short "$author" 244 | encode_string_short "$project" 245 | 246 | # newest (only) version 247 | IFS=. read -ra vparts <<< "$version" 248 | encode_byte "${vparts[0]}" 249 | encode_byte "${vparts[1]}" 250 | encode_byte "${vparts[2]}" 251 | 252 | # number of extra versions 253 | encode_int64 0 254 | done 255 | ) 256 | } 257 | 258 | # generate a registry file appropriate to the Elm version, 259 | # based on previously fetched dependencies 260 | generate() { 261 | echo "generating $registry" 262 | case "$elmversion" in 263 | 0.19.0) 264 | build_versions_dat > "$registry" 265 | ;; 266 | 0.19.1) 267 | build_registry_dat > "$registry" 268 | ;; 269 | esac 270 | } 271 | 272 | 273 | case "$1" in 274 | ""|help|-*) 275 | usage 276 | ;; 277 | fetch) 278 | init 279 | prune_dependencies 280 | fetch_dependencies 281 | generate 282 | ;; 283 | generate) 284 | init 285 | generate 286 | ;; 287 | *) 288 | init 289 | [ ! -f "$registry" ] && fail "registry missing, please fetch and generate first" 290 | HOME="$elmhome" HTTP_PROXY=. elm "$@" 291 | ;; 292 | esac 293 | --------------------------------------------------------------------------------