├── .gitmodules ├── stackage ├── Setup.hs ├── README.md ├── Stackage │ └── Dummy.hs ├── ChangeLog.md ├── LICENSE └── stackage.cabal ├── .gitignore ├── .dockerignore ├── check ├── .dir-locals.el ├── automated ├── .gitignore ├── run-nightly.sh ├── check-cabal.sh └── build.sh ├── .project-settings.yml ├── Dockerfile ├── Stackage └── Config.hs ├── LICENSE ├── .travis.yml ├── debian-bootstrap.sh ├── README.md ├── DATA-FLOW.md ├── MAINTAINERS.md └── CURATORS.md /.gitmodules: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /stackage/Setup.hs: -------------------------------------------------------------------------------- 1 | import Distribution.Simple 2 | main = defaultMain 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /builds/ 2 | /logs/ 3 | nightly-*.yaml 4 | lts-*.yaml 5 | *.swp 6 | check-plan.yaml 7 | -------------------------------------------------------------------------------- /.dockerignore: -------------------------------------------------------------------------------- 1 | dist 2 | builds 3 | logs 4 | .cabal-sandbox 5 | cabal.sandbox.config 6 | tarballs 7 | *.yaml 8 | .git 9 | -------------------------------------------------------------------------------- /stackage/README.md: -------------------------------------------------------------------------------- 1 | This is a dummy wrapper package, forcing installation of other packages which 2 | provide real functionality. 3 | -------------------------------------------------------------------------------- /check: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # Convenience script for checking constraints locally 4 | 5 | cd `dirname $0` 6 | exec stack exec --resolver ghc-8.0.1 stackage-curator check 7 | -------------------------------------------------------------------------------- /.dir-locals.el: -------------------------------------------------------------------------------- 1 | ((haskell-mode . ((haskell-process-type . cabal-repl) 2 | (haskell-indent-spaces . 4) 3 | (hindent-style . "johan-tibell")))) 4 | -------------------------------------------------------------------------------- /automated/.gitignore: -------------------------------------------------------------------------------- 1 | /auth-token 2 | /bin/ 3 | /gitconfig 4 | /hackage-creds 5 | /ssh-nightly/ 6 | /ssh-lts/ 7 | /nighlty/ 8 | /lts-*/ 9 | /stackage-curator/ 10 | /stackage-update/ 11 | -------------------------------------------------------------------------------- /stackage/Stackage/Dummy.hs: -------------------------------------------------------------------------------- 1 | -- | This module does absolutely nothing. It's present so that cabal can more 2 | -- easily track whether the stackage package is installed. 3 | module Stackage.Dummy () where 4 | -------------------------------------------------------------------------------- /.project-settings.yml: -------------------------------------------------------------------------------- 1 | module-template: ! 'module MODULE_NAME where 2 | 3 | ' 4 | extensions: {} 5 | environment: default 6 | cabal-file: project.cabal 7 | version: 1 8 | ghc-args: [] 9 | excluded-modules: 10 | - Setup.hs 11 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:16.04 2 | 3 | ENV HOME /home/stackage 4 | ENV LANG en_US.UTF-8 5 | ENV PATH /opt/ghc/8.0.1/bin:/usr/sbin:/usr/bin:/sbin:/bin 6 | 7 | ADD debian-bootstrap.sh /tmp/debian-bootstrap.sh 8 | RUN /tmp/debian-bootstrap.sh && rm /tmp/debian-bootstrap.sh 9 | -------------------------------------------------------------------------------- /Stackage/Config.hs: -------------------------------------------------------------------------------- 1 | {- 2 | 3 | NOTE: This module is no longer used for tracking packages included in Stackage. 4 | To simplify the codebase, that information is now stored in the 5 | build-constraints.yaml configuration file. The file should be self-explanatory. 6 | Sorry for the inconvenience. 7 | 8 | -} 9 | -------------------------------------------------------------------------------- /stackage/ChangeLog.md: -------------------------------------------------------------------------------- 1 | ## 0.7.3.0 2 | 3 | * Added the executables split off from stackage-cli. 4 | 5 | ## 0.7.0.0 6 | 7 | * First release of this incarnation of the stackage package. Previously, this 8 | package provided completely different functionality. That functionality has 9 | since moved to stackage-curator. 10 | -------------------------------------------------------------------------------- /automated/run-nightly.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | while true; do 4 | /opt/stackage-build/stackage/automated/build.sh nightly-$(date -u +%F) 5 | date 6 | echo 7 | 8 | echo "Running stackage-server-cron..." 9 | echo "('tail -f /home/ubuntu/stackage-server-cron.log' to watch)" 10 | /home/ubuntu/stackage-server-cron.sh >> /home/ubuntu/stackage-server-cron.log 2>&1 11 | echo "done." 12 | date 13 | 14 | sleep 30m 15 | done 16 | -------------------------------------------------------------------------------- /stackage/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 FP Complete 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included 12 | in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 17 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 18 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 19 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 20 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /stackage/stackage.cabal: -------------------------------------------------------------------------------- 1 | name: stackage 2 | version: 0.7.3.2 3 | synopsis: Dummy package forcing installation of other Stackage packages 4 | homepage: https://www.stackage.org/ 5 | license: MIT 6 | license-file: LICENSE 7 | author: Michael Snoyman 8 | maintainer: michael@snoyman.com 9 | category: Development 10 | build-type: Simple 11 | extra-source-files: README.md ChangeLog.md 12 | cabal-version: >=1.10 13 | 14 | library 15 | exposed-modules: Stackage.Dummy 16 | build-depends: base < 10 17 | , stackage-cli >= 0.1.0 18 | , stackage-update 19 | , stackage-upload 20 | , stackage-install 21 | , stackage-build-plan 22 | , stackage-cabal >= 0.1.1 23 | , stackage-sandbox >= 0.1.1 24 | , stackage-setup >= 0.0.1 25 | default-language: Haskell2010 26 | 27 | source-repository head 28 | type: git 29 | location: git://github.com/fpco/stackage.git 30 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 FP Complete, http://www.fpcomplete.com/ 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # Use new container infrastructure to enable caching 2 | sudo: false 3 | 4 | # Choose a lightweight base image; we provide our own build tools. 5 | language: c 6 | 7 | # GHC depends on GMP. You can add other dependencies here as well. 8 | addons: 9 | apt: 10 | packages: 11 | - libgmp-dev 12 | 13 | env: 14 | - GHCVER=8.0.1 15 | 16 | install: 17 | # Download and unpack the stack executable 18 | - mkdir -p ~/.local/bin 19 | - export PATH=$HOME/.local/bin:$PATH 20 | - travis_retry curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack' 21 | 22 | # Get stackage-curator 23 | - wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2 24 | - bunzip2 stackage-curator.bz2 25 | - chmod +x stackage-curator 26 | - mv stackage-curator ~/.local/bin 27 | 28 | # Install GHC and cabal-install 29 | - stack setup $GHCVER 30 | 31 | # Update the index 32 | - travis_retry stack update 33 | 34 | script: 35 | - stack --resolver ghc-$GHCVER exec stackage-curator check 36 | 37 | cache: 38 | directories: 39 | - $HOME/.stack 40 | - $HOME/.stackage/curator/cache 41 | -------------------------------------------------------------------------------- /automated/check-cabal.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script is intended for testing Cabal HEAD against a Stackage snapshot. 4 | 5 | set -eux 6 | 7 | ROOT=$(cd $(dirname $0) ; pwd) 8 | TARGET=$1 9 | 10 | # For nightly-YYYY-MM-DD, tag should be nightly 11 | # For lts-X.Y, tag should be ltsX 12 | SHORTNAME=$(echo $TARGET | cut -d- -f 1) 13 | if [ $SHORTNAME = "lts" ] 14 | then 15 | TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@') 16 | PLAN_URL=https://raw.githubusercontent.com/fpco/lts-haskell/master/$TARGET.yaml 17 | else 18 | TAG=$SHORTNAME 19 | PLAN_URL=https://raw.githubusercontent.com/fpco/stackage-nightly/master/$TARGET.yaml 20 | fi 21 | 22 | IMAGE=snoyberg/stackage:$TAG 23 | 24 | PLAN_FILE=$TARGET-plan.yaml 25 | DOCMAP_FILE=$TARGET-docmap.yaml 26 | BUNDLE_FILE=$TARGET.bundle= 27 | 28 | CABAL_DIR=$ROOT/cabal 29 | STACK_DIR=$ROOT/stack 30 | GHC_DIR=$ROOT/ghc 31 | DOT_STACKAGE_DIR=$ROOT/dot-stackage 32 | WORKDIR=$ROOT/$TAG/work 33 | EXTRA_BIN_DIR=$ROOT/extra-bin 34 | 35 | mkdir -p \ 36 | "$CABAL_DIR" \ 37 | "$STACK_DIR" \ 38 | "$GHC_DIR" \ 39 | "$DOT_STACKAGE_DIR" \ 40 | "$WORKDIR" \ 41 | "$EXTRA_BIN_DIR" 42 | 43 | curl "$PLAN_URL" > $WORKDIR/$PLAN_FILE 44 | 45 | mkdir -p $ROOT/bin 46 | BINDIR=$(cd $ROOT/bin ; pwd) 47 | ( 48 | cd $BINDIR 49 | rm -f stackage-curator stackage-curator.bz2 50 | wget https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2 51 | bunzip2 stackage-curator.bz2 52 | chmod +x stackage-curator 53 | ) 54 | 55 | ARGS_COMMON="--rm -v $WORKDIR:/home/stackage/work -w /home/stackage/work -v $BINDIR/stackage-curator:/usr/bin/stackage-curator:ro -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $EXTRA_BIN_DIR:/home/stackage/bin:ro" 56 | ARGS_PREBUILD="$ARGS_COMMON -u $USER -v $CABAL_DIR:/home/stackage/.cabal -v $STACK_DIR:/home/stackage/.stack -v $GHC_DIR:/home/stackage/.ghc -v $DOT_STACKAGE_DIR:/home/stackage/.stackage" 57 | ARGS_BUILD="$ARGS_COMMON -v $CABAL_DIR:/home/stackage/.cabal:ro -v $STACK_DIR:/home/stackage/.stack:ro -v $GHC_DIR:/home/stackage/.ghc:ro" 58 | 59 | # Get latest stack 60 | curl -L https://www.stackage.org/stack/linux-x86_64 | tar xz --wildcards --strip-components=1 -C $EXTRA_BIN_DIR '*/stack' 61 | 62 | # Do all of the pre-build actions: 63 | # 64 | # * Update the package index 65 | # * Fetch all needed tarballs (the build step does not have write access to the tarball directory) 66 | # * Do a single unpack to create the package index cache (again due to directory perms) 67 | docker run $ARGS_PREBUILD $IMAGE /bin/bash -c "/home/stackage/bin/stack update && stackage-curator fetch --plan-file $PLAN_FILE && cd /tmp && /home/stackage/bin/stack unpack random" 68 | 69 | # Now do the actual build. We need to first set the owner of the home directory 70 | # correctly, so we run the command as root, change owner, and then use sudo to 71 | # switch back to the current user 72 | docker run $ARGS_BUILD $IMAGE /bin/bash -c "chown $USER /home/stackage && sudo -E -u $USER env \"PATH=\$PATH:/home/stackage/bin\" stackage-curator make-bundle --plan-file $PLAN_FILE --docmap-file $DOCMAP_FILE --bundle-file $BUNDLE_FILE --target $TARGET --cabal-from-head" 73 | -------------------------------------------------------------------------------- /debian-bootstrap.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Work in progress: create a list of commands necessary to get Stackage 4 | # up-and-running on a freshly installed Debian-based system (including Ubuntu). 5 | 6 | # Quick start: 7 | # wget -O - https://raw.github.com/fpco/stackage/master/debian-bootstrap.sh | bash -ex 8 | 9 | # NOTE: Requires that GHC and Cabal are installed and on your PATH. For 10 | # instructions, see: 11 | # http://www.stackage.org/install 12 | 13 | set -exu 14 | 15 | mkdir /home/stackage -p 16 | locale-gen en_US.UTF-8 17 | 18 | export DEBIAN_FRONTEND=noninteractive 19 | apt-get update 20 | apt-get install -y software-properties-common 21 | 22 | add-apt-repository ppa:hvr/ghc -y 23 | add-apt-repository -y ppa:marutter/rrutter 24 | # not sure what this was needed for 25 | #add-apt-repository -y ppa:openstack-ubuntu-testing/icehouse 26 | 27 | # Set the GHC version 28 | GHCVER=8.0.1 29 | 30 | # Get Stack 31 | apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 575159689BEFB442 32 | echo 'deb http://download.fpcomplete.com/ubuntu xenial main'|tee /etc/apt/sources.list.d/fpco.list 33 | 34 | apt-get update 35 | apt-get install -y \ 36 | build-essential \ 37 | ghc-$GHCVER \ 38 | ghc-$GHCVER-htmldocs \ 39 | hscolour \ 40 | sudo \ 41 | curl \ 42 | freeglut3-dev \ 43 | git \ 44 | libadns1-dev \ 45 | libasound2-dev \ 46 | libblas-dev \ 47 | libbz2-dev \ 48 | libcairo2-dev \ 49 | libcurl4-openssl-dev \ 50 | libdevil-dev \ 51 | libedit-dev \ 52 | libedit2 \ 53 | libfftw3-dev \ 54 | libfreenect-dev \ 55 | libgd2-xpm-dev \ 56 | libgeoip-dev \ 57 | libgirepository1.0-dev \ 58 | libglib2.0-dev \ 59 | libglu1-mesa-dev \ 60 | libgmp3-dev \ 61 | libgnutls-dev \ 62 | libgsasl7-dev \ 63 | libgsl0-dev \ 64 | libgtk-3-dev \ 65 | libgtk2.0-dev \ 66 | libgtksourceview-3.0-dev \ 67 | libhidapi-dev \ 68 | libicu-dev \ 69 | libjudy-dev \ 70 | liblapack-dev \ 71 | libleveldb-dev \ 72 | liblzma-dev \ 73 | libmagic-dev \ 74 | libmagickcore-dev \ 75 | libmagickwand-dev \ 76 | libmarkdown2-dev \ 77 | libmysqlclient-dev \ 78 | libncurses-dev \ 79 | libnotify-dev \ 80 | libopenal-dev \ 81 | libpango1.0-dev \ 82 | libpcap0.8-dev \ 83 | libpq-dev \ 84 | libsdl2-dev \ 85 | libsnappy-dev \ 86 | libsndfile1-dev \ 87 | libsqlite3-dev \ 88 | libssl-dev \ 89 | libsystemd-dev \ 90 | libtagc0-dev \ 91 | libtre-dev \ 92 | libudev-dev \ 93 | libusb-1.0-0-dev \ 94 | libwebkitgtk-3.0-dev \ 95 | libxau-dev \ 96 | libxml2-dev \ 97 | libxrandr-dev \ 98 | libxss-dev \ 99 | libyaml-dev \ 100 | libzip-dev \ 101 | libzmq3-dev \ 102 | llvm-3.7 \ 103 | m4 \ 104 | nettle-dev \ 105 | nodejs \ 106 | npm \ 107 | openjdk-8-jdk \ 108 | r-base \ 109 | r-base-dev \ 110 | ruby-dev \ 111 | stack \ 112 | wget \ 113 | xclip \ 114 | zip \ 115 | zlib1g-dev 116 | 117 | # Put documentation where we expect it 118 | mv /opt/ghc/$GHCVER/share/doc/ghc-$GHCVER/ /opt/ghc/$GHCVER/share/doc/ghc 119 | 120 | # Buggy versions of ld.bfd fail to link some Haskell packages: 121 | # https://sourceware.org/bugzilla/show_bug.cgi?id=17689. Gold is 122 | # faster anyways and uses less RAM. 123 | update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.gold" 20 124 | update-alternatives --install "/usr/bin/ld" "ld" "/usr/bin/ld.bfd" 10 125 | 126 | # GHC requires a specific LLVM version on the system PATH for its LLVM backend. 127 | # This version is tracked here: 128 | # https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/Backends/LLVM/Installing 129 | # 130 | # GHC 8.0 requires LLVM 3.7 tools (specifically, llc-3.7 and opt-3.7). 131 | update-alternatives --install "/usr/bin/llc" "llc" "/usr/bin/llc-3.7" 50 132 | update-alternatives --install "/usr/bin/opt" "opt" "/usr/bin/opt-3.7" 50 133 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | stackage 2 | ======== 3 | 4 | [![Build Status](https://travis-ci.org/fpco/stackage.svg?branch=master)](https://travis-ci.org/fpco/stackage) 5 | 6 | "Stable Hackage": creating a vetted set of packages from Hackage. 7 | This repository is for package authors and maintainers to get their packages into Stackage. 8 | If you simply want to use Stackage as an end user, please follow the instructions on [https://www.stackage.org/](https://www.stackage.org). 9 | 10 | We strongly recommend using the Haskell tool stack for doing builds, which 11 | includes built-in Stackage support: [stack](https://github.com/commercialhaskell/stack) [![Build Status](https://travis-ci.org/commercialhaskell/stack.svg?branch=master)](https://travis-ci.org/commercialhaskell/stack). 12 | 13 | 14 | Add your package 15 | ---------------- 16 | 17 | We welcome all packages, provided: 18 | 19 | * The package author/maintainer agrees to the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md). 20 | * The package is buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete. 21 | * The package is compatible with the newest versions of all dependencies. 22 | * The package is compatible with the versions of libraries that ship with GHC ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)). 23 | 24 | Full details on how to add a package can be found in the [maintainers agreement](https://github.com/fpco/stackage/blob/master/MAINTAINERS.md#adding-a-package). 25 | 26 | __NOTE__: There is an approximate 30 minute delay between a package uploading 27 | to Hackage and being available to the Travis build script to check upper 28 | bounds. If a pull request is marked as failed due to using an older version, 29 | please close and reopen the PR to retrigger a Travis build. 30 | 31 | Other repos 32 | ----------- 33 | 34 | The Stackage project consists of multiple repositories. This repository 35 | contains the metadata on packages to be included in future builds and some 36 | project information. In addition, we have the following repositories: 37 | 38 | * [stackage-server](https://github.com/fpco/stackage-server) [![Build Status](https://travis-ci.org/fpco/stackage-server.svg?branch=master)](https://travis-ci.org/fpco/stackage-server) 39 | * [stackage-curator](https://github.com/fpco/stackage-curator) [![Build Status](https://travis-ci.org/fpco/stackage-curator.svg?branch=master)](https://travis-ci.org/fpco/stackage-curator) 40 | * [stackage-types](https://github.com/fpco/stackage-types) [![Build Status](https://travis-ci.org/fpco/stackage-types.svg?branch=master)](https://travis-ci.org/fpco/stackage-types) 41 | * [lts-haskell](https://github.com/fpco/lts-haskell) 42 | * [stackage-nightly](https://github.com/fpco/stackage-nightly) 43 | 44 | We also support some add-on tools to cabal-install to make its usage with 45 | Stackage both easier and more secure: 46 | 47 | * [stackage-cli](https://github.com/fpco/stackage-cli) [![Build Status](https://travis-ci.org/fpco/stackage-cli.svg?branch=master)](https://travis-ci.org/fpco/stackage-cli) 48 | * [stackage-update](https://github.com/fpco/stackage-update) [![Build Status](https://travis-ci.org/fpco/stackage-update.svg?branch=master)](https://travis-ci.org/fpco/stackage-update) 49 | * [stackage-upload](https://github.com/fpco/stackage-upload) [![Build Status](https://travis-ci.org/fpco/stackage-upload.svg?branch=master)](https://travis-ci.org/fpco/stackage-upload) 50 | * [stackage-install](https://github.com/fpco/stackage-install) [![Build Status](https://travis-ci.org/fpco/stackage-install.svg?branch=master)](https://travis-ci.org/fpco/stackage-install) 51 | * [stackage-build-plan](https://github.com/fpco/stackage-build-plan) [![Build Status](https://travis-ci.org/fpco/stackage-build-plan.svg?branch=master)](https://travis-ci.org/fpco/stackage-build-plan) 52 | 53 | Curious how it all fits together? See the [Stackage data 54 | flow](https://github.com/fpco/stackage/blob/master/DATA-FLOW.md). 55 | 56 | 57 | Build the package set 58 | --------------------- 59 | 60 | Generally, building the package set should be done only by the Stackage build 61 | machine by the Stackage curation team. If you're interested in trying this 62 | yourself, please check out [the curator 63 | guide](https://github.com/fpco/stackage/blob/master/CURATORS.md), though be 64 | aware that this is not a recommended practice and there likely will be problems 65 | you will need to debug yourself. 66 | 67 | ### Docker 68 | 69 | Note: This method has been disabled for now, but may be enabled again in the future. 70 | 71 | If you'd like to check a build plan, or perform an entire build, without 72 | specially configuring your system, Docker may be a good approach. To check if 73 | some modifications to `build-constraints.yaml` are valid, try the following: 74 | 75 | 1. Create a local clone of the `stackage` repo 76 | 2. Make modifications to your local `build-constraints.yaml` 77 | 3. Inside the `stackage` working directory, run the following: 78 | 79 | ``` 80 | $ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage check' 81 | ``` 82 | 83 | Similarly, if you'd like to perform an entire build, you can replace the last step with: 84 | 85 | ``` 86 | $ docker run -it --rm -v $(pwd):/stackage -w /stackage snoyberg/stackage /bin/bash -c 'cabal update && stackage nightly --skip-upload' 87 | ``` 88 | 89 | ## Processing 90 | 91 | The following describes at a high level the series of steps for processing 92 | 93 | ### Nightlies 94 | 95 | 1. Get list of core packages 96 | 2. Get build constraints from list of maintained packages 97 | 3. Load up package index 98 | 4. Calculate build plan using newest versions of packages 99 | 5. Write out a YAML file with complete build plan 100 | 6. Verify that the build plan can be compiled 101 | 7. Perform the build 102 | 103 | ### LTS 104 | 105 | 1. Load up most recent build plan 106 | 2. Convert build plan into constraints for next build 107 | 3. Continue from step (3) above 108 | -------------------------------------------------------------------------------- /automated/build.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -eux 4 | 5 | ROOT=$(cd $(dirname $0) ; pwd) 6 | TARGET=$1 7 | 8 | # For nightly-YYYY-MM-DD, tag should be nightly 9 | # For lts-X.Y, tag should be ltsX 10 | SHORTNAME=$(echo $TARGET | cut -d- -f 1) 11 | if [ $SHORTNAME = "lts" ] 12 | then 13 | TAG=$(echo $TARGET | sed 's@^lts-\([0-9]*\)\.[0-9]*@lts\1@') 14 | else 15 | TAG=$SHORTNAME 16 | fi 17 | 18 | IMAGE=snoyberg/stackage:$TAG 19 | 20 | PLAN_FILE=current-plan.yaml 21 | DOCMAP_FILE=current-docmap.yaml 22 | BUNDLE_FILE=current.bundle 23 | 24 | CABAL_DIR=$ROOT/cabal 25 | STACK_DIR=$ROOT/stack 26 | GHC_DIR=$ROOT/ghc 27 | DOT_STACKAGE_DIR=$ROOT/dot-stackage 28 | WORKDIR=$ROOT/$TAG/work 29 | EXTRA_BIN_DIR=$ROOT/extra-bin 30 | SSH_DIR=$ROOT/ssh-$SHORTNAME 31 | 32 | mkdir -p \ 33 | "$CABAL_DIR" \ 34 | "$STACK_DIR" \ 35 | "$GHC_DIR" \ 36 | "$DOT_STACKAGE_DIR" \ 37 | "$WORKDIR" \ 38 | "$EXTRA_BIN_DIR" \ 39 | "$SSH_DIR" 40 | 41 | GITCONFIG=$ROOT/gitconfig 42 | cat >$GITCONFIG <$SSH_DIR/known_hosts <= 5.6.7 && < 5.7`. 75 | * When doing a Stackage Nightly build or LTS Haskell major version bump (e.g., 76 | building lts-6.0), it grabs the latest version of the build-constraints.yaml 77 | file. 78 | 79 | By combining these constraints with the current package data, stackage-curator 80 | can generate a build plan and check it. (As an aside, this build plan 81 | generation and checking also occurs every time you make a pull request to the 82 | stackage repo.) If there are version bounds problems, one of the [Stackage 83 | curators](https://github.com/fpco/stackage/blob/master/CURATORS.md) will open 84 | up a Github issue and will add upper bounds, temporarily block a package, or 85 | some other corrective action. 86 | 87 | Once a valid build plan is found, stackage-curator will build all packages, 88 | build docs, and run test suites. Assuming that all succeeds, it generates some 89 | artifacts: 90 | 91 | * Uploads the build plan as a YAML file to either 92 | [stackage-nightly](https://github.com/fpco/stackage-nightly) or 93 | [lts-haskell](https://github.com/fpco/lts-haskell) 94 | * Uploads the generated Haddock docs and a package index (containing all used 95 | .cabal files) to haddock.stackage.org. 96 | 97 | ## stackage-server-cron 98 | 99 | On the Stackage build server, we run the [stackage-server-cron 100 | executable](https://github.com/fpco/stackage-server/blob/master/app/stackage-server-cron.hs) 101 | regularly, which generates: 102 | 103 | * A [SQLite 104 | database](https://github.com/fpco/stackage-server/blob/master/Stackage/Database.hs) 105 | containing information on snapshots, the packages they contain, Hackage 106 | metadata about packages, and a bit more. This database is uploaded to S3. 107 | * A Hoogle database for each snapshot, which is also uploaded to S3 108 | 109 | ## stackage-server 110 | 111 | The [software running stackage.org](https://github.com/fpco/stackage-server) is 112 | a relatively simple Yesod web application. It pulls data from the 113 | stackage-content repo, the SQLite database, the Hoogle databases, and the build 114 | plans for Stackage Nightly and LTS Haskell. It doesn't generate anything 115 | important of its own except for a user interface. 116 | 117 | ## Stack 118 | 119 | [Stack](http://haskellstack.com) takes advantage of many of the pieces listed above as well: 120 | 121 | * It by default uses the all-cabal-hashes repo for getting package metadata, 122 | and downloads package contents from the hackage.fpcomplete.com mirror (using 123 | the hashes in the repo for verification) 124 | * There are some metadata files in stackage-content which contain information 125 | on, for example, where to download GHC tarballs from to make `stack setup` 126 | work 127 | * Stack downloads the raw build plans for Stackage Nightly and LTS Haskell from 128 | the Github repo and uses them when deciding which packages to build for a 129 | given stack.yaml file 130 | -------------------------------------------------------------------------------- /MAINTAINERS.md: -------------------------------------------------------------------------------- 1 | This project is built around the concept of maintainers taking responsibility for making their packages work with the rest of the stable ecosystem, usually meaning the newest version of all dependencies. This is a social contract, and is not reflected in the codebase in any way. 2 | 3 | The idea behind Stackage is that, if all packages work with the newest versions of dependencies, we avoid dependency hell. Specifically, we aim for: 4 | 5 | * All packages are buildable and testable from Hackage. We recommend [the Stack Travis script](http://docs.haskellstack.org/en/stable/GUIDE.html#travis-with-caching), which ensures a package is not accidentally incomplete. 6 | * All packages are compatible with the newest versions of all dependencies. 7 | * All packages in a snapshot are compatible with the versions of libraries that ship with the GHC used in the snapshot ([more information on lenient lower bounds](https://www.fpcomplete.com/blog/2014/05/lenient-lower-bounds)). 8 | 9 | ## Adding a package 10 | 11 | Anyone can add a package to Stackage, but it's highly encouraged that the actual package maintainer is also the Stackage maintainer. If that is not the case, you should drop the package maintainer a note first. 12 | 13 | To add your package, first fork this repository. 14 | In the [`build-constraints.yaml`](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) file, there's a section called `packages`. 15 | To add a set of packages, you would add: 16 | 17 | "My Name myemail@example.com @mygithubuser": 18 | - package1 19 | - package2 20 | - package3 21 | 22 | After doing that, send a pull request (with a commit message like "add foo-bar"). We do not require new submissions to be tested against the rest of Stackage before the pull request (though it is a good idea to do so if you can with `stack --resolver nightly exec stackage-curator check` and `stack --resolver nightly build`), provided you meet the dependency version requirements above. If your library depends on a C library, add a note to your pull request with the Ubuntu library name, or even better edit the `debian-bootstrap.sh` script directly 23 | 24 | **NB** Please use commit messages like "add foo-bar" or "add johndev's packages" 25 | (`build-constraints.yaml` is the most frequently changed file in this git repo 26 | so commit messages like "update build-constraints.yaml" are not helpful). 27 | 28 | **NB2** There can be a delay of up to an hour before package versions newly 29 | uploaded to Hackage appear to our build server. If you just uploaded a package 30 | to Hackage that you're trying to get included, we recommend waiting an hour 31 | before opening the PR. 32 | 33 | 34 | ## Uploading a new package 35 | 36 | When a new version of a package is uploaded to Hackage, we automatically try to include it in Stackage (unless the new version is considered experimental). That can result in a number of possible failures. If there is a failure we temporarily introduce an upper bound, and raise GitHub issue tickets to resolve the issue. 37 | 38 | If the new version doesn't compile then the package author should quickly (within 1 week) upload a fixed version. 39 | 40 | If a package's test suite is failing, the first job is to investigate why. If this is due to a bad interaction with versions of other packages in Stackage, then it is the responsibility of the maintainer to fix the test suite. In some situations, it is acceptable to not run the test suite. 41 | 42 | 43 | ## Following dependency upgrades 44 | 45 | If a new version of a dependency is released, and that stops your package compiling/passing the tests, then it is your responsibility to modify your package. It is highly recommended that all package maintainers follow the dependencies of their packages on [Packdeps](http://packdeps.haskellers.com/), typically using the RSS feeds. 46 | 47 | **If restrictive version bounds are the only problem** then you must quickly (within 1 week) upload a new version with relaxed version bounds. Note that unlike the PVP, Stackage does not require upper bounds. 48 | 49 | **If the new dependency causes breaking changes** then all package authors should quickly assess the likely impact on their package (within 1 week) and then produce a new compatible version. The expected timeline for new versions varies between 1 week and 1 month, depending on the significance of the change, and thus the work required to produce those new versions. 50 | 51 | 52 | ## Failing to meet the time limits 53 | 54 | Maintainers are humans, humans get sick/have babies/go on holiday. If you have regular problems meeting the limits, find a co-maintainer. If you have a one-off problem, respond to the GitHub tickets saying so, and some kind soul might pick up the slack. 55 | 56 | The time limits are intended to stop people being inconvenienced because of problems in other packages. Where such inconvenience happens, we will drop the offending packages from Stackage. While upper bounds are sometimes a temporary solution, they are against the ethos of Stackage, so will not be kept for long. 57 | 58 | ## Upgrading to a new GHC version 59 | 60 | The Stackage curation team tries to move Stackage Nightly to new versions of GHC quickly as they become available, while keeping LTS Haskell on a regular release schedule. For package maintainers, the most important impacts of a new GHC release are: 61 | 62 | * We will typically do a sweep through the Stackage upper bounds and aggressively remove packages that block them. This is because, in most cases, we will need to move to the newest versions of a package to get support for the latest GHC, and asking package maintainers to backport their fixes is an undue burden 63 | * We will definitely do this at a GHC major version release, and may do so at a minor version release 64 | * Packages that are incompatible with the newest GHC version will be temporarily blocked 65 | 66 | If your package ends up being temporarily removed from Stackage Nightly, please simply send a pull request to add it back once it and its dependencies are compatible with the newest GHC version. 67 | 68 | ## Adding a package to an LTS snapshot 69 | 70 | The steps above affect the Stackage Nightly builds, but do not directly affect 71 | LTS Haskell builds. When we build a new LTS Haskell major version (anything 72 | ending in `.0`), the package set is taken from Stackage Nightly. Therefore, by 73 | following the above steps, you can get your package into the next major LTS 74 | Haskell release. 75 | 76 | If you would like to get your package added to an existing LTS Haskell major 77 | release (e.g., if `lts-3.21` is out, you would want your package to appear in 78 | `lts-3.22`), please do the following in addition to the steps above: 79 | 80 | * Open up a new issue on the [lts-haskell repo](https://github.com/fpco/lts-haskell/issues/new) 81 | * Specify the LTS major version you would like your package to go into (e.g., lts-3) 82 | * Provide a list of packages you would like added, and if relevant, any upper bounds on those packages 83 | * Be patient! The LTS releases are by their nature more conservative than nightly, and therefore adding new packages is a more manual process. The Stackage curators will try to get to your issue quickly, but there may be some delay. 84 | -------------------------------------------------------------------------------- /CURATORS.md: -------------------------------------------------------------------------------- 1 | This is a collection of instructions covering the processes that the Stackage curators - the 2 | guys who maintain the Stackage project itself - should be doing on a regular basis. 3 | Originally this was handled largely by Michael Snoyman, 4 | but now we are a team of 4 people handling requests weekly in rotation. 5 | Curation activities are mostly automated, and do not take up a significant amount of time. 6 | 7 | ## Workflow 8 | 9 | This section sketches out at a high level how the entire Stackage build/curation 10 | process works: 11 | 12 | * [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage 13 | * [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly 14 | * stackage-curator can check that build plan to ensure all version bounds are consistent 15 | * The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests 16 | * Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds 17 | * The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh) 18 | * When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly) 19 | * Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell) 20 | * Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS 21 | 22 | ## Pull requests 23 | 24 | The typical story on pull requests is: if Travis accepts it, and the author 25 | only added packages under his/her own name, merge it. If the build later fails 26 | (see below), then block the package until it's fixed. 27 | 28 | ## Fixing bounds issues 29 | 30 | The most common activity you'll deal with in Stackage curation is a version 31 | bound issue, usually a restrictive upper bound. You fix this by opening an 32 | issue on the Stackage repo about the problem, and modifying the 33 | build-constraints.yaml file to work around it in one of the ways below. Be sure 34 | to refer to the issue for workarounds added to that file. 35 | 36 | * __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately 37 | * __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark. 38 | * __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible 39 | 40 | ## Updating the content of the Docker image used for building 41 | 42 | ### Adding Debian packages for required system tools or libraries 43 | Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`. 44 | Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated 45 | since new packages are not added to the current lts release. 46 | 47 | Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to trusty which is the same version as the server). 48 | 49 | Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures. 50 | 51 | ### Upgrading GHC version 52 | The Dockerfile contains information on which GHC versions should be used. You 53 | can modify it and push it to Github to trigger a DockerHub build. The master 54 | branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the 55 | major version number (e.g., lts3 for lts-3.\*). 56 | 57 | Note that when starting a new LTS major release, you'll need to modify Docker 58 | Hub to create a new Docker tag for the relevant branch name. 59 | 60 | ### Getting the new image to the build server 61 | Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see 62 | below). Instead of pulling an unbounded number of images, I typically just 63 | delete all of the old images and let the new ones get downloaded: 64 | 65 | ``` 66 | docker rm $(docker ps -a -q) 67 | docker rmi $(docker images -q) 68 | ``` 69 | 70 | but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say. 71 | 72 | For a new GHC version you should also delete the cache directories on the stackage-build server to 73 | force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg: 74 | ``` 75 | rm -r nightly/work/builds/nightly/ 76 | ``` 77 | This should also be done when moving the Nightly docker image to a new version of Ubuntu. 78 | 79 | ## stackage-build server 80 | 81 | You'll need to get your SSH public key added to the machine. ~/.ssh/config info: 82 | 83 | ``` 84 | Host stackage-build 85 | User ubuntu 86 | Hostname ec2-52-5-20-252.compute-1.amazonaws.com 87 | ``` 88 | 89 | ### Running the build script 90 | 91 | We currently run the builds manually so make it easy to see when there are 92 | bounds issues that need to be corrected. Automated this would be even better, 93 | we're just not there yet. 94 | 95 | ``` 96 | # Run a nightly build 97 | /opt/stackage-build/stackage/automated/run-nightly.sh 98 | 99 | # Run an LTS minor bump 100 | /opt/stackage-build/stackage/automated/build.sh lts-2.17 101 | 102 | # Run an LTS major bump 103 | /opt/stackage-build/stackage/automated/build.sh lts-3.0 104 | ``` 105 | 106 | Recommended: run these from inside a `screen` session. If you get version bound 107 | problems on nightly or LTS major, you need to fix build-constraints.yaml (see 108 | info above). For an LTS minor bump, you'll typically want to use the 109 | `CONSTRAINTS` environment variable, e.g.: 110 | 111 | ``` 112 | CONSTRAINTS='--constraint "conduit < 1.4.5" --constraint "criterion < 1.2.3"' /opt/stackage-build/stackage/automated/build.sh lts-2.17 113 | ``` 114 | 115 | Valid arguments to include in this environment variable: 116 | 117 | * `--constraint` to modify an upper or lower bound 118 | * `--add-package` to add a brand new package 119 | * `--expect-test-failure` to expect tests to fail 120 | * `--expect-haddock-failure` to expect haddocks to fail 121 | 122 | If a build fails for bounds reasons, see all of the advice above. If the code 123 | itself doesn't build, or tests fail, open up an issue and then either put in a 124 | version bound to avoid that version or something else. It's difficult to give 125 | universal advice on how to solve things, since each situation is unique. Let's 126 | develop this advice over time. For now: if you're not sure, ask Michael for 127 | guidance. 128 | 129 | ### Timing 130 | 131 | A looping script on the build server keeps trying to build nightly 132 | with `sleep 30m` interleaved. It only publishes the nightly once per 133 | day. This way new package versions or build failures can be caught 134 | early and hopefully the nightlies will be timely. 135 | 136 | LTS minor bumps typically are run on Sundays. 137 | 138 | ### Website sync debugging (and other out of disk space errors) 139 | 140 | * You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem 141 | * There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally 142 | * You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it 143 | 144 | ### Wiping the cache 145 | 146 | Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`. 147 | --------------------------------------------------------------------------------