├── .gitignore ├── l10n ├── build ├── upload └── setup ├── mingw ├── build ├── reblame ├── upload └── setup ├── rust ├── build ├── reblame ├── upload └── setup ├── comm-central ├── build ├── reblame ├── upload ├── repo_files.py └── setup ├── comm-esr60 ├── build ├── reblame ├── upload └── setup ├── comm-esr68 ├── build ├── reblame ├── upload └── setup ├── comm-esr78 ├── build ├── reblame ├── upload └── setup ├── comm-esr91 ├── build ├── reblame ├── upload └── setup ├── ecma262 ├── build ├── reblame ├── upload └── setup ├── mingw-moz ├── build ├── upload └── setup ├── mozilla-build ├── build ├── reblame ├── upload └── setup ├── mozilla-mobile ├── build ├── upload └── setup ├── whatwg-html ├── build ├── reblame ├── upload └── setup ├── comm-esr102 ├── build ├── reblame ├── upload └── setup ├── comm-esr115 ├── build ├── reblame ├── upload └── setup ├── comm-esr128 ├── build ├── reblame ├── upload └── setup ├── firefox-main ├── disabled-checks │ ├── inputs │ │ └── analysis │ │ │ ├── xpidl │ │ │ ├── xpctest_params__def_testOctet__json │ │ │ └── xpctest_params__def_testOctet__html │ │ │ └── rust │ │ │ ├── third_party__euclid__def__json │ │ │ ├── xpidl__nsichannel__json │ │ │ ├── xpidl__nsichannel__html │ │ │ ├── first_party__webrender__moz2d__def__json │ │ │ ├── stdlib__btreemap__def__json │ │ │ ├── build__webrender_shader__json │ │ │ ├── first_party__webrender__moz2d__def__html │ │ │ ├── stdlib__btreemap__def__html │ │ │ └── build__webrender_shader__html │ └── snapshots │ │ └── analysis │ │ ├── rust │ │ ├── check_glob@third_party__euclid__def__json.snap │ │ ├── check_glob@xpidl__nsichannel__json.snap │ │ ├── check_glob@stdlib__btreemap__def__json.snap │ │ ├── check_glob@build__webrender_shader__json.snap │ │ ├── check_glob@first_party__webrender__moz2d__def__json.snap │ │ ├── check_glob@first_party__webrender__moz2d__def__html.snap │ │ ├── check_glob@stdlib__btreemap__def__html.snap │ │ ├── check_glob@xpidl__nsichannel__html.snap │ │ └── check_glob@build__webrender_shader__html.snap │ │ └── xpidl │ │ ├── check_glob@xpctest_params__def_testOctet__json.snap │ │ └── check_glob@xpctest_params__def_testOctet__html.snap ├── repo_files.py ├── upload └── build ├── mozilla-vpn-client ├── build ├── reblame ├── upload └── setup ├── version-control-tools ├── build ├── reblame ├── upload └── setup ├── mozsearch-mozilla ├── build ├── reblame ├── upload └── setup ├── mozsearch-tests ├── reblame ├── upload ├── build └── setup ├── llvm ├── reblame ├── build ├── upload └── setup ├── nss ├── reblame ├── build ├── upload └── setup ├── glean ├── reblame ├── build ├── upload └── setup ├── graphviz ├── reblame ├── upload ├── build └── setup ├── mozsearch ├── reblame ├── upload ├── build └── setup ├── firefox-beta ├── upload ├── build ├── setup └── repo_files.py ├── firefox-esr115 ├── upload ├── build ├── setup └── repo_files.py ├── firefox-esr128 ├── upload ├── build ├── setup └── repo_files.py ├── firefox-esr140 ├── upload ├── build ├── setup └── repo_files.py ├── firefox-release ├── upload ├── build ├── setup └── repo_files.py ├── mozilla-esr17 ├── build ├── setup └── repo_files.py ├── mozilla-esr31 ├── build ├── setup └── repo_files.py ├── mozilla-esr45 ├── build ├── setup └── repo_files.py ├── mozilla-esr60 ├── build ├── setup └── repo_files.py ├── mozilla-esr102 ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-esr68 ├── build ├── repo_files.py └── setup ├── mozilla-esr78 ├── build ├── repo_files.py └── setup ├── mozilla-beta ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-elm ├── build ├── setup └── repo_files.py ├── mozilla-cedar ├── build ├── setup └── repo_files.py ├── mozilla-cypress ├── build ├── setup └── repo_files.py ├── mozilla-esr115 ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-esr128 ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-esr140 ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-release ├── build ├── upload ├── setup └── repo_files.py ├── mozilla-esr91 ├── build ├── upload ├── setup └── repo_files.py ├── wubkat ├── reblame ├── repo_files.py ├── upload ├── last-successful-build-to-git-commit.sh ├── setup └── build ├── shared ├── git-maintenance.sh ├── resolve-gecko-revs.sh ├── fetch-gecko-tarball.sh ├── process-gecko-analysis.sh ├── collapse-generated-files.sh ├── checkout-gecko-repos.sh └── rebuild-blame.sh ├── firefox-shared ├── git-bare-maintenance.sh ├── resolve-gecko-revs.sh ├── process-gecko-analysis.sh └── collapse-generated-files.sh ├── CODE_OF_CONDUCT.md ├── just-llvm.json ├── mozilla-central ├── upload ├── repo_files.py ├── reblame └── build ├── just-graphviz.json ├── review-build-check-results.sh ├── just-mc.json ├── just-fm.json ├── config5.json ├── tree-list.js ├── just-mozsearch.json ├── config6.json ├── config1.json ├── config2old.json ├── config1old.json └── config2.json /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | /mozilla-central/checks/explanations/ 3 | -------------------------------------------------------------------------------- /l10n/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | -------------------------------------------------------------------------------- /mingw/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /rust/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-central/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | -------------------------------------------------------------------------------- /comm-esr60/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-esr68/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-esr78/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-esr91/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /ecma262/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /mingw-moz/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /mozilla-build/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | -------------------------------------------------------------------------------- /mozilla-mobile/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | -------------------------------------------------------------------------------- /whatwg-html/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | -------------------------------------------------------------------------------- /comm-esr102/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-esr115/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /comm-esr128/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/xpidl/xpctest_params__def_testOctet__json: -------------------------------------------------------------------------------- 1 | filter-analysis js/xpconnect/tests/idl/xpctest_params.idl -k idl -i "nsIXPCTestParams::testOctet" 2 | -------------------------------------------------------------------------------- /mozilla-vpn-client/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/third_party__euclid__def__json: -------------------------------------------------------------------------------- 1 | filter-analysis "third_party/rust/euclid/src/rect.rs" -k def -s "euclid::rect::Rect" | production-filter 2 | -------------------------------------------------------------------------------- /version-control-tools/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | -------------------------------------------------------------------------------- /mozsearch-mozilla/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | -------------------------------------------------------------------------------- /mozsearch-tests/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # no blame 8 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/xpidl__nsichannel__json: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/dist/xpcrs/rt/nsIChannel.rs" -k def -s "xpcom::interfaces::idl::nsIChannel" | production-filter 2 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/xpidl__nsichannel__html: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/dist/xpcrs/rt/nsIChannel.rs" -k def -s "xpcom::interfaces::idl::nsIChannel" | show-html | production-filter 2 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/xpidl/xpctest_params__def_testOctet__html: -------------------------------------------------------------------------------- 1 | filter-analysis js/xpconnect/tests/idl/xpctest_params.idl -k idl -i "nsIXPCTestParams::testOctet" | show-html | production-filter 2 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/first_party__webrender__moz2d__def__json: -------------------------------------------------------------------------------- 1 | filter-analysis "gfx/webrender_bindings/src/moz2d_renderer.rs" -k def -s "webrender_bindings::moz2d_renderer::CacheKey" | production-filter 2 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/stdlib__btreemap__def__json: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/__RUST_STDLIB__/alloc/src/collections/btree/map.rs" -k def -s "alloc::collections::btree::map::BTreeMap" | production-filter 2 | -------------------------------------------------------------------------------- /llvm/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base llvm $* 8 | -------------------------------------------------------------------------------- /nss/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base nss $* 8 | -------------------------------------------------------------------------------- /rust/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base rust $* 8 | -------------------------------------------------------------------------------- /glean/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base glean $* 8 | -------------------------------------------------------------------------------- /mingw/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base mingw $* 8 | -------------------------------------------------------------------------------- /ecma262/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base ecma262 $* 8 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/build__webrender_shader__json: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/__linux64__/__RUST_BUILD_SCRIPT__/webrender/shaders.rs" -k def -s "webrender::shader_source::OPTIMIZED_SHADERS" | production-filter 2 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/first_party__webrender__moz2d__def__html: -------------------------------------------------------------------------------- 1 | filter-analysis "gfx/webrender_bindings/src/moz2d_renderer.rs" -k def -s "webrender_bindings::moz2d_renderer::CacheKey" | show-html | production-filter 2 | -------------------------------------------------------------------------------- /graphviz/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base graphviz $* 8 | -------------------------------------------------------------------------------- /mozsearch/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base mozsearch $* 8 | -------------------------------------------------------------------------------- /comm-esr102/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr102 $* 8 | -------------------------------------------------------------------------------- /comm-esr115/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr115 $* 8 | -------------------------------------------------------------------------------- /comm-esr128/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr128 $* 8 | -------------------------------------------------------------------------------- /comm-esr60/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr60 $* 8 | -------------------------------------------------------------------------------- /comm-esr68/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr68 $* 8 | -------------------------------------------------------------------------------- /comm-esr78/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr78 $* 8 | -------------------------------------------------------------------------------- /comm-esr91/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-esr91 $* 8 | -------------------------------------------------------------------------------- /firefox-beta/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # firefox-main handles all uploads of the firefox-shared tarballs 8 | -------------------------------------------------------------------------------- /firefox-esr115/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # firefox-main handles all uploads of the firefox-shared tarballs 8 | -------------------------------------------------------------------------------- /firefox-esr128/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # firefox-main handles all uploads of the firefox-shared tarballs 8 | -------------------------------------------------------------------------------- /firefox-esr140/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # firefox-main handles all uploads of the firefox-shared tarballs 8 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/stdlib__btreemap__def__html: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/__RUST_STDLIB__/alloc/src/collections/btree/map.rs" -k def -s "alloc::collections::btree::map::BTreeMap" | show-html | production-filter 2 | -------------------------------------------------------------------------------- /mingw-moz/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # Don't upload anything here. The mingw repo will do the uploading 8 | -------------------------------------------------------------------------------- /whatwg-html/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base whatwg-html $* 8 | -------------------------------------------------------------------------------- /comm-central/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base comm-central $* 8 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/inputs/analysis/rust/build__webrender_shader__html: -------------------------------------------------------------------------------- 1 | filter-analysis "__GENERATED__/__linux64__/__RUST_BUILD_SCRIPT__/webrender/shaders.rs" -k def -s "webrender::shader_source::OPTIMIZED_SHADERS" | show-html | production-filter 2 | -------------------------------------------------------------------------------- /firefox-release/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # firefox-main handles all uploads of the firefox-shared tarballs 8 | -------------------------------------------------------------------------------- /mozilla-build/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base mozilla-build $* 8 | -------------------------------------------------------------------------------- /mozsearch-mozilla/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base mozsearch-blame $* 8 | -------------------------------------------------------------------------------- /mozilla-vpn-client/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base mozilla-vpn-client $* 8 | -------------------------------------------------------------------------------- /version-control-tools/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base version-control-tools $* 8 | -------------------------------------------------------------------------------- /mozilla-esr17/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # Don't do anything here. It's not likely that we'll 8 | # want to do Rust/C++ indexing for ESR17, ever. 9 | -------------------------------------------------------------------------------- /mozilla-esr31/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # Don't do anything here. It's not likely that we'll 8 | # want to do Rust/C++ indexing for ESR31, ever. 9 | -------------------------------------------------------------------------------- /mozilla-esr45/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # Don't do anything here. It's not likely that we'll 8 | # want to do Rust/C++ indexing for ESR45, ever. 9 | -------------------------------------------------------------------------------- /mozilla-esr60/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # Don't do anything here. It's not likely that we'll 8 | # want to do Rust/C++ indexing for ESR60, ever. 9 | -------------------------------------------------------------------------------- /glean/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | pushd $GIT_ROOT 8 | rust-analyzer scip . 9 | mkdir -p $OBJDIR 10 | mv index.scip $OBJDIR/rust.scip 11 | popd 12 | -------------------------------------------------------------------------------- /mozilla-esr102/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # ESR102 no longer has taskcluster/treeherder jobs and so we no longer can use 8 | # it for semantic indexing. 9 | -------------------------------------------------------------------------------- /mozilla-esr68/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # ESR68 no longer has taskcluster/treeherder jobs and so we no longer can use it 8 | # for semantic indexing. 9 | -------------------------------------------------------------------------------- /mozilla-esr78/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # ESR78 no longer has taskcluster/treeherder jobs and so we no longer can use it 8 | # for semantic indexing. 9 | -------------------------------------------------------------------------------- /mozilla-beta/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-elm/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-cedar/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-cypress/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-esr115/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-esr128/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-esr140/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-release/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /firefox-beta/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /firefox-esr115/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /firefox-esr128/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /firefox-esr140/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-esr91/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # ESR91 no longer has taskcluster/treeherder jobs and so we no longer can use it 8 | # for semantic indexing. 9 | 10 | -------------------------------------------------------------------------------- /firefox-release/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | -------------------------------------------------------------------------------- /mozilla-beta/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes beta commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozilla-esr102/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes esr102 commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozilla-esr115/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes esr115 commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozilla-esr128/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes esr128 commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozilla-esr140/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes esr140 commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozilla-esr91/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes esr91 commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /mozsearch-tests/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # We reuse the mozsearch tarball for downloads; we are not responsible for 10 | # uploading anything. 11 | -------------------------------------------------------------------------------- /wubkat/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # XXX this won't work with lz4 yet, so just error 8 | exit 1 9 | #$CONFIG_REPO/shared/rebuild-blame.sh --tarball-base wubkat $* 10 | -------------------------------------------------------------------------------- /mozilla-release/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # The mozilla-central repo uploads an updated gecko.tar which 8 | # includes release commits, so we don't need to do anything here. 9 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@third_party__euclid__def__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "NORM:11-15", 8 | "target": 1, 9 | "kind": "def", 10 | "pretty": "euclid::rect::Rect", 11 | "sym": "euclid::rect::Rect" 12 | } 13 | ] 14 | -------------------------------------------------------------------------------- /nss/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # We no longer build NSS for semantic purposes. The tentative plan would be for 8 | # NSS to run the clang indexer on its taskcluster instance and generate 9 | # artifacts like we do for m-c trees. 10 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@xpidl__nsichannel__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "NORM:11-21", 8 | "target": 1, 9 | "kind": "def", 10 | "pretty": "xpcom::interfaces::idl::nsIChannel", 11 | "sym": "xpcom::interfaces::idl::nsIChannel" 12 | } 13 | ] 14 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@stdlib__btreemap__def__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "NORM:11-19", 8 | "target": 1, 9 | "kind": "def", 10 | "pretty": "alloc::collections::btree::map::BTreeMap", 11 | "sym": "alloc::collections::btree::map::BTreeMap" 12 | } 13 | ] 14 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@build__webrender_shader__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "NORM:17-34", 8 | "target": 1, 9 | "kind": "def", 10 | "pretty": "webrender::shader_source::OPTIMIZED_SHADERS", 11 | "sym": "webrender::shader_source::OPTIMIZED_SHADERS" 12 | } 13 | ] 14 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@first_party__webrender__moz2d__def__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "NORM:7-15", 8 | "target": 1, 9 | "kind": "def", 10 | "pretty": "webrender_bindings::moz2d_renderer::CacheKey", 11 | "sym": "webrender_bindings::moz2d_renderer::CacheKey" 12 | } 13 | ] 14 | -------------------------------------------------------------------------------- /shared/git-maintenance.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | GIT_REPO_DIR="$1" 8 | 9 | if [[ $(date +%A) == "Saturday" ]]; then 10 | date 11 | git --git-dir="${GIT_REPO_DIR}/.git" gc 12 | date 13 | git --git-dir="${GIT_REPO_DIR}/.git" cinnabar fsck || true 14 | date 15 | fi 16 | -------------------------------------------------------------------------------- /firefox-shared/git-bare-maintenance.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | GIT_BARE_REPO_DIR="$1" 8 | 9 | if [[ $(date +%A) == "Saturday" ]]; then 10 | date 11 | git --git-dir="${GIT_BARE_REPO_DIR}" gc 12 | date 13 | git --git-dir="${GIT_BARE_REPO_DIR}" cinnabar fsck || true 14 | date 15 | fi 16 | -------------------------------------------------------------------------------- /wubkat/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | # webkit does not have IPDL! 3 | return False 4 | 5 | def filter_webidl(path): 6 | # webkit does not have WebIDL! 7 | return False 8 | 9 | def filter_idl(path): 10 | # webkit does not have XPIDL! 11 | return False 12 | 13 | def filter_js(path): 14 | # webkit does not have XUL, and its `.inc` files appear to at least 15 | # sometimes be metal shaders! 16 | return path.endswith(".js") 17 | -------------------------------------------------------------------------------- /mozilla-esr102/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | # ESR102 isn't getting any new commits, so we can hard-code the last revision 10 | # for simplicity in checkout-gecko-repos.sh. 11 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr102" "esr102" "9df9599ab93f731765afd98cb450c8ffdd3eafdf" 12 | -------------------------------------------------------------------------------- /mozilla-esr91/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | # ESR91 isn't getting any new commits, so we can hard-code the last revision 10 | # for simplicity in checkout-gecko-repos.sh. 11 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr91" "esr91" "f21620428eb73c49891ba403bdb4ed799dc32c21" 12 | 13 | -------------------------------------------------------------------------------- /llvm/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # Add the special clang flags. 10 | $MOZSEARCH_PATH/scripts/indexer-setup.py > $INDEX_ROOT/config 11 | . $INDEX_ROOT/config 12 | 13 | mkdir -p $OBJDIR 14 | 15 | cd $OBJDIR 16 | cmake -S $FILES_ROOT/llvm -B $OBJDIR -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld" -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_LINKER=lld 17 | cmake --build . 18 | 19 | cd - 20 | 21 | date 22 | -------------------------------------------------------------------------------- /mozilla-esr102/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | -------------------------------------------------------------------------------- /mozilla-esr78/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | -------------------------------------------------------------------------------- /mozilla-esr91/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/xpidl/check_glob@xpctest_params__def_testOctet__json.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&json_results" 4 | --- 5 | [ 6 | { 7 | "loc": "24:24-33", 8 | "target": 1, 9 | "kind": "idl", 10 | "pretty": "nsIXPCTestParams::testOctet", 11 | "sym": "XPIDL_nsIXPCTestParams_testOctet" 12 | }, 13 | { 14 | "loc": "24:24-33", 15 | "source": 1, 16 | "syntax": "idl", 17 | "pretty": "IDL method nsIXPCTestParams::testOctet", 18 | "sym": "XPIDL_nsIXPCTestParams_testOctet" 19 | } 20 | ] 21 | -------------------------------------------------------------------------------- /mozilla-esr78/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | # ESR78 isn't getting any new commits, so we can hard-code the last revision 10 | # for simplicity in checkout-gecko-repos.sh. 11 | # Disclaimer: this rev is actually one behind the current tip which was about 12 | # removing the cron jobs. 13 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr78" "esr78" "a326cc6a77c6b3ad702ca999b0058693b10e9f58" 14 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@first_party__webrender__moz2d__def__html.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&aggr_str" 4 | --- 5 |
6 |
7 |
8 |
9 | struct CacheKey { 10 | 11 |
12 | 13 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@stdlib__btreemap__def__html.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&aggr_str" 4 | --- 5 |
6 |
7 |
8 |
9 | pub struct BTreeMap< 10 | 11 |
12 | 13 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@xpidl__nsichannel__html.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&aggr_str" 4 | --- 5 |
6 |
7 |
8 |
9 | pub struct nsIChannel { 10 | 11 |
12 | 13 | -------------------------------------------------------------------------------- /mozilla-beta/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-beta 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "beta" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-esr115/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr115 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "esr115" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-esr128/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr128 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "esr128" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-esr140/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr140 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "esr140" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-release/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-release 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "release" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-esr17/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes esr17 commits, so we don't need our own upload script. 10 | 11 | date 12 | 13 | # ESR17 isn't getting any new commits, so we can hard-code the last revision 14 | # for simplicity in checkout-gecko-repos.sh. 15 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr17" "esr17" "7d97fafba271606958a57f2b9581da4e9de008a4" 16 | 17 | date 18 | -------------------------------------------------------------------------------- /mozilla-esr31/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes esr31 commits, so we don't need our own upload script. 10 | 11 | date 12 | 13 | # ESR31 isn't getting any new commits, so we can hard-code the last revision 14 | # for simplicity in checkout-gecko-repos.sh. 15 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr31" "esr31" "777e60ca88535403b547fcb3271f4baf63c17af2" 16 | 17 | date 18 | -------------------------------------------------------------------------------- /mozilla-esr45/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes esr45 commits, so we don't need our own upload script. 10 | 11 | date 12 | 13 | # ESR45 isn't getting any new commits, so we can hard-code the last revision 14 | # for simplicity in checkout-gecko-repos.sh. 15 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr45" "esr45" "dcc6d7a0dc0056f23d4f0564b708513836961f1a" 16 | 17 | date 18 | -------------------------------------------------------------------------------- /mozilla-esr60/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes esr60 commits, so we don't need our own upload script. 10 | 11 | date 12 | 13 | # ESR60 isn't getting any new commits, so we can hard-code the last revision 14 | # for simplicity in checkout-gecko-repos.sh. 15 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr60" "esr60" "1aeaa33a64f9873fdeb8f986ef71ec35aa672347" 16 | 17 | date 18 | -------------------------------------------------------------------------------- /firefox-beta/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-beta 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/firefox-shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/firefox-shared/checkout-gecko-repos.sh $REVISION_TREE "beta" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/firefox-shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /firefox-esr115/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr115 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/firefox-shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/firefox-shared/checkout-gecko-repos.sh $REVISION_TREE "esr115" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/firefox-shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /firefox-esr128/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr128 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/firefox-shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/firefox-shared/checkout-gecko-repos.sh $REVISION_TREE "esr128" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/firefox-shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /firefox-esr140/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-esr140 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/firefox-shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/firefox-shared/checkout-gecko-repos.sh $REVISION_TREE "esr140" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/firefox-shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /firefox-release/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | 9 | REVISION_TREE=mozilla-release 10 | REVISION_ID=latest 11 | 12 | date 13 | 14 | source $CONFIG_REPO/firefox-shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 15 | 16 | date 17 | 18 | $CONFIG_REPO/firefox-shared/checkout-gecko-repos.sh $REVISION_TREE "release" "$INDEXED_HG_REV" 19 | 20 | date 21 | 22 | $CONFIG_REPO/firefox-shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 23 | 24 | date 25 | -------------------------------------------------------------------------------- /mozilla-esr45/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | 24 | def filter_idl(path): 25 | # ESR45's XPIDL parser is python2 only 26 | return False 27 | -------------------------------------------------------------------------------- /mozilla-esr60/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | 24 | def filter_idl(path): 25 | # ESR60's XPIDL parser is python2 only 26 | return False 27 | -------------------------------------------------------------------------------- /mozilla-esr68/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | 24 | def filter_idl(path): 25 | # ESR68's XPIDL parser is python2 only 26 | return False 27 | -------------------------------------------------------------------------------- /nss/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading NSPR 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf nss.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/nss.tar searchfox.repositories nss.tar 14 | rm nss.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading NSS blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf nss-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/nss-blame.tar searchfox.repositories nss-blame.tar 24 | rm nss-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /llvm/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading llvm 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf llvm.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/llvm.tar searchfox.repositories llvm.tar 14 | rm llvm.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading llvm blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf llvm-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/llvm-blame.tar searchfox.repositories llvm-blame.tar 24 | rm llvm-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /rust/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading rust 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf rust.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/rust.tar searchfox.repositories rust.tar 14 | rm rust.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading rust blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf rust-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/rust-blame.tar searchfox.repositories rust-blame.tar 24 | rm rust-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-esr115/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def modify_file_list(lines, config): 10 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 11 | return lines 12 | 13 | def filter_js(path): 14 | if 'js/src/tests' in path or 'jit-test' in path: 15 | return False 16 | return True 17 | 18 | def filter_html(path): 19 | if 'testing/web-platform/' in path: 20 | return False 21 | return True 22 | 23 | def filter_css(path): 24 | if 'testing/web-platform/' in path: 25 | return False 26 | return True 27 | -------------------------------------------------------------------------------- /glean/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading glean 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf glean.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/glean.tar searchfox.repositories glean.tar 14 | rm glean.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading glean blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf glean-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/glean-blame.tar searchfox.repositories glean-blame.tar 24 | rm glean-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mingw/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mingw 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf mingw.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/mingw.tar searchfox.repositories mingw.tar 14 | rm mingw.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading mingw blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf mingw-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/mingw-blame.tar searchfox.repositories mingw-blame.tar 24 | rm mingw-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /ecma262/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading ecma262 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf ecma262.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/ecma262.tar searchfox.repositories ecma262.tar 14 | rm ecma262.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading ecma262 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf ecma262-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/ecma262-blame.tar searchfox.repositories ecma262-blame.tar 24 | rm ecma262-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-esr17/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | 24 | def filter_idl(path): 25 | # ESR17 doesn't even have an XPIDL parser we'll recognize but let's bypass 26 | # trying to use a modern one. 27 | return False 28 | -------------------------------------------------------------------------------- /mozilla-esr31/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | return False 8 | 9 | def filter_js(path): 10 | if 'js/src/tests' in path or 'jit-test' in path: 11 | return False 12 | return True 13 | 14 | def filter_html(path): 15 | if 'testing/web-platform/' in path: 16 | return False 17 | return True 18 | 19 | def filter_css(path): 20 | if 'testing/web-platform/' in path: 21 | return False 22 | return True 23 | 24 | def filter_idl(path): 25 | # ESR31 doesn't even have an XPIDL parser we'll recognize but let's bypass 26 | # trying to use a modern one. 27 | return False 28 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Community Participation Guidelines 2 | 3 | This repository is governed by Mozilla's code of conduct and etiquette guidelines. 4 | For more details, please read the 5 | [Mozilla Community Participation Guidelines](https://www.mozilla.org/about/governance/policies/participation/). 6 | 7 | ## How to Report 8 | For more information on how to report violations of the Community Participation Guidelines, please read our '[How to Report](https://www.mozilla.org/about/governance/policies/participation/reporting/)' page. 9 | 10 | 16 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/xpidl/check_glob@xpctest_params__def_testOctet__html.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&aggr_str" 4 | --- 5 |
6 |
7 |
8 |
9 | octet testOctet(in octet a, inout octet b); 10 | 11 |
12 | 13 | -------------------------------------------------------------------------------- /mozilla-elm/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes elm commits, so we don't need our own upload script. 10 | 11 | REVISION_TREE=elm 12 | REVISION_ID=latest 13 | 14 | date 15 | 16 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 17 | 18 | date 19 | 20 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "elm" "$INDEXED_HG_REV" 21 | 22 | date 23 | 24 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 25 | 26 | date 27 | -------------------------------------------------------------------------------- /mozilla-esr68/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes esr68 commits, so we don't need our own upload script. 10 | 11 | date 12 | 13 | # ESR68 isn't getting any new commits, so we can hard-code the last revision 14 | # for simplicity in checkout-gecko-repos.sh. 15 | # Disclaimer: this rev is actually one behind the current tip which was about 16 | # removing the cron jobs. 17 | $CONFIG_REPO/shared/checkout-gecko-repos.sh "mozilla-esr68" "esr68" "f72861bb0a1c7914e5f6af56e632f2197c1cd984" 18 | 19 | date 20 | -------------------------------------------------------------------------------- /comm-esr60/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr60 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr60.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr60.tar searchfox.repositories comm-esr60.tar 14 | rm comm-esr60.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr60 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr60-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr60-blame.tar searchfox.repositories comm-esr60-blame.tar 24 | rm comm-esr60-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-esr68/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr68 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr68.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr68.tar searchfox.repositories comm-esr68.tar 14 | rm comm-esr68.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr68 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr68-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr68-blame.tar searchfox.repositories comm-esr68-blame.tar 24 | rm comm-esr68-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-esr78/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr78 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr78.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr78.tar searchfox.repositories comm-esr78.tar 14 | rm comm-esr78.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr78 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr78-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr78-blame.tar searchfox.repositories comm-esr78-blame.tar 24 | rm comm-esr78-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-esr91/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr91 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr91.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr91.tar searchfox.repositories comm-esr91.tar 14 | rm comm-esr91.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr91 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr91-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr91-blame.tar searchfox.repositories comm-esr91-blame.tar 24 | rm comm-esr91-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozsearch-tests/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | ## Attempt to just reuse the `make build-test-repo` target 10 | # 11 | # Our high level goal is to produce output that looks identical to what happens 12 | # when someone does `cd /vagrant; make build-test-repo` in a docker VM. And we 13 | # want to do this without forking the relevant build script. 14 | 15 | # The tests config.json file is here: 16 | TESTS_ROOT=$INDEX_ROOT/git/tests 17 | # The "build" script is in the "tests" dir under that: 18 | TESTS_CONFIG_DIR=$TESTS_ROOT/tests 19 | 20 | # just transfer control into the build script 21 | exec $TESTS_CONFIG_DIR/build 22 | -------------------------------------------------------------------------------- /mozilla-cedar/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes cedar commits, so we don't need our own upload script. 10 | 11 | REVISION_TREE=cedar 12 | REVISION_ID=latest 13 | 14 | date 15 | 16 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 17 | 18 | date 19 | 20 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "cedar" "$INDEXED_HG_REV" 21 | 22 | date 23 | 24 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 25 | 26 | date 27 | -------------------------------------------------------------------------------- /mozilla-mobile/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mozilla-mobile 10 | pushd $INDEX_ROOT 11 | 12 | echo "Running maintenance on mozilla-mobile submodules..." 13 | pushd ./git 14 | git submodule foreach --recursive "${CONFIG_REPO}/shared/git-maintenance.sh \$PWD" 15 | popd 16 | echo "Running maintenance on mozilla-mobile supermodule..." 17 | $CONFIG_REPO/shared/git-maintenance.sh ./git 18 | echo "Maintenance complete!" 19 | 20 | tar cf mozilla-mobile.tar git 21 | $AWS_ROOT/upload.py $INDEX_ROOT/mozilla-mobile.tar searchfox.repositories mozilla-mobile.tar 22 | rm mozilla-mobile.tar 23 | 24 | popd 25 | 26 | date 27 | -------------------------------------------------------------------------------- /comm-esr102/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr102 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr102.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr102.tar searchfox.repositories comm-esr102.tar 14 | rm comm-esr102.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr102 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr102-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr102-blame.tar searchfox.repositories comm-esr102-blame.tar 24 | rm comm-esr102-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-esr115/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr115 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf comm-esr115.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr115.tar searchfox.repositories comm-esr115.tar 14 | rm comm-esr115.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr115 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-esr115-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr115-blame.tar searchfox.repositories comm-esr115-blame.tar 24 | rm comm-esr115-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-cypress/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # See ../mozilla-central/setup for documentation 8 | # The mozilla-central repo also uploads an updated gecko.tar which 9 | # includes cypress commits, so we don't need our own upload script. 10 | 11 | REVISION_TREE=cypress 12 | REVISION_ID=latest 13 | 14 | date 15 | 16 | source $CONFIG_REPO/shared/resolve-gecko-revs.sh $REVISION_TREE $REVISION_ID 17 | 18 | date 19 | 20 | $CONFIG_REPO/shared/checkout-gecko-repos.sh $REVISION_TREE "cypress" "$INDEXED_HG_REV" 21 | 22 | date 23 | 24 | $CONFIG_REPO/shared/fetch-tc-artifacts.sh $REVISION_TREE $INDEXED_HG_REV "$PREEXISTING_HG_REV" 25 | 26 | date 27 | -------------------------------------------------------------------------------- /whatwg-html/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading WHATWG HTML 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf whatwg-html.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/whatwg-html.tar searchfox.repositories whatwg-html.tar 14 | rm whatwg-html.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading WHATWG HTML blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf whatwg-html-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/whatwg-html-blame.tar searchfox.repositories whatwg-html-blame.tar 24 | rm whatwg-html-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /just-llvm.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "llvm", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "llvm": { 9 | "priority": 3, 10 | "on_error": "continue", 11 | "cache": "codesearch", 12 | "index_path": "$WORKING/llvm", 13 | "files_path": "$WORKING/llvm/git", 14 | "git_path": "$WORKING/llvm/git", 15 | "git_blame_path": "$WORKING/llvm/blame", 16 | "github_repo": "https://github.com/llvm/llvm-project", 17 | "objdir_path": "$WORKING/llvm/objdir", 18 | "codesearch_path": "$WORKING/llvm/livegrep.idx", 19 | "codesearch_port": 8082, 20 | "scip_subtrees": {} 21 | } 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /graphviz/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading graphviz 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf - git | lz4 - graphviz.tar.lz4 13 | $AWS_ROOT/upload.py $INDEX_ROOT/graphviz.tar.lz4 searchfox.repositories graphviz.tar.lz4 14 | rm graphviz.tar.lz4 15 | popd 16 | 17 | date 18 | 19 | echo Uploading graphviz blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf - blame | lz4 - graphviz-blame.tar.lz4 23 | $AWS_ROOT/upload.py $INDEX_ROOT/graphviz-blame.tar.lz4 searchfox.repositories graphviz-blame.tar.lz4 24 | rm graphviz-blame.tar.lz4 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-build/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mozilla-build 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf mozilla-build.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/mozilla-build.tar searchfox.repositories mozilla-build.tar 14 | rm mozilla-build.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading mozilla-build blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf mozilla-build-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/mozilla-build-blame.tar searchfox.repositories mozilla-build-blame.tar 24 | rm mozilla-build-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-central/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | date 10 | 11 | echo Uploading Gecko 12 | pushd $INDEX_ROOT 13 | $CONFIG_REPO/shared/git-maintenance.sh ./git 14 | tar cf gecko.tar git 15 | $AWS_ROOT/upload.py $INDEX_ROOT/gecko.tar searchfox.repositories gecko.tar 16 | rm gecko.tar 17 | popd 18 | 19 | date 20 | 21 | echo Uploading Gecko blame 22 | pushd $INDEX_ROOT 23 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 24 | tar cf gecko-blame.tar blame 25 | $AWS_ROOT/upload.py $INDEX_ROOT/gecko-blame.tar searchfox.repositories gecko-blame.tar 26 | rm gecko-blame.tar 27 | popd 28 | 29 | date 30 | -------------------------------------------------------------------------------- /mozsearch/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mozsearch 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf - git | lz4 - mozsearch.tar.lz4 13 | $AWS_ROOT/upload.py $INDEX_ROOT/mozsearch.tar.lz4 searchfox.repositories mozsearch.tar.lz4 14 | rm mozsearch.tar.lz4 15 | popd 16 | 17 | date 18 | 19 | echo Uploading mozsearch blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf - blame | lz4 - mozsearch-blame.tar.lz4 23 | $AWS_ROOT/upload.py $INDEX_ROOT/mozsearch-blame.tar.lz4 searchfox.repositories mozsearch-blame.tar.lz4 24 | rm mozsearch-blame.tar.lz4 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-central/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-central 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar --exclude='git/mozilla' -cf comm-central.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-central.tar searchfox.repositories comm-central.tar 14 | rm comm-central.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-central blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf comm-central-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-central-blame.tar searchfox.repositories comm-central-blame.tar 24 | rm comm-central-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /l10n/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading l10n 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf - git | lz4 - firefox-l10n.tar.lz4 13 | $AWS_ROOT/upload.py $INDEX_ROOT/firefox-l10n.tar.lz4 searchfox.repositories firefox-l10n.tar.lz4 14 | rm firefox-l10n.tar.lz4 15 | popd 16 | 17 | date 18 | 19 | echo Uploading l10n blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf - blame | lz4 - firefox-l10n-blame.tar.lz4 23 | $AWS_ROOT/upload.py $INDEX_ROOT/firefox-l10n-blame.tar.lz4 searchfox.repositories firefox-l10n-blame.tar.lz4 24 | rm firefox-l10n-blame.tar.lz4 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /comm-esr128/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading comm-esr128 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf - git | lz4 - comm-esr128.tar.lz4 13 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr128.tar.lz4 searchfox.repositories comm-esr128.tar.lz4 14 | rm comm-esr128.tar.lz4 15 | popd 16 | 17 | date 18 | 19 | echo Uploading comm-esr128 blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf - blame | lz4 - comm-esr128-blame.tar.lz4 23 | $AWS_ROOT/upload.py $INDEX_ROOT/comm-esr128-blame.tar.lz4 searchfox.repositories comm-esr128-blame.tar.lz4 24 | rm comm-esr128-blame.tar.lz4 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /mozilla-vpn-client/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mozilla-vpn-client 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf mozilla-vpn-client.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/mozilla-vpn-client.tar searchfox.repositories mozilla-vpn-client.tar 14 | rm mozilla-vpn-client.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading mozilla-vpn-client blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf mozilla-vpn-client-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/mozilla-vpn-client-blame.tar searchfox.repositories mozilla-vpn-client-blame.tar 24 | rm mozilla-vpn-client-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /just-graphviz.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "graphviz", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "graphviz": { 9 | "priority": 3, 10 | "on_error": "continue", 11 | "cache": "codesearch", 12 | "index_path": "$WORKING/graphviz", 13 | "files_path": "$WORKING/graphviz/git", 14 | "git_path": "$WORKING/graphviz/git", 15 | "git_blame_path": "$WORKING/graphviz/blame", 16 | "github_repo": "https://gitlab.com/graphviz/graphviz/-", 17 | "walk_submodules": false, 18 | "objdir_path": "$WORKING/graphviz/objdir", 19 | "codesearch_path": "$WORKING/graphviz/livegrep.idx", 20 | "codesearch_port": 8082, 21 | "scip_subtrees": {} 22 | } 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /version-control-tools/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading v-c-t 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf version-control-tools.tar git 13 | $AWS_ROOT/upload.py $INDEX_ROOT/version-control-tools.tar searchfox.repositories version-control-tools.tar 14 | rm version-control-tools.tar 15 | popd 16 | 17 | date 18 | 19 | echo Uploading v-c-t blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf version-control-tools-blame.tar blame 23 | $AWS_ROOT/upload.py $INDEX_ROOT/version-control-tools-blame.tar searchfox.repositories version-control-tools-blame.tar 24 | rm version-control-tools-blame.tar 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /wubkat/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # Note! This ends up including the flatpak artifacts, and we want that. 10 | echo Uploading webkit 11 | pushd $INDEX_ROOT 12 | $CONFIG_REPO/shared/git-maintenance.sh ./git 13 | tar cf - git | lz4 -f - wubkat.tar.lz4 14 | $AWS_ROOT/upload.py $INDEX_ROOT/wubkat.tar.lz4 searchfox.repositories wubkat.tar.lz4 15 | rm wubkat.tar.lz4 16 | popd 17 | 18 | date 19 | 20 | echo Uploading webkit blame 21 | pushd $INDEX_ROOT 22 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 23 | tar cf - blame | lz4 -f - wubkat-blame.tar.lz4 24 | $AWS_ROOT/upload.py $INDEX_ROOT/wubkat-blame.tar.lz4 searchfox.repositories wubkat-blame.tar.lz4 25 | rm wubkat-blame.tar.lz4 26 | popd 27 | 28 | date 29 | -------------------------------------------------------------------------------- /mozsearch-mozilla/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Uploading mozsearch-mozilla 10 | pushd $INDEX_ROOT 11 | $CONFIG_REPO/shared/git-maintenance.sh ./git 12 | tar cf - git | lz4 - mozsearch-mozilla.tar.lz4 13 | $AWS_ROOT/upload.py $INDEX_ROOT/mozsearch-mozilla.tar.lz4 searchfox.repositories mozsearch-mozilla.tar.lz4 14 | rm mozsearch-mozilla.tar.lz4 15 | popd 16 | 17 | date 18 | 19 | echo Uploading mozsearch-mozilla blame 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/git-maintenance.sh ./blame 22 | tar cf - blame | lz4 - mozsearch-mozilla-blame.tar.lz4 23 | $AWS_ROOT/upload.py $INDEX_ROOT/mozsearch-mozilla-blame.tar.lz4 searchfox.repositories mozsearch-mozilla-blame.tar.lz4 24 | rm mozsearch-mozilla-blame.tar.lz4 25 | popd 26 | 27 | date 28 | -------------------------------------------------------------------------------- /graphviz/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # Add the special clang flags. 10 | # See $MOZSEARCH_PATH/scripts/ld-wrapper for WRAPPED_LD and --use-ld-wrapper. 11 | export WRAPPED_LD=ld 12 | $MOZSEARCH_PATH/scripts/indexer-setup.py --use-ld-wrapper > $INDEX_ROOT/config 13 | . $INDEX_ROOT/config 14 | 15 | rm -rf $OBJDIR 16 | mkdir -p $OBJDIR 17 | 18 | cd $FILES_ROOT 19 | # For git checkouts we need to run autogen. 20 | ./autogen.sh 21 | 22 | date 23 | 24 | cd $OBJDIR 25 | # As per https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html 26 | # we want to run configure from the objdir to set up the build tree here. 27 | $FILES_ROOT/configure 28 | 29 | date 30 | 31 | # And then we can run make here. 32 | make 33 | 34 | cd - 35 | 36 | date 37 | -------------------------------------------------------------------------------- /wubkat/last-successful-build-to-git-commit.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # This is evolved from dpino's script from: 3 | # https://gist.github.com/dpino/b324320652bb8b758acde123f9a3dbdc 4 | # with the evolution process happening there, which you can read if you like! 5 | # Many thanks to dpino! 6 | 7 | # Debugging this? Uncomment the following to see the commands as they run! 8 | #set -x 9 | 10 | BUILDER_NAME=${1:-GTK-Linux-64-bit-Release-Ubuntu-2204-Build} 11 | BUILD_INFO=$(curl "https://build.webkit.org/api/v2/builders/$BUILDER_NAME/builds?order=-number&limit=1&complete=true&state_string=build%20successful" 2>/dev/null) 12 | NUMBER=$(jq -Mr '.builds[0].number' <<< "$BUILD_INFO") 13 | BUILDER_ID=$(jq -Mr '.builds[0].builderid' <<< "$BUILD_INFO") 14 | CHANGES_INFO=$(curl "https://build.webkit.org/api/v2/builders/$BUILDER_ID/builds/$NUMBER/changes" 2>/dev/null) 15 | BUILD_REV=$(jq -Mr '.changes[0].revision' <<< "$CHANGES_INFO") 16 | 17 | echo "$BUILD_REV" 18 | -------------------------------------------------------------------------------- /review-build-check-results.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | if [[ $# -ne 3 ]]; then 8 | echo "Usage: $0 " 9 | echo " e.g.: $0 mozilla-central dev" 10 | echo "" 11 | echo "Run this inside the VM, checked out as 'config' under mozsearch" 12 | echo "" 13 | echo "Use the channel name 'release' if you want to check the production server." 14 | exit 1 15 | fi 16 | 17 | CONFIG_FILE=$1 18 | TREE_NAME=$2 19 | CHANNEL=$3 20 | 21 | if [[ $CHANNEL = "release" ]]; then 22 | CHANNEL_PREFIX= 23 | else 24 | CHANNEL_PREFIX=${CHANNEL}. 25 | fi 26 | 27 | export MOZSEARCH_PATH=/vagrant 28 | export CONFIG_REPO=/vagrant/config 29 | 30 | INSTA_FORCE_PASS=1 $MOZSEARCH_PATH/scripts/check-index.sh $CONFIG_FILE $TREE_NAME "" "https://${CHANNEL_PREFIX}searchfox.org/" 31 | cargo insta review --workspace-root=/vagrant/config/ 32 | -------------------------------------------------------------------------------- /firefox-esr115/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | return lines 20 | 21 | def filter_js(path): 22 | if 'js/src/tests' in path or 'jit-test' in path: 23 | return False 24 | return True 25 | 26 | def filter_html(path): 27 | if 'testing/web-platform/' in path: 28 | return False 29 | return True 30 | 31 | def filter_css(path): 32 | if 'testing/web-platform/' in path: 33 | return False 34 | return True 35 | -------------------------------------------------------------------------------- /firefox-esr128/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | return lines 20 | 21 | def filter_js(path): 22 | if 'js/src/tests' in path or 'jit-test' in path: 23 | return False 24 | return True 25 | 26 | def filter_html(path): 27 | if 'testing/web-platform/' in path: 28 | return False 29 | return True 30 | 31 | def filter_css(path): 32 | if 'testing/web-platform/' in path: 33 | return False 34 | return True 35 | -------------------------------------------------------------------------------- /firefox-esr140/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | return lines 20 | 21 | def filter_js(path): 22 | if 'js/src/tests' in path or 'jit-test' in path: 23 | return False 24 | return True 25 | 26 | def filter_html(path): 27 | if 'testing/web-platform/' in path: 28 | return False 29 | return True 30 | 31 | def filter_css(path): 32 | if 'testing/web-platform/' in path: 33 | return False 34 | return True 35 | -------------------------------------------------------------------------------- /mozilla-esr128/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | return lines 20 | 21 | def filter_js(path): 22 | if 'js/src/tests' in path or 'jit-test' in path: 23 | return False 24 | return True 25 | 26 | def filter_html(path): 27 | if 'testing/web-platform/' in path: 28 | return False 29 | return True 30 | 31 | def filter_css(path): 32 | if 'testing/web-platform/' in path: 33 | return False 34 | return True 35 | -------------------------------------------------------------------------------- /mozilla-esr140/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | return lines 20 | 21 | def filter_js(path): 22 | if 'js/src/tests' in path or 'jit-test' in path: 23 | return False 24 | return True 25 | 26 | def filter_html(path): 27 | if 'testing/web-platform/' in path: 28 | return False 29 | return True 30 | 31 | def filter_css(path): 32 | if 'testing/web-platform/' in path: 33 | return False 34 | return True 35 | -------------------------------------------------------------------------------- /llvm/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading llvm repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/llvm.tar 16 | tar xf llvm.tar 17 | rm llvm.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading llvm blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/llvm-blame.tar 30 | tar xf llvm-blame.tar 31 | rm llvm-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin main 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /firefox-main/disabled-checks/snapshots/analysis/rust/check_glob@build__webrender_shader__html.snap: -------------------------------------------------------------------------------- 1 | --- 2 | source: tests/test_check_insta.rs 3 | expression: "&aggr_str" 4 | --- 5 |
6 |
7 |
8 |
9 | pub static ref OPTIMIZED_SHADERS: HashMap<(ShaderVersion, &'static str), OptimizedSourceWithDigest> = { 10 | 11 |
12 | 13 | -------------------------------------------------------------------------------- /glean/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading glean repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/glean.tar 16 | tar xf glean.tar 17 | rm glean.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading glean blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/glean-blame.tar 30 | tar xf glean-blame.tar 31 | rm glean-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin main 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /mingw/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mingw repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mingw.tar 16 | tar xf mingw.tar 17 | rm mingw.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mingw blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mingw-blame.tar 30 | tar xf mingw-blame.tar 31 | rm mingw-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin master 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /ecma262/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading ecma262 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/ecma262.tar 16 | tar xf ecma262.tar 17 | rm ecma262.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading ecma262 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/ecma262-blame.tar 30 | tar xf ecma262-blame.tar 31 | rm ecma262-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin main 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /rust/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading rust repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/rust.tar 16 | tar xf rust.tar 17 | rm rust.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading rust blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/rust-blame.tar 30 | tar xf rust-blame.tar 31 | rm rust-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git fetch origin main 40 | git checkout origin/main 41 | popd 42 | 43 | echo Generating blame information 44 | build-blame $GIT_ROOT $BLAME_ROOT 45 | 46 | date 47 | -------------------------------------------------------------------------------- /firefox-beta/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /firefox-main/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /firefox-release/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-beta/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-cedar/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-central/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-cypress/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-elm/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /mozilla-release/repo_files.py: -------------------------------------------------------------------------------- 1 | def filter_ipdl(path): 2 | if 'ipc/ipdl/test' in path: 3 | return False 4 | return True 5 | 6 | def filter_webidl(path): 7 | if 'dom/bindings/mozwebidlcodegen/test' in path: 8 | return False 9 | if 'dom/bindings/test' in path: 10 | return False 11 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 12 | return False 13 | if 'tools/ts/' in path: 14 | return False 15 | return True 16 | 17 | def modify_file_list(lines, config): 18 | lines.append(b'__GENERATED__/dom/bindings/CSS2Properties.webidl') 19 | lines.append(b'__GENERATED__/dom/bindings/CSSStyleProperties.webidl') 20 | return lines 21 | 22 | def filter_js(path): 23 | if 'js/src/tests' in path or 'jit-test' in path: 24 | return False 25 | return True 26 | 27 | def filter_html(path): 28 | if 'testing/web-platform/' in path: 29 | return False 30 | return True 31 | 32 | def filter_css(path): 33 | if 'testing/web-platform/' in path: 34 | return False 35 | return True 36 | -------------------------------------------------------------------------------- /comm-esr60/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr60 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr60.tar 16 | tar xf comm-esr60.tar 17 | rm comm-esr60.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr60 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr60-blame.tar 30 | tar xf comm-esr60-blame.tar 31 | rm comm-esr60-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr68/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr68 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr68.tar 16 | tar xf comm-esr68.tar 17 | rm comm-esr68.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr68 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr68-blame.tar 30 | tar xf comm-esr68-blame.tar 31 | rm comm-esr68-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr78/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr78 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr78.tar 16 | tar xf comm-esr78.tar 17 | rm comm-esr78.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr78 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr78-blame.tar 30 | tar xf comm-esr78-blame.tar 31 | rm comm-esr78-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr91/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr91 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr91.tar 16 | tar xf comm-esr91.tar 17 | rm comm-esr91.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr91 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr91-blame.tar 30 | tar xf comm-esr91-blame.tar 31 | rm comm-esr91-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr102/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr102 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr102.tar 16 | tar xf comm-esr102.tar 17 | rm comm-esr102.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr102 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr102-blame.tar 30 | tar xf comm-esr102-blame.tar 31 | rm comm-esr102-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr115/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr115 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr115.tar 16 | tar xf comm-esr115.tar 17 | rm comm-esr115.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr115 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr115-blame.tar 30 | tar xf comm-esr115-blame.tar 31 | rm comm-esr115-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /whatwg-html/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading WHATWG HTML repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing git folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/whatwg-html.tar 16 | tar xf whatwg-html.tar 17 | rm whatwg-html.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading WHATWG HTML blame 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/whatwg-html-blame.tar 30 | tar xf whatwg-html-blame.tar 31 | rm whatwg-html-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /mozilla-build/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mozilla-build repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozilla-build.tar 16 | tar xf mozilla-build.tar 17 | rm mozilla-build.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mozilla-build blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozilla-build-blame.tar 30 | tar xf mozilla-build-blame.tar 31 | rm mozilla-build-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-esr128/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-esr128 repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr128.tar.lz4 16 | lz4 -dc comm-esr128.tar.lz4 | tar -x 17 | rm comm-esr128.tar.lz4 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-esr128 blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-esr128-blame.tar.lz4 30 | lz4 -dc comm-esr128-blame.tar.lz4 | tar -x 31 | rm comm-esr128-blame.tar.lz4 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /l10n/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading l10n repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing l10n git folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/firefox-l10n.tar.lz4 16 | lz4 -dc firefox-l10n.tar.lz4 | tar -x 17 | rm firefox-l10n.tar.lz4 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading l10n blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing l10n blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/firefox-l10n-blame.tar.lz4 30 | lz4 -dc firefox-l10n-blame.tar.lz4 | tar -x 31 | rm firefox-l10n-blame.tar.lz4 32 | fi 33 | popd 34 | 35 | echo Updating git 36 | pushd $GIT_ROOT 37 | git pull origin main 38 | 39 | popd 40 | 41 | echo Generating blame information 42 | build-blame $GIT_ROOT $BLAME_ROOT 43 | 44 | date 45 | -------------------------------------------------------------------------------- /mozilla-central/reblame: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | # Build blame for the other branches. For each branch, we create 10 | # a branch in the blame repo from the previous completed branch, so 11 | # as to minimize the amount of new work needed. The order of branches 12 | # in the loop is also selected to reduce unnecessary work; changing the 13 | # order should not affect correctness but may increase redundant work. 14 | 15 | # Note: the pine branch used to be in here, but was removed in bug 1841395. 16 | # If the pine branch gets reset for a new project and we want to 17 | # add it back in here, we might need to manually update the tarball to 18 | # strip the existing pine branch. I don't know if it will work otherwise. 19 | BRANCHES="elm cedar cypress beta release esr140 esr128 esr115 esr102 esr91 esr78 esr68 esr60 esr45 esr31 esr17" 20 | $CONFIG_REPO/shared/rebuild-blame.sh --tarball-base gecko --branches "${BRANCHES}" $* 21 | -------------------------------------------------------------------------------- /mozsearch/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | ## Index the clang-plugin 10 | 11 | # Add the special clang flags. 12 | $MOZSEARCH_PATH/scripts/indexer-setup.py > $INDEX_ROOT/config 13 | . $INDEX_ROOT/config 14 | 15 | date 16 | 17 | rm -rf $OBJDIR 18 | mkdir -p $OBJDIR 19 | 20 | date 21 | 22 | mkdir -p $OBJDIR/clang-plugin 23 | cd $OBJDIR/clang-plugin 24 | 25 | CLANG_PLUGIN_DIR=$FILES_ROOT/clang-plugin 26 | # Passing a VPATH will let make find the source files despite us using an objdir. 27 | # Explicitly passing CC and CXX propagates the special clang flags. 28 | make -f $CLANG_PLUGIN_DIR/Makefile VPATH=$CLANG_PLUGIN_DIR CC="$CC" CXX="$CXX" 29 | 30 | date 31 | 32 | ## Python Indexing 33 | 34 | cd $FILES_ROOT 35 | scip-python index --show-progress-rate-limit 0 --project-name python --output python.scip 36 | mv python.scip $INDEX_ROOT 37 | 38 | date 39 | 40 | ## Rust Indexing 41 | 42 | cd $FILES_ROOT 43 | rust-analyzer scip tools 44 | mv index.scip $INDEX_ROOT/rust.scip 45 | 46 | date 47 | -------------------------------------------------------------------------------- /just-mc.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "mozilla-central", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "mozilla-central": { 9 | "priority": 1000, 10 | "on_error": "halt", 11 | "cache": "everything", 12 | "index_path": "$WORKING/mozilla-central", 13 | "files_path": "$WORKING/mozilla-central/git", 14 | "git_path": "$WORKING/mozilla-central/git", 15 | "git_blame_path": "$WORKING/mozilla-central/blame", 16 | "github_repo": "https://github.com/mozilla/gecko-dev", 17 | "hg_root": "https://hg.mozilla.org/mozilla-central", 18 | "ccov_root": "https://coverage.moz.tools/", 19 | "wpt_root": "testing/web-platform", 20 | "objdir_path": "$WORKING/mozilla-central/objdir", 21 | "codesearch_path": "$WORKING/mozilla-central/livegrep.idx", 22 | "codesearch_port": 8082, 23 | "scip_subtrees": { 24 | "python": { 25 | "scip_index_path": "$WORKING/mozilla-central/python.scip", 26 | "subtree_root": "" 27 | } 28 | } 29 | } 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /version-control-tools/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading v-c-t repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/version-control-tools.tar 16 | tar xf version-control-tools.tar 17 | rm version-control-tools.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading v-c-t blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/version-control-tools-blame.tar 30 | tar xf version-control-tools-blame.tar 31 | rm version-control-tools-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull 40 | popd 41 | 42 | echo Generating blame information 43 | build-blame $GIT_ROOT $BLAME_ROOT 44 | 45 | date 46 | -------------------------------------------------------------------------------- /comm-central/repo_files.py: -------------------------------------------------------------------------------- 1 | import os 2 | from lib import run 3 | 4 | def modify_file_list(files, config=None, **kwargs): 5 | # Also grab the file list from the mozilla/ subrepo 6 | subrepo_path = os.path.join(config['files_path'], 'mozilla') 7 | sub_files = run(['git', 'ls-files'], cwd=subrepo_path).splitlines() 8 | sub_files = [ b'mozilla/' + f for f in sub_files if f ] 9 | return files + sub_files 10 | 11 | def filter_ipdl(path): 12 | return False 13 | 14 | def filter_webidl(path): 15 | if 'dom/bindings/mozwebidlcodegen/test' in path: 16 | return False 17 | if 'dom/bindings/test' in path: 18 | return False 19 | if 'dom/webidl/MozApplicationEvent.webidl' in path: 20 | return False 21 | if 'tools/ts/' in path: 22 | return False 23 | return True 24 | 25 | def filter_js(path): 26 | if 'js/src/tests' in path or 'jit-test' in path: 27 | return False 28 | return True 29 | 30 | def filter_html(path): 31 | if 'testing/web-platform/' in path: 32 | return False 33 | return True 34 | 35 | def filter_css(path): 36 | if 'testing/web-platform/' in path: 37 | return False 38 | return True 39 | -------------------------------------------------------------------------------- /mozsearch-mozilla/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mozsearch-mozilla repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozsearch-mozilla.tar.lz4 16 | lz4 -dc mozsearch-mozilla.tar.lz4 | tar -x 17 | rm mozsearch-mozilla.tar.lz4 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mozsearch-mozilla blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozsearch-mozilla-blame.tar.lz4 30 | lz4 -dc mozsearch-mozilla-blame.tar.lz4 | tar -x 31 | rm mozsearch-mozilla-blame.tar.lz4 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin master 40 | 41 | popd 42 | 43 | echo Generating blame information 44 | build-blame $GIT_ROOT $BLAME_ROOT 45 | 46 | date 47 | -------------------------------------------------------------------------------- /mozilla-vpn-client/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mozilla-vpn-client repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozilla-vpn-client.tar 16 | tar xf mozilla-vpn-client.tar 17 | rm mozilla-vpn-client.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mozilla-vpn-client blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozilla-vpn-client-blame.tar 30 | tar xf mozilla-vpn-client-blame.tar 31 | rm mozilla-vpn-client-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | # Only pull the branch we're going to use 40 | git pull origin main 41 | popd 42 | 43 | echo Generating blame information 44 | build-blame $GIT_ROOT $BLAME_ROOT 45 | 46 | date 47 | -------------------------------------------------------------------------------- /mozsearch/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mozsearch repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozsearch.tar.lz4 16 | lz4 -dc mozsearch.tar.lz4 | tar -x 17 | rm mozsearch.tar.lz4 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mozsearch blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozsearch-blame.tar.lz4 30 | lz4 -dc mozsearch-blame.tar.lz4 | tar -x 31 | rm mozsearch-blame.tar.lz4 32 | fi 33 | popd 34 | 35 | date 36 | 37 | echo Updating git 38 | pushd $GIT_ROOT 39 | git pull origin master 40 | # mozsearch uses submodules 41 | git submodule update --init --recursive 42 | 43 | popd 44 | 45 | echo Generating blame information 46 | build-blame $GIT_ROOT $BLAME_ROOT 47 | 48 | date 49 | -------------------------------------------------------------------------------- /firefox-shared/resolve-gecko-revs.sh: -------------------------------------------------------------------------------- 1 | # This file is intentionally not executable, because it should always be sourced 2 | # into a pre-existing shell. MOZSEARCH_PATH and INDEX_ROOT should be defined prior 3 | # to sourcing. 4 | 5 | if [ -z $MOZSEARCH_PATH ] 6 | then 7 | echo "Error: resolve-gecko-revs.sh is being sourced without MOZSEARCH_PATH defined" 8 | return # leave without aborting the calling script 9 | elif [ -z $INDEX_ROOT ] 10 | then 11 | echo "Error: resolve-gecko-revs.sh is being sourced without INDEX_ROOT defined" 12 | return # leave without aborting the calling script 13 | fi 14 | 15 | REVISION_TREE=$1 16 | REVISION_ID=$2 17 | 18 | date 19 | 20 | REVISION="${REVISION_TREE}.${REVISION_ID}" 21 | CURL="curl -SsfL --compressed" 22 | 23 | pushd $INDEX_ROOT 24 | 25 | PREEXISTING_HG_REV= 26 | if [ -f "target.json" ]; then 27 | PREEXISTING_HG_REV=$(jq -r ".moz_source_stamp" target.json) 28 | echo "Found pre-existing target.json with hg revision ${PREEXISTING_HG_REV}." 29 | fi 30 | 31 | ${CURL} https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.$REVISION.firefox.linux64-searchfox-debug/artifacts/public/build/target.json > target.json 32 | INDEXED_HG_REV=$(jq -r ".moz_source_stamp" target.json) 33 | 34 | popd 35 | -------------------------------------------------------------------------------- /shared/resolve-gecko-revs.sh: -------------------------------------------------------------------------------- 1 | # This file is intentionally not executable, because it should always be sourced 2 | # into a pre-existing shell. MOZSEARCH_PATH and INDEX_ROOT should be defined prior 3 | # to sourcing. 4 | 5 | # This is the pre-https://github.com/mozilla-firefox/firefox support 6 | 7 | if [ -z $MOZSEARCH_PATH ] 8 | then 9 | echo "Error: resolve-gecko-revs.sh is being sourced without MOZSEARCH_PATH defined" 10 | return # leave without aborting the calling script 11 | elif [ -z $INDEX_ROOT ] 12 | then 13 | echo "Error: resolve-gecko-revs.sh is being sourced without INDEX_ROOT defined" 14 | return # leave without aborting the calling script 15 | fi 16 | 17 | REVISION_TREE=$1 18 | REVISION_ID=$2 19 | 20 | date 21 | 22 | REVISION="${REVISION_TREE}.${REVISION_ID}" 23 | CURL="curl -SsfL --compressed" 24 | 25 | pushd $INDEX_ROOT 26 | 27 | PREEXISTING_HG_REV= 28 | if [ -f "target.json" ]; then 29 | PREEXISTING_HG_REV=$(jq -r ".moz_source_stamp" target.json) 30 | echo "Found pre-existing target.json with hg revision ${PREEXISTING_HG_REV}." 31 | fi 32 | 33 | ${CURL} https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.$REVISION.firefox.linux64-searchfox-debug/artifacts/public/build/target.json > target.json 34 | INDEXED_HG_REV=$(jq -r ".moz_source_stamp" target.json) 35 | 36 | popd 37 | -------------------------------------------------------------------------------- /just-fm.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "firefox-main", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "firefox-main": { 9 | "priority": 1000, 10 | "on_error": "halt", 11 | "cache": "everything", 12 | "shared_path": "$WORKING/firefox-shared", 13 | "index_path": "$WORKING/firefox-main", 14 | "files_path": "$WORKING/firefox-main/git", 15 | "git_path": "$WORKING/firefox-main/git", 16 | "git_branch": "main", 17 | "oldtree_name": "mozilla-central", 18 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 19 | "git_blame_path": "$WORKING/firefox-shared/blame", 20 | "github_repo": "https://github.com/mozilla-firefox/firefox", 21 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 22 | "hg_root": "https://hg.mozilla.org/mozilla-central", 23 | "ccov_root": "https://coverage.moz.tools/", 24 | "wpt_root": "testing/web-platform", 25 | "objdir_path": "$WORKING/firefox-main/objdir", 26 | "codesearch_path": "$WORKING/firefox-main/livegrep.idx", 27 | "codesearch_port": 8081, 28 | "scip_subtrees": { 29 | "python": { 30 | "scip_index_path": "$WORKING/firefox-main/python.scip", 31 | "scip_index_path2": "$WORKING/firefox-main/python2.scip", 32 | "subtree_root": "" 33 | } 34 | } 35 | } 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /mingw-moz/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mingw repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mingw.tar 16 | tar xf mingw.tar 17 | rm mingw.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading mingw blame. 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mingw-blame.tar 30 | tar xf mingw-blame.tar 31 | rm mingw-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | MOZ_REV=$(curl -SsfL "https://raw.githubusercontent.com/mozilla-firefox/firefox/refs/heads/main/taskcluster/kinds/fetch/toolchains.yml" | yq -r '.["mingw-w64"].fetch.revision') 38 | 39 | echo Updating git 40 | pushd $GIT_ROOT 41 | git pull origin master 42 | git reset --hard ${MOZ_REV} 43 | popd 44 | 45 | echo Generating blame information 46 | build-blame $GIT_ROOT $BLAME_ROOT 47 | 48 | # Check out the blame rev corresponding to MOZ_REV, since the blame 49 | # repo may be ahead of it. 50 | 51 | pushd $BLAME_ROOT 52 | BLAME_REV=$(git log -1 --grep=$MOZ_REV --pretty=format:%H) 53 | if [ -z "$BLAME_REV" ]; then 54 | echo "mingw-moz: Unable to find blame rev for $MOZ_REV" 55 | exit 1; 56 | fi 57 | git reset --soft $BLAME_REV 58 | popd 59 | 60 | date 61 | -------------------------------------------------------------------------------- /comm-central/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading comm-central... 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing comm-central git folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-central.tar 16 | tar xf comm-central.tar 17 | rm comm-central.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Downloading comm-central blame... 24 | pushd $INDEX_ROOT 25 | if [ -d "blame" ] 26 | then 27 | echo "Found pre-existing blame folder; skipping re-download." 28 | else 29 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/comm-central-blame.tar 30 | tar xf comm-central-blame.tar 31 | rm comm-central-blame.tar 32 | fi 33 | popd 34 | 35 | date 36 | 37 | # We also need to put a copy of mozilla-central into the mozilla/ folder 38 | 39 | echo "Downloading mozilla..." 40 | pushd $INDEX_ROOT/git 41 | if [ -d "mozilla" ] 42 | then 43 | echo "Found pre-existing mozilla subfolder; skipping re-download." 44 | else 45 | $CONFIG_REPO/shared/fetch-gecko-tarball.sh gecko $PWD 46 | mv git mozilla 47 | fi 48 | 49 | date 50 | 51 | echo Updating git 52 | pushd $GIT_ROOT 53 | git pull 54 | pushd mozilla 55 | # The repo may have been uploaded while checked out to some non-master 56 | # revision (to match the hg revision used in taskcluster) 57 | git checkout master 58 | git pull 59 | popd 60 | popd 61 | 62 | echo Generating blame information 63 | build-blame $GIT_ROOT $BLAME_ROOT 64 | 65 | date 66 | -------------------------------------------------------------------------------- /mozsearch-tests/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # We reuse the mozsearch tarball; we don't reuse its blame or do anything with 10 | # blame. We could filter the mozsearch repo down for that, but we want to 11 | # imitate what happens in `make build-test-repo`. 12 | 13 | echo Downloading mozsearch repo 14 | pushd $INDEX_ROOT 15 | if [ -d "git" ] 16 | then 17 | echo "Found pre-existing folder; skipping re-download." 18 | else 19 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozsearch.tar.lz4 20 | lz4 -dc mozsearch.tar.lz4 | tar -x 21 | rm mozsearch.tar.lz4 22 | fi 23 | popd 24 | 25 | date 26 | 27 | # Note that GIT_ROOT is not defined for this tree, but we do need to do an 28 | # update unless we steal the update done by our sibling "mozsearch-mozilla" 29 | # tree. We only expect to be a few commits behind, so we just update manually. 30 | echo Updating git 31 | pushd $INDEX_ROOT/git 32 | git pull origin master 33 | # mozsearch uses submodules 34 | git submodule update --init --recursive 35 | 36 | popd 37 | 38 | date 39 | 40 | ## Invoke the mozsearch tests repo "setup" script for its symlink side-effects 41 | 42 | # The tests config.json file is here: 43 | TESTS_ROOT=$INDEX_ROOT/git/tests 44 | # The "setup" script is in the "tests" dir under that: 45 | TESTS_CONFIG_DIR=$TESTS_ROOT/tests 46 | 47 | # Tell it the config repo is in its tree rather than the real CONFIG_REPO 48 | # because it uses that as the base dir for symlinking. 49 | CONFIG_REPO=$TESTS_ROOT 50 | 51 | # now transfer control into the setup 52 | exec $TESTS_CONFIG_DIR/setup 53 | -------------------------------------------------------------------------------- /graphviz/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading graphviz repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/graphviz.tar.lz4 16 | lz4 -dc graphviz.tar.lz4 | tar -x 17 | rm graphviz.tar.lz4 18 | fi 19 | popd 20 | 21 | # Temporary fix for the git url. 22 | # This can be deleted once this fix is reflected to the uploaded tarball. 23 | sed -i -e 's|https://gitlab.com/graphviz/graphviz$|https://gitlab.com/graphviz/graphviz.git|' $GIT_ROOT/.git/config 24 | 25 | date 26 | 27 | echo Downloading graphviz blame. 28 | pushd $INDEX_ROOT 29 | if [ -d "blame" ] 30 | then 31 | echo "Found pre-existing blame folder; skipping re-download." 32 | else 33 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/graphviz-blame.tar.lz4 34 | lz4 -dc graphviz-blame.tar.lz4 | tar -x 35 | rm graphviz-blame.tar.lz4 36 | fi 37 | popd 38 | 39 | date 40 | 41 | echo Updating git 42 | pushd $GIT_ROOT 43 | git pull origin main 44 | 45 | popd 46 | 47 | echo Generating blame information 48 | build-blame $GIT_ROOT $BLAME_ROOT 49 | 50 | date 51 | 52 | ## Since graphviz is going in config5 and we already do wacky package 53 | ## installation at runtime, there's no real harm in doing more of it, and this 54 | ## avoids us needing to provision autoconf and automake normally. 55 | 56 | # Update the apt index 57 | sudo apt-get update 58 | 59 | sudo apt-get install -y libtool libltdl-dev automake autoconf bison flex 60 | -------------------------------------------------------------------------------- /config5.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "wubkat", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "wubkat": { 9 | "priority": 100, 10 | "on_error": "halt", 11 | "cache": "codesearch", 12 | "index_path": "$WORKING/wubkat", 13 | "files_path": "$WORKING/wubkat/git", 14 | "git_path": "$WORKING/wubkat/git", 15 | "git_blame_path": "$WORKING/wubkat/blame", 16 | "github_repo": "https://github.com/WebKit/Webkit", 17 | "objdir_path": "$WORKING/wubkat/objdir", 18 | "ignore_missing_path_prefixes": [ 19 | "$WORKING/wubkat/objdir/CMakeFiles/CMakeScratch/" 20 | ], 21 | "codesearch_path": "$WORKING/wubkat/livegrep.idx", 22 | "codesearch_port": 8081, 23 | "scip_subtrees": {} 24 | }, 25 | "graphviz": { 26 | "priority": 3, 27 | "on_error": "continue", 28 | "cache": "codesearch", 29 | "index_path": "$WORKING/graphviz", 30 | "files_path": "$WORKING/graphviz/git", 31 | "git_path": "$WORKING/graphviz/git", 32 | "git_blame_path": "$WORKING/graphviz/blame", 33 | "github_repo": "https://gitlab.com/graphviz/graphviz.git", 34 | "walk_submodules": false, 35 | "objdir_path": "$WORKING/graphviz/objdir", 36 | "ignore_missing_path_prefixes": [ 37 | "$WORKING/graphviz/objdir/conftest", 38 | "$WORKING/graphviz/objdir/libltdl/.libs/libltdlcS.c", 39 | "$WORKING/graphviz/objdir/libltdl/confdefs.h", 40 | "$WORKING/graphviz/objdir/libltdl/conftest" 41 | ], 42 | "codesearch_path": "$WORKING/graphviz/livegrep.idx", 43 | "codesearch_port": 8082, 44 | "scip_subtrees": {} 45 | } 46 | } 47 | } 48 | -------------------------------------------------------------------------------- /firefox-main/upload: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # NB: To further increase parallelism we could run all of the aws s3 commands in 10 | # parallel, but these files are generally large enough on their own that they 11 | # will take a while and it probably makes sense to mess with the aws command 12 | # parallelism settings first. 13 | 14 | 15 | echo 'Uploading "git" archive: firefox-shared-git.tar.lz4' 16 | pushd $SHARED_ROOT 17 | $CONFIG_REPO/firefox-shared/git-bare-maintenance.sh ./git 18 | tar cf - git | lz4 -f - firefox-shared-git.tar.lz4 19 | # Note that we're using the "aws" command instead of $AWS_ROOT/upload.py for 20 | # parallelism reasons. 21 | aws s3 cp firefox-shared-git.tar.lz4 s3://searchfox.repositories/firefox-shared-git.tar.lz4 --acl public-read 22 | rm firefox-shared-git.tar.lz4 23 | popd 24 | 25 | date 26 | 27 | echo 'Uploading "oldgit" archive: firefox-shared-oldgit.tar.lz4' 28 | pushd $SHARED_ROOT 29 | $CONFIG_REPO/firefox-shared/git-bare-maintenance.sh ./oldgit 30 | tar cf - oldgit | lz4 -f - firefox-shared-oldgit.tar.lz4 31 | # Note that we're using the "aws" command instead of $AWS_ROOT/upload.py for 32 | # parallelism reasons. 33 | aws s3 cp firefox-shared-oldgit.tar.lz4 s3://searchfox.repositories/firefox-shared-oldgit.tar.lz4 --acl public-read 34 | rm firefox-shared-oldgit.tar.lz4 35 | popd 36 | 37 | date 38 | 39 | echo 'Uploading "blame" archive: firefox-shared-blame.tar.lz4' 40 | pushd $SHARED_ROOT 41 | $CONFIG_REPO/firefox-shared/git-bare-maintenance.sh ./blame 42 | tar cf - blame | lz4 -f - firefox-shared-blame.tar.lz4 43 | # Note that we're using the "aws" command instead of $AWS_ROOT/upload.py for 44 | # parallelism reasons. 45 | aws s3 cp firefox-shared-blame.tar.lz4 s3://searchfox.repositories/firefox-shared-blame.tar.lz4 --acl public-read 46 | rm firefox-shared-blame.tar.lz4 47 | popd 48 | 49 | date 50 | -------------------------------------------------------------------------------- /shared/fetch-gecko-tarball.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | # Helper script to download the gecko or gecko-blame tarball from S3 10 | # into the working dir if it hasn't already been downloaded, 11 | # and then unpack it into the specified destination folder. 12 | # This is a shared helper because we have multiple repos that 13 | # download the same tarball which takes time/bandwidth; doing 14 | # it once and reusing it saves about 3 minutes/7.5 Gb per 15 | # additional repo that uses it. 16 | 17 | if [ $# -ne 2 ]; then 18 | echo "Usage: $0 " 19 | echo " e.g.: $0 gecko-blame \$PWD" 20 | exit 1 21 | fi 22 | 23 | TARBALL="$1" 24 | DESTDIR="$2" 25 | 26 | # This is a bit of a hack, but there are only two allowed values for 27 | # TARBALL so it's not terrible. And anyway no other repo 28 | # respects this convention of having the top-level folder inside the 29 | # tarball match the tarball name, since all repos use "git" and "blame" 30 | # folders inside their tarballs. 31 | if [[ "$TARBALL" == "gecko" ]]; then 32 | TARBALL_FOLDER="git" 33 | elif [[ "$TARBALL" == "gecko-blame" ]]; then 34 | TARBALL_FOLDER="blame" 35 | fi 36 | 37 | if [ -d "${DESTDIR}/${TARBALL_FOLDER}" ]; then 38 | echo "Found pre-existing folder at ${DESTDIR}/${TARBALL_FOLDER}, skipping re-download..." 39 | exit 0 40 | fi 41 | 42 | if [ ! -f "$WORKING/${TARBALL}.tar" ]; then 43 | delete_partial_download() { 44 | rm -f "$WORKING/${TARBALL}.tar" 45 | exit 1 46 | } 47 | 48 | # This download can take a long time. If the user interrupts with ctrl-c, then 49 | # clean up the partial download or it can cause trouble the next time the script 50 | # is run. 51 | trap "delete_partial_download" SIGINT 52 | pushd "$WORKING" 53 | wget -nv "https://s3-us-west-2.amazonaws.com/searchfox.repositories/${TARBALL}.tar" 54 | popd 55 | trap - SIGINT # clear trap now that download is done 56 | fi 57 | 58 | tar xf "$WORKING/${TARBALL}.tar" -C "$DESTDIR" 59 | -------------------------------------------------------------------------------- /tree-list.js: -------------------------------------------------------------------------------- 1 | var TREE_LIST = [ 2 | [ 3 | { 4 | name: "Firefox", 5 | items: [ 6 | { value: "firefox-main" }, 7 | { value: "firefox-beta" }, 8 | { value: "firefox-release" }, 9 | { value: "firefox-esr140" }, 10 | { value: "firefox-esr128" }, 11 | { value: "firefox-esr115" }, 12 | { value: "mozilla-esr102" }, 13 | { value: "mozilla-esr91" }, 14 | { value: "mozilla-esr78" }, 15 | { value: "mozilla-esr68" }, 16 | { value: "mozilla-esr60" }, 17 | { value: "mozilla-esr45" }, 18 | { value: "mozilla-esr31" }, 19 | { value: "mozilla-esr17" }, 20 | ], 21 | }, 22 | ], 23 | [ 24 | { 25 | name: "Firefox other", 26 | items: [ 27 | { value: "mozilla-mobile" }, 28 | { value: "mozilla-elm" }, 29 | { value: "mozilla-cedar" }, 30 | { value: "mozilla-cypress" }, 31 | ], 32 | }, 33 | { 34 | name: "Thunderbird", 35 | items: [ 36 | { value: "comm-central" }, 37 | { value: "comm-esr128" }, 38 | { value: "comm-esr115" }, 39 | { value: "comm-esr102" }, 40 | { value: "comm-esr91" }, 41 | { value: "comm-esr78" }, 42 | { value: "comm-esr68" }, 43 | { value: "comm-esr60" }, 44 | ], 45 | }, 46 | ], 47 | [ 48 | { 49 | name: "Searchfox", 50 | items: [ 51 | { value: "mozsearch" }, 52 | { value: "mozsearch-mozilla" }, 53 | { value: "mozsearch-tests" }, 54 | ], 55 | }, 56 | { 57 | name: "MinGW", 58 | items: [ 59 | { value: "mingw" }, 60 | { value: "mingw-moz" }, 61 | ], 62 | }, 63 | { 64 | name: "Other", 65 | items: [ 66 | { label: "KaiOS", value: "kaios" }, 67 | { value: "glean" }, 68 | { value: "l10n" }, 69 | { label: "LLVM", value: "llvm" }, 70 | { value: "nss" }, 71 | { label: "Rust", value: "rust" }, 72 | { label: "TC39 ECMA262 spec", value: "ecma262" }, 73 | { value: "version-control-tools" }, 74 | { label: "WHATWG HTML spec", value: "whatwg-html" }, 75 | { value: "wubkat" }, 76 | ] 77 | } 78 | ] 79 | ]; 80 | -------------------------------------------------------------------------------- /just-mozsearch.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "mozsearch", 5 | "instance_type": "t3.xlarge", 6 | 7 | "trees": { 8 | "mozsearch": { 9 | "priority": 600, 10 | "on_error": "continue", 11 | "cache": "codesearch", 12 | "index_path": "$WORKING/mozsearch", 13 | "files_path": "$WORKING/mozsearch/git", 14 | "git_path": "$WORKING/mozsearch/git", 15 | "git_blame_path": "$WORKING/mozsearch/blame", 16 | "github_repo": "https://github.com/mozsearch/mozsearch", 17 | "objdir_path": "$WORKING/mozsearch/objdir", 18 | "codesearch_path": "$WORKING/mozsearch/livegrep.idx", 19 | "codesearch_port": 8085, 20 | "scip_subtrees": { 21 | "python": { 22 | "scip_index_path": "$WORKING/mozsearch/python.scip", 23 | "subtree_root": "" 24 | }, 25 | "rust": { 26 | "scip_index_path": "$WORKING/mozsearch/rust.scip", 27 | "subtree_root": "tools" 28 | } 29 | } 30 | }, 31 | 32 | "mozsearch-mozilla": { 33 | "priority": 500, 34 | "on_error": "continue", 35 | "cache": "codesearch", 36 | "index_path": "$WORKING/mozsearch-mozilla", 37 | "files_path": "$WORKING/mozsearch-mozilla/git", 38 | "git_path": "$WORKING/mozsearch-mozilla/git", 39 | "git_blame_path": "$WORKING/mozsearch-mozilla/blame", 40 | "github_repo": "https://github.com/mozsearch/mozsearch-mozilla", 41 | "objdir_path": "$WORKING/mozsearch-mozilla/objdir", 42 | "codesearch_path": "$WORKING/mozsearch-mozilla/livegrep.idx", 43 | "codesearch_port": 8086, 44 | "scip_subtrees": {} 45 | }, 46 | 47 | "mozsearch-tests": { 48 | "priority": 550, 49 | "on_error": "continue", 50 | "cache": "everything", 51 | "index_path": "$WORKING/mozsearch-tests", 52 | "files_path": "$WORKING/mozsearch-tests/files", 53 | "objdir_path": "$WORKING/mozsearch-tests/objdir", 54 | "wpt_root": "testing/web-platform", 55 | "codesearch_path": "$WORKING/mozsearch-tests/livegrep.idx", 56 | "codesearch_port": 8087, 57 | "scip_subtrees": {} 58 | } 59 | } 60 | } 61 | -------------------------------------------------------------------------------- /wubkat/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # Note that our tarball here also includes the flatpak SDK downloads that live 10 | # under git/WebKitBuild/UserFlatpak. 11 | echo Downloading webkit 12 | pushd $INDEX_ROOT 13 | if [ -d "git" ] 14 | then 15 | echo "Found pre-existing git folder; skipping re-download." 16 | else 17 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/wubkat.tar.lz4 18 | lz4 -dc wubkat.tar.lz4 | tar -x 19 | rm wubkat.tar.lz4 20 | fi 21 | popd 22 | 23 | date 24 | 25 | echo Downloading webkit blame 26 | pushd $INDEX_ROOT 27 | if [ -d "blame" ] 28 | then 29 | echo "Found pre-existing blame folder; skipping re-download." 30 | else 31 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/wubkat-blame.tar.lz4 32 | lz4 -dc wubkat-blame.tar.lz4 | tar -x 33 | rm wubkat-blame.tar.lz4 34 | fi 35 | popd 36 | 37 | date 38 | 39 | echo Updating git 40 | pushd $GIT_ROOT 41 | git remote prune origin || true 42 | git pull origin main || handle_tree_error "wubkat git pull" 43 | popd 44 | 45 | date 46 | 47 | echo Figuring out what revision to use. 48 | USE_REV=$($CONFIG_REPO/wubkat/last-successful-build-to-git-commit.sh) 49 | echo Checking out last successfully built revision "${USE_REV}" 50 | 51 | date 52 | 53 | pushd $GIT_ROOT 54 | git checkout "${USE_REV}" 55 | popd 56 | echo Revision should now be checked out. 57 | 58 | date 59 | 60 | echo Generating blame information 61 | build-blame $GIT_ROOT $BLAME_ROOT 62 | 63 | date 64 | 65 | echo Installing/updating system build dependencies 66 | 67 | # Update the apt index 68 | sudo apt-get update 69 | 70 | # gtk/install-dependencies below will install a bunch of stuff, but it seems like 71 | # it maybe missed these things I've added after flatpak, but they may also be 72 | # transitive deps of some of the below. (And now with the return to 73 | # install-dependencies with explicit 22.04 these may not be needed at all, but 74 | # we'll see.) 75 | sudo apt-get install -y libcairo-dev libharfbuzz-dev gi-docgen 76 | 77 | date 78 | 79 | # Support for 22.04 seems to have been officially added in 80 | # https://searchfox.org/wubkat/commit/aa960b1a0c7b980efdbc66718239a449b5966a0c 81 | # so we're now using it directly again. 82 | sudo $FILES_ROOT/Tools/gtk/install-dependencies 83 | 84 | date 85 | -------------------------------------------------------------------------------- /mozilla-mobile/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading mozilla-mobile repo 10 | pushd $INDEX_ROOT 11 | if [ -d "git" ] 12 | then 13 | echo "Found pre-existing mozilla-mobile folder; skipping re-download." 14 | else 15 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/mozilla-mobile.tar 16 | tar xf mozilla-mobile.tar 17 | rm mozilla-mobile.tar 18 | fi 19 | popd 20 | 21 | date 22 | 23 | echo Updating git 24 | pushd $GIT_ROOT 25 | # pull latest moz-mob code. We run this twice because of problems experienced 26 | # at https://bugzilla.mozilla.org/show_bug.cgi?id=1596846#c7. This may be a 27 | # workaround for a git regression, as this didn't happen to 2.20.1 but did 28 | # happen on 2.24.0 and there were various changelog comments about changing when 29 | # various things were resolved. For example, 30 | # https://github.com/git/git/blob/master/Documentation/RelNotes/2.23.0.txt#L36 31 | # 32 | # the prune stuff is to deal with encountering this error: 33 | # error: cannot lock ref 'refs/remotes/origin/mergify': 'refs/remotes/origin/mergify/bp/main/pr-5560' exists; cannot create 'refs/remotes/origin/mergify' 34 | # I'm wrapping it in the and because we still want to do the update. 35 | # 36 | # And now we're also: 37 | # - forcibly cleaning up gc logs before doing stuff 38 | # - run git gc before doing stuff. 39 | # - this is because of: 40 | # `warning: There are too many unreachable loose objects; run 'git prune' to remove them.` 41 | # - despite us running it on saturdays in all repositories via the git-maintenance.sh helper 42 | # - and this may not be enough! 43 | rm -f .git/modules/*/gc.log 44 | git gc 45 | git submodule update --remote || git submodule update --remote || (git submodule foreach --recursive git remote prune origin && git submodule update --remote) 46 | # The previous step will update the submodules of this top-level synthetic superproject, 47 | # and now we want to commit those changes, so that the top-level superproject points 48 | # to the latest version of each submodule. 49 | git add * 50 | git commit --allow-empty -am "Submodule update at $(date)" --author "Searchfox Indexer " 51 | # And then we update the nested submodules to the revision desired by the top-level submodules. 52 | git submodule update --init --recursive 53 | popd 54 | 55 | date 56 | -------------------------------------------------------------------------------- /firefox-shared/process-gecko-analysis.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | # No arguments needed to this script. It just needs $CONFIG_REPO 8 | # and $MOZSEARCH_PATH defined in the environment. 9 | 10 | # Process the downloads for each platform in parallel 11 | echo "linux64 linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64 ios" | tr " " "\n" | 12 | parallel --halt now,fail=1 "$CONFIG_REPO/shared/process-tc-artifacts.sh {}" 13 | 14 | # Combine the per-platform list files 15 | cat generated-files-*.list > generated-files.list 16 | cat analysis-files-*.list > analysis-files.list 17 | cat analysis-dirs-*.list > analysis-dirs.list 18 | 19 | date 20 | 21 | # Special case: xptdata.cpp is a giant file and is different for each platform, but 22 | # the differences are not particularly relevant so let's just keep the Linux one. 23 | for PLATFORM in linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64; do 24 | rm -f generated-${PLATFORM}/xpcom/reflect/xptinfo/xptdata.cpp 25 | rm -f analysis-${PLATFORM}/__GENERATED__/xpcom/reflect/xptinfo/xptdata.cpp 26 | done 27 | 28 | # For each generated file, if all platforms generated the same thing (or didn't 29 | # generate the file at all due to being a platform-specific feature), copy it to 30 | # the merged objdir. 31 | sort generated-files.list | uniq | parallel --halt now,fail=1 "$CONFIG_REPO/shared/collapse-generated-files.sh {}" 32 | 33 | date 34 | 35 | # Finally, merge the analysis files for the non-generated source files. All the files 36 | # are going to be listed in the analysis-files.list, possibly duplicated across 37 | # platforms, so we deduplicate the filenames and merge each filename across platforms. 38 | sort analysis-dirs.list | uniq | parallel --halt now,fail=1 'mkdir -p analysis/{}' 39 | sort analysis-files.list | uniq | parallel --halt now,fail=1 "RUST_LOG=info merge-analyses analysis-*/{} > analysis/{}" 40 | 41 | date 42 | 43 | # Delete the generated-* and analysis-* directories, but retain the tarballs 44 | # for ease of investigation. (The tarballs are still available from taskcluster 45 | # though, and fetch-tc-artifcats.sh knows how to re-fetch them, so if we don't 46 | # want to waste the space, we could just make it easier to re-fetch them so 47 | # there's a one-liner in mozsearch's `docs/aws.md`.) 48 | rm -rf generated-*/ 49 | rm -rf analysis-*/ 50 | -------------------------------------------------------------------------------- /nss/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | echo Downloading NSPR 10 | # We used to have a hand-built tar file at nss-nspr.tar. This included 11 | # pre-configured byproducts and pre-built byproducts that made building NSS 12 | # brittle. The tarball never gets automatically updated, so there's also 13 | # potential for desynchronization versus NSS> 14 | # 15 | # On asuth's machine which is not in an AWS data-center, checking NSPR out from 16 | # hg only takes 4 seconds wall-time, So now we just check the thing out from hg. 17 | # So, if you're seeing this comment, we abandoned the tarball for now. 18 | pushd $INDEX_ROOT 19 | if [ -d "nspr" ] 20 | then 21 | echo "Found pre-existing nspr folder; skipping re-download." 22 | else 23 | hg clone https://hg.mozilla.org/projects/nspr 24 | fi 25 | popd 26 | 27 | date 28 | 29 | echo Downloading NSS 30 | pushd $INDEX_ROOT 31 | if [ -d "git" ] 32 | then 33 | echo "Found pre-existing git folder; skipping re-download." 34 | else 35 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/nss.tar 36 | tar xf nss.tar 37 | rm nss.tar 38 | fi 39 | popd 40 | 41 | date 42 | 43 | echo Downloading NSS blame 44 | pushd $INDEX_ROOT 45 | if [ -d "blame" ] 46 | then 47 | echo "Found pre-existing blame folder; skipping re-download." 48 | else 49 | wget -nv https://s3-us-west-2.amazonaws.com/searchfox.repositories/nss-blame.tar 50 | tar xf nss-blame.tar 51 | rm nss-blame.tar 52 | fi 53 | popd 54 | 55 | date 56 | 57 | echo Updating NSS git 58 | pushd $GIT_ROOT 59 | # The nss repo occasionally experiences situations where someone pushes something 60 | # to github thinking it is canonical, but that push then gets clobbered out of 61 | # existence by the repository syncing mechanism force-pushing the current state 62 | # of hg to the github repo. So just fetch and then reset to the state of the 63 | # origin. 64 | git fetch origin 65 | git reset --hard origin/master 66 | # Since updating to NSS 0.5.9, cinnabar seems upset about the state of the NSS 67 | # repo. Since running these next steps is not necessary for searchfox to update 68 | # (we pulled from github above), this is being disabled. 69 | #git config remote.cinnabar.url || git remote add cinnabar hg::https://hg.mozilla.org/projects/nss 70 | #git config cinnabar.graft true 71 | #git remote update cinnabar 72 | popd 73 | 74 | date 75 | 76 | echo Generating blame information 77 | build-blame $GIT_ROOT $BLAME_ROOT 78 | 79 | date 80 | -------------------------------------------------------------------------------- /shared/process-gecko-analysis.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | # No arguments needed to this script. It just needs $CONFIG_REPO 10 | # and $MOZSEARCH_PATH defined in the environment. 11 | 12 | # Process the downloads for each platform in parallel 13 | echo "linux64 linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64 ios" | tr " " "\n" | 14 | parallel --halt now,fail=1 "$CONFIG_REPO/shared/process-tc-artifacts.sh {}" 15 | 16 | # Combine the per-platform list files 17 | cat generated-files-*.list > generated-files.list 18 | cat analysis-files-*.list > analysis-files.list 19 | cat analysis-dirs-*.list > analysis-dirs.list 20 | 21 | date 22 | 23 | # Special case: xptdata.cpp is a giant file and is different for each platform, but 24 | # the differences are not particularly relevant so let's just keep the Linux one. 25 | for PLATFORM in linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64; do 26 | rm -f generated-${PLATFORM}/xpcom/reflect/xptinfo/xptdata.cpp 27 | rm -f analysis-${PLATFORM}/__GENERATED__/xpcom/reflect/xptinfo/xptdata.cpp 28 | done 29 | 30 | # For each generated file, if all platforms generated the same thing (or didn't 31 | # generate the file at all due to being a platform-specific feature), copy it to 32 | # the merged objdir. 33 | sort generated-files.list | uniq | parallel --halt now,fail=1 "$CONFIG_REPO/shared/collapse-generated-files.sh {}" 34 | 35 | date 36 | 37 | # Finally, merge the analysis files for the non-generated source files. All the files 38 | # are going to be listed in the analysis-files.list, possibly duplicated across 39 | # platforms, so we deduplicate the filenames and merge each filename across platforms. 40 | sort analysis-dirs.list | uniq | parallel --halt now,fail=1 'mkdir -p analysis/{}' 41 | sort analysis-files.list | uniq | parallel --halt now,fail=1 "RUST_LOG=info merge-analyses analysis-*/{} > analysis/{}" 42 | 43 | date 44 | 45 | # Delete the generated-* and analysis-* directories, but retain the tarballs 46 | # for ease of investigation. (The tarballs are still available from taskcluster 47 | # though, and fetch-tc-artifcats.sh knows how to re-fetch them, so if we don't 48 | # want to waste the space, we could just make it easier to re-fetch them so 49 | # there's a one-liner in mozsearch's `docs/aws.md`.) 50 | rm -rf generated-*/ 51 | rm -rf analysis-*/ 52 | -------------------------------------------------------------------------------- /wubkat/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | # the starting point for the build logic here is from: 10 | # https://github.com/dpino/webkit-index-config/blob/master/webkit/build 11 | # 12 | # our primary changes here are: 13 | # - we handle the dependencies and flatpak updating in `setup` so that we can 14 | # persist the downloaded flatpak state in `upload` because the flatpak server 15 | # seems to only be able to talk to our AWS instances at very slow speeds like 16 | # 600kB/s. (It started faster and goes downhill from there. We're talking 17 | # hours for a full download, though.) 18 | # - we dump the indexer setup to "config" like other build scripts so we can 19 | # look at it afterwards rather than sourcing it directly. 20 | 21 | date 22 | 23 | # Add the special clang flags. 24 | $MOZSEARCH_PATH/scripts/indexer-setup.py > $INDEX_ROOT/config 25 | . $INDEX_ROOT/config 26 | 27 | mkdir -p $OBJDIR 28 | cd $OBJDIR 29 | 30 | # Default WebKit port. 31 | PORT=${PORT:-GTK} 32 | 33 | # C Flags are encoded into CC var. 34 | # Remove prefix (clang) and store flags into C_FLAGS. 35 | C_FLAGS=${CC:6} 36 | CC=${CC:0:5} 37 | 38 | # CXX Flags are encoded into CXX var. 39 | # Remove prefix (clang++) and store flags into CXX_FLAGS. 40 | CXX_FLAGS=${CXX:8} 41 | CXX=${CXX:0:7} 42 | 43 | date 44 | 45 | cmake $FILES_ROOT \ 46 | -DCMAKE_C_COMPILER="$CC" \ 47 | -DCMAKE_CXX_COMPILER="$CXX" \ 48 | -DCMAKE_C_FLAGS="$C_FLAGS" \ 49 | -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \ 50 | -DUSE_WPE_RENDERER=OFF \ 51 | -DUSE_LCMS=OFF \ 52 | -DUSE_JPEGXL=OFF \ 53 | -DUSE_LIBBACKTRACE=OFF \ 54 | -DENABLE_BUBBLEWRAP_SANDBOX=OFF \ 55 | -DCMAKE_BUILD_TYPE=Debug \ 56 | -DPORT=$PORT \ 57 | -DUSE_SYSTEM_SYSPROF_CAPTURE=OFF \ 58 | -DENABLE_GPU_PROCESS=OFF \ 59 | -DENABLE_SPEECH_SYNTHESIS=OFF \ 60 | -DENABLE_DOCUMENTATION=OFF \ 61 | -DENABLE_INTROSPECTION=OFF 62 | 63 | date 64 | 65 | make -j$(nproc) 66 | cd - 67 | 68 | date 69 | 70 | # Webkit does something weird where it ends up with duplicate copies of some 71 | # header files in OBJDIR/WebCore/PrivateHeaders/. It seems like they're 72 | # installed there and both the original and the private headers end up with 73 | # equivalent analysis records. Based on extremely cursory investigation, only 74 | # the source-tree file has actual include usages. But since only the source 75 | # tree has blame history and is permalink-able, it seems reasonable to nuke the 76 | # private headers' analysis data to avoid them breaking "go to definition" and 77 | # "go to declaration". 78 | rm -rf $INDEX_ROOT/analysis/WebCore/PrivateHeaders 79 | -------------------------------------------------------------------------------- /config6.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "mozilla-central", 5 | "instance_type": "t3.2xlarge", 6 | 7 | "trees": { 8 | "mozilla-central": { 9 | "priority": 500, 10 | "on_error": "halt", 11 | "cache": "everything", 12 | "index_path": "$WORKING/mozilla-central", 13 | "files_path": "$WORKING/mozilla-central/git", 14 | "git_path": "$WORKING/mozilla-central/git", 15 | "git_blame_path": "$WORKING/mozilla-central/blame", 16 | "github_repo": "https://github.com/mozilla/gecko-dev", 17 | "hg_root": "https://hg.mozilla.org/mozilla-central", 18 | "ccov_root": "https://coverage.moz.tools/", 19 | "wpt_root": "testing/web-platform", 20 | "objdir_path": "$WORKING/mozilla-central/objdir", 21 | "codesearch_path": "$WORKING/mozilla-central/livegrep.idx", 22 | "codesearch_port": 8081, 23 | "scip_subtrees": { 24 | "python": { 25 | "scip_index_path": "$WORKING/mozilla-central/python.scip", 26 | "subtree_root": "" 27 | } 28 | } 29 | }, 30 | 31 | "comm-central": { 32 | "priority": 5, 33 | "on_error": "continue", 34 | "cache": "codesearch", 35 | "index_path": "$WORKING/comm-central", 36 | "files_path": "$WORKING/comm-central/git", 37 | "git_path": "$WORKING/comm-central/git", 38 | "git_blame_path": "$WORKING/comm-central/blame", 39 | "github_repo": "https://github.com/mozilla/releases-comm-central", 40 | "hg_root": "https://hg.mozilla.org/comm-central", 41 | "objdir_path": "$WORKING/comm-central/objdir", 42 | "codesearch_path": "$WORKING/comm-central/livegrep.idx", 43 | "codesearch_port": 8082, 44 | "scip_subtrees": {} 45 | }, 46 | 47 | "comm-esr115": { 48 | "priority": 5, 49 | "on_error": "continue", 50 | "cache": "codesearch", 51 | "index_path": "$WORKING/comm-esr115", 52 | "files_path": "$WORKING/comm-esr115/git", 53 | "git_path": "$WORKING/comm-esr115/git", 54 | "git_blame_path": "$WORKING/comm-esr115/blame", 55 | "hg_root": "https://hg.mozilla.org/releases/comm-esr115", 56 | "objdir_path": "$WORKING/comm-esr115/objdir", 57 | "codesearch_path": "$WORKING/comm-esr115/livegrep.idx", 58 | "codesearch_port": 8083, 59 | "scip_subtrees": {} 60 | }, 61 | 62 | "comm-esr128": { 63 | "priority": 5, 64 | "on_error": "continue", 65 | "cache": "codesearch", 66 | "index_path": "$WORKING/comm-esr128", 67 | "files_path": "$WORKING/comm-esr128/git", 68 | "git_path": "$WORKING/comm-esr128/git", 69 | "git_blame_path": "$WORKING/comm-esr128/blame", 70 | "hg_root": "https://hg.mozilla.org/releases/comm-esr128", 71 | "objdir_path": "$WORKING/comm-esr128/objdir", 72 | "codesearch_path": "$WORKING/comm-esr128/livegrep.idx", 73 | "codesearch_port": 8084, 74 | "scip_subtrees": {} 75 | } 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /firefox-main/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | date 8 | 9 | pushd $INDEX_ROOT 10 | $CONFIG_REPO/firefox-shared/process-gecko-analysis.sh 11 | popd 12 | 13 | date 14 | 15 | ## Analyze the unified python corpus 16 | pushd $GIT_ROOT 17 | 18 | # Build a pyrightconfig.json config (config docs are at 19 | # https://microsoft.github.io/pyright/#/configuration). 20 | # 21 | # In order to reduce the memory pressure, we split the files in to two chunks. 22 | # First one is the top-level directory, excluding the web platform test and some 23 | # third party internal things. 24 | # Second one is the web platform test. 25 | # 26 | # Specific settings: 27 | # - verboseOutput: when true, this will print a lot of output about module 28 | # resolution failures and the paths it tried. Unfortunately this does not 29 | # really help with understanding what is going on when the indexing process 30 | # just sits there and gobbles up CPU. 31 | # - include: Just specify the target directory 32 | # - extraPaths: We include all directories for path resolution purposes 33 | # - exclude: For the top-level directory, exclude some heavy/unrelated things 34 | # 35 | # For the top-level directory, there are ~7000 files. Processing all of them 36 | # consumes a lot of memory, thus we specify --max-old-space-size to increase 37 | # the memory limit. 38 | # At the point of this commit, this takes 7.9GB. 39 | 40 | cat < __pyright_exclude_paths.txt 41 | testing/web-platform 42 | third_party/chromium/build 43 | third_party/libwebrtc 44 | third_party/python/aiohttp/examples 45 | third_party/python/cbor2 46 | third_party/python/dlmanager/examples 47 | third_party/python/gyp/test 48 | third_party/python/pip/pip/_internal 49 | third_party/python/pip/pip/_vendor 50 | third_party/python/ply/example 51 | third_party/python/python-hglib/examples 52 | third_party/python/pyyaml/examples 53 | third_party/python/setuptools/setuptools/_vendor 54 | third_party/python/setuptools/setuptools/_distutils/tests 55 | third_party/rust 56 | EOF 57 | 58 | jo verboseOutput=false \ 59 | include=$(echo "." | jo -a) \ 60 | extraPaths=$(git ls-files '*.py' | xargs -n1 dirname | sort | uniq | grep -vf __pyright_exclude_paths.txt | jo -a) \ 61 | exclude=$(cat __pyright_exclude_paths.txt | jo -a) \ 62 | > pyrightconfig.json 63 | 64 | # Absolute paths aren't treated as absolute but instead concatenated, so we need 65 | # to use just a filename and then move it after. 66 | NODE_OPTIONS="--max-old-space-size=10240" scip-python index --show-progress-rate-limit 0 --project-name python --output python.scip 67 | mv python.scip $INDEX_ROOT 68 | 69 | jo verboseOutput=false \ 70 | include=$(echo testing/web-platform | jo -a) \ 71 | extraPaths=$(echo testing/web-platform | jo -a) \ 72 | > pyrightconfig.json 73 | 74 | NODE_OPTIONS="--max-old-space-size=10240" scip-python index --show-progress-rate-limit 0 --project-name python --output python2.scip 75 | mv python2.scip $INDEX_ROOT 76 | 77 | popd 78 | 79 | date 80 | -------------------------------------------------------------------------------- /mozilla-central/build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | date 10 | 11 | pushd $INDEX_ROOT 12 | $CONFIG_REPO/shared/process-gecko-analysis.sh 13 | popd 14 | 15 | date 16 | 17 | ## Analyze the unified python corpus 18 | pushd $GIT_ROOT 19 | 20 | # Build a pyrightconfig.json config (config docs are at 21 | # https://microsoft.github.io/pyright/#/configuration). 22 | # 23 | # In general it seems like we can get a list of all of the python path entries 24 | # possible from processing the files in 25 | # https://searchfox.org/mozilla-central/source/python/sites and where the 26 | # ingestion logic that consumes those files seems to be 27 | # https://searchfox.org/mozilla-central/rev/da49863c3d6f34038d00f5ba701b9a2ad9cbadba/python/mach/mach/requirements.py#88-149 28 | # and the relevant comment about its ingestion is 29 | # https://searchfox.org/mozilla-central/rev/da49863c3d6f34038d00f5ba701b9a2ad9cbadba/python/mach/mach/requirements.py#31-54 30 | # and so we use ripgrep below to extract out the "pth" and "vendored" directives 31 | # that give us paths in trees. We do not process "packages.txt" which is fine 32 | # because it also doesn't seem to be used and it is arguably unlikely that it 33 | # could ever contain paths relative to the root of m-c as we need. 34 | # 35 | # Because pyright seems to (quite fairly) get overwhelmed if we point it at a 36 | # ton of things, we also filter out everything with "third_party" in the actual 37 | # path. We can't do this based on the basis of "vendored" because there are 38 | # things like wptrunner that are vendored but which we deeply want indexed. 39 | # 40 | # We also filter out roots that potentially entrain a whole bunch of files, 41 | # in particular "testing/web-platform" adds like 2000 python test files that 42 | # seems to be what sets searchfox on fire. 43 | # 44 | # Specific settings: 45 | # - verboseOutput: when true, this will print a lot of output about module 46 | # resolution failures and the paths it tried. Unfortunately this does not 47 | # really help with understanding what is going on when the indexing process 48 | # just sits there and gobbles up CPU. 49 | # - includes: We include every path we figure out from the sites .txt files. 50 | # - extraPaths: We include them again here for path resolution purposes since 51 | # it seemed like the path resolution logic didn't consider what we provided to 52 | # "includes" for this purpose. 53 | # 54 | # 55 | jo verboseOutput=false \ 56 | include=$(rg '^(?:pth|vendored):(.+)$' --no-filename --no-heading -Nor '$1' python/sites/*.txt | grep -v third_party | egrep -v '^(build|tools|testing|testing/web-platform)$' | sort | uniq | jo -a) \ 57 | extraPaths=$(rg '^(?:pth|vendored):(.+)$' --no-filename --no-heading -Nor '$1' python/sites/*.txt | grep -v third_party | egrep -v '^(build|tools|testing|testing/web-platform)$' | sort | uniq | jo -a) \ 58 | > pyrightconfig.json 59 | 60 | # Absolute paths aren't treated as absolute but instead concatenated, so we need 61 | # to use just a filename and then move it after. 62 | scip-python index --show-progress-rate-limit 0 --project-name python --output python.scip 63 | mv python.scip $INDEX_ROOT 64 | popd 65 | 66 | date 67 | -------------------------------------------------------------------------------- /config1.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "firefox-main", 5 | "instance_type": "t3.2xlarge", 6 | 7 | "trees": { 8 | "firefox-main": { 9 | "priority": 1000, 10 | "on_error": "halt", 11 | "cache": "everything", 12 | "shared_path": "$WORKING/firefox-shared", 13 | "index_path": "$WORKING/firefox-main", 14 | "files_path": "$WORKING/firefox-main/git", 15 | "git_path": "$WORKING/firefox-main/git", 16 | "git_branch": "main", 17 | "oldtree_name": "mozilla-central", 18 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 19 | "git_blame_path": "$WORKING/firefox-shared/blame", 20 | "github_repo": "https://github.com/mozilla-firefox/firefox", 21 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 22 | "hg_root": "https://hg.mozilla.org/mozilla-central", 23 | "ccov_root": "https://coverage.moz.tools/", 24 | "wpt_root": "testing/web-platform", 25 | "objdir_path": "$WORKING/firefox-main/objdir", 26 | "codesearch_path": "$WORKING/firefox-main/livegrep.idx", 27 | "codesearch_port": 8081, 28 | "scip_subtrees": { 29 | "python": { 30 | "scip_index_path": "$WORKING/firefox-main/python.scip", 31 | "scip_index_path2": "$WORKING/firefox-main/python2.scip", 32 | "subtree_root": "" 33 | } 34 | } 35 | }, 36 | 37 | "mozilla-mobile": { 38 | "priority": 20, 39 | "on_error": "continue", 40 | "cache": "codesearch", 41 | "index_path": "$WORKING/mozilla-mobile", 42 | "files_path": "$WORKING/mozilla-mobile/git", 43 | "git_path": "$WORKING/mozilla-mobile/git", 44 | "objdir_path": "$WORKING/mozilla-mobile/objdir", 45 | "codesearch_path": "$WORKING/mozilla-mobile/livegrep.idx", 46 | "codesearch_port": 8082, 47 | "scip_subtrees": {} 48 | }, 49 | 50 | "mozsearch": { 51 | "priority": 600, 52 | "on_error": "continue", 53 | "cache": "codesearch", 54 | "index_path": "$WORKING/mozsearch", 55 | "files_path": "$WORKING/mozsearch/git", 56 | "git_path": "$WORKING/mozsearch/git", 57 | "git_blame_path": "$WORKING/mozsearch/blame", 58 | "github_repo": "https://github.com/mozsearch/mozsearch", 59 | "objdir_path": "$WORKING/mozsearch/objdir", 60 | "codesearch_path": "$WORKING/mozsearch/livegrep.idx", 61 | "codesearch_port": 8083, 62 | "scip_subtrees": { 63 | "python": { 64 | "scip_index_path": "$WORKING/mozsearch/python.scip", 65 | "subtree_root": "" 66 | }, 67 | "rust": { 68 | "scip_index_path": "$WORKING/mozsearch/rust.scip", 69 | "subtree_root": "tools" 70 | } 71 | } 72 | }, 73 | 74 | "mozsearch-mozilla": { 75 | "priority": 500, 76 | "on_error": "continue", 77 | "cache": "codesearch", 78 | "index_path": "$WORKING/mozsearch-mozilla", 79 | "files_path": "$WORKING/mozsearch-mozilla/git", 80 | "git_path": "$WORKING/mozsearch-mozilla/git", 81 | "git_blame_path": "$WORKING/mozsearch-mozilla/blame", 82 | "github_repo": "https://github.com/mozsearch/mozsearch-mozilla", 83 | "objdir_path": "$WORKING/mozsearch-mozilla/objdir", 84 | "codesearch_path": "$WORKING/mozsearch-mozilla/livegrep.idx", 85 | "codesearch_port": 8084, 86 | "scip_subtrees": {} 87 | }, 88 | 89 | "mozsearch-tests": { 90 | "priority": 550, 91 | "on_error": "continue", 92 | "cache": "everything", 93 | "index_path": "$WORKING/mozsearch-tests", 94 | "files_path": "$WORKING/mozsearch-tests/files", 95 | "objdir_path": "$WORKING/mozsearch-tests/objdir", 96 | "wpt_root": "testing/web-platform", 97 | "codesearch_path": "$WORKING/mozsearch-tests/livegrep.idx", 98 | "codesearch_port": 8085, 99 | "scip_subtrees": {} 100 | } 101 | } 102 | } 103 | -------------------------------------------------------------------------------- /firefox-shared/collapse-generated-files.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -eu # Errors/undefined vars are fatal 4 | set -o pipefail # Check all commands in a pipeline 5 | 6 | # Given a generated file, this script checks to see if the file was the same 7 | # for all platforms where it was generated. If so, it merges the analysis data 8 | # from all the platforms. If not, it keeps copies of the generated file 9 | # and their analysis data in platform-specific folders. 10 | 11 | if [ $# -ne 1 ]; then 12 | echo "Usage: $0 " 13 | exit 1 14 | fi 15 | 16 | GENERATED_FILE=$1 17 | 18 | 19 | function move_file { 20 | mkdir -p "$(dirname $2)" 21 | mv "$1" "$2" 22 | } 23 | 24 | NORMALIZE_EXPR="s#/builds/worker/checkouts/gecko/##g;s#z:/task_[0-9]*/build/src/##gI" 25 | 26 | # Check that all the files (provided as arguments) are the same, after normalizing 27 | # Windows line endings and paths to UNIX. At least one argument must be provided. 28 | # If all the files match, the name of the first one is echo'd, otherwise the empty 29 | # string is echo'd. 30 | function check_all_same { 31 | if [ $# -eq 0 ]; then 32 | # At least one arg must be provided 33 | return 1; 34 | fi 35 | FIRSTFILE=$1; shift; 36 | # Normalize to UNIX in-place. 37 | dos2unix --quiet "$FIRSTFILE" 38 | sed --in-place -Ee "$NORMALIZE_EXPR" "$FIRSTFILE" 39 | while [ $# -gt 0 ]; do 40 | NEXTFILE=$1; shift; 41 | dos2unix --quiet "$NEXTFILE" 42 | sed --in-place -Ee "$NORMALIZE_EXPR" "$NEXTFILE" 43 | cmp --quiet "$FIRSTFILE" "$NEXTFILE" 44 | if [ $? -ne 0 ]; then 45 | # Files aren't the same, echo empty string 46 | echo "" 47 | return 0; 48 | fi 49 | done 50 | # All files are the same, echo any one of them 51 | echo "$FIRSTFILE" 52 | return 0; 53 | } 54 | 55 | 56 | ALL_SAME_AS=$(check_all_same generated-*/$GENERATED_FILE) 57 | if [ "$ALL_SAME_AS" != "" ]; then 58 | echo "Generated file $GENERATED_FILE was identical across platforms where it was created" 59 | move_file "$ALL_SAME_AS" "objdir/$GENERATED_FILE" 60 | # Also merge the analyses files 61 | MERGED_ANALYSIS="analysis/__GENERATED__/$GENERATED_FILE" 62 | mkdir -p "$(dirname $MERGED_ANALYSIS)" 63 | # TODO: this analysis-* glob seems like it ends up not matching anything 64 | # sometimes and this line or even this whole loop could benefit from use of 65 | # "shopt -s nullglob" as documented at 66 | # https://mywiki.wooledge.org/glob#nullglob but which I'm not doing right 67 | # now because it demands some deeper level of investigation. Also, we 68 | # should probably just consider moving more of this logic directly into 69 | # rust code if only because it could let us provide better instrumentation 70 | # of the path of every file and avoid piles of dos2unix, sed, bash, etc. 71 | RUST_LOG=info merge-analyses analysis-*/__GENERATED__/$GENERATED_FILE > $MERGED_ANALYSIS 72 | exit 0 73 | fi 74 | 75 | # The generated file was not the same across all platforms, so 76 | # put the different versions in __$PLATFORM__ subfolders. 77 | for PLATFORM in linux64 linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64 ios; do 78 | if [ ! -f "generated-$PLATFORM/$GENERATED_FILE" ]; then 79 | # The ios platform is only available on the cedar branch. Don't stop if it's not there. 80 | # The *-opt and macosx64-aarch64 platform may not available in old branch. 81 | # The android-armv7 platform is not available in new branches anymore while android-aarch64 is not available for old branches. 82 | if [ $PLATFORM = linux64-opt -o $PLATFORM = ios -o $PLATFORM = macosx64-aarch64 -o $PLATFORM = macosx64-aarch64-opt -o $PLATFORM = win64-opt -o $PLATFORM = android-armv7 -o $PLATFORM = android-aarch64 ]; then 83 | continue 84 | fi 85 | exit 0 86 | fi 87 | echo "Taking generated file $GENERATED_FILE from $PLATFORM" 88 | move_file "generated-$PLATFORM/$GENERATED_FILE" "objdir/__${PLATFORM}__/$GENERATED_FILE" 89 | if [ -f "analysis-$PLATFORM/__GENERATED__/$GENERATED_FILE" ]; then 90 | # Move the analysis file as well 91 | move_file "analysis-$PLATFORM/__GENERATED__/$GENERATED_FILE" "analysis/__GENERATED__/__${PLATFORM}__/$GENERATED_FILE" 92 | fi 93 | done 94 | -------------------------------------------------------------------------------- /shared/collapse-generated-files.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -eu # Errors/undefined vars are fatal 6 | set -o pipefail # Check all commands in a pipeline 7 | 8 | # Given a generated file, this script checks to see if the file was the same 9 | # for all platforms where it was generated. If so, it merges the analysis data 10 | # from all the platforms. If not, it keeps copies of the generated file 11 | # and their analysis data in platform-specific folders. 12 | 13 | if [ $# -ne 1 ]; then 14 | echo "Usage: $0 " 15 | exit 1 16 | fi 17 | 18 | GENERATED_FILE=$1 19 | 20 | 21 | function move_file { 22 | mkdir -p "$(dirname $2)" 23 | mv "$1" "$2" 24 | } 25 | 26 | NORMALIZE_EXPR="s#/builds/worker/checkouts/gecko/##g;s#z:/task_[0-9]*/build/src/##gI" 27 | 28 | # Check that all the files (provided as arguments) are the same, after normalizing 29 | # Windows line endings and paths to UNIX. At least one argument must be provided. 30 | # If all the files match, the name of the first one is echo'd, otherwise the empty 31 | # string is echo'd. 32 | function check_all_same { 33 | if [ $# -eq 0 ]; then 34 | # At least one arg must be provided 35 | return 1; 36 | fi 37 | FIRSTFILE=$1; shift; 38 | # Normalize to UNIX in-place. 39 | dos2unix --quiet "$FIRSTFILE" 40 | sed --in-place -Ee "$NORMALIZE_EXPR" "$FIRSTFILE" 41 | while [ $# -gt 0 ]; do 42 | NEXTFILE=$1; shift; 43 | dos2unix --quiet "$NEXTFILE" 44 | sed --in-place -Ee "$NORMALIZE_EXPR" "$NEXTFILE" 45 | cmp --quiet "$FIRSTFILE" "$NEXTFILE" 46 | if [ $? -ne 0 ]; then 47 | # Files aren't the same, echo empty string 48 | echo "" 49 | return 0; 50 | fi 51 | done 52 | # All files are the same, echo any one of them 53 | echo "$FIRSTFILE" 54 | return 0; 55 | } 56 | 57 | 58 | ALL_SAME_AS=$(check_all_same generated-*/$GENERATED_FILE) 59 | if [ "$ALL_SAME_AS" != "" ]; then 60 | echo "Generated file $GENERATED_FILE was identical across platforms where it was created" 61 | move_file "$ALL_SAME_AS" "objdir/$GENERATED_FILE" 62 | # Also merge the analyses files 63 | MERGED_ANALYSIS="analysis/__GENERATED__/$GENERATED_FILE" 64 | mkdir -p "$(dirname $MERGED_ANALYSIS)" 65 | # TODO: this analysis-* glob seems like it ends up not matching anything 66 | # sometimes and this line or even this whole loop could benefit from use of 67 | # "shopt -s nullglob" as documented at 68 | # https://mywiki.wooledge.org/glob#nullglob but which I'm not doing right 69 | # now because it demands some deeper level of investigation. Also, we 70 | # should probably just consider moving more of this logic directly into 71 | # rust code if only because it could let us provide better instrumentation 72 | # of the path of every file and avoid piles of dos2unix, sed, bash, etc. 73 | RUST_LOG=info merge-analyses analysis-*/__GENERATED__/$GENERATED_FILE > $MERGED_ANALYSIS 74 | exit 0 75 | fi 76 | 77 | # The generated file was not the same across all platforms, so 78 | # put the different versions in __$PLATFORM__ subfolders. 79 | for PLATFORM in linux64 linux64-opt macosx64 macosx64-aarch64 macosx64-aarch64-opt win64 win64-opt android-armv7 android-aarch64 ios; do 80 | if [ ! -f "generated-$PLATFORM/$GENERATED_FILE" ]; then 81 | # The ios platform is only available on the cedar branch. Don't stop if it's not there. 82 | # The *-opt and macosx64-aarch64 platform may not available in old branch. 83 | # The android-armv7 platform is not available in new branches anymore while android-aarch64 is not available for old branches. 84 | if [ $PLATFORM = linux64-opt -o $PLATFORM = ios -o $PLATFORM = macosx64-aarch64 -o $PLATFORM = macosx64-aarch64-opt -o $PLATFORM = win64-opt -o $PLATFORM = android-armv7 -o $PLATFORM = android-aarch64 ]; then 85 | continue 86 | fi 87 | exit 0 88 | fi 89 | echo "Taking generated file $GENERATED_FILE from $PLATFORM" 90 | move_file "generated-$PLATFORM/$GENERATED_FILE" "objdir/__${PLATFORM}__/$GENERATED_FILE" 91 | if [ -f "analysis-$PLATFORM/__GENERATED__/$GENERATED_FILE" ]; then 92 | # Move the analysis file as well 93 | move_file "analysis-$PLATFORM/__GENERATED__/$GENERATED_FILE" "analysis/__GENERATED__/__${PLATFORM}__/$GENERATED_FILE" 94 | fi 95 | done 96 | -------------------------------------------------------------------------------- /config2old.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "instance_type": "t3.2xlarge", 5 | 6 | "trees": { 7 | "mozilla-beta": { 8 | "priority": 120, 9 | "on_error": "continue", 10 | "cache": "codesearch", 11 | "index_path": "$WORKING/mozilla-beta", 12 | "files_path": "$WORKING/mozilla-beta/git", 13 | "git_path": "$WORKING/mozilla-beta/git", 14 | "git_blame_path": "$WORKING/mozilla-beta/blame", 15 | "github_repo": "https://github.com/mozilla/gecko-dev", 16 | "hg_root": "https://hg.mozilla.org/releases/mozilla-beta", 17 | "objdir_path": "$WORKING/mozilla-beta/objdir", 18 | "codesearch_path": "$WORKING/mozilla-beta/livegrep.idx", 19 | "codesearch_port": 8081, 20 | "scip_subtrees": {} 21 | }, 22 | 23 | "mozilla-release": { 24 | "priority": 110, 25 | "on_error": "continue", 26 | "cache": "codesearch", 27 | "index_path": "$WORKING/mozilla-release", 28 | "files_path": "$WORKING/mozilla-release/git", 29 | "git_path": "$WORKING/mozilla-release/git", 30 | "git_blame_path": "$WORKING/mozilla-release/blame", 31 | "github_repo": "https://github.com/mozilla/gecko-dev", 32 | "hg_root": "https://hg.mozilla.org/releases/mozilla-release", 33 | "objdir_path": "$WORKING/mozilla-release/objdir", 34 | "codesearch_path": "$WORKING/mozilla-release/livegrep.idx", 35 | "codesearch_port": 8082, 36 | "scip_subtrees": {} 37 | }, 38 | 39 | "mozilla-esr115": { 40 | "priority": 20, 41 | "on_error": "continue", 42 | "cache": "codesearch", 43 | "index_path": "$WORKING/mozilla-esr115", 44 | "files_path": "$WORKING/mozilla-esr115/git", 45 | "git_path": "$WORKING/mozilla-esr115/git", 46 | "git_blame_path": "$WORKING/mozilla-esr115/blame", 47 | "github_repo": "https://github.com/mozilla/gecko-dev", 48 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr115", 49 | "objdir_path": "$WORKING/mozilla-esr115/objdir", 50 | "codesearch_path": "$WORKING/mozilla-esr115/livegrep.idx", 51 | "codesearch_port": 8083, 52 | "scip_subtrees": {} 53 | }, 54 | 55 | "comm-esr115": { 56 | "priority": 5, 57 | "on_error": "continue", 58 | "cache": "codesearch", 59 | "index_path": "$WORKING/comm-esr115", 60 | "files_path": "$WORKING/comm-esr115/git", 61 | "git_path": "$WORKING/comm-esr115/git", 62 | "git_blame_path": "$WORKING/comm-esr115/blame", 63 | "hg_root": "https://hg.mozilla.org/releases/comm-esr115", 64 | "objdir_path": "$WORKING/comm-esr115/objdir", 65 | "codesearch_path": "$WORKING/comm-esr115/livegrep.idx", 66 | "codesearch_port": 8084, 67 | "scip_subtrees": {} 68 | }, 69 | 70 | "mozilla-esr128": { 71 | "priority": 20, 72 | "on_error": "continue", 73 | "cache": "codesearch", 74 | "index_path": "$WORKING/mozilla-esr128", 75 | "files_path": "$WORKING/mozilla-esr128/git", 76 | "git_path": "$WORKING/mozilla-esr128/git", 77 | "git_blame_path": "$WORKING/mozilla-esr128/blame", 78 | "github_repo": "https://github.com/mozilla/gecko-dev", 79 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr128", 80 | "objdir_path": "$WORKING/mozilla-esr128/objdir", 81 | "codesearch_path": "$WORKING/mozilla-esr128/livegrep.idx", 82 | "codesearch_port": 8085, 83 | "scip_subtrees": {} 84 | }, 85 | 86 | "comm-esr128": { 87 | "priority": 5, 88 | "on_error": "continue", 89 | "cache": "codesearch", 90 | "index_path": "$WORKING/comm-esr128", 91 | "files_path": "$WORKING/comm-esr128/git", 92 | "git_path": "$WORKING/comm-esr128/git", 93 | "git_blame_path": "$WORKING/comm-esr128/blame", 94 | "hg_root": "https://hg.mozilla.org/releases/comm-esr128", 95 | "objdir_path": "$WORKING/comm-esr128/objdir", 96 | "codesearch_path": "$WORKING/comm-esr128/livegrep.idx", 97 | "codesearch_port": 8086, 98 | "scip_subtrees": {} 99 | }, 100 | 101 | "mozilla-esr140": { 102 | "priority": 20, 103 | "on_error": "continue", 104 | "cache": "codesearch", 105 | "index_path": "$WORKING/mozilla-esr140", 106 | "files_path": "$WORKING/mozilla-esr140/git", 107 | "git_path": "$WORKING/mozilla-esr140/git", 108 | "git_blame_path": "$WORKING/mozilla-esr140/blame", 109 | "github_repo": "https://github.com/mozilla/gecko-dev", 110 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr140", 111 | "objdir_path": "$WORKING/mozilla-esr140/objdir", 112 | "codesearch_path": "$WORKING/mozilla-esr140/livegrep.idx", 113 | "codesearch_port": 8087, 114 | "scip_subtrees": {} 115 | } 116 | } 117 | } 118 | -------------------------------------------------------------------------------- /config1old.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "default_tree": "mozilla-central", 5 | "instance_type": "t3.2xlarge", 6 | 7 | "trees": { 8 | "mozilla-central": { 9 | "priority": 500, 10 | "on_error": "halt", 11 | "cache": "everything", 12 | "index_path": "$WORKING/mozilla-central", 13 | "files_path": "$WORKING/mozilla-central/git", 14 | "git_path": "$WORKING/mozilla-central/git", 15 | "git_blame_path": "$WORKING/mozilla-central/blame", 16 | "github_repo": "https://github.com/mozilla/gecko-dev", 17 | "hg_root": "https://hg.mozilla.org/mozilla-central", 18 | "ccov_root": "https://coverage.moz.tools/", 19 | "wpt_root": "testing/web-platform", 20 | "objdir_path": "$WORKING/mozilla-central/objdir", 21 | "codesearch_path": "$WORKING/mozilla-central/livegrep.idx", 22 | "codesearch_port": 8082, 23 | "scip_subtrees": { 24 | "python": { 25 | "scip_index_path": "$WORKING/mozilla-central/python.scip", 26 | "subtree_root": "" 27 | } 28 | } 29 | }, 30 | 31 | "comm-central": { 32 | "priority": 5, 33 | "on_error": "continue", 34 | "cache": "codesearch", 35 | "index_path": "$WORKING/comm-central", 36 | "files_path": "$WORKING/comm-central/git", 37 | "git_path": "$WORKING/comm-central/git", 38 | "git_blame_path": "$WORKING/comm-central/blame", 39 | "github_repo": "https://github.com/mozilla/releases-comm-central", 40 | "hg_root": "https://hg.mozilla.org/comm-central", 41 | "objdir_path": "$WORKING/comm-central/objdir", 42 | "codesearch_path": "$WORKING/comm-central/livegrep.idx", 43 | "codesearch_port": 8083, 44 | "scip_subtrees": {} 45 | }, 46 | 47 | "mozilla-mobile": { 48 | "priority": 20, 49 | "on_error": "continue", 50 | "cache": "codesearch", 51 | "index_path": "$WORKING/mozilla-mobile", 52 | "files_path": "$WORKING/mozilla-mobile/git", 53 | "git_path": "$WORKING/mozilla-mobile/git", 54 | "objdir_path": "$WORKING/mozilla-mobile/objdir", 55 | "codesearch_path": "$WORKING/mozilla-mobile/livegrep.idx", 56 | "codesearch_port": 8084, 57 | "scip_subtrees": {} 58 | }, 59 | 60 | "mozsearch": { 61 | "priority": 600, 62 | "on_error": "continue", 63 | "cache": "codesearch", 64 | "index_path": "$WORKING/mozsearch", 65 | "files_path": "$WORKING/mozsearch/git", 66 | "git_path": "$WORKING/mozsearch/git", 67 | "git_blame_path": "$WORKING/mozsearch/blame", 68 | "github_repo": "https://github.com/mozsearch/mozsearch", 69 | "objdir_path": "$WORKING/mozsearch/objdir", 70 | "codesearch_path": "$WORKING/mozsearch/livegrep.idx", 71 | "codesearch_port": 8085, 72 | "scip_subtrees": { 73 | "python": { 74 | "scip_index_path": "$WORKING/mozsearch/python.scip", 75 | "subtree_root": "" 76 | }, 77 | "rust": { 78 | "scip_index_path": "$WORKING/mozsearch/rust.scip", 79 | "subtree_root": "tools" 80 | } 81 | } 82 | }, 83 | 84 | "mozsearch-mozilla": { 85 | "priority": 500, 86 | "on_error": "continue", 87 | "cache": "codesearch", 88 | "index_path": "$WORKING/mozsearch-mozilla", 89 | "files_path": "$WORKING/mozsearch-mozilla/git", 90 | "git_path": "$WORKING/mozsearch-mozilla/git", 91 | "git_blame_path": "$WORKING/mozsearch-mozilla/blame", 92 | "github_repo": "https://github.com/mozsearch/mozsearch-mozilla", 93 | "objdir_path": "$WORKING/mozsearch-mozilla/objdir", 94 | "codesearch_path": "$WORKING/mozsearch-mozilla/livegrep.idx", 95 | "codesearch_port": 8086, 96 | "scip_subtrees": {} 97 | }, 98 | 99 | "mozsearch-tests": { 100 | "priority": 550, 101 | "on_error": "continue", 102 | "cache": "everything", 103 | "index_path": "$WORKING/mozsearch-tests", 104 | "files_path": "$WORKING/mozsearch-tests/files", 105 | "objdir_path": "$WORKING/mozsearch-tests/objdir", 106 | "wpt_root": "testing/web-platform", 107 | "codesearch_path": "$WORKING/mozsearch-tests/livegrep.idx", 108 | "codesearch_port": 8087, 109 | "scip_subtrees": {} 110 | }, 111 | 112 | "nss": { 113 | "priority": 10, 114 | "on_error": "continue", 115 | "cache": "codesearch", 116 | "index_path": "$WORKING/nss", 117 | "files_path": "$WORKING/nss/git", 118 | "git_path": "$WORKING/nss/git", 119 | "git_blame_path": "$WORKING/nss/blame", 120 | "github_repo": "https://github.com/nss-dev/nss", 121 | "hg_root": "https://hg.mozilla.org/projects/nss", 122 | "objdir_path": "$WORKING/nss/dist", 123 | "codesearch_path": "$WORKING/nss/livegrep.idx", 124 | "codesearch_port": 8088, 125 | "scip_subtrees": {} 126 | } 127 | } 128 | } 129 | -------------------------------------------------------------------------------- /config2.json: -------------------------------------------------------------------------------- 1 | { 2 | "mozsearch_path": "$MOZSEARCH_PATH", 3 | "config_repo": "$CONFIG_REPO", 4 | "instance_type": "t3.2xlarge", 5 | 6 | "trees": { 7 | "firefox-beta": { 8 | "priority": 120, 9 | "on_error": "continue", 10 | "cache": "codesearch", 11 | "shared_path": "$WORKING/firefox-shared", 12 | "index_path": "$WORKING/firefox-beta", 13 | "files_path": "$WORKING/firefox-beta/git", 14 | "git_path": "$WORKING/firefox-beta/git", 15 | "git_branch": "beta", 16 | "oldtree_name": "mozilla-beta", 17 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 18 | "git_blame_path": "$WORKING/firefox-shared/blame", 19 | "github_repo": "https://github.com/mozilla-firefox/firefox", 20 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 21 | "hg_root": "https://hg.mozilla.org/releases/mozilla-beta", 22 | "wpt_root": "testing/web-platform", 23 | "objdir_path": "$WORKING/firefox-beta/objdir", 24 | "codesearch_path": "$WORKING/firefox-beta/livegrep.idx", 25 | "codesearch_port": 8081, 26 | "scip_subtrees": {} 27 | }, 28 | 29 | "firefox-release": { 30 | "priority": 110, 31 | "on_error": "continue", 32 | "cache": "codesearch", 33 | "shared_path": "$WORKING/firefox-shared", 34 | "index_path": "$WORKING/firefox-release", 35 | "files_path": "$WORKING/firefox-release/git", 36 | "git_path": "$WORKING/firefox-release/git", 37 | "git_branch": "release", 38 | "oldtree_name": "mozilla-release", 39 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 40 | "git_blame_path": "$WORKING/firefox-shared/blame", 41 | "github_repo": "https://github.com/mozilla-firefox/firefox", 42 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 43 | "hg_root": "https://hg.mozilla.org/releases/mozilla-release", 44 | "wpt_root": "testing/web-platform", 45 | "objdir_path": "$WORKING/firefox-release/objdir", 46 | "codesearch_path": "$WORKING/firefox-release/livegrep.idx", 47 | "codesearch_port": 8082, 48 | "scip_subtrees": {} 49 | }, 50 | 51 | "firefox-esr140": { 52 | "priority": 100, 53 | "on_error": "continue", 54 | "cache": "codesearch", 55 | "shared_path": "$WORKING/firefox-shared", 56 | "index_path": "$WORKING/firefox-esr140", 57 | "files_path": "$WORKING/firefox-esr140/git", 58 | "git_path": "$WORKING/firefox-esr140/git", 59 | "git_branch": "esr140", 60 | "oldtree_name": "mozilla-esr140", 61 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 62 | "git_blame_path": "$WORKING/firefox-shared/blame", 63 | "github_repo": "https://github.com/mozilla-firefox/firefox", 64 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 65 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr140", 66 | "wpt_root": "testing/web-platform", 67 | "objdir_path": "$WORKING/firefox-esr140/objdir", 68 | "codesearch_path": "$WORKING/firefox-esr140/livegrep.idx", 69 | "codesearch_port": 8083, 70 | "scip_subtrees": {} 71 | }, 72 | 73 | "firefox-esr128": { 74 | "priority": 90, 75 | "on_error": "continue", 76 | "cache": "codesearch", 77 | "shared_path": "$WORKING/firefox-shared", 78 | "index_path": "$WORKING/firefox-esr128", 79 | "files_path": "$WORKING/firefox-esr128/git", 80 | "git_path": "$WORKING/firefox-esr128/git", 81 | "git_branch": "esr128", 82 | "oldtree_name": "mozilla-esr128", 83 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 84 | "git_blame_path": "$WORKING/firefox-shared/blame", 85 | "github_repo": "https://github.com/mozilla-firefox/firefox", 86 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 87 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr128", 88 | "wpt_root": "testing/web-platform", 89 | "objdir_path": "$WORKING/firefox-esr128/objdir", 90 | "codesearch_path": "$WORKING/firefox-esr128/livegrep.idx", 91 | "codesearch_port": 8084, 92 | "scip_subtrees": {} 93 | }, 94 | 95 | "firefox-esr115": { 96 | "priority": 80, 97 | "on_error": "continue", 98 | "cache": "codesearch", 99 | "shared_path": "$WORKING/firefox-shared", 100 | "index_path": "$WORKING/firefox-esr115", 101 | "files_path": "$WORKING/firefox-esr115/git", 102 | "git_path": "$WORKING/firefox-esr115/git", 103 | "git_branch": "esr115", 104 | "oldtree_name": "mozilla-esr115", 105 | "oldgit_path": "$WORKING/firefox-shared/oldgit", 106 | "git_blame_path": "$WORKING/firefox-shared/blame", 107 | "github_repo": "https://github.com/mozilla-firefox/firefox", 108 | "oldgithub_repo": "https://github.com/mozilla/gecko-dev", 109 | "hg_root": "https://hg.mozilla.org/releases/mozilla-esr115", 110 | "wpt_root": "testing/web-platform", 111 | "objdir_path": "$WORKING/firefox-esr115/objdir", 112 | "codesearch_path": "$WORKING/firefox-esr115/livegrep.idx", 113 | "codesearch_port": 8085, 114 | "scip_subtrees": {} 115 | } 116 | } 117 | } 118 | -------------------------------------------------------------------------------- /shared/checkout-gecko-repos.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This is the pre-https://github.com/mozilla-firefox/firefox support 4 | 5 | set -x # Show commands 6 | set -eu # Errors/undefined vars are fatal 7 | set -o pipefail # Check all commands in a pipeline 8 | 9 | if [ $# -ne 3 ]; then 10 | echo "Usage: $0 " 11 | echo " e.g.: $0 mozilla-central master 26bd1e060c5bf1f2f3f3c7f34fae152380cda29c" 12 | exit 1 13 | fi 14 | 15 | REVISION_TREE=$1 16 | BRANCH=$2 17 | INDEXED_HG_REV=$3 18 | 19 | echo Downloading Gecko 20 | pushd $INDEX_ROOT 21 | $CONFIG_REPO/shared/fetch-gecko-tarball.sh gecko $PWD 22 | popd 23 | 24 | date 25 | 26 | echo Downloading Gecko blame 27 | pushd $INDEX_ROOT 28 | $CONFIG_REPO/shared/fetch-gecko-tarball.sh gecko-blame $PWD 29 | popd 30 | 31 | date 32 | 33 | echo Updating git 34 | pushd $GIT_ROOT 35 | # Fetch the m-c repos that we care about with non-grafted cinnabar, so it will have all the necessary hg metadata. 36 | # This could be simplified by using mozilla-unified, but currently mozilla-unified is updated with some amount 37 | # of latency and that can still leave us with stale data. It's better to pull from the individual source-of-truth 38 | # repos directly. Note we only pull the "default/tip" branch from these remotes, rather than all the random tags 39 | # and offshoot branches/closed heads that have accumulated in these repos. 40 | # If we need to fetch project branches in the future, we can fetch those also with cinnabar here. 41 | # Note that this repo may still have a 'projects' and a 'cinnabar' remote left over that we don't use any more. 42 | # Note: the pine branch used to be in here, but was removed in bug 1841395. 43 | # If the pine branch gets reset for a new project and we want to 44 | # add it back in here, we might need to manually update the tarball to 45 | # strip the existing pine branch. I don't know if it will work otherwise. 46 | git config remote.central.url || git remote add -t branches/default/tip central hg::https://hg.mozilla.org/mozilla-central 47 | git config remote.elm.url || git remote add -t branches/default/tip elm hg::https://hg.mozilla.org/projects/elm 48 | git config remote.cedar.url || git remote add -t branches/default/tip cedar hg::https://hg.mozilla.org/projects/cedar 49 | git config remote.cypress.url || git remote add -t branches/default/tip cypress hg::https://hg.mozilla.org/projects/cypress 50 | git config remote.beta.url || git remote add -t branches/default/tip beta hg::https://hg.mozilla.org/releases/mozilla-beta 51 | git config remote.release.url || git remote add -t branches/default/tip release hg::https://hg.mozilla.org/releases/mozilla-release 52 | git config remote.esr140.url || git remote add -t branches/default/tip esr140 hg::https://hg.mozilla.org/releases/mozilla-esr140 53 | git config remote.esr128.url || git remote add -t branches/default/tip esr128 hg::https://hg.mozilla.org/releases/mozilla-esr128 54 | git config remote.esr115.url || git remote add -t branches/default/tip esr115 hg::https://hg.mozilla.org/releases/mozilla-esr115 55 | git config remote.esr102.url || git remote add -t branches/default/tip esr102 hg::https://hg.mozilla.org/releases/mozilla-esr102 56 | git config remote.esr91.url || git remote add -t branches/default/tip esr91 hg::https://hg.mozilla.org/releases/mozilla-esr91 57 | git config remote.esr78.url || git remote add -t branches/default/tip esr78 hg::https://hg.mozilla.org/releases/mozilla-esr78 58 | git config remote.esr68.url || git remote add -t branches/default/tip esr68 hg::https://hg.mozilla.org/releases/mozilla-esr68 59 | git config remote.esr60.url || git remote add -t branches/default/tip esr60 hg::https://hg.mozilla.org/releases/mozilla-esr60 60 | git config remote.esr45.url || git remote add -t branches/default/tip esr45 hg::https://hg.mozilla.org/releases/mozilla-esr45 61 | git config remote.esr31.url || git remote add -t branches/default/tip esr31 hg::https://hg.mozilla.org/releases/mozilla-esr31 62 | git config remote.esr17.url || git remote add -t branches/default/tip esr17 hg::https://hg.mozilla.org/releases/mozilla-esr17 63 | git config cinnabar.graft false 64 | git config fetch.prune true 65 | git -c cinnabar.check=traceback fetch --multiple central elm cedar cypress beta release esr140 esr128 esr115 esr102 esr91 esr78 esr68 esr60 esr45 esr31 esr17 66 | 67 | # If a try push was specified, pull it in non-graft mode so we actually pull those changes. 68 | if [ "$REVISION_TREE" == "try" ]; then 69 | git cinnabar fetch hg::https://hg.mozilla.org/try $INDEXED_HG_REV 70 | fi 71 | 72 | INDEXED_GIT_REV=$(git cinnabar hg2git $INDEXED_HG_REV) 73 | 74 | # If INDEXED_GIT_REV gets set to 40*"0", that means the gecko-dev repo is lagging 75 | # lagging behind the canonical hg repo, and we don't have the source corresponding 76 | # to the indexing run on taskcluster. In that case we error out and abort. 77 | 78 | if [ "$INDEXED_GIT_REV" == "0000000000000000000000000000000000000000" ]; then 79 | echo "ERROR: Unable to find git equivalent for hg rev $INDEXED_HG_REV; please fix cinnabar and retry." 80 | exit 1 81 | fi 82 | 83 | git checkout -B "$BRANCH" $INDEXED_GIT_REV 84 | popd 85 | 86 | date 87 | 88 | # Generate the blame information after checking out the GIT_ROOT to appropriate 89 | # revision above, so that the blame repo's head matches the git repo's head. 90 | echo "Generating blame information..." 91 | pushd $BLAME_ROOT 92 | git reset --soft "$BRANCH" 93 | popd 94 | build-blame $GIT_ROOT $BLAME_ROOT 95 | 96 | date 97 | 98 | # Point the blame repo's HEAD to the commit matching what we have in in the src repo. Note 99 | # that we use `git reset --soft` because we don't need anything in the repo's working dir. 100 | pushd $BLAME_ROOT 101 | BLAME_REV=$(git log -1 --grep=$INDEXED_GIT_REV --pretty=format:%H) 102 | if [ -z "$BLAME_REV" ]; then 103 | echo "Unable to find blame rev for $INDEXED_GIT_REV" 104 | exit 1; 105 | fi 106 | git reset --soft $BLAME_REV 107 | popd 108 | -------------------------------------------------------------------------------- /shared/rebuild-blame.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -x # Show commands 4 | set -eu # Errors/undefined vars are fatal 5 | set -o pipefail # Check all commands in a pipeline 6 | 7 | if [[ -z $MOZSEARCH_PATH ]]; then 8 | echo "MOZSEARCH_PATH needs to be defined for this script" > /dev/stderr 9 | exit 1 10 | fi 11 | 12 | # Options 13 | UPLOAD=no 14 | UPLOAD_INPLACE=no 15 | GIT_REPO_DIR=git 16 | BLAME_REPO_DIR=blame 17 | TARBALL_BASE= 18 | BRANCHES= 19 | 20 | # PArse command-line options 21 | while [[ $# -gt 0 ]]; do 22 | case "$1" in 23 | --help) 24 | set +x 25 | ( 26 | echo "$0 --tarball-base [--git-repo-dir ] [--blame-repo-dir ]" 27 | echo " [--upload [--in-place]] [--branches 'branch1 branch2 [...]']" 28 | echo "" 29 | echo "This script downloads the git tarball from the S3 bucket and builds a corresponding blame" 30 | echo "repo tarball, optionally uploading it back to the S3 bucket." 31 | echo "Arguments:" 32 | echo " --tarball-base The base name of the git repo tarball. e.g. provide 'foo' if the tarball" 33 | echo " holding the git repo is called foo.tar. The generated blame tarball will then" 34 | echo " be named foo-blame.tar. This argument is required" 35 | echo " --git-repo-dir The folder holding the git repo inside the git tarball. Defaults to 'git'" 36 | echo " --blame-repo-dir The folder to hold the blame repo inside the blame tarball. Defaults to 'blame'" 37 | echo " --upload Flag that enables re-uploading of the generated blame tarball back to the S3 bucket." 38 | echo " See also the --in-place option for details on the upload." 39 | echo " --in-place If this flag is specified, any pre-existing blame tarball in the S3 bucket will be" 40 | echo " copied to the backups/ folder, and then the newly generated blame tarball will" 41 | echo " overwrite the original." 42 | echo " If this flag is not specified, the blame tarball will be uploaded as -blame-new.tar," 43 | echo " so that it does not clobber the existing blame tarball." 44 | echo " --branches A space-separate ordered list of branches to build blame for, after the HEAD" 45 | echo " branch is built" 46 | echo "" 47 | )>/dev/stderr 48 | set -x 49 | exit 0 50 | ;; 51 | --tarball-base) 52 | TARBALL_BASE="$2" 53 | shift 54 | shift 55 | ;; 56 | --git-repo-dir) 57 | GIT_REPO_DIR="$2" 58 | shift 59 | shift 60 | ;; 61 | --blame-repo-dir) 62 | BLAME_REPO_DIR="$2" 63 | shift 64 | shift 65 | ;; 66 | --upload) 67 | UPLOAD=yes 68 | shift 69 | ;; 70 | --in-place) 71 | UPLOAD_INPLACE=yes 72 | shift 73 | ;; 74 | --branches) 75 | BRANCHES="$2" 76 | shift 77 | shift 78 | ;; 79 | *) 80 | echo "Unknown argument $1. Try running $0 --help" > /dev/stderr 81 | exit 1 82 | ;; 83 | esac 84 | done 85 | 86 | if [[ -z "$TARBALL_BASE" ]]; then 87 | echo "No tarball base provided. Try running $0 --help" > /dev/stderr 88 | exit 1 89 | fi 90 | 91 | curl -sSfL "https://s3-us-west-2.amazonaws.com/searchfox.repositories/${TARBALL_BASE}.tar" -o "${TARBALL_BASE}.tar" 92 | tar xf "${TARBALL_BASE}.tar" 93 | 94 | # Init a new blame repo 95 | mkdir "${BLAME_REPO_DIR}" 96 | pushd "${BLAME_REPO_DIR}" 97 | git init . 98 | popd 99 | 100 | # Build blame for HEAD, i.e. master branch 101 | build-blame "${GIT_REPO_DIR}" "${BLAME_REPO_DIR}" 102 | 103 | LASTBRANCH="HEAD" 104 | for BRANCH in $BRANCHES; do 105 | # Start the new branch in the blame repo, using the last done 106 | # branch as the starting point so as to maximally reuse previous 107 | # results. 108 | pushd "${BLAME_REPO_DIR}" 109 | git branch "${BRANCH}" "${LASTBRANCH}" 110 | popd 111 | 112 | echo "Generating blame information for ${BRANCH}..." 113 | BLAME_REF="refs/heads/${BRANCH}" build-blame "${GIT_REPO_DIR}" "${BLAME_REPO_DIR}" 114 | 115 | LASTBRANCH="${BRANCH}" 116 | done 117 | 118 | pushd "${BLAME_REPO_DIR}" 119 | # After building the blame with fast-import the pack delta compression is 120 | # generally quite bad. We can reduce the space requirement by asking 121 | # git to repack the repository and compute new delta compression chains. 122 | # See https://stackoverflow.com/a/28721047 for a more detailed discussion. 123 | # This will often reduce space requirement by 2x or more. However, it has 124 | # a tendency to OOM, so we restrict the amount of memory allowed to 3G 125 | # per thread. On our current indexer instances (8 cores, 32G memory) this 126 | # works well. It still causes an OOM on the whatwg-html blame repo, because 127 | # it happens while writing the repacked objects rather than during 128 | # the repacking itself. So what we do is first try with 3G per thread, and 129 | # if that fails, try again with 2G per thread. This gives us the benefit 130 | # of faster/better repacking on repos that don't OOM with 3G, and falls 131 | # back to 2G if that fails. If we find repos that OOM even with 2G we can 132 | # reduce that further on the fallback call. 133 | git repack -f -a -d --depth=250 --window=250 --window-memory=3g || git repack -f -a -d --depth=250 --window=250 --window-memory=2g 134 | popd 135 | 136 | tar cf "${TARBALL_BASE}-blame.tar" "${BLAME_REPO_DIR}" 137 | rm -rf "${GIT_REPO_DIR}" "${BLAME_REPO_DIR}" 138 | 139 | if [[ "$UPLOAD" == "yes" ]]; then 140 | if [[ "$UPLOAD_INPLACE" == "yes" ]]; then 141 | # Copy the existing blame tarball into the backups folder 142 | aws s3 cp "s3://searchfox.repositories/${TARBALL_BASE}-blame.tar" "s3://searchfox.repositories/backups/${TARBALL_BASE}-blame.tar" --acl public-read || true 143 | aws s3 cp "./${TARBALL_BASE}-blame.tar" "s3://searchfox.repositories/${TARBALL_BASE}-blame.tar" --acl public-read 144 | else 145 | aws s3 cp "./${TARBALL_BASE}-blame.tar" "s3://searchfox.repositories/${TARBALL_BASE}-blame-new.tar" --acl public-read 146 | fi 147 | fi 148 | --------------------------------------------------------------------------------