2 |
3 | Baler (bad link reporter) is a [GitHub Action](https://docs.github.com/actions) that tests the URLs inside Markdown files of your GitHub repository. If any of them are invalid, Baler automatically opens a GitHub issue to report the problem(s).
4 |
5 | [](https://github.com/caltechlibrary/baler/blob/main/LICENSE)
6 | 
7 | [](https://github.com/caltechlibrary/baler/releases)
8 | [](https://data.caltech.edu/records/0qetp-p3g60/latest)
9 | [](https://github.com/marketplace/actions/baler-bad-link-reporter)
10 |
11 |
12 | ## Table of contents
13 |
14 | * [Introduction](#introduction)
15 | * [Installation](#installation)
16 | * [Quick start](#quick-start)
17 | * [Usage](#usage)
18 | * [Known issues and limitations](#known-issues-and-limitations)
19 | * [Getting help](#getting-help)
20 | * [Contributing](#contributing)
21 | * [License](#license)
22 | * [Acknowledgments](#acknowledgments)
23 |
24 |
25 | ## Introduction
26 |
27 | The URLs of hyperlinks inside Markdown files may be invalid for any number of reasons, including inaccuracies, typographical errors, and destinations that disappear over time. Manually testing the validity of links on a regular basis is laborious and error-prone. This is clearly a situation where automation helps, and that's where Baler comes in.
28 |
29 | Baler (Bad link reporter) is a [GitHub Action](https://docs.github.com/actions) for automatically testing the links inside Markdown files in your repository, and filing issue reports when problems are found. It's designed to run when changes are pushed to a repository as well as on a regular schedule; the latter helps detect when previously-valid links stop working because of [link rot](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0115253) or other problems. Though it’s not the only GitHub Action available for this purpose, some features set Baler apart from the others:
30 |
31 | * _Simplicity_: a single, short workflow handles both testing files and opening an issue.
32 | * _Smart issue handling_: before it opens a new issue, Baler looks at open issues in the repository. If any previously reported the same URLs, Baler doesn't open a new issue.
33 | * _Informative issue reports_: the issues opened by Baler not only list the URLs that failed; they also describe the reasons for the failures.
34 | * _Simple exclusion list_: fake URLs meant as examples, or real URLs that nevertheless fail when tested from GitHub’s cloud runners, can be skipped by adding them to a file in your repository.
35 |
36 | Baler lets you follow the proverb [“make hay while the sun shines”](https://grammarist.com/make-hay/) – take advantage of opportunities (in this case, easy automated testing) when they’re available.
37 |
38 |
39 | ## Installation
40 |
41 | To use Baler, you need to create a GitHub Actions workflow file in your repository. Follow these simple steps:
42 |
43 | 1. In the main branch of your repository, create a `.github/workflows` directory if one does not already exist.
44 | 2. In the `.github/workflows` directory, create a file named `bad-link-reporter.yml`.
45 | 3. Copy and paste the [contents of `sample-workflow.yml`](https://raw.githubusercontent.com/caltechlibrary/baler/main/sample-workflow.yml) into your `bad-link-reporter.yml` file:
46 |
47 | ```yml
48 | # GitHub Actions workflow for Baler (BAd Link reportER) version 2.0.4.
49 | # This is available as the file "sample-workflow.yml" from the source
50 | # code repository for Baler: https://github.com/caltechlibrary/baler
51 |
52 | name: Bad Link Reporter
53 |
54 | # Configure this section ─────────────────────────────────────────────
55 |
56 | env:
57 | # Files to check. (Put patterns on separate lines, no leading dash.)
58 | files: |
59 | **/*.md
60 |
61 | # Label assigned to issues created by this workflow:
62 | labels: bug
63 |
64 | # Number of previous issues to check for duplicate reports.
65 | lookback: 10
66 |
67 | # Time (sec) to wait on an unresponsive URL before trying once more.
68 | timeout: 15
69 |
70 | # Optional file containing a list of URLs to ignore, one per line:
71 | ignore: .github/workflows/ignored-urls.txt
72 |
73 | on:
74 | schedule: # Cron syntax is: "min hr day-of-month month day-of-week"
75 | - cron: 00 04 * * 1
76 | push:
77 | paths: ['**.md']
78 | workflow_dispatch:
79 |
80 | # The rest of this file should be left as-is ─────────────────────────
81 |
82 | run-name: Test links in Markdown files
83 | jobs:
84 | Baler:
85 | name: Link checker and reporter
86 | runs-on: ubuntu-latest
87 | permissions:
88 | issues: write
89 | steps:
90 | - uses: caltechlibrary/baler@v2
91 | with:
92 | files: ${{github.event.inputs.files || env.files}}
93 | labels: ${{github.event.inputs.labels || env.labels}}
94 | ignore: ${{github.event.inputs.ignore || env.ignore}}
95 | timeout: ${{github.event.inputs.timeout || env.timeout}}
96 | lookback: ${{github.event.inputs.lookback || env.lookback}}
97 | ```
98 |
99 | 4. Save the file, add it to your git repository, and commit the changes.
100 | 5. (If you did the steps above outside of GitHub) Push your repository changes to GitHub.
101 |
102 |
103 | ## Quick start
104 |
105 | Once the workflow is installed in your repository on GitHub, Baler will run whenever a configured trigger event occurs. The trigger conditions are specified in the `on` statement of the `bad-link-reporter.yml` workflow file. The default workflow sets the conditions to be pull requests, a scheduled run once a week, and manual execution.
106 |
107 | Right after installing the workflow in your GitHub repository, it's wise to do a manual test run in order to check that things are working as expected.
108 |
109 | 1. Go to the _Actions_ tab in your repository and click on the workflow named "Bad Link Reporter" in the sidebar on the left:
110 | 
111 | 
114 | 
240 |
241 | 242 |
243 |
244 |
245 |
--------------------------------------------------------------------------------
/SUPPORT.md:
--------------------------------------------------------------------------------
1 | Support
2 | =======
3 |
4 | Thank you for your interest in this project. If you are experiencing problems or have questions, the following are the preferred methods of reaching someone:
5 |
6 | 1. Report a new issue using the [issue tracker](https://github.com/caltechlibrary/baler/issues).
7 | 2. Send email to the Caltech Library: [helpdesk@library.caltech.edu](mailto:helpdesk@library.caltech.edu).
8 | 3. Send email to an individual involved in the project. People's names appear in the top-level `README.md` file in the source code repository.
9 |
--------------------------------------------------------------------------------
/action.yml:
--------------------------------------------------------------------------------
1 | # Summary: GitHub Action definition file for Baler – the Bad Link Reporter.
2 | #
3 | # Copyright 2024 California Institute of Technology.
4 | # License: Modified BSD 3-clause – see file "LICENSE" in the project website.
5 | # Website: https://github.com/caltechlibrary/baler
6 |
7 | name: Baler – BAd Link reportER
8 | description: Test URLs in Markdown files and open an issue for problems found
9 |
10 | inputs:
11 |
12 | # User-level parameters.
13 |
14 | files:
15 | description: Regexes for files to test. Can be comma-separated list.
16 | default: '*.md'
17 | required: true
18 | type: string
19 | ignore:
20 | description: File containing a list of URLs to ignore, one per line.
21 | default: .github/workflows/ignored-urls.txt
22 | required: false
23 | type: string
24 | labels:
25 | description: Labels assigned to GitHub issues created by this workflow.
26 | default: bug
27 | required: false
28 | type: string
29 | lookback:
30 | description: How many previous issues to check for duplicate reports.
31 | default: 5
32 | required: false
33 | type: number
34 | timeout:
35 | description: Number of seconds to wait on unresponsive URL.
36 | default: 15
37 | required: false
38 | type: number
39 |
40 | # Advanced parameters.
41 |
42 | debug:
43 | description: Run in debug mode.
44 | default: false
45 | required: false
46 | type: boolean
47 |
48 | # Internal parameters.
49 |
50 | lyVersion:
51 | description: Version of Lychee link checker program.
52 | default: 0.14.1
53 | required: false
54 | lyDownloadBase:
55 | description: Base download URL for Lychee binaries.
56 | default: https://github.com/lycheeverse/lychee/releases/download
57 | required: false
58 | lyRetryWaitTime:
59 | description: Time to wait between retries, in seconds.
60 | default: 10
61 | required: false
62 |
63 | # Explanation for how different GitHub events are handled, and why:
64 | #
65 | # workflow_dispatch: Tests all .md files matched by ${{inputs.files}},
66 | # regardless of whether they have been modified in the latest commit.
67 | # Rationale: if you're invoking the action manually, you probably intend
68 | # to test the files as they exist in the repository now, and not relative
69 | # to a past commit or other past event.
70 | #
71 | # schedule: Tests all .md files matched by ${{inputs.files}}, regardless
72 | # of whether they have been modified in the latest commit. Rationale:
73 | # (1) it wouldn't make sense to have a nightly run test only the files
74 | # modified in the latest commit, because a previous commit might also
75 | # have modified some Markdown files, which means the latest commit is
76 | # not a good reference point for this purpose; and (2) regularly testing
77 | # all Markdown files, regardless of recent edits, is good for discovering
78 | # links that worked in the past but stopped working due to (e.g.) link
79 | # rot. In case #2, there are no local file changes to trigger on.
80 | #
81 | # all other events: Test the .md files that are being changed as a result of
82 | # the request, compared to the versions of the files in the destination
83 | # branch. The trigger/invocation is controlled by the trigger rules of
84 | # the invoking workflow. (So, for example, in the sample workflow
85 | # provided for users, push events result in testing .md files that were
86 | # modified by the push.)
87 |
88 | runs:
89 | using: composite
90 | steps:
91 | - name: Set preliminary run-time configuration variables.
92 | shell: bash
93 | run: |
94 | if [[ ${{github.event_name == 'push'}} ]]; then
95 | echo "depth_needed=2" >> $GITHUB_ENV
96 | elif [[ ${{github.event_name == 'pull_request'}} ]]; then
97 | echo "depth_needed=1" >> $GITHUB_ENV
98 | else
99 | echo "depth_needed=0" >> $GITHUB_ENV
100 | fi
101 |
102 | - name: Check out source repository.
103 | uses: actions/checkout@v4
104 | with:
105 | fetch-depth: ${{env.depth_needed}}
106 | persist-credentials: true
107 |
108 | - name: Get list of all files matching the desired pattern.
109 | uses: tj-actions/glob@v20
110 | id: match-files
111 | with:
112 | files: ${{inputs.files}}
113 |
114 | - name: Check if any files matching the pattern have been modified.
115 | id: test-changes
116 | uses: tj-actions/changed-files@v42
117 | with:
118 | files: ${{inputs.files}}
119 |
120 | - name: Construct list of files to act on.
121 | shell: bash
122 | run: |
123 | event=${{github.event_name}}
124 | if [[ "$event" == "workflow_dispatch" || "$event" == "schedule" ]]; then
125 | echo "considered=(${{steps.match-files.outputs.paths}})" >> $GITHUB_ENV
126 | elif [[ "${{steps.test-changes.outputs.any_changed}}" == "true" ]]; then
127 | echo "considered=(${{steps.test-changes.outputs.all_changed_files}})" >> $GITHUB_ENV
128 | fi
129 |
130 | - name: Decide whether to proceed.
131 | shell: bash
132 | run: |
133 | if [[ "${{steps.test-changes.outputs.any_changed}}" == "true" \
134 | || "${{github.event_name}}" == "workflow_dispatch" ]]; then
135 | echo "Continuing workflow."
136 | echo "continue=true" >> $GITHUB_ENV
137 | else
138 | echo "Stopping workflow."
139 |
140 | # Report that we have nothing to do.
141 | msg="241 | 242 |
243 |
244 | Relevant files are unchanged – nothing to do
142 |Link testing was skipped because the relevant
143 | files have not changed. The files that were considered are the
144 | following:${{inputs.files}}.
Run-time error
251 |Baler encountered an exception. This was most likely caused by a 252 | bug in Baler itself. Please report this to the developers. You can 253 | report it by opening an issue in the GitHub repository at 254 | 255 | https://github.com/caltechlibrary/baler.
" 256 | echo "$msg" >> $GITHUB_STEP_SUMMARY 257 | # Bail with an error. 258 | exit 2 259 | fi 260 | 261 | - name: Check if we've already opened a duplicate issue. 262 | if: env.continue == 'true' && env.lychee_exit_code != 0 263 | shell: bash 264 | run: | 265 | # First, save the current issue text body, to use for comparisons. 266 | # Strip the end of the body b/c it has this workflow's unique run 267 | # id, which can never match any other issue. Also, sort the body; 268 | # this is a way to normalize the contents to avoid false positives 269 | # when, e.g., two runs end up putting URLs in different orders. 270 | sed_cmd='s/\(.*\)This content was produced by.*/\1/' 271 | current=$(sed "$sed_cmd" < ${{env.lychee_output}} | sort) 272 | # Get the issue numbers for the last N issues, where N = lookback. 273 | endpoint="https://api.github.com/repos/${{github.repository}}/issues" 274 | query="q=state:open&per_page=${{inputs.lookback}}" 275 | accept="Accept: application/vnd.github+json" 276 | auth="Authorization: Bearer ${{github.token}}" 277 | issues=$(curl -s -H "$accept" -H "$auth" "$endpoint?$query") 278 | issue_numbers=($(jq '.[].number' <<<"$issues")) 279 | # Iterate over the issues & compare the bodies. 280 | for number in ${issue_numbers[@]}; do 281 | previous=$(curl -s -H "$accept" -H "$auth" "$endpoint/$number" | jq -r '.body') 282 | previous=$(echo "$previous" | sed 's/\\n/\n/g' | sed "$sed_cmd" | sort) 283 | if [[ "$current" == "$previous" ]]; then 284 | link="https://github.com/${{github.repository}}/issues/$number" 285 | 286 | # Report that we found a match. 287 | msg="Bad links found, but issue creation skipped
288 |One or more invalid URL(s) have been found; however, no new 289 | issue has been opened in the repository because the same URL(s) 290 | were reported in issue #$number.
" 291 | echo "$msg" >> $GITHUB_STEP_SUMMARY 292 | 293 | # Skip the rest of the workflow. 294 | echo "continue=false" >> $GITHUB_ENV 295 | break 296 | fi 297 | done 298 | 299 | - name: Open a new issue/ticket to report the problems. 300 | if: env.continue == 'true' && env.lychee_exit_code != 0 301 | id: create-issue 302 | uses: peter-evans/create-issue-from-file@v5.0.0 303 | with: 304 | title: Invalid URLs in Markdown files 305 | content-filepath: ${{env.lychee_output}} 306 | labels: ${{env.labels}} 307 | token: ${{github.token}} 308 | 309 | - name: Put a link to the issue in the workflow output. 310 | if: env.continue == 'true' && env.lychee_exit_code != 0 311 | env: 312 | issue-number: ${{steps.create-issue.outputs.issue-number}} 313 | shell: bash 314 | run: | 315 | number="${{env.issue-number}}" 316 | link="https://github.com/${{github.repository}}/issues/$number" 317 | msg="
Invalid URLs found
318 |Ticket #${{env.issue-number}} has been 319 | created.
" 320 | echo "$msg" >> $GITHUB_STEP_SUMMARY 321 | # Report a failure. 322 | exit 1 323 | 324 | - name: Log internal variable values in debug mode or if a failure occurs. 325 | if: inputs.debug == 'true' || failure() 326 | shell: bash 327 | run: | 328 | echo "Summary of run-time variables" 329 | echo "=============================" 330 | echo "repository_owner = ${{github.repository_owner}}" 331 | echo "repository = ${{github.event.repository.name}}" 332 | echo "event name = ${{github.event_name}}" 333 | echo "files: ${{inputs.files}}" 334 | echo "ignore: ${{inputs.ignore}}" 335 | echo "labels: ${{inputs.labels}}" 336 | echo "lookback: ${{inputs.lookback}}" 337 | echo "timeout: ${{inputs.timeout}}" 338 | echo "debug: ${{inputs.debug}}" 339 | echo "depth_needed (for git history) = ${{env.depth_needed}}" 340 | echo "files considered = ${{env.considered}}" 341 | echo "continue = ${{env.continue}}" 342 | echo "env.lychee_exit_code = ${{env.lychee_exit_code}}" 343 | echo "" 344 | echo "Content of lychee config file" 345 | echo "=============================" 346 | cat ${lychee_config} 347 | echo "=============================" 348 | 349 | - name: Report if checks were successful. 350 | if: env.continue == 'true' && env.lychee_exit_code == 0 351 | shell: bash 352 | run: | 353 | msg="No invalid URLs found
354 |No bad links were found in the files matched by this pattern:
355 | ${{inputs.files}}