Reports for GitHub
One review's result is written for GitHub in a few ways, each a format of --format (-f) and each from the public view, so no sink a stranger reads holds a model's reasoning, a discarded finding, a model's name or the usage: a summary for a pull request or the job summary, the payload that quorum-action posts, and SARIF for code scanning. This page lists the formats and describes the summary, the payload and SARIF.
The formats
| Format | What it writes | Public view only |
|---|---|---|
text | The review as a person reads it: The report | No |
json | The whole result as JSON: The report | No |
brief | One line of counts and code words: The report | No |
markdown | The summary of the review, below | Yes |
payload | The payload quorum-action posts, below | Yes |
sarif | SARIF for GitHub's code scanning, below | Yes |
A format marked public only renders the public view of whatever it is given, a full result included, so no way of calling it publishes the reasoning: Public-only formats. --view does not change it. The three that are not marked write what they are given, so --view public is what makes them the public view.
The summary
-f markdown writes the summary, which a CI job posts as a pull request comment or appends to its job summary. For a review of a range that kept one finding, left one unreviewed and had a gap, it is:
<!-- quorum:summary:v1 -->
### quorum review
**exit 1, findings:** a gating profile kept a finding at or above its fail-on level
| Profile | Fail-on | Kept | Unreviewed | Discarded | Outside the diff | Coverage |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| security | high | 1 | 1 | 1 | 0 | 1 of 3 files reviewed, 1 in part, 1 not |
#### Findings
- **HIGH** security · `src/auth/login.ts` line 42 · `CWE-89` · kept by 3 of 3 lenses
`SQL built from the request's id parameter reaches db.query`
```text
41 const sql = 'SELECT * FROM users WHERE id = ' + id
> 42 + db.query(sql)
43 return rows
```
- **UNREVIEWED** security · `src/session.js` line 20 · `CWE-338` · kept by 1 of 3 lenses, 1 vote failed
`Session tokens come from Math.random, which is not a cryptographic generator.`
```text
- return crypto.randomBytes(32).toString('hex')
> 20 + return Math.random().toString(36).slice(2)
21 }
```
#### Gaps
- security, binary: the diff carries no content for a binary file
#### Files left out
- `assets/logo.png` (binary)
#### Flags
- hidden-characters at `src/session.js` line 20: `U+202E on 1 added line(s)`
---
quorum 0.6.0 · range `1f3a9c2aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa`...`7b2e4d0aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa`- The first line is the marker,
<!-- quorum:summary:v1 -->, the only raw HTML the summary holds. It names the summary and its version, so a program can find its own comment. It proves nothing without its author: anyone can write the marker, so check who wrote the comment too. - The verdict says what the exit code says, in the words of the last line of the text report:
exit 1, findings,exit 4, incompleteorexit 0, pass. - The table has a row for each profile: the level it fails at (
nonefor an advisory profile), how many findings it kept and left unreviewed, how many the panel discarded (counted, never listed), how many findings it dropped for citing code outside the diff, and how much of the diff it reviewed. - The findings are the kept and the unreviewed ones, each with its level, its profile, its file and line, its category, how its lenses voted, its claim and its evidence, numbered as the text report numbers it: the cited row is marked
>. A finding that fails its profile comes first, then the high ones, the medium, the unreviewed and the low, each group by file and then by line, so the budget drops the least important last. - The gaps are one line for each profile and reason, in fixed words that say nothing of the diff, with a count in parentheses when a profile records a reason more than once.
- The files left out and the flags list the first 20 each, then how many more. A file with no name, the part of a diff that no header names, is
(no path). A held file's line carries its reason and its rule:(secret: name). - The trailer is the version and what was reviewed: a range's two commits, the commit a local or staged review sits on, the commit of a path review, or
diff.
A section with nothing to list is left out, so the review of a blank diff is a heading, a verdict, a table and a trailer.
Why every text is in a code span
A claim, a category, a file name, a row of code and a flag's detail are text that a model or the diff wrote, and an attacker can influence it. Written bare, a name or a claim could mention a person, link an issue or a commit, load an image, become an emoji or math, or hold HTML. A code span is the one Markdown construct that stops all of that at once, and a backslash does not stop a mention, so the summary puts each such text in a span, whose fence quorum computes to be longer than any run of backticks inside, and the evidence in a fenced block that no row can close. No such text is in a table cell, since a | splits a cell even inside a span. Every line passes inertLine, so a log that echoes the summary starts no workflow command. The rule, and what the summary never writes, are in Markdown.
The size
A summary is at most 60,000 UTF-16 code units, inside GitHub's limit on a comment and the job summary's. A large review can hold more: 100 findings with their evidence are about 280,000 characters. The summary drops whole pieces until it fits, in this order, and counts what it dropped:
- the evidence of the findings, from the last one up, with a note that says how many findings lost theirs;
- the flags, last first;
- the files left out, last first;
- the findings, last first, so those that fail their profile go last;
- the gap lines;
- the profiles' rows.
Each list that lost entries ends with a line that counts them, as - and 5 more, listed in the review's JSON result, and the findings' line names their levels. The marker, the heading, the verdict, the section headings, the count lines, the note and the trailer stay whatever the budget, since they hold only words quorum wrote, numbers, the version and commit ids. A string is never cut, so a code span or a block is whole or gone.
Writing it
quorum --range "$BASE...$HEAD" -f markdown >> "$GITHUB_STEP_SUMMARY"-f markdown writes the summary on stdout, and --output markdown=<path> writes it to a file as well as the format on stdout: --output markdown="$RUNNER_TEMP/summary.md". Either is the public view whatever --view says. The exit code is the review's, so the step fails on a finding as the review does; the summary is written first. A program calls renderReport(result, { format: 'markdown' }): renderReport.
The payload
-f payload writes the document that quorum-action posts as it is: one versioned JSON object that holds the commit the review is about (head), what it concluded (conclusion: pass, findings or incomplete, the exit code's meaning), the summary above, and a line comment for each kept or unreviewed finding that sits on a file of the diff, with the count of the findings no line can hold (unplaced). Each comment's body starts with a marker that holds the finding's fingerprint, then the level, the profile, the category, how the lenses voted and the claim, the category and the claim in code spans. The JSON is written by inertJson, so every text in it is inert and the Action composes none of its own. It is the public view whatever --view says.
quorum --range "$BASE...$HEAD" -f brief --output payload="$RUNNER_TEMP/payload.json"Every field, the body and what the Action owns: The Action's payload.
SARIF
-f sarif writes SARIF 2.1.0, the format GitHub's code scanning reads, as one run of the public view:
- The tool is
quorum, with its version, and a rule for each profile,quorum/<profile>, whose texts quorum's code writes and whose help says what each level means for that profile'sfailOn. - A result for each kept or unreviewed finding that sits on a file of the diff, at its path, its line and the columns of its cited row. Its message is the level, how the lenses voted, the category and the claim, so a reviewer sees what the summary says. Its
quorumFingerprint/v1is the finding's fingerprint. A kept finding also gets its confidence asprecision. A finding that names no file of the diff, as a review of a diff with no per-file headers has, or a path that is not a clean relative path (it starts with/, or has an empty,.or..segment), has no place for a result and is counted in a notification. - The run's commit, when the source has one: a range's head, a local or staged review's head, or a path review's commit.
quorum --range "$BASE...$HEAD" -f brief --output sarif="$RUNNER_TEMP/quorum.sarif"The levels
A result's level agrees with the exit code. It is error exactly when the finding fails its profile, the rule that sets exit 1: a kept finding at or above the profile's failOn. Another kept high or medium finding and an unreviewed one are warning, a kept low one is note, and a profile at none writes no error. GitHub's check fails on an error alert, so it fails where quorum's gate does.
A result sets no security-severity. GitHub reads that number before the level, so a number quorum chose would decide the check instead of the profile's failOn, which is what the owner configured.
The notifications
The run's executionSuccessful is false whenever a gating profile has a gap or an unreviewed finding, even when a kept finding sets exit 1 or --allow-incomplete lifted the gaps, since a run that did not review everything is not complete. A gap's notification has the level error for a gating profile and warning for an advisory one, the gap's fixed words, and the files it is about, at most 100 with their count. Only a file that is a clean relative path, as for a result, gets a location: another is counted and never located, so no location names another place. A file left out with no gap, which is one that an exclude pattern or a path review's skip rule left out, or a deleted secret, is a note, and so is the count of the findings no result holds. They come in that order: the gaps, the files left out, the count.
GitHub does not show notifications. They are in the file for the tools that read it and for the person who opens it, and the summary is where a gap is read.
What GitHub does with it
- Duplicate alerts. GitHub matches an alert across uploads by
primaryLocationLineHash, whichupload-sarifcomputes from the code at the result's line, by reading the files under its checkout, when the file leaves it out. Without a checkout, as in the post job below, no line hash is computed, so each push can open new alerts. Quorum's own fingerprint is a different hash, so it sits under its own key, which GitHub ignores. A finding whose line moves in a later push can open a new alert beside the old one. This is the price of keeping an alert from being matched to a dismissed one by a reading quorum did not foresee. - Private repositories. Code scanning is free for a public repository, and a private one needs GitHub Code Security. A review is then never only SARIF: the summary and the exit code say the same, where code scanning is not there.
- Nothing uploads it. quorum writes the file and uploads nothing;
github/codeql-action/upload-sarifdoes.
quorum report
The post job of a two-job workflow holds no model key and no checkout of the pull request: it downloads the result the review job saved and renders it with quorum report, which calls no model, reads no config and runs no git.
quorum report result.json -f payload --output markdown="$GITHUB_STEP_SUMMARY" --output sarif=quorum.sarifIt reads the full result or the public view, a file or stdin, only from the same release, through a strict schema, and writes any format as the review command would have from the same result: the summary, the payload and the SARIF are the public view whatever --view says. It exits 0 whatever the review found, since the review job's exit code is the gate, and 2 for a result it refuses, which fails the post job: a result of another release, a truncated download and a public view with a field added are each one message that names the cause and holds no value, no text and no key but a code word of the file. Its flags, its refusals and its exit codes: quorum report.
A GitHub workflow
Two jobs keep the model key, the write token and the pull request's code apart: a review job that holds the key and a post job that holds the token. A fork's pull request runs neither.
name: Quorum
on:
pull_request:
permissions: {}
env:
QUORUM_VERSION: 0.7.0
jobs:
review:
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Check out the base, with the history the range needs
uses: actions/checkout@v7
with:
ref: ${{ github.event.pull_request.base.sha }}
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: 24
- name: Install quorum
run: npm install -g "@tobi-or-not/quorum@$QUORUM_VERSION"
- name: Review the pull request's commits
env:
BASE: ${{ github.event.pull_request.base.sha }}
HEAD: ${{ github.event.pull_request.head.sha }}
QUORUM_RESEARCH_PRESET: anthropic
QUORUM_RESEARCH_MODEL: ${{ vars.QUORUM_MODEL }}
QUORUM_RESEARCH_CONTEXT_TOKENS: ${{ vars.QUORUM_CONTEXT_TOKENS }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: quorum --range "$BASE...$HEAD" --view public -f brief --output json="$RUNNER_TEMP/quorum-result.json"
- name: Keep the public view for the post job
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: quorum-result
path: ${{ runner.temp }}/quorum-result.json
retention-days: 1
post:
needs: review
if: ${{ !cancelled() && needs.review.result != 'skipped' }}
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
steps:
- uses: actions/download-artifact@v8
with:
name: quorum-result
path: ${{ runner.temp }}
- uses: actions/setup-node@v7
with:
node-version: 24
- name: Install quorum
run: npm install -g "@tobi-or-not/quorum@$QUORUM_VERSION"
- name: Render the summary and SARIF from the saved result
run: |
quorum report "$RUNNER_TEMP/quorum-result.json" -f brief \
--output markdown="$RUNNER_TEMP/quorum-summary.md" --output sarif="$RUNNER_TEMP/quorum.sarif"
cat "$RUNNER_TEMP/quorum-summary.md" >> "$GITHUB_STEP_SUMMARY"
- uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: ${{ runner.temp }}/quorum.sarif
category: quorum
ref: refs/pull/${{ github.event.pull_request.number }}/head
sha: ${{ github.event.pull_request.head.sha }}Set QUORUM_VERSION to the release you run. The model variables are the example's: use the provider, model and window of your own config, and a config file from the base checkout, as Trust says.
The review job
- It checks out the base, not the pull request.
refis the base commit, so no checkout of the pull request's head exists in a job that holds a key, and a config file the job reads is the base's.--rangereads the head's commits with plumbing alone,rev-parse,merge-baseanddiffof two commits, whose objects the fetch brought: A pull request from a fork explains the reading.persist-credentials: falseleaves no token in the checkout's git config. fetch-depth: 0brings the history where the two sides meet, which a shallow clone lacks: A range of commits.--view publickeeps the reasoning, the discarded findings, the models and the usage out of the artifact and the log, which anyone signed in can read on a public repository: The public view.- The result goes to
$RUNNER_TEMP, a directory of the runner that a pull request cannot plant a file in, as it can in the workspace. The step's exit code is the gate:0,1,2,3or4, as Using it in CI lists. The upload runs whatever it was, and the post job renders a review that wrote a result: a review that found something (1) or left a gap (4) did. An exit2or3reviewed nothing and wrote none, so the post job fails at its download, and the review job's own failure is the cause to read.
The post job
- It holds no model key, and it runs nothing a model wrote: it downloads the saved result and renders it with
quorum report, which calls no model, reads no config and runs no git. - It installs the same release, since the reader takes only a result of its own release: quorum report.
- The summary is appended to the job summary, and the SARIF goes to code scanning with
github/codeql-action/upload-sarif. A private repository needs GitHub Code Security for that step: What GitHub does with it. - The upload names the head.
refandshaare the pull request's head,refs/pull/<number>/headand its commit, because quorum reviews the changes from the merge base to the head, so every line of a result is a line of the head. Without themupload-sarifon apull_requestuses the merge commit, which the post job has no checkout to correct, and an alert can land on another line, or outside the pull request's diff, when the base has changed the same file since the merge base. actions: readandcontents: readstay. On a private repositoryupload-sarifreads the run and the repository with them, so a job trimmed tosecurity-events: writealone fails at the upload there. They grant no write.
What the workflow settles
- A fork's pull request runs neither job. The review job's
ifcompares the pull request's head repository with this one and skips a fork's, and the post job'sneedsskips with it. A fork's run gets no secret, andpull_request_target, which would give it one, runs the base's workflow beside the fork's code, so the workflow never uses it. A maintainer reviews a fork's pull request from a clone of their own: A pull request from a fork. - Every value reaches a script through
env:. Norun:holds a${{ }}expression: the commit ids, the key and the variables are set inenv:and read as$BASEand$HEAD, so none of them is parsed as shell code. - Pin each action to a commit, as this repository's workflows do, with the tag in a comment:
uses: actions/checkout@<commit> # v7.0.1. The tags above keep the example short, and a tag can be moved. - Line comments on the pull request are
quorum-action's, from the payload: The payload. The workflow above posts the summary and SARIF only.
Next
Threat model — the attacker, the assets, the residual risk stated plainly, the controls in place today, and what the panel's independence is worth.
