Scoring methodology v2

The scan protocol

This directory measures sscsb-control adoption — how much of the supply-chain posture sscsb can bootstrap and verify a repository has actually committed to. It is not a general security audit. In methodology v2, a repository using an equivalent tool sscsb doesn't model scores a gap for that control — Dependabot in place of Renovate, say. Tool-equivalence mapping is a roadmap item for a future methodology version. Every version bump is recorded here and displayed on each repo's page.

1. Shallow-clone the repository's default branch. The target's code is never executed.
2. Snapshot the committed file list (git ls-files).
3. Run sscsb init, then sscsb verify --format json and sscsb report --format json.
4. Reclassify: any control whose passing evidence was created by init scores gap, per the class rules below.
5. Delete the clone. A maintainer reviews every record before it publishes.

Evidence classes

Not every check can be answered from the same place. These five groups say who could see what, and each repository page names the group beside every check.

Scope

A control is in scope when it is enabled by sscsb's defaults or by the repository's own committed .sscsb/config.toml. Disabling a default-on control scores a gap — the denominator cannot be shrunk. Enabling an optional control puts it in scope against real evidence — never free points.

Authenticated records: the trust chain

Three things can produce a record, and they see different amounts. That is the evidence source — who ran the scan, and therefore how much of the project they could see, shown as a badge on every listing. It decides nothing about the score. It tells you how far the scanner could see. Where a build leaves a signed receipt for what it produced, that receipt is an attestation — a signed receipt saying which build produced which file.

The checks that matter most — branch protection, Actions token permissions, security-feature enablement — are repository settings, readable only with repository credentials. The only place a complete scan can run is the repository's own CI, which raises the obvious question: when a repository hands in its own report card, why believe it? The answer is OpenSSF Scorecard's, adopted deliberately.

1. Verified scanner. The action installs an sscsb release only after checking its Sigstore bundle. The bundle is checked against the tool repository's own release-workflow identity, at that exact tag. The scanner proves its own provenance before assessing anyone else's.
2. Signed record. With id-token: write, the action signs scan-record.json using keyless signing — signing with a short-lived certificate tied to the build, so there is no long-lived key to steal. Fulcio issues a short-lived certificate whose identity is the producing workflow: repository, workflow path, and branch ref. GitHub's OIDC issuer burns those in; the record does not assert them. The signature is then logged to Rekor.
3. Pinned verification. Ingest runs cosign verify-blob pinned to https://github.com/OWNER/REPO/.github/workflows/sscsb-scan.yml@refs/heads/<default branch> — the default branch fetched live from GitHub, never read from the record — and to the commit the record claims. A third party cannot mint that identity. A feature-branch or renamed-workflow signature does not verify. A record whose bundle fails verification is rejected outright. And the directory publishes the bundle beside every verified record, so anyone can re-run the check.
4. Human gate, still. Verified or not, nothing lists without a maintainer's publish label. An action-lane record that arrives unsigned is listed only as an unverified claim.

What this deliberately does not prove: that the workflow which signed the record was unmodified. A maintainer who edits their own sscsb-scan.yml can sign whatever it emits. OpenSSF Scorecard closes that gap. It fetches the producing workflow at the certificate's commit and rule-checks it: allow-listed steps only, no environment redirection, hosted runners. That is this directory's ingestion-hardening milestone. Until then, the human publish gate is the backstop.

Local records: the third lane

About a dozen checks describe a developer's own machine, where no scan can look. A maintainer answers them by running the scan there and signing the result. The key they sign with is one the repository already publishes, in its signer list — the file a project commits naming the keys it accepts scans from.

Ten or eleven controls per repository are class C: they describe the development environment — commit signing configuration, AI-assistant trailers, dependency gates, package-trust hooks. No repository scan can observe them, from inside CI or out, so they score unverified and sit outside every denominator. For a repository that has actually adopted them, that is a permanent ceiling on evidence coverage. It is why a well-run repository can still read provisional.

The local lane closes exactly that gap. A maintainer runs

sscsb scan --local --submit

which scans the working tree, writes a record, and signs it with the maintainer's git signing key. That is the same user.signingkey / gpg.format=ssh / gpg.ssh.program configuration git itself uses, so a 1Password- or hardware-backed key works untouched.

The contract

The tool and this directory are two programs that have to agree exactly — down to a namespace string and a file path. Otherwise every submission fails in a way neither side can see. So the agreement is written down once, in the tool's docs/local-scan.md, and mirrored here byte for byte. Both trees test their own copy against these values, and both pin the same digest over the block. An edit on one side that the other does not mirror fails a test, instead of shipping a lane that does not work.

sscsb local-lane contract v1
command              sscsb scan --local --submit
sshsig-namespace     sscsb-scan-record
record-path          .sscsb/scan-record.local.json
signature-path       .sscsb/scan-record.local.json.sig
anchor-path          .sscsb/policy/allowed_signers
anchor-namespaces    git,sscsb-scan-record
signed-bytes         the bytes of .sscsb/scan-record.local.json, verbatim
record-shape         ScanRecord
schema-version       1
methodology-version  2
record-fields        schema_version methodology_version repo scanned_at scanner request_issue controls score
repo-fields          owner name url default_branch commit description
control-fields       id phase in_scope raw_outcome scan_outcome reclassified reason messages
score-fields         grade provisional overall_percent evidence_coverage_percent phases
submission-label     local-scan-result

Two lines carry most of the weight. record-path and signature-path are committed paths. The submission is a pointer: the directory reads the record, the signature and the trust anchor out of the public repository. Nothing a submitter types reaches the bytes that get verified. And record-shape is ScanRecord — the directory's own schema, the same one every other lane produces, with every required field including methodology_version. A signature makes bytes unreshapeable, so the shape has to be right at signing time.

The trust anchor

A local scan has no OIDC identity: there is no CI run to issue a certificate against. What it does have is a file the repository already commits. .sscsb/policy/allowed_signers — generated from .sscsb/policy/signers.toml — is the anchor the commit-signing control itself uses. So ingest verifies the record with

ssh-keygen -Y verify -f allowed_signers \
  -I "<signer>" -n sscsb-scan-record \
  -s .sscsb/scan-record.local.json.sig < .sscsb/scan-record.local.json

against an allowed_signers fetched from the public repository at the scanned commit. That is committed content, read through the GitHub contents API at that ref, never anything the submitter hands us. A record whose signature does not verify is refused outright — exactly as a Sigstore bundle that fails cosign verify-blob is refused.

Only a human-class approved signer may assert a local record

sscsb grants the sscsb-scan-record namespace to class = "human" signers only. A ci or ai entry in .sscsb/policy/signers.toml keeps namespaces="git" and nothing else, so its commits still verify and its scan records cannot.

Three reasons, none of them tidiness. A local record is a person's attested word about a machine nobody can inspect. It is the one lane whose local-environment verdicts count with no corroboration. What makes that acceptable is that a named human put their key behind it. CI does not need the grant: it has the authenticated lane, which proves strictly more. And an AI-class signer asserting one would contradict the invariant the signing policy is built on: an agent may draft anything and signs nothing.

The refusal is structural, not advisory. Because the namespace is simply absent from the anchor line, the ssh-keygen -Y verify above fails: the tool refuses to produce the record, and this directory refuses to ingest one, without either of them re-implementing the rule.

The same fact has a consequence. A repository anchored before this lane existed cannot submit yet. A listing in that state is told to regenerate and commit its anchor first, rather than promised a one-line fix that would refuse.

What this proves — and what it does not

Proves: a holder of a key this repository commits as an approved signer asserts this result at commit X. That is attributable, auditable, and re-checkable by anyone: the directory publishes the local record and its signature beside the listing, and the command above is the whole verification.

Does not prove: that the scan ran on a clean machine, that the tools were the ones claimed, or that the working tree matched the commit. Nobody can check a workstation. This is weaker than the authenticated lane, which proves the repository's own CI produced the record under an identity GitHub's OIDC issuer burned in. It never earns the ✓ verified mark.

The scoring rule

A repository's grade takes account of its GitHub Actions-emitted results and its local scan-emitted results. These are not mutually exclusive lanes, and neither one is a second-class citizen. For each control the directory collects a verdict from every evidence source it holds. There are three of them: the newest action-lane record whose signature verified, the newest local-lane record whose signature verified, and the external record the directory produced itself. Then:

1. Two or more sources give different countable verdicts (pass, fail or gap) → the control scores gap. It also carries a contradiction flag naming each source and the verdict it gave. The flag appears on the record, on the listing row and on the detail page. Scoring a disagreement down while saying nothing about it would hide the most interesting fact the directory holds about that repository.
2. Exactly one distinct countable verdict across sources → that verdict, whichever lane produced it. A local pass counts; so does a local fail.
3. No countable verdict → unverified or info, outside every denominator, exactly as before.

A contradiction therefore costs the repository — a gap sits in the denominator without passing. That is deliberate. Erring on the side of caution removes any incentive to submit a flattering local scan: the flattering answer never wins, it only ever converts a row into a gap.

…and where a lane could not have looked, its verdict is not evidence

The same principle runs backwards. A repository-observable lane cannot observe class C: the development environment is not in the checkout. So a class-C verdict carried by an action-lane or external record is dropped before anything is counted, whatever it says. It is not countable evidence. It does not satisfy the independence requirement above for other rows. And it cannot contradict a genuine local verdict into a gap. A verdict a source could not have made is not a verdict.

The record's own score is the submitter's, not ours

A local record is a complete, self-describing ScanRecord, so it carries a top-level score block: a grade, a percentage, a coverage figure. Those were computed on the maintainer's machine, over the controls that machine had in scope. This directory republishes the record byte for byte. The signature covers those exact bytes, and rewriting them would destroy the only thing that makes the record evidence.

So two scores can be reached from one listing, and only one of them is ours. The grade on the listing and on the detail page is the directory's, computed here from every evidence source under the published methodology. The number inside scan-record.local.json is the submitter's self-report. Every listing carrying a local record says which is which, in both places, and names both numbers. Quoting the embedded grade as a directory grade should take work, not inattention.

Where someone else could have checked, we require that someone else

One requirement makes that union safe, and it is not "local counts less".

Classes A, A′ and B are by definition observable from a repository scan — a committed artifact, a committed workflow, a live GitHub setting. For those rows a maintainer's self-report alone is not countable. With no independent source the control stays unverified, outside the denominator. It becomes countable the moment a CI or external record exists to agree or disagree with it.

Class C is by definition not independently observable — it lives on the workstation and nowhere else. There the maintainer's signed word is the best evidence that can exist, and it counts on its own.

The practical consequence, and the reason the requirement exists: a repository whose only evidence is a local record publishes with its class-C rows scored and everything else unverified. Its evidence coverage is low, so it reads NA — insufficient evidence, never A+. A local record lifts a real score only alongside a scan somebody else could run. The scope such a listing is measured against is the directory's control set, not the record's: a record cannot shrink its own denominator by declining to mention a control.

Gates

A local submission passes through everything an authenticated one does. The record must be a valid v1 ScanRecord for the repository the request names. The signature must verify against the repository's own committed allowed_signers. And a maintainer must still apply the publish label. The authenticated lane's auto-publish gate is not available to it. A listing that used the local lane says so, on the listing and on the detail page. It names the controls the local record actually resolved. It says so again when that record describes a different commit than the repository scan it sits beside. A local record has no expiry. A stale one quietly filling holes in a much newer scan is something a reader has to be told about.

Coverage below 75% still reads provisional after a local scan if the remaining holes are elsewhere. Where that is the case the listing says so rather than promising a fix the command cannot deliver.

Changelog

• v1 — initial methodology: diff-based init reclassification, five evidence classes, and an academic grade scale with A+ reserved for exactly 100%.
• v1, 2026-09 — authenticated records are signed and verified against the producing workflow's identity. Scoring is unchanged: provenance is displayed, not scored.
• v1, 2026-09 — the local lane. A maintainer-signed workstation record, verified against the repository's own committed allowed_signers, joins the action and external records as an evidence source. Verdicts are merged per control. Sources that disagree score a gap with a named contradiction. A local assertion about a control a repository scan could observe is not counted until an independent record agrees with it. Class rules, the formula and the grade scale are unchanged.
• v2, 2026-09 — three controls join the registry: binary-artifacts, webhooks and dependency-pinning. All three are on by default, so every score has more checks in it. The sbom and vuln-scan gates now run their tools instead of checking that a tool exists. A scanner that ran and could not verify is recorded as unverified; only an absent tool lets the committed artifacts stand in. A record made under v1 says so until its next scan.