First public release
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,12 @@
|
|||||||
|
root = true
|
||||||
|
|
||||||
|
[*]
|
||||||
|
charset = utf-8
|
||||||
|
end_of_line = lf
|
||||||
|
insert_final_newline = true
|
||||||
|
trim_trailing_whitespace = true
|
||||||
|
indent_style = space
|
||||||
|
indent_size = 2
|
||||||
|
|
||||||
|
[*.md]
|
||||||
|
trim_trailing_whitespace = false
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
.npmrc
|
||||||
|
dist-cjs/
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
stages: [test]
|
||||||
|
|
||||||
|
test:
|
||||||
|
stage: test
|
||||||
|
image: node:22-alpine
|
||||||
|
script: [npm ci, npm run build, npm test]
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.2.0 - 2026-09-28
|
||||||
|
|
||||||
|
First public release under MIT.
|
||||||
|
|
||||||
|
- `scoreDimension()`, `scoreProduct()` and `scoreSuite()` apply a
|
||||||
|
five-dimension readiness rubric with hard caps and six bands.
|
||||||
|
- The lowest open cap on a dimension wins; an unknown cap id throws.
|
||||||
|
- A score above 70 with no evidence is held at 70 and reported as NOT
|
||||||
|
ASSESSED.
|
||||||
|
- The overall score is the weakest dimension; the average is reported
|
||||||
|
alongside and labelled as such.
|
||||||
|
- Suite-wide caps apply before the tier-weighted average.
|
||||||
|
- A cap can cite the evidence that establishes it; an uncited cap is still
|
||||||
|
applied and the report says so.
|
||||||
|
- `productReport()`, `suiteReport()`, `formatProduct()` and `formatSuite()`
|
||||||
|
produce an evidence-record report.
|
||||||
|
- Worked example in `examples/cap-by-worst-defect.ts`.
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Contributing
|
||||||
|
|
||||||
|
Issues and merge requests are welcome at
|
||||||
|
https://gitlab.com/extant2000/sustain-score.
|
||||||
|
|
||||||
|
## Ground rules
|
||||||
|
|
||||||
|
- A test that cannot fail proves nothing. If you fix a bug, add a test and
|
||||||
|
check that it fails against the unfixed code before you submit.
|
||||||
|
- Measure, don't assume. Two modules with the same line count can still be
|
||||||
|
different programs.
|
||||||
|
- Keep dependencies minimal. Every new runtime dependency needs a reason.
|
||||||
|
- Explain why in comments, not what. The what is already in the code.
|
||||||
|
- A breaking change needs a major version bump and a note in CHANGELOG.md.
|
||||||
|
|
||||||
|
## Before you open a merge request
|
||||||
|
|
||||||
|
Run both of these and make sure they pass:
|
||||||
|
|
||||||
|
npm run build
|
||||||
|
npm test
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Extant 2000 LLC
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# SustainScore
|
||||||
|
|
||||||
|
A production-readiness score that separates demo-ware from software that is
|
||||||
|
ready to hand over, with the rubric's caps enforced in code.
|
||||||
|
|
||||||
|
SustainScore encodes a five-dimension readiness rubric (security, bugs,
|
||||||
|
compliance, consistency, usability) with six bands from Prototype to Mature
|
||||||
|
and a table of hard caps. You supply the scorer's raw judgement per dimension,
|
||||||
|
the defects that are currently open, and the evidence. It returns scores that
|
||||||
|
apply the rubric's anti-inflation rules mechanically.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
A rubric kept as prose gets applied charitably. A scorer can write down, in
|
||||||
|
the same row, that an open defect caps Security at 45 and still record 70.
|
||||||
|
Nothing in a document stops that. Encoding the rubric does.
|
||||||
|
|
||||||
|
The worked example in `examples/cap-by-worst-defect.ts` scores a fictional
|
||||||
|
product whose security review notes an exposed API key that was never
|
||||||
|
rotated. Run it with `npx tsx examples/cap-by-worst-defect.ts`:
|
||||||
|
|
||||||
|
```
|
||||||
|
Hand-scored vs rubric-enforced
|
||||||
|
|
||||||
|
security raw 70 -> 45 Beta <- CAPPED by sec.leaked-secret (45)
|
||||||
|
bugs raw 68 -> 68 Launch-capable, with debt
|
||||||
|
compliance raw 73 -> 73 Launch-capable, with debt
|
||||||
|
consistency raw 68 -> 68 Launch-capable, with debt
|
||||||
|
usability raw 68 -> 68 Launch-capable, with debt
|
||||||
|
|
||||||
|
hand-scored overall : ~69
|
||||||
|
enforced overall : 45 (Beta)
|
||||||
|
weakest : security
|
||||||
|
average : 64.4 (reported, never the headline)
|
||||||
|
delta : -24
|
||||||
|
```
|
||||||
|
|
||||||
|
A 24-point overstatement and a band change, from one cap a human reader
|
||||||
|
skipped.
|
||||||
|
|
||||||
|
## What it enforces
|
||||||
|
|
||||||
|
| Rule | Enforcement |
|
||||||
|
|---|---|
|
||||||
|
| Cap by the worst defect | The lowest open cap on a dimension wins. One serious hole is not averaged away by ten strengths. |
|
||||||
|
| Evidence required above 70 | A score above 70 with no cited evidence is held at 70 and reported as NOT ASSESSED, not as a pass. Unverified is treated as absent. |
|
||||||
|
| Overall is the lowest dimension | `overall` is the minimum. The average is reported alongside it and labelled so it cannot be read as the headline. |
|
||||||
|
| Suite caps apply before averaging | A shared platform flaw hits every product. Averaging first would let healthy products dilute it. |
|
||||||
|
|
||||||
|
An unknown cap id throws. Dropping a mistyped cap without a word would
|
||||||
|
inflate the score, which is the exact failure this library prevents. A cap
|
||||||
|
assigned to the wrong dimension also throws, and so does a product with a
|
||||||
|
missing dimension.
|
||||||
|
|
||||||
|
Caps can cite what establishes them: `openCaps` takes a bare id or
|
||||||
|
`{ id, evidence }`. A cap that cites nothing is still applied, and the report
|
||||||
|
says it cited nothing. An unproven cap is a smaller error than an unapplied
|
||||||
|
one.
|
||||||
|
|
||||||
|
The full cap table is exported as `CAP_RULES`, the bands as `BANDS`.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { scoreProduct, scoreSuite, formatProduct } from '@extant2000/sustain-score'
|
||||||
|
|
||||||
|
const result = scoreProduct({
|
||||||
|
product: 'example-app',
|
||||||
|
tier: 'A',
|
||||||
|
dimensions: {
|
||||||
|
security: { raw: 70, openCaps: ['sec.leaked-secret'], evidence: [{ ref: 'docs/security-review.md:14' }] },
|
||||||
|
bugs: { raw: 68, evidence: [{ ref: 'test/api.test.ts' }] },
|
||||||
|
compliance: { raw: 73, evidence: [{ ref: 'docs/compliance-checklist.md' }] },
|
||||||
|
consistency: { raw: 68, evidence: [{ ref: 'shared/ui/' }] },
|
||||||
|
usability: { raw: 68, evidence: [{ ref: 'reports/accessibility-scan.json' }] },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
result.overall // 45, the lowest dimension
|
||||||
|
result.weakest // 'security'
|
||||||
|
result.band.name // 'Beta'
|
||||||
|
|
||||||
|
console.log(formatProduct(result))
|
||||||
|
```
|
||||||
|
|
||||||
|
`scoreSuite(products, { suiteCaps })` rolls several products up with tier
|
||||||
|
weighting (A = 2, B = 1.5, C = 1). `productReport()` and `suiteReport()`
|
||||||
|
return a `CapabilityReport` from `@extant2000/evidence-record`, and
|
||||||
|
`formatProduct()` and `formatSuite()` render one. Evidence can be the
|
||||||
|
evidence-record shape or a plain `{ ref, note }`, where a `path:line` ref
|
||||||
|
becomes a file citation.
|
||||||
|
|
||||||
|
To rank several scored products by what to fix next, see
|
||||||
|
`@extant2000/portfolio-advisor`.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```
|
||||||
|
npm install @extant2000/sustain-score
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT. Copyright (c) 2026 Extant 2000 LLC. See [LICENSE](LICENSE).
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
# Security policy
|
||||||
|
|
||||||
|
## Reporting a vulnerability
|
||||||
|
|
||||||
|
Please report vulnerabilities privately by email to security@extant2000.com.
|
||||||
|
Do not open a public issue or merge request for a security problem.
|
||||||
|
|
||||||
|
Include the affected version, a description of the issue, and steps or a test
|
||||||
|
that reproduce it if you have them.
|
||||||
|
|
||||||
|
We aim to acknowledge every report within 5 business days.
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
/**
|
||||||
|
* Worked example: cap by the worst defect.
|
||||||
|
*
|
||||||
|
* A hand-scored assessment of a fictional product. The scorer rated Security
|
||||||
|
* at 70 and noted, in the same row, an open defect that the rubric caps at 45
|
||||||
|
* (a secret that was exposed and never rotated). A human reading prose can be
|
||||||
|
* charitable and skip the cap. The library cannot: the lowest open cap wins,
|
||||||
|
* and the product's overall score is its weakest dimension, not the average.
|
||||||
|
*
|
||||||
|
* Run: npx tsx examples/cap-by-worst-defect.ts
|
||||||
|
*/
|
||||||
|
import { scoreProduct, type DimensionAssessment, type DimensionKey } from '../src/index.js'
|
||||||
|
|
||||||
|
/** The overall score a hand-scored summary might report: roughly the average. */
|
||||||
|
const HAND_SCORED_OVERALL = 69
|
||||||
|
|
||||||
|
const dimensions: Record<DimensionKey, DimensionAssessment> = {
|
||||||
|
security: {
|
||||||
|
raw: 70,
|
||||||
|
// The scorer named this cap and still wrote 70.
|
||||||
|
openCaps: [{
|
||||||
|
id: 'sec.leaked-secret',
|
||||||
|
evidence: [{ ref: 'docs/security-review.md:14', note: 'exposed API key not yet rotated' }],
|
||||||
|
}],
|
||||||
|
evidence: [{ ref: 'docs/security-review.md' }],
|
||||||
|
bindingConstraint: 'exposed API key not yet rotated',
|
||||||
|
nextAction: 'rotate the key and redeploy',
|
||||||
|
},
|
||||||
|
bugs: {
|
||||||
|
raw: 68,
|
||||||
|
evidence: [{ ref: 'test/api.test.ts' }],
|
||||||
|
bindingConstraint: 'API handlers lack behaviour tests',
|
||||||
|
},
|
||||||
|
compliance: {
|
||||||
|
raw: 73,
|
||||||
|
evidence: [{ ref: 'docs/compliance-checklist.md' }],
|
||||||
|
},
|
||||||
|
consistency: {
|
||||||
|
raw: 68,
|
||||||
|
evidence: [{ ref: 'shared/ui/' }],
|
||||||
|
},
|
||||||
|
usability: {
|
||||||
|
raw: 68,
|
||||||
|
evidence: [{ ref: 'reports/accessibility-scan.json' }],
|
||||||
|
bindingConstraint: 'accessibility findings not yet triaged',
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
const result = scoreProduct({ product: 'example-app', tier: 'A', dimensions })
|
||||||
|
|
||||||
|
console.log('Hand-scored vs rubric-enforced\n')
|
||||||
|
for (const d of result.dimensions) {
|
||||||
|
const flag = d.boundBy ? ` <- CAPPED by ${d.boundBy.id} (${d.boundBy.max})` : ''
|
||||||
|
console.log(
|
||||||
|
` ${d.dimension.padEnd(12)} raw ${String(d.raw).padStart(3)} -> ${String(d.score).padStart(3)} ${d.band.name}${flag}`,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
console.log(`\n hand-scored overall : ~${HAND_SCORED_OVERALL}`)
|
||||||
|
console.log(` enforced overall : ${result.overall} (${result.band.name})`)
|
||||||
|
console.log(` weakest : ${result.weakest}`)
|
||||||
|
console.log(` average : ${result.average} (reported, never the headline)`)
|
||||||
|
console.log(` delta : ${result.overall - HAND_SCORED_OVERALL}`)
|
||||||
|
|
||||||
|
if (result.overall < HAND_SCORED_OVERALL) {
|
||||||
|
console.log('\n The hand-scored number is higher than the rubric permits.')
|
||||||
|
}
|
||||||
Generated
+1504
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,44 @@
|
|||||||
|
{
|
||||||
|
"name": "@extant2000/sustain-score",
|
||||||
|
"version": "0.2.0",
|
||||||
|
"private": false,
|
||||||
|
"description": "Maturity score separating demo-ware from transition-ready.",
|
||||||
|
"license": "MIT",
|
||||||
|
"type": "module",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"files": [
|
||||||
|
"dist",
|
||||||
|
"src",
|
||||||
|
"examples",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
],
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "https://gitlab.com/extant2000/sustain-score.git"
|
||||||
|
},
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"import": "./dist/index.js"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc",
|
||||||
|
"test": "vitest run",
|
||||||
|
"prepublishOnly": "npm run build"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"typescript": "^5.6.0",
|
||||||
|
"vitest": "^2.1.0"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@extant2000/evidence-record": "^0.1.2"
|
||||||
|
},
|
||||||
|
"author": "Extant 2000 LLC",
|
||||||
|
"homepage": "https://gitlab.com/extant2000/sustain-score",
|
||||||
|
"bugs": {
|
||||||
|
"url": "https://gitlab.com/extant2000/sustain-score/-/issues"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
export * from './rubric.js'
|
||||||
|
export * from './score.js'
|
||||||
|
export * from './report.js'
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
import {
|
||||||
|
type CapabilityReport,
|
||||||
|
type Finding,
|
||||||
|
type FormatOptions,
|
||||||
|
evidence as ev,
|
||||||
|
formatReport as renderReport,
|
||||||
|
} from '@extant2000/evidence-record'
|
||||||
|
import type { DimensionScore, ProductScore, SuiteScore } from './score.js'
|
||||||
|
import { EVIDENCE_THRESHOLD } from './rubric.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One finding per dimension, carrying what set the number.
|
||||||
|
*
|
||||||
|
* The previous output said `CAPPED sec.leaked-secret` and stopped there. A
|
||||||
|
* reader could not tell whether the cap was still warranted, which is how a
|
||||||
|
* cap ends up recorded but not applied. A cap now cites what establishes it,
|
||||||
|
* and a cap that cites nothing says so.
|
||||||
|
*/
|
||||||
|
function findingFor(product: string, d: DimensionScore): Finding {
|
||||||
|
const id = `${product}.${d.dimension}`
|
||||||
|
|
||||||
|
if (d.boundBy) {
|
||||||
|
return {
|
||||||
|
id,
|
||||||
|
summary: `${d.dimension} capped at ${d.score} (assessed ${d.raw}) — ${d.boundBy.description}`,
|
||||||
|
determination: 'fail',
|
||||||
|
severity: d.score < 50 ? 'critical' : 'high',
|
||||||
|
detail: d.nextAction ?? d.bindingConstraint,
|
||||||
|
evidence: d.boundBy.evidence.length > 0
|
||||||
|
? d.boundBy.evidence
|
||||||
|
// Loud rather than silent: an uncited cap is still applied, because
|
||||||
|
// the cap being wrong is a better failure than the cap being skipped.
|
||||||
|
: [ev.document(`Cap rule ${d.boundBy.id}`, undefined, undefined,
|
||||||
|
'No evidence was recorded for this cap. It is applied anyway — an unproven cap is a smaller error than an unapplied one.')],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (d.evidenceWithheld) {
|
||||||
|
return {
|
||||||
|
id,
|
||||||
|
summary: `${d.dimension} held at ${EVIDENCE_THRESHOLD} (assessed ${d.raw}) — no evidence was cited`,
|
||||||
|
determination: 'not-assessed',
|
||||||
|
severity: 'high',
|
||||||
|
detail: `A score above ${EVIDENCE_THRESHOLD} requires evidence. Unverified is treated as absent, so the assessed value was not granted.`,
|
||||||
|
evidence: [],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
id,
|
||||||
|
summary: `${d.dimension} scored ${d.score} (${d.band.name})`,
|
||||||
|
determination: 'pass',
|
||||||
|
severity: 'info',
|
||||||
|
detail: d.bindingConstraint,
|
||||||
|
evidence: d.evidence.length > 0
|
||||||
|
? d.evidence
|
||||||
|
: [ev.document(`${d.dimension} assessment`, undefined, undefined,
|
||||||
|
`At or below ${EVIDENCE_THRESHOLD}, evidence is not required by the rubric.`)],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Convert one product's score into the portfolio-standard report shape. */
|
||||||
|
export function productReport(p: ProductScore): CapabilityReport {
|
||||||
|
return {
|
||||||
|
capability: 'SustainScore',
|
||||||
|
scope: `${p.product} (${p.tier}) — ${p.dimensions.length} dimensions`,
|
||||||
|
examined: p.dimensions.length,
|
||||||
|
findings: p.dimensions.map(d => findingFor(p.product, d)),
|
||||||
|
notes: [
|
||||||
|
`overall ${p.overall} (${p.band.name}), set by the weakest dimension: ${p.weakest}`,
|
||||||
|
// Printed second and labelled, so it cannot be mistaken for the headline.
|
||||||
|
`average ${p.average} — reported because the rubric asks for it, never as the headline`,
|
||||||
|
],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Convert a suite roll-up into the portfolio-standard report shape. */
|
||||||
|
export function suiteReport(s: SuiteScore): CapabilityReport {
|
||||||
|
return {
|
||||||
|
capability: 'SustainScore',
|
||||||
|
scope: `${s.products.length} products, weighted by tier`,
|
||||||
|
examined: s.products.reduce((n, p) => n + p.dimensions.length, 0),
|
||||||
|
findings: s.products.flatMap(p => p.dimensions.map(d => findingFor(p.product, d))),
|
||||||
|
notes: [
|
||||||
|
`suite overall ${s.overall} (${s.band.name}), set by the weakest dimension: ${s.weakest}`,
|
||||||
|
`average ${s.average} — suite-wide caps are applied BEFORE averaging, so healthy products cannot dilute a shared flaw`,
|
||||||
|
],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatProduct(p: ProductScore, opts: FormatOptions = {}): string {
|
||||||
|
return renderReport(productReport(p), opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatSuite(s: SuiteScore, opts: FormatOptions = {}): string {
|
||||||
|
return renderReport(suiteReport(s), opts)
|
||||||
|
}
|
||||||
+118
@@ -0,0 +1,118 @@
|
|||||||
|
/**
|
||||||
|
* Production-Readiness Rubric v1.0 — encoded.
|
||||||
|
*
|
||||||
|
* A rubric kept as prose and applied by hand gets applied charitably. A scorer
|
||||||
|
* can write down a defect that caps a dimension at 45 and still record 70 in
|
||||||
|
* the same row. Encoding the rubric is not a formatting exercise: a charitable
|
||||||
|
* scorer is the failure mode this module removes.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type BandName =
|
||||||
|
| 'Prototype' | 'Alpha' | 'Beta'
|
||||||
|
| 'Launch-capable, with debt' | 'Production' | 'Mature'
|
||||||
|
|
||||||
|
export interface Band {
|
||||||
|
name: BandName
|
||||||
|
min: number
|
||||||
|
max: number
|
||||||
|
meaning: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Identical for every dimension. Ordered low → high. */
|
||||||
|
export const BANDS: readonly Band[] = Object.freeze([
|
||||||
|
{ name: 'Prototype', min: 0, max: 19,
|
||||||
|
meaning: 'Unsafe or broken in ordinary use. Would harm a user or the business if strangers used it today.' },
|
||||||
|
{ name: 'Alpha', min: 20, max: 39,
|
||||||
|
meaning: 'Works on the happy path. Known defects that actively harm users or expose the company. Not sellable.' },
|
||||||
|
{ name: 'Beta', min: 40, max: 59,
|
||||||
|
meaning: 'Genuinely usable, but ≥1 gap that blocks charging money or inviting scrutiny.' },
|
||||||
|
{ name: 'Launch-capable, with debt', min: 60, max: 74,
|
||||||
|
meaning: 'No blocking defect. Real, catalogued debt a competitor or auditor would notice.' },
|
||||||
|
{ name: 'Production', min: 75, max: 89,
|
||||||
|
meaning: 'No known blockers. Controls exist AND are verified working. Failures are monitored and recoverable.' },
|
||||||
|
{ name: 'Mature', min: 90, max: 100,
|
||||||
|
meaning: 'Independently verified (external audit, pen test, or a real incident survived). Evidence, not assertion.' },
|
||||||
|
])
|
||||||
|
|
||||||
|
export function bandFor(score: number): Band {
|
||||||
|
const s = Math.max(0, Math.min(100, Math.round(score)))
|
||||||
|
// Walk high → low so the first match is the highest band the score reaches.
|
||||||
|
for (let i = BANDS.length - 1; i >= 0; i--) {
|
||||||
|
const b = BANDS[i]!
|
||||||
|
if (s >= b.min) return b
|
||||||
|
}
|
||||||
|
return BANDS[0]!
|
||||||
|
}
|
||||||
|
|
||||||
|
export type DimensionKey =
|
||||||
|
| 'security' | 'bugs' | 'compliance' | 'consistency' | 'usability'
|
||||||
|
|
||||||
|
export const DIMENSIONS: readonly DimensionKey[] = Object.freeze([
|
||||||
|
'security', 'bugs', 'compliance', 'consistency', 'usability',
|
||||||
|
])
|
||||||
|
|
||||||
|
export interface CapRule {
|
||||||
|
/** Stable id so a scorecard can reference which cap bound a score. */
|
||||||
|
id: string
|
||||||
|
dimension: DimensionKey
|
||||||
|
/** Ceiling this defect imposes on its dimension. */
|
||||||
|
max: number
|
||||||
|
description: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every hard cap in rubric v1.0, verbatim. */
|
||||||
|
export const CAP_RULES: readonly CapRule[] = Object.freeze([
|
||||||
|
// Dimension 1 — SECURITY
|
||||||
|
{ id: 'sec.cross-tenant', dimension: 'security', max: 30,
|
||||||
|
description: 'Any live cross-tenant data exposure' },
|
||||||
|
{ id: 'sec.leaked-secret', dimension: 'security', max: 45,
|
||||||
|
description: 'Any secret in the repo, in logs, or known-leaked and unrotated' },
|
||||||
|
{ id: 'sec.no-restore', dimension: 'security', max: 70,
|
||||||
|
description: 'No verified backup restore' },
|
||||||
|
{ id: 'sec.no-dep-monitoring', dimension: 'security', max: 75,
|
||||||
|
description: 'No dependency-vulnerability monitoring' },
|
||||||
|
// Dimension 2 — BUGS
|
||||||
|
{ id: 'bugs.data-loss', dimension: 'bugs', max: 35,
|
||||||
|
description: 'Any known defect causing silent data loss or corruption' },
|
||||||
|
{ id: 'bugs.takes-money-no-deliver', dimension: 'bugs', max: 40,
|
||||||
|
description: "Any user-facing flow that takes money and doesn't deliver" },
|
||||||
|
{ id: 'bugs.no-error-monitoring', dimension: 'bugs', max: 60,
|
||||||
|
description: 'No error monitoring in production' },
|
||||||
|
{ id: 'bugs.smoke-only', dimension: 'bugs', max: 70,
|
||||||
|
description: 'Smoke-tests-only (tests assert files exist, not behavior)' },
|
||||||
|
// Dimension 3 — COMPLIANCE
|
||||||
|
{ id: 'comp.false-claim', dimension: 'compliance', max: 50,
|
||||||
|
description: 'Any live false or unsubstantiated public claim' },
|
||||||
|
{ id: 'comp.missing-registration', dimension: 'compliance', max: 60,
|
||||||
|
description: 'Any missing registration that is legally required NOW' },
|
||||||
|
{ id: 'comp.legal-contradicts-code', dimension: 'compliance', max: 65,
|
||||||
|
description: 'Legal pages that contradict code behavior' },
|
||||||
|
// Dimension 4 — CONSISTENCY
|
||||||
|
{ id: 'cons.per-repo-fix', dimension: 'consistency', max: 65,
|
||||||
|
description: 'Same critical bug class needing a separate fix per repo' },
|
||||||
|
{ id: 'cons.docs-contradict', dimension: 'consistency', max: 70,
|
||||||
|
description: 'Docs/CLAUDE.md materially contradicting the code' },
|
||||||
|
// Dimension 5 — USABILITY
|
||||||
|
{ id: 'use.keyboard-inoperable', dimension: 'usability', max: 55,
|
||||||
|
description: 'Keyboard-inoperable or unlabeled primary flow' },
|
||||||
|
{ id: 'use.contrast', dimension: 'usability', max: 65,
|
||||||
|
description: 'Body-copy contrast below AA' },
|
||||||
|
{ id: 'use.no-empty-error-states', dimension: 'usability', max: 70,
|
||||||
|
description: 'No empty/error states on primary screens' },
|
||||||
|
])
|
||||||
|
|
||||||
|
const CAPS_BY_ID = new Map(CAP_RULES.map(c => [c.id, c]))
|
||||||
|
|
||||||
|
export function capRule(id: string): CapRule | undefined {
|
||||||
|
return CAPS_BY_ID.get(id)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Score at or above which evidence is mandatory (anti-inflation rule 5:
|
||||||
|
* "Evidence required above 70").
|
||||||
|
*/
|
||||||
|
export const EVIDENCE_THRESHOLD = 70
|
||||||
|
|
||||||
|
/** Revenue/exposure weights for the suite average. */
|
||||||
|
export const TIER_WEIGHTS = Object.freeze({ A: 2, B: 1.5, C: 1 })
|
||||||
|
export type Tier = keyof typeof TIER_WEIGHTS
|
||||||
+267
@@ -0,0 +1,267 @@
|
|||||||
|
import {
|
||||||
|
DIMENSIONS,
|
||||||
|
EVIDENCE_THRESHOLD,
|
||||||
|
TIER_WEIGHTS,
|
||||||
|
bandFor,
|
||||||
|
capRule,
|
||||||
|
type Band,
|
||||||
|
type DimensionKey,
|
||||||
|
type Tier,
|
||||||
|
} from './rubric.js'
|
||||||
|
|
||||||
|
import { type Evidence as SourcedEvidence, evidence as ev } from '@extant2000/evidence-record'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The original shape: a free-text reference.
|
||||||
|
*
|
||||||
|
* @deprecated Pass `@extant2000/evidence-record` evidence instead — `ref` is a
|
||||||
|
* string a reader has to trust, and this library's own rubric penalises
|
||||||
|
* unverified claims under "Unverified = absent". Still accepted, and
|
||||||
|
* converted: `path:42` becomes a file citation, anything else a document one.
|
||||||
|
*/
|
||||||
|
export interface Evidence {
|
||||||
|
/** file:line, a passing test name, a live check, or a production query. */
|
||||||
|
ref: string
|
||||||
|
note?: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export type EvidenceLike = Evidence | SourcedEvidence
|
||||||
|
|
||||||
|
/** Normalise either shape into a citation the shared renderer can print. */
|
||||||
|
export function toSourcedEvidence(e: EvidenceLike): SourcedEvidence {
|
||||||
|
if ('source' in e) return e
|
||||||
|
const m = /^(.+):(\d+)$/.exec(e.ref)
|
||||||
|
return m
|
||||||
|
? ev.file(m[1]!, Number(m[2]), undefined, e.note)
|
||||||
|
: ev.document(e.ref, undefined, undefined, e.note)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A cap that is currently open, and what establishes that.
|
||||||
|
*
|
||||||
|
* The bare-string form is what the first version accepted, and it is exactly
|
||||||
|
* the gap this release closes: the output said `CAPPED sec.leaked-secret`
|
||||||
|
* without citing anything, so a reader had no way to check whether the cap
|
||||||
|
* was still warranted, which is how a cap goes unapplied in the first place.
|
||||||
|
*/
|
||||||
|
export type CapClaim = string | { id: string, evidence?: EvidenceLike[] }
|
||||||
|
|
||||||
|
export function capId(c: CapClaim): string {
|
||||||
|
return typeof c === 'string' ? c : c.id
|
||||||
|
}
|
||||||
|
|
||||||
|
function capEvidence(c: CapClaim): SourcedEvidence[] {
|
||||||
|
return typeof c === 'string' ? [] : (c.evidence ?? []).map(toSourcedEvidence)
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DimensionAssessment {
|
||||||
|
/** The scorer's judgement BEFORE caps are applied. */
|
||||||
|
raw: number
|
||||||
|
/** Cap rules whose defect is currently open, ideally with what proves it. */
|
||||||
|
openCaps?: CapClaim[]
|
||||||
|
/** Required to hold a final score above EVIDENCE_THRESHOLD. */
|
||||||
|
evidence?: EvidenceLike[]
|
||||||
|
/** What is holding the score down. Free text, carried through. */
|
||||||
|
bindingConstraint?: string
|
||||||
|
/** Single highest-leverage action to raise it. */
|
||||||
|
nextAction?: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DimensionScore {
|
||||||
|
dimension: DimensionKey
|
||||||
|
raw: number
|
||||||
|
score: number
|
||||||
|
band: Band
|
||||||
|
/**
|
||||||
|
* The cap that actually bound the score, if any — with whatever established
|
||||||
|
* it, so "CAPPED sec.leaked-secret" is checkable rather than asserted.
|
||||||
|
*/
|
||||||
|
boundBy?: { id: string, max: number, description: string, evidence: SourcedEvidence[] }
|
||||||
|
/** Citations supporting the raw judgement, normalised. */
|
||||||
|
evidence: SourcedEvidence[]
|
||||||
|
/** True when the score was reduced to EVIDENCE_THRESHOLD for lack of evidence. */
|
||||||
|
evidenceWithheld: boolean
|
||||||
|
bindingConstraint?: string
|
||||||
|
nextAction?: string
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Score one dimension, applying the anti-inflation rules mechanically.
|
||||||
|
*
|
||||||
|
* Rule 3 — cap by the WORST defect: the lowest open cap wins. "One serious hole
|
||||||
|
* is not averaged away by ten strengths."
|
||||||
|
*
|
||||||
|
* Rule 5 — evidence required above 70: a score that cannot cite evidence is
|
||||||
|
* reduced to the threshold rather than rejected, so an un-evidenced assessment
|
||||||
|
* still produces a usable (and honest) number.
|
||||||
|
*
|
||||||
|
* These two are the whole point. Applied by hand, rule 3 is the one that gets
|
||||||
|
* skipped: a scorer notes an open defect that caps Security at 45 and still
|
||||||
|
* records 70.
|
||||||
|
*/
|
||||||
|
export function scoreDimension(
|
||||||
|
dimension: DimensionKey,
|
||||||
|
input: DimensionAssessment,
|
||||||
|
): DimensionScore {
|
||||||
|
const raw = clamp(input.raw)
|
||||||
|
let score = raw
|
||||||
|
let boundBy: DimensionScore['boundBy']
|
||||||
|
|
||||||
|
for (const claim of input.openCaps ?? []) {
|
||||||
|
const id = capId(claim)
|
||||||
|
const rule = capRule(id)
|
||||||
|
if (!rule) throw new Error(`Unknown cap rule: ${id}`)
|
||||||
|
if (rule.dimension !== dimension) {
|
||||||
|
throw new Error(`Cap ${id} belongs to ${rule.dimension}, not ${dimension}`)
|
||||||
|
}
|
||||||
|
if (rule.max < score) {
|
||||||
|
score = rule.max
|
||||||
|
boundBy = {
|
||||||
|
id: rule.id,
|
||||||
|
max: rule.max,
|
||||||
|
description: rule.description,
|
||||||
|
evidence: capEvidence(claim),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const evidence = (input.evidence ?? []).map(toSourcedEvidence)
|
||||||
|
const hasEvidence = evidence.length > 0
|
||||||
|
let evidenceWithheld = false
|
||||||
|
if (score > EVIDENCE_THRESHOLD && !hasEvidence) {
|
||||||
|
score = EVIDENCE_THRESHOLD
|
||||||
|
evidenceWithheld = true
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
dimension,
|
||||||
|
raw,
|
||||||
|
score,
|
||||||
|
band: bandFor(score),
|
||||||
|
boundBy,
|
||||||
|
evidence,
|
||||||
|
evidenceWithheld,
|
||||||
|
bindingConstraint: input.bindingConstraint,
|
||||||
|
nextAction: input.nextAction,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProductAssessment {
|
||||||
|
product: string
|
||||||
|
tier: Tier
|
||||||
|
dimensions: Record<DimensionKey, DimensionAssessment>
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProductScore {
|
||||||
|
product: string
|
||||||
|
tier: Tier
|
||||||
|
weight: number
|
||||||
|
dimensions: DimensionScore[]
|
||||||
|
/** The headline number: the LOWEST dimension, not the average. */
|
||||||
|
overall: number
|
||||||
|
/** Reported alongside, never instead of, `overall`. */
|
||||||
|
average: number
|
||||||
|
band: Band
|
||||||
|
/** Which dimension set the overall score. */
|
||||||
|
weakest: DimensionKey
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Score one product.
|
||||||
|
*
|
||||||
|
* The overall score is the MINIMUM across dimensions — "a product is only as
|
||||||
|
* shippable as its weakest dimension." The average is reported too, because the
|
||||||
|
* rubric says to, but it is deliberately not the headline: averaging is how a
|
||||||
|
* single disqualifying hole disappears behind four healthy numbers.
|
||||||
|
*/
|
||||||
|
export function scoreProduct(input: ProductAssessment): ProductScore {
|
||||||
|
const dimensions = DIMENSIONS.map(d => {
|
||||||
|
const a = input.dimensions[d]
|
||||||
|
if (!a) throw new Error(`Missing assessment for dimension: ${d}`)
|
||||||
|
return scoreDimension(d, a)
|
||||||
|
})
|
||||||
|
|
||||||
|
let weakest = dimensions[0]!
|
||||||
|
for (const d of dimensions) if (d.score < weakest.score) weakest = d
|
||||||
|
|
||||||
|
const overall = weakest.score
|
||||||
|
const average = round(dimensions.reduce((s, d) => s + d.score, 0) / dimensions.length)
|
||||||
|
|
||||||
|
return {
|
||||||
|
product: input.product,
|
||||||
|
tier: input.tier,
|
||||||
|
weight: TIER_WEIGHTS[input.tier],
|
||||||
|
dimensions,
|
||||||
|
overall,
|
||||||
|
average,
|
||||||
|
band: bandFor(overall),
|
||||||
|
weakest: weakest.dimension,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SuiteOptions {
|
||||||
|
/**
|
||||||
|
* Cap ids that apply suite-wide (a shared platform flaw hits everything).
|
||||||
|
* Applied BEFORE averaging, per the rubric's scoring procedure — averaging
|
||||||
|
* first would let healthy products dilute a flaw that affects them all.
|
||||||
|
*/
|
||||||
|
suiteCaps?: CapClaim[]
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SuiteScore {
|
||||||
|
products: ProductScore[]
|
||||||
|
/** Weighted average per dimension, suite caps applied first. */
|
||||||
|
dimensions: Record<DimensionKey, number>
|
||||||
|
/** Lowest suite dimension score. The headline. */
|
||||||
|
overall: number
|
||||||
|
average: number
|
||||||
|
band: Band
|
||||||
|
weakest: DimensionKey
|
||||||
|
}
|
||||||
|
|
||||||
|
export function scoreSuite(
|
||||||
|
inputs: ProductAssessment[],
|
||||||
|
options: SuiteOptions = {},
|
||||||
|
): SuiteScore {
|
||||||
|
if (inputs.length === 0) throw new Error('scoreSuite requires at least one product')
|
||||||
|
const products = inputs.map(scoreProduct)
|
||||||
|
|
||||||
|
const dimensions = {} as Record<DimensionKey, number>
|
||||||
|
for (const d of DIMENSIONS) {
|
||||||
|
let totalWeight = 0
|
||||||
|
let acc = 0
|
||||||
|
for (const p of products) {
|
||||||
|
const ds = p.dimensions.find(x => x.dimension === d)!
|
||||||
|
// Suite-wide cap applied to each product's score BEFORE it enters the
|
||||||
|
// weighted mean.
|
||||||
|
let s = ds.score
|
||||||
|
for (const claim of options.suiteCaps ?? []) {
|
||||||
|
const id = capId(claim)
|
||||||
|
const rule = capRule(id)
|
||||||
|
if (rule && rule.dimension === d && rule.max < s) s = rule.max
|
||||||
|
}
|
||||||
|
acc += s * p.weight
|
||||||
|
totalWeight += p.weight
|
||||||
|
}
|
||||||
|
dimensions[d] = round(acc / totalWeight)
|
||||||
|
}
|
||||||
|
|
||||||
|
let weakest: DimensionKey = DIMENSIONS[0]!
|
||||||
|
for (const d of DIMENSIONS) if (dimensions[d] < dimensions[weakest]) weakest = d
|
||||||
|
|
||||||
|
const overall = dimensions[weakest]
|
||||||
|
const average = round(
|
||||||
|
DIMENSIONS.reduce((s, d) => s + dimensions[d], 0) / DIMENSIONS.length,
|
||||||
|
)
|
||||||
|
|
||||||
|
return { products, dimensions, overall, average, band: bandFor(overall), weakest }
|
||||||
|
}
|
||||||
|
|
||||||
|
function clamp(n: number): number {
|
||||||
|
if (!Number.isFinite(n)) throw new Error(`Score must be a finite number, got ${n}`)
|
||||||
|
return Math.max(0, Math.min(100, Math.round(n)))
|
||||||
|
}
|
||||||
|
|
||||||
|
function round(n: number): number {
|
||||||
|
return Math.round(n * 10) / 10
|
||||||
|
}
|
||||||
@@ -0,0 +1,289 @@
|
|||||||
|
import { describe, expect, it } from 'vitest'
|
||||||
|
import {
|
||||||
|
BANDS,
|
||||||
|
CAP_RULES,
|
||||||
|
DIMENSIONS,
|
||||||
|
bandFor,
|
||||||
|
scoreDimension,
|
||||||
|
scoreProduct,
|
||||||
|
scoreSuite,
|
||||||
|
type DimensionAssessment,
|
||||||
|
type DimensionKey,
|
||||||
|
type ProductAssessment,
|
||||||
|
formatProduct,
|
||||||
|
productReport,
|
||||||
|
} from '../src/index.js'
|
||||||
|
import { validateReport } from '@extant2000/evidence-record'
|
||||||
|
|
||||||
|
const ev = [{ ref: 'test/foo.test.ts:12' }]
|
||||||
|
|
||||||
|
/** A product scoring `n` on every dimension, with evidence. */
|
||||||
|
function flat(product: string, n: number, tier: 'A' | 'B' | 'C' = 'B'): ProductAssessment {
|
||||||
|
const dimensions = {} as Record<DimensionKey, DimensionAssessment>
|
||||||
|
for (const d of DIMENSIONS) dimensions[d] = { raw: n, evidence: ev }
|
||||||
|
return { product, tier, dimensions }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A healthy product with one dimension replaced, for isolating its output. */
|
||||||
|
function assessmentWith(dim: DimensionKey, a: DimensionAssessment): ProductAssessment {
|
||||||
|
const p = flat('app', 60)
|
||||||
|
p.dimensions[dim] = a
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('bandFor', () => {
|
||||||
|
it('maps the documented boundaries', () => {
|
||||||
|
expect(bandFor(0).name).toBe('Prototype')
|
||||||
|
expect(bandFor(19).name).toBe('Prototype')
|
||||||
|
expect(bandFor(20).name).toBe('Alpha')
|
||||||
|
expect(bandFor(40).name).toBe('Beta')
|
||||||
|
expect(bandFor(60).name).toBe('Launch-capable, with debt')
|
||||||
|
expect(bandFor(75).name).toBe('Production')
|
||||||
|
expect(bandFor(90).name).toBe('Mature')
|
||||||
|
expect(bandFor(100).name).toBe('Mature')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('covers 0-100 with no gaps', () => {
|
||||||
|
for (let s = 0; s <= 100; s++) expect(bandFor(s)).toBeDefined()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('has contiguous non-overlapping bands', () => {
|
||||||
|
for (let i = 1; i < BANDS.length; i++) {
|
||||||
|
expect(BANDS[i]!.min).toBe(BANDS[i - 1]!.max + 1)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('scoreDimension — anti-inflation rule 3 (cap by worst defect)', () => {
|
||||||
|
it('CATCHES THE CHARITABLE SCORE: an unrotated leaked secret caps Security at 45', () => {
|
||||||
|
// A hand scorer can note that a known-leaked, unrotated secret caps the
|
||||||
|
// dimension at 45 and still record ~70. Charity is not available here.
|
||||||
|
const r = scoreDimension('security', {
|
||||||
|
raw: 70,
|
||||||
|
openCaps: ['sec.leaked-secret'],
|
||||||
|
evidence: ev,
|
||||||
|
})
|
||||||
|
expect(r.raw).toBe(70)
|
||||||
|
expect(r.score).toBe(45)
|
||||||
|
expect(r.band.name).toBe('Beta')
|
||||||
|
expect(r.boundBy?.id).toBe('sec.leaked-secret')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('applies the LOWEST open cap, not the first or last', () => {
|
||||||
|
const r = scoreDimension('security', {
|
||||||
|
raw: 90,
|
||||||
|
openCaps: ['sec.no-dep-monitoring', 'sec.cross-tenant', 'sec.no-restore'],
|
||||||
|
evidence: ev,
|
||||||
|
})
|
||||||
|
expect(r.score).toBe(30)
|
||||||
|
expect(r.boundBy?.id).toBe('sec.cross-tenant')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leaves a score below the cap untouched', () => {
|
||||||
|
const r = scoreDimension('security', {
|
||||||
|
raw: 25, openCaps: ['sec.leaked-secret'], evidence: ev,
|
||||||
|
})
|
||||||
|
expect(r.score).toBe(25)
|
||||||
|
expect(r.boundBy).toBeUndefined()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects a cap belonging to another dimension', () => {
|
||||||
|
expect(() => scoreDimension('bugs', { raw: 80, openCaps: ['sec.leaked-secret'] }))
|
||||||
|
.toThrow(/belongs to security/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rejects an unknown cap id rather than ignoring it', () => {
|
||||||
|
// Silently ignoring a typo'd cap would inflate the score — the exact
|
||||||
|
// failure mode this library exists to prevent.
|
||||||
|
expect(() => scoreDimension('security', { raw: 80, openCaps: ['sec.tpyo'] }))
|
||||||
|
.toThrow(/Unknown cap rule/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('scoreDimension — anti-inflation rule 5 (evidence above 70)', () => {
|
||||||
|
it('withholds an un-evidenced score above the threshold', () => {
|
||||||
|
const r = scoreDimension('bugs', { raw: 85 })
|
||||||
|
expect(r.score).toBe(70)
|
||||||
|
expect(r.evidenceWithheld).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('allows an evidenced score above the threshold', () => {
|
||||||
|
const r = scoreDimension('bugs', { raw: 85, evidence: ev })
|
||||||
|
expect(r.score).toBe(85)
|
||||||
|
expect(r.evidenceWithheld).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not touch an un-evidenced score at or below the threshold', () => {
|
||||||
|
const r = scoreDimension('bugs', { raw: 70 })
|
||||||
|
expect(r.score).toBe(70)
|
||||||
|
expect(r.evidenceWithheld).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('an empty evidence array is not evidence', () => {
|
||||||
|
expect(scoreDimension('bugs', { raw: 85, evidence: [] }).score).toBe(70)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('scoreProduct', () => {
|
||||||
|
it('reports the LOWEST dimension as overall, not the average', () => {
|
||||||
|
const a = flat('app', 80)
|
||||||
|
a.dimensions.security = { raw: 40, evidence: ev }
|
||||||
|
const r = scoreProduct(a)
|
||||||
|
expect(r.overall).toBe(40)
|
||||||
|
expect(r.weakest).toBe('security')
|
||||||
|
expect(r.average).toBe(72) // (40+80*4)/5
|
||||||
|
expect(r.band.name).toBe('Beta')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a single capped dimension pins the whole product', () => {
|
||||||
|
// Four strong dimensions must not average away one disqualifying hole.
|
||||||
|
const a = flat('api', 88)
|
||||||
|
a.dimensions.security = { raw: 88, openCaps: ['sec.cross-tenant'], evidence: ev }
|
||||||
|
const r = scoreProduct(a)
|
||||||
|
expect(r.overall).toBe(30)
|
||||||
|
expect(r.band.name).toBe('Alpha')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('assigns the tier weight', () => {
|
||||||
|
expect(scoreProduct(flat('x', 80, 'A')).weight).toBe(2)
|
||||||
|
expect(scoreProduct(flat('x', 80, 'B')).weight).toBe(1.5)
|
||||||
|
expect(scoreProduct(flat('x', 80, 'C')).weight).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('throws on a missing dimension rather than scoring a partial product', () => {
|
||||||
|
const a = flat('x', 80)
|
||||||
|
delete (a.dimensions as Partial<Record<DimensionKey, DimensionAssessment>>).usability
|
||||||
|
expect(() => scoreProduct(a)).toThrow(/Missing assessment/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('carries the binding constraint and next action through', () => {
|
||||||
|
const a = flat('x', 80)
|
||||||
|
a.dimensions.bugs = {
|
||||||
|
raw: 68, evidence: ev,
|
||||||
|
bindingConstraint: 'server/api/** handlers untested',
|
||||||
|
nextAction: 'behaviour-asserting coverage',
|
||||||
|
}
|
||||||
|
const bugs = scoreProduct(a).dimensions.find(d => d.dimension === 'bugs')!
|
||||||
|
expect(bugs.bindingConstraint).toBe('server/api/** handlers untested')
|
||||||
|
expect(bugs.nextAction).toBe('behaviour-asserting coverage')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('scoreSuite', () => {
|
||||||
|
it('weights by tier', () => {
|
||||||
|
// A-tier (2) at 90, C-tier (1) at 60 → (90*2 + 60*1) / 3 = 80
|
||||||
|
const r = scoreSuite([flat('big', 90, 'A'), flat('small', 60, 'C')])
|
||||||
|
expect(r.dimensions.security).toBe(80)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('applies a suite-wide cap BEFORE averaging', () => {
|
||||||
|
// Averaging first would let healthy products dilute a flaw affecting all
|
||||||
|
// of them: (90+90)/2 = 90, capped after → 90. Capping first → 45.
|
||||||
|
const r = scoreSuite(
|
||||||
|
[flat('a', 90, 'A'), flat('b', 90, 'A')],
|
||||||
|
{ suiteCaps: ['sec.leaked-secret'] },
|
||||||
|
)
|
||||||
|
expect(r.dimensions.security).toBe(45)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('leads with the lowest suite dimension', () => {
|
||||||
|
const a = flat('a', 85)
|
||||||
|
a.dimensions.consistency = { raw: 63, evidence: ev }
|
||||||
|
const r = scoreSuite([a])
|
||||||
|
expect(r.overall).toBe(63)
|
||||||
|
expect(r.weakest).toBe('consistency')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('throws on an empty suite instead of returning a flattering zero', () => {
|
||||||
|
expect(() => scoreSuite([])).toThrow(/at least one product/)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('rubric integrity', () => {
|
||||||
|
it('every cap rule has a unique id', () => {
|
||||||
|
const ids = CAP_RULES.map(c => c.id)
|
||||||
|
expect(new Set(ids).size).toBe(ids.length)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('every cap belongs to a known dimension and sits in 0-100', () => {
|
||||||
|
for (const c of CAP_RULES) {
|
||||||
|
expect(DIMENSIONS).toContain(c.dimension)
|
||||||
|
expect(c.max).toBeGreaterThanOrEqual(0)
|
||||||
|
expect(c.max).toBeLessThanOrEqual(100)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('every dimension carries at least one cap', () => {
|
||||||
|
for (const d of DIMENSIONS) {
|
||||||
|
expect(CAP_RULES.some(c => c.dimension === d)).toBe(true)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('conformance with the evidence-record standard', () => {
|
||||||
|
const withCapEvidence = scoreDimension('security', {
|
||||||
|
raw: 70,
|
||||||
|
openCaps: [{
|
||||||
|
id: CAP_RULES.find(c => c.dimension === 'security')!.id,
|
||||||
|
evidence: [{ ref: 'docs/secrets-review.md:12', note: 'exposed key, never rotated' }],
|
||||||
|
}],
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a cap cites what establishes it', () => {
|
||||||
|
// The gap this closes: "CAPPED sec.leaked-secret" with nothing behind it.
|
||||||
|
expect(withCapEvidence.boundBy!.evidence).toHaveLength(1)
|
||||||
|
const text = formatProduct(scoreProduct(assessmentWith('security', {
|
||||||
|
raw: 70,
|
||||||
|
openCaps: [{
|
||||||
|
id: CAP_RULES.find(c => c.dimension === 'security')!.id,
|
||||||
|
evidence: [{ ref: 'docs/secrets-review.md:12', note: 'never rotated' }],
|
||||||
|
}],
|
||||||
|
})), { evidence: 'full' })
|
||||||
|
expect(text).toContain('docs/secrets-review.md:12')
|
||||||
|
expect(text).toContain('never rotated')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('still applies a cap that cites nothing, and says it cited nothing', () => {
|
||||||
|
// An unproven cap is a smaller error than an unapplied one.
|
||||||
|
const bare = scoreDimension('security', {
|
||||||
|
raw: 90,
|
||||||
|
openCaps: [CAP_RULES.find(c => c.dimension === 'security')!.id],
|
||||||
|
})
|
||||||
|
expect(bare.score).toBeLessThan(90)
|
||||||
|
const text = formatProduct(scoreProduct(assessmentWith('security', {
|
||||||
|
raw: 90,
|
||||||
|
openCaps: [CAP_RULES.find(c => c.dimension === 'security')!.id],
|
||||||
|
})), { evidence: 'full' })
|
||||||
|
expect(text).toContain('No evidence was recorded for this cap')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('converts a legacy path:line ref into a real citation', () => {
|
||||||
|
const d = scoreDimension('security', { raw: 50, evidence: [{ ref: 'server/api/x.ts:9' }] })
|
||||||
|
expect(d.evidence[0]!.source).toMatchObject({ kind: 'file', path: 'server/api/x.ts', line: 9 })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a score withheld for lack of evidence is not-assessed, not a pass', () => {
|
||||||
|
const p = scoreProduct(assessmentWith('security', { raw: 95 }))
|
||||||
|
const f = productReport(p).findings.find(x => x.id.endsWith('.security'))!
|
||||||
|
expect(f.determination).toBe('not-assessed')
|
||||||
|
expect(formatProduct(p)).toContain('no evidence was cited')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('every conclusion carries a citation', () => {
|
||||||
|
expect(validateReport(productReport(scoreProduct(assessmentWith('security', { raw: 60 }))))).toEqual([])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('labels the average so it cannot be read as the headline', () => {
|
||||||
|
const text = formatProduct(scoreProduct(assessmentWith('security', { raw: 60 })))
|
||||||
|
expect(text).toContain('never as the headline')
|
||||||
|
expect(text).toContain('set by the weakest dimension')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('band rendering', () => {
|
||||||
|
it('prints the band name, not a stringified object', () => {
|
||||||
|
const text = formatProduct(scoreProduct(flat('app', 80)))
|
||||||
|
expect(text).not.toContain('[object Object]')
|
||||||
|
expect(text).toContain('Production')
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022", "module": "ES2022", "moduleResolution": "bundler",
|
||||||
|
"declaration": true, "outDir": "dist", "rootDir": "src",
|
||||||
|
"strict": true, "skipLibCheck": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts"]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user