First public release
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
root = true
|
||||
|
||||
[*]
|
||||
charset = utf-8
|
||||
end_of_line = lf
|
||||
insert_final_newline = true
|
||||
trim_trailing_whitespace = true
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
|
||||
[*.md]
|
||||
trim_trailing_whitespace = false
|
||||
@@ -0,0 +1,8 @@
|
||||
node_modules/
|
||||
dist/
|
||||
*.log
|
||||
.DS_Store
|
||||
.env*
|
||||
!.env.example
|
||||
.npmrc
|
||||
dist-cjs/
|
||||
@@ -0,0 +1,6 @@
|
||||
stages: [test]
|
||||
|
||||
test:
|
||||
stage: test
|
||||
image: node:22-alpine
|
||||
script: [npm ci, npm run build, npm test]
|
||||
@@ -0,0 +1,14 @@
|
||||
# Changelog
|
||||
|
||||
## 0.1.0 - 2026-09-28
|
||||
|
||||
First public release under MIT.
|
||||
|
||||
- `generateCases()` derives edge, misuse and abuse cases from an operation's
|
||||
parameters: empty and single-item collections, absent and null fields,
|
||||
boundary + 1, zero vs absent, string edge cases, malformed, nonexistent and
|
||||
wrong-tenant identifiers, idempotency replays, double submit and partial
|
||||
failure.
|
||||
- `assessCoverage()` compares the derived cases with the case ids your tests
|
||||
cover. An uncovered case is reported as not assessed, never as a failure.
|
||||
- `formatSuite()` renders the report for a terminal or CI log.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Contributing
|
||||
|
||||
Issues and merge requests are welcome at
|
||||
https://gitlab.com/extant2000/adversarial-suite.
|
||||
|
||||
## Ground rules
|
||||
|
||||
- **A test that cannot fail proves nothing.** If you fix a bug, add a test
|
||||
that fails without your fix, and check that it does fail before you submit.
|
||||
- Measure, do not assume. Two modules with the same line count can be
|
||||
different programs.
|
||||
- Keep dependencies minimal. Every new dependency needs a reason.
|
||||
- Explain why in comments, not what.
|
||||
|
||||
## Before you open a merge request
|
||||
|
||||
Run both of these and make sure they pass:
|
||||
|
||||
```
|
||||
npm run build
|
||||
npm test
|
||||
```
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Extant 2000 LLC
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,95 @@
|
||||
# AdversarialSuite
|
||||
|
||||
Derives the edge, misuse and abuse cases an interface implies, and reports
|
||||
which of them have no test.
|
||||
|
||||
## What it does and why
|
||||
|
||||
A test suite can pass while missing the inputs that break the code. The
|
||||
failure this library targets is not a failing test. It is a green suite with a
|
||||
hole in it.
|
||||
|
||||
AdversarialSuite does not run tests. You describe your operations and their
|
||||
parameters, and it derives the cases those parameters imply. Then it compares
|
||||
them with the list of case ids your tests already cover. It has no test runner
|
||||
and makes no network calls.
|
||||
|
||||
Each generator comes from a class of defect that shows up in real systems:
|
||||
|
||||
| Generator | Defect class |
|
||||
|---|---|
|
||||
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
|
||||
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
|
||||
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
|
||||
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
|
||||
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
|
||||
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
|
||||
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
|
||||
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
|
||||
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
|
||||
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
|
||||
|
||||
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
|
||||
case. `idempotent: true` produces the replay cases. `mutates: true` produces
|
||||
double-submit and partial-failure.
|
||||
|
||||
## Uncovered is not failed
|
||||
|
||||
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
|
||||
has been shown to be broken. What has been shown is that nobody checked, and
|
||||
that is the state a passing suite cannot tell apart from working code.
|
||||
|
||||
Reports follow the `@extant2000/evidence-record` format: every finding cites
|
||||
the file where the operation is defined, and an empty description of
|
||||
operations is reported as not assessed instead of clean.
|
||||
|
||||
## Usage
|
||||
|
||||
```ts
|
||||
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
|
||||
|
||||
const operations: Operation[] = [{
|
||||
name: 'listContacts',
|
||||
where: 'src/api/contacts.ts',
|
||||
params: [
|
||||
{ name: 'ids', kind: 'collection', max: 400 },
|
||||
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
||||
],
|
||||
}]
|
||||
|
||||
// Every case the interface implies, with input, expectation and rationale.
|
||||
const cases = generateCases(operations)
|
||||
|
||||
// Compare against the case ids your tests already cover.
|
||||
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
|
||||
|
||||
// Or render it for a terminal or CI log.
|
||||
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
|
||||
```
|
||||
|
||||
Parameter kinds are `collection`, `string`, `number`, `boolean`,
|
||||
`identifier` and `optional`. `max` sets the documented limit that the
|
||||
boundary cases use.
|
||||
|
||||
Output of the example above, shortened (the header line is omitted):
|
||||
|
||||
```
|
||||
NOT ASSESSED · 8 items examined · 8 findings
|
||||
1 of 8 implied cases covered; 7 with no test found
|
||||
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
|
||||
|
||||
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
|
||||
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
|
||||
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
|
||||
◦ info edge: ids = [] [MET]
|
||||
```
|
||||
|
||||
## Install
|
||||
|
||||
```
|
||||
npm install @extant2000/adversarial-suite
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
@@ -0,0 +1,6 @@
|
||||
# Security
|
||||
|
||||
Please report vulnerabilities privately by email to security@extant2000.com.
|
||||
Do not open a public issue for a security problem.
|
||||
|
||||
We aim to acknowledge every report within 5 business days.
|
||||
Generated
+1503
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"name": "@extant2000/adversarial-suite",
|
||||
"version": "0.1.0",
|
||||
"private": false,
|
||||
"description": "Derives the edge, misuse and abuse cases an interface implies and reports which have no test.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
"main": "./dist-cjs/index.js",
|
||||
"files": [
|
||||
"dist",
|
||||
"dist-cjs",
|
||||
"src",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://gitlab.com/extant2000/adversarial-suite.git"
|
||||
},
|
||||
"module": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js",
|
||||
"require": "./dist-cjs/index.js"
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc && tsc -p tsconfig.cjs.json && node -e \"require('fs').writeFileSync('dist-cjs/package.json', JSON.stringify({type:'commonjs'}))\"",
|
||||
"test": "vitest run",
|
||||
"prepublishOnly": "npm run build"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.6.0",
|
||||
"vitest": "^2.1.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@extant2000/evidence-record": "^0.1.2"
|
||||
},
|
||||
"author": "Extant 2000 LLC",
|
||||
"homepage": "https://gitlab.com/extant2000/adversarial-suite",
|
||||
"bugs": {
|
||||
"url": "https://gitlab.com/extant2000/adversarial-suite/-/issues"
|
||||
}
|
||||
}
|
||||
+294
@@ -0,0 +1,294 @@
|
||||
/**
|
||||
* AdversarialSuite — generate the edge, misuse and abuse cases a happy-path
|
||||
* test suite does not contain, and report which of them nobody has covered.
|
||||
*
|
||||
* This library does not run tests. It derives the cases an interface implies
|
||||
* and checks them against the cases that exist, because the failure it
|
||||
* targets is not a failing test — it is a passing suite with a hole in it.
|
||||
* Every generator below comes from a class of defect seen in real systems:
|
||||
*
|
||||
* - **Empty collection.** `[].every(...)` is `true`, so a requirement with
|
||||
* zero verifications reported as verified, and a gate with zero matching
|
||||
* files reported as passing. The empty case is the single most productive
|
||||
* input in this whole set.
|
||||
* - **Absent optional field.** A provider silently dropped a requested
|
||||
* expansion; the response stayed valid and the field was simply missing.
|
||||
* Nothing asserted its presence.
|
||||
* - **Boundary + 1.** A URL filter died past roughly 400 identifiers and a
|
||||
* result set capped silently at 1000 rows. Both are fine at the sizes
|
||||
* anyone tests with.
|
||||
* - **Null-vs-zero.** Zero samples is not 100%. A null composite became a
|
||||
* truthy all-null object.
|
||||
* - **Repeated identical call.** An idempotency key derived from a literal
|
||||
* replayed a deleted object for 24 hours.
|
||||
* - **Wrong-tenant identifier.** A well-formed id belonging to somebody
|
||||
* else is the authorisation case that syntactic validation waves through.
|
||||
*
|
||||
* Pure: no test runner, no network.
|
||||
*/
|
||||
|
||||
import {
|
||||
type CapabilityReport,
|
||||
type Finding,
|
||||
type FormatOptions,
|
||||
type Severity,
|
||||
evidence,
|
||||
formatReport as renderReport,
|
||||
} from '@extant2000/evidence-record'
|
||||
|
||||
export type ParamKind =
|
||||
| 'collection'
|
||||
| 'string'
|
||||
| 'number'
|
||||
| 'boolean'
|
||||
| 'identifier'
|
||||
| 'optional'
|
||||
|
||||
export interface Param {
|
||||
name: string
|
||||
kind: ParamKind
|
||||
required?: boolean
|
||||
/** Documented maximum — length, count, or value. Boundary cases use it. */
|
||||
max?: number
|
||||
/** True when the value is scoped to a tenant, user or organisation. */
|
||||
tenantScoped?: boolean
|
||||
}
|
||||
|
||||
export interface Operation {
|
||||
name: string
|
||||
params: Param[]
|
||||
/** True when repeating the call must not repeat its effect. */
|
||||
idempotent?: boolean
|
||||
/** True when the operation writes. Abuse cases weigh heavier. */
|
||||
mutates?: boolean
|
||||
/** Where the operation is defined — cited on every generated case. */
|
||||
where: string
|
||||
}
|
||||
|
||||
export type CaseCategory = 'edge' | 'misuse' | 'abuse'
|
||||
|
||||
export interface AdversarialCase {
|
||||
id: string
|
||||
operation: string
|
||||
category: CaseCategory
|
||||
/** What to do. */
|
||||
input: string
|
||||
/** What must happen. Written as a requirement, not a hope. */
|
||||
expectation: string
|
||||
/** Why this case exists — the defect class it comes from. */
|
||||
rationale: string
|
||||
severity: Severity
|
||||
}
|
||||
|
||||
const CAT_SEVERITY: Record<CaseCategory, Severity> = {
|
||||
edge: 'medium',
|
||||
misuse: 'high',
|
||||
abuse: 'critical',
|
||||
}
|
||||
|
||||
function mk(
|
||||
op: Operation,
|
||||
suffix: string,
|
||||
category: CaseCategory,
|
||||
input: string,
|
||||
expectation: string,
|
||||
rationale: string,
|
||||
severity?: Severity,
|
||||
): AdversarialCase {
|
||||
return {
|
||||
id: `${op.name}/${suffix}`,
|
||||
operation: op.name,
|
||||
category,
|
||||
input,
|
||||
expectation,
|
||||
rationale,
|
||||
severity: severity ?? CAT_SEVERITY[category],
|
||||
}
|
||||
}
|
||||
|
||||
function casesForParam(op: Operation, p: Param): AdversarialCase[] {
|
||||
const out: AdversarialCase[] = []
|
||||
|
||||
if (p.kind === 'collection') {
|
||||
// The single most productive input in this set.
|
||||
out.push(mk(op, `${p.name}/empty`, 'edge',
|
||||
`${p.name} = []`,
|
||||
'Reports "nothing was checked", never a pass.',
|
||||
'[].every(...) is true, so an empty collection satisfies every universal assertion made about it. A requirement with zero verifications reported as verified this way.',
|
||||
'high'))
|
||||
|
||||
out.push(mk(op, `${p.name}/single`, 'edge',
|
||||
`${p.name} = [one item]`,
|
||||
'Reports that a comparison needs at least two members, rather than a clean result.',
|
||||
'One member cannot disagree with itself, so a comparison over a single element trivially succeeds.'))
|
||||
|
||||
if (p.max) {
|
||||
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||
`${p.name} with ${p.max + 1} items`,
|
||||
'Rejects, paginates or chunks — never silently truncates.',
|
||||
`Documented maximum is ${p.max}. A URL filter can fail past a few hundred identifiers, and a result set can be capped silently at 1000 rows; both behave correctly at the sizes anyone tests with.`))
|
||||
}
|
||||
}
|
||||
|
||||
if (p.kind === 'number') {
|
||||
out.push(mk(op, `${p.name}/zero`, 'edge',
|
||||
`${p.name} = 0`,
|
||||
'Distinguishes zero from absent. Zero samples is not 100%.',
|
||||
'A zero denominator produced a flattering default rather than an abstention.',
|
||||
'high'))
|
||||
out.push(mk(op, `${p.name}/negative`, 'misuse',
|
||||
`${p.name} = -1`, 'Rejects rather than coercing.',
|
||||
'Negative values reach arithmetic that assumed non-negative input.'))
|
||||
if (p.max !== undefined) {
|
||||
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||
`${p.name} = ${p.max + 1}`, `Rejects above the documented maximum of ${p.max}.`,
|
||||
'Off-by-one at a documented boundary.'))
|
||||
}
|
||||
}
|
||||
|
||||
if (p.kind === 'optional' || !p.required) {
|
||||
out.push(mk(op, `${p.name}/absent`, 'edge',
|
||||
`${p.name} omitted entirely`,
|
||||
'Behaves correctly, and does not report success for work it skipped.',
|
||||
'A provider silently dropped a requested expansion. The response stayed structurally valid and the field was simply absent, so schema validation passed while the feature was dead.',
|
||||
'high'))
|
||||
out.push(mk(op, `${p.name}/null`, 'edge',
|
||||
`${p.name} = null`,
|
||||
'Treats null as absent, not as a present empty value.',
|
||||
'A null composite deserialised into a truthy object with all-null fields, which passed every presence check made on it.'))
|
||||
}
|
||||
|
||||
if (p.kind === 'string') {
|
||||
out.push(mk(op, `${p.name}/empty-string`, 'edge',
|
||||
`${p.name} = ""`, 'Distinguishes empty from absent.',
|
||||
'An empty string is falsy and present at the same time.'))
|
||||
out.push(mk(op, `${p.name}/unicode`, 'misuse',
|
||||
`${p.name} with combining marks, RTL overrides and a 4-byte emoji`,
|
||||
'Stores and returns the value unchanged; length limits count characters consistently.',
|
||||
'Byte-length limits truncate mid-codepoint.'))
|
||||
if (p.max) {
|
||||
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||
`${p.name} of length ${p.max + 1}`, `Rejects above ${p.max} rather than truncating silently.`,
|
||||
'Silent truncation is data loss that looks like success.'))
|
||||
}
|
||||
}
|
||||
|
||||
if (p.kind === 'identifier') {
|
||||
out.push(mk(op, `${p.name}/malformed`, 'misuse',
|
||||
`${p.name} = "not-an-id"`, 'Rejects with a clear error.',
|
||||
'Syntactic validation is the easy half.'))
|
||||
out.push(mk(op, `${p.name}/nonexistent`, 'misuse',
|
||||
`${p.name} = a well-formed id that does not exist`,
|
||||
'Returns not-found, and does not leak whether the id exists elsewhere.',
|
||||
'Distinguishing "absent" from "not yours" through error text is an enumeration oracle.'))
|
||||
|
||||
if (p.tenantScoped) {
|
||||
out.push(mk(op, `${p.name}/wrong-tenant`, 'abuse',
|
||||
`${p.name} = a well-formed id belonging to another tenant`,
|
||||
'Denies. The response is indistinguishable from the not-found case.',
|
||||
'The authorisation case syntactic validation waves through: the id is valid, it exists, and it is not yours. This is the check a shared-table suite exists to enforce.'))
|
||||
}
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
function operationCases(op: Operation): AdversarialCase[] {
|
||||
const out = op.params.flatMap(p => casesForParam(op, p))
|
||||
|
||||
if (op.idempotent) {
|
||||
out.push(mk(op, 'replay/identical', 'edge',
|
||||
'The same call twice with the same idempotency key',
|
||||
'The second call returns the first result. It does not repeat the effect, and it does not return a deleted object.',
|
||||
'A key derived from a literal rather than from the parameters replayed a stale — and in one case deleted — object for 24 hours.',
|
||||
'high'))
|
||||
out.push(mk(op, 'replay/changed-params', 'misuse',
|
||||
'The same idempotency key with different parameters',
|
||||
'Rejects. It does not silently serve the first response for a different request.',
|
||||
'A literal key with changed parameters failed in-flight requests for a full day.'))
|
||||
}
|
||||
|
||||
if (op.mutates) {
|
||||
out.push(mk(op, 'concurrent/double-submit', 'abuse',
|
||||
'Two identical writes issued simultaneously',
|
||||
'Exactly one takes effect, or both are rejected. Never two.',
|
||||
'A double submit is the cheapest abuse there is and the least often tested.'))
|
||||
out.push(mk(op, 'partial-failure', 'edge',
|
||||
'A failure injected midway through a multi-step write',
|
||||
'Leaves no partially-applied state, and reports the failure rather than a success.',
|
||||
'A partially applied write reports success and is discovered later by its inconsistency.',
|
||||
'high'))
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
/** Derive the adversarial cases an operation implies. */
|
||||
export function generateCases(operations: Operation[]): AdversarialCase[] {
|
||||
return (operations ?? []).flatMap(operationCases)
|
||||
}
|
||||
|
||||
export interface Coverage {
|
||||
/** Case ids that already have a test. */
|
||||
covered: Iterable<string>
|
||||
}
|
||||
|
||||
/**
|
||||
* Report which implied cases are covered.
|
||||
*
|
||||
* An uncovered case is `not-assessed`, not `fail`: nothing has been shown to
|
||||
* be broken. What has been shown is that nobody looked — which is the state
|
||||
* a passing suite is least able to distinguish from working.
|
||||
*/
|
||||
export function assessCoverage(
|
||||
operations: Operation[],
|
||||
coverage: Coverage = { covered: [] },
|
||||
): CapabilityReport {
|
||||
const cases = generateCases(operations)
|
||||
const covered = new Set(coverage.covered)
|
||||
|
||||
if (cases.length === 0) {
|
||||
return {
|
||||
capability: 'AdversarialSuite',
|
||||
scope: 'no operations described',
|
||||
examined: 0,
|
||||
findings: [],
|
||||
notes: ['No operations were described, so no cases were derived.'],
|
||||
}
|
||||
}
|
||||
|
||||
const findings: Finding[] = cases.map(c => {
|
||||
const isCovered = covered.has(c.id)
|
||||
const op = operations.find(o => o.name === c.operation)!
|
||||
return {
|
||||
id: c.id,
|
||||
summary: `${c.category}: ${c.input}`,
|
||||
determination: isCovered ? 'pass' : 'not-assessed',
|
||||
severity: isCovered ? 'info' : c.severity,
|
||||
detail: isCovered ? undefined : `Expected behaviour: ${c.expectation} — ${c.rationale}`,
|
||||
evidence: [evidence.file(op.where, undefined, c.operation,
|
||||
isCovered ? 'case covered' : 'case implied by the interface, no test found')],
|
||||
}
|
||||
})
|
||||
|
||||
const gap = cases.length - cases.filter(c => covered.has(c.id)).length
|
||||
return {
|
||||
capability: 'AdversarialSuite',
|
||||
scope: `${cases.length} case(s) implied by ${operations.length} operation(s)`,
|
||||
examined: cases.length,
|
||||
findings,
|
||||
notes: [
|
||||
`${cases.length - gap} of ${cases.length} implied cases covered; ${gap} with no test found`,
|
||||
'An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.',
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
/** Render the derived suite for a terminal, CI log or test plan. */
|
||||
export function formatSuite(
|
||||
operations: Operation[],
|
||||
coverage: Coverage = { covered: [] },
|
||||
opts: FormatOptions = {},
|
||||
): string {
|
||||
return renderReport(assessCoverage(operations, coverage), opts)
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { overall, validateReport } from '@extant2000/evidence-record'
|
||||
import { assessCoverage, formatSuite, generateCases, type Operation } from '../src/index.js'
|
||||
|
||||
const listContacts: Operation = {
|
||||
name: 'listContacts',
|
||||
where: 'server/api/contacts/index.get.ts',
|
||||
params: [
|
||||
{ name: 'ids', kind: 'collection', max: 400 },
|
||||
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
||||
{ name: 'limit', kind: 'number', max: 1000 },
|
||||
],
|
||||
}
|
||||
|
||||
const createSubscription: Operation = {
|
||||
name: 'createSubscription',
|
||||
where: 'server/api/billing/subscribe.post.ts',
|
||||
idempotent: true,
|
||||
mutates: true,
|
||||
params: [{ name: 'priceId', kind: 'identifier', required: true }],
|
||||
}
|
||||
|
||||
const ids = (op: Operation) => generateCases([op]).map(c => c.id)
|
||||
|
||||
describe('the cases come from real defect classes', () => {
|
||||
it('derives the empty-collection case — the most productive input in the set', () => {
|
||||
// [].every(...) is true, so an empty collection satisfies every
|
||||
// universal assertion made about it.
|
||||
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/empty')!
|
||||
expect(c.expectation).toContain('never a pass')
|
||||
expect(c.rationale).toContain('[].every(...) is true')
|
||||
expect(c.severity).toBe('high')
|
||||
})
|
||||
|
||||
it('derives the boundary+1 case from a documented maximum', () => {
|
||||
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/over-max')!
|
||||
expect(c.input).toContain('401 items')
|
||||
expect(c.expectation).toContain('never silently truncates')
|
||||
})
|
||||
|
||||
it('derives the wrong-tenant case only for tenant-scoped identifiers', () => {
|
||||
expect(ids(listContacts)).toContain('listContacts/orgId/wrong-tenant')
|
||||
expect(ids(createSubscription)).not.toContain('createSubscription/priceId/wrong-tenant')
|
||||
})
|
||||
|
||||
it('rates the wrong-tenant case as abuse, the highest severity', () => {
|
||||
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/orgId/wrong-tenant')!
|
||||
expect(c.category).toBe('abuse')
|
||||
expect(c.severity).toBe('critical')
|
||||
expect(c.expectation).toContain('indistinguishable from the not-found case')
|
||||
})
|
||||
|
||||
it('derives the null-vs-zero case for numbers', () => {
|
||||
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/limit/zero')!
|
||||
expect(c.expectation).toContain('Zero samples is not 100%')
|
||||
})
|
||||
|
||||
it('derives the absent-optional case — the silently dropped expansion', () => {
|
||||
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/absent')!
|
||||
expect(c.rationale).toContain('silently dropped a requested expansion')
|
||||
})
|
||||
|
||||
it('derives replay cases only for idempotent operations', () => {
|
||||
expect(ids(createSubscription)).toContain('createSubscription/replay/identical')
|
||||
expect(ids(createSubscription)).toContain('createSubscription/replay/changed-params')
|
||||
expect(ids(listContacts)).not.toContain('listContacts/replay/identical')
|
||||
})
|
||||
|
||||
it('derives concurrency and partial-failure cases only for mutating operations', () => {
|
||||
expect(ids(createSubscription)).toContain('createSubscription/concurrent/double-submit')
|
||||
expect(ids(createSubscription)).toContain('createSubscription/partial-failure')
|
||||
expect(ids(listContacts)).not.toContain('listContacts/concurrent/double-submit')
|
||||
})
|
||||
|
||||
it('writes expectations as requirements, not hopes', () => {
|
||||
for (const c of generateCases([listContacts, createSubscription])) {
|
||||
expect(c.expectation.length).toBeGreaterThan(10)
|
||||
expect(c.rationale.length).toBeGreaterThan(10)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('an uncovered case is an absence of a check, not a failure', () => {
|
||||
it('reports uncovered cases as not-assessed, never fail', () => {
|
||||
const r = assessCoverage([listContacts])
|
||||
expect(r.findings.every(f => f.determination === 'not-assessed')).toBe(true)
|
||||
expect(overall(r)).toBe('not-assessed')
|
||||
expect(formatSuite([listContacts])).toContain('cannot distinguish from working')
|
||||
})
|
||||
|
||||
it('passes a case that has a test', () => {
|
||||
const all = generateCases([listContacts]).map(c => c.id)
|
||||
const r = assessCoverage([listContacts], { covered: all })
|
||||
expect(overall(r)).toBe('pass')
|
||||
})
|
||||
|
||||
it('counts the gap', () => {
|
||||
const all = generateCases([listContacts]).map(c => c.id)
|
||||
const text = formatSuite([listContacts], { covered: all.slice(0, 2) })
|
||||
expect(text).toContain(`2 of ${all.length} implied cases covered`)
|
||||
})
|
||||
|
||||
it('cites the operation definition on every case', () => {
|
||||
expect(formatSuite([listContacts], { covered: [] }, { evidence: 'full' }))
|
||||
.toContain('server/api/contacts/index.get.ts')
|
||||
})
|
||||
})
|
||||
|
||||
describe('conformance with the evidence-record standard', () => {
|
||||
it('no operations described is not-assessed, never clean', () => {
|
||||
expect(overall(assessCoverage([]))).toBe('not-assessed')
|
||||
})
|
||||
|
||||
it('every conclusion carries a citation', () => {
|
||||
expect(validateReport(assessCoverage([listContacts, createSubscription]))).toEqual([])
|
||||
})
|
||||
|
||||
it('produces unique case ids', () => {
|
||||
const all = generateCases([listContacts, createSubscription]).map(c => c.id)
|
||||
expect(new Set(all).size).toBe(all.length)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"extends": "./tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"module": "CommonJS",
|
||||
"moduleResolution": "Node",
|
||||
"outDir": "dist-cjs",
|
||||
"declaration": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022", "module": "ES2022", "moduleResolution": "bundler",
|
||||
"declaration": true, "outDir": "dist", "rootDir": "src",
|
||||
"strict": true, "skipLibCheck": true
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
Reference in New Issue
Block a user