First public release
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,12 @@
|
|||||||
|
root = true
|
||||||
|
|
||||||
|
[*]
|
||||||
|
charset = utf-8
|
||||||
|
end_of_line = lf
|
||||||
|
insert_final_newline = true
|
||||||
|
trim_trailing_whitespace = true
|
||||||
|
indent_style = space
|
||||||
|
indent_size = 2
|
||||||
|
|
||||||
|
[*.md]
|
||||||
|
trim_trailing_whitespace = false
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
|
.env*
|
||||||
|
!.env.example
|
||||||
|
.npmrc
|
||||||
|
dist-cjs/
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
stages: [test]
|
||||||
|
|
||||||
|
test:
|
||||||
|
stage: test
|
||||||
|
image: node:22-alpine
|
||||||
|
script: [npm ci, npm run build, npm test]
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.1.0 - 2026-09-28
|
||||||
|
|
||||||
|
First public release under MIT.
|
||||||
|
|
||||||
|
- `generateCases()` derives edge, misuse and abuse cases from an operation's
|
||||||
|
parameters: empty and single-item collections, absent and null fields,
|
||||||
|
boundary + 1, zero vs absent, string edge cases, malformed, nonexistent and
|
||||||
|
wrong-tenant identifiers, idempotency replays, double submit and partial
|
||||||
|
failure.
|
||||||
|
- `assessCoverage()` compares the derived cases with the case ids your tests
|
||||||
|
cover. An uncovered case is reported as not assessed, never as a failure.
|
||||||
|
- `formatSuite()` renders the report for a terminal or CI log.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Contributing
|
||||||
|
|
||||||
|
Issues and merge requests are welcome at
|
||||||
|
https://gitlab.com/extant2000/adversarial-suite.
|
||||||
|
|
||||||
|
## Ground rules
|
||||||
|
|
||||||
|
- **A test that cannot fail proves nothing.** If you fix a bug, add a test
|
||||||
|
that fails without your fix, and check that it does fail before you submit.
|
||||||
|
- Measure, do not assume. Two modules with the same line count can be
|
||||||
|
different programs.
|
||||||
|
- Keep dependencies minimal. Every new dependency needs a reason.
|
||||||
|
- Explain why in comments, not what.
|
||||||
|
|
||||||
|
## Before you open a merge request
|
||||||
|
|
||||||
|
Run both of these and make sure they pass:
|
||||||
|
|
||||||
|
```
|
||||||
|
npm run build
|
||||||
|
npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Extant 2000 LLC
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# AdversarialSuite
|
||||||
|
|
||||||
|
Derives the edge, misuse and abuse cases an interface implies, and reports
|
||||||
|
which of them have no test.
|
||||||
|
|
||||||
|
## What it does and why
|
||||||
|
|
||||||
|
A test suite can pass while missing the inputs that break the code. The
|
||||||
|
failure this library targets is not a failing test. It is a green suite with a
|
||||||
|
hole in it.
|
||||||
|
|
||||||
|
AdversarialSuite does not run tests. You describe your operations and their
|
||||||
|
parameters, and it derives the cases those parameters imply. Then it compares
|
||||||
|
them with the list of case ids your tests already cover. It has no test runner
|
||||||
|
and makes no network calls.
|
||||||
|
|
||||||
|
Each generator comes from a class of defect that shows up in real systems:
|
||||||
|
|
||||||
|
| Generator | Defect class |
|
||||||
|
|---|---|
|
||||||
|
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
|
||||||
|
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
|
||||||
|
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
|
||||||
|
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
|
||||||
|
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
|
||||||
|
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
|
||||||
|
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
|
||||||
|
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
|
||||||
|
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
|
||||||
|
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
|
||||||
|
|
||||||
|
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
|
||||||
|
case. `idempotent: true` produces the replay cases. `mutates: true` produces
|
||||||
|
double-submit and partial-failure.
|
||||||
|
|
||||||
|
## Uncovered is not failed
|
||||||
|
|
||||||
|
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
|
||||||
|
has been shown to be broken. What has been shown is that nobody checked, and
|
||||||
|
that is the state a passing suite cannot tell apart from working code.
|
||||||
|
|
||||||
|
Reports follow the `@extant2000/evidence-record` format: every finding cites
|
||||||
|
the file where the operation is defined, and an empty description of
|
||||||
|
operations is reported as not assessed instead of clean.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
|
||||||
|
|
||||||
|
const operations: Operation[] = [{
|
||||||
|
name: 'listContacts',
|
||||||
|
where: 'src/api/contacts.ts',
|
||||||
|
params: [
|
||||||
|
{ name: 'ids', kind: 'collection', max: 400 },
|
||||||
|
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
||||||
|
],
|
||||||
|
}]
|
||||||
|
|
||||||
|
// Every case the interface implies, with input, expectation and rationale.
|
||||||
|
const cases = generateCases(operations)
|
||||||
|
|
||||||
|
// Compare against the case ids your tests already cover.
|
||||||
|
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
|
||||||
|
|
||||||
|
// Or render it for a terminal or CI log.
|
||||||
|
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
|
||||||
|
```
|
||||||
|
|
||||||
|
Parameter kinds are `collection`, `string`, `number`, `boolean`,
|
||||||
|
`identifier` and `optional`. `max` sets the documented limit that the
|
||||||
|
boundary cases use.
|
||||||
|
|
||||||
|
Output of the example above, shortened (the header line is omitted):
|
||||||
|
|
||||||
|
```
|
||||||
|
NOT ASSESSED · 8 items examined · 8 findings
|
||||||
|
1 of 8 implied cases covered; 7 with no test found
|
||||||
|
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
|
||||||
|
|
||||||
|
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
|
||||||
|
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
|
||||||
|
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
|
||||||
|
◦ info edge: ids = [] [MET]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```
|
||||||
|
npm install @extant2000/adversarial-suite
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT. See [LICENSE](LICENSE).
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# Security
|
||||||
|
|
||||||
|
Please report vulnerabilities privately by email to security@extant2000.com.
|
||||||
|
Do not open a public issue for a security problem.
|
||||||
|
|
||||||
|
We aim to acknowledge every report within 5 business days.
|
||||||
Generated
+1503
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,46 @@
|
|||||||
|
{
|
||||||
|
"name": "@extant2000/adversarial-suite",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"private": false,
|
||||||
|
"description": "Derives the edge, misuse and abuse cases an interface implies and reports which have no test.",
|
||||||
|
"license": "MIT",
|
||||||
|
"type": "module",
|
||||||
|
"main": "./dist-cjs/index.js",
|
||||||
|
"files": [
|
||||||
|
"dist",
|
||||||
|
"dist-cjs",
|
||||||
|
"src",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
],
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "https://gitlab.com/extant2000/adversarial-suite.git"
|
||||||
|
},
|
||||||
|
"module": "./dist/index.js",
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"import": "./dist/index.js",
|
||||||
|
"require": "./dist-cjs/index.js"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc && tsc -p tsconfig.cjs.json && node -e \"require('fs').writeFileSync('dist-cjs/package.json', JSON.stringify({type:'commonjs'}))\"",
|
||||||
|
"test": "vitest run",
|
||||||
|
"prepublishOnly": "npm run build"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"typescript": "^5.6.0",
|
||||||
|
"vitest": "^2.1.0"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@extant2000/evidence-record": "^0.1.2"
|
||||||
|
},
|
||||||
|
"author": "Extant 2000 LLC",
|
||||||
|
"homepage": "https://gitlab.com/extant2000/adversarial-suite",
|
||||||
|
"bugs": {
|
||||||
|
"url": "https://gitlab.com/extant2000/adversarial-suite/-/issues"
|
||||||
|
}
|
||||||
|
}
|
||||||
+294
@@ -0,0 +1,294 @@
|
|||||||
|
/**
|
||||||
|
* AdversarialSuite — generate the edge, misuse and abuse cases a happy-path
|
||||||
|
* test suite does not contain, and report which of them nobody has covered.
|
||||||
|
*
|
||||||
|
* This library does not run tests. It derives the cases an interface implies
|
||||||
|
* and checks them against the cases that exist, because the failure it
|
||||||
|
* targets is not a failing test — it is a passing suite with a hole in it.
|
||||||
|
* Every generator below comes from a class of defect seen in real systems:
|
||||||
|
*
|
||||||
|
* - **Empty collection.** `[].every(...)` is `true`, so a requirement with
|
||||||
|
* zero verifications reported as verified, and a gate with zero matching
|
||||||
|
* files reported as passing. The empty case is the single most productive
|
||||||
|
* input in this whole set.
|
||||||
|
* - **Absent optional field.** A provider silently dropped a requested
|
||||||
|
* expansion; the response stayed valid and the field was simply missing.
|
||||||
|
* Nothing asserted its presence.
|
||||||
|
* - **Boundary + 1.** A URL filter died past roughly 400 identifiers and a
|
||||||
|
* result set capped silently at 1000 rows. Both are fine at the sizes
|
||||||
|
* anyone tests with.
|
||||||
|
* - **Null-vs-zero.** Zero samples is not 100%. A null composite became a
|
||||||
|
* truthy all-null object.
|
||||||
|
* - **Repeated identical call.** An idempotency key derived from a literal
|
||||||
|
* replayed a deleted object for 24 hours.
|
||||||
|
* - **Wrong-tenant identifier.** A well-formed id belonging to somebody
|
||||||
|
* else is the authorisation case that syntactic validation waves through.
|
||||||
|
*
|
||||||
|
* Pure: no test runner, no network.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import {
|
||||||
|
type CapabilityReport,
|
||||||
|
type Finding,
|
||||||
|
type FormatOptions,
|
||||||
|
type Severity,
|
||||||
|
evidence,
|
||||||
|
formatReport as renderReport,
|
||||||
|
} from '@extant2000/evidence-record'
|
||||||
|
|
||||||
|
export type ParamKind =
|
||||||
|
| 'collection'
|
||||||
|
| 'string'
|
||||||
|
| 'number'
|
||||||
|
| 'boolean'
|
||||||
|
| 'identifier'
|
||||||
|
| 'optional'
|
||||||
|
|
||||||
|
export interface Param {
|
||||||
|
name: string
|
||||||
|
kind: ParamKind
|
||||||
|
required?: boolean
|
||||||
|
/** Documented maximum — length, count, or value. Boundary cases use it. */
|
||||||
|
max?: number
|
||||||
|
/** True when the value is scoped to a tenant, user or organisation. */
|
||||||
|
tenantScoped?: boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Operation {
|
||||||
|
name: string
|
||||||
|
params: Param[]
|
||||||
|
/** True when repeating the call must not repeat its effect. */
|
||||||
|
idempotent?: boolean
|
||||||
|
/** True when the operation writes. Abuse cases weigh heavier. */
|
||||||
|
mutates?: boolean
|
||||||
|
/** Where the operation is defined — cited on every generated case. */
|
||||||
|
where: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export type CaseCategory = 'edge' | 'misuse' | 'abuse'
|
||||||
|
|
||||||
|
export interface AdversarialCase {
|
||||||
|
id: string
|
||||||
|
operation: string
|
||||||
|
category: CaseCategory
|
||||||
|
/** What to do. */
|
||||||
|
input: string
|
||||||
|
/** What must happen. Written as a requirement, not a hope. */
|
||||||
|
expectation: string
|
||||||
|
/** Why this case exists — the defect class it comes from. */
|
||||||
|
rationale: string
|
||||||
|
severity: Severity
|
||||||
|
}
|
||||||
|
|
||||||
|
const CAT_SEVERITY: Record<CaseCategory, Severity> = {
|
||||||
|
edge: 'medium',
|
||||||
|
misuse: 'high',
|
||||||
|
abuse: 'critical',
|
||||||
|
}
|
||||||
|
|
||||||
|
function mk(
|
||||||
|
op: Operation,
|
||||||
|
suffix: string,
|
||||||
|
category: CaseCategory,
|
||||||
|
input: string,
|
||||||
|
expectation: string,
|
||||||
|
rationale: string,
|
||||||
|
severity?: Severity,
|
||||||
|
): AdversarialCase {
|
||||||
|
return {
|
||||||
|
id: `${op.name}/${suffix}`,
|
||||||
|
operation: op.name,
|
||||||
|
category,
|
||||||
|
input,
|
||||||
|
expectation,
|
||||||
|
rationale,
|
||||||
|
severity: severity ?? CAT_SEVERITY[category],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function casesForParam(op: Operation, p: Param): AdversarialCase[] {
|
||||||
|
const out: AdversarialCase[] = []
|
||||||
|
|
||||||
|
if (p.kind === 'collection') {
|
||||||
|
// The single most productive input in this set.
|
||||||
|
out.push(mk(op, `${p.name}/empty`, 'edge',
|
||||||
|
`${p.name} = []`,
|
||||||
|
'Reports "nothing was checked", never a pass.',
|
||||||
|
'[].every(...) is true, so an empty collection satisfies every universal assertion made about it. A requirement with zero verifications reported as verified this way.',
|
||||||
|
'high'))
|
||||||
|
|
||||||
|
out.push(mk(op, `${p.name}/single`, 'edge',
|
||||||
|
`${p.name} = [one item]`,
|
||||||
|
'Reports that a comparison needs at least two members, rather than a clean result.',
|
||||||
|
'One member cannot disagree with itself, so a comparison over a single element trivially succeeds.'))
|
||||||
|
|
||||||
|
if (p.max) {
|
||||||
|
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||||
|
`${p.name} with ${p.max + 1} items`,
|
||||||
|
'Rejects, paginates or chunks — never silently truncates.',
|
||||||
|
`Documented maximum is ${p.max}. A URL filter can fail past a few hundred identifiers, and a result set can be capped silently at 1000 rows; both behave correctly at the sizes anyone tests with.`))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (p.kind === 'number') {
|
||||||
|
out.push(mk(op, `${p.name}/zero`, 'edge',
|
||||||
|
`${p.name} = 0`,
|
||||||
|
'Distinguishes zero from absent. Zero samples is not 100%.',
|
||||||
|
'A zero denominator produced a flattering default rather than an abstention.',
|
||||||
|
'high'))
|
||||||
|
out.push(mk(op, `${p.name}/negative`, 'misuse',
|
||||||
|
`${p.name} = -1`, 'Rejects rather than coercing.',
|
||||||
|
'Negative values reach arithmetic that assumed non-negative input.'))
|
||||||
|
if (p.max !== undefined) {
|
||||||
|
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||||
|
`${p.name} = ${p.max + 1}`, `Rejects above the documented maximum of ${p.max}.`,
|
||||||
|
'Off-by-one at a documented boundary.'))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (p.kind === 'optional' || !p.required) {
|
||||||
|
out.push(mk(op, `${p.name}/absent`, 'edge',
|
||||||
|
`${p.name} omitted entirely`,
|
||||||
|
'Behaves correctly, and does not report success for work it skipped.',
|
||||||
|
'A provider silently dropped a requested expansion. The response stayed structurally valid and the field was simply absent, so schema validation passed while the feature was dead.',
|
||||||
|
'high'))
|
||||||
|
out.push(mk(op, `${p.name}/null`, 'edge',
|
||||||
|
`${p.name} = null`,
|
||||||
|
'Treats null as absent, not as a present empty value.',
|
||||||
|
'A null composite deserialised into a truthy object with all-null fields, which passed every presence check made on it.'))
|
||||||
|
}
|
||||||
|
|
||||||
|
if (p.kind === 'string') {
|
||||||
|
out.push(mk(op, `${p.name}/empty-string`, 'edge',
|
||||||
|
`${p.name} = ""`, 'Distinguishes empty from absent.',
|
||||||
|
'An empty string is falsy and present at the same time.'))
|
||||||
|
out.push(mk(op, `${p.name}/unicode`, 'misuse',
|
||||||
|
`${p.name} with combining marks, RTL overrides and a 4-byte emoji`,
|
||||||
|
'Stores and returns the value unchanged; length limits count characters consistently.',
|
||||||
|
'Byte-length limits truncate mid-codepoint.'))
|
||||||
|
if (p.max) {
|
||||||
|
out.push(mk(op, `${p.name}/over-max`, 'misuse',
|
||||||
|
`${p.name} of length ${p.max + 1}`, `Rejects above ${p.max} rather than truncating silently.`,
|
||||||
|
'Silent truncation is data loss that looks like success.'))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (p.kind === 'identifier') {
|
||||||
|
out.push(mk(op, `${p.name}/malformed`, 'misuse',
|
||||||
|
`${p.name} = "not-an-id"`, 'Rejects with a clear error.',
|
||||||
|
'Syntactic validation is the easy half.'))
|
||||||
|
out.push(mk(op, `${p.name}/nonexistent`, 'misuse',
|
||||||
|
`${p.name} = a well-formed id that does not exist`,
|
||||||
|
'Returns not-found, and does not leak whether the id exists elsewhere.',
|
||||||
|
'Distinguishing "absent" from "not yours" through error text is an enumeration oracle.'))
|
||||||
|
|
||||||
|
if (p.tenantScoped) {
|
||||||
|
out.push(mk(op, `${p.name}/wrong-tenant`, 'abuse',
|
||||||
|
`${p.name} = a well-formed id belonging to another tenant`,
|
||||||
|
'Denies. The response is indistinguishable from the not-found case.',
|
||||||
|
'The authorisation case syntactic validation waves through: the id is valid, it exists, and it is not yours. This is the check a shared-table suite exists to enforce.'))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
function operationCases(op: Operation): AdversarialCase[] {
|
||||||
|
const out = op.params.flatMap(p => casesForParam(op, p))
|
||||||
|
|
||||||
|
if (op.idempotent) {
|
||||||
|
out.push(mk(op, 'replay/identical', 'edge',
|
||||||
|
'The same call twice with the same idempotency key',
|
||||||
|
'The second call returns the first result. It does not repeat the effect, and it does not return a deleted object.',
|
||||||
|
'A key derived from a literal rather than from the parameters replayed a stale — and in one case deleted — object for 24 hours.',
|
||||||
|
'high'))
|
||||||
|
out.push(mk(op, 'replay/changed-params', 'misuse',
|
||||||
|
'The same idempotency key with different parameters',
|
||||||
|
'Rejects. It does not silently serve the first response for a different request.',
|
||||||
|
'A literal key with changed parameters failed in-flight requests for a full day.'))
|
||||||
|
}
|
||||||
|
|
||||||
|
if (op.mutates) {
|
||||||
|
out.push(mk(op, 'concurrent/double-submit', 'abuse',
|
||||||
|
'Two identical writes issued simultaneously',
|
||||||
|
'Exactly one takes effect, or both are rejected. Never two.',
|
||||||
|
'A double submit is the cheapest abuse there is and the least often tested.'))
|
||||||
|
out.push(mk(op, 'partial-failure', 'edge',
|
||||||
|
'A failure injected midway through a multi-step write',
|
||||||
|
'Leaves no partially-applied state, and reports the failure rather than a success.',
|
||||||
|
'A partially applied write reports success and is discovered later by its inconsistency.',
|
||||||
|
'high'))
|
||||||
|
}
|
||||||
|
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Derive the adversarial cases an operation implies. */
|
||||||
|
export function generateCases(operations: Operation[]): AdversarialCase[] {
|
||||||
|
return (operations ?? []).flatMap(operationCases)
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface Coverage {
|
||||||
|
/** Case ids that already have a test. */
|
||||||
|
covered: Iterable<string>
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Report which implied cases are covered.
|
||||||
|
*
|
||||||
|
* An uncovered case is `not-assessed`, not `fail`: nothing has been shown to
|
||||||
|
* be broken. What has been shown is that nobody looked — which is the state
|
||||||
|
* a passing suite is least able to distinguish from working.
|
||||||
|
*/
|
||||||
|
export function assessCoverage(
|
||||||
|
operations: Operation[],
|
||||||
|
coverage: Coverage = { covered: [] },
|
||||||
|
): CapabilityReport {
|
||||||
|
const cases = generateCases(operations)
|
||||||
|
const covered = new Set(coverage.covered)
|
||||||
|
|
||||||
|
if (cases.length === 0) {
|
||||||
|
return {
|
||||||
|
capability: 'AdversarialSuite',
|
||||||
|
scope: 'no operations described',
|
||||||
|
examined: 0,
|
||||||
|
findings: [],
|
||||||
|
notes: ['No operations were described, so no cases were derived.'],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const findings: Finding[] = cases.map(c => {
|
||||||
|
const isCovered = covered.has(c.id)
|
||||||
|
const op = operations.find(o => o.name === c.operation)!
|
||||||
|
return {
|
||||||
|
id: c.id,
|
||||||
|
summary: `${c.category}: ${c.input}`,
|
||||||
|
determination: isCovered ? 'pass' : 'not-assessed',
|
||||||
|
severity: isCovered ? 'info' : c.severity,
|
||||||
|
detail: isCovered ? undefined : `Expected behaviour: ${c.expectation} — ${c.rationale}`,
|
||||||
|
evidence: [evidence.file(op.where, undefined, c.operation,
|
||||||
|
isCovered ? 'case covered' : 'case implied by the interface, no test found')],
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
const gap = cases.length - cases.filter(c => covered.has(c.id)).length
|
||||||
|
return {
|
||||||
|
capability: 'AdversarialSuite',
|
||||||
|
scope: `${cases.length} case(s) implied by ${operations.length} operation(s)`,
|
||||||
|
examined: cases.length,
|
||||||
|
findings,
|
||||||
|
notes: [
|
||||||
|
`${cases.length - gap} of ${cases.length} implied cases covered; ${gap} with no test found`,
|
||||||
|
'An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.',
|
||||||
|
],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Render the derived suite for a terminal, CI log or test plan. */
|
||||||
|
export function formatSuite(
|
||||||
|
operations: Operation[],
|
||||||
|
coverage: Coverage = { covered: [] },
|
||||||
|
opts: FormatOptions = {},
|
||||||
|
): string {
|
||||||
|
return renderReport(assessCoverage(operations, coverage), opts)
|
||||||
|
}
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
import { describe, expect, it } from 'vitest'
|
||||||
|
import { overall, validateReport } from '@extant2000/evidence-record'
|
||||||
|
import { assessCoverage, formatSuite, generateCases, type Operation } from '../src/index.js'
|
||||||
|
|
||||||
|
const listContacts: Operation = {
|
||||||
|
name: 'listContacts',
|
||||||
|
where: 'server/api/contacts/index.get.ts',
|
||||||
|
params: [
|
||||||
|
{ name: 'ids', kind: 'collection', max: 400 },
|
||||||
|
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
||||||
|
{ name: 'limit', kind: 'number', max: 1000 },
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
const createSubscription: Operation = {
|
||||||
|
name: 'createSubscription',
|
||||||
|
where: 'server/api/billing/subscribe.post.ts',
|
||||||
|
idempotent: true,
|
||||||
|
mutates: true,
|
||||||
|
params: [{ name: 'priceId', kind: 'identifier', required: true }],
|
||||||
|
}
|
||||||
|
|
||||||
|
const ids = (op: Operation) => generateCases([op]).map(c => c.id)
|
||||||
|
|
||||||
|
describe('the cases come from real defect classes', () => {
|
||||||
|
it('derives the empty-collection case — the most productive input in the set', () => {
|
||||||
|
// [].every(...) is true, so an empty collection satisfies every
|
||||||
|
// universal assertion made about it.
|
||||||
|
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/empty')!
|
||||||
|
expect(c.expectation).toContain('never a pass')
|
||||||
|
expect(c.rationale).toContain('[].every(...) is true')
|
||||||
|
expect(c.severity).toBe('high')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives the boundary+1 case from a documented maximum', () => {
|
||||||
|
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/over-max')!
|
||||||
|
expect(c.input).toContain('401 items')
|
||||||
|
expect(c.expectation).toContain('never silently truncates')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives the wrong-tenant case only for tenant-scoped identifiers', () => {
|
||||||
|
expect(ids(listContacts)).toContain('listContacts/orgId/wrong-tenant')
|
||||||
|
expect(ids(createSubscription)).not.toContain('createSubscription/priceId/wrong-tenant')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rates the wrong-tenant case as abuse, the highest severity', () => {
|
||||||
|
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/orgId/wrong-tenant')!
|
||||||
|
expect(c.category).toBe('abuse')
|
||||||
|
expect(c.severity).toBe('critical')
|
||||||
|
expect(c.expectation).toContain('indistinguishable from the not-found case')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives the null-vs-zero case for numbers', () => {
|
||||||
|
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/limit/zero')!
|
||||||
|
expect(c.expectation).toContain('Zero samples is not 100%')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives the absent-optional case — the silently dropped expansion', () => {
|
||||||
|
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/absent')!
|
||||||
|
expect(c.rationale).toContain('silently dropped a requested expansion')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives replay cases only for idempotent operations', () => {
|
||||||
|
expect(ids(createSubscription)).toContain('createSubscription/replay/identical')
|
||||||
|
expect(ids(createSubscription)).toContain('createSubscription/replay/changed-params')
|
||||||
|
expect(ids(listContacts)).not.toContain('listContacts/replay/identical')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('derives concurrency and partial-failure cases only for mutating operations', () => {
|
||||||
|
expect(ids(createSubscription)).toContain('createSubscription/concurrent/double-submit')
|
||||||
|
expect(ids(createSubscription)).toContain('createSubscription/partial-failure')
|
||||||
|
expect(ids(listContacts)).not.toContain('listContacts/concurrent/double-submit')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('writes expectations as requirements, not hopes', () => {
|
||||||
|
for (const c of generateCases([listContacts, createSubscription])) {
|
||||||
|
expect(c.expectation.length).toBeGreaterThan(10)
|
||||||
|
expect(c.rationale.length).toBeGreaterThan(10)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('an uncovered case is an absence of a check, not a failure', () => {
|
||||||
|
it('reports uncovered cases as not-assessed, never fail', () => {
|
||||||
|
const r = assessCoverage([listContacts])
|
||||||
|
expect(r.findings.every(f => f.determination === 'not-assessed')).toBe(true)
|
||||||
|
expect(overall(r)).toBe('not-assessed')
|
||||||
|
expect(formatSuite([listContacts])).toContain('cannot distinguish from working')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('passes a case that has a test', () => {
|
||||||
|
const all = generateCases([listContacts]).map(c => c.id)
|
||||||
|
const r = assessCoverage([listContacts], { covered: all })
|
||||||
|
expect(overall(r)).toBe('pass')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('counts the gap', () => {
|
||||||
|
const all = generateCases([listContacts]).map(c => c.id)
|
||||||
|
const text = formatSuite([listContacts], { covered: all.slice(0, 2) })
|
||||||
|
expect(text).toContain(`2 of ${all.length} implied cases covered`)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('cites the operation definition on every case', () => {
|
||||||
|
expect(formatSuite([listContacts], { covered: [] }, { evidence: 'full' }))
|
||||||
|
.toContain('server/api/contacts/index.get.ts')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('conformance with the evidence-record standard', () => {
|
||||||
|
it('no operations described is not-assessed, never clean', () => {
|
||||||
|
expect(overall(assessCoverage([]))).toBe('not-assessed')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('every conclusion carries a citation', () => {
|
||||||
|
expect(validateReport(assessCoverage([listContacts, createSubscription]))).toEqual([])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('produces unique case ids', () => {
|
||||||
|
const all = generateCases([listContacts, createSubscription]).map(c => c.id)
|
||||||
|
expect(new Set(all).size).toBe(all.length)
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"extends": "./tsconfig.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"module": "CommonJS",
|
||||||
|
"moduleResolution": "Node",
|
||||||
|
"outDir": "dist-cjs",
|
||||||
|
"declaration": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022", "module": "ES2022", "moduleResolution": "bundler",
|
||||||
|
"declaration": true, "outDir": "dist", "rootDir": "src",
|
||||||
|
"strict": true, "skipLibCheck": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts"]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user