First public release

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-28 14:49:00 -04:00
co-authored by Claude Opus 5.5
commit a8226bd889
14 changed files with 2166 additions and 0 deletions
+12
View File
@@ -0,0 +1,12 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 2
[*.md]
trim_trailing_whitespace = false
+8
View File
@@ -0,0 +1,8 @@
node_modules/
dist/
*.log
.DS_Store
.env*
!.env.example
.npmrc
dist-cjs/
+6
View File
@@ -0,0 +1,6 @@
stages: [test]
test:
stage: test
image: node:22-alpine
script: [npm ci, npm run build, npm test]
+14
View File
@@ -0,0 +1,14 @@
# Changelog
## 0.1.0 - 2026-09-28
First public release under MIT.
- `generateCases()` derives edge, misuse and abuse cases from an operation's
parameters: empty and single-item collections, absent and null fields,
boundary + 1, zero vs absent, string edge cases, malformed, nonexistent and
wrong-tenant identifiers, idempotency replays, double submit and partial
failure.
- `assessCoverage()` compares the derived cases with the case ids your tests
cover. An uncovered case is reported as not assessed, never as a failure.
- `formatSuite()` renders the report for a terminal or CI log.
+22
View File
@@ -0,0 +1,22 @@
# Contributing
Issues and merge requests are welcome at
https://gitlab.com/extant2000/adversarial-suite.
## Ground rules
- **A test that cannot fail proves nothing.** If you fix a bug, add a test
that fails without your fix, and check that it does fail before you submit.
- Measure, do not assume. Two modules with the same line count can be
different programs.
- Keep dependencies minimal. Every new dependency needs a reason.
- Explain why in comments, not what.
## Before you open a merge request
Run both of these and make sure they pass:
```
npm run build
npm test
```
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Extant 2000 LLC
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+95
View File
@@ -0,0 +1,95 @@
# AdversarialSuite
Derives the edge, misuse and abuse cases an interface implies, and reports
which of them have no test.
## What it does and why
A test suite can pass while missing the inputs that break the code. The
failure this library targets is not a failing test. It is a green suite with a
hole in it.
AdversarialSuite does not run tests. You describe your operations and their
parameters, and it derives the cases those parameters imply. Then it compares
them with the list of case ids your tests already cover. It has no test runner
and makes no network calls.
Each generator comes from a class of defect that shows up in real systems:
| Generator | Defect class |
|---|---|
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
case. `idempotent: true` produces the replay cases. `mutates: true` produces
double-submit and partial-failure.
## Uncovered is not failed
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
has been shown to be broken. What has been shown is that nobody checked, and
that is the state a passing suite cannot tell apart from working code.
Reports follow the `@extant2000/evidence-record` format: every finding cites
the file where the operation is defined, and an empty description of
operations is reported as not assessed instead of clean.
## Usage
```ts
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
const operations: Operation[] = [{
name: 'listContacts',
where: 'src/api/contacts.ts',
params: [
{ name: 'ids', kind: 'collection', max: 400 },
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
],
}]
// Every case the interface implies, with input, expectation and rationale.
const cases = generateCases(operations)
// Compare against the case ids your tests already cover.
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
// Or render it for a terminal or CI log.
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
```
Parameter kinds are `collection`, `string`, `number`, `boolean`,
`identifier` and `optional`. `max` sets the documented limit that the
boundary cases use.
Output of the example above, shortened (the header line is omitted):
```
NOT ASSESSED · 8 items examined · 8 findings
1 of 8 implied cases covered; 7 with no test found
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
◦ info edge: ids = [] [MET]
```
## Install
```
npm install @extant2000/adversarial-suite
```
## License
MIT. See [LICENSE](LICENSE).
+6
View File
@@ -0,0 +1,6 @@
# Security
Please report vulnerabilities privately by email to security@extant2000.com.
Do not open a public issue for a security problem.
We aim to acknowledge every report within 5 business days.
+1503
View File
File diff suppressed because it is too large Load Diff
+46
View File
@@ -0,0 +1,46 @@
{
"name": "@extant2000/adversarial-suite",
"version": "0.1.0",
"private": false,
"description": "Derives the edge, misuse and abuse cases an interface implies and reports which have no test.",
"license": "MIT",
"type": "module",
"main": "./dist-cjs/index.js",
"files": [
"dist",
"dist-cjs",
"src",
"README.md",
"LICENSE"
],
"repository": {
"type": "git",
"url": "https://gitlab.com/extant2000/adversarial-suite.git"
},
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js",
"require": "./dist-cjs/index.js"
}
},
"scripts": {
"build": "tsc && tsc -p tsconfig.cjs.json && node -e \"require('fs').writeFileSync('dist-cjs/package.json', JSON.stringify({type:'commonjs'}))\"",
"test": "vitest run",
"prepublishOnly": "npm run build"
},
"devDependencies": {
"typescript": "^5.6.0",
"vitest": "^2.1.0"
},
"dependencies": {
"@extant2000/evidence-record": "^0.1.2"
},
"author": "Extant 2000 LLC",
"homepage": "https://gitlab.com/extant2000/adversarial-suite",
"bugs": {
"url": "https://gitlab.com/extant2000/adversarial-suite/-/issues"
}
}
+294
View File
@@ -0,0 +1,294 @@
/**
* AdversarialSuite — generate the edge, misuse and abuse cases a happy-path
* test suite does not contain, and report which of them nobody has covered.
*
* This library does not run tests. It derives the cases an interface implies
* and checks them against the cases that exist, because the failure it
* targets is not a failing test — it is a passing suite with a hole in it.
* Every generator below comes from a class of defect seen in real systems:
*
* - **Empty collection.** `[].every(...)` is `true`, so a requirement with
* zero verifications reported as verified, and a gate with zero matching
* files reported as passing. The empty case is the single most productive
* input in this whole set.
* - **Absent optional field.** A provider silently dropped a requested
* expansion; the response stayed valid and the field was simply missing.
* Nothing asserted its presence.
* - **Boundary + 1.** A URL filter died past roughly 400 identifiers and a
* result set capped silently at 1000 rows. Both are fine at the sizes
* anyone tests with.
* - **Null-vs-zero.** Zero samples is not 100%. A null composite became a
* truthy all-null object.
* - **Repeated identical call.** An idempotency key derived from a literal
* replayed a deleted object for 24 hours.
* - **Wrong-tenant identifier.** A well-formed id belonging to somebody
* else is the authorisation case that syntactic validation waves through.
*
* Pure: no test runner, no network.
*/
import {
type CapabilityReport,
type Finding,
type FormatOptions,
type Severity,
evidence,
formatReport as renderReport,
} from '@extant2000/evidence-record'
export type ParamKind =
| 'collection'
| 'string'
| 'number'
| 'boolean'
| 'identifier'
| 'optional'
export interface Param {
name: string
kind: ParamKind
required?: boolean
/** Documented maximum — length, count, or value. Boundary cases use it. */
max?: number
/** True when the value is scoped to a tenant, user or organisation. */
tenantScoped?: boolean
}
export interface Operation {
name: string
params: Param[]
/** True when repeating the call must not repeat its effect. */
idempotent?: boolean
/** True when the operation writes. Abuse cases weigh heavier. */
mutates?: boolean
/** Where the operation is defined — cited on every generated case. */
where: string
}
export type CaseCategory = 'edge' | 'misuse' | 'abuse'
export interface AdversarialCase {
id: string
operation: string
category: CaseCategory
/** What to do. */
input: string
/** What must happen. Written as a requirement, not a hope. */
expectation: string
/** Why this case exists — the defect class it comes from. */
rationale: string
severity: Severity
}
const CAT_SEVERITY: Record<CaseCategory, Severity> = {
edge: 'medium',
misuse: 'high',
abuse: 'critical',
}
function mk(
op: Operation,
suffix: string,
category: CaseCategory,
input: string,
expectation: string,
rationale: string,
severity?: Severity,
): AdversarialCase {
return {
id: `${op.name}/${suffix}`,
operation: op.name,
category,
input,
expectation,
rationale,
severity: severity ?? CAT_SEVERITY[category],
}
}
function casesForParam(op: Operation, p: Param): AdversarialCase[] {
const out: AdversarialCase[] = []
if (p.kind === 'collection') {
// The single most productive input in this set.
out.push(mk(op, `${p.name}/empty`, 'edge',
`${p.name} = []`,
'Reports "nothing was checked", never a pass.',
'[].every(...) is true, so an empty collection satisfies every universal assertion made about it. A requirement with zero verifications reported as verified this way.',
'high'))
out.push(mk(op, `${p.name}/single`, 'edge',
`${p.name} = [one item]`,
'Reports that a comparison needs at least two members, rather than a clean result.',
'One member cannot disagree with itself, so a comparison over a single element trivially succeeds.'))
if (p.max) {
out.push(mk(op, `${p.name}/over-max`, 'misuse',
`${p.name} with ${p.max + 1} items`,
'Rejects, paginates or chunks — never silently truncates.',
`Documented maximum is ${p.max}. A URL filter can fail past a few hundred identifiers, and a result set can be capped silently at 1000 rows; both behave correctly at the sizes anyone tests with.`))
}
}
if (p.kind === 'number') {
out.push(mk(op, `${p.name}/zero`, 'edge',
`${p.name} = 0`,
'Distinguishes zero from absent. Zero samples is not 100%.',
'A zero denominator produced a flattering default rather than an abstention.',
'high'))
out.push(mk(op, `${p.name}/negative`, 'misuse',
`${p.name} = -1`, 'Rejects rather than coercing.',
'Negative values reach arithmetic that assumed non-negative input.'))
if (p.max !== undefined) {
out.push(mk(op, `${p.name}/over-max`, 'misuse',
`${p.name} = ${p.max + 1}`, `Rejects above the documented maximum of ${p.max}.`,
'Off-by-one at a documented boundary.'))
}
}
if (p.kind === 'optional' || !p.required) {
out.push(mk(op, `${p.name}/absent`, 'edge',
`${p.name} omitted entirely`,
'Behaves correctly, and does not report success for work it skipped.',
'A provider silently dropped a requested expansion. The response stayed structurally valid and the field was simply absent, so schema validation passed while the feature was dead.',
'high'))
out.push(mk(op, `${p.name}/null`, 'edge',
`${p.name} = null`,
'Treats null as absent, not as a present empty value.',
'A null composite deserialised into a truthy object with all-null fields, which passed every presence check made on it.'))
}
if (p.kind === 'string') {
out.push(mk(op, `${p.name}/empty-string`, 'edge',
`${p.name} = ""`, 'Distinguishes empty from absent.',
'An empty string is falsy and present at the same time.'))
out.push(mk(op, `${p.name}/unicode`, 'misuse',
`${p.name} with combining marks, RTL overrides and a 4-byte emoji`,
'Stores and returns the value unchanged; length limits count characters consistently.',
'Byte-length limits truncate mid-codepoint.'))
if (p.max) {
out.push(mk(op, `${p.name}/over-max`, 'misuse',
`${p.name} of length ${p.max + 1}`, `Rejects above ${p.max} rather than truncating silently.`,
'Silent truncation is data loss that looks like success.'))
}
}
if (p.kind === 'identifier') {
out.push(mk(op, `${p.name}/malformed`, 'misuse',
`${p.name} = "not-an-id"`, 'Rejects with a clear error.',
'Syntactic validation is the easy half.'))
out.push(mk(op, `${p.name}/nonexistent`, 'misuse',
`${p.name} = a well-formed id that does not exist`,
'Returns not-found, and does not leak whether the id exists elsewhere.',
'Distinguishing "absent" from "not yours" through error text is an enumeration oracle.'))
if (p.tenantScoped) {
out.push(mk(op, `${p.name}/wrong-tenant`, 'abuse',
`${p.name} = a well-formed id belonging to another tenant`,
'Denies. The response is indistinguishable from the not-found case.',
'The authorisation case syntactic validation waves through: the id is valid, it exists, and it is not yours. This is the check a shared-table suite exists to enforce.'))
}
}
return out
}
function operationCases(op: Operation): AdversarialCase[] {
const out = op.params.flatMap(p => casesForParam(op, p))
if (op.idempotent) {
out.push(mk(op, 'replay/identical', 'edge',
'The same call twice with the same idempotency key',
'The second call returns the first result. It does not repeat the effect, and it does not return a deleted object.',
'A key derived from a literal rather than from the parameters replayed a stale — and in one case deleted — object for 24 hours.',
'high'))
out.push(mk(op, 'replay/changed-params', 'misuse',
'The same idempotency key with different parameters',
'Rejects. It does not silently serve the first response for a different request.',
'A literal key with changed parameters failed in-flight requests for a full day.'))
}
if (op.mutates) {
out.push(mk(op, 'concurrent/double-submit', 'abuse',
'Two identical writes issued simultaneously',
'Exactly one takes effect, or both are rejected. Never two.',
'A double submit is the cheapest abuse there is and the least often tested.'))
out.push(mk(op, 'partial-failure', 'edge',
'A failure injected midway through a multi-step write',
'Leaves no partially-applied state, and reports the failure rather than a success.',
'A partially applied write reports success and is discovered later by its inconsistency.',
'high'))
}
return out
}
/** Derive the adversarial cases an operation implies. */
export function generateCases(operations: Operation[]): AdversarialCase[] {
return (operations ?? []).flatMap(operationCases)
}
export interface Coverage {
/** Case ids that already have a test. */
covered: Iterable<string>
}
/**
* Report which implied cases are covered.
*
* An uncovered case is `not-assessed`, not `fail`: nothing has been shown to
* be broken. What has been shown is that nobody looked — which is the state
* a passing suite is least able to distinguish from working.
*/
export function assessCoverage(
operations: Operation[],
coverage: Coverage = { covered: [] },
): CapabilityReport {
const cases = generateCases(operations)
const covered = new Set(coverage.covered)
if (cases.length === 0) {
return {
capability: 'AdversarialSuite',
scope: 'no operations described',
examined: 0,
findings: [],
notes: ['No operations were described, so no cases were derived.'],
}
}
const findings: Finding[] = cases.map(c => {
const isCovered = covered.has(c.id)
const op = operations.find(o => o.name === c.operation)!
return {
id: c.id,
summary: `${c.category}: ${c.input}`,
determination: isCovered ? 'pass' : 'not-assessed',
severity: isCovered ? 'info' : c.severity,
detail: isCovered ? undefined : `Expected behaviour: ${c.expectation} — ${c.rationale}`,
evidence: [evidence.file(op.where, undefined, c.operation,
isCovered ? 'case covered' : 'case implied by the interface, no test found')],
}
})
const gap = cases.length - cases.filter(c => covered.has(c.id)).length
return {
capability: 'AdversarialSuite',
scope: `${cases.length} case(s) implied by ${operations.length} operation(s)`,
examined: cases.length,
findings,
notes: [
`${cases.length - gap} of ${cases.length} implied cases covered; ${gap} with no test found`,
'An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.',
],
}
}
/** Render the derived suite for a terminal, CI log or test plan. */
export function formatSuite(
operations: Operation[],
coverage: Coverage = { covered: [] },
opts: FormatOptions = {},
): string {
return renderReport(assessCoverage(operations, coverage), opts)
}
+122
View File
@@ -0,0 +1,122 @@
import { describe, expect, it } from 'vitest'
import { overall, validateReport } from '@extant2000/evidence-record'
import { assessCoverage, formatSuite, generateCases, type Operation } from '../src/index.js'
const listContacts: Operation = {
name: 'listContacts',
where: 'server/api/contacts/index.get.ts',
params: [
{ name: 'ids', kind: 'collection', max: 400 },
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
{ name: 'limit', kind: 'number', max: 1000 },
],
}
const createSubscription: Operation = {
name: 'createSubscription',
where: 'server/api/billing/subscribe.post.ts',
idempotent: true,
mutates: true,
params: [{ name: 'priceId', kind: 'identifier', required: true }],
}
const ids = (op: Operation) => generateCases([op]).map(c => c.id)
describe('the cases come from real defect classes', () => {
it('derives the empty-collection case — the most productive input in the set', () => {
// [].every(...) is true, so an empty collection satisfies every
// universal assertion made about it.
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/empty')!
expect(c.expectation).toContain('never a pass')
expect(c.rationale).toContain('[].every(...) is true')
expect(c.severity).toBe('high')
})
it('derives the boundary+1 case from a documented maximum', () => {
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/over-max')!
expect(c.input).toContain('401 items')
expect(c.expectation).toContain('never silently truncates')
})
it('derives the wrong-tenant case only for tenant-scoped identifiers', () => {
expect(ids(listContacts)).toContain('listContacts/orgId/wrong-tenant')
expect(ids(createSubscription)).not.toContain('createSubscription/priceId/wrong-tenant')
})
it('rates the wrong-tenant case as abuse, the highest severity', () => {
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/orgId/wrong-tenant')!
expect(c.category).toBe('abuse')
expect(c.severity).toBe('critical')
expect(c.expectation).toContain('indistinguishable from the not-found case')
})
it('derives the null-vs-zero case for numbers', () => {
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/limit/zero')!
expect(c.expectation).toContain('Zero samples is not 100%')
})
it('derives the absent-optional case — the silently dropped expansion', () => {
const c = generateCases([listContacts]).find(c => c.id === 'listContacts/ids/absent')!
expect(c.rationale).toContain('silently dropped a requested expansion')
})
it('derives replay cases only for idempotent operations', () => {
expect(ids(createSubscription)).toContain('createSubscription/replay/identical')
expect(ids(createSubscription)).toContain('createSubscription/replay/changed-params')
expect(ids(listContacts)).not.toContain('listContacts/replay/identical')
})
it('derives concurrency and partial-failure cases only for mutating operations', () => {
expect(ids(createSubscription)).toContain('createSubscription/concurrent/double-submit')
expect(ids(createSubscription)).toContain('createSubscription/partial-failure')
expect(ids(listContacts)).not.toContain('listContacts/concurrent/double-submit')
})
it('writes expectations as requirements, not hopes', () => {
for (const c of generateCases([listContacts, createSubscription])) {
expect(c.expectation.length).toBeGreaterThan(10)
expect(c.rationale.length).toBeGreaterThan(10)
}
})
})
describe('an uncovered case is an absence of a check, not a failure', () => {
it('reports uncovered cases as not-assessed, never fail', () => {
const r = assessCoverage([listContacts])
expect(r.findings.every(f => f.determination === 'not-assessed')).toBe(true)
expect(overall(r)).toBe('not-assessed')
expect(formatSuite([listContacts])).toContain('cannot distinguish from working')
})
it('passes a case that has a test', () => {
const all = generateCases([listContacts]).map(c => c.id)
const r = assessCoverage([listContacts], { covered: all })
expect(overall(r)).toBe('pass')
})
it('counts the gap', () => {
const all = generateCases([listContacts]).map(c => c.id)
const text = formatSuite([listContacts], { covered: all.slice(0, 2) })
expect(text).toContain(`2 of ${all.length} implied cases covered`)
})
it('cites the operation definition on every case', () => {
expect(formatSuite([listContacts], { covered: [] }, { evidence: 'full' }))
.toContain('server/api/contacts/index.get.ts')
})
})
describe('conformance with the evidence-record standard', () => {
it('no operations described is not-assessed, never clean', () => {
expect(overall(assessCoverage([]))).toBe('not-assessed')
})
it('every conclusion carries a citation', () => {
expect(validateReport(assessCoverage([listContacts, createSubscription]))).toEqual([])
})
it('produces unique case ids', () => {
const all = generateCases([listContacts, createSubscription]).map(c => c.id)
expect(new Set(all).size).toBe(all.length)
})
})
+9
View File
@@ -0,0 +1,9 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"module": "CommonJS",
"moduleResolution": "Node",
"outDir": "dist-cjs",
"declaration": false
}
}
+8
View File
@@ -0,0 +1,8 @@
{
"compilerOptions": {
"target": "ES2022", "module": "ES2022", "moduleResolution": "bundler",
"declaration": true, "outDir": "dist", "rootDir": "src",
"strict": true, "skipLibCheck": true
},
"include": ["src/**/*.ts"]
}