First public release

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-28 14:49:00 -04:00
co-authored by Claude Opus 5.5
commit a8226bd889
14 changed files with 2166 additions and 0 deletions
+95
View File
@@ -0,0 +1,95 @@
# AdversarialSuite
Derives the edge, misuse and abuse cases an interface implies, and reports
which of them have no test.
## What it does and why
A test suite can pass while missing the inputs that break the code. The
failure this library targets is not a failing test. It is a green suite with a
hole in it.
AdversarialSuite does not run tests. You describe your operations and their
parameters, and it derives the cases those parameters imply. Then it compares
them with the list of case ids your tests already cover. It has no test runner
and makes no network calls.
Each generator comes from a class of defect that shows up in real systems:
| Generator | Defect class |
|---|---|
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
case. `idempotent: true` produces the replay cases. `mutates: true` produces
double-submit and partial-failure.
## Uncovered is not failed
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
has been shown to be broken. What has been shown is that nobody checked, and
that is the state a passing suite cannot tell apart from working code.
Reports follow the `@extant2000/evidence-record` format: every finding cites
the file where the operation is defined, and an empty description of
operations is reported as not assessed instead of clean.
## Usage
```ts
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
const operations: Operation[] = [{
name: 'listContacts',
where: 'src/api/contacts.ts',
params: [
{ name: 'ids', kind: 'collection', max: 400 },
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
],
}]
// Every case the interface implies, with input, expectation and rationale.
const cases = generateCases(operations)
// Compare against the case ids your tests already cover.
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
// Or render it for a terminal or CI log.
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
```
Parameter kinds are `collection`, `string`, `number`, `boolean`,
`identifier` and `optional`. `max` sets the documented limit that the
boundary cases use.
Output of the example above, shortened (the header line is omitted):
```
NOT ASSESSED · 8 items examined · 8 findings
1 of 8 implied cases covered; 7 with no test found
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
◦ info edge: ids = [] [MET]
```
## Install
```
npm install @extant2000/adversarial-suite
```
## License
MIT. See [LICENSE](LICENSE).