AdversarialSuite

Derives the edge, misuse and abuse cases an interface implies, and reports which of them have no test.

What it does and why

A test suite can pass while missing the inputs that break the code. The failure this library targets is not a failing test. It is a green suite with a hole in it.

AdversarialSuite does not run tests. You describe your operations and their parameters, and it derives the cases those parameters imply. Then it compares them with the list of case ids your tests already cover. It has no test runner and makes no network calls.

Each generator comes from a class of defect that shows up in real systems:

Generator	Defect class
Empty collection	[].every(...) is true, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified.
Single item	One member cannot disagree with itself, so a comparison over one element always succeeds.
Absent or null optional field	An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken.
Boundary + 1	A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them.
Zero vs absent	Zero samples is not 100%. A zero denominator produces a flattering default.
Empty string, Unicode, over-length string	Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success.
Malformed and nonexistent identifiers	Error text that tells "absent" apart from "not yours" becomes an enumeration oracle.
Wrong-tenant identifier	The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through.
Replayed idempotency key	A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request.
Double submit, partial failure	Two identical writes both take effect, or a half-applied write reports success.

Setting tenantScoped: true on an identifier produces the wrong-tenant abuse case. idempotent: true produces the replay cases. mutates: true produces double-submit and partial-failure.

Uncovered is not failed

An uncovered case is reported as NOT ASSESSED, never as a failure. Nothing has been shown to be broken. What has been shown is that nobody checked, and that is the state a passing suite cannot tell apart from working code.

Reports follow the @extant2000/evidence-record format: every finding cites the file where the operation is defined, and an empty description of operations is reported as not assessed instead of clean.

Usage

import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'

const operations: Operation[] = [{
  name: 'listContacts',
  where: 'src/api/contacts.ts',
  params: [
    { name: 'ids', kind: 'collection', max: 400 },
    { name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
  ],
}]

// Every case the interface implies, with input, expectation and rationale.
const cases = generateCases(operations)

// Compare against the case ids your tests already cover.
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })

// Or render it for a terminal or CI log.
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))

Parameter kinds are collection, string, number, boolean, identifier and optional. max sets the documented limit that the boundary cases use.

Output of the example above, shortened (the header line is omitted):

  NOT ASSESSED  ·  8 items examined  ·  8 findings
  1 of 8 implied cases covered; 7 with no test found
  An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.

✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant   [NOT ASSESSED]
             Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
✗ HIGH     misuse: ids with 401 items   [NOT ASSESSED]
◦ info     edge: ids = []   [MET]

Install

The package is published to the GitLab package registry, so point the @extant2000 scope there first:

npm config set @extant2000:registry https://gitlab.com/api/v4/packages/npm/
npm install @extant2000/adversarial-suite

License

MIT. See LICENSE.