96 lines
4.2 KiB
Markdown
96 lines
4.2 KiB
Markdown
# AdversarialSuite
|
|
|
|
Derives the edge, misuse and abuse cases an interface implies, and reports
|
|
which of them have no test.
|
|
|
|
## What it does and why
|
|
|
|
A test suite can pass while missing the inputs that break the code. The
|
|
failure this library targets is not a failing test. It is a green suite with a
|
|
hole in it.
|
|
|
|
AdversarialSuite does not run tests. You describe your operations and their
|
|
parameters, and it derives the cases those parameters imply. Then it compares
|
|
them with the list of case ids your tests already cover. It has no test runner
|
|
and makes no network calls.
|
|
|
|
Each generator comes from a class of defect that shows up in real systems:
|
|
|
|
| Generator | Defect class |
|
|
|---|---|
|
|
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
|
|
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
|
|
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
|
|
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
|
|
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
|
|
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
|
|
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
|
|
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
|
|
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
|
|
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
|
|
|
|
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
|
|
case. `idempotent: true` produces the replay cases. `mutates: true` produces
|
|
double-submit and partial-failure.
|
|
|
|
## Uncovered is not failed
|
|
|
|
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
|
|
has been shown to be broken. What has been shown is that nobody checked, and
|
|
that is the state a passing suite cannot tell apart from working code.
|
|
|
|
Reports follow the `@extant2000/evidence-record` format: every finding cites
|
|
the file where the operation is defined, and an empty description of
|
|
operations is reported as not assessed instead of clean.
|
|
|
|
## Usage
|
|
|
|
```ts
|
|
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
|
|
|
|
const operations: Operation[] = [{
|
|
name: 'listContacts',
|
|
where: 'src/api/contacts.ts',
|
|
params: [
|
|
{ name: 'ids', kind: 'collection', max: 400 },
|
|
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
|
],
|
|
}]
|
|
|
|
// Every case the interface implies, with input, expectation and rationale.
|
|
const cases = generateCases(operations)
|
|
|
|
// Compare against the case ids your tests already cover.
|
|
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
|
|
|
|
// Or render it for a terminal or CI log.
|
|
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
|
|
```
|
|
|
|
Parameter kinds are `collection`, `string`, `number`, `boolean`,
|
|
`identifier` and `optional`. `max` sets the documented limit that the
|
|
boundary cases use.
|
|
|
|
Output of the example above, shortened (the header line is omitted):
|
|
|
|
```
|
|
NOT ASSESSED · 8 items examined · 8 findings
|
|
1 of 8 implied cases covered; 7 with no test found
|
|
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
|
|
|
|
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
|
|
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
|
|
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
|
|
◦ info edge: ids = [] [MET]
|
|
```
|
|
|
|
## Install
|
|
|
|
```
|
|
npm install @extant2000/adversarial-suite
|
|
```
|
|
|
|
## License
|
|
|
|
MIT. See [LICENSE](LICENSE).
|