First public release
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# AdversarialSuite
|
||||
|
||||
Derives the edge, misuse and abuse cases an interface implies, and reports
|
||||
which of them have no test.
|
||||
|
||||
## What it does and why
|
||||
|
||||
A test suite can pass while missing the inputs that break the code. The
|
||||
failure this library targets is not a failing test. It is a green suite with a
|
||||
hole in it.
|
||||
|
||||
AdversarialSuite does not run tests. You describe your operations and their
|
||||
parameters, and it derives the cases those parameters imply. Then it compares
|
||||
them with the list of case ids your tests already cover. It has no test runner
|
||||
and makes no network calls.
|
||||
|
||||
Each generator comes from a class of defect that shows up in real systems:
|
||||
|
||||
| Generator | Defect class |
|
||||
|---|---|
|
||||
| Empty collection | `[].every(...)` is `true`, so an empty list satisfies every universal check. A requirement with zero verifications can report as verified. |
|
||||
| Single item | One member cannot disagree with itself, so a comparison over one element always succeeds. |
|
||||
| Absent or null optional field | An upstream service drops a field. The response is still structurally valid, so schema validation passes while the feature is broken. |
|
||||
| Boundary + 1 | A filter or result set is capped at some size. It works at the sizes people test with and truncates without warning above them. |
|
||||
| Zero vs absent | Zero samples is not 100%. A zero denominator produces a flattering default. |
|
||||
| Empty string, Unicode, over-length string | Empty is falsy and present at once; byte limits cut through a codepoint; silent truncation is data loss that looks like success. |
|
||||
| Malformed and nonexistent identifiers | Error text that tells "absent" apart from "not yours" becomes an enumeration oracle. |
|
||||
| Wrong-tenant identifier | The id is well formed, it exists, and it belongs to someone else. Syntactic validation lets it through. |
|
||||
| Replayed idempotency key | A key derived from a constant can replay a stale or deleted object, or serve an old response for a different request. |
|
||||
| Double submit, partial failure | Two identical writes both take effect, or a half-applied write reports success. |
|
||||
|
||||
Setting `tenantScoped: true` on an identifier produces the wrong-tenant abuse
|
||||
case. `idempotent: true` produces the replay cases. `mutates: true` produces
|
||||
double-submit and partial-failure.
|
||||
|
||||
## Uncovered is not failed
|
||||
|
||||
An uncovered case is reported as **NOT ASSESSED**, never as a failure. Nothing
|
||||
has been shown to be broken. What has been shown is that nobody checked, and
|
||||
that is the state a passing suite cannot tell apart from working code.
|
||||
|
||||
Reports follow the `@extant2000/evidence-record` format: every finding cites
|
||||
the file where the operation is defined, and an empty description of
|
||||
operations is reported as not assessed instead of clean.
|
||||
|
||||
## Usage
|
||||
|
||||
```ts
|
||||
import { generateCases, assessCoverage, formatSuite, type Operation } from '@extant2000/adversarial-suite'
|
||||
|
||||
const operations: Operation[] = [{
|
||||
name: 'listContacts',
|
||||
where: 'src/api/contacts.ts',
|
||||
params: [
|
||||
{ name: 'ids', kind: 'collection', max: 400 },
|
||||
{ name: 'orgId', kind: 'identifier', required: true, tenantScoped: true },
|
||||
],
|
||||
}]
|
||||
|
||||
// Every case the interface implies, with input, expectation and rationale.
|
||||
const cases = generateCases(operations)
|
||||
|
||||
// Compare against the case ids your tests already cover.
|
||||
const report = assessCoverage(operations, { covered: ['listContacts/ids/empty'] })
|
||||
|
||||
// Or render it for a terminal or CI log.
|
||||
console.log(formatSuite(operations, { covered: ['listContacts/ids/empty'] }))
|
||||
```
|
||||
|
||||
Parameter kinds are `collection`, `string`, `number`, `boolean`,
|
||||
`identifier` and `optional`. `max` sets the documented limit that the
|
||||
boundary cases use.
|
||||
|
||||
Output of the example above, shortened (the header line is omitted):
|
||||
|
||||
```
|
||||
NOT ASSESSED · 8 items examined · 8 findings
|
||||
1 of 8 implied cases covered; 7 with no test found
|
||||
An uncovered case is not a failure. It is the absence of a check, which a passing suite cannot distinguish from working.
|
||||
|
||||
✗ CRITICAL abuse: orgId = a well-formed id belonging to another tenant [NOT ASSESSED]
|
||||
Expected behaviour: Denies. The response is indistinguishable from the not-found case. ...
|
||||
✗ HIGH misuse: ids with 401 items [NOT ASSESSED]
|
||||
◦ info edge: ids = [] [MET]
|
||||
```
|
||||
|
||||
## Install
|
||||
|
||||
```
|
||||
npm install @extant2000/adversarial-suite
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT. See [LICENSE](LICENSE).
|
||||
Reference in New Issue
Block a user