First public release
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,12 @@
|
|||||||
|
root = true
|
||||||
|
|
||||||
|
[*]
|
||||||
|
charset = utf-8
|
||||||
|
end_of_line = lf
|
||||||
|
insert_final_newline = true
|
||||||
|
trim_trailing_whitespace = true
|
||||||
|
indent_style = space
|
||||||
|
indent_size = 2
|
||||||
|
|
||||||
|
[*.md]
|
||||||
|
trim_trailing_whitespace = false
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
|
.env*
|
||||||
|
!.env.example
|
||||||
|
.npmrc
|
||||||
|
dist-cjs/
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
stages: [test]
|
||||||
|
|
||||||
|
test:
|
||||||
|
stage: test
|
||||||
|
image: node:22-alpine
|
||||||
|
script: [npm ci, npm run build, npm test]
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.2.1 - 2026-09-28
|
||||||
|
|
||||||
|
First public release under MIT.
|
||||||
|
|
||||||
|
- `assertShape()` validates a payload with any zod-style parser and throws a
|
||||||
|
framework-independent `ContractDriftError`. Zod 4 symbol paths are handled.
|
||||||
|
- `expectFields()` and `assertFields()` check that requested fields actually
|
||||||
|
came back, treating `null` as present and `undefined` as absent.
|
||||||
|
- `coverage()` measures contract-check coverage against a floor and lists
|
||||||
|
the uncovered endpoints.
|
||||||
|
- `fieldReport()`, `coverageReport()`, `formatFields()` and
|
||||||
|
`formatCoverage()` produce `@extant2000/evidence-record` reports. An empty
|
||||||
|
surface is not assessed.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Contributing
|
||||||
|
|
||||||
|
Issues and merge requests are welcome at
|
||||||
|
https://gitlab.com/extant2000/contract-check.
|
||||||
|
|
||||||
|
## Ground rules
|
||||||
|
|
||||||
|
- **A test that cannot fail proves nothing.** If you fix a bug, add a test
|
||||||
|
that fails without your fix, and check that it does fail before you submit.
|
||||||
|
- Measure, do not assume. Two modules with the same line count can be
|
||||||
|
different programs.
|
||||||
|
- Keep dependencies minimal. Every new dependency needs a reason.
|
||||||
|
- Explain why in comments, not what.
|
||||||
|
|
||||||
|
## Before you open a merge request
|
||||||
|
|
||||||
|
Run both of these and make sure they pass:
|
||||||
|
|
||||||
|
```
|
||||||
|
npm run build
|
||||||
|
npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Extant 2000 LLC
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# ContractCheck
|
||||||
|
|
||||||
|
Catches interface drift between two systems: a response whose shape changed,
|
||||||
|
a requested field that never came back, and endpoints with no contract check.
|
||||||
|
It is used in Hitt Hosting products.
|
||||||
|
|
||||||
|
## What it does and why
|
||||||
|
|
||||||
|
**A response can be perfectly valid while the field you asked for is
|
||||||
|
missing.** Stripe removed `invoice.payment_intent` in its 2025-03-31 (Basil)
|
||||||
|
API version, and Stripe ignores an unknown `expand` path instead of rejecting
|
||||||
|
it. A stale expand keeps succeeding and returns a well-formed invoice without
|
||||||
|
the field. There is no error and no warning; the client secret is absent and
|
||||||
|
the payment form cannot mount, while typecheck, lint and the test suite all
|
||||||
|
pass. A schema validator cannot catch this, because the invoice is valid. The
|
||||||
|
only way to detect it is to check that what you requested actually came back.
|
||||||
|
|
||||||
|
ContractCheck has three parts.
|
||||||
|
|
||||||
|
- `assertShape()` validates a payload against any schema with a zod-style
|
||||||
|
`safeParse` and throws a `ContractDriftError` naming the context and the
|
||||||
|
offending paths. It fails loudly because the bug it targets is an endpoint
|
||||||
|
dropping a field the UI needs: nothing errors, the page renders, and the
|
||||||
|
number is just wrong. A 500 naming the path beats a plausible screen. The
|
||||||
|
error is a plain `Error`, not tied to a web framework, so you map it to your
|
||||||
|
own transport.
|
||||||
|
- `expectFields()` and `assertFields()` check that each requested dot path is
|
||||||
|
present in the response. An explicit `null` counts as present: `null` means
|
||||||
|
the provider answered "no value", while `undefined` means it never
|
||||||
|
acknowledged the request. A `why` note on each expectation is quoted in the
|
||||||
|
failure, so it says what breaks and not only what is missing.
|
||||||
|
- `coverage()` measures how much of an API surface is contract-checked,
|
||||||
|
against a floor you can raise over time so coverage cannot slip back. It
|
||||||
|
lists the uncovered endpoints, because a bare percentage tells you there is
|
||||||
|
a gap, not where it is.
|
||||||
|
|
||||||
|
The `Parser` type accepts Zod 4 schemas, whose issue paths may include
|
||||||
|
symbols, and symbol path segments are rendered without crashing.
|
||||||
|
|
||||||
|
## Reports
|
||||||
|
|
||||||
|
`fieldReport()` and `formatFields()` make each requested path its own
|
||||||
|
finding, so "the response is a valid object" and "the field you asked for is
|
||||||
|
absent" never collapse into one verdict. If no fields were asserted, the
|
||||||
|
report is not assessed.
|
||||||
|
|
||||||
|
`coverageReport()` and `formatCoverage()` report an empty surface as not
|
||||||
|
assessed, not 0%. "0% covered" for a surface with no endpoints describes a
|
||||||
|
failure that did not happen. Uncovered endpoints are listed one by one as not
|
||||||
|
assessed. Reports use the `@extant2000/evidence-record` format.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { assertShape, expectFields, assertFields, formatFields, coverage, formatCoverage } from '@extant2000/contract-check'
|
||||||
|
import { z } from 'zod' // or any parser with a zod-style safeParse
|
||||||
|
|
||||||
|
// 1. Response shape
|
||||||
|
const payload: unknown = { id: 'm1', name: 'Survey' } // e.g. a parsed response body
|
||||||
|
const Mission = z.object({ id: z.string(), name: z.string() })
|
||||||
|
const mission = assertShape(Mission, payload, 'GET /missions/:id')
|
||||||
|
|
||||||
|
// 2. Requested fields actually returned
|
||||||
|
const subscription = { id: 'sub_1', latest_invoice: { id: 'in_1' } } // response from the provider
|
||||||
|
const expectations = [
|
||||||
|
{ path: 'latest_invoice.payment_intent', why: 'the client secret is needed to mount the payment form' },
|
||||||
|
'id',
|
||||||
|
]
|
||||||
|
const result = expectFields(subscription, expectations)
|
||||||
|
console.log(formatFields(result, { context: 'stripe.subscriptions.create', expectations }))
|
||||||
|
assertFields(subscription, expectations, 'stripe.subscriptions.create') // or throw
|
||||||
|
|
||||||
|
// 3. Coverage against a floor
|
||||||
|
const c = coverage(['GET /a', 'GET /b', 'POST /c', 'GET /d'], ['GET /a', 'GET /b', 'POST /c'], 80)
|
||||||
|
console.log(formatCoverage(c, 80))
|
||||||
|
```
|
||||||
|
|
||||||
|
For a subscription whose invoice lacks `payment_intent`, the field report
|
||||||
|
fails at critical severity on `latest_invoice.payment_intent`, quotes the
|
||||||
|
`why` note, and passes `id`. The coverage example prints:
|
||||||
|
|
||||||
|
```
|
||||||
|
✗ HIGH 3 of 4 endpoints contract-checked (75%) [NOT MET]
|
||||||
|
Below the ratchet floor of 80%.
|
||||||
|
evidence: Contract coverage = 75% (4 samples)
|
||||||
|
· MEDIUM GET /d has no contract check [NOT ASSESSED]
|
||||||
|
```
|
||||||
|
|
||||||
|
ContractCheck does not depend on zod. Any object with a compatible
|
||||||
|
`safeParse` works.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```
|
||||||
|
npm install @extant2000/contract-check
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT. See [LICENSE](LICENSE).
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# Security
|
||||||
|
|
||||||
|
Please report vulnerabilities privately by email to security@extant2000.com.
|
||||||
|
Do not open a public issue for a security problem.
|
||||||
|
|
||||||
|
We aim to acknowledge every report within 5 business days.
|
||||||
Generated
+1504
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"name": "@extant2000/contract-check",
|
||||||
|
"version": "0.2.1",
|
||||||
|
"private": false,
|
||||||
|
"description": "Catches interface drift between two systems before deployment.",
|
||||||
|
"license": "MIT",
|
||||||
|
"type": "module",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"files": [
|
||||||
|
"dist",
|
||||||
|
"src",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
],
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "https://gitlab.com/extant2000/contract-check.git"
|
||||||
|
},
|
||||||
|
"types": "dist/index.d.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"import": "./dist/index.js"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc",
|
||||||
|
"test": "vitest run",
|
||||||
|
"prepublishOnly": "npm run build"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"typescript": "^5.6.0",
|
||||||
|
"vitest": "^2.1.0"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@extant2000/evidence-record": "^0.1.2"
|
||||||
|
},
|
||||||
|
"author": "Extant 2000 LLC",
|
||||||
|
"homepage": "https://gitlab.com/extant2000/contract-check",
|
||||||
|
"bugs": {
|
||||||
|
"url": "https://gitlab.com/extant2000/contract-check/-/issues"
|
||||||
|
}
|
||||||
|
}
|
||||||
+161
@@ -0,0 +1,161 @@
|
|||||||
|
/**
|
||||||
|
* Detect when two subsystems' interface assumptions drift.
|
||||||
|
*
|
||||||
|
* Two design points:
|
||||||
|
*
|
||||||
|
* 1. **Framework-agnostic.** A helper that throws a framework-specific HTTP
|
||||||
|
* error ties itself to that framework. This throws a plain
|
||||||
|
* `ContractDriftError` the caller maps to its own transport.
|
||||||
|
* 2. **Request contracts, not just response shapes.** A shape validator
|
||||||
|
* checks what came back. It cannot catch a provider that SILENTLY IGNORES
|
||||||
|
* part of the request — see `expectFields` below.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** Anything with a zod-style `safeParse`. Avoids depending on zod itself. */
|
||||||
|
export interface Parser<T> {
|
||||||
|
safeParse: (data: unknown) => { success: true, data: T } | {
|
||||||
|
success: false
|
||||||
|
error: {
|
||||||
|
/**
|
||||||
|
* Zod 4 widened issue paths to `PropertyKey[]`, which includes symbol.
|
||||||
|
* Narrowing to string|number here made a Zod 4 schema unassignable to
|
||||||
|
* this interface — and where typecheck gates a pipeline, that failure
|
||||||
|
* can block a deploy without anyone reading the log.
|
||||||
|
*/
|
||||||
|
issues: Array<{ path: ReadonlyArray<PropertyKey>, message: string }>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export class ContractDriftError extends Error {
|
||||||
|
readonly context: string
|
||||||
|
readonly issues: string[]
|
||||||
|
constructor(context: string, issues: string[]) {
|
||||||
|
super(`Response shape drift in ${context} — ${issues.slice(0, 3).join('; ')}`)
|
||||||
|
this.name = 'ContractDriftError'
|
||||||
|
this.context = context
|
||||||
|
this.issues = issues
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate a payload against a schema, throwing on drift.
|
||||||
|
*
|
||||||
|
* Fails LOUDLY on purpose. The bug class this exists for is an endpoint
|
||||||
|
* silently dropping a column the UI needs: nothing errors, the page renders,
|
||||||
|
* and the number is just wrong. A 500 naming the offending path is strictly
|
||||||
|
* better than a plausible screen.
|
||||||
|
*/
|
||||||
|
export function assertShape<T>(schema: Parser<T>, data: unknown, context: string): T {
|
||||||
|
const result = schema.safeParse(data)
|
||||||
|
if (!result.success) {
|
||||||
|
// `.join()` throws on a symbol element, so map through String first —
|
||||||
|
// a symbol path renders as "Symbol(x)" rather than crashing the handler
|
||||||
|
// that was trying to report a validation error.
|
||||||
|
const issues = result.error.issues.map(i => `${i.path.map(String).join('.')}: ${i.message}`)
|
||||||
|
throw new ContractDriftError(context, issues)
|
||||||
|
}
|
||||||
|
return result.data
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FieldExpectation {
|
||||||
|
/** Dot path, e.g. `latest_invoice.confirmation_secret`. */
|
||||||
|
path: string
|
||||||
|
/** Human note on why this field was requested. */
|
||||||
|
why?: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FieldCheckResult {
|
||||||
|
ok: boolean
|
||||||
|
missing: string[]
|
||||||
|
present: string[]
|
||||||
|
}
|
||||||
|
|
||||||
|
function readPath(obj: unknown, path: string): unknown {
|
||||||
|
return path.split('.').reduce<unknown>(
|
||||||
|
(acc, k) => (acc == null || typeof acc !== 'object' ? undefined : (acc as Record<string, unknown>)[k]),
|
||||||
|
obj,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Assert a response actually contains the fields the request asked for.
|
||||||
|
*
|
||||||
|
* THE FAILURE THIS EXISTS FOR: Stripe removed `invoice.payment_intent` in the
|
||||||
|
* 2025-03-31 (Basil) API version, and Stripe **silently ignores an unknown
|
||||||
|
* `expand` path** rather than rejecting it. So a stale expand keeps succeeding
|
||||||
|
* and simply returns an invoice without the field. No error, no warning — the
|
||||||
|
* client secret is absent and the payment form cannot mount, while typecheck,
|
||||||
|
* lint and a full test suite all pass.
|
||||||
|
*
|
||||||
|
* A response-shape validator cannot catch this: the response is a perfectly
|
||||||
|
* valid invoice. The only way to detect it is to assert that what you ASKED FOR
|
||||||
|
* actually came back.
|
||||||
|
*/
|
||||||
|
export function expectFields(
|
||||||
|
data: unknown,
|
||||||
|
expectations: Array<string | FieldExpectation>,
|
||||||
|
): FieldCheckResult {
|
||||||
|
const missing: string[] = []
|
||||||
|
const present: string[] = []
|
||||||
|
for (const e of expectations) {
|
||||||
|
const path = typeof e === 'string' ? e : e.path
|
||||||
|
if (readPath(data, path) === undefined) missing.push(path)
|
||||||
|
else present.push(path)
|
||||||
|
}
|
||||||
|
return { ok: missing.length === 0, missing, present }
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Throwing form of `expectFields`.
|
||||||
|
*
|
||||||
|
* Use at an integration boundary where a silently-ignored request parameter
|
||||||
|
* would otherwise surface as a confusing downstream failure far from its cause.
|
||||||
|
*/
|
||||||
|
export function assertFields(
|
||||||
|
data: unknown,
|
||||||
|
expectations: Array<string | FieldExpectation>,
|
||||||
|
context: string,
|
||||||
|
): void {
|
||||||
|
const r = expectFields(data, expectations)
|
||||||
|
if (r.ok) return
|
||||||
|
const notes = expectations
|
||||||
|
.filter((e): e is FieldExpectation => typeof e !== 'string' && !!e.why)
|
||||||
|
.filter(e => r.missing.includes(e.path))
|
||||||
|
.map(e => `${e.path} (${e.why})`)
|
||||||
|
throw new ContractDriftError(
|
||||||
|
context,
|
||||||
|
notes.length ? notes : r.missing.map(m => `${m}: requested but absent from the response`),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CoverageReport {
|
||||||
|
total: number
|
||||||
|
covered: number
|
||||||
|
/** 0–100, one decimal. */
|
||||||
|
pct: number
|
||||||
|
uncovered: string[]
|
||||||
|
/** True when below the floor — the caller decides whether that fails a build. */
|
||||||
|
belowFloor: boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Measure how much of a surface is contract-checked, against a ratchet floor.
|
||||||
|
*
|
||||||
|
* Raise the floor over time (a ratchet) so coverage cannot silently regress. Reporting `uncovered` rather than just a percentage is deliberate: a
|
||||||
|
* bare number tells you there is a gap, not where it is.
|
||||||
|
*/
|
||||||
|
export function coverage(
|
||||||
|
all: string[],
|
||||||
|
checked: Iterable<string>,
|
||||||
|
floorPct = 0,
|
||||||
|
): CoverageReport {
|
||||||
|
const set = new Set(checked)
|
||||||
|
const uncovered = all.filter(x => !set.has(x))
|
||||||
|
const covered = all.length - uncovered.length
|
||||||
|
const pct = all.length === 0 ? 0 : Math.round((covered / all.length) * 1000) / 10
|
||||||
|
return { total: all.length, covered, pct, uncovered, belowFloor: pct < floorPct }
|
||||||
|
}
|
||||||
|
|
||||||
|
export { fieldReport, coverageReport, formatFields, formatCoverage } from './report.js'
|
||||||
|
export type { FieldReportOptions } from './report.js'
|
||||||
+115
@@ -0,0 +1,115 @@
|
|||||||
|
import {
|
||||||
|
type CapabilityReport,
|
||||||
|
type Finding,
|
||||||
|
type FormatOptions,
|
||||||
|
evidence,
|
||||||
|
formatReport as renderReport,
|
||||||
|
} from '@extant2000/evidence-record'
|
||||||
|
import type { CoverageReport, FieldCheckResult, FieldExpectation } from './index.js'
|
||||||
|
|
||||||
|
export interface FieldReportOptions {
|
||||||
|
/** The call being checked, e.g. "stripe.subscriptions.create". */
|
||||||
|
context?: string
|
||||||
|
/** The expectations, so a finding can quote why a field was requested. */
|
||||||
|
expectations?: Array<string | FieldExpectation>
|
||||||
|
}
|
||||||
|
|
||||||
|
const why = (path: string, exps: Array<string | FieldExpectation> = []): string | undefined => {
|
||||||
|
const e = exps.find(x => typeof x !== 'string' && x.path === path)
|
||||||
|
return typeof e === 'object' ? e.why : undefined
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Report a field-presence check.
|
||||||
|
*
|
||||||
|
* The distinction this library exists for survives into the vocabulary: a
|
||||||
|
* response can be a perfectly valid object (schema PASSES) while the field
|
||||||
|
* you asked for is absent (assertion FAILS), because the provider silently
|
||||||
|
* ignored part of the request. Each requested path is its own finding so the
|
||||||
|
* two cannot be collapsed into one verdict.
|
||||||
|
*/
|
||||||
|
export function fieldReport(r: FieldCheckResult, opts: FieldReportOptions = {}): CapabilityReport {
|
||||||
|
const context = opts.context ?? 'response'
|
||||||
|
const findings: Finding[] = [
|
||||||
|
...r.missing.map((p): Finding => ({
|
||||||
|
id: `missing:${p}`,
|
||||||
|
summary: `${p} was requested but is absent from the response`,
|
||||||
|
determination: 'fail',
|
||||||
|
severity: 'critical',
|
||||||
|
detail: why(p, opts.expectations)
|
||||||
|
?? 'The provider may be silently ignoring this request parameter. A schema check cannot catch this: the response is a valid object.',
|
||||||
|
evidence: [evidence.document(context, p, undefined, 'requested, not returned')],
|
||||||
|
})),
|
||||||
|
...r.present.map((p): Finding => ({
|
||||||
|
id: `present:${p}`,
|
||||||
|
summary: `${p} present`,
|
||||||
|
determination: 'pass',
|
||||||
|
severity: 'info',
|
||||||
|
evidence: [evidence.document(context, p, undefined, 'requested and returned')],
|
||||||
|
})),
|
||||||
|
]
|
||||||
|
|
||||||
|
const total = r.missing.length + r.present.length
|
||||||
|
return {
|
||||||
|
capability: 'ContractCheck',
|
||||||
|
scope: `${total} requested field(s) on ${context}`,
|
||||||
|
examined: total,
|
||||||
|
findings,
|
||||||
|
notes: total === 0
|
||||||
|
? ['No fields were asserted, so nothing was checked. A response that is never inspected cannot be said to conform.']
|
||||||
|
: [],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Report contract coverage against a ratchet floor.
|
||||||
|
*
|
||||||
|
* An empty surface is `not-assessed`, not 0%. `coverage([], …)` correctly
|
||||||
|
* returns zero, but rendering "0% covered" for a surface with no endpoints
|
||||||
|
* describes a failure that did not happen — the same error as reporting 100%
|
||||||
|
* uptime from zero samples, inverted.
|
||||||
|
*/
|
||||||
|
export function coverageReport(c: CoverageReport, floorPct = 0): CapabilityReport {
|
||||||
|
if (c.total === 0) {
|
||||||
|
return {
|
||||||
|
capability: 'ContractCheck',
|
||||||
|
scope: 'no surface supplied',
|
||||||
|
examined: 0,
|
||||||
|
findings: [],
|
||||||
|
notes: ['No endpoints were supplied, so coverage could not be computed. This is not 0% coverage — nothing was measured.'],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
capability: 'ContractCheck',
|
||||||
|
scope: `${c.total} endpoint(s)`,
|
||||||
|
examined: c.total,
|
||||||
|
findings: [
|
||||||
|
{
|
||||||
|
id: 'coverage',
|
||||||
|
summary: `${c.covered} of ${c.total} endpoints contract-checked (${c.pct}%)`,
|
||||||
|
determination: c.belowFloor ? 'fail' : 'pass',
|
||||||
|
severity: c.belowFloor ? 'high' : 'info',
|
||||||
|
detail: c.belowFloor ? `Below the ratchet floor of ${floorPct}%.` : undefined,
|
||||||
|
evidence: [evidence.measurement('Contract coverage', c.pct, c.total, { unit: '%' })],
|
||||||
|
},
|
||||||
|
// Named individually rather than left as a percentage: a bare number
|
||||||
|
// tells you there is a gap, not where it is.
|
||||||
|
...c.uncovered.map((path): Finding => ({
|
||||||
|
id: `uncovered:${path}`,
|
||||||
|
summary: `${path} has no contract check`,
|
||||||
|
determination: 'not-assessed',
|
||||||
|
severity: 'medium',
|
||||||
|
evidence: [],
|
||||||
|
})),
|
||||||
|
],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatFields(r: FieldCheckResult, opts: FieldReportOptions & FormatOptions = {}): string {
|
||||||
|
return renderReport(fieldReport(r, opts), opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
export function formatCoverage(c: CoverageReport, floorPct = 0, opts: FormatOptions = {}): string {
|
||||||
|
return renderReport(coverageReport(c, floorPct), opts)
|
||||||
|
}
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
import { describe, expect, it } from 'vitest'
|
||||||
|
import { overall, validateReport } from '@extant2000/evidence-record'
|
||||||
|
import {
|
||||||
|
ContractDriftError,
|
||||||
|
fieldReport,
|
||||||
|
coverageReport,
|
||||||
|
formatFields,
|
||||||
|
formatCoverage,
|
||||||
|
assertFields,
|
||||||
|
assertShape,
|
||||||
|
coverage,
|
||||||
|
expectFields,
|
||||||
|
type Parser,
|
||||||
|
} from '../src/index.js'
|
||||||
|
|
||||||
|
/** Minimal zod-shaped parser so the library stays dependency-free. */
|
||||||
|
function requireKeys(keys: string[]): Parser<Record<string, unknown>> {
|
||||||
|
return {
|
||||||
|
safeParse(data: unknown) {
|
||||||
|
if (data == null || typeof data !== 'object' || Array.isArray(data)) {
|
||||||
|
return { success: false as const, error: { issues: [{ path: [], message: 'expected an object' }] } }
|
||||||
|
}
|
||||||
|
const obj = data as Record<string, unknown>
|
||||||
|
const issues = keys.filter(k => !(k in obj)).map(k => ({ path: [k], message: 'required' }))
|
||||||
|
return issues.length
|
||||||
|
? { success: false as const, error: { issues } }
|
||||||
|
: { success: true as const, data: obj }
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('assertShape', () => {
|
||||||
|
it('returns the parsed data when the shape holds', () => {
|
||||||
|
expect(assertShape(requireKeys(['id']), { id: 1 }, 'GET /x')).toEqual({ id: 1 })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('throws naming the context and the offending path', () => {
|
||||||
|
try {
|
||||||
|
assertShape(requireKeys(['id', 'name']), { id: 1 }, 'GET /missions')
|
||||||
|
expect.unreachable('should have thrown')
|
||||||
|
} catch (e) {
|
||||||
|
expect(e).toBeInstanceOf(ContractDriftError)
|
||||||
|
const err = e as ContractDriftError
|
||||||
|
expect(err.context).toBe('GET /missions')
|
||||||
|
expect(err.message).toContain('name: required')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('catches null where a list was expected — the empty-table failure', () => {
|
||||||
|
expect(() => assertShape(requireKeys([]), null, 'GET /rows')).toThrow(ContractDriftError)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('is NOT a framework HTTP error', () => {
|
||||||
|
// A framework-specific HTTP error would tie the library to one framework.
|
||||||
|
try {
|
||||||
|
assertShape(requireKeys(['a']), {}, 'ctx')
|
||||||
|
} catch (e) {
|
||||||
|
expect(e).toBeInstanceOf(Error)
|
||||||
|
expect((e as { statusCode?: number }).statusCode).toBeUndefined()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('expectFields — the silently-ignored request parameter', () => {
|
||||||
|
// The motivating case: Stripe ignores an unknown `expand` path
|
||||||
|
// rather than rejecting it, so a stale expand returns a VALID invoice that
|
||||||
|
// simply lacks the field. A response-shape validator passes this.
|
||||||
|
const basilInvoice = {
|
||||||
|
id: 'in_1',
|
||||||
|
latest_invoice: { id: 'inv_1', confirmation_secret: { client_secret: 'cs_x' } },
|
||||||
|
}
|
||||||
|
const preBasilInvoice = {
|
||||||
|
id: 'in_1',
|
||||||
|
latest_invoice: { id: 'inv_1', payment_intent: { client_secret: 'pi_x' } },
|
||||||
|
}
|
||||||
|
|
||||||
|
it('detects the field that was requested but never came back', () => {
|
||||||
|
const r = expectFields(basilInvoice, ['latest_invoice.payment_intent'])
|
||||||
|
expect(r.ok).toBe(false)
|
||||||
|
expect(r.missing).toEqual(['latest_invoice.payment_intent'])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('passes when the requested field is present', () => {
|
||||||
|
expect(expectFields(preBasilInvoice, ['latest_invoice.payment_intent']).ok).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a schema validator would NOT catch this — the object is valid', () => {
|
||||||
|
// Both shapes are perfectly well-formed invoices. Only asking "did what I
|
||||||
|
// requested come back" distinguishes them.
|
||||||
|
expect(assertShape(requireKeys(['id', 'latest_invoice']), basilInvoice, 'stripe')).toBeTruthy()
|
||||||
|
expect(expectFields(basilInvoice, ['latest_invoice.payment_intent']).ok).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('handles a missing intermediate segment without throwing', () => {
|
||||||
|
expect(expectFields({}, ['a.b.c']).missing).toEqual(['a.b.c'])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('treats an explicit null as PRESENT — absent and null are different', () => {
|
||||||
|
// null means the provider answered "no value"; undefined means it never
|
||||||
|
// acknowledged the request. Conflating them hides the bug this catches.
|
||||||
|
expect(expectFields({ a: null }, ['a']).ok).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reports partial results, not just a boolean', () => {
|
||||||
|
const r = expectFields(basilInvoice, [
|
||||||
|
'latest_invoice.confirmation_secret',
|
||||||
|
'latest_invoice.payment_intent',
|
||||||
|
])
|
||||||
|
expect(r.present).toEqual(['latest_invoice.confirmation_secret'])
|
||||||
|
expect(r.missing).toEqual(['latest_invoice.payment_intent'])
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('assertFields', () => {
|
||||||
|
it('surfaces the why-note so the failure explains itself', () => {
|
||||||
|
try {
|
||||||
|
assertFields({}, [{ path: 'latest_invoice.payment_intent', why: 'needed to mount Elements' }], 'stripe.subscriptions.create')
|
||||||
|
expect.unreachable('should have thrown')
|
||||||
|
} catch (e) {
|
||||||
|
expect((e as ContractDriftError).message).toContain('needed to mount Elements')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not throw when everything requested came back', () => {
|
||||||
|
expect(() => assertFields({ a: 1 }, ['a'], 'ctx')).not.toThrow()
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('coverage', () => {
|
||||||
|
it('reports which items are uncovered, not just a percentage', () => {
|
||||||
|
const r = coverage(['a', 'b', 'c', 'd'], ['a', 'b'], 75)
|
||||||
|
expect(r.pct).toBe(50)
|
||||||
|
expect(r.uncovered).toEqual(['c', 'd'])
|
||||||
|
expect(r.belowFloor).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('passes at exactly the floor', () => {
|
||||||
|
expect(coverage(['a', 'b'], ['a'], 50).belowFloor).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('an empty surface is 0%, never a flattering 100', () => {
|
||||||
|
// Nothing to check is not full coverage.
|
||||||
|
const r = coverage([], [], 0)
|
||||||
|
expect(r.pct).toBe(0)
|
||||||
|
expect(r.covered).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('ignores checked items that are not part of the surface', () => {
|
||||||
|
expect(coverage(['a'], ['a', 'ghost'], 100).pct).toBe(100)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('report — schema pass and assertion fail are different verdicts', () => {
|
||||||
|
it('keeps a valid-object pass separate from a missing-field failure', () => {
|
||||||
|
// The Basil failure: a perfectly valid invoice missing the field that
|
||||||
|
// was asked for. One verdict cannot carry both facts.
|
||||||
|
const r = expectFields({ id: 'in_1', latest_invoice: {} }, [
|
||||||
|
{ path: 'latest_invoice.confirmation_secret', why: 'Elements cannot mount without it' },
|
||||||
|
'id',
|
||||||
|
])
|
||||||
|
expect(r.ok).toBe(false)
|
||||||
|
const rep = fieldReport(r, { context: 'stripe.subscriptions.create', expectations: [
|
||||||
|
{ path: 'latest_invoice.confirmation_secret', why: 'Elements cannot mount without it' },
|
||||||
|
] })
|
||||||
|
expect(overall(rep)).toBe('fail')
|
||||||
|
const text = formatFields(r, {
|
||||||
|
context: 'stripe.subscriptions.create',
|
||||||
|
expectations: [{ path: 'latest_invoice.confirmation_secret', why: 'Elements cannot mount without it' }],
|
||||||
|
evidence: 'full',
|
||||||
|
})
|
||||||
|
expect(text).toContain('latest_invoice.confirmation_secret')
|
||||||
|
expect(text).toContain('Elements cannot mount without it')
|
||||||
|
expect(text).toContain('stripe.subscriptions.create')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('treats an empty surface as not-assessed, not as 0% coverage', () => {
|
||||||
|
const rep = coverageReport(coverage([], []))
|
||||||
|
expect(overall(rep)).toBe('not-assessed')
|
||||||
|
expect(formatCoverage(coverage([], []))).toContain('This is not 0% coverage')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('names the uncovered endpoints, not just the percentage', () => {
|
||||||
|
const c = coverage(['/a', '/b', '/c'], ['/a'], 80)
|
||||||
|
const text = formatCoverage(c, 80)
|
||||||
|
expect(text).toContain('/b')
|
||||||
|
expect(text).toContain('/c')
|
||||||
|
expect(text).toContain('Below the ratchet floor')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('every conclusion carries a citation', () => {
|
||||||
|
const r = expectFields({ id: 'x' }, ['id', 'missing_one'])
|
||||||
|
expect(validateReport(fieldReport(r))).toEqual([])
|
||||||
|
expect(validateReport(coverageReport(coverage(['/a'], ['/a'])))).toEqual([])
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('Parser accepts a Zod 4 schema', () => {
|
||||||
|
it('accepts PropertyKey paths, including symbols', () => {
|
||||||
|
// Zod 4 types issue paths as PropertyKey[]. Narrowing to string|number
|
||||||
|
// made a Zod 4 schema unassignable, and where typecheck gates the
|
||||||
|
// pipeline that mismatch blocks a deploy.
|
||||||
|
const sym = Symbol('secret')
|
||||||
|
const schema = {
|
||||||
|
safeParse: () => ({
|
||||||
|
success: false as const,
|
||||||
|
error: { issues: [{ path: ['a', 1, sym], message: 'bad' }] },
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
expect(() => assertShape(schema, {}, 'ctx')).toThrow(ContractDriftError)
|
||||||
|
try {
|
||||||
|
assertShape(schema, {}, 'ctx')
|
||||||
|
} catch (e) {
|
||||||
|
// `.join()` alone throws on a symbol; this must render, not crash.
|
||||||
|
expect((e as ContractDriftError).message).toContain('Symbol(secret)')
|
||||||
|
expect((e as ContractDriftError).message).toContain('a.1.')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('still renders a plain string path unchanged', () => {
|
||||||
|
const schema = {
|
||||||
|
safeParse: () => ({
|
||||||
|
success: false as const,
|
||||||
|
error: { issues: [{ path: ['latest_invoice', 'id'], message: 'required' }] },
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
assertShape(schema, {}, 'ctx')
|
||||||
|
} catch (e) {
|
||||||
|
expect((e as ContractDriftError).message).toContain('latest_invoice.id: required')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022", "module": "ES2022", "moduleResolution": "bundler",
|
||||||
|
"declaration": true, "outDir": "dist", "rootDir": "src",
|
||||||
|
"strict": true, "skipLibCheck": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts"]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user