Every other reduction surface answers a question about one frame. The question an operator actually wakes up with is "did last night get processed", and answering it meant opening frames one at a time until you found the broken one. The failure this is really built for is the quiet one. The sidecar runs in its own container; when it stops claiming, every job stays pending, no error is ever written, and a stalled queue looks exactly like a busy one in the counts. So the page leads with which workers have claimed and when, checked per side: a healthy local Node worker says nothing about a dead sidecar, and the warning names the side that stopped along with the number of jobs waiting on it. Below that, the stages in dependency order with their counts and the age of the oldest waiting job (ten jobs a minute old is a healthy queue, one job from last Tuesday is a broken one), then the failures with the error the worker reported and a batch retry. Server side, GET /reduction/queue grew from bare counts into the whole payload, and POST /reduction/retry requeues up to 100 frames under one click, authorizing each frame individually and reporting what it refused rather than failing the batch. Two bugs found while verifying against a live queue rather than a fixture: - The failure scope SQL checked site_members but not site ownership. A site's owner has no site_members row, so the list hid an operator's own failures on their own telescope, which is the single case the page exists for. - A retried frame never left the list. Reprocessing creates a new run and keeps the old one, which is the provenance model working, but the old failed rows stay in the table. Failures are now scoped to each frame's current run, so acting on one clears it while the history stays intact. Verified against an isolated stack seeded with a real mix: one stage done, two frames failed on different stages with real errors, five sidecar jobs waiting, and a worker whose last claim was 40 minutes old. The page named the stalled sidecar rather than the failures, the batch retry reported "Queued 2 frames", the superseded failures dropped off, and the new run's genuine ENOENT failures took their place. Research is now full at three items, which the nav invariant tests enforce. DESIGN.md says what each of the three answers and that anything else has to displace one of them. 253 server tests, 22 client tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
16 KiB
Reduction worker protocol
The contract between the Caelum API and a reduction sidecar. Mounted at
/api/reduction/worker. Source:
server/src/routes/reduction-worker.js.
This document exists so someone can implement a worker without reading the server. The reference implementation lives in sidecar/.
Shape of the thing
Node owns the queue, decides which calibration masters apply to which frame, and keeps the provenance record. The worker does arithmetic on pixels. It polls for work, does one stage, reports back, repeats.
worker API
| POST /claim |
|------------------------------------>| claim a job, build a work order
|<------------------------------------| 200 work order, or 204 when idle
| |
| GET /frames/:id/file |
|------------------------------------>| download the pixels
| GET /masters/:id/file |
|------------------------------------>| download each master it was given
| |
| POST /jobs/:id/heartbeat | (repeat on a long stage)
|------------------------------------>|
| |
| POST /jobs/:id/complete |
|------------------------------------>| result validated, then stored
A claim response is a complete work order. A worker should never need a
second request to learn a filter name or a site's latitude. Same principle as
the observatory-daemon payload in routes/jobs.js.
Authentication
Every endpoint requires an Authorization: Bearer cae_... API token whose
scopes include reduction. Nothing else is accepted: a user session (JWT) is
refused with 401, and a write- or daemon-scoped token is refused with 403.
The reduction scope is not account-wide write. A token holding only that
scope is refused by the normal auth middleware for every mutation outside these
routes, so it cannot delete a frame or post a comment. That is deliberate: a
sidecar runs unattended in a container, which makes its credential the most
likely one to leak, and it should be able to do exactly its job and nothing
else.
Create one from the Caelum UI or:
curl -X POST https://caelum.network/api/tokens \
-H "Authorization: Bearer <your session token>" \
-H 'Content-Type: application/json' \
-d '{"name":"reduction sidecar","scopes":"reduction"}'
The token is shown once. Only its SHA-256 hash is stored, so a lost token is revoked rather than recovered.
Stages
| Stage | Runs on | Depends on |
|---|---|---|
header |
the Node server | nothing |
timing |
sidecar | header |
calibrate |
sidecar | header |
astrometry |
sidecar | calibrate |
extract |
sidecar | calibrate |
photometry |
sidecar | extract |
header is not claimable by a worker. A job is only offered once its
prerequisite stage is done or skipped, so a worker never has to check
ordering itself.
astrometry and extract share a prerequisite and can arrive in either
order. Extraction measures the frame off its pixel grid, so it needs calibrated
pixels rather than a solution; a WCS, where the frame has one, only adds sky
coordinates to the sources it found.
POST /claim
{ "worker_id": "sidecar-1", "stages": ["calibrate", "astrometry"] }
stages is optional and declares what this worker can run. Naming a stage the
server owns (header) is a 400 rather than a silent empty claim, because it is
a configuration mistake worth seeing.
Declaring stages matters more than it looks. The filter is applied inside the
claim query, so a worker is never handed a stage it cannot run. An earlier
version claimed first and handed the job back, which cost the job one of its
three attempts and a 30-second backoff; with two specialized workers polling, a
job could burn every attempt without anyone trying to run it.
204 when there is no work. Not an empty 200: a worker polling every few seconds should not have to parse a body to learn the queue is idle.
200 with a work order:
{
"job_id": "...", "run_id": "...", "stage": "calibrate", "attempt": 1,
"frame": {
"id": "...", "filename": "m31_Ha_001.fits",
"download_path": "/api/reduction/worker/frames/<id>/file",
"frame_type": "EXPOSE", "target_name": "M31", "filter": "Ha",
"exposure_time": 600, "date_obs": "2026-09-12T22:14:03",
"ccd_temp_c": -14.8, "gain": 1.6, "binning": "1x1"
},
"site": { "id": "...", "latitude": 34.5, "longitude": -120.5, "altitude_m": 1500 },
"telescope": { "id": "...", "aperture_mm": 355.6, "focal_length_mm": 2563 },
"instrument": { "id": "...", "pixel_size_um": 9.0, "filters": ["L","R","G","B","Ha"] }
}
download_path is relative. Resolve it against your own configured base URL,
so the same payload works whether you reach the API over the compose network or
a tunnel.
Plus a stage-specific part:
calibrate carries the masters to apply and why they were chosen.
{
"masters": {
"bias": { "id": "...", "kind": "BIAS", "download_path": "..." },
"dark": { "id": "...", "kind": "DARK", "exposure_time": 300, "download_path": "...", "scale": 2.0 },
"flat": { "id": "...", "kind": "FLAT", "filter": "Ha", "download_path": "..." }
},
"complete": true,
"decisions": [{ "kind": "FLAT", "master_id": "...", "reason": "matched on ..." }]
}
scale appears only when the dark's exposure did not match exactly. Subtract
the bias before scaling. Dark current scales with exposure time and the bias
does not, so scaling a bias-inclusive dark multiplies the bias too. The
reference implementation refuses to scale a dark not marked bias-subtracted, and
so should yours.
decisions includes the misses. "No flat matched, and here is why" is part of
how a frame was reduced, and it is recorded on the run at claim time rather
than when you report back, so a worker that dies mid-stage still leaves a
debuggable trail.
astrometry carries solver hints so the solve need not be blind.
{
"hints": {
"ra_deg": 10.68, "dec_deg": 41.27, "search_radius_deg": 5,
"scale_arcsec_per_px": 0.724, "scale_low": 0.579, "scale_high": 0.869
},
"reference_catalog": "gaia_dr3",
"already_solved": false
}
The scale comes from pixel size and focal length; the 20% window absorbs an unknown binning or an approximate focal length. The search radius is generous because the mount model is the weak link. Solve against Gaia DR3, not the solver's bundled index: sub-arcsecond astrometry is what makes MPC submission possible (TODOS.md item R), and a bundled index will not get you there.
timing carries date_obs, exposure_time, timescale_in,
timescale_out, epoch_convention and can_run. Report BJD_TDB at
mid-exposure. A 600 s exposure timed at the shutter is wrong by 300 s,
which is larger than the entire light-travel term.
photometry carries aperture radii as multiples of the measured FWHM,
rather than fixed pixels, because the right aperture depends on that night's
seeing. It also sets require_uncertainties: true, which is not advisory.
POST /jobs/:id/heartbeat
{ "worker_id": "sidecar-1" }
Extends the claim by the claim TTL (10 minutes). Call it on any stage that might run long: a wide-field plate solve can outlast the TTL, and without a heartbeat the reclaimer hands the job to a second worker while the first is still solving.
409 when the claim is no longer yours, with a reason: the job is not running, or another worker holds it. 409 rather than 403 because your credentials are fine; stop working on it.
POST /jobs/:id/complete
{ "worker_id": "sidecar-1", "result": { } }
Only the worker currently holding the claim may complete it. Everything in
result is merged into the run's provenance under the stage name, so include
what a reader would want: counts, residuals, versions, notes.
Some stages have required fields, checked at the boundary and rejected with 400 rather than stored:
-
astrometryneedsresult.wcswith at leastCTYPE1,CTYPE2,CRVAL1,CRVAL2,CRPIX1,CRPIX2. A partial WCS is refused and the frame is not marked solved, because a partial solution produces confidently wrong sky coordinates, which is worse than no solution. On success the frame'swcs_jsonis stored,wcs_solvedis set, andraanddec_coordare overwritten fromCRVAL1/2: the solved position beats the mount's reported pointing. -
timingneeds a finite numericresult.bjd_tdb. -
photometryneedsresult.measurements, an array in which every entry has a numericmagandmag_err. This is not negotiable (TODOS.md item N). A magnitude without an uncertainty is not a measurement, and accepting one is how an archive fills with numbers nobody can use. -
extractneeds an integerresult.source_count(zero is a valid answer: a frame taken through cloud legitimately contains nothing) and numericresult.sky_aduandresult.sky_rms_adu. A frame that was measured has to say what the sky was doing, otherwise nothing downstream can tell "clear but empty" from "never measured". Ifresult.fwhm_arcsecis present it must be accompanied by theresult.pixel_scale_arcsecit was converted with, because a seeing figure in the wrong units is the kind of number that survives into a paper.On success the measured values land in their own frame columns:
seeing_fwhm_arcsec,seeing_fwhm_px,ellipticity,sky_adu,sky_rms_adu,limiting_flux_adu,source_countandextracted_at. The frame's existingfwhm_arcsecis left alone: that column holds whatever the acquisition software claimed in the FITS header, which is a guess, is absent on most frames and is not comparable between two cameras. One column cannot mean both things.Those columns are what
GET /api/archivefilters and sorts on:max_seeing,max_ellipticity,min_sources,measured=1, andsort=seeing|depth|sources. Every one of them reads the measured column and ignoresfwhm_arcsec, so a frame carrying only a header claim never satisfies a quality filter.Report instrumental flux in ADU and nothing magnitude-shaped. A magnitude needs a zero point, which photometric standardisation supplies later; a number labelled as a magnitude before then is a lie that survives into a published result.
POST /jobs/:id/fail
{ "worker_id": "sidecar-1", "error": "solve failed: too few stars", "retryable": false }
retryable defaults to true. You are the only party that knows whether the
cause was a wedged solve (retry) or a corrupt file (do not), so decide rather
than letting the queue burn three attempts on something deterministic.
A job that runs out of attempts fails for good, and every stage downstream of
it is marked blocked rather than failed. Blocked means nothing is wrong
with that stage and it becomes runnable once the blocker is fixed.
POST /jobs/:id/skip
{ "worker_id": "sidecar-1", "reason": "frame already carried a complete WCS" }
reason is required. A skip satisfies the prerequisite for downstream stages,
so extraction still follows a skipped astrometry.
GET /frames/:id/file and GET /masters/:id/file
The bytes, as application/fits.
409 on a frame that has no local file, which is a legitimate state for a frame registered by URL. 404 when it should have one and the file is missing. The distinction matters: the first is "nothing to do here", the second is "something deleted the data".
Watching the queue from outside
GET /api/reduction/queue (a normal session endpoint, not a worker one)
returns pipeline health in one request: counts by stage and status, the
canonical stage order with each stage's runs_on, the oldest waiting job per
stage, the workers that have claimed with their last claim time and the stages
they serve, and the failures with their frames attached.
The worker list is the part worth having. A sidecar that stops claiming writes
no error: every job stays pending, and a stalled queue and a busy one look
identical in the counts. A last-claim time per side is what separates them.
Failures are scoped two ways. To frames the caller could actually reprocess, since a failure you cannot act on is noise. And to each frame's current run, because reprocessing keeps the old run rather than editing it, and without that scope the list would accumulate every failure anyone ever fixed.
GET /master-requests
Groups of raw calibration frames that are ready to combine and do not already have a current master.
GET /master-requests?telescope_id=<id>&min_frames=5
Master building is per-group, not per-frame, so it does not fit the job queue; poll this instead. Groups are keyed on instrument, binning, gain, bucketed sensor temperature, exposure time (darks) and filter (flats). A group is considered covered when a non-superseded master already exists built from the same set of source frames, so adding one more flat produces a new request and the master gets rebuilt deeper.
POST /masters
Multipart. file is the combined FITS; meta is a JSON string:
{
"telescope_id": "...", "instrument_id": "...", "kind": "FLAT",
"filter": "Ha", "exposure_time": 4.0, "ccd_temp_c": -15, "binning": "1x1", "gain": 1.6,
"combine_method": "sigma_clip_median", "sigma_low": 5.0, "sigma_high": 3.0,
"frame_ids": ["...", "..."],
"mean_level": 21000, "std_level": 140,
"valid_from": "2026-09-10 19:00:00", "valid_to": "2026-09-10 19:30:00"
}
Rules, enforced:
kindis BIAS, DARK or FLAT.- A FLAT requires a
filter. A flat is filter-specific by definition, and matching refuses a filter mismatch outright, so a filterless flat would be a master nothing could ever select. frame_idsneeds at least two entries.- Idempotent on the source frame set. Posting the same
frame_idstwice returns200 {"created": false}with the existing id rather than creating a duplicate that later selection would have to choose between. - A new master supersedes any older one sharing its grouping keys, rather than deleting it. Selection ignores superseded rows, so the rebuild takes effect immediately, while a historical run still resolves the master it actually used.
valid_from and valid_to bound the master in time. Set them from the source
frames' timestamps. Matching uses them: a flat goes stale after 30 days because
it describes the optical train as it was that night, while a dark lasts 90
because nothing physical about the sensor changed.
POST /runs/:id/environment
{
"solver": { "name": "astap", "version": "2024.03", "index": "g18" },
"software": { "astropy": "8.0.1", "photutils": "3.0.0", "image": "sha256:..." }
}
Report this once per run. It is what makes a run re-executable rather than merely described, and it is the foundation for TODOS.md item V. A version string is the minimum; a container digest is better, so send one if you have it.
Writing a worker
The loop is small:
POST /claimwith yourworker_idand the stages you implement.- On 204, sleep a few seconds and poll again.
- On 200, download the frame and any masters, do the work, heartbeat if it runs long.
POST .../completewith the result, or.../failwith a reason and an honestretryable.- Separately and less often, poll
/master-requestsand post any masters you build.
Three things to get right, all of which the API will catch but which are cheaper to get right first: subtract bias before scaling a dark, never report a partial WCS, and never report a magnitude without an uncertainty.