Files
Paul HittandClaude Opus 5.5 37b64af3ba
ci / client (pull_request) Successful in 28s
ci / lint (pull_request) Successful in 11s
ci / nox (pull_request) Successful in 12s
ci / server (mysql) (pull_request) Successful in 13s
ci / server (sqlite) (pull_request) Successful in 12s
ci / sidecar (pull_request) Successful in 34s
ci / image (pull_request) Skipped
ci / sourcemaps (pull_request) Skipped
ci / deploy (pull_request) Skipped
sidecar image: fetch Debian packages over https
The mac1/mac2 build machines cannot reach port 80 on the Debian mirror
CDN, so apt-get update failed and the reduction image never built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:13:17 -04:00
..

Caelum reduction sidecar

The pixel-handling half of the reduction pipeline. Node owns the job queue, the calibration-matching rules and the provenance record; this package does the arithmetic.

The split is deliberate. astropy, photutils and ccdproc are validated and widely trusted, and a bug in a hand-rolled aperture sum produces plausible wrong answers, which is the worst failure mode available to a science platform. See TODOS.md item L.

What is implemented

Stage Module State
timing caelum_reduce/timing.py Done. UTC to BJD_TDB at mid-exposure.
calibrate caelum_reduce/calibrate.py Done. Master bias/dark/flat, and applying them with uncertainties propagated.
extract caelum_reduce/extract.py Done. Source catalogue, plus the frame's own seeing, ellipticity, sky level and depth.
astrometry not yet Needs a solver binary (ASTAP or astrometry.net) in the image.
photometry not yet Aperture and differential photometry.

The worker loop, the protocol client and the container are done, so this runs as a service rather than sitting as a library:

docker compose --profile reduction up -d

It has been verified end to end against a live API: the container claims a queued timing job, computes BJD_TDB, reports it, and the run comes back complete with the software versions recorded for provenance.

Running it

Configuration is entirely environmental, because an env var works the same in compose, in a shell, and in a scheduler.

Variable Required Notes
CAELUM_API_URL yes e.g. http://caelum:3000
CAELUM_WORKER_TOKEN yes a cae_ token whose scopes are exactly reduction
CAELUM_WORKER_ID no defaults to the hostname
CAELUM_WORKER_STAGES no defaults to the stages this build implements
CAELUM_POLL_INTERVAL no seconds between empty claims, default 5
CAELUM_HEARTBEAT_INTERVAL no default 120; the claim TTL is 10 minutes
CAELUM_MASTER_POLL_INTERVAL no seconds between master sweeps, default 900
CAELUM_LOG_LEVEL no default INFO

Give it a reduction-scoped token and nothing wider. That scope is not account-wide write, so the container can claim and report reduction jobs and nothing else. A daemon or write token would also work and would hand an unattended container the owner's whole account.

A missing or malformed token fails at startup with a message saying which variable is wrong, rather than on the first claim with a 401 to interpret.

SIGTERM sets a stop flag rather than killing the process, so a container stop finishes the stage it is in instead of leaving a truncated master.

Running the tests

Wheels exist for every package on macOS, so the venv path needs no compiler. (The Docker build is the exception; see the photutils note below.)

cd sidecar
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt
PYTHONPATH=. .venv/bin/python -m pytest

154 tests, a few seconds. They run against synthetic frames with known truth and a faked HTTP transport, so each one asserts the right answer rather than self-consistency.

Why photutils and not SEP

The original plan named SEP for source extraction. photutils does the same job, is the astropy-affiliated package we already depend on, and builds reliably from source when a wheel is missing. SEP had no wheel for Python 3.14 at the time of writing and no source build that worked there either.

One correction to an earlier version of this file: photutils is not pure Python. It has C extensions and publishes no linux/aarch64 wheel, which is why the Dockerfile is multi-stage. On an arm64 host pip falls back to compiling it, and putting that compiler in a build stage keeps it out of the runtime image while letting the build work on amd64 and arm64 alike.

What the tests are actually checking

Worth knowing, because several of these caught mistakes during development.

Timing asserts physics, not just that a number comes back. The barycentric correction is bounded by the light-crossing time of Earth's orbit, it reverses sign between the solstices for a target in the ecliptic plane, it nearly vanishes at quadrature, it almost disappears toward the ecliptic poles, and two sites on opposite sides of Earth agree to within the light travel time across Earth's diameter. A correction that failed any of those would still look like a plausible timestamp.

The reason for that thoroughness: a hand-rolled approximation is wrong by around 0.1 s, and the error is periodic with a one-year cycle because it tracks Earth's orbit. A periodic error in a data set about periodic phenomena does not read as a bug, it reads as signal.

Calibration asserts the reduction equation recovers a known injected signal, that a cosmic ray in one input frame does not reach the master while a genuine hot pixel in every frame survives, that a dead column does not become a NaN hole, and that uncertainties can only grow under subtraction.

It also pins two refusals:

  • Scaling a dark that is not marked bias_subtracted is rejected. Dark current scales with exposure time and the bias does not, so scaling a bias-inclusive dark multiplies the bias too. This is the most common calibration mistake and the result looks entirely reasonable.
  • A master whose shape does not match the frame is rejected rather than allowed to broadcast. A master built at 2x2 binning silently applied to a 1x1 frame would be a disaster that no later step could detect.

And one documented limitation, pinned as a test so it stays visible: single-pass sigma clipping does not catch an evenly split stack. Combining eight frames at one illumination level with eight at another clips nothing, because the split inflates the stack's own standard deviation until both groups look like one-sigma deviations. Preventing that is the Node-side grouping rules' job, not the combiner's.