The mac1/mac2 build machines cannot reach port 80 on the Debian mirror CDN, so apt-get update failed and the reduction image never built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caelum reduction sidecar
The pixel-handling half of the reduction pipeline. Node owns the job queue, the calibration-matching rules and the provenance record; this package does the arithmetic.
The split is deliberate. astropy, photutils and ccdproc are validated and widely trusted, and a bug in a hand-rolled aperture sum produces plausible wrong answers, which is the worst failure mode available to a science platform. See TODOS.md item L.
What is implemented
| Stage | Module | State |
|---|---|---|
timing |
caelum_reduce/timing.py |
Done. UTC to BJD_TDB at mid-exposure. |
calibrate |
caelum_reduce/calibrate.py |
Done. Master bias/dark/flat, and applying them with uncertainties propagated. |
extract |
caelum_reduce/extract.py |
Done. Source catalogue, plus the frame's own seeing, ellipticity, sky level and depth. |
astrometry |
not yet | Needs a solver binary (ASTAP or astrometry.net) in the image. |
photometry |
not yet | Aperture and differential photometry. |
The worker loop, the protocol client and the container are done, so this runs as a service rather than sitting as a library:
docker compose --profile reduction up -d
It has been verified end to end against a live API: the container claims a
queued timing job, computes BJD_TDB, reports it, and the run comes back
complete with the software versions recorded for provenance.
Running it
Configuration is entirely environmental, because an env var works the same in compose, in a shell, and in a scheduler.
| Variable | Required | Notes |
|---|---|---|
CAELUM_API_URL |
yes | e.g. http://caelum:3000 |
CAELUM_WORKER_TOKEN |
yes | a cae_ token whose scopes are exactly reduction |
CAELUM_WORKER_ID |
no | defaults to the hostname |
CAELUM_WORKER_STAGES |
no | defaults to the stages this build implements |
CAELUM_POLL_INTERVAL |
no | seconds between empty claims, default 5 |
CAELUM_HEARTBEAT_INTERVAL |
no | default 120; the claim TTL is 10 minutes |
CAELUM_MASTER_POLL_INTERVAL |
no | seconds between master sweeps, default 900 |
CAELUM_LOG_LEVEL |
no | default INFO |
Give it a reduction-scoped token and nothing wider. That scope is not
account-wide write, so the container can claim and report reduction jobs and
nothing else. A daemon or write token would also work and would hand an
unattended container the owner's whole account.
A missing or malformed token fails at startup with a message saying which variable is wrong, rather than on the first claim with a 401 to interpret.
SIGTERM sets a stop flag rather than killing the process, so a container stop finishes the stage it is in instead of leaving a truncated master.
Running the tests
Wheels exist for every package on macOS, so the venv path needs no compiler. (The Docker build is the exception; see the photutils note below.)
cd sidecar
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt
PYTHONPATH=. .venv/bin/python -m pytest
154 tests, a few seconds. They run against synthetic frames with known truth and a faked HTTP transport, so each one asserts the right answer rather than self-consistency.
Why photutils and not SEP
The original plan named SEP for source extraction. photutils does the same job, is the astropy-affiliated package we already depend on, and builds reliably from source when a wheel is missing. SEP had no wheel for Python 3.14 at the time of writing and no source build that worked there either.
One correction to an earlier version of this file: photutils is not pure
Python. It has C extensions and publishes no linux/aarch64 wheel, which is
why the Dockerfile is multi-stage. On an arm64 host pip falls back to compiling
it, and putting that compiler in a build stage keeps it out of the runtime
image while letting the build work on amd64 and arm64 alike.
What the tests are actually checking
Worth knowing, because several of these caught mistakes during development.
Timing asserts physics, not just that a number comes back. The barycentric correction is bounded by the light-crossing time of Earth's orbit, it reverses sign between the solstices for a target in the ecliptic plane, it nearly vanishes at quadrature, it almost disappears toward the ecliptic poles, and two sites on opposite sides of Earth agree to within the light travel time across Earth's diameter. A correction that failed any of those would still look like a plausible timestamp.
The reason for that thoroughness: a hand-rolled approximation is wrong by around 0.1 s, and the error is periodic with a one-year cycle because it tracks Earth's orbit. A periodic error in a data set about periodic phenomena does not read as a bug, it reads as signal.
Calibration asserts the reduction equation recovers a known injected signal, that a cosmic ray in one input frame does not reach the master while a genuine hot pixel in every frame survives, that a dead column does not become a NaN hole, and that uncertainties can only grow under subtraction.
It also pins two refusals:
- Scaling a dark that is not marked
bias_subtractedis rejected. Dark current scales with exposure time and the bias does not, so scaling a bias-inclusive dark multiplies the bias too. This is the most common calibration mistake and the result looks entirely reasonable. - A master whose shape does not match the frame is rejected rather than allowed to broadcast. A master built at 2x2 binning silently applied to a 1x1 frame would be a disaster that no later step could detect.
And one documented limitation, pinned as a test so it stays visible: single-pass sigma clipping does not catch an evenly split stack. Combining eight frames at one illumination level with eight at another clips nothing, because the split inflates the stack's own standard deviation until both groups look like one-sigma deviations. Preventing that is the Node-side grouping rules' job, not the combiner's.