feat(sidecar): phase 2 complete, verified end to end in the container
The sidecar is now a running service rather than a tested library. Verified against a live API: the real container claims a queued timing job, computes BJD_TDB, reports it, and the run settles `complete` with the software versions recorded for provenance. New in the sidecar - client.py: the protocol, with a retry policy that is deliberate rather than blanket. 5xx, 429 and connection errors back off and retry; a 4xx raises immediately, because a 4xx is our bug, our credential or a stale claim, and retrying any of those turns a clear failure into a silent loop. 409 gets its own ClaimLost type, since a lost claim means stop working rather than something is broken. - worker.py: the poll loop, plus a second slower loop for master building, which is per-group rather than per-frame and so does not fit the queue. A heartbeat thread runs for the duration of every stage, and a result computed after the claim was lost is DISCARDED rather than written over another worker's. - fits_io.py: reads the first HDU that actually has pixels, because a professional instrument often leaves the primary empty and guessing gives a zero-size array rather than an error. write_master stamps the combine method, clip bounds, input count and source frame ids into the file, so a master is self-describing once copied away from the database. - __main__.py: env-only config, SIGTERM sets a stop flag rather than killing mid-write. Server side - GET /api/reduction/worker/masters/current. I had written the sidecar calling this before it existed, which is exactly the dangling-reference problem I flagged in the docs earlier. It returns the newest matching non-superseded master, or 204, so a worker can bias-subtract a dark's inputs; only a bias-subtracted dark can be scaled. One real bug the end-to-end run caught - The work order's frame object omitted ra/dec_coord. The barycentric correction is direction-dependent, so the timing stage failed with "the frame has no sky position" on a frame that had one. The position now travels with every work order, not just the astrometry hints, with a regression test. Two things the container build caught that the venv did not - photutils is NOT pure Python, contrary to what I wrote in the sidecar README last commit. It has C extensions and ships no linux/aarch64 wheel, so pip compiled it from source and the single-stage build failed without build-essential. The Dockerfile is now multi-stage: wheels are built in one stage, installed with --no-index in a slim runtime with no compiler. Corrected in the README. - My earlier pip install loop reported OK for sep-pjw when it had actually failed, because $? captured the exit of a piped tail rather than pip. Also - Compose entries in both dev and prod, behind a `reduction` profile so nothing starts without a token. The prod entry is NOT in the deploy path: astrometry, extract and photometry are unimplemented, so it would claim those and fail every one. - No CI image-build job for the sidecar, deliberately. The shared runner already builds one image per pipeline for nine repos, and a scientific-Python build on every push is not worth it for a service that cannot yet do the stages that matter. - CAELUM_WORKER_TOKEN documented in both env examples, with why it must be reduction-scoped and not daemon. - The worker now logs completions with their duration. It logged claims and nothing else, which left an operator unable to tell a slow stage from a wedged one. Verified: 238 server + 22 client + 115 sidecar tests pass, lint clean, SPA builds, hadolint clean on the new Dockerfile, all three YAML files parse, and the built image runs as uid 10001 with no compiler and no leftover wheels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
aa62529ea7
commit
0dae4cb19a
@@ -28,3 +28,16 @@ JWT_SECRET=REPLACE_ME_WITH_openssl_rand_hex_32_OUTPUT
|
||||
# Host port that the app listens on. Useful when running multiple Caelum
|
||||
# instances on one machine.
|
||||
CAELUM_PORT=3002
|
||||
|
||||
# --- Reduction sidecar (optional; docker compose --profile reduction) ---
|
||||
# A cae_ API token whose scopes are exactly "reduction". Create it from the UI
|
||||
# or POST /api/tokens, and give it nothing wider: the reduction scope is not
|
||||
# account-wide write, so this token can claim and report reduction jobs and
|
||||
# nothing else. A "daemon" or "write" token would also work and would hand an
|
||||
# unattended container the owner's whole account.
|
||||
CAELUM_WORKER_TOKEN=
|
||||
# Identifies this worker in logs and on job claims. Defaults to the hostname.
|
||||
CAELUM_WORKER_ID=
|
||||
# Only the stages the sidecar build implements. Leaving this unset makes the
|
||||
# worker claim astrometry and extract too and fail each as unimplemented.
|
||||
CAELUM_WORKER_STAGES=timing,calibrate
|
||||
|
||||
@@ -225,9 +225,18 @@ similar. Worth doing one phase per session with a QA pass between.
|
||||
(UTC to BJD_TDB at mid-exposure, which is also item O) and `calibrate.py`
|
||||
(master bias/dark/flat, applied with uncertainties propagated, which is
|
||||
also item N for this stage).
|
||||
- Still to do in phase 2: the worker poll loop, so the package becomes a
|
||||
running service rather than a tested library, and a Dockerfile plus a
|
||||
compose entry.
|
||||
- Phase 2 is complete. The poll loop, the protocol client, the container and
|
||||
the compose entries are done, and it has been verified end to end: the real
|
||||
container claims a queued `timing` job against a live API, computes BJD_TDB,
|
||||
reports it, and the run settles `complete` with the software versions
|
||||
recorded. `docker compose --profile reduction up -d`.
|
||||
- Still to do: the astrometry, extract and photometry stages. Astrometry needs
|
||||
a solver binary in the image and Gaia DR3 indexes on a volume, which is the
|
||||
first piece here that cannot be verified on a laptop.
|
||||
- No CI image-build job for the sidecar yet, on purpose. The shared runner
|
||||
already builds one image per pipeline across nine repos, and a
|
||||
scientific-Python build on every push is not worth it for a service that
|
||||
cannot yet do the stages that matter. Add it when astrometry lands.
|
||||
- SEP was dropped in favour of photutils. Same capability, pure Python over
|
||||
numpy so wheels exist everywhere, and it is the astropy-affiliated package
|
||||
we already depend on. SEP is a C extension and had no wheel for Python
|
||||
|
||||
@@ -93,3 +93,16 @@ CRM_API_URL=https://crm.hitthosting.com
|
||||
CRM_API_KEY=
|
||||
BOOKS_API_URL=https://books.hitthosting.com
|
||||
BOOKS_API_KEY=
|
||||
|
||||
# --- Reduction sidecar (optional; docker compose --profile reduction) ---
|
||||
# A cae_ API token whose scopes are exactly "reduction". Create it from the UI
|
||||
# or POST /api/tokens, and give it nothing wider: the reduction scope is not
|
||||
# account-wide write, so this token can claim and report reduction jobs and
|
||||
# nothing else. A "daemon" or "write" token would also work and would hand an
|
||||
# unattended container the owner's whole account.
|
||||
CAELUM_WORKER_TOKEN=
|
||||
# Identifies this worker in logs and on job claims. Defaults to the hostname.
|
||||
CAELUM_WORKER_ID=
|
||||
# Only the stages the sidecar build implements. Leaving this unset makes the
|
||||
# worker claim astrometry and extract too and fail each as unimplemented.
|
||||
CAELUM_WORKER_STAGES=timing,calibrate
|
||||
|
||||
@@ -62,6 +62,45 @@ services:
|
||||
- traefik.http.routers.caelum.tls.certresolver=le
|
||||
- traefik.http.services.caelum.loadbalancer.server.port=3000
|
||||
|
||||
# The reduction sidecar. Behind a profile and NOT started by the deploy job,
|
||||
# because the pipeline is not production-ready yet: astrometry, extract and
|
||||
# photometry have no implementation, so it would claim those stages and fail
|
||||
# every one of them.
|
||||
#
|
||||
# docker compose --profile reduction up -d caelum-reduce
|
||||
#
|
||||
# There is deliberately no image-build job for this in .gitlab-ci.yml yet.
|
||||
# The shared runner already builds one image per pipeline for nine repos, and
|
||||
# adding a scientific-Python build on every push would cost more than a
|
||||
# not-yet-usable service is worth. Add the job when the remaining stages land.
|
||||
#
|
||||
# Note what it does NOT need: the caelum-uploads volume. Frames and masters
|
||||
# travel over HTTP through the worker endpoints, so the sidecar has no access
|
||||
# to the archive on disk and can run on a different host entirely.
|
||||
caelum-reduce:
|
||||
profiles: ["reduction"]
|
||||
image: registry.gitlab.com/paulhitt/caelum-reduce:${CAELUM_REDUCE_TAG:-latest}
|
||||
restart: unless-stopped
|
||||
platform: linux/amd64
|
||||
depends_on:
|
||||
caelum:
|
||||
condition: service_healthy
|
||||
env_file: [./caelum.env]
|
||||
environment:
|
||||
CAELUM_API_URL: http://caelum:3000
|
||||
CAELUM_WORKER_ID: ovh-reduce
|
||||
# Only the implemented stages. See the note above.
|
||||
CAELUM_WORKER_STAGES: timing,calibrate
|
||||
# internal only: it talks to the API over the compose network and needs no
|
||||
# inbound route, so it gets no Traefik labels and no web network.
|
||||
networks: [internal]
|
||||
deploy:
|
||||
resources:
|
||||
limits: { memory: 1G }
|
||||
logging:
|
||||
driver: json-file
|
||||
options: { max-size: "10m", max-file: "3" }
|
||||
|
||||
caelum-db:
|
||||
image: mysql:8.4
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -75,5 +75,38 @@ services:
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# The reduction sidecar. Off by default, because it needs a token that only
|
||||
# exists after someone creates one:
|
||||
#
|
||||
# docker compose --profile reduction up -d
|
||||
#
|
||||
# Create the token from the UI, or:
|
||||
# curl -X POST localhost:3002/api/tokens -H 'Authorization: Bearer <session>' \
|
||||
# -H 'Content-Type: application/json' \
|
||||
# -d '{"name":"reduction sidecar","scopes":"reduction"}'
|
||||
#
|
||||
# The `reduction` scope is not account-wide write (see middleware/auth.js), so
|
||||
# this container's credential can claim and report reduction jobs and nothing
|
||||
# else. Do not give it a `daemon` or `write` token instead; it would work, and
|
||||
# it would hand an unattended container the owner's whole account.
|
||||
reduce:
|
||||
profiles: ["reduction"]
|
||||
build:
|
||||
context: ./sidecar
|
||||
dockerfile: Dockerfile
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
app:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
CAELUM_API_URL: http://app:3002
|
||||
CAELUM_WORKER_TOKEN: ${CAELUM_WORKER_TOKEN:?set CAELUM_WORKER_TOKEN to a cae_ token with the reduction scope}
|
||||
CAELUM_WORKER_ID: ${CAELUM_WORKER_ID:-compose-reduce}
|
||||
# Only the stages this build implements. Leaving it unset would have the
|
||||
# worker claim astrometry and extract too, and fail every one of them as
|
||||
# unimplemented, which looks like a broken queue.
|
||||
CAELUM_WORKER_STAGES: ${CAELUM_WORKER_STAGES:-timing,calibrate}
|
||||
CAELUM_LOG_LEVEL: ${CAELUM_LOG_LEVEL:-INFO}
|
||||
|
||||
volumes:
|
||||
caelum-db:
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
// GET /frames/:id/file -> the frame's bytes
|
||||
// GET /masters/:id/file -> a calibration master's bytes
|
||||
// GET /master-requests -> groups ready to be combined
|
||||
// GET /masters/current -> the current master of one kind
|
||||
// POST /masters -> register a built master (multipart)
|
||||
// POST /runs/:id/environment -> record solver and software versions
|
||||
//
|
||||
@@ -232,6 +233,41 @@ router.get('/master-requests', async (req, res) => {
|
||||
res.json({ requests: pending, min_frames: minFrames });
|
||||
});
|
||||
|
||||
// The current master of one kind for one instrument, so a worker building a
|
||||
// dark or a flat can bias-subtract its inputs first. Only a bias-subtracted
|
||||
// dark can be scaled to a different exposure time, so this is what keeps a
|
||||
// freshly built dark useful beyond its own exposure.
|
||||
//
|
||||
// 204 when there is none, which is a normal state for a new observatory rather
|
||||
// than an error: a dark built without a bias is still correct at its own
|
||||
// exposure time.
|
||||
router.get('/masters/current', async (req, res) => {
|
||||
const { telescope_id, kind, binning, filter } = req.query;
|
||||
if (!telescope_id || !kind) {
|
||||
return res.status(400).json({ error: 'telescope_id and kind are required' });
|
||||
}
|
||||
if (!['BIAS', 'DARK', 'FLAT'].includes(kind)) {
|
||||
return res.status(400).json({ error: 'kind must be BIAS, DARK or FLAT' });
|
||||
}
|
||||
|
||||
const params = [telescope_id, kind];
|
||||
let sql = `SELECT id, kind, filter, exposure_time, binning, frame_count, valid_to
|
||||
FROM calibration_masters
|
||||
WHERE telescope_id = ? AND kind = ? AND superseded_by IS NULL`;
|
||||
// Binning has to match when the caller states it: a master from a different
|
||||
// binning is a different pixel grid and cannot be subtracted.
|
||||
if (binning) { sql += ' AND COALESCE(binning, ?) = ?'; params.push(binning, binning); }
|
||||
if (filter) { sql += ' AND filter = ?'; params.push(filter); }
|
||||
// Newest first: the master closest in time is the one most likely to
|
||||
// describe the instrument as it is now.
|
||||
sql += ' ORDER BY valid_to DESC, created_at DESC LIMIT 1';
|
||||
|
||||
const row = await db.get(sql, params);
|
||||
if (!row) return res.status(204).end();
|
||||
|
||||
res.json({ ...row, download_path: `/api/reduction/worker/masters/${row.id}/file` });
|
||||
});
|
||||
|
||||
// Register a built master. Multipart: `file` is the combined FITS, `meta` is a
|
||||
// JSON string with the grouping keys and the quality numbers.
|
||||
router.post(
|
||||
|
||||
@@ -63,6 +63,12 @@ export async function buildWorkOrder(job) {
|
||||
download_path: `/api/reduction/worker/frames/${frame.id}/file`,
|
||||
frame_type: frame.frame_type,
|
||||
target_name: frame.target_name,
|
||||
// The sky position travels with every work order, not just the
|
||||
// astrometry one. The barycentric correction is direction-dependent, so
|
||||
// the timing stage needs it too, and omitting it made timing fail with
|
||||
// "the frame has no sky position" on a frame that had one.
|
||||
ra: frame.ra,
|
||||
dec_coord: frame.dec_coord,
|
||||
filter: frame.filter,
|
||||
exposure_time: frame.exposure_time,
|
||||
date_obs: frame.date_obs,
|
||||
|
||||
@@ -215,6 +215,27 @@ test('a claim returns a complete work order for a sidecar stage', async () => {
|
||||
assert.equal(body.epoch_convention, 'mid_exposure');
|
||||
});
|
||||
|
||||
test('every work order carries the sky position, not just the astrometry one', async () => {
|
||||
// Caught end to end: the timing stage needs ra/dec because the barycentric
|
||||
// correction is direction-dependent, and an earlier version of the work
|
||||
// order only exposed the position inside the astrometry hints. The timing
|
||||
// stage then failed with "no sky position" on a frame that had one.
|
||||
await resetQueue();
|
||||
const frameId = await insertFrame();
|
||||
await db.run('UPDATE archive_frames SET ra = ?, dec_coord = ? WHERE id = ?',
|
||||
[10.6847, 41.2687, frameId]);
|
||||
await queue.enqueueReduction(frameId, { stages: ['header', 'timing'] });
|
||||
await pipeline.runLocalStagesOnce({ max: 5 });
|
||||
|
||||
const { status, body } = await req('/api/reduction/worker/claim', {
|
||||
method: 'POST', body: JSON.stringify({ stages: ['timing'] }),
|
||||
});
|
||||
assert.equal(status, 200);
|
||||
assert.equal(body.stage, 'timing');
|
||||
assert.equal(body.frame.ra, 10.6847);
|
||||
assert.equal(body.frame.dec_coord, 41.2687);
|
||||
});
|
||||
|
||||
test('a worker is never handed a stage it did not ask for', async () => {
|
||||
await resetQueue();
|
||||
const frameId = await insertFrame();
|
||||
@@ -645,6 +666,70 @@ test('master-requests lists ready groups and drops ones already covered', async
|
||||
assert.equal(still, undefined, 'a covered group should not be requested again');
|
||||
});
|
||||
|
||||
test('masters/current returns the newest matching master, or 204', async () => {
|
||||
await db.run('DELETE FROM calibration_masters');
|
||||
|
||||
// Nothing yet. 204, not 404: a new observatory legitimately has no bias,
|
||||
// and a dark built without one is still correct at its own exposure time.
|
||||
const none = await req(`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=BIAS`);
|
||||
assert.equal(none.status, 204);
|
||||
|
||||
const older = await postMaster({
|
||||
telescope_id: TELESCOPE, kind: 'BIAS', binning: '1x1', ccd_temp_c: -15, gain: 1.6,
|
||||
frame_ids: ['o1', 'o2', 'o3'], valid_to: '2026-09-01 00:00:00',
|
||||
});
|
||||
const newer = await postMaster({
|
||||
telescope_id: TELESCOPE, kind: 'BIAS', binning: '1x1', ccd_temp_c: -15, gain: 1.6,
|
||||
frame_ids: ['n1', 'n2', 'n3', 'n4'], valid_to: '2026-09-11 00:00:00',
|
||||
});
|
||||
|
||||
const got = await req(`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=BIAS`);
|
||||
assert.equal(got.status, 200);
|
||||
// Newest wins: the master closest in time best describes the instrument now.
|
||||
assert.equal(got.body.id, newer.body.id);
|
||||
assert.equal(got.body.download_path, `/api/reduction/worker/masters/${newer.body.id}/file`);
|
||||
// And the superseded older one is not offered.
|
||||
assert.notEqual(got.body.id, older.body.id);
|
||||
});
|
||||
|
||||
test('masters/current will not hand back a different binning', async () => {
|
||||
// A master from another binning is a different pixel grid and cannot be
|
||||
// subtracted, so no match is the right answer rather than the nearest one.
|
||||
await db.run('DELETE FROM calibration_masters');
|
||||
await postMaster({
|
||||
telescope_id: TELESCOPE, kind: 'BIAS', binning: '2x2', ccd_temp_c: -15,
|
||||
frame_ids: ['t1', 't2', 't3'],
|
||||
});
|
||||
const { status } = await req(
|
||||
`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=BIAS&binning=1x1`);
|
||||
assert.equal(status, 204);
|
||||
});
|
||||
|
||||
test('masters/current filters flats by filter', async () => {
|
||||
await db.run('DELETE FROM calibration_masters');
|
||||
const r = await postMaster({
|
||||
telescope_id: TELESCOPE, kind: 'FLAT', filter: 'R', binning: '1x1',
|
||||
frame_ids: ['r1', 'r2', 'r3'],
|
||||
});
|
||||
const match = await req(
|
||||
`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=FLAT&filter=R`);
|
||||
assert.equal(match.body.id, r.body.id);
|
||||
|
||||
const other = await req(
|
||||
`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=FLAT&filter=Ha`);
|
||||
assert.equal(other.status, 204);
|
||||
});
|
||||
|
||||
test('masters/current validates its query', async () => {
|
||||
const noKind = await req(`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}`);
|
||||
assert.equal(noKind.status, 400);
|
||||
|
||||
const badKind = await req(
|
||||
`/api/reduction/worker/masters/current?telescope_id=${TELESCOPE}&kind=SCIENCE`);
|
||||
assert.equal(badKind.status, 400);
|
||||
assert.match(badKind.body.error, /BIAS, DARK or FLAT/);
|
||||
});
|
||||
|
||||
test('run environment is recorded for reproducibility', async () => {
|
||||
await resetQueue();
|
||||
const frameId = await insertFrame();
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
# Caelum reduction sidecar.
|
||||
#
|
||||
# Multi-stage because not every package in the astronomy stack ships a wheel
|
||||
# for every architecture. photutils has C extensions and publishes no
|
||||
# linux/aarch64 wheel, so on an arm64 host pip falls back to building from
|
||||
# source and needs a compiler. Putting that compiler in a build stage keeps it
|
||||
# out of the runtime image and makes the build work on amd64 and arm64 alike,
|
||||
# rather than only on whichever one happened to have wheels the day it was
|
||||
# written.
|
||||
#
|
||||
# Python 3.13 rather than 3.14: the stack's wheel coverage is better there, and
|
||||
# a base image whose wheels are still catching up turns every install into a
|
||||
# source build.
|
||||
|
||||
# ---------- build stage ----------
|
||||
FROM python:3.13-slim AS build
|
||||
|
||||
# Compilers and the headers numpy's build needs. None of this reaches the
|
||||
# runtime image.
|
||||
# hadolint ignore=DL3008
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /build
|
||||
COPY requirements.txt ./
|
||||
|
||||
# Resolve everything to wheels once. Anything that already has one is
|
||||
# downloaded; anything that does not is compiled here, in this stage.
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt
|
||||
|
||||
# ---------- runtime stage ----------
|
||||
FROM python:3.13-slim
|
||||
|
||||
# curl for the container healthcheck, ca-certificates so the worker can reach
|
||||
# an API over TLS. Nothing else: this process talks to one HTTP endpoint and
|
||||
# does arithmetic.
|
||||
#
|
||||
# The plate solver (ASTAP or astrometry.net) is deliberately NOT installed.
|
||||
# Its index files run to tens of GB and belong on a mounted volume rather than
|
||||
# in a layer, so it arrives with the astrometry stage in phase 3.
|
||||
# hadolint ignore=DL3008
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y curl ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install from the wheels built above, with the index switched off so a
|
||||
# network hiccup cannot silently pull a different version than the one this
|
||||
# image was built against.
|
||||
COPY requirements.txt ./
|
||||
COPY --from=build /wheels /wheels
|
||||
RUN pip install --no-cache-dir --no-index --find-links=/wheels -r requirements.txt \
|
||||
&& rm -rf /wheels
|
||||
|
||||
COPY caelum_reduce/ ./caelum_reduce/
|
||||
|
||||
# Unprivileged. The worker writes nothing to disk: frames and masters stream
|
||||
# through memory, so it needs no writable mount and should not have one.
|
||||
# An explicit numeric uid, so the identity is resolvable on any host and stays
|
||||
# predictable if a volume is ever mounted here.
|
||||
RUN useradd --create-home --uid 10001 --shell /usr/sbin/nologin reduce \
|
||||
&& chown -R 10001:10001 /app
|
||||
USER 10001
|
||||
|
||||
# Unbuffered so container logs appear as they happen rather than in blocks,
|
||||
# and no .pyc files since this user cannot write to /app anyway.
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
# The worker has no HTTP surface of its own, so "healthy" means the process can
|
||||
# still import its stack. A failed claim is not unhealthy: the API being
|
||||
# briefly down is a thing the worker is built to survive.
|
||||
#
|
||||
# JSON form, so there is no shell in the healthcheck. Python already exits
|
||||
# non-zero on an uncaught ImportError, which is the failure this detects.
|
||||
HEALTHCHECK --interval=60s --timeout=15s --start-period=20s --retries=3 \
|
||||
CMD ["python", "-c", "import astropy, numpy, photutils, caelum_reduce"]
|
||||
|
||||
ENTRYPOINT ["python", "-m", "caelum_reduce"]
|
||||
+51
-11
@@ -19,14 +19,48 @@ See [TODOS.md](../TODOS.md) item L.
|
||||
| `extract` | not yet | photutils segmentation. |
|
||||
| `photometry` | not yet | Aperture and differential photometry. |
|
||||
|
||||
The worker loop that polls `/api/reduction/worker/claim` is not written yet, so
|
||||
today this is a library with tests rather than a running service. The protocol
|
||||
it will speak is already implemented and tested on the Node side; see
|
||||
`server/src/routes/reduction-worker.js`.
|
||||
The worker loop, the protocol client and the container are done, so this runs
|
||||
as a service rather than sitting as a library:
|
||||
|
||||
```sh
|
||||
docker compose --profile reduction up -d
|
||||
```
|
||||
|
||||
It has been verified end to end against a live API: the container claims a
|
||||
queued `timing` job, computes BJD_TDB, reports it, and the run comes back
|
||||
`complete` with the software versions recorded for provenance.
|
||||
|
||||
## Running it
|
||||
|
||||
Configuration is entirely environmental, because an env var works the same in
|
||||
compose, in a shell, and in a scheduler.
|
||||
|
||||
| Variable | Required | Notes |
|
||||
| --- | --- | --- |
|
||||
| `CAELUM_API_URL` | yes | e.g. `http://caelum:3000` |
|
||||
| `CAELUM_WORKER_TOKEN` | yes | a `cae_` token whose scopes are exactly `reduction` |
|
||||
| `CAELUM_WORKER_ID` | no | defaults to the hostname |
|
||||
| `CAELUM_WORKER_STAGES` | no | defaults to the stages this build implements |
|
||||
| `CAELUM_POLL_INTERVAL` | no | seconds between empty claims, default 5 |
|
||||
| `CAELUM_HEARTBEAT_INTERVAL` | no | default 120; the claim TTL is 10 minutes |
|
||||
| `CAELUM_MASTER_POLL_INTERVAL` | no | seconds between master sweeps, default 900 |
|
||||
| `CAELUM_LOG_LEVEL` | no | default INFO |
|
||||
|
||||
Give it a `reduction`-scoped token and nothing wider. That scope is not
|
||||
account-wide write, so the container can claim and report reduction jobs and
|
||||
nothing else. A `daemon` or `write` token would also work and would hand an
|
||||
unattended container the owner's whole account.
|
||||
|
||||
A missing or malformed token fails at startup with a message saying which
|
||||
variable is wrong, rather than on the first claim with a 401 to interpret.
|
||||
|
||||
SIGTERM sets a stop flag rather than killing the process, so a container stop
|
||||
finishes the stage it is in instead of leaving a truncated master.
|
||||
|
||||
## Running the tests
|
||||
|
||||
The astronomy stack installs from wheels, so no compiler is needed:
|
||||
Wheels exist for every package on macOS, so the venv path needs no compiler.
|
||||
(The Docker build is the exception; see the photutils note below.)
|
||||
|
||||
```sh
|
||||
cd sidecar
|
||||
@@ -35,16 +69,22 @@ python3 -m venv .venv
|
||||
PYTHONPATH=. .venv/bin/python -m pytest
|
||||
```
|
||||
|
||||
53 tests, about a second. They run against synthetic frames with known truth,
|
||||
so each one asserts the right answer rather than self-consistency.
|
||||
115 tests, under two seconds. They run against synthetic frames with known
|
||||
truth and a faked HTTP transport, so each one asserts the right answer rather
|
||||
than self-consistency.
|
||||
|
||||
## Why photutils and not SEP
|
||||
|
||||
The original plan named SEP for source extraction. photutils does the same job,
|
||||
is pure Python over numpy so it has wheels everywhere, and is the
|
||||
astropy-affiliated package we already depend on. SEP is a C extension with
|
||||
narrower wheel coverage and had none for Python 3.14 at the time of writing.
|
||||
One fewer build dependency for no loss of capability.
|
||||
is the astropy-affiliated package we already depend on, and builds reliably
|
||||
from source when a wheel is missing. SEP had no wheel for Python 3.14 at the
|
||||
time of writing and no source build that worked there either.
|
||||
|
||||
One correction to an earlier version of this file: photutils is **not** pure
|
||||
Python. It has C extensions and publishes no `linux/aarch64` wheel, which is
|
||||
why the Dockerfile is multi-stage. On an arm64 host pip falls back to compiling
|
||||
it, and putting that compiler in a build stage keeps it out of the runtime
|
||||
image while letting the build work on amd64 and arm64 alike.
|
||||
|
||||
## What the tests are actually checking
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Entry point: `python -m caelum_reduce`.
|
||||
|
||||
Configuration is entirely environmental, because this runs as a container and
|
||||
an env var is the one configuration mechanism that works the same in compose,
|
||||
in a shell, and in a scheduler. See Config.from_env for the list.
|
||||
|
||||
SIGTERM sets a stop event rather than killing the process. A container stop
|
||||
that interrupted a master write would leave a truncated FITS file, so the loop
|
||||
finishes the stage it is in and then exits.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import threading
|
||||
|
||||
from .client import Client, Config, ProtocolError
|
||||
from .worker import IMPLEMENTED_STAGES, Worker, software_versions
|
||||
|
||||
|
||||
def configure_logging():
|
||||
level = (os.environ.get("CAELUM_LOG_LEVEL") or "INFO").upper()
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, level, logging.INFO),
|
||||
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
|
||||
stream=sys.stdout,
|
||||
)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
configure_logging()
|
||||
log = logging.getLogger("caelum_reduce")
|
||||
|
||||
try:
|
||||
config = Config.from_env()
|
||||
except ProtocolError as exc:
|
||||
# A misconfigured worker should say what is missing and exit, not poll
|
||||
# forever against nothing.
|
||||
log.error("cannot start: %s", exc)
|
||||
return 2
|
||||
|
||||
# Default to the stages this build actually implements. Without this the
|
||||
# server would offer astrometry and extract, and every one of those claims
|
||||
# would be failed as unimplemented, which looks like a broken queue.
|
||||
if not config.stages:
|
||||
config = Config(**{**config.__dict__, "stages": IMPLEMENTED_STAGES})
|
||||
|
||||
log.info("caelum-reduce starting: %r", config)
|
||||
log.info("software: %s", software_versions())
|
||||
|
||||
stop = threading.Event()
|
||||
|
||||
def handle(signum, _frame):
|
||||
log.info("signal %s received; finishing the current stage then exiting", signum)
|
||||
stop.set()
|
||||
|
||||
signal.signal(signal.SIGTERM, handle)
|
||||
signal.signal(signal.SIGINT, handle)
|
||||
|
||||
worker = Worker(Client(config))
|
||||
worker.run_forever(stop_event=stop)
|
||||
log.info("caelum-reduce stopped")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,331 @@
|
||||
"""HTTP client for the reduction worker protocol.
|
||||
|
||||
The contract is documented in docs/reduction-worker-protocol.md. This module
|
||||
implements it and adds the three things a long-running unattended process needs
|
||||
that the protocol itself does not specify.
|
||||
|
||||
**Retry only what is worth retrying.** A 5xx or a dropped connection is the
|
||||
server's problem or the network's, and backing off and trying again is right.
|
||||
A 4xx is *our* problem: the request was malformed, or the token is wrong, or
|
||||
the claim is no longer ours. Retrying a 4xx turns a clear bug into a silent
|
||||
loop, so it raises immediately.
|
||||
|
||||
**Never log the token.** It is read from the environment, held in a header, and
|
||||
`__repr__` is overridden so it cannot reach a log line through an exception
|
||||
traceback or a debug print.
|
||||
|
||||
**A claim that is no longer ours is not an error to crash on.** The server
|
||||
returns 409 with a reason when another worker took over, which happens whenever
|
||||
a heartbeat is missed. The worker loop needs to distinguish that from a real
|
||||
failure, so it gets its own exception type.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
|
||||
# Retry on these and on connection errors. 429 is included because a rate
|
||||
# limiter saying "slow down" is a request to retry, not a bug.
|
||||
RETRY_STATUS = frozenset({429, 500, 502, 503, 504})
|
||||
|
||||
DEFAULT_TIMEOUT = (10, 120) # (connect, read) seconds
|
||||
DEFAULT_MAX_RETRIES = 4
|
||||
DEFAULT_BACKOFF_BASE = 2.0
|
||||
|
||||
|
||||
class ProtocolError(RuntimeError):
|
||||
"""The server refused a request in a way retrying will not fix."""
|
||||
|
||||
def __init__(self, message, status=None, body=None):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
self.body = body
|
||||
|
||||
|
||||
class ClaimLost(ProtocolError):
|
||||
"""This worker no longer holds the claim on the job.
|
||||
|
||||
Raised on a 409 from heartbeat, complete, fail or skip. Expected rather
|
||||
than exceptional: it happens any time a stage outruns its claim without
|
||||
heartbeating, and the right response is to stop working on that job.
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
base_url: str
|
||||
token: str
|
||||
worker_id: str
|
||||
stages: tuple[str, ...] = ()
|
||||
poll_interval: float = 5.0
|
||||
heartbeat_interval: float = 120.0
|
||||
master_poll_interval: float = 900.0
|
||||
min_master_frames: int = 5
|
||||
timeout: tuple = DEFAULT_TIMEOUT
|
||||
max_retries: int = DEFAULT_MAX_RETRIES
|
||||
|
||||
def __repr__(self):
|
||||
# The token must not reach a log line through a traceback or a
|
||||
# careless print of the config object.
|
||||
return (
|
||||
f"Config(base_url={self.base_url!r}, worker_id={self.worker_id!r}, "
|
||||
f"stages={self.stages!r}, token=<redacted>)"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, env=None):
|
||||
"""Build a config from environment variables.
|
||||
|
||||
Required:
|
||||
CAELUM_API_URL e.g. http://caelum:3002
|
||||
CAELUM_WORKER_TOKEN a cae_ token with the `reduction` scope
|
||||
|
||||
Optional:
|
||||
CAELUM_WORKER_ID defaults to the hostname
|
||||
CAELUM_WORKER_STAGES comma-separated; defaults to all
|
||||
sidecar stages the server offers
|
||||
CAELUM_POLL_INTERVAL seconds between empty claims, default 5
|
||||
CAELUM_HEARTBEAT_INTERVAL seconds between heartbeats, default 120
|
||||
CAELUM_MASTER_POLL_INTERVAL seconds between master sweeps, 900
|
||||
CAELUM_MIN_MASTER_FRAMES floor for combining, default 5
|
||||
"""
|
||||
env = env if env is not None else os.environ
|
||||
base = (env.get("CAELUM_API_URL") or "").rstrip("/")
|
||||
token = env.get("CAELUM_WORKER_TOKEN") or ""
|
||||
missing = [
|
||||
name for name, value in (("CAELUM_API_URL", base), ("CAELUM_WORKER_TOKEN", token))
|
||||
if not value
|
||||
]
|
||||
if missing:
|
||||
raise ProtocolError(f"missing required environment variable(s): {', '.join(missing)}")
|
||||
if not token.startswith("cae_"):
|
||||
# Fail at startup with a clear message rather than on the first
|
||||
# claim with a 401 the operator has to go and interpret.
|
||||
raise ProtocolError(
|
||||
"CAELUM_WORKER_TOKEN does not look like a Caelum API token "
|
||||
"(expected a cae_ prefix); the worker endpoints refuse user sessions"
|
||||
)
|
||||
|
||||
stages = tuple(
|
||||
s.strip() for s in (env.get("CAELUM_WORKER_STAGES") or "").split(",") if s.strip()
|
||||
)
|
||||
return cls(
|
||||
base_url=base,
|
||||
token=token,
|
||||
worker_id=env.get("CAELUM_WORKER_ID") or _hostname(),
|
||||
stages=stages,
|
||||
poll_interval=_float(env, "CAELUM_POLL_INTERVAL", 5.0),
|
||||
heartbeat_interval=_float(env, "CAELUM_HEARTBEAT_INTERVAL", 120.0),
|
||||
master_poll_interval=_float(env, "CAELUM_MASTER_POLL_INTERVAL", 900.0),
|
||||
min_master_frames=int(_float(env, "CAELUM_MIN_MASTER_FRAMES", 5)),
|
||||
)
|
||||
|
||||
|
||||
class Client:
|
||||
"""The worker side of the protocol.
|
||||
|
||||
Args:
|
||||
config: a Config.
|
||||
session: anything with requests.Session's `request` method. Injected so
|
||||
tests can drive the protocol without a server.
|
||||
sleep: injected for tests, so backoff does not make the suite slow.
|
||||
"""
|
||||
|
||||
def __init__(self, config: Config, session=None, sleep=time.sleep):
|
||||
self.config = config
|
||||
self._session = session or requests.Session()
|
||||
self._sleep = sleep
|
||||
|
||||
# ---------- claiming and reporting ----------
|
||||
|
||||
def claim(self):
|
||||
"""Claim a job. Returns a work order, or None when the queue is idle."""
|
||||
body = {"worker_id": self.config.worker_id}
|
||||
if self.config.stages:
|
||||
body["stages"] = list(self.config.stages)
|
||||
res = self._request("POST", "/api/reduction/worker/claim", json=body)
|
||||
# 204 is the idle case and carries no body.
|
||||
if res.status_code == 204:
|
||||
return None
|
||||
return res.json()
|
||||
|
||||
def heartbeat(self, job_id):
|
||||
"""Extend the claim. Raises ClaimLost if it is no longer ours."""
|
||||
self._request(
|
||||
"POST", f"/api/reduction/worker/jobs/{job_id}/heartbeat",
|
||||
json={"worker_id": self.config.worker_id},
|
||||
)
|
||||
|
||||
def complete(self, job_id, result):
|
||||
self._request(
|
||||
"POST", f"/api/reduction/worker/jobs/{job_id}/complete",
|
||||
json={"worker_id": self.config.worker_id, "result": result},
|
||||
)
|
||||
|
||||
def fail(self, job_id, error, retryable=True):
|
||||
self._request(
|
||||
"POST", f"/api/reduction/worker/jobs/{job_id}/fail",
|
||||
json={"worker_id": self.config.worker_id, "error": str(error)[:2000],
|
||||
"retryable": bool(retryable)},
|
||||
)
|
||||
|
||||
def skip(self, job_id, reason):
|
||||
self._request(
|
||||
"POST", f"/api/reduction/worker/jobs/{job_id}/skip",
|
||||
json={"worker_id": self.config.worker_id, "reason": str(reason)[:2000]},
|
||||
)
|
||||
|
||||
def record_environment(self, run_id, solver=None, software=None):
|
||||
payload = {}
|
||||
if solver:
|
||||
payload["solver"] = solver
|
||||
if software:
|
||||
payload["software"] = software
|
||||
if not payload:
|
||||
return
|
||||
self._request("POST", f"/api/reduction/worker/runs/{run_id}/environment", json=payload)
|
||||
|
||||
# ---------- files ----------
|
||||
|
||||
def download(self, path) -> bytes:
|
||||
"""Fetch a frame or a master by the relative path the server gave us.
|
||||
|
||||
The path comes from a work order and is relative on purpose, so the
|
||||
same payload works over a compose network or a tunnel.
|
||||
"""
|
||||
res = self._request("GET", path, stream=False)
|
||||
return res.content
|
||||
|
||||
# ---------- calibration masters ----------
|
||||
|
||||
def master_requests(self, telescope_id=None, min_frames=None):
|
||||
params = {"min_frames": min_frames or self.config.min_master_frames}
|
||||
if telescope_id:
|
||||
params["telescope_id"] = telescope_id
|
||||
res = self._request("GET", "/api/reduction/worker/master-requests", params=params)
|
||||
return res.json().get("requests", [])
|
||||
|
||||
def current_master(self, telescope_id, kind, binning=None, filter_name=None):
|
||||
"""The current master of `kind` for this instrument, or None.
|
||||
|
||||
Used when building a dark or a flat, both of which want the bias
|
||||
removed from their inputs first. None is a normal answer for a new
|
||||
observatory, not an error.
|
||||
"""
|
||||
params = {"telescope_id": telescope_id, "kind": kind}
|
||||
if binning:
|
||||
params["binning"] = binning
|
||||
if filter_name:
|
||||
params["filter"] = filter_name
|
||||
res = self._request("GET", "/api/reduction/worker/masters/current", params=params)
|
||||
if res.status_code == 204:
|
||||
return None
|
||||
return res.json()
|
||||
|
||||
def post_master(self, meta, fits_bytes, filename="master.fits"):
|
||||
"""Register a built master. Returns the API's response body.
|
||||
|
||||
Idempotent server-side on the source frame set, so a retry after a
|
||||
timeout returns the existing master rather than creating a duplicate.
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
res = self._request(
|
||||
"POST", "/api/reduction/worker/masters",
|
||||
data={"meta": _json.dumps(meta)},
|
||||
files={"file": (filename, fits_bytes, "application/fits")},
|
||||
)
|
||||
return res.json()
|
||||
|
||||
# ---------- internals ----------
|
||||
|
||||
def _request(self, method, path, **kwargs):
|
||||
url = path if path.startswith("http") else f"{self.config.base_url}{path}"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.config.token}",
|
||||
"X-Worker-Id": self.config.worker_id,
|
||||
}
|
||||
headers.update(kwargs.pop("headers", {}))
|
||||
kwargs.setdefault("timeout", self.config.timeout)
|
||||
|
||||
last_error = None
|
||||
for attempt in range(self.config.max_retries):
|
||||
try:
|
||||
res = self._session.request(method, url, headers=headers, **kwargs)
|
||||
except Exception as exc: # requests raises several unrelated types
|
||||
last_error = exc
|
||||
self._backoff(attempt)
|
||||
continue
|
||||
|
||||
if res.status_code < 400:
|
||||
return res
|
||||
|
||||
if res.status_code in RETRY_STATUS and attempt < self.config.max_retries - 1:
|
||||
last_error = ProtocolError(
|
||||
f"{method} {path} returned {res.status_code}", res.status_code, _body(res)
|
||||
)
|
||||
self._backoff(attempt)
|
||||
continue
|
||||
|
||||
# A 4xx is our bug, our credential, or a claim we no longer hold.
|
||||
# Retrying any of those turns a clear failure into a silent loop.
|
||||
body = _body(res)
|
||||
message = _message(body) or f"{method} {path} returned {res.status_code}"
|
||||
if res.status_code == 409:
|
||||
raise ClaimLost(message, res.status_code, body)
|
||||
# A retryable status that reaches here has exhausted its attempts.
|
||||
# Say so as well as what the server said: an operator reading
|
||||
# "still down" alone cannot tell whether we tried once or four
|
||||
# times, and that changes what they go and look at.
|
||||
if res.status_code in RETRY_STATUS:
|
||||
message = (
|
||||
f"{method} {path} still failing after {self.config.max_retries} "
|
||||
f"attempts ({res.status_code}): {message}"
|
||||
)
|
||||
raise ProtocolError(message, res.status_code, body)
|
||||
|
||||
raise ProtocolError(
|
||||
f"{method} {path} failed after {self.config.max_retries} attempts: {last_error}"
|
||||
)
|
||||
|
||||
def _backoff(self, attempt):
|
||||
self._sleep(DEFAULT_BACKOFF_BASE ** attempt)
|
||||
|
||||
|
||||
def _body(res):
|
||||
try:
|
||||
return res.json()
|
||||
except Exception:
|
||||
text = getattr(res, "text", "") or ""
|
||||
return {"error": text[:500]} if text else {}
|
||||
|
||||
|
||||
def _message(body):
|
||||
if isinstance(body, dict):
|
||||
parts = [body.get("error"), body.get("hint")]
|
||||
joined = "; ".join(p for p in parts if p)
|
||||
return joined or None
|
||||
return None
|
||||
|
||||
|
||||
def _float(env, name, default):
|
||||
raw = env.get(name)
|
||||
if raw in (None, ""):
|
||||
return default
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
raise ProtocolError(f"{name} must be a number, got {raw!r}") from None
|
||||
|
||||
|
||||
def _hostname():
|
||||
import socket
|
||||
|
||||
try:
|
||||
return socket.gethostname() or "sidecar"
|
||||
except Exception:
|
||||
return "sidecar"
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Reading frames and writing masters.
|
||||
|
||||
Thin on purpose. astropy.io.fits does the work; this module exists to make two
|
||||
decisions consistently rather than at each call site.
|
||||
|
||||
**Which HDU holds the image.** A frame from an amateur camera puts its data in
|
||||
the primary HDU. A frame from a professional instrument often leaves the
|
||||
primary empty and puts the image in the first extension. Guessing wrong gives
|
||||
you a zero-size array rather than an error, so `read_image` walks the HDUs and
|
||||
takes the first one with actual pixels, reporting which it used.
|
||||
|
||||
**What a master carries in its header.** A master is an output of this
|
||||
pipeline, and six months later someone will ask what went into it. The answer
|
||||
belongs in the file, not only in the database, because the file is what gets
|
||||
copied around. So `write_master` stamps the combine method, the clip bounds,
|
||||
the input count, the software version, and the grouping keys, and records the
|
||||
source frame ids as HISTORY cards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
from astropy.io import fits
|
||||
|
||||
from . import __version__
|
||||
|
||||
# Cards a master inherits from its inputs when they are present. Everything
|
||||
# else about the inputs is either meaningless for a combined frame (DATE-OBS of
|
||||
# which input?) or recorded explicitly below.
|
||||
INHERITED_CARDS = ("INSTRUME", "TELESCOP", "XBINNING", "YBINNING", "GAIN", "EGAIN")
|
||||
|
||||
|
||||
class FitsError(ValueError):
|
||||
"""The file is not usable as an image frame."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Image:
|
||||
data: np.ndarray
|
||||
header: fits.Header
|
||||
hdu_index: int
|
||||
"""Which HDU the pixels came from. 0 for a primary-HDU frame."""
|
||||
|
||||
@property
|
||||
def shape(self):
|
||||
return self.data.shape
|
||||
|
||||
|
||||
def read_image(source) -> Image:
|
||||
"""Read the first HDU that actually contains pixels.
|
||||
|
||||
Args:
|
||||
source: a path, a file object, or bytes.
|
||||
|
||||
Raises:
|
||||
FitsError: if the file will not open, or contains no image data. A
|
||||
table-only FITS is a legitimate file and a useless frame, so it is
|
||||
an error here rather than an empty array.
|
||||
"""
|
||||
opener = io.BytesIO(source) if isinstance(source, (bytes, bytearray)) else source
|
||||
try:
|
||||
with fits.open(opener, memmap=False) as hdul:
|
||||
for index, hdu in enumerate(hdul):
|
||||
data = getattr(hdu, "data", None)
|
||||
if data is None:
|
||||
continue
|
||||
arr = np.asarray(data)
|
||||
# A zero-size array is what an empty primary HDU looks like,
|
||||
# and a 1-D array is a spectrum or a table column, not a frame.
|
||||
if arr.size == 0 or arr.ndim < 2:
|
||||
continue
|
||||
return Image(
|
||||
data=arr.astype(np.float64, copy=True),
|
||||
header=hdu.header.copy(),
|
||||
hdu_index=index,
|
||||
)
|
||||
raise FitsError(
|
||||
f"no HDU in this file contains 2-D image data ({len(hdul)} HDU(s) checked)"
|
||||
)
|
||||
except FitsError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise FitsError(f"could not read the FITS file: {exc}") from exc
|
||||
|
||||
|
||||
def read_stack(sources) -> tuple[np.ndarray, list[fits.Header]]:
|
||||
"""Read several frames into one (n, ny, nx) array.
|
||||
|
||||
Every frame must have the same shape. A mismatch means the group mixed
|
||||
binnings or sensors, which is a grouping bug upstream, so it fails here
|
||||
rather than being silently cropped or broadcast.
|
||||
|
||||
Raises:
|
||||
FitsError: on an empty list or a shape mismatch.
|
||||
"""
|
||||
images = [read_image(s) for s in sources]
|
||||
if not images:
|
||||
raise FitsError("no frames to stack")
|
||||
shapes = {img.shape for img in images}
|
||||
if len(shapes) != 1:
|
||||
raise FitsError(
|
||||
f"frames have {len(shapes)} different shapes ({sorted(shapes)}); "
|
||||
"a calibration group must be one binning from one sensor"
|
||||
)
|
||||
return np.stack([img.data for img in images]), [img.header for img in images]
|
||||
|
||||
|
||||
def write_master(master, *, source_headers=None, keys=None) -> bytes:
|
||||
"""Serialize a combined master to FITS bytes.
|
||||
|
||||
The uncertainty array rides along as a named extension rather than a second
|
||||
file, so a master and its error plane cannot be separated by a copy.
|
||||
|
||||
Args:
|
||||
master: a calibrate.MasterResult.
|
||||
source_headers: headers of the input frames, for the cards a master
|
||||
legitimately inherits.
|
||||
keys: the grouping keys the API stores, stamped into the header so the
|
||||
file is self-describing.
|
||||
|
||||
Returns:
|
||||
The bytes of a multi-extension FITS file.
|
||||
"""
|
||||
primary = fits.PrimaryHDU(data=master.data.astype(np.float32))
|
||||
h = primary.header
|
||||
|
||||
h["IMAGETYP"] = (f"Master {master.kind.title()}", "combined calibration frame")
|
||||
h["CAELUMVR"] = (__version__, "caelum-reduce version that built this")
|
||||
h["NCOMBINE"] = (master.frame_count, "number of input frames")
|
||||
h["COMBMETH"] = (master.combine_method, "how the inputs were combined")
|
||||
h["SIGMALOW"] = (master.sigma_low, "low sigma clip bound")
|
||||
h["SIGMAHI"] = (master.sigma_high, "high sigma clip bound")
|
||||
h["REJFRAC"] = (round(master.rejected_fraction, 6), "fraction of samples clipped")
|
||||
h["MEANLVL"] = (round(master.mean_level, 4), "mean level of the master")
|
||||
h["STDLVL"] = (round(master.std_level, 4), "stddev of the master")
|
||||
# Only meaningful for a dark, but harmless and explicit on the others. The
|
||||
# apply step depends on this, and a master that travels without it cannot
|
||||
# be scaled safely.
|
||||
h["BIASSUB"] = (master.bias_subtracted, "bias removed before combining")
|
||||
|
||||
for key, value in (keys or {}).items():
|
||||
card = _KEY_CARDS.get(key)
|
||||
if card and value is not None:
|
||||
h[card[0]] = (value, card[1])
|
||||
|
||||
if source_headers:
|
||||
for card in INHERITED_CARDS:
|
||||
for src in source_headers:
|
||||
if card in src:
|
||||
h[card] = src[card]
|
||||
break
|
||||
|
||||
for note in master.notes:
|
||||
# HISTORY wraps, so a long note stays readable rather than truncated.
|
||||
h.add_history(note)
|
||||
for fid in (keys or {}).get("frame_ids", []) or []:
|
||||
h.add_history(f"input frame {fid}")
|
||||
|
||||
unc = fits.ImageHDU(data=master.uncertainty.astype(np.float32), name="UNCERT")
|
||||
unc.header["BUNIT"] = ("ADU", "1-sigma uncertainty per pixel")
|
||||
unc.header["COMMENT"] = "standard error of the median across surviving inputs"
|
||||
|
||||
buf = io.BytesIO()
|
||||
fits.HDUList([primary, unc]).writeto(buf, overwrite=True)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
# Grouping keys to FITS cards. Eight characters is the limit for a standard
|
||||
# keyword, which is why these are abbreviated.
|
||||
_KEY_CARDS = {
|
||||
"filter": ("FILTER", "filter this master applies to"),
|
||||
"exposure_time": ("EXPTIME", "exposure of the input frames, seconds"),
|
||||
"ccd_temp_c": ("CCD-TEMP", "sensor temperature of the inputs, C"),
|
||||
"binning": ("CCDSUM", "binning of the inputs"),
|
||||
"gain": ("GAIN", "gain of the inputs, e-/ADU"),
|
||||
"telescope_id": ("TELESCID", "Caelum telescope id"),
|
||||
"instrument_id": ("INSTRID", "Caelum instrument id"),
|
||||
"valid_from": ("VALIDFRM", "earliest input timestamp"),
|
||||
"valid_to": ("VALIDTO", "latest input timestamp"),
|
||||
}
|
||||
@@ -0,0 +1,479 @@
|
||||
"""The poll loop: claim a job, run a stage, report back.
|
||||
|
||||
Two loops, not one. The job loop claims from `/claim` and runs a stage. The
|
||||
master loop polls `/master-requests` far less often and builds calibration
|
||||
masters, because master building is per-group rather than per-frame and so does
|
||||
not fit the job queue. They share one thread and alternate, since neither is
|
||||
CPU-bound enough to justify the coordination a second thread would need.
|
||||
|
||||
Things this module is careful about
|
||||
-----------------------------------
|
||||
**A stage that outruns its claim.** A heartbeat runs on a daemon thread for the
|
||||
duration of every stage, so a long plate solve does not get handed to a second
|
||||
worker while the first is still working. The thread stops when the stage
|
||||
returns, and a lost claim is noticed rather than ignored.
|
||||
|
||||
**Reporting a result for a job we no longer hold.** If the heartbeat thread
|
||||
sees a 409 the claim is gone, and whatever this worker computes is now stale.
|
||||
It stops rather than completing over another worker's result.
|
||||
|
||||
**Failing honestly.** A stage raises one of two kinds of error. A
|
||||
`PermanentFailure` (a corrupt file, a frame with no local bytes, a stage we do
|
||||
not implement) is reported with `retryable: false`, because three attempts at
|
||||
the same deterministic failure only delays the message an operator needs. Every
|
||||
other exception is reported retryable, which is the safer default.
|
||||
|
||||
**Never dying.** An unexpected exception in one job must not take the process
|
||||
down, because the process is the thing that drains the queue. Errors are logged
|
||||
and the loop continues.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
from . import __version__
|
||||
from .calibrate import (
|
||||
CalibrationError,
|
||||
apply_masters,
|
||||
build_master_dark,
|
||||
build_master_flat,
|
||||
combine_frames,
|
||||
)
|
||||
from .client import ClaimLost, Client, Config, ProtocolError
|
||||
from .fits_io import FitsError, read_image, read_stack, write_master
|
||||
from .timing import TimingError, compute_bjd_tdb
|
||||
|
||||
log = logging.getLogger("caelum_reduce")
|
||||
|
||||
# Stages this worker implements. A claim for anything else is failed
|
||||
# non-retryably with a clear message rather than left to time out, so a
|
||||
# misconfigured deployment is visible immediately.
|
||||
IMPLEMENTED_STAGES = ("timing", "calibrate")
|
||||
|
||||
|
||||
class PermanentFailure(RuntimeError):
|
||||
"""A failure that will recur identically on every retry."""
|
||||
|
||||
|
||||
def software_versions():
|
||||
"""Versions to report to /runs/:id/environment.
|
||||
|
||||
What makes a run re-executable rather than merely described. Collected
|
||||
defensively because a missing optional dependency should not stop a stage
|
||||
from reporting the versions it does know.
|
||||
"""
|
||||
import importlib
|
||||
|
||||
out = {"caelum_reduce": __version__}
|
||||
for name in ("numpy", "scipy", "astropy", "photutils", "ccdproc"):
|
||||
try:
|
||||
out[name] = getattr(importlib.import_module(name), "__version__", "unknown")
|
||||
except Exception:
|
||||
out[name] = "absent"
|
||||
return out
|
||||
|
||||
|
||||
class Heartbeat:
|
||||
"""Keeps a claim alive for as long as a stage is running.
|
||||
|
||||
A daemon thread so it can never hold the process open at shutdown. Exposes
|
||||
`lost` so the caller can check, after the stage finishes, whether the claim
|
||||
survived; a result computed after the claim was reassigned must not be
|
||||
reported.
|
||||
"""
|
||||
|
||||
def __init__(self, client: Client, job_id: str, interval: float):
|
||||
self._client = client
|
||||
self._job_id = job_id
|
||||
self._interval = max(5.0, float(interval))
|
||||
self._stop = threading.Event()
|
||||
self._thread = None
|
||||
self.lost = False
|
||||
|
||||
def __enter__(self):
|
||||
self._thread = threading.Thread(target=self._run, name="heartbeat", daemon=True)
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc_info):
|
||||
self._stop.set()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5.0)
|
||||
return False
|
||||
|
||||
def _run(self):
|
||||
while not self._stop.wait(self._interval):
|
||||
try:
|
||||
self._client.heartbeat(self._job_id)
|
||||
except ClaimLost as exc:
|
||||
log.warning("claim on job %s lost: %s", self._job_id, exc)
|
||||
self.lost = True
|
||||
return
|
||||
except ProtocolError as exc:
|
||||
# A transient failure here is survivable: the claim has a TTL
|
||||
# measured in minutes and the next beat may well succeed.
|
||||
log.warning("heartbeat for job %s failed: %s", self._job_id, exc)
|
||||
|
||||
|
||||
# ---------- stage handlers ----------
|
||||
|
||||
|
||||
def run_timing(client: Client, order: dict) -> dict:
|
||||
"""The `timing` stage: BJD_TDB at mid-exposure."""
|
||||
if not order.get("can_run"):
|
||||
raise PermanentFailure(
|
||||
"the frame lacks DATE-OBS or EXPTIME, so a mid-exposure epoch cannot be computed"
|
||||
)
|
||||
frame = order["frame"]
|
||||
site = order["site"]
|
||||
# Position: prefer the solved one the server put on the frame, and fall
|
||||
# back to the work order's own fields. Either way a missing position is a
|
||||
# permanent failure, because the correction is direction-dependent.
|
||||
ra = frame.get("ra") if frame.get("ra") is not None else order.get("ra_deg")
|
||||
dec = frame.get("dec_coord") if frame.get("dec_coord") is not None else order.get("dec_deg")
|
||||
if ra is None or dec is None:
|
||||
raise PermanentFailure(
|
||||
"the frame has no sky position, and the barycentric correction is direction-dependent"
|
||||
)
|
||||
try:
|
||||
result = compute_bjd_tdb(
|
||||
date_obs=order.get("date_obs") or frame.get("date_obs"),
|
||||
exposure_time=order.get("exposure_time", frame.get("exposure_time")),
|
||||
ra_deg=float(ra),
|
||||
dec_deg=float(dec),
|
||||
latitude=site["latitude"],
|
||||
longitude=site["longitude"],
|
||||
altitude_m=site.get("altitude_m", 0.0),
|
||||
)
|
||||
except TimingError as exc:
|
||||
# Every TimingError is a refusal over bad input, which will not change
|
||||
# on a retry.
|
||||
raise PermanentFailure(str(exc)) from exc
|
||||
return result.to_dict()
|
||||
|
||||
|
||||
def run_calibrate(client: Client, order: dict) -> dict:
|
||||
"""The `calibrate` stage: apply the masters the server selected."""
|
||||
frame = order["frame"]
|
||||
masters = order.get("masters") or {}
|
||||
|
||||
raw = _download_image(client, frame.get("download_path"), "frame")
|
||||
|
||||
loaded = {}
|
||||
for kind in ("bias", "dark", "flat"):
|
||||
ref = masters.get(kind)
|
||||
if not ref:
|
||||
continue
|
||||
loaded[kind] = _download_image(client, ref.get("download_path"), f"{kind} master")
|
||||
|
||||
if not loaded:
|
||||
# The server normally skips this case at claim time; reaching it means
|
||||
# the masters were superseded between the claim and now.
|
||||
raise PermanentFailure("no calibration masters were available for this frame")
|
||||
|
||||
dark_ref = masters.get("dark") or {}
|
||||
try:
|
||||
out = apply_masters(
|
||||
raw.data,
|
||||
bias=loaded.get("bias").data if "bias" in loaded else None,
|
||||
dark=_as_master(loaded.get("dark"), dark_ref),
|
||||
flat=loaded.get("flat").data if "flat" in loaded else None,
|
||||
dark_scale=dark_ref.get("scale"),
|
||||
gain=frame.get("gain"),
|
||||
read_noise_e=_read_noise(order),
|
||||
)
|
||||
except CalibrationError as exc:
|
||||
# A shape mismatch or a refused dark scale is deterministic.
|
||||
raise PermanentFailure(str(exc)) from exc
|
||||
|
||||
finite = np.isfinite(out.data)
|
||||
return {
|
||||
"steps": out.steps,
|
||||
"dark_scale": out.dark_scale,
|
||||
"masters_applied": {k: masters[k]["id"] for k in loaded if masters.get(k)},
|
||||
# Summary statistics rather than the pixels. The calibrated frame is
|
||||
# not stored yet (that is phase 3, alongside extraction); what matters
|
||||
# here is evidence the arithmetic did something sane.
|
||||
"calibrated_median": float(np.nanmedian(out.data)),
|
||||
"calibrated_std": float(np.nanstd(out.data)),
|
||||
"median_uncertainty": float(np.nanmedian(out.uncertainty)),
|
||||
"nan_pixels": int((~finite).sum()),
|
||||
"software": software_versions(),
|
||||
}
|
||||
|
||||
|
||||
STAGE_HANDLERS = {
|
||||
"timing": run_timing,
|
||||
"calibrate": run_calibrate,
|
||||
}
|
||||
|
||||
|
||||
# ---------- the loops ----------
|
||||
|
||||
|
||||
class Worker:
|
||||
def __init__(self, client: Client, clock=time.monotonic, sleep=time.sleep):
|
||||
self.client = client
|
||||
self._clock = clock
|
||||
self._sleep = sleep
|
||||
self._next_master_sweep = 0.0
|
||||
|
||||
def run_forever(self, stop_event=None):
|
||||
"""Poll until told to stop.
|
||||
|
||||
`stop_event` is injected in tests and wired to SIGTERM in __main__, so
|
||||
a container stop drains cleanly rather than killing a stage mid-write.
|
||||
"""
|
||||
log.info("worker %s starting; stages=%s",
|
||||
self.client.config.worker_id,
|
||||
",".join(self.client.config.stages or IMPLEMENTED_STAGES))
|
||||
while not (stop_event and stop_event.is_set()):
|
||||
worked = False
|
||||
try:
|
||||
worked = self.run_once()
|
||||
except ProtocolError as exc:
|
||||
# Includes an expired token and an unreachable API. Log and
|
||||
# keep polling: the operator may be fixing it right now, and a
|
||||
# worker that exits on the first 401 has to be restarted by
|
||||
# hand.
|
||||
log.error("protocol error: %s", exc)
|
||||
except Exception:
|
||||
log.exception("unexpected error in the job loop")
|
||||
|
||||
try:
|
||||
self.maybe_build_masters()
|
||||
except ProtocolError as exc:
|
||||
log.error("master sweep failed: %s", exc)
|
||||
except Exception:
|
||||
log.exception("unexpected error in the master sweep")
|
||||
|
||||
if not worked:
|
||||
self._sleep(self.client.config.poll_interval)
|
||||
|
||||
def run_once(self) -> bool:
|
||||
"""Claim and run at most one job. Returns True if it did work."""
|
||||
order = self.client.claim()
|
||||
if not order:
|
||||
return False
|
||||
|
||||
job_id = order["job_id"]
|
||||
stage = order.get("stage")
|
||||
log.info("claimed job %s stage=%s frame=%s", job_id, stage, order.get("frame", {}).get("id"))
|
||||
|
||||
started = self._clock()
|
||||
handler = STAGE_HANDLERS.get(stage)
|
||||
if handler is None:
|
||||
# Not implemented here. Report it as permanent so the queue stops
|
||||
# offering it and an operator sees a clear message, rather than the
|
||||
# job cycling through three attempts and a timeout.
|
||||
message = (
|
||||
f"this worker does not implement the {stage!r} stage "
|
||||
f"(implemented: {', '.join(IMPLEMENTED_STAGES)})"
|
||||
)
|
||||
log.warning("%s", message)
|
||||
self.client.fail(job_id, message, retryable=False)
|
||||
return True
|
||||
|
||||
with Heartbeat(self.client, job_id, self.client.config.heartbeat_interval) as hb:
|
||||
try:
|
||||
result = handler(self.client, order)
|
||||
except PermanentFailure as exc:
|
||||
log.warning("job %s failed permanently: %s", job_id, exc)
|
||||
self._report(hb, self.client.fail, job_id, str(exc), retryable=False)
|
||||
return True
|
||||
except (FitsError, TimingError, CalibrationError) as exc:
|
||||
# These are deterministic even when not raised as
|
||||
# PermanentFailure, so do not spend retries on them.
|
||||
log.warning("job %s failed: %s", job_id, exc)
|
||||
self._report(hb, self.client.fail, job_id, str(exc), retryable=False)
|
||||
return True
|
||||
except Exception as exc:
|
||||
log.exception("job %s raised", job_id)
|
||||
self._report(hb, self.client.fail, job_id, f"{type(exc).__name__}: {exc}")
|
||||
return True
|
||||
|
||||
if hb.lost:
|
||||
# Another worker owns this job now, and its result is the one
|
||||
# that counts. Reporting ours would overwrite work that may
|
||||
# already be finished.
|
||||
log.warning("discarding the result for job %s: the claim was lost", job_id)
|
||||
return True
|
||||
|
||||
self._report(hb, self.client.complete, job_id, result)
|
||||
# Log the outcome, not just the claim. Without this an operator
|
||||
# watching the logs sees work start and never sees it finish, and
|
||||
# cannot tell a slow stage from a wedged one.
|
||||
log.info("completed job %s stage=%s in %.1fs", job_id, stage, self._clock() - started)
|
||||
if order.get("run_id"):
|
||||
try:
|
||||
self.client.record_environment(order["run_id"], software=software_versions())
|
||||
except ProtocolError as exc:
|
||||
# Provenance is valuable, but losing it must not turn a
|
||||
# successful stage into a failed one.
|
||||
log.warning("could not record the run environment: %s", exc)
|
||||
return True
|
||||
|
||||
def maybe_build_masters(self):
|
||||
"""Build any calibration masters that are ready, on a slow cadence."""
|
||||
now = self._clock()
|
||||
if now < self._next_master_sweep:
|
||||
return
|
||||
self._next_master_sweep = now + self.client.config.master_poll_interval
|
||||
|
||||
requests_ = self.client.master_requests()
|
||||
if not requests_:
|
||||
return
|
||||
log.info("%d calibration group(s) ready to combine", len(requests_))
|
||||
for group in requests_:
|
||||
try:
|
||||
self.build_master(group)
|
||||
except (FitsError, CalibrationError, ProtocolError) as exc:
|
||||
# One bad group must not stop the sweep. The group stays in
|
||||
# the request list and will be retried next sweep, which is
|
||||
# right: the cause is often a frame still being uploaded.
|
||||
log.warning("could not build a %s master: %s", group.get("kind"), exc)
|
||||
except Exception:
|
||||
log.exception("unexpected error building a %s master", group.get("kind"))
|
||||
|
||||
def build_master(self, group: dict):
|
||||
"""Combine one group of raw calibration frames and register the result."""
|
||||
kind = group["kind"]
|
||||
frame_ids = group.get("frame_ids") or []
|
||||
if len(frame_ids) < 2:
|
||||
raise CalibrationError(f"group has {len(frame_ids)} frame(s); need at least 2")
|
||||
|
||||
sources = [
|
||||
self.client.download(f"/api/reduction/worker/frames/{fid}/file") for fid in frame_ids
|
||||
]
|
||||
stack, headers = read_stack(sources)
|
||||
|
||||
if kind == "BIAS":
|
||||
master = combine_frames(stack, "BIAS")
|
||||
elif kind == "DARK":
|
||||
# Bias-subtract if a bias master is available, because only a
|
||||
# bias-subtracted dark can be scaled to another exposure time.
|
||||
bias = self._find_master_data(group, "BIAS")
|
||||
master = build_master_dark(stack, bias)
|
||||
elif kind == "FLAT":
|
||||
bias = self._find_master_data(group, "BIAS")
|
||||
master = build_master_flat(stack, master_bias=bias)
|
||||
else:
|
||||
raise CalibrationError(f"unknown master kind {kind!r}")
|
||||
|
||||
keys = {
|
||||
"telescope_id": group.get("telescope_id"),
|
||||
"instrument_id": group.get("instrument_id"),
|
||||
"filter": group.get("filter"),
|
||||
"exposure_time": group.get("exposure_time"),
|
||||
"ccd_temp_c": group.get("ccd_temp_c"),
|
||||
"binning": group.get("binning"),
|
||||
"gain": group.get("gain"),
|
||||
"valid_from": group.get("valid_from"),
|
||||
"valid_to": group.get("valid_to"),
|
||||
"frame_ids": frame_ids,
|
||||
}
|
||||
payload = write_master(master, source_headers=headers, keys=keys)
|
||||
|
||||
meta = {
|
||||
**{k: v for k, v in keys.items() if k != "frame_ids"},
|
||||
"kind": kind,
|
||||
"frame_ids": frame_ids,
|
||||
"combine_method": master.combine_method,
|
||||
"sigma_low": master.sigma_low,
|
||||
"sigma_high": master.sigma_high,
|
||||
"mean_level": master.mean_level,
|
||||
"std_level": master.std_level,
|
||||
}
|
||||
out = self.client.post_master(meta, payload, filename=f"master_{kind.lower()}.fits")
|
||||
if out.get("created"):
|
||||
log.info("registered %s master %s from %d frames", kind, out.get("id"), len(frame_ids))
|
||||
else:
|
||||
log.info("%s master already existed as %s", kind, out.get("id"))
|
||||
return out
|
||||
|
||||
# ---------- internals ----------
|
||||
|
||||
def _find_master_data(self, group, kind):
|
||||
"""The pixels of the current master of `kind` matching this group.
|
||||
|
||||
Used when building a dark or a flat, both of which want the bias
|
||||
removed from their inputs first. Returns None rather than failing when
|
||||
there is no bias yet: a dark without one is still correct at its own
|
||||
exposure time, and refusing to build anything until a bias exists
|
||||
would stall a fresh observatory on its first night.
|
||||
"""
|
||||
try:
|
||||
ref = self.client.current_master(
|
||||
group.get("telescope_id"), kind, binning=group.get("binning"),
|
||||
)
|
||||
except ProtocolError as exc:
|
||||
log.warning("could not look up a current %s master: %s", kind, exc)
|
||||
return None
|
||||
path = (ref or {}).get("download_path")
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
return read_image(self.client.download(path)).data
|
||||
except FitsError as exc:
|
||||
log.warning("the current %s master %s would not read: %s", kind, ref.get("id"), exc)
|
||||
return None
|
||||
|
||||
def _report(self, hb, fn, *args, **kwargs):
|
||||
"""Report an outcome, tolerating a claim that has already gone."""
|
||||
try:
|
||||
fn(*args, **kwargs)
|
||||
except ClaimLost as exc:
|
||||
hb.lost = True
|
||||
log.warning("could not report: %s", exc)
|
||||
|
||||
|
||||
def _download_image(client: Client, path, what):
|
||||
if not path:
|
||||
raise PermanentFailure(f"the work order carries no download path for the {what}")
|
||||
try:
|
||||
return read_image(client.download(path))
|
||||
except FitsError as exc:
|
||||
raise PermanentFailure(f"could not read the {what}: {exc}") from exc
|
||||
|
||||
|
||||
def _as_master(image, ref):
|
||||
"""Wrap a downloaded master so apply_masters knows whether it is scalable.
|
||||
|
||||
A bare array carries no `bias_subtracted` flag, and apply_masters refuses
|
||||
to scale a dark without one. The flag lives in the master's FITS header
|
||||
(BIASSUB), written by write_master, so it survives the round trip.
|
||||
"""
|
||||
if image is None:
|
||||
return None
|
||||
from .calibrate import MasterResult
|
||||
|
||||
bias_subtracted = bool(image.header.get("BIASSUB", False))
|
||||
return MasterResult(
|
||||
data=image.data,
|
||||
uncertainty=np.zeros_like(image.data),
|
||||
kind=ref.get("kind", "DARK"),
|
||||
frame_count=int(image.header.get("NCOMBINE", 0) or 0),
|
||||
combine_method=str(image.header.get("COMBMETH", "unknown")),
|
||||
sigma_low=float(image.header.get("SIGMALOW", 0) or 0),
|
||||
sigma_high=float(image.header.get("SIGMAHI", 0) or 0),
|
||||
mean_level=float(image.header.get("MEANLVL", 0) or 0),
|
||||
std_level=float(image.header.get("STDLVL", 0) or 0),
|
||||
bias_subtracted=bias_subtracted,
|
||||
)
|
||||
|
||||
|
||||
def _read_noise(order):
|
||||
"""Read noise in electrons, if anything in the work order knows it.
|
||||
|
||||
Nothing populates this yet: `instruments` has no read-noise column. Left
|
||||
as a single place to change when it does, rather than scattered defaults.
|
||||
A missing value means the uncertainty is Poisson-only, which understates
|
||||
the error in the sky background.
|
||||
"""
|
||||
instrument = order.get("instrument") or {}
|
||||
return instrument.get("read_noise_e")
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Tests for the protocol client, driven by a fake transport.
|
||||
|
||||
No HTTP and no server. The transport is a list of canned responses, so these
|
||||
tests pin the *decisions* the client makes: what it retries, what it refuses to
|
||||
retry, what it treats as a lost claim, and that the token never appears in a
|
||||
string representation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from caelum_reduce.client import (
|
||||
ClaimLost,
|
||||
Client,
|
||||
Config,
|
||||
ProtocolError,
|
||||
)
|
||||
|
||||
TOKEN = "cae_" + "0" * 48
|
||||
|
||||
|
||||
def cfg(**over):
|
||||
base = dict(base_url="http://api.test", token=TOKEN, worker_id="sidecar-1",
|
||||
stages=("timing",), max_retries=4)
|
||||
base.update(over)
|
||||
return Config(**base)
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, status, body=None, content=b"", text=None):
|
||||
self.status_code = status
|
||||
self._body = body
|
||||
self.content = content
|
||||
self.text = text if text is not None else (json.dumps(body) if body is not None else "")
|
||||
|
||||
def json(self):
|
||||
if self._body is None:
|
||||
raise ValueError("no json")
|
||||
return self._body
|
||||
|
||||
|
||||
class FakeSession:
|
||||
"""Returns queued responses in order, recording what it was asked."""
|
||||
|
||||
def __init__(self, responses):
|
||||
self._responses = list(responses)
|
||||
self.calls = []
|
||||
|
||||
def request(self, method, url, **kwargs):
|
||||
self.calls.append({"method": method, "url": url, **kwargs})
|
||||
if not self._responses:
|
||||
raise AssertionError(f"unexpected extra request: {method} {url}")
|
||||
nxt = self._responses.pop(0)
|
||||
if isinstance(nxt, Exception):
|
||||
raise nxt
|
||||
return nxt
|
||||
|
||||
|
||||
def client_with(responses, **over):
|
||||
session = FakeSession(responses)
|
||||
# sleep is a no-op so backoff does not make the suite slow.
|
||||
return Client(cfg(**over), session=session, sleep=lambda _s: None), session
|
||||
|
||||
|
||||
# ---------- configuration ----------
|
||||
|
||||
|
||||
def test_config_reads_the_environment():
|
||||
got = Config.from_env({
|
||||
"CAELUM_API_URL": "http://api.test/",
|
||||
"CAELUM_WORKER_TOKEN": TOKEN,
|
||||
"CAELUM_WORKER_ID": "box-3",
|
||||
"CAELUM_WORKER_STAGES": "timing, calibrate",
|
||||
"CAELUM_POLL_INTERVAL": "2.5",
|
||||
})
|
||||
# The trailing slash is stripped so paths concatenate cleanly.
|
||||
assert got.base_url == "http://api.test"
|
||||
assert got.worker_id == "box-3"
|
||||
assert got.stages == ("timing", "calibrate")
|
||||
assert got.poll_interval == 2.5
|
||||
|
||||
|
||||
def test_a_missing_url_or_token_fails_at_startup():
|
||||
with pytest.raises(ProtocolError, match="CAELUM_API_URL"):
|
||||
Config.from_env({"CAELUM_WORKER_TOKEN": TOKEN})
|
||||
with pytest.raises(ProtocolError, match="CAELUM_WORKER_TOKEN"):
|
||||
Config.from_env({"CAELUM_API_URL": "http://api.test"})
|
||||
|
||||
|
||||
def test_a_session_token_is_rejected_at_startup():
|
||||
# The worker endpoints refuse a JWT with a 401. Catching it here means the
|
||||
# operator gets told why instead of watching a 401 loop.
|
||||
with pytest.raises(ProtocolError, match="cae_ prefix"):
|
||||
Config.from_env({"CAELUM_API_URL": "http://api.test",
|
||||
"CAELUM_WORKER_TOKEN": "eyJhbGciOiJIUzI1NiJ9.abc.def"})
|
||||
|
||||
|
||||
def test_a_non_numeric_interval_fails_with_a_clear_message():
|
||||
with pytest.raises(ProtocolError, match="must be a number"):
|
||||
Config.from_env({"CAELUM_API_URL": "http://api.test",
|
||||
"CAELUM_WORKER_TOKEN": TOKEN,
|
||||
"CAELUM_POLL_INTERVAL": "soon"})
|
||||
|
||||
|
||||
def test_the_token_never_appears_in_a_repr():
|
||||
# The single most likely way a credential leaks into a log is a traceback
|
||||
# or a debug print of the config object.
|
||||
config = cfg()
|
||||
assert TOKEN not in repr(config)
|
||||
assert "redacted" in repr(config)
|
||||
# And inside an f-string, which is how it would actually reach a log line.
|
||||
assert TOKEN not in f"{config}"
|
||||
assert TOKEN not in f"{config!r}"
|
||||
|
||||
|
||||
# ---------- claiming ----------
|
||||
|
||||
|
||||
def test_claim_sends_the_worker_id_and_stages():
|
||||
client, session = client_with([FakeResponse(200, {"job_id": "j1", "stage": "timing"})])
|
||||
order = client.claim()
|
||||
assert order["job_id"] == "j1"
|
||||
sent = session.calls[0]
|
||||
assert sent["method"] == "POST"
|
||||
assert sent["url"] == "http://api.test/api/reduction/worker/claim"
|
||||
assert sent["json"] == {"worker_id": "sidecar-1", "stages": ["timing"]}
|
||||
assert sent["headers"]["Authorization"] == f"Bearer {TOKEN}"
|
||||
|
||||
|
||||
def test_an_idle_queue_returns_none_rather_than_raising():
|
||||
client, _ = client_with([FakeResponse(204)])
|
||||
assert client.claim() is None
|
||||
|
||||
|
||||
def test_stages_are_omitted_when_not_configured():
|
||||
# An empty list would be a different request than not asking at all.
|
||||
client, session = client_with([FakeResponse(204)], stages=())
|
||||
client.claim()
|
||||
assert "stages" not in session.calls[0]["json"]
|
||||
|
||||
|
||||
# ---------- retry policy ----------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [500, 502, 503, 504, 429])
|
||||
def test_a_retryable_status_is_retried_then_succeeds(status):
|
||||
client, session = client_with([
|
||||
FakeResponse(status, {"error": "later"}),
|
||||
FakeResponse(200, {"job_id": "j2", "stage": "timing"}),
|
||||
])
|
||||
assert client.claim()["job_id"] == "j2"
|
||||
assert len(session.calls) == 2
|
||||
|
||||
|
||||
def test_a_connection_error_is_retried():
|
||||
client, session = client_with([
|
||||
ConnectionError("dns went away"),
|
||||
FakeResponse(200, {"job_id": "j3", "stage": "timing"}),
|
||||
])
|
||||
assert client.claim()["job_id"] == "j3"
|
||||
assert len(session.calls) == 2
|
||||
|
||||
|
||||
def test_retries_are_bounded_and_the_last_error_is_reported():
|
||||
client, session = client_with(
|
||||
[FakeResponse(503, {"error": "still down"})] * 4, max_retries=4,
|
||||
)
|
||||
with pytest.raises(ProtocolError, match="after 4 attempts|503"):
|
||||
client.claim()
|
||||
assert len(session.calls) == 4
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [400, 401, 403, 404, 422])
|
||||
def test_a_client_error_is_not_retried(status):
|
||||
# A 4xx is our bug, our credential, or a stale claim. Retrying turns a
|
||||
# clear failure into a silent loop.
|
||||
client, session = client_with([FakeResponse(status, {"error": "nope"})])
|
||||
with pytest.raises(ProtocolError) as exc:
|
||||
client.claim()
|
||||
assert exc.value.status == status
|
||||
assert len(session.calls) == 1, "a 4xx must not be retried"
|
||||
|
||||
|
||||
def test_the_servers_error_and_hint_reach_the_exception_message():
|
||||
client, _ = client_with([
|
||||
FakeResponse(403, {"error": 'scopes do not include "reduction"',
|
||||
"hint": "create a token with the reduction scope"}),
|
||||
])
|
||||
with pytest.raises(ProtocolError, match="reduction.*reduction scope"):
|
||||
client.claim()
|
||||
|
||||
|
||||
def test_a_non_json_error_body_still_produces_a_message():
|
||||
client, _ = client_with([FakeResponse(502, None, text="<html>bad gateway</html>")])
|
||||
with pytest.raises(ProtocolError):
|
||||
client.claim()
|
||||
|
||||
|
||||
# ---------- lost claims ----------
|
||||
|
||||
|
||||
def test_a_409_is_a_lost_claim_not_a_generic_error():
|
||||
# The worker loop has to tell these apart: a lost claim means stop working
|
||||
# on this job, not that something is broken.
|
||||
client, _ = client_with([FakeResponse(409, {"error": "claim is held by another worker"})])
|
||||
with pytest.raises(ClaimLost, match="another worker"):
|
||||
client.heartbeat("j1")
|
||||
|
||||
|
||||
def test_a_lost_claim_is_also_a_protocol_error():
|
||||
# So a caller that only cares about "it failed" can catch one type.
|
||||
client, _ = client_with([FakeResponse(409, {"error": "job is failed, not running"})])
|
||||
with pytest.raises(ProtocolError):
|
||||
client.complete("j1", {})
|
||||
|
||||
|
||||
# ---------- reporting ----------
|
||||
|
||||
|
||||
def test_complete_sends_the_result_under_a_result_key():
|
||||
client, session = client_with([FakeResponse(200, {"ok": True})])
|
||||
client.complete("j1", {"bjd_tdb": 2461296.43})
|
||||
assert session.calls[0]["json"] == {
|
||||
"worker_id": "sidecar-1", "result": {"bjd_tdb": 2461296.43},
|
||||
}
|
||||
|
||||
|
||||
def test_fail_defaults_to_retryable_and_truncates_a_huge_error():
|
||||
client, session = client_with([FakeResponse(200, {"ok": True})])
|
||||
client.fail("j1", "x" * 5000)
|
||||
body = session.calls[0]["json"]
|
||||
assert body["retryable"] is True
|
||||
assert len(body["error"]) == 2000
|
||||
|
||||
|
||||
def test_fail_can_declare_a_permanent_failure():
|
||||
client, session = client_with([FakeResponse(200, {"ok": True})])
|
||||
client.fail("j1", "corrupt file", retryable=False)
|
||||
assert session.calls[0]["json"]["retryable"] is False
|
||||
|
||||
|
||||
def test_skip_requires_going_through_the_reason_field():
|
||||
client, session = client_with([FakeResponse(200, {"ok": True})])
|
||||
client.skip("j1", "already solved")
|
||||
assert session.calls[0]["json"]["reason"] == "already solved"
|
||||
|
||||
|
||||
def test_record_environment_sends_nothing_when_there_is_nothing_to_send():
|
||||
# Guards against a pointless request on every job.
|
||||
client, session = client_with([])
|
||||
client.record_environment("run-1")
|
||||
assert session.calls == []
|
||||
|
||||
|
||||
def test_record_environment_sends_what_it_has():
|
||||
client, session = client_with([FakeResponse(200, {"ok": True})])
|
||||
client.record_environment("run-1", software={"astropy": "8.0.1"})
|
||||
assert session.calls[0]["url"].endswith("/runs/run-1/environment")
|
||||
assert session.calls[0]["json"] == {"software": {"astropy": "8.0.1"}}
|
||||
|
||||
|
||||
# ---------- files and masters ----------
|
||||
|
||||
|
||||
def test_download_returns_raw_bytes_and_resolves_a_relative_path():
|
||||
client, session = client_with([FakeResponse(200, content=b"SIMPLE =")])
|
||||
assert client.download("/api/reduction/worker/frames/f1/file") == b"SIMPLE ="
|
||||
assert session.calls[0]["url"] == "http://api.test/api/reduction/worker/frames/f1/file"
|
||||
|
||||
|
||||
def test_master_requests_unwraps_the_requests_array():
|
||||
client, session = client_with([
|
||||
FakeResponse(200, {"requests": [{"kind": "BIAS", "frame_count": 5}], "min_frames": 5}),
|
||||
])
|
||||
got = client.master_requests(telescope_id="t1")
|
||||
assert got == [{"kind": "BIAS", "frame_count": 5}]
|
||||
assert session.calls[0]["params"]["telescope_id"] == "t1"
|
||||
|
||||
|
||||
def test_current_master_returns_none_on_204():
|
||||
# A new observatory legitimately has no bias yet, and that is not an error.
|
||||
client, _ = client_with([FakeResponse(204)])
|
||||
assert client.current_master("t1", "BIAS") is None
|
||||
|
||||
|
||||
def test_current_master_returns_the_reference():
|
||||
client, session = client_with([
|
||||
FakeResponse(200, {"id": "m1", "download_path": "/api/reduction/worker/masters/m1/file"}),
|
||||
])
|
||||
got = client.current_master("t1", "BIAS", binning="1x1")
|
||||
assert got["id"] == "m1"
|
||||
assert session.calls[0]["params"] == {"telescope_id": "t1", "kind": "BIAS", "binning": "1x1"}
|
||||
|
||||
|
||||
def test_post_master_sends_meta_as_json_text_and_the_file_as_multipart():
|
||||
client, session = client_with([FakeResponse(201, {"id": "m9", "created": True})])
|
||||
out = client.post_master({"kind": "BIAS", "frame_ids": ["a", "b"]}, b"FITSBYTES")
|
||||
assert out == {"id": "m9", "created": True}
|
||||
call = session.calls[0]
|
||||
# meta travels as a form field holding JSON, because the request is
|
||||
# multipart and cannot also be a JSON body.
|
||||
assert json.loads(call["data"]["meta"])["kind"] == "BIAS"
|
||||
assert call["files"]["file"][1] == b"FITSBYTES"
|
||||
assert call["files"]["file"][2] == "application/fits"
|
||||
@@ -0,0 +1,496 @@
|
||||
"""Tests for the poll loop and the stage dispatch.
|
||||
|
||||
The client is faked, so these pin the worker's decisions rather than HTTP:
|
||||
which failures are reported permanent, that a result is discarded when the
|
||||
claim was lost, that one bad job does not kill the loop, and that a master
|
||||
gets built and registered end to end.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from astropy.io import fits
|
||||
|
||||
from caelum_reduce.calibrate import combine_frames
|
||||
from caelum_reduce.client import ClaimLost, Config, ProtocolError
|
||||
from caelum_reduce.fits_io import read_image, write_master
|
||||
from caelum_reduce.worker import (
|
||||
IMPLEMENTED_STAGES,
|
||||
Heartbeat,
|
||||
PermanentFailure,
|
||||
Worker,
|
||||
run_calibrate,
|
||||
run_timing,
|
||||
software_versions,
|
||||
)
|
||||
|
||||
SHAPE = (12, 14)
|
||||
|
||||
|
||||
def fits_bytes(data, **cards):
|
||||
hdu = fits.PrimaryHDU(data=np.asarray(data, dtype=np.float32))
|
||||
for k, v in cards.items():
|
||||
hdu.header[k] = v
|
||||
import io
|
||||
|
||||
buf = io.BytesIO()
|
||||
fits.HDUList([hdu]).writeto(buf, overwrite=True)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
class FakeClient:
|
||||
"""Records what the worker did, and serves canned downloads."""
|
||||
|
||||
def __init__(self, orders=None, files=None, masters=None, requests_=None):
|
||||
self.config = Config(base_url="http://api.test", token="cae_" + "0" * 48,
|
||||
worker_id="sidecar-1", stages=IMPLEMENTED_STAGES,
|
||||
poll_interval=0, heartbeat_interval=3600,
|
||||
master_poll_interval=0)
|
||||
self._orders = list(orders or [])
|
||||
self.files = dict(files or {})
|
||||
self._current_masters = dict(masters or {})
|
||||
self._requests = list(requests_ or [])
|
||||
self.completed = []
|
||||
self.failed = []
|
||||
self.skipped = []
|
||||
self.posted_masters = []
|
||||
self.environments = []
|
||||
self.heartbeats = 0
|
||||
self.raise_on_heartbeat = None
|
||||
|
||||
def claim(self):
|
||||
return self._orders.pop(0) if self._orders else None
|
||||
|
||||
def heartbeat(self, job_id):
|
||||
self.heartbeats += 1
|
||||
if self.raise_on_heartbeat:
|
||||
raise self.raise_on_heartbeat
|
||||
|
||||
def complete(self, job_id, result):
|
||||
self.completed.append((job_id, result))
|
||||
|
||||
def fail(self, job_id, error, retryable=True):
|
||||
self.failed.append((job_id, str(error), retryable))
|
||||
|
||||
def skip(self, job_id, reason):
|
||||
self.skipped.append((job_id, reason))
|
||||
|
||||
def record_environment(self, run_id, solver=None, software=None):
|
||||
self.environments.append((run_id, solver, software))
|
||||
|
||||
def download(self, path):
|
||||
if path not in self.files:
|
||||
raise AssertionError(f"unexpected download: {path}")
|
||||
return self.files[path]
|
||||
|
||||
def master_requests(self, telescope_id=None, min_frames=None):
|
||||
return list(self._requests)
|
||||
|
||||
def current_master(self, telescope_id, kind, binning=None, filter_name=None):
|
||||
return self._current_masters.get(kind)
|
||||
|
||||
def post_master(self, meta, payload, filename="master.fits"):
|
||||
self.posted_masters.append((meta, payload, filename))
|
||||
return {"id": f"m{len(self.posted_masters)}", "created": True}
|
||||
|
||||
|
||||
def timing_order(**over):
|
||||
order = {
|
||||
"job_id": "j-timing", "run_id": "r1", "stage": "timing", "attempt": 1,
|
||||
"frame": {"id": "f1", "download_path": "/f1", "exposure_time": 600,
|
||||
"date_obs": "2026-09-12T22:14:03", "ra": 10.6847, "dec_coord": 41.2687},
|
||||
"site": {"latitude": 34.5, "longitude": -120.5, "altitude_m": 1500},
|
||||
"date_obs": "2026-09-12T22:14:03", "exposure_time": 600,
|
||||
"can_run": True, "epoch_convention": "mid_exposure",
|
||||
}
|
||||
order.update(over)
|
||||
return order
|
||||
|
||||
|
||||
# ---------- the timing stage ----------
|
||||
|
||||
|
||||
def test_the_timing_stage_returns_a_bjd():
|
||||
out = run_timing(FakeClient(), timing_order())
|
||||
assert out["epoch_convention"] == "mid_exposure"
|
||||
assert 2461296 < out["bjd_tdb"] < 2461297, out["bjd_tdb"]
|
||||
assert "light_travel_sec" in out
|
||||
|
||||
|
||||
def test_timing_refuses_permanently_when_it_cannot_run():
|
||||
with pytest.raises(PermanentFailure, match="DATE-OBS or EXPTIME"):
|
||||
run_timing(FakeClient(), timing_order(can_run=False))
|
||||
|
||||
|
||||
def test_timing_refuses_permanently_without_a_position():
|
||||
order = timing_order()
|
||||
order["frame"]["ra"] = None
|
||||
order["frame"]["dec_coord"] = None
|
||||
with pytest.raises(PermanentFailure, match="direction-dependent"):
|
||||
run_timing(FakeClient(), order)
|
||||
|
||||
|
||||
def test_a_timing_refusal_over_bad_input_is_permanent_not_retryable():
|
||||
# Every TimingError is a refusal over input that will not change, so
|
||||
# spending three attempts on it only delays the message.
|
||||
order = timing_order(date_obs="2026-09-12T22:14:03+02:00")
|
||||
with pytest.raises(PermanentFailure, match="offset"):
|
||||
run_timing(FakeClient(), order)
|
||||
|
||||
|
||||
# ---------- the calibrate stage ----------
|
||||
|
||||
|
||||
def calibrate_setup(dark_scale=None, bias_subtracted=True):
|
||||
bias = combine_frames(np.full((6, *SHAPE), 500.0), "BIAS")
|
||||
dark = combine_frames(np.full((6, *SHAPE), 40.0), "DARK", bias_subtracted=bias_subtracted)
|
||||
flat = combine_frames(np.full((6, *SHAPE), 1.0), "FLAT")
|
||||
|
||||
files = {
|
||||
"/f1": fits_bytes(np.full(SHAPE, 500.0 + 40.0 + 800.0), GAIN=2.0),
|
||||
"/mb": write_master(bias),
|
||||
"/md": write_master(dark),
|
||||
"/mf": write_master(flat),
|
||||
}
|
||||
order = {
|
||||
"job_id": "j-cal", "run_id": "r1", "stage": "calibrate", "attempt": 1,
|
||||
"frame": {"id": "f1", "download_path": "/f1", "gain": 2.0},
|
||||
"site": {"latitude": 34.5, "longitude": -120.5, "altitude_m": 0},
|
||||
"instrument": {"id": "cam", "pixel_size_um": 9.0},
|
||||
"masters": {
|
||||
"bias": {"id": "mb", "kind": "BIAS", "download_path": "/mb"},
|
||||
"dark": {"id": "md", "kind": "DARK", "download_path": "/md", **({"scale": dark_scale} if dark_scale else {})},
|
||||
"flat": {"id": "mf", "kind": "FLAT", "download_path": "/mf"},
|
||||
},
|
||||
"complete": True,
|
||||
}
|
||||
return FakeClient(files=files), order
|
||||
|
||||
|
||||
def test_the_calibrate_stage_applies_the_masters_and_reports_statistics():
|
||||
client, order = calibrate_setup()
|
||||
out = run_calibrate(client, order)
|
||||
assert abs(out["calibrated_median"] - 800.0) < 1.0, out["calibrated_median"]
|
||||
assert out["median_uncertainty"] > 0
|
||||
assert out["nan_pixels"] == 0
|
||||
assert set(out["masters_applied"]) == {"bias", "dark", "flat"}
|
||||
assert "bias subtracted" in out["steps"]
|
||||
assert out["software"]["caelum_reduce"]
|
||||
|
||||
|
||||
def test_the_bias_subtracted_flag_survives_the_fits_round_trip():
|
||||
# This is what makes a downloaded dark scalable. The flag lives in the
|
||||
# master's BIASSUB header card, written by write_master, and without it
|
||||
# apply_masters would refuse every scale.
|
||||
client, order = calibrate_setup(dark_scale=1.5, bias_subtracted=True)
|
||||
out = run_calibrate(client, order)
|
||||
assert out["dark_scale"] == 1.5
|
||||
assert any("scaled by 1.500" in s for s in out["steps"]), out["steps"]
|
||||
|
||||
|
||||
def test_scaling_a_dark_that_was_not_bias_subtracted_is_a_permanent_failure():
|
||||
# The round trip has to preserve the refusal too, not just the success.
|
||||
client, order = calibrate_setup(dark_scale=2.0, bias_subtracted=False)
|
||||
with pytest.raises(PermanentFailure, match="not marked bias_subtracted"):
|
||||
run_calibrate(client, order)
|
||||
|
||||
|
||||
def test_an_unreadable_frame_is_a_permanent_failure():
|
||||
client = FakeClient(files={"/f1": b"not a fits file at all"})
|
||||
order = {"job_id": "j", "stage": "calibrate",
|
||||
"frame": {"id": "f1", "download_path": "/f1"},
|
||||
"masters": {"bias": {"id": "b", "download_path": "/f1"}}}
|
||||
with pytest.raises(PermanentFailure, match="could not read"):
|
||||
run_calibrate(client, order)
|
||||
|
||||
|
||||
def test_a_work_order_with_no_download_path_is_a_permanent_failure():
|
||||
client = FakeClient()
|
||||
order = {"job_id": "j", "stage": "calibrate", "frame": {"id": "f1"}, "masters": {}}
|
||||
with pytest.raises(PermanentFailure, match="no download path"):
|
||||
run_calibrate(client, order)
|
||||
|
||||
|
||||
# ---------- the loop ----------
|
||||
|
||||
|
||||
def test_run_once_returns_false_on_an_idle_queue():
|
||||
worker = Worker(FakeClient())
|
||||
assert worker.run_once() is False
|
||||
|
||||
|
||||
def test_a_successful_job_is_completed_and_the_environment_recorded():
|
||||
client = FakeClient(orders=[timing_order()])
|
||||
worker = Worker(client)
|
||||
assert worker.run_once() is True
|
||||
assert len(client.completed) == 1
|
||||
job_id, result = client.completed[0]
|
||||
assert job_id == "j-timing"
|
||||
assert "bjd_tdb" in result
|
||||
# Provenance: what produced this number.
|
||||
assert client.environments and client.environments[0][0] == "r1"
|
||||
|
||||
|
||||
def test_a_completion_is_logged_with_its_duration(caplog):
|
||||
# An operator watching logs must be able to tell a slow stage from a
|
||||
# wedged one, which means the outcome has to be logged, not just the claim.
|
||||
import logging
|
||||
|
||||
client = FakeClient(orders=[timing_order()])
|
||||
with caplog.at_level(logging.INFO, logger="caelum_reduce"):
|
||||
Worker(client).run_once()
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any("claimed job" in m for m in messages), messages
|
||||
assert any("completed job" in m and "timing" in m for m in messages), messages
|
||||
|
||||
|
||||
def test_an_unimplemented_stage_is_failed_as_permanent_with_a_clear_message():
|
||||
# Not left to time out. A misconfigured deployment should be obvious.
|
||||
client = FakeClient(orders=[{"job_id": "j-x", "stage": "astrometry", "frame": {"id": "f"}}])
|
||||
Worker(client).run_once()
|
||||
assert len(client.failed) == 1
|
||||
job_id, error, retryable = client.failed[0]
|
||||
assert job_id == "j-x"
|
||||
assert retryable is False
|
||||
assert "does not implement" in error
|
||||
assert "timing" in error, "the message should say what IS implemented"
|
||||
|
||||
|
||||
def test_a_permanent_stage_failure_is_reported_non_retryable():
|
||||
client = FakeClient(orders=[timing_order(can_run=False)])
|
||||
Worker(client).run_once()
|
||||
assert client.failed[0][2] is False
|
||||
|
||||
|
||||
def test_an_unexpected_exception_is_reported_retryable():
|
||||
# The safer default: we do not know it will recur.
|
||||
client = FakeClient(orders=[timing_order()])
|
||||
order = client._orders[0]
|
||||
del order["site"] # provokes a KeyError inside the stage
|
||||
Worker(client).run_once()
|
||||
assert len(client.failed) == 1
|
||||
assert client.failed[0][2] is True
|
||||
assert "KeyError" in client.failed[0][1]
|
||||
|
||||
|
||||
def test_a_result_is_discarded_when_the_claim_was_lost():
|
||||
# Another worker owns the job now and its result is the one that counts.
|
||||
# Completing over it could overwrite work that already finished.
|
||||
client = FakeClient(orders=[timing_order()])
|
||||
|
||||
class LosingWorker(Worker):
|
||||
def run_once(self):
|
||||
# Drive the same path but force the heartbeat to have lost it.
|
||||
order = self.client.claim()
|
||||
from caelum_reduce.worker import STAGE_HANDLERS
|
||||
|
||||
hb = Heartbeat(self.client, order["job_id"], 3600)
|
||||
result = STAGE_HANDLERS[order["stage"]](self.client, order)
|
||||
hb.lost = True
|
||||
if hb.lost:
|
||||
return True
|
||||
self.client.complete(order["job_id"], result)
|
||||
return True
|
||||
|
||||
LosingWorker(client).run_once()
|
||||
assert client.completed == [], "a stale result must not be reported"
|
||||
|
||||
|
||||
def test_reporting_a_result_tolerates_a_claim_that_has_already_gone():
|
||||
# complete() raising ClaimLost must not crash the loop.
|
||||
client = FakeClient(orders=[timing_order()])
|
||||
|
||||
def losing_complete(job_id, result):
|
||||
raise ClaimLost("claim is held by another worker", 409, {})
|
||||
|
||||
client.complete = losing_complete
|
||||
Worker(client).run_once() # must not raise
|
||||
|
||||
|
||||
def test_the_loop_survives_a_protocol_error_and_keeps_polling():
|
||||
calls = {"n": 0}
|
||||
client = FakeClient()
|
||||
|
||||
def flaky_claim():
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
raise ProtocolError("api unreachable")
|
||||
return None
|
||||
|
||||
client.claim = flaky_claim
|
||||
stop = threading.Event()
|
||||
worker = Worker(client, sleep=lambda _s: stop.set())
|
||||
worker.run_forever(stop_event=stop)
|
||||
assert calls["n"] >= 1, "the loop should have kept going after the error"
|
||||
|
||||
|
||||
def test_run_forever_exits_when_the_stop_event_is_set():
|
||||
client = FakeClient()
|
||||
stop = threading.Event()
|
||||
stop.set()
|
||||
# Returns immediately rather than polling once first.
|
||||
Worker(client, sleep=lambda _s: None).run_forever(stop_event=stop)
|
||||
assert client.completed == []
|
||||
|
||||
|
||||
# ---------- master building ----------
|
||||
|
||||
|
||||
def bias_group(n=5):
|
||||
return {
|
||||
"kind": "BIAS", "telescope_id": "t1", "instrument_id": "cam",
|
||||
"filter": None, "exposure_time": None, "ccd_temp_c": -15.0,
|
||||
"binning": "1x1", "gain": 1.6,
|
||||
"frame_ids": [f"b{i}" for i in range(n)],
|
||||
"frame_count": n,
|
||||
"valid_from": "2026-09-10 19:00:00", "valid_to": "2026-09-10 19:05:00",
|
||||
}
|
||||
|
||||
|
||||
def test_a_bias_master_is_built_and_registered_with_its_keys():
|
||||
group = bias_group()
|
||||
files = {
|
||||
f"/api/reduction/worker/frames/{fid}/file": fits_bytes(np.full(SHAPE, 500.0))
|
||||
for fid in group["frame_ids"]
|
||||
}
|
||||
client = FakeClient(files=files, requests_=[group])
|
||||
Worker(client).maybe_build_masters()
|
||||
|
||||
assert len(client.posted_masters) == 1
|
||||
meta, payload, filename = client.posted_masters[0]
|
||||
assert meta["kind"] == "BIAS"
|
||||
assert meta["frame_ids"] == group["frame_ids"]
|
||||
assert meta["telescope_id"] == "t1"
|
||||
assert meta["binning"] == "1x1"
|
||||
assert abs(meta["mean_level"] - 500.0) < 1.0
|
||||
assert filename == "master_bias.fits"
|
||||
|
||||
# The registered file is a readable FITS with an uncertainty extension.
|
||||
img = read_image(payload)
|
||||
assert img.shape == SHAPE
|
||||
assert img.header["NCOMBINE"] == 5
|
||||
assert img.header["IMAGETYP"].lower().startswith("master")
|
||||
|
||||
|
||||
def test_a_dark_master_is_bias_subtracted_when_a_bias_exists():
|
||||
# Only a bias-subtracted dark can be scaled, so building one this way is
|
||||
# what keeps it useful beyond its own exposure time.
|
||||
group = {**bias_group(), "kind": "DARK", "exposure_time": 300.0,
|
||||
"frame_ids": [f"d{i}" for i in range(5)]}
|
||||
files = {
|
||||
f"/api/reduction/worker/frames/{fid}/file": fits_bytes(np.full(SHAPE, 560.0))
|
||||
for fid in group["frame_ids"]
|
||||
}
|
||||
files["/api/reduction/worker/masters/mb/file"] = write_master(
|
||||
combine_frames(np.full((6, *SHAPE), 500.0), "BIAS")
|
||||
)
|
||||
client = FakeClient(
|
||||
files=files, requests_=[group],
|
||||
masters={"BIAS": {"id": "mb", "download_path": "/api/reduction/worker/masters/mb/file"}},
|
||||
)
|
||||
Worker(client).maybe_build_masters()
|
||||
|
||||
meta, payload, _ = client.posted_masters[0]
|
||||
assert meta["kind"] == "DARK"
|
||||
# 560 raw minus a 500 bias leaves 60 ADU of dark current.
|
||||
assert abs(meta["mean_level"] - 60.0) < 1.0, meta["mean_level"]
|
||||
assert read_image(payload).header["BIASSUB"] is True
|
||||
|
||||
|
||||
def test_a_dark_still_builds_when_there_is_no_bias_yet():
|
||||
# A fresh observatory must not be stalled waiting for a bias. The dark is
|
||||
# correct at its own exposure time, and the master records that it cannot
|
||||
# be scaled.
|
||||
group = {**bias_group(), "kind": "DARK", "exposure_time": 300.0,
|
||||
"frame_ids": [f"d{i}" for i in range(5)]}
|
||||
files = {
|
||||
f"/api/reduction/worker/frames/{fid}/file": fits_bytes(np.full(SHAPE, 560.0))
|
||||
for fid in group["frame_ids"]
|
||||
}
|
||||
client = FakeClient(files=files, requests_=[group], masters={})
|
||||
Worker(client).maybe_build_masters()
|
||||
|
||||
meta, payload, _ = client.posted_masters[0]
|
||||
assert abs(meta["mean_level"] - 560.0) < 1.0, "the bias should still be in there"
|
||||
assert read_image(payload).header["BIASSUB"] is False
|
||||
|
||||
|
||||
def test_a_flat_master_is_normalized_before_registration():
|
||||
group = {**bias_group(), "kind": "FLAT", "filter": "R", "exposure_time": 4.0,
|
||||
"frame_ids": [f"fl{i}" for i in range(5)]}
|
||||
files = {
|
||||
f"/api/reduction/worker/frames/{fid}/file": fits_bytes(np.full(SHAPE, 18000.0))
|
||||
for fid in group["frame_ids"]
|
||||
}
|
||||
client = FakeClient(files=files, requests_=[group], masters={})
|
||||
Worker(client).maybe_build_masters()
|
||||
|
||||
meta, _, _ = client.posted_masters[0]
|
||||
assert meta["filter"] == "R"
|
||||
assert abs(meta["mean_level"] - 1.0) < 1e-6, "a flat must be normalized to a median of 1"
|
||||
|
||||
|
||||
def test_a_group_of_mismatched_shapes_is_skipped_without_stopping_the_sweep():
|
||||
# A frame still being uploaded, or a grouping bug upstream. One bad group
|
||||
# must not stop the others, and it stays in the request list for next time.
|
||||
bad = {**bias_group(), "frame_ids": ["x1", "x2"]}
|
||||
good = {**bias_group(), "frame_ids": ["g1", "g2", "g3"]}
|
||||
files = {
|
||||
"/api/reduction/worker/frames/x1/file": fits_bytes(np.full((10, 10), 500.0)),
|
||||
"/api/reduction/worker/frames/x2/file": fits_bytes(np.full((12, 12), 500.0)),
|
||||
}
|
||||
for fid in good["frame_ids"]:
|
||||
files[f"/api/reduction/worker/frames/{fid}/file"] = fits_bytes(np.full(SHAPE, 500.0))
|
||||
|
||||
client = FakeClient(files=files, requests_=[bad, good], masters={})
|
||||
Worker(client).maybe_build_masters()
|
||||
assert len(client.posted_masters) == 1, "the good group should still have been built"
|
||||
assert client.posted_masters[0][0]["frame_ids"] == good["frame_ids"]
|
||||
|
||||
|
||||
def test_a_group_with_one_frame_is_refused():
|
||||
group = {**bias_group(), "frame_ids": ["only"]}
|
||||
client = FakeClient(files={}, requests_=[group], masters={})
|
||||
# Raises inside build_master, and the sweep swallows it. No master posted.
|
||||
Worker(client).maybe_build_masters()
|
||||
assert client.posted_masters == []
|
||||
|
||||
|
||||
def test_the_master_sweep_respects_its_interval():
|
||||
group = bias_group()
|
||||
files = {
|
||||
f"/api/reduction/worker/frames/{fid}/file": fits_bytes(np.full(SHAPE, 500.0))
|
||||
for fid in group["frame_ids"]
|
||||
}
|
||||
client = FakeClient(files=files, requests_=[group], masters={})
|
||||
client.config = Config(**{**client.config.__dict__, "master_poll_interval": 900.0})
|
||||
|
||||
now = {"t": 1000.0}
|
||||
worker = Worker(client, clock=lambda: now["t"])
|
||||
worker.maybe_build_masters()
|
||||
assert len(client.posted_masters) == 1
|
||||
|
||||
# Immediately after, the sweep must not run again: re-combining the same
|
||||
# group every few seconds would be pure waste.
|
||||
worker.maybe_build_masters()
|
||||
assert len(client.posted_masters) == 1
|
||||
|
||||
now["t"] += 901.0
|
||||
worker.maybe_build_masters()
|
||||
assert len(client.posted_masters) == 2
|
||||
|
||||
|
||||
# ---------- provenance ----------
|
||||
|
||||
|
||||
def test_software_versions_reports_what_is_installed_and_does_not_crash_on_gaps():
|
||||
got = software_versions()
|
||||
assert got["caelum_reduce"]
|
||||
assert got["numpy"] != "absent"
|
||||
assert got["astropy"] != "absent"
|
||||
# Every key is a string, so the whole dict is JSON-serializable for the
|
||||
# environment POST.
|
||||
assert all(isinstance(v, str) for v in got.values())
|
||||
Reference in New Issue
Block a user