feat(sidecar): the Python reduction sidecar, timing and calibration done
Phase 2 of item L. Two stages implemented with 53 tests, plus the protocol document so someone can write a worker without reading the server. sidecar/caelum_reduce/timing.py (item O) - UTC at the observatory to BJD_TDB at mid-exposure, via astropy. - Refuses rather than guesses. A missing exposure time, a missing target position, a DATE-OBS carrying a UTC offset: all rejected, because a wrong epoch is undetectable downstream. - The tests assert physics, not just that a float comes back: the correction is bounded by the light-crossing time of Earth's orbit, reverses sign between the solstices for a target in the ecliptic plane, nearly vanishes at quadrature, almost disappears toward the ecliptic poles, and two sites on opposite sides of Earth agree to within the light travel time across Earth's diameter. - That thoroughness is the point. A hand-rolled approximation is wrong by ~0.1s and the error is PERIODIC with a one-year cycle, because it tracks Earth's orbit. A periodic error in a data set about periodic phenomena does not read as a bug, it reads as signal. sidecar/caelum_reduce/calibrate.py (items L and N) - Master bias/dark/flat by sigma-clipped median, and the full reduction equation applied with uncertainties propagated: var = counts_e + rn^2 through the gain, variances adding under subtraction, relative errors adding under division by the flat. - Two refusals that matter more than the arithmetic. Scaling a dark not marked bias_subtracted is rejected, because dark current scales with exposure time and the bias does not, so scaling a bias-inclusive dark multiplies the bias too; this is the most common calibration mistake and the result looks entirely reasonable. And a master whose shape does not match the frame is rejected rather than allowed to broadcast. - A dead column gets kept rather than clipped to a NaN hole, since a hole would propagate into every frame the master touches. Three things learned while testing, all recorded in comments - Single-pass sigma clipping does NOT catch an evenly split stack: 8 frames at one level plus 8 at another clips nothing, because the split inflates the stack's own standard deviation until both groups look like one-sigma deviations. Pinned as a test so the limitation stays visible; preventing it is the Node-side grouping rules' job. - The barycentric correction extremes fall at the SOLSTICES for a target at RA 6h, not the equinoxes, where Earth is perpendicular to the line of sight. My first version of that test asserted the wrong dates. - A flat normalizes by its median, not its peak, so the centre comes out above 1. Assert the centre-to-corner ratio, which is the physical quantity, not either absolute value. SEP dropped in favour of photutils. Same capability, pure Python over numpy so wheels exist everywhere, already an astropy-affiliated dependency. SEP is a C extension with no wheel for Python 3.14. docs/reduction-worker-protocol.md - All 10 endpoints, the stage dependency table, the per-stage payloads, and the three things a worker author must get right: subtract bias before scaling a dark, never report a partial WCS, never report a magnitude without an uncertainty. Not done in phase 2 yet: the worker poll loop (so this is a tested library, not a running service), a Dockerfile, and a compose entry. astrometry, extract and photometry stages remain unimplemented; astrometry needs a solver binary in the image. Verified: 233 server + 22 client + 53 sidecar tests pass, lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
67661c8569
commit
aa62529ea7
@@ -62,3 +62,8 @@ Thumbs.db
|
||||
# Local secrets (GitLab PAT, VPS credential) — never commit
|
||||
.pat
|
||||
.vps
|
||||
|
||||
# Sidecar: local venv and pytest cache
|
||||
sidecar/.venv/
|
||||
sidecar/.pytest_cache/
|
||||
sidecar/**/__pycache__/
|
||||
|
||||
@@ -193,6 +193,26 @@ cd server && npm run migrate -- --dump ../sql
|
||||
|
||||
These dumps live locally only (gitignored) so they never end up in the public repo.
|
||||
|
||||
## Reduction pipeline
|
||||
|
||||
Frames that land in the archive go through a reduction pipeline: header
|
||||
ingest, calibration against master bias/dark/flat frames, plate solving into a
|
||||
stored WCS, source extraction, photometry, and barycentric time correction.
|
||||
|
||||
Node owns the job queue, the rules deciding which calibration masters apply to
|
||||
which frame, and the provenance record. A Python sidecar
|
||||
([sidecar/](sidecar/)) does the arithmetic on pixels, because astropy and
|
||||
photutils are validated and a hand-rolled aperture sum produces plausible wrong
|
||||
answers.
|
||||
|
||||
Every measurement points at the `reduction_runs` row that produced it, which
|
||||
records the pipeline version, the exact masters applied, and the solver and
|
||||
software versions. Re-reducing a frame creates a new run rather than editing
|
||||
the old one, so a published measurement keeps pointing at what made it.
|
||||
|
||||
To write your own worker, see
|
||||
[docs/reduction-worker-protocol.md](docs/reduction-worker-protocol.md).
|
||||
|
||||
## Repo layout
|
||||
|
||||
```
|
||||
@@ -200,6 +220,8 @@ caelum/
|
||||
├── client/ Vue 3 + Vite SPA
|
||||
├── server/ Express API + telescope drivers
|
||||
├── nox/ The Nox language (parser, runtime, vscode, docs, examples)
|
||||
├── sidecar/ Python reduction sidecar (astropy, photutils)
|
||||
├── docs/ Protocol and architecture references
|
||||
├── sql/ Database dumps and migration helpers
|
||||
├── scripts/ Dev and release tooling
|
||||
├── server.js Top-level entry point (run by the Docker image)
|
||||
|
||||
@@ -217,6 +217,22 @@ sidecar is told which masters to apply rather than deciding.
|
||||
**Estimate:** Phase 2 is a substantial session on its own. Phases 3 and 4 each
|
||||
similar. Worth doing one phase per session with a QA pass between.
|
||||
|
||||
**Progress (2026-09-12):**
|
||||
- The worker protocol is done and tested: `routes/reduction-worker.js`, 10
|
||||
endpoints, behind a `reduction`-scoped token that is deliberately not
|
||||
account-wide write. See [docs/reduction-worker-protocol.md](docs/reduction-worker-protocol.md).
|
||||
- `sidecar/` exists with two stages implemented and 53 tests: `timing.py`
|
||||
(UTC to BJD_TDB at mid-exposure, which is also item O) and `calibrate.py`
|
||||
(master bias/dark/flat, applied with uncertainties propagated, which is
|
||||
also item N for this stage).
|
||||
- Still to do in phase 2: the worker poll loop, so the package becomes a
|
||||
running service rather than a tested library, and a Dockerfile plus a
|
||||
compose entry.
|
||||
- SEP was dropped in favour of photutils. Same capability, pure Python over
|
||||
numpy so wheels exist everywhere, and it is the astropy-affiliated package
|
||||
we already depend on. SEP is a C extension and had no wheel for Python
|
||||
3.14.
|
||||
|
||||
## M. Cross-site photometric standardization
|
||||
|
||||
**What:** Turn instrumental magnitudes into standard-system magnitudes. Per-frame
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
# Reduction worker protocol
|
||||
|
||||
The contract between the Caelum API and a reduction sidecar. Mounted at
|
||||
`/api/reduction/worker`. Source:
|
||||
[server/src/routes/reduction-worker.js](../server/src/routes/reduction-worker.js).
|
||||
|
||||
This document exists so someone can implement a worker without reading the
|
||||
server. The reference implementation lives in [sidecar/](../sidecar/).
|
||||
|
||||
## Shape of the thing
|
||||
|
||||
Node owns the queue, decides which calibration masters apply to which frame,
|
||||
and keeps the provenance record. The worker does arithmetic on pixels. It polls
|
||||
for work, does one stage, reports back, repeats.
|
||||
|
||||
```
|
||||
worker API
|
||||
| POST /claim |
|
||||
|------------------------------------>| claim a job, build a work order
|
||||
|<------------------------------------| 200 work order, or 204 when idle
|
||||
| |
|
||||
| GET /frames/:id/file |
|
||||
|------------------------------------>| download the pixels
|
||||
| GET /masters/:id/file |
|
||||
|------------------------------------>| download each master it was given
|
||||
| |
|
||||
| POST /jobs/:id/heartbeat | (repeat on a long stage)
|
||||
|------------------------------------>|
|
||||
| |
|
||||
| POST /jobs/:id/complete |
|
||||
|------------------------------------>| result validated, then stored
|
||||
```
|
||||
|
||||
A claim response is a *complete* work order. A worker should never need a
|
||||
second request to learn a filter name or a site's latitude. Same principle as
|
||||
the observatory-daemon payload in `routes/jobs.js`.
|
||||
|
||||
## Authentication
|
||||
|
||||
Every endpoint requires an `Authorization: Bearer cae_...` API token whose
|
||||
scopes include `reduction`. Nothing else is accepted: a user session (JWT) is
|
||||
refused with 401, and a `write`- or `daemon`-scoped token is refused with 403.
|
||||
|
||||
The `reduction` scope is **not** account-wide write. A token holding only that
|
||||
scope is refused by the normal auth middleware for every mutation outside these
|
||||
routes, so it cannot delete a frame or post a comment. That is deliberate: a
|
||||
sidecar runs unattended in a container, which makes its credential the most
|
||||
likely one to leak, and it should be able to do exactly its job and nothing
|
||||
else.
|
||||
|
||||
Create one from the Caelum UI or:
|
||||
|
||||
```sh
|
||||
curl -X POST https://caelum.network/api/tokens \
|
||||
-H "Authorization: Bearer <your session token>" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"reduction sidecar","scopes":"reduction"}'
|
||||
```
|
||||
|
||||
The token is shown once. Only its SHA-256 hash is stored, so a lost token is
|
||||
revoked rather than recovered.
|
||||
|
||||
## Stages
|
||||
|
||||
| Stage | Runs on | Depends on |
|
||||
| --- | --- | --- |
|
||||
| `header` | the Node server | nothing |
|
||||
| `timing` | sidecar | `header` |
|
||||
| `calibrate` | sidecar | `header` |
|
||||
| `astrometry` | sidecar | `calibrate` |
|
||||
| `extract` | sidecar | `astrometry` |
|
||||
| `photometry` | sidecar | `extract` |
|
||||
|
||||
`header` is not claimable by a worker. A job is only offered once its
|
||||
prerequisite stage is `done` or `skipped`, so a worker never has to check
|
||||
ordering itself.
|
||||
|
||||
## `POST /claim`
|
||||
|
||||
```json
|
||||
{ "worker_id": "sidecar-1", "stages": ["calibrate", "astrometry"] }
|
||||
```
|
||||
|
||||
`stages` is optional and declares what this worker can run. Naming a stage the
|
||||
server owns (`header`) is a 400 rather than a silent empty claim, because it is
|
||||
a configuration mistake worth seeing.
|
||||
|
||||
Declaring `stages` matters more than it looks. The filter is applied inside the
|
||||
claim query, so a worker is never handed a stage it cannot run. An earlier
|
||||
version claimed first and handed the job back, which cost the job one of its
|
||||
three attempts and a 30-second backoff; with two specialized workers polling, a
|
||||
job could burn every attempt without anyone trying to run it.
|
||||
|
||||
**204** when there is no work. Not an empty 200: a worker polling every few
|
||||
seconds should not have to parse a body to learn the queue is idle.
|
||||
|
||||
**200** with a work order:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "...", "run_id": "...", "stage": "calibrate", "attempt": 1,
|
||||
"frame": {
|
||||
"id": "...", "filename": "m31_Ha_001.fits",
|
||||
"download_path": "/api/reduction/worker/frames/<id>/file",
|
||||
"frame_type": "EXPOSE", "target_name": "M31", "filter": "Ha",
|
||||
"exposure_time": 600, "date_obs": "2026-09-12T22:14:03",
|
||||
"ccd_temp_c": -14.8, "gain": 1.6, "binning": "1x1"
|
||||
},
|
||||
"site": { "id": "...", "latitude": 34.5, "longitude": -120.5, "altitude_m": 1500 },
|
||||
"telescope": { "id": "...", "aperture_mm": 355.6, "focal_length_mm": 2563 },
|
||||
"instrument": { "id": "...", "pixel_size_um": 9.0, "filters": ["L","R","G","B","Ha"] }
|
||||
}
|
||||
```
|
||||
|
||||
`download_path` is relative. Resolve it against your own configured base URL,
|
||||
so the same payload works whether you reach the API over the compose network or
|
||||
a tunnel.
|
||||
|
||||
Plus a stage-specific part:
|
||||
|
||||
**`calibrate`** carries the masters to apply and why they were chosen.
|
||||
|
||||
```json
|
||||
{
|
||||
"masters": {
|
||||
"bias": { "id": "...", "kind": "BIAS", "download_path": "..." },
|
||||
"dark": { "id": "...", "kind": "DARK", "exposure_time": 300, "download_path": "...", "scale": 2.0 },
|
||||
"flat": { "id": "...", "kind": "FLAT", "filter": "Ha", "download_path": "..." }
|
||||
},
|
||||
"complete": true,
|
||||
"decisions": [{ "kind": "FLAT", "master_id": "...", "reason": "matched on ..." }]
|
||||
}
|
||||
```
|
||||
|
||||
`scale` appears only when the dark's exposure did not match exactly. **Subtract
|
||||
the bias before scaling.** Dark current scales with exposure time and the bias
|
||||
does not, so scaling a bias-inclusive dark multiplies the bias too. The
|
||||
reference implementation refuses to scale a dark not marked bias-subtracted, and
|
||||
so should yours.
|
||||
|
||||
`decisions` includes the misses. "No flat matched, and here is why" is part of
|
||||
how a frame was reduced, and it is recorded on the run at claim time rather
|
||||
than when you report back, so a worker that dies mid-stage still leaves a
|
||||
debuggable trail.
|
||||
|
||||
**`astrometry`** carries solver hints so the solve need not be blind.
|
||||
|
||||
```json
|
||||
{
|
||||
"hints": {
|
||||
"ra_deg": 10.68, "dec_deg": 41.27, "search_radius_deg": 5,
|
||||
"scale_arcsec_per_px": 0.724, "scale_low": 0.579, "scale_high": 0.869
|
||||
},
|
||||
"reference_catalog": "gaia_dr3",
|
||||
"already_solved": false
|
||||
}
|
||||
```
|
||||
|
||||
The scale comes from pixel size and focal length; the 20% window absorbs an
|
||||
unknown binning or an approximate focal length. The search radius is generous
|
||||
because the mount model is the weak link. Solve against **Gaia DR3**, not the
|
||||
solver's bundled index: sub-arcsecond astrometry is what makes MPC submission
|
||||
possible (TODOS.md item R), and a bundled index will not get you there.
|
||||
|
||||
**`timing`** carries `date_obs`, `exposure_time`, `timescale_in`,
|
||||
`timescale_out`, `epoch_convention` and `can_run`. Report BJD_TDB at
|
||||
**mid-exposure**. A 600 s exposure timed at the shutter is wrong by 300 s,
|
||||
which is larger than the entire light-travel term.
|
||||
|
||||
**`photometry`** carries aperture radii as multiples of the measured FWHM,
|
||||
rather than fixed pixels, because the right aperture depends on that night's
|
||||
seeing. It also sets `require_uncertainties: true`, which is not advisory.
|
||||
|
||||
## `POST /jobs/:id/heartbeat`
|
||||
|
||||
```json
|
||||
{ "worker_id": "sidecar-1" }
|
||||
```
|
||||
|
||||
Extends the claim by the claim TTL (10 minutes). Call it on any stage that
|
||||
might run long: a wide-field plate solve can outlast the TTL, and without a
|
||||
heartbeat the reclaimer hands the job to a second worker while the first is
|
||||
still solving.
|
||||
|
||||
**409** when the claim is no longer yours, with a reason: the job is not
|
||||
running, or another worker holds it. 409 rather than 403 because your
|
||||
credentials are fine; stop working on it.
|
||||
|
||||
## `POST /jobs/:id/complete`
|
||||
|
||||
```json
|
||||
{ "worker_id": "sidecar-1", "result": { } }
|
||||
```
|
||||
|
||||
Only the worker currently holding the claim may complete it. Everything in
|
||||
`result` is merged into the run's provenance under the stage name, so include
|
||||
what a reader would want: counts, residuals, versions, notes.
|
||||
|
||||
Some stages have required fields, checked at the boundary and rejected with 400
|
||||
rather than stored:
|
||||
|
||||
- **`astrometry`** needs `result.wcs` with at least `CTYPE1`, `CTYPE2`,
|
||||
`CRVAL1`, `CRVAL2`, `CRPIX1`, `CRPIX2`. A partial WCS is refused and the
|
||||
frame is *not* marked solved, because a partial solution produces
|
||||
confidently wrong sky coordinates, which is worse than no solution.
|
||||
On success the frame's `wcs_json` is stored, `wcs_solved` is set, and `ra`
|
||||
and `dec_coord` are overwritten from `CRVAL1/2`: the solved position beats
|
||||
the mount's reported pointing.
|
||||
- **`timing`** needs a finite numeric `result.bjd_tdb`.
|
||||
- **`photometry`** needs `result.measurements`, an array in which every entry
|
||||
has a numeric `mag` **and** `mag_err`. This is not negotiable
|
||||
(TODOS.md item N). A magnitude without an uncertainty is not a measurement,
|
||||
and accepting one is how an archive fills with numbers nobody can use.
|
||||
- **`extract`** optionally sets the frame's `fwhm_arcsec` from
|
||||
`result.fwhm_arcsec`.
|
||||
|
||||
## `POST /jobs/:id/fail`
|
||||
|
||||
```json
|
||||
{ "worker_id": "sidecar-1", "error": "solve failed: too few stars", "retryable": false }
|
||||
```
|
||||
|
||||
`retryable` defaults to true. You are the only party that knows whether the
|
||||
cause was a wedged solve (retry) or a corrupt file (do not), so decide rather
|
||||
than letting the queue burn three attempts on something deterministic.
|
||||
|
||||
A job that runs out of attempts fails for good, and every stage downstream of
|
||||
it is marked `blocked` rather than `failed`. Blocked means nothing is wrong
|
||||
with that stage and it becomes runnable once the blocker is fixed.
|
||||
|
||||
## `POST /jobs/:id/skip`
|
||||
|
||||
```json
|
||||
{ "worker_id": "sidecar-1", "reason": "frame already carried a complete WCS" }
|
||||
```
|
||||
|
||||
`reason` is required. A skip satisfies the prerequisite for downstream stages,
|
||||
so extraction still follows a skipped astrometry.
|
||||
|
||||
## `GET /frames/:id/file` and `GET /masters/:id/file`
|
||||
|
||||
The bytes, as `application/fits`.
|
||||
|
||||
**409** on a frame that has no local file, which is a legitimate state for a
|
||||
frame registered by URL. **404** when it should have one and the file is
|
||||
missing. The distinction matters: the first is "nothing to do here", the
|
||||
second is "something deleted the data".
|
||||
|
||||
## `GET /master-requests`
|
||||
|
||||
Groups of raw calibration frames that are ready to combine and do not already
|
||||
have a current master.
|
||||
|
||||
```
|
||||
GET /master-requests?telescope_id=<id>&min_frames=5
|
||||
```
|
||||
|
||||
Master building is per-group, not per-frame, so it does not fit the job queue;
|
||||
poll this instead. Groups are keyed on instrument, binning, gain, bucketed
|
||||
sensor temperature, exposure time (darks) and filter (flats). A group is
|
||||
considered covered when a non-superseded master already exists built from the
|
||||
same set of source frames, so adding one more flat produces a new request and
|
||||
the master gets rebuilt deeper.
|
||||
|
||||
## `POST /masters`
|
||||
|
||||
Multipart. `file` is the combined FITS; `meta` is a JSON string:
|
||||
|
||||
```json
|
||||
{
|
||||
"telescope_id": "...", "instrument_id": "...", "kind": "FLAT",
|
||||
"filter": "Ha", "exposure_time": 4.0, "ccd_temp_c": -15, "binning": "1x1", "gain": 1.6,
|
||||
"combine_method": "sigma_clip_median", "sigma_low": 5.0, "sigma_high": 3.0,
|
||||
"frame_ids": ["...", "..."],
|
||||
"mean_level": 21000, "std_level": 140,
|
||||
"valid_from": "2026-09-10 19:00:00", "valid_to": "2026-09-10 19:30:00"
|
||||
}
|
||||
```
|
||||
|
||||
Rules, enforced:
|
||||
|
||||
- `kind` is BIAS, DARK or FLAT.
|
||||
- A FLAT requires a `filter`. A flat is filter-specific by definition, and
|
||||
matching refuses a filter mismatch outright, so a filterless flat would be a
|
||||
master nothing could ever select.
|
||||
- `frame_ids` needs at least two entries.
|
||||
- **Idempotent on the source frame set.** Posting the same `frame_ids` twice
|
||||
returns `200 {"created": false}` with the existing id rather than creating a
|
||||
duplicate that later selection would have to choose between.
|
||||
- A new master **supersedes** any older one sharing its grouping keys, rather
|
||||
than deleting it. Selection ignores superseded rows, so the rebuild takes
|
||||
effect immediately, while a historical run still resolves the master it
|
||||
actually used.
|
||||
|
||||
`valid_from` and `valid_to` bound the master in time. Set them from the source
|
||||
frames' timestamps. Matching uses them: a flat goes stale after 30 days because
|
||||
it describes the optical train as it was that night, while a dark lasts 90
|
||||
because nothing physical about the sensor changed.
|
||||
|
||||
## `POST /runs/:id/environment`
|
||||
|
||||
```json
|
||||
{
|
||||
"solver": { "name": "astap", "version": "2024.03", "index": "g18" },
|
||||
"software": { "astropy": "8.0.1", "photutils": "3.0.0", "image": "sha256:..." }
|
||||
}
|
||||
```
|
||||
|
||||
Report this once per run. It is what makes a run re-executable rather than
|
||||
merely described, and it is the foundation for TODOS.md item V. A version
|
||||
string is the minimum; a container digest is better, so send one if you have it.
|
||||
|
||||
## Writing a worker
|
||||
|
||||
The loop is small:
|
||||
|
||||
1. `POST /claim` with your `worker_id` and the stages you implement.
|
||||
2. On 204, sleep a few seconds and poll again.
|
||||
3. On 200, download the frame and any masters, do the work, heartbeat if it
|
||||
runs long.
|
||||
4. `POST .../complete` with the result, or `.../fail` with a reason and an
|
||||
honest `retryable`.
|
||||
5. Separately and less often, poll `/master-requests` and post any masters you
|
||||
build.
|
||||
|
||||
Three things to get right, all of which the API will catch but which are
|
||||
cheaper to get right first: subtract bias before scaling a dark, never report a
|
||||
partial WCS, and never report a magnitude without an uncertainty.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Caelum reduction sidecar
|
||||
|
||||
The pixel-handling half of the reduction pipeline. Node owns the job queue, the
|
||||
calibration-matching rules and the provenance record; this package does the
|
||||
arithmetic.
|
||||
|
||||
The split is deliberate. astropy, photutils and ccdproc are validated and
|
||||
widely trusted, and a bug in a hand-rolled aperture sum produces plausible
|
||||
wrong answers, which is the worst failure mode available to a science platform.
|
||||
See [TODOS.md](../TODOS.md) item L.
|
||||
|
||||
## What is implemented
|
||||
|
||||
| Stage | Module | State |
|
||||
| --- | --- | --- |
|
||||
| `timing` | `caelum_reduce/timing.py` | Done. UTC to BJD_TDB at mid-exposure. |
|
||||
| `calibrate` | `caelum_reduce/calibrate.py` | Done. Master bias/dark/flat, and applying them with uncertainties propagated. |
|
||||
| `astrometry` | not yet | Needs a solver binary (ASTAP or astrometry.net) in the image. |
|
||||
| `extract` | not yet | photutils segmentation. |
|
||||
| `photometry` | not yet | Aperture and differential photometry. |
|
||||
|
||||
The worker loop that polls `/api/reduction/worker/claim` is not written yet, so
|
||||
today this is a library with tests rather than a running service. The protocol
|
||||
it will speak is already implemented and tested on the Node side; see
|
||||
`server/src/routes/reduction-worker.js`.
|
||||
|
||||
## Running the tests
|
||||
|
||||
The astronomy stack installs from wheels, so no compiler is needed:
|
||||
|
||||
```sh
|
||||
cd sidecar
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements-dev.txt
|
||||
PYTHONPATH=. .venv/bin/python -m pytest
|
||||
```
|
||||
|
||||
53 tests, about a second. They run against synthetic frames with known truth,
|
||||
so each one asserts the right answer rather than self-consistency.
|
||||
|
||||
## Why photutils and not SEP
|
||||
|
||||
The original plan named SEP for source extraction. photutils does the same job,
|
||||
is pure Python over numpy so it has wheels everywhere, and is the
|
||||
astropy-affiliated package we already depend on. SEP is a C extension with
|
||||
narrower wheel coverage and had none for Python 3.14 at the time of writing.
|
||||
One fewer build dependency for no loss of capability.
|
||||
|
||||
## What the tests are actually checking
|
||||
|
||||
Worth knowing, because several of these caught mistakes during development.
|
||||
|
||||
**Timing** asserts physics, not just that a number comes back. The barycentric
|
||||
correction is bounded by the light-crossing time of Earth's orbit, it reverses
|
||||
sign between the solstices for a target in the ecliptic plane, it nearly
|
||||
vanishes at quadrature, it almost disappears toward the ecliptic poles, and two
|
||||
sites on opposite sides of Earth agree to within the light travel time across
|
||||
Earth's diameter. A correction that failed any of those would still look like a
|
||||
plausible timestamp.
|
||||
|
||||
The reason for that thoroughness: a hand-rolled approximation is wrong by
|
||||
around 0.1 s, and the error is periodic with a one-year cycle because it tracks
|
||||
Earth's orbit. A periodic error in a data set about periodic phenomena does not
|
||||
read as a bug, it reads as signal.
|
||||
|
||||
**Calibration** asserts the reduction equation recovers a known injected
|
||||
signal, that a cosmic ray in one input frame does not reach the master while a
|
||||
genuine hot pixel in every frame survives, that a dead column does not become a
|
||||
NaN hole, and that uncertainties can only grow under subtraction.
|
||||
|
||||
It also pins two refusals:
|
||||
|
||||
- Scaling a dark that is not marked `bias_subtracted` is rejected. Dark current
|
||||
scales with exposure time and the bias does not, so scaling a bias-inclusive
|
||||
dark multiplies the bias too. This is the most common calibration mistake and
|
||||
the result looks entirely reasonable.
|
||||
- A master whose shape does not match the frame is rejected rather than allowed
|
||||
to broadcast. A master built at 2x2 binning silently applied to a 1x1 frame
|
||||
would be a disaster that no later step could detect.
|
||||
|
||||
And one documented limitation, pinned as a test so it stays visible: single-pass
|
||||
sigma clipping does not catch an evenly split stack. Combining eight frames at
|
||||
one illumination level with eight at another clips nothing, because the split
|
||||
inflates the stack's own standard deviation until both groups look like
|
||||
one-sigma deviations. Preventing that is the Node-side grouping rules' job, not
|
||||
the combiner's.
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Caelum reduction sidecar.
|
||||
|
||||
The pixel-handling half of the reduction pipeline. Node owns the queue, the
|
||||
calibration-matching rules and the provenance record; this package does the
|
||||
arithmetic, because astropy, photutils and ccdproc are validated and a
|
||||
hand-rolled aperture sum produces plausible wrong answers.
|
||||
|
||||
Modules:
|
||||
timing UTC at the observatory to BJD_TDB at mid-exposure
|
||||
calibrate build master bias/dark/flat, and apply them with uncertainties
|
||||
|
||||
See ../TODOS.md item L for the phase plan and ../docs/reduction-worker-protocol.md
|
||||
for the contract this speaks.
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,433 @@
|
||||
"""Build calibration masters, and apply them to a science frame.
|
||||
|
||||
The reduction equation this implements:
|
||||
|
||||
calibrated = (raw - bias - dark_scaled) / flat_normalized
|
||||
|
||||
Each term has a reason to be careful about it.
|
||||
|
||||
**bias** is the sensor's zero point, a zero-length read. Combined with a
|
||||
sigma-clipped median so a cosmic ray in one input frame does not become a
|
||||
permanent artifact in every frame the master touches.
|
||||
|
||||
**dark** is accumulated dark current plus the bias. If the master dark still
|
||||
contains the bias (which it does, unless it was built bias-subtracted), then
|
||||
subtracting both bias and dark removes the bias twice. This module builds
|
||||
darks *bias-subtracted* and records that fact on the master, so the apply step
|
||||
knows which arithmetic to use rather than assuming.
|
||||
|
||||
**dark scaling** only works on a bias-subtracted dark, because dark current
|
||||
scales with exposure time and the bias does not. Scaling a dark that still
|
||||
contains its bias multiplies the bias too, which is wrong by the scale factor.
|
||||
This is the single most common calibration mistake, so the code refuses to
|
||||
scale a dark not marked bias-subtracted.
|
||||
|
||||
**flat** is the multiplicative response. Normalized by its own median before
|
||||
division so the operation preserves the science frame's flux scale instead of
|
||||
rescaling every pixel by whatever the flat's illumination level happened to be.
|
||||
|
||||
Uncertainties (TODOS.md item N) are propagated rather than bolted on later. The
|
||||
variance model is the standard one:
|
||||
|
||||
var_electrons = counts_e + read_noise_e^2
|
||||
|
||||
converted back to ADU through the gain, then carried through each arithmetic
|
||||
step with the usual rules. Subtraction adds variances; division by the flat
|
||||
scales the relative error. Every array returned has a matching uncertainty
|
||||
array, and `apply_masters` refuses to produce a calibrated frame without one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, asdict, field
|
||||
|
||||
import numpy as np
|
||||
|
||||
# Sigma-clip bounds for combining. Asymmetric on purpose: the contaminants are
|
||||
# cosmic rays and satellite trails, which are positive excursions, so the high
|
||||
# side is clipped harder than the low.
|
||||
DEFAULT_SIGMA_LOW = 5.0
|
||||
DEFAULT_SIGMA_HIGH = 3.0
|
||||
|
||||
# Below this many inputs, combining barely reduces noise and a single outlier
|
||||
# can dominate the median. The API enforces the same floor when registering a
|
||||
# master; this is the second half of that rule.
|
||||
MIN_COMBINE_FRAMES = 2
|
||||
|
||||
# A dark may be scaled by at most this factor either way. Beyond it the noise
|
||||
# the scaling introduces outweighs the dark current it removes. Matches
|
||||
# DARK_SCALE_MAX_RATIO in the Node calibration service, and the two must stay
|
||||
# in step: Node decides the scale, this applies it.
|
||||
MAX_DARK_SCALE = 4.0
|
||||
|
||||
|
||||
class CalibrationError(ValueError):
|
||||
"""The inputs cannot produce a trustworthy calibrated frame."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class MasterResult:
|
||||
"""A combined calibration frame, with the metadata the API stores."""
|
||||
|
||||
data: np.ndarray
|
||||
uncertainty: np.ndarray
|
||||
kind: str
|
||||
frame_count: int
|
||||
combine_method: str
|
||||
sigma_low: float
|
||||
sigma_high: float
|
||||
mean_level: float
|
||||
std_level: float
|
||||
# True when the bias was removed before combining. Only meaningful for a
|
||||
# dark, and the apply step depends on it, so it travels with the master
|
||||
# rather than being inferred.
|
||||
bias_subtracted: bool = False
|
||||
# Fraction of pixels rejected by the clip, per frame. A master whose
|
||||
# rejection fraction is wildly high was built from mismatched inputs.
|
||||
rejected_fraction: float = 0.0
|
||||
notes: list[str] = field(default_factory=list)
|
||||
|
||||
def metadata(self) -> dict:
|
||||
d = asdict(self)
|
||||
d.pop("data")
|
||||
d.pop("uncertainty")
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
class CalibratedFrame:
|
||||
data: np.ndarray
|
||||
uncertainty: np.ndarray
|
||||
steps: list[str]
|
||||
dark_scale: float | None = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"steps": self.steps, "dark_scale": self.dark_scale}
|
||||
|
||||
|
||||
def combine_frames(
|
||||
stack: np.ndarray,
|
||||
kind: str,
|
||||
sigma_low: float = DEFAULT_SIGMA_LOW,
|
||||
sigma_high: float = DEFAULT_SIGMA_HIGH,
|
||||
bias_subtracted: bool = False,
|
||||
) -> MasterResult:
|
||||
"""Sigma-clipped median combine of a stack of like frames.
|
||||
|
||||
Args:
|
||||
stack: shape (n_frames, ny, nx).
|
||||
kind: BIAS, DARK or FLAT. Recorded, and used for the notes only; the
|
||||
arithmetic is the same for all three.
|
||||
sigma_low, sigma_high: clip bounds in units of the per-pixel standard
|
||||
deviation across the stack.
|
||||
bias_subtracted: pass True when building a dark from already
|
||||
bias-subtracted inputs. Recorded on the master, because
|
||||
`apply_masters` needs it and cannot work it out from the pixels.
|
||||
|
||||
Returns:
|
||||
MasterResult, whose `uncertainty` is the standard error of the median
|
||||
across the surviving inputs, per pixel.
|
||||
|
||||
Raises:
|
||||
CalibrationError: on too few frames or mismatched shapes.
|
||||
"""
|
||||
stack = np.asarray(stack, dtype=np.float64)
|
||||
if stack.ndim != 3:
|
||||
raise CalibrationError(f"expected a stack of shape (n, ny, nx), got {stack.shape}")
|
||||
n = stack.shape[0]
|
||||
if n < MIN_COMBINE_FRAMES:
|
||||
raise CalibrationError(
|
||||
f"{n} frame(s) is not enough to combine; need at least {MIN_COMBINE_FRAMES}"
|
||||
)
|
||||
|
||||
# Median and spread across the stack, per pixel. The median rather than the
|
||||
# mean because it is what makes a single cosmic ray harmless before any
|
||||
# clipping happens at all.
|
||||
median = np.median(stack, axis=0)
|
||||
spread = np.std(stack, axis=0)
|
||||
|
||||
# A pixel with zero spread across the stack (a dead or saturated column)
|
||||
# would make every deviation infinite. Treat it as "no outliers here".
|
||||
safe_spread = np.where(spread > 0, spread, np.inf)
|
||||
deviation = (stack - median) / safe_spread
|
||||
keep = (deviation >= -sigma_low) & (deviation <= sigma_high)
|
||||
|
||||
counts = keep.sum(axis=0)
|
||||
# Never reject every input for a pixel. If the clip would, fall back to
|
||||
# keeping all of them: a noisy value is better than a hole, and a hole
|
||||
# would propagate as a NaN into every frame this master touches.
|
||||
everything_rejected = counts == 0
|
||||
if np.any(everything_rejected):
|
||||
keep[:, everything_rejected] = True
|
||||
counts = keep.sum(axis=0)
|
||||
|
||||
masked = np.where(keep, stack, np.nan)
|
||||
combined = np.nanmedian(masked, axis=0)
|
||||
|
||||
# Standard error of the median, approximated as 1.253 * sigma / sqrt(n).
|
||||
# The constant is the asymptotic ratio of the median's standard error to
|
||||
# the mean's for a normal distribution.
|
||||
surviving_std = np.nanstd(masked, axis=0)
|
||||
uncertainty = 1.253 * surviving_std / np.sqrt(np.maximum(counts, 1))
|
||||
|
||||
rejected = float(1.0 - (counts.sum() / keep.size))
|
||||
|
||||
notes = []
|
||||
if np.any(everything_rejected):
|
||||
notes.append(
|
||||
f"{int(everything_rejected.sum())} pixel(s) had every input clipped; "
|
||||
"kept all inputs for those rather than leaving a hole"
|
||||
)
|
||||
if rejected > 0.05:
|
||||
notes.append(
|
||||
f"rejected {rejected:.1%} of pixel samples, which is high; "
|
||||
"check that the inputs really are the same kind of frame"
|
||||
)
|
||||
|
||||
return MasterResult(
|
||||
data=combined,
|
||||
uncertainty=uncertainty,
|
||||
kind=kind,
|
||||
frame_count=n,
|
||||
combine_method="sigma_clip_median",
|
||||
sigma_low=sigma_low,
|
||||
sigma_high=sigma_high,
|
||||
mean_level=float(np.mean(combined)),
|
||||
std_level=float(np.std(combined)),
|
||||
bias_subtracted=bias_subtracted,
|
||||
rejected_fraction=rejected,
|
||||
notes=notes,
|
||||
)
|
||||
|
||||
|
||||
def build_master_dark(
|
||||
stack: np.ndarray,
|
||||
master_bias: np.ndarray | None,
|
||||
sigma_low: float = DEFAULT_SIGMA_LOW,
|
||||
sigma_high: float = DEFAULT_SIGMA_HIGH,
|
||||
) -> MasterResult:
|
||||
"""Combine dark frames, subtracting the bias first when one is available.
|
||||
|
||||
A bias-subtracted dark is the only kind that can be scaled to a different
|
||||
exposure time, which is why this is the default path. When no bias is
|
||||
available the dark still combines, but it is marked `bias_subtracted=False`
|
||||
and `apply_masters` will refuse to scale it.
|
||||
"""
|
||||
stack = np.asarray(stack, dtype=np.float64)
|
||||
if master_bias is None:
|
||||
result = combine_frames(stack, "DARK", sigma_low, sigma_high, bias_subtracted=False)
|
||||
result.notes.append(
|
||||
"no master bias available, so this dark still contains the bias; "
|
||||
"it can only be used at its own exposure time, never scaled"
|
||||
)
|
||||
return result
|
||||
|
||||
bias = np.asarray(master_bias, dtype=np.float64)
|
||||
_require_same_shape(stack[0], bias, "dark frames", "master bias")
|
||||
return combine_frames(stack - bias, "DARK", sigma_low, sigma_high, bias_subtracted=True)
|
||||
|
||||
|
||||
def build_master_flat(
|
||||
stack: np.ndarray,
|
||||
master_bias: np.ndarray | None = None,
|
||||
master_dark: np.ndarray | None = None,
|
||||
sigma_low: float = DEFAULT_SIGMA_LOW,
|
||||
sigma_high: float = DEFAULT_SIGMA_HIGH,
|
||||
) -> MasterResult:
|
||||
"""Combine flat frames and normalize to a median of 1.
|
||||
|
||||
Normalizing here rather than at apply time means the division in the
|
||||
reduction equation preserves the science frame's flux scale. A flat left
|
||||
un-normalized would rescale every pixel by the flat's own illumination
|
||||
level, which is an arbitrary number that depends on how bright the twilight
|
||||
was.
|
||||
"""
|
||||
stack = np.asarray(stack, dtype=np.float64)
|
||||
work = stack
|
||||
steps = []
|
||||
if master_bias is not None:
|
||||
bias = np.asarray(master_bias, dtype=np.float64)
|
||||
_require_same_shape(stack[0], bias, "flat frames", "master bias")
|
||||
work = work - bias
|
||||
steps.append("bias subtracted")
|
||||
if master_dark is not None:
|
||||
dark = np.asarray(master_dark, dtype=np.float64)
|
||||
_require_same_shape(stack[0], dark, "flat frames", "master dark")
|
||||
work = work - dark
|
||||
steps.append("dark subtracted")
|
||||
|
||||
result = combine_frames(work, "FLAT", sigma_low, sigma_high)
|
||||
|
||||
level = float(np.median(result.data))
|
||||
if not np.isfinite(level) or level <= 0:
|
||||
raise CalibrationError(
|
||||
f"the combined flat has a median of {level}, which cannot be normalized; "
|
||||
"the inputs are probably not flats, or are fully saturated"
|
||||
)
|
||||
result.uncertainty = result.uncertainty / level
|
||||
result.data = result.data / level
|
||||
result.mean_level = float(np.mean(result.data))
|
||||
result.std_level = float(np.std(result.data))
|
||||
result.notes.append(f"normalized by its median of {level:.2f} ADU")
|
||||
if steps:
|
||||
result.notes.append("before combining: " + ", ".join(steps))
|
||||
return result
|
||||
|
||||
|
||||
def frame_uncertainty(
|
||||
data: np.ndarray,
|
||||
gain: float | None,
|
||||
read_noise_e: float | None,
|
||||
) -> np.ndarray:
|
||||
"""Per-pixel uncertainty in ADU for a raw frame.
|
||||
|
||||
The standard CCD noise model:
|
||||
|
||||
variance_electrons = signal_electrons + read_noise_electrons^2
|
||||
signal_electrons = counts_adu * gain
|
||||
|
||||
converted back to ADU by dividing by the gain squared. Negative counts
|
||||
(a pixel below bias after subtraction) are clamped to zero signal rather
|
||||
than producing a negative variance.
|
||||
|
||||
When the gain is unknown there is no way to convert counts to electrons, so
|
||||
this falls back to a pure read-noise floor and the caller is told, because
|
||||
silently pretending gain is 1.0 would produce uncertainties that look
|
||||
precise and are not.
|
||||
"""
|
||||
data = np.asarray(data, dtype=np.float64)
|
||||
rn = float(read_noise_e or 0.0)
|
||||
if gain is None or float(gain) <= 0:
|
||||
# No gain: the Poisson term is unknowable. Report the read noise alone,
|
||||
# which is a floor rather than an estimate.
|
||||
return np.full_like(data, rn, dtype=np.float64)
|
||||
g = float(gain)
|
||||
signal_e = np.maximum(data * g, 0.0)
|
||||
var_e = signal_e + rn * rn
|
||||
return np.sqrt(var_e) / g
|
||||
|
||||
|
||||
def apply_masters(
|
||||
raw: np.ndarray,
|
||||
bias: MasterResult | np.ndarray | None = None,
|
||||
dark: MasterResult | np.ndarray | None = None,
|
||||
flat: MasterResult | np.ndarray | None = None,
|
||||
dark_scale: float | None = None,
|
||||
gain: float | None = None,
|
||||
read_noise_e: float | None = None,
|
||||
) -> CalibratedFrame:
|
||||
"""Calibrate one science frame, propagating uncertainties.
|
||||
|
||||
Args:
|
||||
raw: the science frame in ADU.
|
||||
bias, dark, flat: masters, either MasterResult (preferred, since it
|
||||
carries `bias_subtracted`) or bare arrays.
|
||||
dark_scale: exposure-time ratio to apply to the dark. Refused unless
|
||||
the dark is known to be bias-subtracted, because scaling a dark
|
||||
that still contains its bias multiplies the bias too.
|
||||
gain, read_noise_e: for the noise model. Missing gain degrades the
|
||||
uncertainty to a read-noise floor and is noted in `steps`.
|
||||
|
||||
Raises:
|
||||
CalibrationError: on a shape mismatch, an out-of-range dark scale, or
|
||||
an attempt to scale a dark that is not bias-subtracted.
|
||||
"""
|
||||
raw = np.asarray(raw, dtype=np.float64)
|
||||
data = raw.copy()
|
||||
variance = frame_uncertainty(raw, gain, read_noise_e) ** 2
|
||||
steps: list[str] = []
|
||||
|
||||
if gain is None or float(gain or 0) <= 0:
|
||||
steps.append("gain unknown: uncertainty is a read-noise floor, not a Poisson estimate")
|
||||
|
||||
bias_data, bias_unc, _ = _unpack(bias)
|
||||
dark_data, dark_unc, dark_meta = _unpack(dark)
|
||||
flat_data, flat_unc, _ = _unpack(flat)
|
||||
|
||||
if bias_data is not None:
|
||||
_require_same_shape(raw, bias_data, "science frame", "master bias")
|
||||
data = data - bias_data
|
||||
variance = variance + _sq(bias_unc, bias_data)
|
||||
steps.append("bias subtracted")
|
||||
|
||||
if dark_data is not None:
|
||||
_require_same_shape(raw, dark_data, "science frame", "master dark")
|
||||
scale = 1.0 if dark_scale is None else float(dark_scale)
|
||||
if scale != 1.0:
|
||||
if not (1.0 / MAX_DARK_SCALE <= scale <= MAX_DARK_SCALE):
|
||||
raise CalibrationError(
|
||||
f"dark scale {scale:.3f} is outside the allowed range "
|
||||
f"[{1 / MAX_DARK_SCALE:.3f}, {MAX_DARK_SCALE:.1f}]"
|
||||
)
|
||||
# The mistake this refusal exists to prevent. A dark that still
|
||||
# contains its bias, scaled by 2, contributes twice the bias.
|
||||
if not (dark_meta and dark_meta.get("bias_subtracted")):
|
||||
raise CalibrationError(
|
||||
"refusing to scale a dark that is not marked bias_subtracted: "
|
||||
"dark current scales with exposure time but the bias does not, "
|
||||
"so scaling would multiply the bias too"
|
||||
)
|
||||
scaled = dark_data * scale
|
||||
data = data - scaled
|
||||
# Scaling multiplies the uncertainty by the same factor.
|
||||
variance = variance + _sq(dark_unc, dark_data) * (scale ** 2)
|
||||
steps.append("dark subtracted" if scale == 1.0 else f"dark subtracted, scaled by {scale:.3f}")
|
||||
|
||||
if flat_data is not None:
|
||||
_require_same_shape(raw, flat_data, "science frame", "master flat")
|
||||
# A zero or negative flat pixel means no information about that pixel.
|
||||
# Division would give an infinity that then contaminates anything
|
||||
# computed from a region containing it, so mark it NaN and say how many.
|
||||
bad = ~np.isfinite(flat_data) | (flat_data <= 0)
|
||||
safe_flat = np.where(bad, np.nan, flat_data)
|
||||
# Relative variances add under division.
|
||||
with np.errstate(divide="ignore", invalid="ignore"):
|
||||
rel_sci = variance / np.square(data, where=data != 0, out=np.full_like(data, np.nan))
|
||||
rel_flat = _sq(flat_unc, flat_data) / np.square(safe_flat)
|
||||
data = data / safe_flat
|
||||
variance = np.square(data) * (np.nan_to_num(rel_sci, nan=0.0) + np.nan_to_num(rel_flat, nan=0.0))
|
||||
steps.append("divided by the normalized flat")
|
||||
if np.any(bad):
|
||||
steps.append(f"{int(bad.sum())} flat pixel(s) were non-positive and are NaN in the output")
|
||||
|
||||
uncertainty = np.sqrt(np.maximum(variance, 0.0))
|
||||
return CalibratedFrame(
|
||||
data=data,
|
||||
uncertainty=uncertainty,
|
||||
steps=steps,
|
||||
dark_scale=None if dark_scale is None else float(dark_scale),
|
||||
)
|
||||
|
||||
|
||||
# ---------- internals ----------
|
||||
|
||||
|
||||
def _unpack(master):
|
||||
"""Return (data, uncertainty, metadata) for a master given either form."""
|
||||
if master is None:
|
||||
return None, None, None
|
||||
if isinstance(master, MasterResult):
|
||||
return master.data, master.uncertainty, master.metadata()
|
||||
return np.asarray(master, dtype=np.float64), None, None
|
||||
|
||||
|
||||
def _sq(uncertainty, reference):
|
||||
"""Variance contribution of a master, zero when it reports none.
|
||||
|
||||
A bare array master has no uncertainty, so it contributes nothing to the
|
||||
error budget. That is a real understatement of the true error and it is
|
||||
why MasterResult is the preferred form; passing an array is supported for
|
||||
tests and for a master built elsewhere.
|
||||
"""
|
||||
if uncertainty is None:
|
||||
return np.zeros_like(reference, dtype=np.float64)
|
||||
return np.square(np.asarray(uncertainty, dtype=np.float64))
|
||||
|
||||
|
||||
def _require_same_shape(a, b, a_name, b_name):
|
||||
if np.shape(a) != np.shape(b):
|
||||
raise CalibrationError(
|
||||
f"{a_name} is {np.shape(a)} but {b_name} is {np.shape(b)}; "
|
||||
"a master built at a different binning or from a different sensor "
|
||||
"cannot be applied"
|
||||
)
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Barycentric time correction: UTC at the observatory to BJD_TDB.
|
||||
|
||||
Why this is a sidecar stage and not a few lines of JavaScript
|
||||
------------------------------------------------------------
|
||||
Getting from a shutter timestamp to a barycentric one needs two things that
|
||||
are not arithmetic. The UTC-to-TDB conversion needs the leap-second table and
|
||||
the relativistic terms that relate a clock on Earth's geoid to a clock at the
|
||||
solar-system barycentre. The light-travel correction needs Earth's position in
|
||||
the solar system at the moment of observation, which needs an ephemeris.
|
||||
|
||||
A hand-rolled approximation lands around 0.1 s. That sounds close until you
|
||||
notice the error is *periodic with a one-year cycle*, because it tracks Earth's
|
||||
orbit. A periodic timing error in a data set about periodic phenomena does not
|
||||
look like a bug, it looks like signal. Transit-timing variation studies work at
|
||||
the tens-of-seconds level, so a 0.1 s systematic that walks over a season is
|
||||
exactly the wrong kind of wrong.
|
||||
|
||||
Conventions this module commits to, and records
|
||||
-----------------------------------------------
|
||||
* The reported epoch is **mid-exposure**, not shutter open. Half the exposure
|
||||
time is added before the correction. A 600 s exposure timed at its start is
|
||||
wrong by 300 s, which dwarfs every other term here.
|
||||
* The output scale is **TDB**, and the output is a **BJD**, not an HJD.
|
||||
Heliocentric times are still common in older variable-star literature and
|
||||
differ from barycentric by up to ~4 s, so the distinction is stated rather
|
||||
than assumed.
|
||||
* Times are returned with the intermediate values (JD_UTC, JD_TDB, the
|
||||
light-travel time in seconds) so a reader can check the arithmetic without
|
||||
rerunning it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, asdict
|
||||
|
||||
import astropy.units as u
|
||||
from astropy.coordinates import EarthLocation, SkyCoord
|
||||
from astropy.time import Time
|
||||
|
||||
# The light-travel time across 1 AU is about 499 s, so a barycentric
|
||||
# correction can never exceed roughly that in magnitude. Anything larger means
|
||||
# the coordinates, the site, or the timestamp is wrong, and it is better to
|
||||
# refuse than to emit a plausible-looking bad epoch.
|
||||
MAX_LIGHT_TRAVEL_SEC = 520.0
|
||||
|
||||
# 'builtin' is astropy's bundled low-precision ephemeris. It is accurate to
|
||||
# well under a millisecond of light-travel time, which is far below anything
|
||||
# a small-telescope network can time, and it needs no download at runtime.
|
||||
# Switch to 'jpl' only if a science case ever needs better, and record the
|
||||
# choice in the result either way so a reader knows which was used.
|
||||
DEFAULT_EPHEMERIS = "builtin"
|
||||
|
||||
|
||||
class TimingError(ValueError):
|
||||
"""The inputs cannot produce a trustworthy barycentric epoch."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class TimingResult:
|
||||
bjd_tdb: float
|
||||
jd_utc_mid: float
|
||||
jd_tdb_mid: float
|
||||
light_travel_sec: float
|
||||
tdb_minus_utc_sec: float
|
||||
epoch_convention: str
|
||||
timescale_out: str
|
||||
kind: str
|
||||
ephemeris: str
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def compute_bjd_tdb(
|
||||
date_obs: str,
|
||||
exposure_time: float,
|
||||
ra_deg: float,
|
||||
dec_deg: float,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
altitude_m: float = 0.0,
|
||||
ephemeris: str = DEFAULT_EPHEMERIS,
|
||||
) -> TimingResult:
|
||||
"""Barycentric Julian Date in TDB at mid-exposure.
|
||||
|
||||
Args:
|
||||
date_obs: shutter-open time, ISO-8601, interpreted as UTC. A trailing
|
||||
"Z" is accepted and stripped; an offset other than Z is rejected
|
||||
rather than silently misread.
|
||||
exposure_time: seconds. Half of it is added to reach mid-exposure.
|
||||
ra_deg, dec_deg: target position, ICRS.
|
||||
latitude, longitude: site position in degrees, longitude east-positive.
|
||||
altitude_m: site height above the ellipsoid.
|
||||
ephemeris: passed to astropy; recorded in the result.
|
||||
|
||||
Raises:
|
||||
TimingError: if any input is missing or out of range. This stage
|
||||
refuses rather than guesses, because a wrong epoch is not
|
||||
detectable downstream.
|
||||
"""
|
||||
if not date_obs:
|
||||
raise TimingError("date_obs is required; a frame with no timestamp cannot be corrected")
|
||||
if exposure_time is None:
|
||||
raise TimingError("exposure_time is required to reach mid-exposure")
|
||||
if exposure_time < 0:
|
||||
raise TimingError(f"exposure_time must not be negative, got {exposure_time}")
|
||||
if ra_deg is None or dec_deg is None:
|
||||
raise TimingError("the target position is required; the correction is direction-dependent")
|
||||
if not (0.0 <= float(ra_deg) <= 360.0):
|
||||
raise TimingError(f"ra_deg out of range: {ra_deg}")
|
||||
if not (-90.0 <= float(dec_deg) <= 90.0):
|
||||
raise TimingError(f"dec_deg out of range: {dec_deg}")
|
||||
if latitude is None or longitude is None:
|
||||
raise TimingError("the site position is required; the correction is observer-dependent")
|
||||
|
||||
location = EarthLocation(
|
||||
lat=float(latitude) * u.deg,
|
||||
lon=float(longitude) * u.deg,
|
||||
height=float(altitude_m or 0.0) * u.m,
|
||||
)
|
||||
|
||||
start = _parse_utc(date_obs, location)
|
||||
# Mid-exposure. Doing this before the correction rather than after matters
|
||||
# at the sub-millisecond level only, but doing it at all matters at the
|
||||
# half-exposure level, which is the whole point.
|
||||
mid = start + (float(exposure_time) / 2.0) * u.s
|
||||
|
||||
target = SkyCoord(ra=float(ra_deg) * u.deg, dec=float(dec_deg) * u.deg, frame="icrs")
|
||||
ltt = mid.light_travel_time(target, kind="barycentric", ephemeris=ephemeris)
|
||||
ltt_sec = float(ltt.sec)
|
||||
|
||||
if abs(ltt_sec) > MAX_LIGHT_TRAVEL_SEC:
|
||||
raise TimingError(
|
||||
f"light-travel correction of {ltt_sec:.1f}s exceeds the physical maximum "
|
||||
f"of ~{MAX_LIGHT_TRAVEL_SEC:.0f}s; check the coordinates and the site"
|
||||
)
|
||||
|
||||
bjd = float((mid.tdb + ltt).jd)
|
||||
|
||||
return TimingResult(
|
||||
bjd_tdb=bjd,
|
||||
jd_utc_mid=float(mid.jd),
|
||||
jd_tdb_mid=float(mid.tdb.jd),
|
||||
light_travel_sec=ltt_sec,
|
||||
tdb_minus_utc_sec=float((mid.tdb.jd - mid.jd) * 86400.0),
|
||||
epoch_convention="mid_exposure",
|
||||
timescale_out="tdb",
|
||||
# Stated explicitly because heliocentric times remain common in older
|
||||
# variable-star literature and differ by up to about 4 seconds.
|
||||
kind="barycentric",
|
||||
ephemeris=ephemeris,
|
||||
)
|
||||
|
||||
|
||||
def _parse_utc(date_obs: str, location: EarthLocation) -> Time:
|
||||
"""Parse a FITS DATE-OBS as UTC.
|
||||
|
||||
FITS says DATE-OBS is UTC unless a TIMESYS card says otherwise, and it
|
||||
carries no offset. Cameras nonetheless sometimes append "Z", which is
|
||||
harmless and means the same thing. Any other offset is ambiguous about
|
||||
whether the value was already converted, so it is refused.
|
||||
"""
|
||||
raw = str(date_obs).strip().replace(" ", "T")
|
||||
if raw.endswith("Z"):
|
||||
raw = raw[:-1]
|
||||
if "+" in raw[10:] or raw[10:].count("-") > 0:
|
||||
raise TimingError(
|
||||
f"date_obs carries a UTC offset ({date_obs!r}); FITS DATE-OBS must be UTC "
|
||||
"with no offset, and guessing would put a fixed error in every epoch"
|
||||
)
|
||||
try:
|
||||
return Time(raw, format="isot", scale="utc", location=location)
|
||||
except Exception as exc: # astropy raises a variety of types here
|
||||
raise TimingError(f"could not parse date_obs {date_obs!r}: {exc}") from exc
|
||||
@@ -0,0 +1,11 @@
|
||||
[project]
|
||||
name = "caelum-reduce"
|
||||
version = "0.1.0"
|
||||
description = "Reduction sidecar for Caelum: calibration, astrometry, photometry and timing"
|
||||
requires-python = ">=3.11"
|
||||
license = { text = "MIT" }
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
# Quiet, with the reason a test failed rather than a wall of locals.
|
||||
addopts = "-q --tb=short"
|
||||
@@ -0,0 +1,2 @@
|
||||
-r requirements.txt
|
||||
pytest~=9.0
|
||||
@@ -0,0 +1,11 @@
|
||||
# Pinned to the minor, because a numerics change between minors is exactly the
|
||||
# kind of thing that moves a measurement without anything failing. The exact
|
||||
# resolved versions belong in the image, and the sidecar reports them to
|
||||
# /runs/:id/environment so a run records what actually produced it.
|
||||
numpy~=2.5
|
||||
scipy~=1.18
|
||||
astropy~=8.0
|
||||
photutils~=3.0
|
||||
ccdproc~=2.5
|
||||
astroquery~=0.4
|
||||
requests~=2.32
|
||||
@@ -0,0 +1,360 @@
|
||||
"""Tests for master building and frame calibration.
|
||||
|
||||
Written against synthetic frames with known truth, so each test can assert the
|
||||
*right answer* rather than just self-consistency. The refusals matter as much
|
||||
as the arithmetic: a wrongly calibrated frame looks fine and produces wrong
|
||||
science, so the cases this module declines to handle are pinned here too.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from caelum_reduce.calibrate import (
|
||||
MAX_DARK_SCALE,
|
||||
CalibrationError,
|
||||
apply_masters,
|
||||
build_master_dark,
|
||||
build_master_flat,
|
||||
combine_frames,
|
||||
frame_uncertainty,
|
||||
)
|
||||
|
||||
SHAPE = (16, 20)
|
||||
RNG = np.random.default_rng(1234)
|
||||
|
||||
|
||||
def noisy(level, sigma, n=8, shape=SHAPE):
|
||||
"""A stack of n frames at `level` with gaussian noise."""
|
||||
return RNG.normal(level, sigma, size=(n, *shape))
|
||||
|
||||
|
||||
# ---------- combining ----------
|
||||
|
||||
|
||||
def test_combining_recovers_the_true_level():
|
||||
stack = noisy(500.0, 10.0, n=20)
|
||||
master = combine_frames(stack, "BIAS")
|
||||
assert abs(master.mean_level - 500.0) < 1.0, master.mean_level
|
||||
assert master.frame_count == 20
|
||||
assert master.combine_method == "sigma_clip_median"
|
||||
|
||||
|
||||
def test_combining_reduces_noise_by_roughly_root_n():
|
||||
# The reason to combine at all. The master's pixel-to-pixel scatter should
|
||||
# be far below a single frame's.
|
||||
stack = noisy(1000.0, 50.0, n=25)
|
||||
single_sigma = float(np.std(stack[0]))
|
||||
master = combine_frames(stack, "BIAS")
|
||||
assert float(np.std(master.data)) < single_sigma / 3, (
|
||||
f"master scatter {np.std(master.data):.1f} vs single frame {single_sigma:.1f}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_cosmic_ray_in_one_frame_does_not_reach_the_master():
|
||||
# The failure this prevents: an artifact baked into a master contaminates
|
||||
# every science frame the master is ever applied to.
|
||||
stack = noisy(500.0, 5.0, n=15)
|
||||
stack[3, 8, 10] = 60000.0 # a saturated hit in one input
|
||||
master = combine_frames(stack, "BIAS")
|
||||
assert abs(master.data[8, 10] - 500.0) < 10.0, (
|
||||
f"the hit leaked through: pixel is {master.data[8, 10]:.1f}, expected ~500"
|
||||
)
|
||||
|
||||
|
||||
def test_a_real_hot_pixel_present_in_every_frame_is_preserved():
|
||||
# The complement: clipping must not erase genuine sensor structure, which
|
||||
# is the whole thing a dark exists to record.
|
||||
stack = noisy(500.0, 5.0, n=15)
|
||||
stack[:, 4, 4] += 3000.0
|
||||
master = combine_frames(stack, "DARK")
|
||||
assert master.data[4, 4] > 3000.0, master.data[4, 4]
|
||||
|
||||
|
||||
def test_a_pixel_with_no_spread_does_not_become_a_hole():
|
||||
# A dead or saturated column has zero variance across the stack, which
|
||||
# would make every deviation infinite and clip everything. The result must
|
||||
# be the value, not a NaN that propagates into every calibrated frame.
|
||||
stack = noisy(500.0, 5.0, n=10)
|
||||
stack[:, 0, 0] = 0.0 # dead pixel, identical in every frame
|
||||
stack[:, 1, 1] = 65535.0 # saturated, identical in every frame
|
||||
master = combine_frames(stack, "BIAS")
|
||||
assert np.isfinite(master.data).all(), "a master must contain no NaNs"
|
||||
assert master.data[0, 0] == 0.0
|
||||
assert master.data[1, 1] == 65535.0
|
||||
|
||||
|
||||
def test_combining_too_few_frames_is_refused():
|
||||
with pytest.raises(CalibrationError, match="not enough to combine"):
|
||||
combine_frames(noisy(500.0, 5.0, n=1), "BIAS")
|
||||
|
||||
|
||||
def test_a_non_stack_is_refused():
|
||||
with pytest.raises(CalibrationError, match="expected a stack"):
|
||||
combine_frames(np.zeros(SHAPE), "BIAS")
|
||||
|
||||
|
||||
def test_the_master_reports_an_uncertainty_per_pixel():
|
||||
stack = noisy(500.0, 20.0, n=16)
|
||||
master = combine_frames(stack, "BIAS")
|
||||
assert master.uncertainty.shape == SHAPE
|
||||
# Standard error of the median for sigma=20, n=16 is about 1.253*20/4 = 6.3.
|
||||
assert 2.0 < float(np.median(master.uncertainty)) < 12.0, np.median(master.uncertainty)
|
||||
|
||||
|
||||
def test_a_high_rejection_fraction_is_flagged_in_the_notes():
|
||||
# A stack riddled with cosmic rays clips heavily, and saying so beats
|
||||
# silently handing back a master built from mostly-rejected samples.
|
||||
#
|
||||
# Note what does NOT trigger this, because it surprised me: combining two
|
||||
# different illumination levels in equal numbers (8 frames at 500 and 8 at
|
||||
# 900) clips nothing at all. The stack's own standard deviation is inflated
|
||||
# by the split, so each group sits about 1 sigma from the median and passes
|
||||
# the cut. Single-pass sigma clipping only catches a minority of outliers,
|
||||
# which is a real limitation of the method rather than a bug here.
|
||||
# Two frames of twenty are bad everywhere, the way a pair taken with the
|
||||
# dome still shut would be. Every pixel then has 2 of 20 samples clipped,
|
||||
# which is 10% of the stack and comfortably over the 5% threshold.
|
||||
stack = noisy(500.0, 3.0, n=20)
|
||||
stack[7, :, :] = 30000.0
|
||||
stack[13, :, :] = 30000.0
|
||||
master = combine_frames(stack, "FLAT")
|
||||
assert master.rejected_fraction > 0.05, master.rejected_fraction
|
||||
assert any("high" in n for n in master.notes), master.notes
|
||||
# And the good frames still set the level, which is the point of clipping.
|
||||
assert abs(master.mean_level - 500.0) < 2.0, master.mean_level
|
||||
|
||||
|
||||
def test_an_evenly_split_stack_clips_nothing_and_says_nothing():
|
||||
# The counterexample above, pinned so the limitation stays documented in
|
||||
# code rather than only in a comment. The master is the midpoint of two
|
||||
# unrelated groups, which is wrong, and single-pass clipping cannot see it.
|
||||
# Guarding against that is the Node-side grouping rules' job, not this
|
||||
# function's.
|
||||
stack = np.concatenate([noisy(500.0, 2.0, n=8), noisy(900.0, 2.0, n=8)])
|
||||
master = combine_frames(stack, "FLAT")
|
||||
assert master.rejected_fraction < 0.01, master.rejected_fraction
|
||||
|
||||
|
||||
# ---------- darks ----------
|
||||
|
||||
|
||||
def test_a_dark_built_with_a_bias_is_bias_subtracted_and_scalable():
|
||||
bias = np.full(SHAPE, 500.0)
|
||||
darks = noisy(500.0 + 120.0, 3.0, n=10) # bias + 120 ADU of dark current
|
||||
master = build_master_dark(darks, bias)
|
||||
assert master.bias_subtracted is True
|
||||
assert abs(master.mean_level - 120.0) < 2.0, master.mean_level
|
||||
|
||||
|
||||
def test_a_dark_built_without_a_bias_says_it_cannot_be_scaled():
|
||||
darks = noisy(620.0, 3.0, n=10)
|
||||
master = build_master_dark(darks, None)
|
||||
assert master.bias_subtracted is False
|
||||
assert any("never scaled" in n for n in master.notes), master.notes
|
||||
|
||||
|
||||
def test_a_dark_and_a_bias_of_different_shapes_are_refused():
|
||||
with pytest.raises(CalibrationError, match="cannot be applied"):
|
||||
build_master_dark(noisy(600.0, 3.0, n=5), np.full((4, 4), 500.0))
|
||||
|
||||
|
||||
# ---------- flats ----------
|
||||
|
||||
|
||||
def test_a_flat_is_normalized_to_a_median_of_one():
|
||||
# Normalizing is what makes the division preserve the science frame's flux
|
||||
# scale. An un-normalized flat rescales everything by how bright the
|
||||
# twilight happened to be.
|
||||
flats = noisy(20000.0, 100.0, n=12)
|
||||
master = build_master_flat(flats)
|
||||
assert abs(float(np.median(master.data)) - 1.0) < 1e-6, np.median(master.data)
|
||||
assert any("normalized" in n for n in master.notes)
|
||||
|
||||
|
||||
def test_a_flat_preserves_the_illumination_pattern():
|
||||
# Vignetting is the signal a flat carries. After normalization the centre
|
||||
# should read near 1 and the corner below it.
|
||||
base = np.ones(SHAPE)
|
||||
yy, xx = np.mgrid[0:SHAPE[0], 0:SHAPE[1]]
|
||||
cy, cx = (SHAPE[0] - 1) / 2, (SHAPE[1] - 1) / 2
|
||||
radius = np.sqrt(((yy - cy) / cy) ** 2 + ((xx - cx) / cx) ** 2)
|
||||
pattern = base * (1.0 - 0.3 * np.clip(radius, 0, 1)) # 30% vignette
|
||||
flats = np.array([pattern * 20000.0 + RNG.normal(0, 50, SHAPE) for _ in range(12)])
|
||||
|
||||
master = build_master_flat(flats)
|
||||
centre = master.data[int(cy), int(cx)]
|
||||
corner = master.data[0, 0]
|
||||
assert centre > corner, f"centre {centre:.3f} should exceed corner {corner:.3f}"
|
||||
# Assert the RATIO, not either absolute value. The flat is normalized by
|
||||
# its median, so the centre comes out above 1 and the corner below it, and
|
||||
# both depend on the median of the vignette pattern rather than on its
|
||||
# peak. The ratio is the physical quantity: 1.0 at centre over 0.7 at the
|
||||
# corner is 1.43, whatever the normalization constant happens to be.
|
||||
assert abs((centre / corner) - (1.0 / 0.7)) < 0.05, centre / corner
|
||||
|
||||
|
||||
def test_a_flat_that_cannot_be_normalized_is_refused():
|
||||
# All-zero inputs are not flats. Dividing by a zero-median flat would make
|
||||
# every science pixel infinite.
|
||||
with pytest.raises(CalibrationError, match="cannot be normalized"):
|
||||
build_master_flat(np.zeros((6, *SHAPE)))
|
||||
|
||||
|
||||
def test_a_flat_can_be_bias_and_dark_corrected_before_combining():
|
||||
bias = np.full(SHAPE, 500.0)
|
||||
dark = np.full(SHAPE, 20.0)
|
||||
flats = noisy(500.0 + 20.0 + 15000.0, 80.0, n=10)
|
||||
master = build_master_flat(flats, master_bias=bias, master_dark=dark)
|
||||
assert abs(float(np.median(master.data)) - 1.0) < 1e-6
|
||||
assert any("bias subtracted" in n for n in master.notes), master.notes
|
||||
|
||||
|
||||
# ---------- the noise model ----------
|
||||
|
||||
|
||||
def test_uncertainty_follows_the_ccd_noise_model():
|
||||
# var_e = signal_e + rn^2, converted back to ADU. For 1000 ADU at gain 2
|
||||
# and 5 e- read noise: signal 2000 e-, var 2025 e-, sigma 45 e- = 22.5 ADU.
|
||||
unc = frame_uncertainty(np.full((2, 2), 1000.0), gain=2.0, read_noise_e=5.0)
|
||||
assert abs(float(unc[0, 0]) - 22.5) < 0.1, unc[0, 0]
|
||||
|
||||
|
||||
def test_uncertainty_grows_with_signal():
|
||||
unc = frame_uncertainty(np.array([[100.0, 10000.0]]), gain=1.5, read_noise_e=4.0)
|
||||
assert unc[0, 1] > unc[0, 0]
|
||||
|
||||
|
||||
def test_a_negative_pixel_does_not_produce_a_negative_variance():
|
||||
# After bias subtraction some pixels sit below zero. The Poisson term is
|
||||
# clamped rather than allowed to make the variance negative, which would
|
||||
# come out as a NaN from the square root.
|
||||
unc = frame_uncertainty(np.array([[-50.0]]), gain=2.0, read_noise_e=5.0)
|
||||
assert np.isfinite(unc).all()
|
||||
assert unc[0, 0] > 0
|
||||
|
||||
|
||||
def test_an_unknown_gain_falls_back_to_a_read_noise_floor():
|
||||
# Pretending gain is 1.0 would produce uncertainties that look like
|
||||
# estimates and are not. A floor is honest.
|
||||
unc = frame_uncertainty(np.full((2, 2), 5000.0), gain=None, read_noise_e=6.0)
|
||||
assert np.allclose(unc, 6.0)
|
||||
|
||||
|
||||
# ---------- applying ----------
|
||||
|
||||
|
||||
def test_the_full_reduction_equation_recovers_the_true_signal():
|
||||
# Truth: 800 ADU of star light, on a 500 ADU bias, 40 ADU of dark, through
|
||||
# a flat that transmits 80% at this pixel.
|
||||
bias = combine_frames(noisy(500.0, 1.0, n=20), "BIAS")
|
||||
dark = build_master_dark(noisy(540.0, 1.0, n=20), bias.data)
|
||||
flat_stack = np.array([np.full(SHAPE, 16000.0) + RNG.normal(0, 20, SHAPE) for _ in range(20)])
|
||||
flat_stack[:, 5, 5] *= 0.8
|
||||
flat = build_master_flat(flat_stack)
|
||||
|
||||
raw = np.full(SHAPE, 500.0 + 40.0 + 800.0)
|
||||
raw[5, 5] = 500.0 + 40.0 + 800.0 * 0.8 # the flat attenuates the star too
|
||||
|
||||
out = apply_masters(raw, bias=bias, dark=dark, flat=flat, gain=2.0, read_noise_e=5.0)
|
||||
# Both the unvignetted and the vignetted pixel should come back to 800.
|
||||
assert abs(out.data[0, 0] - 800.0) < 15.0, out.data[0, 0]
|
||||
assert abs(out.data[5, 5] - 800.0) < 25.0, out.data[5, 5]
|
||||
assert "bias subtracted" in out.steps
|
||||
assert "divided by the normalized flat" in out.steps
|
||||
|
||||
|
||||
def test_every_calibrated_frame_carries_an_uncertainty():
|
||||
# TODOS.md item N. Non-negotiable: a frame without an error array is not a
|
||||
# measurement, and the API rejects a photometry result without one.
|
||||
raw = np.full(SHAPE, 1500.0)
|
||||
out = apply_masters(raw, gain=2.0, read_noise_e=5.0)
|
||||
assert out.uncertainty.shape == raw.shape
|
||||
assert np.isfinite(out.uncertainty).all()
|
||||
assert (out.uncertainty > 0).all()
|
||||
|
||||
|
||||
def test_subtracting_a_master_increases_the_uncertainty():
|
||||
# Variances add under subtraction, so calibration can never reduce the
|
||||
# error bar. A pipeline that reported a smaller uncertainty after
|
||||
# subtracting something would be wrong.
|
||||
raw = np.full(SHAPE, 2000.0)
|
||||
bare = apply_masters(raw, gain=2.0, read_noise_e=5.0)
|
||||
bias = combine_frames(noisy(500.0, 20.0, n=4), "BIAS")
|
||||
with_bias = apply_masters(raw, bias=bias, gain=2.0, read_noise_e=5.0)
|
||||
assert float(np.median(with_bias.uncertainty)) > float(np.median(bare.uncertainty))
|
||||
|
||||
|
||||
def test_scaling_a_bias_subtracted_dark_is_allowed_and_scales_the_dark():
|
||||
bias = np.full(SHAPE, 500.0)
|
||||
dark = build_master_dark(np.full((10, *SHAPE), 500.0 + 50.0), bias)
|
||||
raw = np.full(SHAPE, 500.0 + 100.0 + 300.0) # twice the dark, plus signal
|
||||
out = apply_masters(raw, bias=bias, dark=dark, dark_scale=2.0, gain=2.0, read_noise_e=5.0)
|
||||
assert abs(out.data[0, 0] - 300.0) < 1.0, out.data[0, 0]
|
||||
assert out.dark_scale == 2.0
|
||||
assert any("scaled by 2.000" in s for s in out.steps), out.steps
|
||||
|
||||
|
||||
def test_scaling_a_dark_that_still_contains_its_bias_is_refused():
|
||||
# The single most common calibration mistake. Scaling by 2 a dark that
|
||||
# includes a 500 ADU bias subtracts 1000 ADU of bias, and the result looks
|
||||
# like a perfectly plausible frame.
|
||||
unsubtracted = build_master_dark(np.full((10, *SHAPE), 550.0), None)
|
||||
assert unsubtracted.bias_subtracted is False
|
||||
with pytest.raises(CalibrationError, match="not marked bias_subtracted"):
|
||||
apply_masters(np.full(SHAPE, 1200.0), dark=unsubtracted, dark_scale=2.0)
|
||||
|
||||
|
||||
def test_an_unscaled_dark_is_fine_even_if_it_contains_its_bias():
|
||||
# At its own exposure time a bias-inclusive dark is exactly right, because
|
||||
# it removes bias and dark in one step.
|
||||
unsubtracted = build_master_dark(np.full((10, *SHAPE), 550.0), None)
|
||||
out = apply_masters(np.full(SHAPE, 550.0 + 400.0), dark=unsubtracted, gain=2.0, read_noise_e=5.0)
|
||||
assert abs(out.data[0, 0] - 400.0) < 1.0, out.data[0, 0]
|
||||
|
||||
|
||||
def test_a_dark_scale_beyond_the_limit_is_refused():
|
||||
bias = np.full(SHAPE, 500.0)
|
||||
dark = build_master_dark(np.full((6, *SHAPE), 520.0), bias)
|
||||
with pytest.raises(CalibrationError, match="outside the allowed range"):
|
||||
apply_masters(np.full(SHAPE, 1000.0), dark=dark, dark_scale=MAX_DARK_SCALE + 1)
|
||||
|
||||
|
||||
def test_a_master_of_the_wrong_shape_is_refused():
|
||||
# A master built at 2x2 binning applied to a 1x1 frame is a silent
|
||||
# disaster if numpy broadcasting were allowed to happen.
|
||||
with pytest.raises(CalibrationError, match="cannot be applied"):
|
||||
apply_masters(np.full(SHAPE, 1000.0), bias=np.full((4, 4), 500.0))
|
||||
|
||||
|
||||
def test_a_non_positive_flat_pixel_becomes_nan_and_is_counted():
|
||||
# Division would give an infinity that contaminates any statistic computed
|
||||
# over a region containing it. NaN is contained and countable.
|
||||
flat = np.ones(SHAPE)
|
||||
flat[2, 2] = 0.0
|
||||
flat[3, 3] = -0.5
|
||||
out = apply_masters(np.full(SHAPE, 1000.0), flat=flat, gain=2.0, read_noise_e=5.0)
|
||||
assert np.isnan(out.data[2, 2])
|
||||
assert np.isnan(out.data[3, 3])
|
||||
assert np.isfinite(out.data[0, 0])
|
||||
assert any("non-positive" in s for s in out.steps), out.steps
|
||||
|
||||
|
||||
def test_applying_nothing_is_a_no_op_that_still_reports_uncertainty():
|
||||
raw = np.full(SHAPE, 1234.0)
|
||||
out = apply_masters(raw, gain=1.5, read_noise_e=3.0)
|
||||
assert np.allclose(out.data, raw)
|
||||
assert out.steps == [] or all("gain unknown" not in s for s in out.steps)
|
||||
assert (out.uncertainty > 0).all()
|
||||
|
||||
|
||||
def test_master_metadata_excludes_the_pixel_arrays():
|
||||
# The metadata goes into a JSON POST to the API, so it must not carry
|
||||
# megabytes of pixels.
|
||||
master = combine_frames(noisy(500.0, 5.0, n=6), "BIAS")
|
||||
meta = master.metadata()
|
||||
assert "data" not in meta
|
||||
assert "uncertainty" not in meta
|
||||
assert meta["kind"] == "BIAS"
|
||||
assert meta["frame_count"] == 6
|
||||
@@ -0,0 +1,224 @@
|
||||
"""Tests for the barycentric time correction.
|
||||
|
||||
The failure this suite exists to prevent is a *periodic* timing error, because
|
||||
that is the kind that does not look like a bug in a data set about periodic
|
||||
things. So the tests check physics rather than just "it returned a number":
|
||||
the correction has a bounded magnitude, it flips sign over half a year, it
|
||||
vanishes toward the ecliptic poles, and mid-exposure is actually the middle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from caelum_reduce.timing import (
|
||||
MAX_LIGHT_TRAVEL_SEC,
|
||||
TimingError,
|
||||
compute_bjd_tdb,
|
||||
)
|
||||
|
||||
# A site and a target used throughout. The site is the one the test fixtures
|
||||
# in the Node suite use, so numbers can be compared across the two.
|
||||
SITE = dict(latitude=34.5, longitude=-120.5, altitude_m=1500.0)
|
||||
M31 = dict(ra_deg=10.6847, dec_deg=41.2687)
|
||||
|
||||
# Pinned from astropy 8.0.1 with the builtin ephemeris. This is a regression
|
||||
# guard, not an independent check: the physics assertions below are what
|
||||
# establish the answer is right. If an astropy upgrade moves this by more than
|
||||
# a millisecond, that is worth knowing about deliberately.
|
||||
REFERENCE = dict(
|
||||
date_obs="2026-09-12T22:14:03",
|
||||
exposure_time=0.0,
|
||||
bjd_tdb=2461296.4310303824,
|
||||
light_travel_sec=328.8425343026767,
|
||||
)
|
||||
|
||||
|
||||
def test_matches_the_pinned_reference():
|
||||
got = compute_bjd_tdb(date_obs=REFERENCE["date_obs"],
|
||||
exposure_time=REFERENCE["exposure_time"],
|
||||
**M31, **SITE)
|
||||
# One millisecond in days. Tighter than any science case here needs, loose
|
||||
# enough not to fail on a float-formatting change.
|
||||
assert abs(got.bjd_tdb - REFERENCE["bjd_tdb"]) < 1e-3 / 86400.0
|
||||
assert abs(got.light_travel_sec - REFERENCE["light_travel_sec"]) < 1e-3
|
||||
|
||||
|
||||
def test_the_correction_never_exceeds_the_light_crossing_time_of_the_orbit():
|
||||
# Light takes about 499 s to cross 1 AU, so no barycentric correction can
|
||||
# be larger than roughly that. Sampled around a full year and across the
|
||||
# sky, because the extreme depends on both date and direction.
|
||||
for month in range(1, 13):
|
||||
for ra in (0.0, 90.0, 180.0, 270.0):
|
||||
for dec in (-80.0, -30.0, 0.0, 30.0, 80.0):
|
||||
got = compute_bjd_tdb(
|
||||
date_obs=f"2026-{month:02d}-15T00:00:00",
|
||||
exposure_time=0.0, ra_deg=ra, dec_deg=dec, **SITE,
|
||||
)
|
||||
assert abs(got.light_travel_sec) <= MAX_LIGHT_TRAVEL_SEC
|
||||
|
||||
|
||||
def test_the_correction_reverses_sign_over_half_a_year():
|
||||
# Earth sits on opposite sides of its orbit six months apart, so a target
|
||||
# in the ecliptic plane goes from maximally near to maximally far. A
|
||||
# correction that did NOT flip sign would mean the ephemeris is not being
|
||||
# consulted at all, which is the most likely way to get this silently
|
||||
# wrong.
|
||||
#
|
||||
# The extremes fall at the SOLSTICES for this target, not the equinoxes.
|
||||
# The target is at RA 6h; at the March equinox the Sun is at RA 0h, so
|
||||
# Earth lies toward RA 12h, which is 90 degrees away and projects to
|
||||
# nearly nothing. At the solstices Earth lies along RA 6h or 18h, in line
|
||||
# with the target, which is where the projection is largest.
|
||||
on_ecliptic = dict(ra_deg=90.0, dec_deg=23.4) # the summer solstice point
|
||||
june = compute_bjd_tdb(date_obs="2026-06-21T00:00:00", exposure_time=0.0,
|
||||
**on_ecliptic, **SITE)
|
||||
december = compute_bjd_tdb(date_obs="2026-12-21T00:00:00", exposure_time=0.0,
|
||||
**on_ecliptic, **SITE)
|
||||
assert june.light_travel_sec * december.light_travel_sec < 0, (
|
||||
f"expected opposite signs, got {june.light_travel_sec:.1f} and "
|
||||
f"{december.light_travel_sec:.1f}"
|
||||
)
|
||||
# Both near the physical maximum, because the target is in the plane and
|
||||
# Earth is in line with it.
|
||||
assert abs(june.light_travel_sec) > 400, june.light_travel_sec
|
||||
assert abs(december.light_travel_sec) > 400, december.light_travel_sec
|
||||
|
||||
|
||||
def test_the_correction_nearly_vanishes_at_quadrature():
|
||||
# The complement of the test above, and the reason its dates are what they
|
||||
# are: at the equinoxes Earth is perpendicular to this target, so the
|
||||
# projection onto the line of sight is near zero. Getting this backwards
|
||||
# is how you end up "verifying" a correction that is actually constant.
|
||||
on_ecliptic = dict(ra_deg=90.0, dec_deg=23.4)
|
||||
for equinox in ("2026-03-20T00:00:00", "2026-09-22T00:00:00"):
|
||||
got = compute_bjd_tdb(date_obs=equinox, exposure_time=0.0, **on_ecliptic, **SITE)
|
||||
assert abs(got.light_travel_sec) < 60, (equinox, got.light_travel_sec)
|
||||
|
||||
|
||||
def test_a_target_at_the_ecliptic_pole_barely_moves_over_a_year():
|
||||
# Earth's orbital motion is perpendicular to this direction all year, so
|
||||
# the annual swing should be small. This is the complement of the test
|
||||
# above: it pins that the correction is direction-dependent in the right
|
||||
# way, not just that it varies.
|
||||
pole = dict(ra_deg=270.0, dec_deg=66.56) # north ecliptic pole
|
||||
values = [
|
||||
compute_bjd_tdb(date_obs=f"2026-{m:02d}-15T00:00:00", exposure_time=0.0,
|
||||
**pole, **SITE).light_travel_sec
|
||||
for m in range(1, 13)
|
||||
]
|
||||
swing = max(values) - min(values)
|
||||
on_ecliptic = [
|
||||
compute_bjd_tdb(date_obs=f"2026-{m:02d}-15T00:00:00", exposure_time=0.0,
|
||||
ra_deg=90.0, dec_deg=23.4, **SITE).light_travel_sec
|
||||
for m in range(1, 13)
|
||||
]
|
||||
ecliptic_swing = max(on_ecliptic) - min(on_ecliptic)
|
||||
assert swing < ecliptic_swing / 10, (
|
||||
f"pole swing {swing:.1f}s should be far smaller than ecliptic {ecliptic_swing:.1f}s"
|
||||
)
|
||||
|
||||
|
||||
def test_mid_exposure_is_actually_the_middle():
|
||||
# A 600 s exposure timed at the shutter is wrong by 300 s, which is larger
|
||||
# than the entire light-travel term. This is the single most likely
|
||||
# mistake in the whole module.
|
||||
zero = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=0.0, **M31, **SITE)
|
||||
six_hundred = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=600.0, **M31, **SITE)
|
||||
delta_sec = (six_hundred.bjd_tdb - zero.bjd_tdb) * 86400.0
|
||||
# 300 s, plus a few milliseconds because Earth moved during the exposure.
|
||||
assert abs(delta_sec - 300.0) < 0.05, f"offset was {delta_sec:.4f}s, expected ~300"
|
||||
assert six_hundred.epoch_convention == "mid_exposure"
|
||||
|
||||
|
||||
def test_tdb_minus_utc_is_the_expected_scale_offset():
|
||||
# TDB runs ahead of UTC by the accumulated leap seconds plus about 32.184s
|
||||
# of TAI-to-TT offset, so the total sits near 69s in this era. A value far
|
||||
# from that means the time scales are not being converted at all.
|
||||
got = compute_bjd_tdb(date_obs="2026-09-12T22:14:03", exposure_time=0.0, **M31, **SITE)
|
||||
assert 60.0 < got.tdb_minus_utc_sec < 80.0, got.tdb_minus_utc_sec
|
||||
|
||||
|
||||
def test_the_result_states_its_conventions():
|
||||
# A consumer must never have to guess whether an epoch is heliocentric or
|
||||
# barycentric, or which ephemeris produced it.
|
||||
got = compute_bjd_tdb(date_obs="2026-09-12T22:14:03", exposure_time=60.0, **M31, **SITE)
|
||||
assert got.kind == "barycentric"
|
||||
assert got.timescale_out == "tdb"
|
||||
assert got.epoch_convention == "mid_exposure"
|
||||
assert got.ephemeris
|
||||
d = got.to_dict()
|
||||
assert set(d) >= {"bjd_tdb", "jd_utc_mid", "jd_tdb_mid", "light_travel_sec"}
|
||||
|
||||
|
||||
def test_a_trailing_z_is_accepted():
|
||||
with_z = compute_bjd_tdb(date_obs="2026-09-12T22:14:03Z", exposure_time=0.0, **M31, **SITE)
|
||||
without = compute_bjd_tdb(date_obs="2026-09-12T22:14:03", exposure_time=0.0, **M31, **SITE)
|
||||
assert with_z.bjd_tdb == without.bjd_tdb
|
||||
|
||||
|
||||
def test_a_space_separated_timestamp_is_accepted():
|
||||
# The API hands back what SQLite stores, which may be space-separated.
|
||||
spaced = compute_bjd_tdb(date_obs="2026-09-12 22:14:03", exposure_time=0.0, **M31, **SITE)
|
||||
iso = compute_bjd_tdb(date_obs="2026-09-12T22:14:03", exposure_time=0.0, **M31, **SITE)
|
||||
assert spaced.bjd_tdb == iso.bjd_tdb
|
||||
|
||||
|
||||
def test_an_offset_other_than_z_is_refused():
|
||||
# Ambiguous: it could mean the camera already converted, or not. Guessing
|
||||
# puts a fixed multi-hour error into every epoch from that camera.
|
||||
with pytest.raises(TimingError, match="offset"):
|
||||
compute_bjd_tdb(date_obs="2026-09-12T22:14:03+02:00", exposure_time=0.0, **M31, **SITE)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"kwargs, match",
|
||||
[
|
||||
(dict(date_obs="", exposure_time=1.0, **M31, **SITE), "date_obs is required"),
|
||||
(dict(date_obs="2026-09-12T22:00:00", exposure_time=None, **M31, **SITE), "exposure_time is required"),
|
||||
(dict(date_obs="2026-09-12T22:00:00", exposure_time=-5.0, **M31, **SITE), "negative"),
|
||||
(dict(date_obs="2026-09-12T22:00:00", exposure_time=1.0, ra_deg=None, dec_deg=41.0, **SITE), "target position"),
|
||||
(dict(date_obs="2026-09-12T22:00:00", exposure_time=1.0, ra_deg=400.0, dec_deg=41.0, **SITE), "ra_deg out of range"),
|
||||
(dict(date_obs="2026-09-12T22:00:00", exposure_time=1.0, ra_deg=10.0, dec_deg=-120.0, **SITE), "dec_deg out of range"),
|
||||
(dict(date_obs="not-a-date", exposure_time=1.0, **M31, **SITE), "could not parse"),
|
||||
],
|
||||
)
|
||||
def test_bad_input_is_refused_with_a_specific_reason(kwargs, match):
|
||||
# Refusing is the right behaviour: a wrong epoch is undetectable
|
||||
# downstream, so there is nothing to gain by guessing.
|
||||
with pytest.raises(TimingError, match=match):
|
||||
compute_bjd_tdb(**kwargs)
|
||||
|
||||
|
||||
def test_a_missing_site_is_refused():
|
||||
with pytest.raises(TimingError, match="site position"):
|
||||
compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=1.0,
|
||||
latitude=None, longitude=None, **M31)
|
||||
|
||||
|
||||
def test_altitude_changes_the_answer_only_slightly():
|
||||
# Sanity that the site is being used at all, and that its height is a
|
||||
# small term: the difference between sea level and 1500 m is nanoseconds,
|
||||
# far below anything measurable here.
|
||||
low = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=0.0,
|
||||
latitude=34.5, longitude=-120.5, altitude_m=0.0, **M31)
|
||||
high = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=0.0,
|
||||
latitude=34.5, longitude=-120.5, altitude_m=1500.0, **M31)
|
||||
diff_sec = abs(high.bjd_tdb - low.bjd_tdb) * 86400.0
|
||||
assert diff_sec < 1e-3, diff_sec
|
||||
|
||||
|
||||
def test_two_sites_observing_the_same_target_agree_to_milliseconds():
|
||||
# The point of barycentric time: it removes the observer. Two telescopes
|
||||
# on opposite sides of Earth timing the same event must agree, and the
|
||||
# residual is the light-travel difference across Earth's radius, about
|
||||
# 21 ms at most.
|
||||
east = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=0.0,
|
||||
latitude=0.0, longitude=0.0, altitude_m=0.0, **M31)
|
||||
west = compute_bjd_tdb(date_obs="2026-09-12T22:00:00", exposure_time=0.0,
|
||||
latitude=0.0, longitude=180.0, altitude_m=0.0, **M31)
|
||||
diff_ms = abs(east.bjd_tdb - west.bjd_tdb) * 86400.0 * 1000.0
|
||||
assert diff_ms < 50.0, f"{diff_ms:.1f} ms apart; expected under ~43 ms (2 Earth radii of light time)"
|
||||
assert not math.isclose(diff_ms, 0.0, abs_tol=1e-9), "a real difference is expected, just a small one"
|
||||
Reference in New Issue
Block a user