Files
caelum/docs/running-it-yourself.md
Paul HittandClaude Opus 5 def8e518e2 docs and landing: photometry runs, and how to use the differential curve
The landing page still listed photometry as being built and the setup guide
stopped at the raw curve. Both were true this morning.

The landing copy now says astrometry is the one unbuilt stage, that photometry
does not wait on it because most acquisition software plate-solves at capture,
and that the magnitudes are instrumental. De-slopped per the standing rule:
clean on the pattern catalog, no em dashes, no tier words.

The guide gains a section on choosing comparison stars, which is the part that
decides whether a differential curve means anything. It says what the scatter
test is for in the terms that matter to an observer: a variable comparison puts
its own variation into every point with the sign flipped, so the target appears
to vary in antiphase with a star nobody was looking at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 22:20:52 -04:00

12 KiB

Running and testing Caelum yourself

Everything below has been run. Where something has not, this file says so instead of implying otherwise.

Three pieces run separately and none of them needs the others to start:

Piece Where it runs What it does
The API and web app Wherever you host it Stores frames, schedules, decides
The reduction sidecar Anywhere with Python Calibrates frames, times them, finds sources
The sequencer bridge At the observatory Drives the telescope

The sidecar and the bridge both poll the API over HTTPS and accept nothing inbound, so neither needs a port opened to it.

Before you start

Node 22 or later, because Chevrotain needs Object.groupBy and Node 20 has not got it. Python 3.11 or later for the sidecar. Docker if you want the containers rather than the processes.

The API

npm install
npm run dev:seed          # one-time, creates a SQLite database with sample data
npm run dev               # API on :3002, web app on :5173 with hot reload

SQLite is the development default and needs no setup. For MySQL set DB_KIND=mysql and the MYSQL_* variables in server/.env.

Check it came up:

curl -s localhost:3002/api/health

db.app should read up. If it reads down, the message says why.

Tokens

The sidecar and the bridge each need an API token, and the scopes are deliberately narrow. Create them in Settings, or over the API:

curl -s -X POST localhost:3002/api/tokens \
  -H "Authorization: Bearer $YOUR_SESSION_JWT" \
  -H 'Content-Type: application/json' \
  -d '{"name":"reduction sidecar","scopes":"reduction"}'

Valid scopes are read, write, daemon, reduction and sequencer. The token is shown once, at creation, and stored hashed. If you lose it, make another and revoke the old one.

Give the sidecar reduction and the bridge sequencer, nothing more. Neither scope grants account-wide write, so a token that leaks off an observatory PC cannot touch your archive. There is a test for that: an account-wide write token, far more privileged, still cannot drive a mount.

The reduction sidecar

cd sidecar
python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt

CAELUM_API_URL=http://localhost:3002 \
CAELUM_WORKER_TOKEN=cae_... \
PYTHONPATH=. .venv/bin/python -m caelum_reduce

Or in Docker, which is how it runs in production:

CAELUM_WORKER_TOKEN=cae_... docker compose --profile reduction up -d
Variable Default Notes
CAELUM_API_URL required No trailing slash needed
CAELUM_WORKER_TOKEN required Must start with cae_
CAELUM_WORKER_ID hostname Appears on claims and in logs
CAELUM_WORKER_STAGES all implemented timing,calibrate,extract,photometry
CAELUM_POLL_INTERVAL 5 Seconds between empty claims
CAELUM_MIN_MASTER_FRAMES 5 Floor for combining a master
CAELUM_ASTROMETRY_CHECK off Cross-check solved frames against Gaia

Leave CAELUM_WORKER_STAGES unset unless you mean it. Setting it to something this build does not implement makes the worker claim those jobs and fail every one, which looks like a broken queue rather than a misconfiguration.

Checking it works

Upload a science frame and at least five bias, dark and flat frames for the same telescope. Within a minute or so the log should show a master built for each kind, then the stages running:

registered BIAS master … from 6 frames
claimed job … stage=calibrate frame=…
completed job … stage=calibrate in 0.0s

Open the frame in the Archive. The reduction panel names each stage, what it produced, and which calibration masters were applied. A stage that did not run says why it did not.

If nothing is claimed, the usual causes are a token without the reduction scope (the API answers 403 and says which scopes the token does carry) and a CAELUM_API_URL the container cannot reach.

The sequencer bridge

This one drives a telescope, so read the protocol before pointing it at hardware. The bridge owns everything physical, Caelum owns the decision, and safety never depends on the network.

Simulated devices, which move nothing:

cd sidecar
CAELUM_API_URL=http://localhost:3002 \
CAELUM_BRIDGE_TOKEN=cae_... \
CAELUM_TELESCOPE_ID=<telescope uuid> \
PYTHONPATH=. .venv/bin/python -m caelum_bridge

It warns on every start that the devices are simulated. Without that warning, somebody eventually reads a clean log and credits a night the telescope spent parked.

Real hardware through ASCOM Remote or any Alpaca server:

ALPACA_URL=http://obs-pc:11111 \
ALPACA_TELESCOPE=0 ALPACA_CAMERA=0 ALPACA_FILTER_WHEEL=0 \
ALPACA_SAFETY_DEVICE=0 \
CAELUM_API_URL=https://caelum.network \
CAELUM_BRIDGE_TOKEN=cae_... \
CAELUM_TELESCOPE_ID=<telescope uuid> \
PYTHONPATH=. .venv/bin/python -m caelum_bridge

ALPACA_SAFETY_DEVICE has no default and the bridge refuses to start without it. Assume the weather is fine because nobody wired up a sensor and you observe through a thunderstorm.

Variable Default Notes
CAELUM_TELESCOPE_ID required Which telescope this bridge drives
CAELUM_BRIDGE_ID hostname Identifies this bridge on leases
CAELUM_POLL_INTERVAL 15 Seconds between polls
CAELUM_WEATHER_LATCH 600 Seconds of continuous safe weather before resuming
CAELUM_DISPATCH_MODE normal daynight dispatches regardless of the Sun

Checking out an observatory in daylight

CAELUM_DISPATCH_MODE=daynight is ACP's mode for exactly this: with the roof shut, the mount slews and the camera exposes so you can prove the hardware works without spending a night on it. Altitude, airmass and Moon limits still apply, so it will not point at the ground.

Checking it works

Open Observe for that telescope. The top of the page reports one of five things, and they are kept apart because they call for different actions:

  • No bridge has ever connected
  • A bridge connected once and has not polled recently
  • Connected and polling, but reporting it cannot observe
  • Ready and idle
  • Observing, with the target and the reason it was chosen

The last two are what you want to see. "Has not polled recently" is the state worth watching for: a bridge that stops polling writes no error anywhere, and that page is the only place it shows.

Measuring something

Photometry needs two things the archive cannot work out for itself, so it declines to run until you supply them. Set them on the camera in Settings:

Field What it is
gain_e_per_adu Electrons per ADU, the FITS EGAIN
read_noise_e Read noise in electrons, at the gain you observe at
saturation_adu Full well in ADU, above which the sensor stops being linear

gain_e_per_adu is not the CMOS GAIN setting. That one is a camera menu number, commonly 0 or 100 or 120, and is not a conversion factor at all. Using it would scale every uncertainty by about a hundred, in the direction that makes the data look better. That is why the header's GAIN is deliberately not a fallback here.

Without the gain and the read noise there is no uncertainty, and a magnitude without an error bar is refused, not stored. The stage names the missing field and stops there.

Then open Analysis, search a target, and the light curve is at the bottom. Under it are four counts: how many frames exist, how many are solved, how many are measured, how many landed on the target. A curve of nine points drawn from two hundred frames is a different claim from nine drawn from nine, and the plot alone does not distinguish them.

Differential photometry

The raw curve still carries the sky. Extinction, thin cloud and changes in transparency move every star in the frame together, and on most nights that motion is larger than anything you are trying to measure.

Open Comparison stars under the chart and press Suggest comparison stars. Each candidate is scored on how much its own magnitude moves across the very frames your curve is built from, compared with how much its error bars say it should. A star that moves more than three times its own errors is variable, and the list says so and does not offer it: a variable comparison puts its own variation into every point of your curve with the sign flipped, so your target appears to vary in antiphase with a star nobody was looking at.

Candidates are also rejected for being absent from some frames, for saturating in any of them, and for sitting more than two magnitudes from the target. Each rejection says which, so you can see why the obvious bright star nearby was passed over.

Add two or three, tick Show the differential curve, and the atmosphere divides out. Frames missing any comparison are dropped from the differential series rather than measured against a smaller ensemble, and the count of what was dropped sits under the chart.

The scheduler

Nothing to run. Open Observe and it ranks candidate targets for the selected telescope, with each score broken into the terms that produced it. Every declined target keeps its reason and either the time it becomes possible or the fact that it never will from your latitude.

The weight sliders are the coefficients from Denny's dispatch-scheduling paper, the design ACP Scheduler used. Moving one re-queries, so you can see what a different balance would have picked. Setting Priority to zero and leaving Transit shows what nearness to the meridian does on its own, which is a quick way to satisfy yourself the thing is doing what it claims.

Running the tests

npm run lint                                      # the whole repo
cd server      && npm test                        # 398 tests
cd client      && npm test                        # 46 tests
cd nox/parser  && npm test
cd nox/runtime && npm test
cd sidecar     && PYTHONPATH=. .venv/bin/python -m pytest    # 243 tests

Those counts are from 2026-09-12 and will drift upward. They are here so you can tell a suite that passed from a suite that collected nothing and reported success, which looks identical until you read the number.

None of these need a network, a database server, or hardware. The sidecar tests fake the HTTP transport and the devices; the server tests run against a temporary SQLite file they clean up after themselves.

Checking the ephemeris against JPL

The observability engine is tested against astropy in the normal suite. To check it against JPL Horizons instead, which is the ephemeris astropy's own reference is built on:

cd sidecar
PYTHONPATH=. .venv/bin/python tools/check_ephemeris.py

It prints the disagreement at six epochs, two of them outside the range the checked-in fixture covers so a series that drifts away from its fitting epoch cannot pass. Last run: worst Sun 35 arcsec, worst Moon 448, against tolerances of 180 and 1080.

This one needs the network and is not in CI, because Horizons is a shared public service and should not be hit on every push. Run it when you change the engine.

What none of this proves

No telescope has been driven by this code. The bridge has been run end to end against a live API with simulated devices, and the Alpaca adapter is tested against a faithful fake: correct envelope, correct units, correct asynchronous behaviour, correct error reporting. That covers the adapter's logic. A mount is a separate question, so before pointing this at equipment, run it against ASCOM's own Alpaca simulator drivers: a real server, not our idea of one.

Astrometry has no worker. It needs a solver and Gaia index files running to tens of gigabytes. Frames sit at reducing until that lands, and the reduction panel labels each unbuilt stage.

That does not block photometry, which measures sky positions from a WCS the frame already carries. Most acquisition software plate-solves at capture time and writes the solution into the header. A frame that arrives unsolved stays unsolved, and the light curve counts it as excluded rather than leaving it out of the total.

The magnitudes are instrumental. -2.5 log10(flux per second), no zero point. Two frames from the same telescope and filter compare to each other, which is what a variable star or a transit needs. Comparing them to a catalogue magnitude, or to another telescope, needs photometric standardisation. That is not built.

docs/verifying-accuracy.md records what each measurement is checked against, how good that reference is, and what each check has caught.