feat(research): add the Pipeline page

Every other reduction surface answers a question about one frame. The
question an operator actually wakes up with is "did last night get
processed", and answering it meant opening frames one at a time until
you found the broken one.

The failure this is really built for is the quiet one. The sidecar runs
in its own container; when it stops claiming, every job stays pending,
no error is ever written, and a stalled queue looks exactly like a busy
one in the counts. So the page leads with which workers have claimed and
when, checked per side: a healthy local Node worker says nothing about a
dead sidecar, and the warning names the side that stopped along with the
number of jobs waiting on it.

Below that, the stages in dependency order with their counts and the age
of the oldest waiting job (ten jobs a minute old is a healthy queue, one
job from last Tuesday is a broken one), then the failures with the error
the worker reported and a batch retry.

Server side, GET /reduction/queue grew from bare counts into the whole
payload, and POST /reduction/retry requeues up to 100 frames under one
click, authorizing each frame individually and reporting what it
refused rather than failing the batch.

Two bugs found while verifying against a live queue rather than a
fixture:

- The failure scope SQL checked site_members but not site ownership. A
  site's owner has no site_members row, so the list hid an operator's
  own failures on their own telescope, which is the single case the page
  exists for.
- A retried frame never left the list. Reprocessing creates a new run
  and keeps the old one, which is the provenance model working, but the
  old failed rows stay in the table. Failures are now scoped to each
  frame's current run, so acting on one clears it while the history
  stays intact.

Verified against an isolated stack seeded with a real mix: one stage
done, two frames failed on different stages with real errors, five
sidecar jobs waiting, and a worker whose last claim was 40 minutes old.
The page named the stalled sidecar rather than the failures, the batch
retry reported "Queued 2 frames", the superseded failures dropped off,
and the new run's genuine ENOENT failures took their place.

Research is now full at three items, which the nav invariant tests
enforce. DESIGN.md says what each of the three answers and that anything
else has to displace one of them.

253 server tests, 22 client tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
2026-09-12 16:49:55 -04:00
co-authored by Claude Opus 5
parent f9cd88ad73
commit 3915e879cd
8 changed files with 844 additions and 4 deletions
+7 -1
View File
@@ -27,7 +27,7 @@ telescope goes on the network → people book it → it runs live → the data l
| **Get Time** | Schedule, Discover, Proposals | Acquiring time, yours or someone else's |
| **Prepare** | Targets, Observation Plans, Nox Console | Deciding and scripting what to observe |
| **Archive** | Archive, SpaceDB | The data that lands, and the catalog to compare it to |
| **Research** | Analysis, Calibration | Turning those frames into measurements |
| **Research** | Analysis, Calibration, Pipeline | Turning those frames into measurements |
| **Community** | Activity, Leaderboard, Messages | The people around it |
| **Gear** | Shop, Shop Catalog | The store. An appendix, below the arc |
@@ -48,6 +48,12 @@ The same rule put Calibration in Research rather than Archive. Archive answers
is science work. Research sits after Archive because a measurement needs frames
before it can exist.
Research is now full at three, and the three answer different questions.
Analysis asks whether a target can be measured yet. Calibration asks whether
the frames can be trusted. Pipeline asks whether the processing is running at
all, which is the operational question underneath both. Anything else that
wants to live here has to displace one of them.
**No icons in the rail.** The rail is 240px and the labels carry it. A mixed
set of emoji and geometric glyphs read as unfinished and told the reader
nothing the label did not. The command palette is text-only for the same
+5 -1
View File
@@ -85,7 +85,11 @@ export const NAV_GROUPS = [
label: 'Research',
items: [
{ to: '/analysis', name: 'analysis', label: 'Analysis', keywords: 'science measure readiness light curve photometry timing baseline seeing target research' },
{ to: '/calibration', name: 'calibration', label: 'Calibration', keywords: 'master bias dark flat reduction pipeline calibrate frames stale library' },
{ to: '/calibration', name: 'calibration', label: 'Calibration', keywords: 'master bias dark flat calibrate frames stale library masters' },
// Third and last: the group is full. Pipeline answers "is my data being
// processed", which is a different question from Calibration's "can
// these frames be trusted" and Analysis's "is this target measurable".
{ to: '/pipeline', name: 'pipeline', label: 'Pipeline', keywords: 'reduction queue jobs failed stuck retry worker sidecar stages processing status health backlog' },
],
},
{
+1
View File
@@ -21,6 +21,7 @@ export const routes = [
{ path: '/spacedb', name: 'spacedb', component: () => import('./views/SpaceDB.vue'), meta: { title: 'SpaceDB' } },
{ path: '/analysis', name: 'analysis', component: () => import('./views/Analysis.vue'), meta: { title: 'Analysis', description: 'Whether a target has enough calibrated, solved and timed frames to measure.' } },
{ path: '/calibration', name: 'calibration', component: () => import('./views/Calibration.vue'), meta: { title: 'Calibration', description: 'Master bias, dark and flat frames per telescope, and the groups nearly ready to build.' } },
{ path: '/pipeline', name: 'pipeline', component: () => import('./views/Pipeline.vue'), meta: { title: 'Pipeline', description: 'What the reduction queue is doing, which workers are claiming, and which jobs failed.' } },
// Reached from a frame's reduction panel rather than the nav: a run record
// is about one frame, so it has no standalone entry point.
{ path: '/reduction/run/:id', name: 'reduction-run', component: () => import('./views/ReductionRun.vue'), props: true, meta: { title: 'Reduction run' } },
+456
View File
@@ -0,0 +1,456 @@
<script setup>
// The reduction pipeline as a whole, rather than one frame at a time.
//
// Why this page exists: every other reduction surface answers a question about
// a single frame. The question an operator actually wakes up with is "did last
// night get processed", and answering it meant opening frames one by one until
// you found the broken one.
//
// The failure this is really built for is the silent one. The sidecar runs in
// its own container; when it stops claiming, every job stays `pending`, no
// error is ever written, and a queue with a backlog looks exactly like a queue
// being worked through. A last-claim time is the only thing that separates
// them, so it is the first thing on the page.
import { ref, computed, onMounted, onUnmounted } from 'vue';
import { api } from '../services/api.js';
import { log } from '../utils/log.js';
const data = ref(null);
const loading = ref(false);
const error = ref('');
const retrying = ref(false);
const retryNote = ref('');
const selected = ref(new Set());
// A queue moves on its own, so the page refreshes itself. Thirty seconds is
// slow enough to be free and fast enough that a watched reprocess visibly
// finishes.
const REFRESH_MS = 30_000;
let timer = null;
// A worker that has not claimed in this long is not working through a backlog,
// it has stopped. Generous on purpose: an idle queue means a worker has
// nothing to claim, so this only reads as trouble alongside pending work.
const WORKER_STALE_MS = 10 * 60_000;
// Matches MAX_RETRY_BATCH in server/src/routes/reduction.js.
const MAX_RETRY_BATCH = 100;
const STATUS_ORDER = ['running', 'pending', 'blocked', 'failed', 'done', 'skipped'];
async function load() {
loading.value = true;
error.value = '';
try {
data.value = await api.get('/reduction/queue');
} catch (e) {
error.value = e.message || String(e);
} finally {
loading.value = false;
}
}
onMounted(() => {
load();
timer = setInterval(load, REFRESH_MS);
});
onUnmounted(() => clearInterval(timer));
const stages = computed(() => {
// The order and the local/sidecar split come from the API, so this page
// never carries a second copy of the dependency chain that can drift.
const order = data.value?.order || [];
const counts = data.value?.stages || {};
const pending = data.value?.pending || {};
return order.map(o => ({
...o,
counts: counts[o.name] || {},
total: Object.values(counts[o.name] || {}).reduce((a, b) => a + b, 0),
waiting: pending[o.name] || null,
}));
});
const totals = computed(() => {
const out = {};
for (const s of Object.values(data.value?.stages || {})) {
for (const [status, n] of Object.entries(s)) out[status] = (out[status] || 0) + n;
}
return out;
});
const failures = computed(() => data.value?.failures || []);
const workers = computed(() => data.value?.workers || []);
const anyPending = computed(() => (totals.value.pending || 0) > 0);
// How many jobs are waiting on each side of the split. The local stages run
// inside the API process and the rest need the sidecar container, so a healthy
// Node worker says nothing about whether the sidecar is alive. Counting them
// apart is what makes the stalled-sidecar warning possible.
const waitingBySide = computed(() => {
const out = { server: 0, sidecar: 0 };
for (const s of stages.value) {
out[s.runs_on] = (out[s.runs_on] || 0) + (s.counts.pending || 0);
}
return out;
});
// The headline. Deliberately few answers, because an operator reading this
// wants to know whether to go back to bed.
const health = computed(() => {
if (!data.value) return null;
const waiting = totals.value.pending || 0;
const running = totals.value.running || 0;
const failed = totals.value.failed || 0;
// Checked per side and before the failure count, because a stalled worker is
// the more urgent problem and the one nothing else on the page reports. It
// is also the quiet one: no error is ever written, so a stalled queue and a
// busy queue look identical in the counts.
const stalled = ['sidecar', 'server'].find(
side => waitingBySide.value[side] > 0
&& !workers.value.some(w => w.runs_on === side && !isStale(w)),
);
if (stalled) {
const n = waitingBySide.value[stalled];
return {
state: 'failed',
title: stalled === 'sidecar' ? 'The reduction sidecar is not claiming' : 'The local stage worker is not claiming',
body: `${n} ${stalled} job${n === 1 ? '' : 's'} waiting, and no ${stalled} worker has claimed anything in the last 10 minutes. `
+ (stalled === 'sidecar'
? 'The sidecar is a separate container: check that it is running, that it can reach this API, and that its reduction token is still valid.'
: 'The local stages run inside the API process, so this usually means the reduction tick is disabled or erroring.'),
};
}
if (failed > 0) {
return {
state: 'blocked',
title: `${failed} job${failed === 1 ? '' : 's'} failed`,
body: 'Listed below with the error the worker reported. Fix the cause, then retry the frames.',
};
}
if (running > 0 || waiting > 0) {
return {
state: 'running',
title: `${running} running, ${waiting} waiting`,
body: 'Work is moving. This page refreshes itself every 30 seconds.',
};
}
return { state: 'done', title: 'Queue is clear', body: 'Nothing waiting, nothing running, nothing failed.' };
});
function isStale(w) {
if (!w.last_claim_at) return true;
const t = new Date(w.last_claim_at).getTime();
return !Number.isFinite(t) || Date.now() - t > WORKER_STALE_MS;
}
function ago(v) {
if (!v) return 'never';
const ms = Date.now() - new Date(v).getTime();
if (!Number.isFinite(ms)) return 'unknown';
if (ms < 60_000) return 'just now';
const mins = Math.round(ms / 60_000);
if (mins < 60) return `${mins}m ago`;
const hours = Math.round(mins / 60);
if (hours < 48) return `${hours}h ago`;
return `${Math.round(hours / 24)}d ago`;
}
function statusesOf(stage) {
return STATUS_ORDER.filter(k => stage.counts[k]).map(k => ({ k, n: stage.counts[k] }));
}
function toggle(frameId) {
const next = new Set(selected.value);
if (next.has(frameId)) next.delete(frameId);
else next.add(frameId);
selected.value = next;
}
function selectAll() {
// One row per frame even when several of its stages failed, since a retry
// requeues the whole frame.
selected.value = new Set(failures.value.slice(0, MAX_RETRY_BATCH).map(f => f.frame_id));
}
function clearSelection() { selected.value = new Set(); }
async function retrySelected() {
const ids = [...selected.value];
if (!ids.length) return;
retrying.value = true;
retryNote.value = '';
try {
const out = await api.post('/reduction/retry', { frame_ids: ids });
const q = out.queued?.length || 0;
const r = out.refused?.length || 0;
// Say what was refused rather than reporting a clean success over it.
retryNote.value = r
? `Queued ${q}. ${r} refused: ${out.refused.map(x => x.reason).filter((v, i, a) => a.indexOf(v) === i).join(', ')}.`
: `Queued ${q} frame${q === 1 ? '' : 's'} for a fresh run.`;
clearSelection();
await load();
} catch (e) {
log.warn('[pipeline] retry failed', e);
retryNote.value = e.message || String(e);
} finally {
retrying.value = false;
}
}
</script>
<template>
<div>
<div class="page-header">
<div>
<h2>Pipeline</h2>
<p style="color:var(--text-secondary);font-size:var(--text-sm);margin-top:0.25rem;max-width:64ch">
What the reduction queue is doing, and what it needs from you. Only frames you can
reprocess are listed, because the rest are not yours to fix.
</p>
</div>
<button class="btn btn-ghost btn-sm" @click="load" :disabled="loading">Refresh</button>
</div>
<p v-if="error" style="color:var(--danger);font-size:var(--text-sm)">{{ error }}</p>
<p v-if="loading && !data" style="color:var(--text-muted);font-size:var(--text-sm)">Loading…</p>
<template v-if="data">
<!-- The one-line answer, before any table. -->
<div v-if="health" class="pipe-health" :class="'is-' + health.state">
<div class="pipe-health-title">{{ health.title }}</div>
<p class="pipe-health-body">{{ health.body }}</p>
</div>
<div class="card">
<div class="card-header">
<h3>Workers</h3>
<span style="font-size:var(--text-xs);color:var(--text-muted)">
who has claimed, and when
</span>
</div>
<p v-if="!workers.length" style="font-size:var(--text-sm);color:var(--text-secondary);margin:0">
No worker has ever claimed a job on this instance.
<template v-if="anyPending">
With work waiting, that means nothing is running: the local stages need the API
process up, and every other stage needs the reduction sidecar.
</template>
<template v-else>
Nothing has been queued yet, so there has been nothing to claim.
</template>
</p>
<div v-else class="table-scroll">
<table class="data-table">
<thead>
<tr><th>Worker</th><th>Serves</th><th class="num">Running</th><th>Last claim</th><th></th></tr>
</thead>
<tbody>
<tr v-for="w in workers" :key="w.worker_id">
<td style="font-family:monospace;font-size:var(--text-xs)">{{ w.worker_id }}</td>
<td>
{{ w.runs_on }}
<div class="pipe-sub">{{ w.stages.join(', ') }}</div>
</td>
<td class="num">{{ w.running }}</td>
<td>{{ ago(w.last_claim_at) }}</td>
<td>
<span class="pill" :class="isStale(w) ? 'pill-skipped' : 'pill-done'">
{{ isStale(w) ? 'idle' : 'active' }}
</span>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="card">
<div class="card-header">
<h3>Stages</h3>
<span style="font-size:var(--text-xs);color:var(--text-muted)">
in dependency order
</span>
</div>
<div v-for="s in stages" :key="s.name" class="pipe-stage">
<div class="pipe-stage-id">
<span class="pipe-stage-name">{{ s.name }}</span>
<span class="pipe-stage-where">{{ s.runs_on }}</span>
</div>
<div class="pipe-stage-counts">
<span v-for="c in statusesOf(s)" :key="c.k" class="pill" :class="'pill-' + c.k">
{{ c.n }} {{ c.k }}
</span>
<span v-if="!s.total" class="pipe-stage-empty">no jobs</span>
</div>
<!-- A depth alone says nothing. Ten jobs a minute old is healthy;
one job from last week is stuck. -->
<div class="pipe-stage-wait">
<template v-if="s.waiting">oldest waiting {{ ago(s.waiting.since) }}</template>
</div>
</div>
</div>
<div class="card">
<div class="card-header">
<h3>Needs you</h3>
<span style="font-size:var(--text-xs);color:var(--text-muted)">
{{ failures.length }} failed job{{ failures.length === 1 ? '' : 's' }}
</span>
</div>
<p v-if="!failures.length" style="font-size:var(--text-sm);color:var(--text-muted);margin:0">
Nothing has failed. A blocked stage is not listed here: it is waiting on the stage
above it and clears itself when that one does.
</p>
<template v-else>
<div class="pipe-actions">
<button class="btn btn-primary btn-sm"
:disabled="!selected.size || retrying"
@click="retrySelected">
{{ retrying ? 'Queueing…' : `Retry ${selected.size || ''} selected` }}
</button>
<button class="btn btn-ghost btn-sm" @click="selectAll" :disabled="retrying">
Select all
</button>
<button class="btn btn-ghost btn-sm" @click="clearSelection" :disabled="!selected.size || retrying">
Clear
</button>
<span v-if="retryNote" class="pipe-note">{{ retryNote }}</span>
</div>
<div class="table-scroll">
<table class="data-table">
<thead>
<tr>
<th></th>
<th>Frame</th>
<th>Stage</th>
<th>Error</th>
<th class="num">Attempts</th>
<th>Failed</th>
<th></th>
</tr>
</thead>
<tbody>
<tr v-for="f in failures" :key="f.id">
<td>
<input type="checkbox"
:checked="selected.has(f.frame_id)"
@change="toggle(f.frame_id)"
:aria-label="`Select ${f.target_name || f.filename}`" />
</td>
<td>
<div>{{ f.target_name || f.filename }}</div>
<div class="pipe-sub">{{ f.telescope_name }}</div>
</td>
<td><span class="pill pill-failed">{{ f.stage }}</span></td>
<td class="pipe-err">{{ f.error }}</td>
<td class="num">{{ f.attempts }} of {{ f.max_attempts }}</td>
<td style="white-space:nowrap">{{ ago(f.updated_at) }}</td>
<td>
<RouterLink class="btn btn-ghost btn-sm" :to="`/archive?frame=${f.frame_id}`">
Frame
</RouterLink>
<RouterLink v-if="f.run_id" class="btn btn-ghost btn-sm" :to="`/reduction/run/${f.run_id}`">
Run
</RouterLink>
</td>
</tr>
</tbody>
</table>
</div>
<p class="pipe-foot">
Retrying queues a fresh run for the whole frame, not just the stage that failed,
so the new run is a complete record rather than a patch on an old one. The previous
run is kept.
<template v-if="failures.length >= MAX_RETRY_BATCH">
Select all takes the first {{ MAX_RETRY_BATCH }}; run it again for the rest.
</template>
</p>
</template>
</div>
</template>
</div>
</template>
<style scoped>
.table-scroll { overflow-x: auto; }
.pipe-health {
border-left: 3px solid var(--border);
background: var(--bg-card);
padding: var(--space-4);
margin-bottom: var(--space-4);
}
.pipe-health.is-failed { border-left-color: var(--danger); }
.pipe-health.is-blocked { border-left-color: var(--warning); }
.pipe-health.is-running { border-left-color: var(--accent); }
.pipe-health.is-done { border-left-color: var(--success); }
.pipe-health-title {
font-size: var(--text-md);
font-weight: 600;
color: var(--text-primary);
}
.pipe-health-body {
font-size: var(--text-sm);
color: var(--text-secondary);
margin: 0.25rem 0 0;
max-width: 70ch;
}
.pipe-stage {
display: grid;
grid-template-columns: 10rem minmax(0, 1fr) auto;
gap: var(--space-3);
align-items: baseline;
padding: var(--space-3) 0;
border-bottom: 1px solid var(--border);
}
.pipe-stage:last-child { border-bottom: none; }
.pipe-stage-id { display: flex; flex-direction: column; }
.pipe-stage-name {
font-family: monospace;
font-size: var(--text-sm);
color: var(--text-primary);
}
.pipe-stage-where { font-size: var(--text-xs); color: var(--text-muted); }
.pipe-stage-counts { display: flex; gap: var(--space-2); flex-wrap: wrap; }
.pipe-stage-empty { font-size: var(--text-xs); color: var(--text-muted); }
.pipe-stage-wait {
font-size: var(--text-xs);
color: var(--text-muted);
white-space: nowrap;
}
.pipe-actions {
display: flex;
gap: var(--space-2);
align-items: center;
flex-wrap: wrap;
margin-bottom: var(--space-3);
}
.pipe-note { font-size: var(--text-xs); color: var(--text-secondary); }
.pipe-sub { font-size: var(--text-xs); color: var(--text-muted); }
.pipe-err {
font-size: var(--text-xs);
color: var(--text-secondary);
max-width: 34rem;
overflow-wrap: anywhere;
}
.pipe-foot {
font-size: var(--text-xs);
color: var(--text-muted);
margin: var(--space-3) 0 0;
max-width: 74ch;
}
@media (max-width: 768px) {
.pipe-stage { grid-template-columns: minmax(0, 1fr); }
.pipe-stage-id { flex-direction: row; gap: var(--space-2); align-items: baseline; }
}
</style>
+17
View File
@@ -276,6 +276,23 @@ frame registered by URL. **404** when it should have one and the file is
missing. The distinction matters: the first is "nothing to do here", the
second is "something deleted the data".
## Watching the queue from outside
`GET /api/reduction/queue` (a normal session endpoint, not a worker one)
returns pipeline health in one request: counts by stage and status, the
canonical stage order with each stage's `runs_on`, the oldest waiting job per
stage, the workers that have claimed with their last claim time and the stages
they serve, and the failures with their frames attached.
The worker list is the part worth having. A sidecar that stops claiming writes
no error: every job stays `pending`, and a stalled queue and a busy one look
identical in the counts. A last-claim time per side is what separates them.
Failures are scoped two ways. To frames the caller could actually reprocess,
since a failure you cannot act on is noise. And to each frame's *current* run,
because reprocessing keeps the old run rather than editing it, and without that
scope the list would accumulate every failure anyone ever fixed.
## `GET /master-requests`
Groups of raw calibration frames that are ready to combine and do not already
+111 -2
View File
@@ -4,11 +4,12 @@
//
// GET /api/reduction/frames/:id -> run, per-stage state, jobs
// POST /api/reduction/frames/:id/reprocess -> queue a fresh run
// POST /api/reduction/retry -> requeue a batch of frames
// GET /api/reduction/frames/:id/masters -> which masters would apply
// GET /api/reduction/masters -> the calibration library
// GET /api/reduction/candidates -> raw cal frames ready to combine
// GET /api/reduction/runs/:id -> one provenance record
// GET /api/reduction/queue -> counts by stage and status
// GET /api/reduction/queue -> pipeline health and what failed
//
// Read access follows the frame's own visibility, so a public frame's
// provenance is public: a measurement nobody can audit is not much of a
@@ -25,6 +26,9 @@ import {
getFrameReduction,
enqueueReduction,
queueStats,
workerActivity,
failingJobs,
oldestPending,
STAGES,
} from '../services/reduction-queue.js';
@@ -123,6 +127,59 @@ router.post('/frames/:id/reprocess', authenticate, async (req, res) => {
}
});
// The most common shape of a pipeline failure is one cause and forty frames:
// a master went missing, the sidecar ran out of disk, a version changed. Making
// an operator click each one is how a backlog stays broken, so a batch retry
// exists. Capped, because an unbounded requeue is a way to melt the queue by
// accident.
const MAX_RETRY_BATCH = 100;
router.post('/retry', authenticate, async (req, res) => {
const ids = req.body?.frame_ids;
if (!Array.isArray(ids) || !ids.length) {
return res.status(400).json({ error: 'frame_ids must be a non-empty array' });
}
if (ids.length > MAX_RETRY_BATCH) {
return res.status(400).json({ error: `at most ${MAX_RETRY_BATCH} frames per request` });
}
if (ids.some(id => typeof id !== 'string')) {
return res.status(400).json({ error: 'frame_ids must be strings' });
}
const unique = [...new Set(ids)];
const queued = [];
const refused = [];
for (const id of unique) {
const frame = await db.get(
`SELECT af.id, af.uploader_id, af.telescope_id, e.site_id
FROM archive_frames af
JOIN telescopes t ON af.telescope_id = t.id
JOIN enclosures e ON t.enclosure_id = e.id
WHERE af.id = ?`,
[id],
);
// Each frame is authorized on its own. A batch is a convenience for the
// caller, never a way to act on one frame under another's permission.
if (!frame) { refused.push({ frame_id: id, reason: 'no such frame' }); continue; }
if (!(await canEditFrame(frame, req.user))) {
refused.push({ frame_id: id, reason: 'not authorized' });
continue;
}
try {
const out = await enqueueReduction(frame.id, { triggeredBy: req.user.id, force: true });
queued.push({ frame_id: id, run_id: out.run_id });
} catch (e) {
refused.push({ frame_id: id, reason: e.message });
}
}
// 202 even with refusals in the body: the accepted frames really were
// queued, and reporting the partial result beats failing the whole batch
// over one frame someone else owns.
res.status(202).json({ queued, refused });
});
// Which masters the pipeline WOULD pick for this frame, and why. Answers
// "why is my frame not calibrating?" without running anything, which is the
// question an operator actually has.
@@ -215,10 +272,62 @@ router.get('/runs/:id', authenticate, async (req, res) => {
});
});
/**
* Pipeline health, in one request.
*
* Counts alone do not answer the question an operator actually has, which is
* "is my data moving, and if not, what do I do about it". So this also returns
* which workers are claiming (a silent sidecar leaves every job pending with
* no error recorded), how long the oldest waiting job has waited, and the
* failures with their frames attached.
*/
router.get('/queue', authenticate, async (req, res) => {
res.json({ stages: await queueStats() });
// Failures name a filename and a target, so they are scoped to frames the
// caller could reprocess. Showing a stranger's broken frame would be both a
// leak and useless: there is nothing they could do with it.
const { sql, params } = editableFrameScope(req.user);
const [stages, workers, failures, pending] = await Promise.all([
queueStats(),
workerActivity(),
failingJobs(sql, params, 50),
oldestPending(),
]);
res.json({
stages,
// The canonical order and the local/sidecar split, so the client does not
// hardcode a second copy of the stage list.
order: STAGES.map(s => ({ name: s.name, needs: s.needs, runs_on: s.local ? 'server' : 'sidecar' })),
workers,
pending,
failures,
});
});
/**
* A SQL predicate over `af` and `e` matching the frames this user may act on.
*
* The same rule as canEditFrame in middleware/perms.js: the uploader, a member
* of the frame's site, or an admin. Expressed as SQL rather than looped per
* row, because the queue view reads many frames at once and a per-row
* permission check would be one query per job.
*/
function editableFrameScope(user) {
if (user.role === 'admin') return { sql: '1 = 1', params: [] };
return {
// Site ownership is a separate branch from membership, because a site's
// owner has no site_members row. Leaving it out made the list hide an
// operator's own failures on their own telescope, which is the one case
// this page exists for.
sql: `(af.uploader_id = ?
OR e.site_id IN (SELECT id FROM sites WHERE owner_id = ?)
OR e.site_id IN (SELECT site_id FROM site_members
WHERE user_id = ? AND role IN ('owner', 'operator', 'observer')))`,
params: [user.id, user.id, user.id],
};
}
// The frame table has no instrument_id: a telescope's camera is reached
// through the instruments table. Calibration is per-camera, so resolve it
// here rather than matching on the telescope alone.
+112
View File
@@ -455,6 +455,118 @@ export async function queueStats() {
return out;
}
/**
* Which workers have claimed anything, and when.
*
* The sidecar runs in its own container, so the failure nobody notices is the
* one where it stops claiming: every job stays `pending`, no error is ever
* recorded, and the queue looks merely busy. A last-claim timestamp is what
* separates "working through a backlog" from "nothing is running".
*
* `local: true` covers the stages Node runs in-process, which is a different
* question from whether the sidecar is alive, so the two are reported apart.
*
* @returns {Promise<Array<{ worker_id: string, running: number, last_claim_at: string }>>}
*/
export async function workerActivity() {
// Grouped by worker AND stage, then folded in JS. Knowing which stages a
// worker takes is what lets a caller ask the useful question: not "is any
// worker alive" but "is anything claiming the stages that are waiting". The
// local Node worker being healthy says nothing about a stopped sidecar.
// Folding in JS rather than GROUP_CONCAT keeps it identical on both backends.
const rows = await db.all(
`SELECT claimed_by AS worker_id, stage,
SUM(CASE WHEN status = 'running' THEN 1 ELSE 0 END) AS running,
MAX(claimed_at) AS last_claim_at
FROM reduction_jobs
WHERE claimed_by IS NOT NULL
GROUP BY claimed_by, stage`,
);
const byWorker = new Map();
for (const r of rows) {
const w = byWorker.get(r.worker_id) || {
worker_id: r.worker_id, running: 0, last_claim_at: null, stages: [],
};
w.running += Number(r.running) || 0;
if (!w.last_claim_at || String(r.last_claim_at) > String(w.last_claim_at)) {
w.last_claim_at = r.last_claim_at;
}
if (r.stage && !w.stages.includes(r.stage)) w.stages.push(r.stage);
byWorker.set(r.worker_id, w);
}
const localStages = new Set(STAGES.filter(s => s.local).map(s => s.name));
return [...byWorker.values()]
.map(w => ({
...w,
stages: w.stages.sort(),
// A worker that has only ever claimed Node-owned stages is the local
// one. Reported rather than inferred from the id, which is a hostname.
runs_on: w.stages.every(st => localStages.has(st)) ? 'server' : 'sidecar',
}))
.sort((a, b) => String(b.last_claim_at).localeCompare(String(a.last_claim_at)));
}
/**
* The jobs an operator can do something about, newest trouble first.
*
* Only `failed`, because that is the set where a person is the next step: a
* `blocked` job resolves itself when its blocker does, and a `pending` one is
* waiting on a worker rather than on a decision. Each row carries the frame it
* belongs to so the page can link straight to it.
*
* Only failures from the frame's CURRENT run. Reprocessing creates a new run
* and leaves the old one intact, which is the point of the provenance model,
* but it means the old failed rows never go away. Without this the list a
* caller is told to act on keeps every failure they already fixed, and stops
* meaning "needs you" by the end of the first bad night.
*
* @param {string} scopeSql a SQL predicate over `af` and `e` limiting which
* frames the caller may see. Callers pass their own, because the rule
* lives with the route's permission model rather than here.
* @param {Array} scopeParams bindings for that predicate
* @param {number} [limit]
*/
export async function failingJobs(scopeSql, scopeParams = [], limit = 50) {
const rows = await db.all(
`SELECT j.id, j.run_id, j.frame_id, j.stage, j.error, j.attempts, j.max_attempts,
j.updated_at, j.claimed_by,
af.filename, af.target_name, af.date_obs, af.reduction_state,
t.name AS telescope_name
FROM reduction_jobs j
JOIN archive_frames af ON j.frame_id = af.id
JOIN telescopes t ON af.telescope_id = t.id
JOIN enclosures e ON t.enclosure_id = e.id
WHERE j.status = 'failed'
AND j.run_id = af.reduction_run_id
AND (${scopeSql})
ORDER BY j.updated_at DESC
LIMIT ?`,
[...scopeParams, limit],
);
return rows;
}
/**
* The oldest job still waiting, per stage.
*
* A queue depth on its own does not say whether anything is stuck: ten pending
* jobs a minute old is a healthy pipeline, and one pending job from last
* Tuesday is a broken one.
*/
export async function oldestPending() {
const rows = await db.all(
`SELECT stage, MIN(created_at) AS since, COUNT(*) AS n
FROM reduction_jobs
WHERE status = 'pending'
GROUP BY stage`,
);
const out = {};
for (const r of rows) out[r.stage] = { since: r.since, n: Number(r.n) };
return out;
}
// ---------- internals ----------
// A stage runs only once its prerequisite has succeeded. 'skipped' counts as
+135
View File
@@ -23,6 +23,7 @@ const TEST_JWT_SECRET = 'a'.repeat(64);
const USER = 'red-user';
const OTHER_USER = 'red-outsider';
const TELESCOPE = 'red-tel';
const OTHER_TELESCOPE = 'out-tel';
let server, base, token, otherToken, db, queue, pipeline;
let uploadDir;
@@ -142,6 +143,15 @@ before(async () => {
[TELESCOPE, 'red-enc', 'Reduction Scope', 'RS-1']);
await db.run('INSERT INTO instruments (id, telescope_id, name, code, type) VALUES (?, ?, ?, ?, ?)',
['red-cam', TELESCOPE, 'Main Camera', 'CAM', 'camera']);
// A second site, owned by the outsider and with USER not a member. Needed
// because a frame on USER's OWN telescope is reprocessable by USER whoever
// uploaded it, so uploader id alone cannot express "somebody else's frame".
await db.run('INSERT INTO sites (id, name, code, owner_id, latitude, longitude) VALUES (?, ?, ?, ?, ?, ?)',
['out-site', 'Outsider Site', 'OUT', OTHER_USER, 10.0, 10.0]);
await db.run('INSERT INTO enclosures (id, site_id, name, code) VALUES (?, ?, ?, ?)',
['out-enc', 'out-site', 'Dome', 'D']);
await db.run('INSERT INTO telescopes (id, enclosure_id, name, code) VALUES (?, ?, ?, ?)',
[OTHER_TELESCOPE, 'out-enc', 'Outsider Scope', 'OS-1']);
token = jwt.sign({ id: USER, email: 'red@test', username: 'red-tester', role: 'observer' }, TEST_JWT_SECRET, { expiresIn: '1h' });
otherToken = jwt.sign({ id: OTHER_USER, email: 'out@test', username: 'red-outsider', role: 'observer' }, TEST_JWT_SECRET, { expiresIn: '1h' });
@@ -570,6 +580,131 @@ test('GET /queue reports counts by stage and status', async () => {
assert.equal(status, 200);
assert.ok(body.stages.header, 'no header stage counts');
assert.ok(Object.values(body.stages.header).some(n => n > 0));
// The stage list travels with the response so the client does not keep a
// second copy of the dependency chain that can drift from this one.
assert.deepEqual(body.order.map(o => o.name), queue.STAGES.map(s => s.name));
assert.equal(body.order.find(o => o.name === 'header').runs_on, 'server');
assert.equal(body.order.find(o => o.name === 'calibrate').runs_on, 'sidecar');
});
test('GET /queue names the workers that claimed, so a silent sidecar is visible', async () => {
// The failure nobody notices: the sidecar container stops claiming, every
// job stays pending, no error is ever written, and the queue looks busy.
await resetQueue();
const name = writeFits('red-q1.fits', [kv('OBJECT', "'M13'")]);
const frameId = await insertFrame(name);
await queue.enqueueReduction(frameId, { stages: ['header', 'calibrate'] });
const before = await fetchJson('/api/reduction/queue');
assert.equal(before.body.workers.length, 0, 'nothing claimed yet');
assert.ok(before.body.pending.header, 'header should be waiting');
assert.ok(before.body.pending.header.since, 'a waiting stage needs a since, not just a count');
await queue.claimNextJob('probe-worker', { local: true });
const after = await fetchJson('/api/reduction/queue');
const worker = after.body.workers.find(w => w.worker_id === 'probe-worker');
assert.ok(worker, `probe-worker missing from ${JSON.stringify(after.body.workers)}`);
assert.equal(worker.running, 1);
assert.ok(worker.last_claim_at);
});
test('GET /queue lists failures with their frame, and only ones the caller could fix', async () => {
await resetQueue();
const mine = await insertFrame(writeFits('red-q2.fits', [kv('OBJECT', "'M51'")]));
const theirs = await insertFrame(writeFits('red-q3.fits', [kv('OBJECT', "'M64'")]),
{ uploader_id: OTHER_USER, telescope_id: OTHER_TELESCOPE, visibility: 'public' });
for (const id of [mine, theirs]) {
await queue.enqueueReduction(id, { stages: ['header'] });
}
// Burn both jobs to a final failure.
for (let i = 0; i < 2; i++) {
const job = await queue.claimNextJob('w', { local: true });
await db.run('UPDATE reduction_jobs SET attempts = max_attempts WHERE id = ?', [job.id]);
await queue.failJob(job.id, new Error('disk went away'));
}
const { body } = await fetchJson('/api/reduction/queue');
const frames = body.failures.map(f => f.frame_id);
assert.ok(frames.includes(mine), 'my own failure should be listed');
// The other user's frame is PUBLIC, so it is readable, and it sits on a site
// the caller has nothing to do with. It is excluded because this list is
// "what needs you", and there is nothing the caller could do about it.
//
// Note a frame someone else uploaded onto the CALLER'S telescope does belong
// here: the site operator can reprocess it, so it is their problem to fix.
assert.ok(!frames.includes(theirs), 'a frame the caller cannot reprocess should not be listed');
const row = body.failures.find(f => f.frame_id === mine);
assert.equal(row.stage, 'header');
assert.match(row.error, /disk went away/);
assert.ok(row.filename, 'a failure without its frame is not actionable');
assert.equal(row.telescope_name, 'Reduction Scope');
});
test('POST /retry requeues a batch and refuses the frames that are not the callers', async () => {
await resetQueue();
const mine = await insertFrame(writeFits('red-q4.fits', [kv('OBJECT', "'M81'")]));
const theirs = await insertFrame(writeFits('red-q5.fits', [kv('OBJECT', "'M82'")]),
{ uploader_id: OTHER_USER, telescope_id: OTHER_TELESCOPE, visibility: 'public' });
const { status, body } = await fetchJson('/api/reduction/retry', {
method: 'POST',
body: JSON.stringify({ frame_ids: [mine, theirs, 'no-such-frame'] }),
});
assert.equal(status, 202);
assert.deepEqual(body.queued.map(q => q.frame_id), [mine]);
assert.equal(body.refused.length, 2);
assert.ok(body.refused.find(r => r.frame_id === theirs).reason.match(/not authorized/));
assert.ok(body.refused.find(r => r.frame_id === 'no-such-frame').reason.match(/no such frame/));
// The accepted one really was queued, rather than merely reported.
const jobs = await db.all("SELECT stage FROM reduction_jobs WHERE frame_id = ? AND status = 'pending'", [mine]);
assert.ok(jobs.length > 0, 'the retried frame should have pending work');
});
test('a retried frame drops off the needs-you list', async () => {
// Reprocessing creates a new run and keeps the old one, which is the whole
// provenance model. It also means the old failed rows are still in the
// table, so without scoping to the frame's current run the list keeps every
// failure the operator already dealt with and stops meaning "needs you".
await resetQueue();
const frameId = await insertFrame(writeFits('red-q6.fits', [kv('OBJECT', "'M27'")]));
await queue.enqueueReduction(frameId, { stages: ['header'] });
const job = await queue.claimNextJob('w', { local: true });
await db.run('UPDATE reduction_jobs SET attempts = max_attempts WHERE id = ?', [job.id]);
await queue.failJob(job.id, new Error('transient mount error'));
const before = await fetchJson('/api/reduction/queue');
assert.ok(before.body.failures.some(f => f.frame_id === frameId), 'the failure should be listed');
const retry = await fetchJson('/api/reduction/retry', {
method: 'POST', body: JSON.stringify({ frame_ids: [frameId] }),
});
assert.equal(retry.status, 202);
const after = await fetchJson('/api/reduction/queue');
assert.ok(!after.body.failures.some(f => f.frame_id === frameId),
'the superseded failure should be gone once a fresh run is queued');
// The old job row is kept, so the history is not rewritten. It is simply no
// longer something the operator is being asked to act on.
const old = await db.get('SELECT status FROM reduction_jobs WHERE id = ?', [job.id]);
assert.equal(old.status, 'failed');
});
test('POST /retry validates the batch before touching anything', async () => {
const empty = await fetchJson('/api/reduction/retry', { method: 'POST', body: JSON.stringify({ frame_ids: [] }) });
assert.equal(empty.status, 400);
assert.match(empty.body.error, /non-empty array/);
const huge = await fetchJson('/api/reduction/retry', {
method: 'POST',
body: JSON.stringify({ frame_ids: Array.from({ length: 101 }, (_, i) => `f${i}`) }),
});
assert.equal(huge.status, 400);
assert.match(huge.body.error, /at most 100/);
});
test('GET /runs/:id returns the parsed provenance record', async () => {