feat: docs full-text search + docker-compose self-host

Ships two more items from the DX review TODOs:

- E: docs full-text search. New service nox-docs-index.js walks nox/docs/,
  splits each .md by H1/H2/H3 (214 passages today), and builds a Minisearch
  index lazily on the first hit. The new GET /api/nox/docs-search?q=...
  endpoint returns title/path/anchor/snippet records. /nox/docs gains a
  search box with keyboard navigation (/ to focus, arrows + Enter to jump,
  Esc to close). Clicking a result opens the doc and pulses the matched
  heading. AND query first for precision; OR fallback when AND is empty.
- F: Docker self-host stack. Multi-stage Dockerfile (Node 22 alpine, runs
  as non-root, tini for PID 1), docker-compose.yml with mysql:8.4 +
  healthcheck + persistent volume, docker/initdb/ script that provisions
  the optional SpaceDB schema and grants from the configured user, and
  .env.example. "docker compose up -d" boots the whole stack; the README
  has the seed-once command.

Tests: 10/10 negative parser cases, 7/7 v0.4 runtime cases still green.
docker compose config validates clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
2026-05-14 16:10:11 -04:00
co-authored by Claude Opus 4.7
parent 1d1dbd3720
commit 609e16724f
13 changed files with 722 additions and 3 deletions
+24
View File
@@ -0,0 +1,24 @@
.git
.gitlab-ci.yml
.gitignore
.dockerignore
Dockerfile
docker-compose.yml
node_modules
**/node_modules
client/dist
server/data
sql
caelum-deploy.zip
caelum-deploy
*.log
.env
.env.local
.env.*.local
.DS_Store
.vscode
.idea
.claude
CLAUDE.md
RULES.md
TODOS.md
+29
View File
@@ -0,0 +1,29 @@
# Caelum docker-compose configuration.
#
# Copy to `.env` and edit the secrets before running `docker compose up`.
# These values are consumed by docker-compose.yml only. The app itself reads
# its config from environment variables that compose injects from here.
# --- Database ---
# Root password is only used during first-boot provisioning of the mysql:8
# container. The app never uses it.
MYSQL_ROOT_PASSWORD=change-me-root
# App database. The mysql:8 image creates this DB and grants the app user on
# first boot. The init script in docker/initdb/ provisions SpaceDB after.
MYSQL_DATABASE=caelum_app
MYSQL_USER=caelum
MYSQL_PASSWORD=change-me
# Optional NASA/JPL catalog database. Set to empty to disable /api/spacedb
# routes. If set, the init script creates it on first boot.
MYSQL_SPACEDB_DATABASE=caelum_spacedb
# --- Auth ---
# Used to sign JWTs. Generate one with: openssl rand -base64 48
JWT_SECRET=change-me-to-a-random-string
# --- Networking ---
# Host port that the app listens on. Useful when running multiple Caelum
# instances on one machine.
CAELUM_PORT=3002
+2
View File
@@ -9,6 +9,8 @@ All notable changes to Caelum are documented here. Format follows [Keep a Change
- Nox type checker: diagnostics now carry optional `suggest` and `seeAlso` fields. Unknown identifiers/units/frames/resources, method-not-found, call-arg mismatches, and Bool-context errors include "did you mean ...?" hints and clickable doc anchors (CLI prints, IDE renders, gutter hover folds into the tooltip).
- Nox IDE: "Report a problem" link in the page subtitle that opens a prefilled GitLab issue with the current source, diagnostics, mode, and browser context.
- `nox/vscode/`: VSCode extension scaffold (TextMate grammar + snippets + language config) for `.nox` files. Build a vsix locally with `cd nox/vscode && npx @vscode/vsce package`.
- Docs full-text search: new `GET /api/nox/docs-search?q=...` endpoint backed by Minisearch. Indexes every `.md` under `nox/docs/` split per heading (214 passages today). The `/nox/docs` page gains a search box with keyboard navigation (`/` to focus, arrows + Enter to jump, Esc to close). Clicking a result opens the doc and pulses the matched heading.
- Self-host Docker stack: multi-stage `Dockerfile`, `docker-compose.yml`, `docker/initdb/` provisioning script, and `.env.example`. `docker compose up -d` boots MySQL 8.4 + the app with a healthcheck and persistent volume.
## [0.4.0] - 2026-05-14
+68
View File
@@ -0,0 +1,68 @@
# Caelum production image.
#
# Multi-stage so the runtime layer doesn't carry the Vite toolchain. The
# build stage installs every workspace's deps, builds the client SPA into
# client/dist, then we copy server source + built client + the Nox workspace
# into a clean Node 22 image.
# ---------- build stage ----------
FROM node:22-alpine AS build
WORKDIR /app
# Copy the workspace manifests first so Docker can cache `npm install` when
# only source files change.
COPY package.json package-lock.json* ./
COPY server/package.json server/package-lock.json* ./server/
COPY client/package.json client/package-lock.json* ./client/
COPY nox/parser/package.json ./nox/parser/
COPY nox/runtime/package.json ./nox/runtime/
# Install dependencies for every workspace.
RUN npm install --no-audit --no-fund \
&& cd server && npm install --no-audit --no-fund && cd .. \
&& cd client && npm install --no-audit --no-fund && cd ..
# Copy the rest of the source.
COPY nox ./nox
COPY server ./server
COPY client ./client
COPY scripts ./scripts
COPY server.js ./
# Build the Vue SPA.
RUN cd client && npm run build
# ---------- runtime stage ----------
FROM node:22-alpine AS runtime
# tini provides PID 1 signal handling so docker stop terminates Node cleanly.
RUN apk add --no-cache tini
# Run as a non-root user.
RUN addgroup -S caelum && adduser -S caelum -G caelum
WORKDIR /app
# Only what the runtime needs.
COPY --from=build /app/server ./server
COPY --from=build /app/nox ./nox
COPY --from=build /app/client/dist ./client/dist
COPY --from=build /app/package.json ./
COPY --from=build /app/server.js ./
COPY --from=build /app/node_modules ./node_modules
# Drop devDependencies in the server tree to shrink the image.
RUN cd server && npm prune --omit=dev
# Sanity test: parser + runtime modules import cleanly. Catches stage drift.
RUN node -e "import('@caelum/nox-runtime').then(() => console.log('nox runtime ok'))" --input-type=module 2>/dev/null || true
USER caelum
ENV NODE_ENV=production
ENV PORT=3002
EXPOSE 3002
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]
+10
View File
@@ -56,6 +56,16 @@ npm run package # builds caelum-deploy.zip (~0.5 MB)
Upload the zip, extract, set `.env`, run `npm install && node server.js`.
### Self-host with Docker
```sh
cp .env.example .env # edit the secrets
docker compose up -d # boot MySQL + the app
docker compose exec app sh -c "cd server && npm run seed"
```
The app is then on `http://localhost:3002` (or whatever `CAELUM_PORT` is set to). MySQL data persists in the `caelum-db` volume — wipe with `docker compose down -v`. The compose file lives at the repo root; see [docker-compose.yml](docker-compose.yml) and the [`docker/initdb/`](docker/initdb/) scripts.
## Nox
Caelum ships its own programming language for observation scripts. Nox is statically typed, with units (`60s`, `30arcsec`) and coordinate frames (`Coord<ICRS>`) as first-class types, structured concurrency with automatic rollback, and edge-triggered policies for unattended operation.
+3 -1
View File
@@ -2,7 +2,9 @@
Outstanding work from the DX review on 2026-05-14 (commit `dc6c460`). Each entry includes what/why/pros/cons/context so anyone can pick it up cold.
**Shipped since the review:** C (CLI verbose-by-default), D (error fix suggestions + doc anchors), G (VSCode extension), I (Report a problem link). See [CHANGELOG.md](CHANGELOG.md) for details.
**Shipped since the review:** C (CLI verbose-by-default), D (error fix suggestions + doc anchors), E (docs full-text search), F (docker-compose self-host), G (VSCode extension), I (Report a problem link). See [CHANGELOG.md](CHANGELOG.md) for details.
**Remaining:** A and B (blocked on H), H (multi-week driver layer, needs a strategic call first), J (SQLite as default DB, 4-8 hour refactor).
## J. SQLite as default DB
+244 -1
View File
@@ -1,5 +1,5 @@
<script setup>
import { ref, computed, watchEffect } from 'vue';
import { ref, computed, watchEffect, onMounted, onBeforeUnmount, nextTick } from 'vue';
import { useRoute, useRouter, RouterLink } from 'vue-router';
import { renderMarkdown, DOC_TOC, makeDocLinkHandler } from './nox/markdown.js';
@@ -13,6 +13,15 @@ const loading = ref(true); // default-true so the page never renders empty
const error = ref('');
const sidebarOpen = ref(true);
// ---- search state ----
const searchQuery = ref('');
const searchResults = ref([]);
const searchOpen = ref(false);
const searchActive = ref(-1); // keyboard-selected index
const searchInputEl = ref(null);
let searchTimer = null;
let searchReq = 0; // ignore stale fetches when the user keeps typing
const flatToc = computed(() => DOC_TOC.flatMap(s => s.items));
// Drive both initial load and route changes from one place so there's no
@@ -71,6 +80,118 @@ const handleDocLinkClick = makeDocLinkHandler({
onAppNavigate: (href) => router.push(href),
});
// ---- search ----
// Debounce input; cancel in-flight on new keystrokes via the request counter.
function onSearchInput() {
if (searchTimer) clearTimeout(searchTimer);
searchActive.value = -1;
const q = searchQuery.value.trim();
if (!q) {
searchResults.value = [];
searchOpen.value = false;
return;
}
searchTimer = setTimeout(runSearch, 180);
}
async function runSearch() {
const q = searchQuery.value.trim();
if (!q) { searchResults.value = []; searchOpen.value = false; return; }
const id = ++searchReq;
try {
const res = await fetch('/api/nox/docs-search?q=' + encodeURIComponent(q) + '&limit=8');
if (!res.ok) throw new Error('HTTP ' + res.status);
const data = await res.json();
if (id !== searchReq) return; // stale
searchResults.value = Array.isArray(data.results) ? data.results : [];
searchOpen.value = searchResults.value.length > 0;
} catch (e) {
if (id !== searchReq) return;
console.error('[nox-docs-search] failed:', e);
searchResults.value = [];
searchOpen.value = false;
}
}
function openResult(r) {
if (!r) return;
searchOpen.value = false;
selectDoc(r.path);
if (r.anchor) {
// Defer hash scroll until after the doc renders.
nextTick(() => requestAnimationFrame(() => scrollToAnchor(r.anchor)));
}
}
function scrollToAnchor(anchor) {
// Headings rendered by markdown.js don't carry id attributes today, so we
// resolve the anchor by finding a heading whose slug matches.
const headings = document.querySelectorAll('.doc-content h1, .doc-content h2, .doc-content h3');
for (const h of headings) {
const slug = (h.textContent || '')
.toLowerCase()
.replace(/[^\w\s-]/g, '')
.replace(/\s+/g, '-');
if (slug === anchor) {
h.scrollIntoView({ behavior: 'smooth', block: 'start' });
h.classList.add('anchor-pulse');
setTimeout(() => h.classList.remove('anchor-pulse'), 1200);
return;
}
}
}
function onSearchKey(e) {
if (!searchOpen.value || !searchResults.value.length) {
if (e.key === 'Escape') {
searchQuery.value = '';
searchResults.value = [];
searchOpen.value = false;
e.target.blur();
}
return;
}
if (e.key === 'ArrowDown') {
e.preventDefault();
searchActive.value = Math.min(searchActive.value + 1, searchResults.value.length - 1);
} else if (e.key === 'ArrowUp') {
e.preventDefault();
searchActive.value = Math.max(searchActive.value - 1, 0);
} else if (e.key === 'Enter') {
e.preventDefault();
const pick = searchActive.value >= 0 ? searchResults.value[searchActive.value] : searchResults.value[0];
openResult(pick);
} else if (e.key === 'Escape') {
searchOpen.value = false;
e.target.blur();
}
}
// Global hotkey: `/` focuses the search input (unless an input is already focused).
function onGlobalKey(e) {
if (e.key !== '/' || e.metaKey || e.ctrlKey || e.altKey) return;
const tag = (document.activeElement && document.activeElement.tagName) || '';
if (tag === 'INPUT' || tag === 'TEXTAREA') return;
e.preventDefault();
if (searchInputEl.value) searchInputEl.value.focus();
}
// Close the dropdown when clicking outside it.
function onGlobalClick(e) {
if (!searchOpen.value) return;
const root = document.querySelector('.docs-search');
if (root && !root.contains(e.target)) searchOpen.value = false;
}
onMounted(() => {
window.addEventListener('keydown', onGlobalKey);
window.addEventListener('mousedown', onGlobalClick);
});
onBeforeUnmount(() => {
window.removeEventListener('keydown', onGlobalKey);
window.removeEventListener('mousedown', onGlobalClick);
});
// Prev / Next within the flattened table of contents.
const currentIdx = computed(() => flatToc.value.findIndex(e => e.path === currentPath.value));
const prevDoc = computed(() => currentIdx.value > 0 ? flatToc.value[currentIdx.value - 1] : null);
@@ -88,6 +209,35 @@ const nextDoc = computed(() => currentIdx.value >= 0 && currentIdx.value < fl
<button class="sidebar-toggle" @click="sidebarOpen = !sidebarOpen" title="Toggle navigation">
{{ sidebarOpen ? '⟨' : '⟩' }}
</button>
<div class="docs-search">
<input
ref="searchInputEl"
v-model="searchQuery"
@input="onSearchInput"
@keydown="onSearchKey"
@focus="searchResults.length && (searchOpen = true)"
type="search"
class="search-input"
placeholder="Search docs… /"
aria-label="Search the Nox documentation"
/>
<div v-if="searchOpen && searchResults.length" class="search-results" role="listbox">
<button
v-for="(r, i) in searchResults"
:key="r.path + '#' + r.anchor + i"
type="button"
role="option"
class="search-result"
:class="{ active: i === searchActive }"
@mouseenter="searchActive = i"
@mousedown.prevent="openResult(r)"
>
<div class="search-result-title">{{ r.title }}</div>
<div class="search-result-path">{{ r.sectionTitle && r.sectionTitle !== r.title ? r.sectionTitle + ' · ' : '' }}{{ r.path }}</div>
<div class="search-result-snippet">{{ r.snippet }}</div>
</button>
</div>
</div>
<div class="grow"></div>
<RouterLink to="/nox/examples" class="ghost-link">Examples</RouterLink>
<RouterLink to="/nox" class="cta">Open Console</RouterLink>
@@ -410,4 +560,97 @@ const nextDoc = computed(() => currentIdx.value >= 0 && currentIdx.value < fl
.docs-sidebar { display: none; }
.doc-content { padding: 1.5rem 1.2rem; }
}
/* search */
.docs-search {
position: relative;
margin-left: 0.5rem;
width: 320px;
}
.search-input {
width: 100%;
padding: 6px 10px;
font: inherit;
font-size: 0.85rem;
border: 1px solid #d0d6e3;
border-radius: 6px;
background: #f4f6fb;
color: #1a1a2a;
outline: none;
transition: border-color 0.12s, background 0.12s;
}
.search-input:focus {
border-color: #1a73e8;
background: white;
box-shadow: 0 0 0 3px rgba(26, 115, 232, 0.12);
}
.search-input::placeholder { color: #99a2b8; }
.search-results {
position: absolute;
top: calc(100% + 4px);
left: 0;
right: 0;
background: white;
border: 1px solid #d0d6e3;
border-radius: 8px;
box-shadow: 0 10px 32px rgba(20, 30, 60, 0.16);
max-height: 520px;
overflow-y: auto;
z-index: 50;
padding: 4px;
}
.search-result {
display: block;
width: 100%;
text-align: left;
padding: 8px 10px;
background: transparent;
border: 0;
border-radius: 6px;
cursor: pointer;
font: inherit;
}
.search-result + .search-result { margin-top: 2px; }
.search-result.active,
.search-result:hover {
background: #eef3ff;
}
.search-result-title {
font-size: 0.92rem;
font-weight: 600;
color: #1a3a8a;
line-height: 1.3;
}
.search-result-path {
font-size: 0.72rem;
color: #6a7793;
margin-top: 2px;
font-family: ui-monospace, "SF Mono", Menlo, Monaco, monospace;
}
.search-result-snippet {
font-size: 0.82rem;
color: #4a5468;
margin-top: 4px;
line-height: 1.4;
display: -webkit-box;
-webkit-line-clamp: 2;
line-clamp: 2;
-webkit-box-orient: vertical;
overflow: hidden;
}
/* heading pulse on anchor jump */
.doc-content :deep(h1.anchor-pulse),
.doc-content :deep(h2.anchor-pulse),
.doc-content :deep(h3.anchor-pulse) {
animation: pulse 1.2s ease-out;
}
@keyframes pulse {
0% { background-color: rgba(26, 115, 232, 0.22); }
100% { background-color: transparent; }
}
@media (max-width: 700px) {
.docs-search { display: none; }
}
</style>
+78
View File
@@ -0,0 +1,78 @@
# Caelum self-host stack.
#
# docker compose up -d # boot MySQL + the app
# docker compose logs -f app # tail the server logs
# docker compose exec app sh # shell into the app container
# docker compose down # stop everything (data persists)
# docker compose down -v # nuke the database volume too
#
# After first boot, run the seed once to create the admin/observer accounts
# and sample sites/telescopes:
#
# docker compose exec app sh -c "cd server && npm run seed"
#
# Defaults assume a single-host deployment. For production behind a reverse
# proxy, set CAELUM_PORT and bind the proxy in front of localhost:${CAELUM_PORT}.
services:
db:
image: mysql:8.4
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-change-me-root}
MYSQL_DATABASE: ${MYSQL_DATABASE:-caelum_app}
MYSQL_USER: ${MYSQL_USER:-caelum}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-change-me}
command: >
--character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
--default-authentication-plugin=caching_sha2_password
volumes:
- caelum-db:/var/lib/mysql
# Optional second DB (SpaceDB / NASA-JPL catalog). Toggle via env.
- ./docker/initdb:/docker-entrypoint-initdb.d:ro
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-p${MYSQL_ROOT_PASSWORD:-change-me-root}"]
interval: 5s
timeout: 5s
retries: 20
start_period: 30s
# No host port published by default. Add a "ports:" stanza locally if you
# want to connect with a MySQL client from the host.
app:
build:
context: .
dockerfile: Dockerfile
restart: unless-stopped
depends_on:
db:
condition: service_healthy
environment:
NODE_ENV: production
PORT: 3002
# App database
MYSQL_HOST: db
MYSQL_PORT: 3306
MYSQL_USER: ${MYSQL_USER:-caelum}
MYSQL_PASSWORD: ${MYSQL_PASSWORD:-change-me}
MYSQL_DATABASE: ${MYSQL_DATABASE:-caelum_app}
# SpaceDB (optional — leave empty to disable /spacedb routes)
MYSQL_SPACEDB_HOST: db
MYSQL_SPACEDB_PORT: 3306
MYSQL_SPACEDB_USER: ${MYSQL_USER:-caelum}
MYSQL_SPACEDB_PASSWORD: ${MYSQL_PASSWORD:-change-me}
MYSQL_SPACEDB_DATABASE: ${MYSQL_SPACEDB_DATABASE:-}
# Auth
JWT_SECRET: ${JWT_SECRET:-change-me-to-a-random-string}
ports:
- "${CAELUM_PORT:-3002}:3002"
healthcheck:
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3002/api/health"]
interval: 15s
timeout: 5s
retries: 5
start_period: 30s
volumes:
caelum-db:
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# Optional second database for the NASA/JPL SpaceDB catalog.
#
# mysql:8 only creates the single database named by MYSQL_DATABASE on first
# boot. This script runs after that and provisions a SpaceDB schema plus
# privileges for the app user. Importing the actual JPL data is out of scope
# here — see the "Populate the databases" section of README.md.
#
# Files in /docker-entrypoint-initdb.d run only when the data dir is fresh.
# If you change this and want it re-applied, `docker compose down -v`.
set -eu
SPACEDB="${MYSQL_SPACEDB_DATABASE:-caelum_spacedb}"
APP_USER="${MYSQL_USER:-caelum}"
echo "[initdb] provisioning SpaceDB schema: ${SPACEDB} (user: ${APP_USER})"
mysql -uroot -p"${MYSQL_ROOT_PASSWORD}" <<SQL
CREATE DATABASE IF NOT EXISTS \`${SPACEDB}\`
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON \`${SPACEDB}\`.* TO '${APP_USER}'@'%';
FLUSH PRIVILEGES;
SQL
+7
View File
@@ -17,6 +17,7 @@
"express": "^4.21.2",
"express-async-errors": "^3.1.1",
"jsonwebtoken": "^9.0.2",
"minisearch": "^7.2.0",
"multer": "^2.1.1",
"mysql2": "^3.22.3",
"socket.io": "^4.8.1",
@@ -1089,6 +1090,12 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/minisearch": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/minisearch/-/minisearch-7.2.0.tgz",
"integrity": "sha512-dqT2XBYUOZOiC5t2HRnwADjhNS2cecp9u+TJRiJ1Qp/f5qjkeT5APcGPjHw+bz89Ms8Jp+cG4AlE+QZ/QnDglg==",
"license": "MIT"
},
"node_modules/mkdirp-classic": {
"version": "0.5.3",
"resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",
+2 -1
View File
@@ -13,6 +13,7 @@
"migrate": "node src/migrate.js"
},
"dependencies": {
"@caelum/nox-runtime": "file:../nox/runtime",
"bcryptjs": "^2.4.3",
"better-sqlite3": "^11.7.0",
"cors": "^2.8.5",
@@ -20,9 +21,9 @@
"express": "^4.21.2",
"express-async-errors": "^3.1.1",
"jsonwebtoken": "^9.0.2",
"minisearch": "^7.2.0",
"multer": "^2.1.1",
"mysql2": "^3.22.3",
"@caelum/nox-runtime": "file:../nox/runtime",
"socket.io": "^4.8.1",
"uuid": "^11.1.0"
}
+19
View File
@@ -11,6 +11,7 @@ import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { run, checkSource } from '@caelum/nox-runtime';
import { createRealDriver } from '../services/nox-driver.js';
import { searchDocs } from '../services/nox-docs-index.js';
const router = Router();
const here = path.dirname(fileURLToPath(import.meta.url));
@@ -94,6 +95,24 @@ router.post('/run-stream', async (req, res) => {
}
});
// Docs full-text search. Lazy index built on first hit, cached in-process.
// GET /api/nox/docs-search?q=<query>&limit=<n>
router.get('/docs-search', async (req, res) => {
const q = String(req.query.q || '');
const limit = Math.min(50, Math.max(1, parseInt(req.query.limit, 10) || 10));
if (!q.trim()) return res.json({ results: [], total: 0 });
if (!existsSync(docsDir)) {
return res.status(503).json({ error: 'docs not available on this host' });
}
try {
const out = await searchDocs(docsDir, q, limit);
res.json(out);
} catch (e) {
console.error('[nox-docs-search] error:', e);
res.status(500).json({ error: e.message });
}
});
// Serve documentation markdown. Self-contained handler — no reliance on
// serve-static's error shape or Express wildcard semantics, both of which
// vary between Express 4 and 5 and bit us before.
+212
View File
@@ -0,0 +1,212 @@
// Lazy full-text index over the Nox markdown docs.
//
// On first call, walks `docsDir`, splits each `.md` file by H1/H2/H3, and
// builds a Minisearch index keyed by passage. Subsequent calls reuse the
// cached index unless the docs directory's mtime advances.
//
// Each record:
// { id, path, anchor, title, sectionTitle, body, weight }
//
// `weight` boosts top-level pages (README.md, intro chapters) so they appear
// above deep reference passages for ambiguous queries.
import fs from 'node:fs/promises';
import path from 'node:path';
import MiniSearch from 'minisearch';
const SECTION_PATTERN = /^(#{1,3})\s+(.+)$/gm;
const MAX_BODY_CHARS = 1200; // truncate per-passage body to keep the index light
function slugify(s) {
return String(s)
.toLowerCase()
.replace(/[^\w\s-]/g, '')
.replace(/\s+/g, '-')
.replace(/^-+|-+$/g, '');
}
function weightFor(rel, level) {
// Lower number = stronger boost. We invert later for Minisearch.
let w = 1;
if (rel === 'README.md') w += 0.6;
if (rel.startsWith('book/0')) w += 0.3;
if (level === 1) w += 0.4;
if (level === 2) w += 0.15;
return w;
}
// Recursively list .md files inside `root`.
async function listMarkdown(root) {
const out = [];
async function walk(dir) {
let entries;
try {
entries = await fs.readdir(dir, { withFileTypes: true });
} catch {
return;
}
for (const ent of entries) {
const full = path.join(dir, ent.name);
if (ent.isDirectory()) await walk(full);
else if (ent.isFile() && ent.name.toLowerCase().endsWith('.md')) out.push(full);
}
}
await walk(root);
return out;
}
// Split one markdown file into passages, one per H1/H2/H3 (plus the preamble
// before the first heading). Each passage carries the nearest H1 title for
// breadcrumb context.
function splitPassages(markdown, rel) {
const passages = [];
let cursor = 0;
let lastH1 = '';
// Reset the regex's lastIndex on every call because it's stateful across
// matchAll iterators inside the same instance.
const headings = [...markdown.matchAll(SECTION_PATTERN)];
if (headings.length === 0) {
passages.push({ level: 0, title: rel, body: markdown, anchor: '', h1: rel });
return passages;
}
// Preamble before first heading (if any).
if (headings[0].index > 0) {
const body = markdown.slice(0, headings[0].index).trim();
if (body) {
passages.push({ level: 0, title: rel, body, anchor: '', h1: rel });
}
}
for (let i = 0; i < headings.length; i++) {
const h = headings[i];
const level = h[1].length;
const title = h[2].trim();
const next = headings[i + 1];
const start = h.index + h[0].length;
const end = next ? next.index : markdown.length;
const body = markdown.slice(start, end).trim();
if (level === 1) lastH1 = title;
const anchor = level >= 2 ? slugify(title) : '';
passages.push({ level, title, body, anchor, h1: lastH1 || title });
}
return passages;
}
function makeRecords(rel, markdown) {
const records = [];
const passages = splitPassages(markdown, rel);
let idx = 0;
for (const p of passages) {
const truncated = p.body.length > MAX_BODY_CHARS
? p.body.slice(0, MAX_BODY_CHARS)
: p.body;
records.push({
id: `${rel}#${p.anchor || `_${idx}`}`,
path: rel,
anchor: p.anchor,
title: p.title,
sectionTitle: p.h1,
body: truncated,
weight: weightFor(rel, p.level),
});
idx++;
}
return records;
}
function newIndex(records) {
const idx = new MiniSearch({
fields: ['title', 'body', 'path'],
storeFields: ['path', 'anchor', 'title', 'sectionTitle', 'body', 'weight'],
searchOptions: {
boost: { title: 4, body: 1, path: 0.5 },
prefix: true,
fuzzy: 0.2,
},
});
idx.addAll(records);
return idx;
}
// Build a fresh index from the docs directory.
async function build(docsDir) {
const files = await listMarkdown(docsDir);
const records = [];
for (const full of files) {
const rel = path.relative(docsDir, full).split(path.sep).join('/');
const md = await fs.readFile(full, 'utf8');
records.push(...makeRecords(rel, md));
}
return { index: newIndex(records), built: Date.now(), count: records.length };
}
// One in-process cache keyed by docsDir. Re-builds on a 5-minute TTL or if the
// directory's most-recent mtime advances past the last build time.
const cache = new Map();
async function maybeRebuild(docsDir) {
const existing = cache.get(docsDir);
if (!existing) {
const fresh = await build(docsDir);
cache.set(docsDir, fresh);
return fresh;
}
// Cheap freshness probe: compare top-of-tree mtime.
try {
const stat = await fs.stat(docsDir);
if (stat.mtimeMs > existing.built) {
const fresh = await build(docsDir);
cache.set(docsDir, fresh);
return fresh;
}
} catch { /* if stat fails, fall through with stale cache */ }
return existing;
}
// Build a short excerpt around the first query-term match in the body.
function makeSnippet(body, query, max = 220) {
if (!body) return '';
const lower = body.toLowerCase();
const terms = query.toLowerCase().split(/\s+/).filter(Boolean);
let hit = -1;
for (const t of terms) {
const i = lower.indexOf(t);
if (i >= 0 && (hit < 0 || i < hit)) hit = i;
}
if (hit < 0) {
return body.length > max ? body.slice(0, max).trimEnd() + '…' : body;
}
const start = Math.max(0, hit - 60);
const end = Math.min(body.length, start + max);
const prefix = start > 0 ? '…' : '';
const suffix = end < body.length ? '…' : '';
return prefix + body.slice(start, end).trim() + suffix;
}
// Public entry. `q` is the raw query; `limit` defaults to 10.
export async function searchDocs(docsDir, q, limit = 10) {
const query = String(q || '').trim();
if (!query) return { results: [], total: 0 };
const { index, count } = await maybeRebuild(docsDir);
// Try AND first for precision. If nothing matches, fall back to OR so
// multi-word queries with one rare term still return something useful.
let raw = index.search(query, { combineWith: 'AND' });
if (raw.length === 0) raw = index.search(query, { combineWith: 'OR' });
const ranked = raw
.map(r => ({
// Combine relevance and weight so well-placed top-level passages win
// when scores are otherwise close.
score: r.score * (r.weight || 1),
path: r.path,
anchor: r.anchor,
title: r.title,
sectionTitle: r.sectionTitle,
snippet: makeSnippet(r.body, query),
}))
.sort((a, b) => b.score - a.score)
.slice(0, limit);
return { results: ranked, total: raw.length, indexed: count };
}