Range requests for large attachments (relay#37) #104

2026-09-04T15:50:43Z

paulhitt commented

2026-09-04 15:50:43 +00:00

(Migrated from gitlab.com)

Half of this already worked and no client could have found out. Vapor's
streamFile has always read the Range header and answered 206 with a
Content-Range, so the directory store has been capable of partial responses
the whole time — but nothing ever sent Accept-Ranges: bytes, and a client
that is not told does not speculatively try one. The capability existed and
the feature did not, which is why this read as unimplemented from outside.

Three things were actually missing.

The advertisement, which is the one that made the rest reachable.

The refusal, which has to be 416 and was 400. A phone resuming a download
asks for bytes=900- because it already holds 900 bytes; if the blob was
replaced or truncated, 400 Bad Request tells it the request was malformed,
which is untrue and gives it nothing to do — it retries the same broken resume
forever. 416 with Content-Range: bytes */<total> hands it the real length,
which is the one fact that lets it start again from zero.

Object storage, which ignored Range entirely and returned whole objects. The
range now goes to the bucket rather than being applied after the fact —
fetching everything and slicing here would answer correctly while paying the
full transfer on every partial request, which is the exact cost a resumable
download exists to avoid, moved from the phone to the relay's egress bill. The
upstream header is rebuilt from the decision rather than forwarded, so a
bucket is never handed a string a stranger wrote: buckets will happily answer
the multi-range and suffix forms this relay declines, and forwarding would
mean the rules applied to local blobs and not to remote ones.

The range is measured against the length recorded at upload, not the file's
own — a truncated blob answering ranges against its truncated size would look
perfectly self-consistent while being the wrong file. The relay#26 integrity
check still runs and still refuses, in pieces as well as whole, so ranges do
not become the way around it. Both are tested.

The privacy cost is real, and sharper than the issue states. A partial
fetch says somebody stopped listening to a voice note after three seconds;
a whole-file fetch cannot say that. There is no way to offer resumable
downloads without the server seeing the offset, so the only honest mitigation
is to not write it down: not logged, not counted per blob, not stored on the
row. docs/API.md says so plainly, including that a relay you do not run could
choose otherwise.

647 relay tests in 57 suites green, core paths 7/7.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Half of this already worked and no client could have found out. Vapor's `streamFile` has always read the `Range` header and answered 206 with a `Content-Range`, so the directory store has been capable of partial responses the whole time — but nothing ever sent `Accept-Ranges: bytes`, and a client that is not told does not speculatively try one. The capability existed and the feature did not, which is why this read as unimplemented from outside. Three things were actually missing. The advertisement, which is the one that made the rest reachable. The refusal, which has to be 416 and was 400. A phone resuming a download asks for `bytes=900-` because it already holds 900 bytes; if the blob was replaced or truncated, `400 Bad Request` tells it the request was malformed, which is untrue and gives it nothing to do — it retries the same broken resume forever. `416` with `Content-Range: bytes */<total>` hands it the real length, which is the one fact that lets it start again from zero. Object storage, which ignored `Range` entirely and returned whole objects. The range now goes to the bucket rather than being applied after the fact — fetching everything and slicing here would answer correctly while paying the full transfer on every partial request, which is the exact cost a resumable download exists to avoid, moved from the phone to the relay's egress bill. The upstream header is rebuilt from the decision rather than forwarded, so a bucket is never handed a string a stranger wrote: buckets will happily answer the multi-range and suffix forms this relay declines, and forwarding would mean the rules applied to local blobs and not to remote ones. The range is measured against the length recorded at upload, not the file's own — a truncated blob answering ranges against its truncated size would look perfectly self-consistent while being the wrong file. The relay#26 integrity check still runs and still refuses, in pieces as well as whole, so ranges do not become the way around it. Both are tested. The privacy cost is real, and sharper than the issue states. A partial fetch says somebody stopped listening to a voice note after three seconds; a whole-file fetch cannot say that. There is no way to offer resumable downloads without the server seeing the offset, so the only honest mitigation is to not write it down: not logged, not counted per blob, not stored on the row. docs/API.md says so plainly, including that a relay you do not run could choose otherwise. 647 relay tests in 57 suites green, core paths 7/7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

paulhitt commented

2026-09-04 15:50:43 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt commented

2026-09-04 20:41:03 +00:00

(Migrated from gitlab.com)

deleted the feat/pooling-and-read-path branch. This merge request now targets the main branch

deleted the `feat/pooling-and-read-path` branch. This merge request now targets the `main` branch

paulhitt commented

2026-09-04 20:41:05 +00:00

(Migrated from gitlab.com)

mentioned in commit b34c3aea46

mentioned in commit b34c3aea46755807b393d16d2ff69089a43d885f

paulhitt (Migrated from gitlab.com) merged commit b34c3aea46 into main

2026-09-04 20:41:05 +00:00

paulhitt commented

2026-09-09 18:59:06 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#104