vincent/gerbeur
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v3: comments say what the code can't — the documentation pass wanted JSDoc on every export and every field of an exported interface, which filled the code with restatements ("The dump.", "Whether it is loading.", "The props of X.", "Renders the X.") and module docs that told a reader nothing ("The hot and new feeds, and the props every main feed of the index takes."); some were wrong as well as vague. Every comment in api/, shared/ and src/ is rewritten under one rule, now stated in the README's "Comments" section: every module and export still has a JSDoc, and it gives context (who uses it and where), usage (what a caller must know: a precondition, what it leaves to the caller, a side effect) or its place in the chain (where its input comes from, where its output goes); a field, parameter, private function or local gets one only when it adds to its parent's, and context about a whole type goes on the type; a type's doc is phrased from the value's side ("Returned to Y by X when Z", "Passed to Y by X"); full sentences, as short as they can be, no label-colon-list; // goes wherever the code isn't obvious and says why, and the section banners and narration go. Every claim was checked against the code, which turned up some forty comments that were wrong: DumpCard's claimed it was only used by the hot and new feeds, the WebSocket welcome claimed to be a socket's first message (presence comes first), UpdateUserRequest said null removes a description (it keeps it), deletedDumpIds leaves out that it also holds dumps made private, THEME_COLOR claimed to mirror index.html's, and the like. The routers' JSDoc now says where they are mounted and what becomes of a thrown APIException. doc-lint.ts lets a field without a doc through; a module or export without one still fails. No code changes; 204 backend and 628 frontend tests pass, and doc:lint, deno check, deno lint and tsc are clean.

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
khannurien
2026-09-29 14:31:14 +00:00
parent 42e0356d21
commit 0d4e344d77
287 changed files with 5679 additions and 5135 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
api/remote/fetch.ts
View File

@@ -1,7 +1,8 @@
/**
* The guarded fetch every request to another site goes through: each hop must
* resolve to a public address, the whole exchange has a deadline, and pages are
* read up to a cap.
* Fetches pages and images from other sites for link previews, the image proxy
* and the Bandcamp resolver, without letting a user-supplied URL reach the
* server's own network. Each hop must resolve to a public address, the whole
* exchange has a deadline, and pages are read up to a cap.
*
* @module
*/
@@ -20,15 +21,11 @@ const FETCH_HEADERS = {
const MAX_REDIRECTS = 5;
/**
* Fetches `url` through a `curl --insecure` subprocess and returns a minimal
* Response.
* Used as a last resort when Deno's rustls rejects the server's TLS certificate
* due to an unsupported algorithm (e.g. UnsupportedSignatureAlgorithm).
* Returns null if curl is unavailable or exits non-zero.
*
* `address` pins the connection to the IP already checked by
* resolvePublicAddress, and curl doesn't follow redirects itself: a 3xx comes
* back with its Location so fetchWithTimeout can check the next hop too.
* Fetches `url` with `curl --insecure`, for a site whose TLS certificate Deno
* rejects. Returns a Response carrying only the `Location` and `Content-Type`
* headers, or null when curl is missing or fails. The connection is pinned to
* `address`, the one {@link resolvePublicAddress} checked, and redirects come
* back unfollowed so that {@link fetchWithTimeout} checks each hop.
*/
async function fetchViaCurl(
url: URL,
@@ -49,6 +46,8 @@ async function fetchViaCurl(
`${url.hostname.replace(/^\[(.*)\]$/, "$1")}:${port}:${pinned}`,
"--max-time",
String(Math.ceil(timeoutMs / 1000)),
// curl fails on a larger file rather than stopping at a cap, so this is
// looser than MAX_PAGE_BYTES, where readTextCapped still cuts.
"--max-filesize",
String(2 * MAX_PAGE_BYTES),
"--user-agent",
@@ -81,8 +80,8 @@ async function fetchViaCurl(
if (!contentType.startsWith("text/")) {
return new Response(bytes, { status: statusCode, headers });
}
// Decode using the charset declared in the Content-Type header so that
// pages served in ISO-8859-1, windows-1252, etc. are read correctly.
// readTextCapped decodes as UTF-8, so a page in another charset is
// re-encoded here.
const charset = /charset=([\w-]+)/i.exec(contentType)?.[1] ?? "utf-8";
let bodyText: string;
try {
@@ -100,15 +99,15 @@ async function fetchViaCurl(
}
/**
* The most bytes of a fetched page that are read. Metadata lives in the <head>,
* and a "page" past this is a download, not something to scrape.
* The default cap of {@link readTextCapped}. The metadata previews read is in
* the `<head>`, and a page past this is a download.
*/
export const MAX_PAGE_BYTES = 4 * 1024 * 1024;
/**
* Hands `body` on, and clears the request's deadline once it has been read to
* the end or cancelled. Until then the deadline still applies: a server that
* answers at once and then drips its body can't hold the request open.
* Wraps `body` so that the request's deadline `timer` is cleared only once the
* body is read to the end or cancelled. A server that answers at once and then
* drips its body is still cut off.
*/
function withDeadline(
body: ReadableStream<Uint8Array>,
@@ -138,8 +137,9 @@ function withDeadline(
}
/**
* Reads a response's body as text, stopping at `maxBytes` (the rest is
* cancelled, never downloaded).
* Reads the body of `res` as UTF-8 text, up to `maxBytes`; the rest is
* cancelled, never downloaded. The providers and the Bandcamp and YouTube
* resolvers read pages through it.
*/
export async function readTextCapped(
res: Response,
@@ -170,7 +170,10 @@ export async function readTextCapped(
return new TextDecoder().decode(bytes);
}
/** Makes one request, following no redirects. The timeout covers the body too. */
/**
* Makes one request to `url`, following no redirects, with a deadline that
* covers the body too.
*/
async function fetchOnce(
url: URL,
address: string,
@@ -191,10 +194,9 @@ async function fetchOnce(
throw err;
}
// Deno/rustls rejected the server's TLS certificate (expired, self-signed,
// or an unsupported signature algorithm) and offers no way to override
// verification programmatically. Fall back to a `curl --insecure`
// subprocess, which uses its own TLS stack.
// Deno can't be told to skip certificate checks for one request, so a
// site with an expired, self-signed or unusually signed certificate is
// fetched with curl instead.
const curlRes = await fetchViaCurl(url, address, timeoutMs);
if (curlRes) return curlRes;
@@ -212,10 +214,10 @@ async function fetchOnce(
}
/**
* Fetches a user-supplied URL, following up to five redirects. Every hop — the
* URL itself and each redirect — must resolve to a public address (see
* {@link resolvePublicAddress}), so a link or a redirect can't point the
* server at its own network.
* Fetches a user-supplied URL, following up to five redirects, each of which
* must resolve to a public address too. `timeoutMs` bounds each hop, body
* included. The caller must read or cancel the body, or its connection stays
* open.
*
* @throws {TypeError} When a hop isn't http(s), there are too many redirects,
* or the request fails.
@@ -242,16 +244,17 @@ export async function fetchWithTimeout(
}
/**
* Checks whether `raw` is an http(s) URL a dump may link to: not on a private
* host, unless it is one of this site's own pages.
* Checks whether `raw` is an http(s) URL on a host that isn't private, or a
* link to one of this site's own pages. Routes and services check a URL with it
* before storing, previewing or proxying it; it does no DNS lookup, which
* {@link fetchWithTimeout} does.
*/
export function isValidHttpUrl(raw: string): boolean {
try {
const u = new URL(raw);
if (u.protocol !== "http:" && u.protocol !== "https:") return false;
// This site's own pages are fine even on a private host (e.g. localhost in
// development): selfProvider reads them from the DB, no HTTP involved.
// Every actual fetch is checked again, after DNS, in fetchWithTimeout.
// This site may run on a private host (localhost in development), and the
// self provider reads its pages from the database without fetching them.
return !isPrivateHostname(u) || isSelfUrl(u);
} catch {
return false;