vincent/gerbeur
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v3: comments say what the code can't — the documentation pass wanted JSDoc on every export and every field of an exported interface, which filled the code with restatements ("The dump.", "Whether it is loading.", "The props of X.", "Renders the X.") and module docs that told a reader nothing ("The hot and new feeds, and the props every main feed of the index takes."); some were wrong as well as vague. Every comment in api/, shared/ and src/ is rewritten under one rule, now stated in the README's "Comments" section: every module and export still has a JSDoc, and it gives context (who uses it and where), usage (what a caller must know: a precondition, what it leaves to the caller, a side effect) or its place in the chain (where its input comes from, where its output goes); a field, parameter, private function or local gets one only when it adds to its parent's, and context about a whole type goes on the type; a type's doc is phrased from the value's side ("Returned to Y by X when Z", "Passed to Y by X"); full sentences, as short as they can be, no label-colon-list; // goes wherever the code isn't obvious and says why, and the section banners and narration go. Every claim was checked against the code, which turned up some forty comments that were wrong: DumpCard's claimed it was only used by the hot and new feeds, the WebSocket welcome claimed to be a socket's first message (presence comes first), UpdateUserRequest said null removes a description (it keeps it), deletedDumpIds leaves out that it also holds dumps made private, THEME_COLOR claimed to mirror index.html's, and the like. The routers' JSDoc now says where they are mounted and what becomes of a thrown APIException. doc-lint.ts lets a field without a doc through; a module or export without one still fails. No code changes; 204 backend and 628 frontend tests pass, and doc:lint, deno check, deno lint and tsc are clean.

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
khannurien
2026-09-29 14:31:14 +00:00
parent 42e0356d21
commit 0d4e344d77
287 changed files with 5679 additions and 5135 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
api/remote/html.ts
View File

@@ -1,11 +1,17 @@
/**
* Scraping helpers: the metadata a link preview reads from a page's HTML —
* meta and OG tags, the title, JSON-LD, images, icons and brand colors.
* Reads a link preview's metadata out of a page's HTML with regular
* expressions, without parsing the document. The generic provider uses most of
* it; the Bandcamp, SoundCloud and YouTube providers read only OG tags. Each
* function returns undefined when the page doesn't say.
*
* @module
*/
/** Decodes the named and numeric HTML entities in `str`. */
/**
* Decodes the five XML entities and numeric ones in `str`. `&amp;` goes first,
* so `&amp;quot;` decodes twice; `bandcamp-tralbum.ts`, which decodes JSON,
* has its own single-pass decoder.
*/
function decodeHtmlEntities(str: string): string {
return str
.replace(/&amp;/gi, "&")
@@ -21,9 +27,9 @@ function decodeHtmlEntities(str: string): string {
}
/**
* Returns the `content` of the first `<meta>` tag whose `attr` equals `value`
* (either attribute ordering). Each tag is matched in isolation so
* the `content` capture can never span across tag boundaries.
* Returns the `content` of the first `<meta>` tag whose `attr` equals `value`,
* in whichever order the attributes come. Each tag is matched on its own, so
* the capture can't run into the next tag.
*/
function extractMetaContent(
html: string,
@@ -44,7 +50,7 @@ function extractMetaContent(
return undefined;
}
/** Returns the content of `<meta property="og:<tag>">`. */
/** Returns the content of `<meta property="og:<tag>">`, entities decoded. */
export function extractOgTag(
html: string,
tag: string,
@@ -53,8 +59,8 @@ export function extractOgTag(
}
/**
* Returns the content of `<meta name="…" content="…">` (both attribute
* orderings).
* Returns the content of `<meta name="<name>">`, entities decoded. Twitter
* card tags use `name`, not `property`.
*/
export function extractMetaName(
html: string,
@@ -63,20 +69,16 @@ export function extractMetaName(
return extractMetaContent(html, "name", name);
}
/** Returns the text content of the `<title>` element. */
/** Returns the text of the `<title>` element, trimmed and entities decoded. */
export function extractPageTitle(html: string): string | undefined {
const match = html.match(/<title[^>]*>([^<]+)<\/title>/i);
return match ? decodeHtmlEntities(match[1].trim()) : undefined;
}
// ── Brand color helpers ───────────────────────────────────────────────────────
/**
* Returns the page's `theme-color`.
*
* Sites routinely ship the tag twice, scoped by `media`, and often list the
* dark-scheme one first — so unlike `extractMetaName` this can't just take the
* first match. Prefer the unscoped tag, then the light-scheme one, then any.
* Returns the page's `theme-color`, as written: pass it to
* {@link normalizeCssColor}. Sites often ship one per color scheme with the
* dark one first, so the unscoped tag wins, then the light one, then any.
*/
export function extractThemeColor(html: string): string | undefined {
const tagRe = /<meta\b[^>]*>/gi;
@@ -100,7 +102,10 @@ export function extractThemeColor(html: string): string | undefined {
return unscoped ?? light ?? any;
}
/** Returns the `color` of `<link rel="mask-icon">` (Safari pinned tabs). */
/**
* Returns the `color` of `<link rel="mask-icon">`, Safari's pinned-tab icon,
* as written. The generic provider's last resort for a brand color.
*/
export function extractMaskIconColor(html: string): string | undefined {
const linkRe = /<link[^>]+>/gi;
let m: RegExpExecArray | null;
@@ -114,8 +119,8 @@ export function extractMaskIconColor(html: string): string | undefined {
}
/**
* The handful of named colors that actually show up in `theme-color`. The full
* 148-name table isn't worth carrying for the long tail.
* The named colors sites actually put in `theme-color`; the rest of CSS's names
* aren't worth carrying.
*/
const NAMED_COLORS: Record<string, string> = {
black: "#000000",
@@ -145,9 +150,10 @@ function clampByte(n: number): number {
}
/**
* Normalizes a CSS color to canonical lowercase `#rrggbb`, or `undefined` when
* it isn't one of the forms sites actually use. Alpha is dropped — the color is
* only ever used as a tint over the app's own surface.
* Returns `raw` as a lowercase `#rrggbb`, from a hex, `rgb()`, `rgba()` or
* common named color; undefined for any other form. The generic provider
* stores it as the preview's `accentColor`, a tint over the app's own surface,
* so alpha is dropped.
*/
export function normalizeCssColor(raw: string | undefined): string | undefined {
if (!raw) return undefined;
@@ -182,17 +188,17 @@ export function normalizeCssColor(raw: string | undefined): string | undefined {
return undefined;
}
// ── JSON-LD helpers ───────────────────────────────────────────────────────────
/** What a page's JSON-LD says about it. */
/**
* Returned to the generic provider by {@link extractJsonLd}, which ranks each
* field below the page's OG and Twitter tags.
*/
export interface JsonLdResult {
/** The node's `name` or `headline`. */
/** From `name`, or else `headline`. */
title?: string;
/** The node's `description`. */
description?: string;
/**
* The node's `image`, `thumbnailUrl` or `thumbnail`, when it is an absolute
* URL.
* From `image`, or else `thumbnailUrl` or `thumbnail`, and only when it is
* an absolute URL.
*/
thumbnailUrl?: string;
}
@@ -237,9 +243,10 @@ function ldExtractNode(data: unknown): JsonLdResult {
}
/**
* Parses every `<script type="application/ld+json">` block and returns the first
* node that yields a title or image. Handles `@graph`, arrays, and the common
* `image` shapes (string, string[], ImageObject).
* Returns the first JSON-LD node in `html` that has a title or an image,
* looking through `@graph` and arrays; empty when there is none. An image may
* be a URL, a list of them or an `ImageObject`. Blocks that aren't valid JSON
* are skipped.
*/
export function extractJsonLd(html: string): JsonLdResult {
const pattern =
@@ -249,14 +256,15 @@ export function extractJsonLd(html: string): JsonLdResult {
try {
const result = ldExtractNode(JSON.parse(match[1]));
if (result.title || result.thumbnailUrl) return result;
} catch { /* invalid JSON — skip */ }
} catch { /* invalid JSON: try the next block */ }
}
return {};
}
/**
* Returns the `src` of the first `<img>` whose declared width or height is at
* least `minSize` pixels (default 200). Skips data URIs. Resolves relative URLs.
* Returns the absolute URL of the first `<img>` whose declared width and height
* are both at least `minSize` pixels; an image without them doesn't count.
* Data URIs are skipped.
*/
export function extractLargeImage(
html: string,
@@ -283,13 +291,12 @@ export function extractLargeImage(
}
/**
* Collects all `<link rel="icon">` / `<link rel="apple-touch-icon">` tags, ranks
* them by declared size (largest wins), and returns the best resolved URL.
* Falls back to the first match when no `sizes` attribute is present.
* Returns the absolute URL of the page's largest icon, among its `icon` and
* `apple-touch-icon` links, by declared `sizes`; the first one when none
* declares a size. The generic provider stores it as the preview's
* `faviconUrl`, which the thumbnail placeholder draws.
*
* SVG icons are ranked above everything: they carry no `sizes` attribute, so
* they'd otherwise score 0 and lose to a 16×16 PNG, yet they scale cleanly to
* whatever size the placeholder renders them at.
* An SVG icon wins over any size: it declares none, yet scales to any.
*/
export function extractBestIcon(
html: string,
@@ -330,7 +337,7 @@ export function extractBestIcon(
return candidates[0].href;
}
/** Filename tokens that mark an image as UI chrome rather than content. */
/** Filename words that mark an image as page furniture rather than content. */
const UI_IMAGE_KEYWORDS = new Set([
"logo",
"icon",
@@ -350,24 +357,22 @@ const UI_IMAGE_KEYWORDS = new Set([
]);
/**
* Checks whether `src` looks like UI chrome: an SVG, or a filename with one of
* the {@link UI_IMAGE_KEYWORDS}.
* Checks whether `src` looks like page furniture: an SVG, or a filename with
* one of the {@link UI_IMAGE_KEYWORDS} between its `-`, `_` or `.`.
*/
function isUiImage(src: string): boolean {
if (/\.svg(\?|$)/i.test(src)) return true;
const filename = src.split("?")[0].split("/").pop() ?? "";
const baseName = filename.replace(/\.[^.]+$/, ""); // strip extension
// Split on common filename separators (-, _, .) and check each token
const baseName = filename.replace(/\.[^.]+$/, "");
return baseName.toLowerCase().split(/[-_.]/).some((t) =>
UI_IMAGE_KEYWORDS.has(t)
);
}
/**
* Returns the `src` of the first `<img>` that looks like content rather than UI
* chrome. Skips SVGs, data URIs, and images whose filename tokens match common
* icon/logo/nav patterns (logo, icon, sprite, etc.).
* Resolves relative and protocol-relative URLs.
* Returns the absolute URL of the first `<img>` that looks like content rather
* than a logo, icon or other page furniture. Data URIs are skipped. The generic
* provider's last resort for a thumbnail.
*/
export function extractFirstContentImage(
html: string,
@@ -390,8 +395,8 @@ export function extractFirstContentImage(
}
/**
* Returns the `href` of the first `<link rel="…">` whose rel contains
* `relFragment`, resolved to an absolute URL using `baseUrl`.
* Returns the absolute URL of the first `<link>` whose `rel` contains
* `relFragment`. No provider calls it; only its tests do.
*/
export function extractLinkHref(
html: string,