Fetches pages and images from other sites for link previews, the image proxy and the Bandcamp resolver, without letting a user-supplied URL reach the server's own network. Each hop must resolve to a public address, the whole exchange has a deadline, and pages are read up to a cap.
Fetches a page for its link preview, through fetchWithTimeout. A response that isn't HTML is cancelled unread.
Fetches a user-supplied URL, following up to five redirects, each of which
must resolve to a public address too. timeoutMs bounds each hop, body
included. The caller must read or cancel the body, or its connection stays
open.
Checks whether raw is an http(s) URL on a host that isn't private, or a
link to one of this site's own pages. Routes and services check a URL with it
before storing, previewing or proxying it; it does no DNS lookup, which
fetchWithTimeout does.
Reads the body of res up to maxBytes; the rest is cancelled, never
downloaded. The image proxy reads through it, and readTextCapped.
Reads the body of res as text, up to maxBytes, in the charset the page
declares; the rest is cancelled, never downloaded. fetchPage and the
Bandcamp and YouTube resolvers read pages through it.
Returned by fetchPage to the link preview providers.
-
html: string
The page's text, up to MAX_PAGE_BYTES; missing when it isn't HTML.
-
url: string
The URL the page was served from, after redirects: the base its relative links resolve against.
The default cap of readTextCapped. The metadata previews read is in
the <head>, and a page past this is a download.
Usage
import * as mod from "api/remote/fetch.ts";