Spots reposts and links to known dumps, by reducing a URL to a key shared by every link to the same page or video. dump-service stores it in dumps.url_canonical when a dump is posted or edited, GET /api/dumps?url= matches on it to warn the create form of a repost, and backlink-service compares links by it. The key is never fetched or shown, so it may be lossy: it forces https, drops www., tracking parameters and timestamps.

Migration 0013 backfilled existing rows with a frozen copy of these rules. Changing them needs a new backfill migration, or old rows keep old keys.

Functions

f
canonicalizeUrl(raw: string): string | null

Returns the canonical key of raw, or null when it isn't an http(s) URL. A YouTube video or playlist becomes one fixed URL whatever shape it came in.

f
youtubeHost(u: URL): string | null

Returns u's host in lower case and without "www.", when it is one of YouTube's (youtu.be aside), and null otherwise. The YouTube preview provider uses it too, so both agree on what a YouTube link is.

f
youtubeVideoId(
host: string,
pathname: string,
params: URLSearchParams
): string | null

Returns the id of the video a YouTube URL points at, in any of the shapes people paste (youtu.be/ID, /watch?v=ID, /embed/ID, /shorts/ID, /live/ID), and null for any other URL. host is lower case, without "www.". The canonical key drops the timestamp, since the same video linked at 2:30 is still a repost; the YouTube provider reads it separately.

Usage

import * as mod from "api/lib/canonical-url.ts";