vincent/gerbeur
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vincent:5f0cf86fe68550b5e63f5b94f87cbaefd38849bc
Branches Tags
vincent:main vincent:gh-pages
...
compare: vincent:a2ec163ee5673c9799b19b032d63e0d776161dff
Branches Tags
vincent:gh-pages vincent:main

59 Commits
5f0cf86fe6 ... a2ec163ee5

Author SHA1 Message Date
khannurien
a2ec163ee5 v4: ESLint and deno fmt pass on the whole tree again — html.ts's attribute pattern escaped a "/" its character class doesn't need escaped (no-useless-escape), and a line of journalLayout.test.ts ran past the width.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 57s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 25s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:43:28 +00:00
khannurien
7c00db2b84 v4: what comes into a live list goes where its date puts it — a dump edited into a category stayed out of the index's feed of it until a reload, and a dump or a playlist made public went on top of the others' lists as if just posted. The index's feed now takes a dump edited into its scope, and orders the dumps that arrive by date, as the server pages them; a created playlist goes by date in the profile's and /playlists' lists, a new one first as before. Both use withByDate (utils/feed/pagedList.ts), which leaves an item older than every one loaded to a later page. The search runs again after a reconnection, for every page of dumps loaded, with its users and playlists; it shares refetchPages with usePagedList's resync. injectDump's doc said a private dump is never broadcast, while its poster gets it. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:43:28 +00:00
khannurien
308a8299f3 v4: every live update reaches every page that shows what it changed — the client kept each kind of event in one "latest" slot, so two arriving before a render lost the first (two notifications of one dump going public showed one); they now go through a subscription, useWSEvent, and each reaches every subscriber. A playlist made private sent its new title and description to everyone, and kept them on a visitor's page; playlists now follow the dumps' rule, broadcastChange: others get playlist_deleted, and playlist_created when it goes public again. A dump deleted or turned private sends its playlists' dumps again, so their counts and pages follow. Comments on a private dump reach its poster's other tabs. Reading the notifications clears the bell in every tab (notifications_read), and a category change makes every client fetch the categories again (categories_changed). The dump page gives way to a notice when its dump is deleted or made private. A profile gains the user's new dumps, the followed feed those of the users followed and the playlists followed, the search results follow edits and deletions, the index's category feed drops a dump edited out of it, and a renamed user's name follows on cards and comments. After a reconnection, every live list fetches the pages it holds again (usePagedList's resync), and the playlist, the profile, the dump, the categories and the follows are fetched again. The position memory of usePositionAwareSync is now inside the list sync hooks, so a returning item is put back without depending on the order the hooks are called. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:34:49 +00:00
khannurien
1c5a449d84 v4: API.md says what the URLs are again — it lacked the /playlists pages and GET /api/playlists, a playlist's categories, the ?category= of a user's lists and of the search, and ?chat=<message id>, and its reserved words lacked playlists. It now also states the rules the pages kept without saying: a category is a segment where the page is the category's own and ?category= where it narrows a list on another page; switching tabs takes the place of the history entry, while picking a category or opening another page or an editor adds one; a signed-in page opened signed out comes back after /login. canonical-url.ts named GET /api/dumps/by-url, which is GET /api/dumps?url=. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:25:08 +00:00
khannurien
71df0268e3 v4: asking for a password reset link is a page, /reset-password — "Forgot password?" unfolded the request form on /login as a state of the page, which a reload or a link lost, while /reset-password, where the emailed link leads, answered without its token "Invalid link". The page now holds both steps: without a token it asks for the link, with one it sets the new password, and "Forgot password?" links to it. The login page's unfolding panel and its styles are gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:24:13 +00:00
khannurien
9a215b0cc5 v4: a playlist's editor has a URL, as a dump's has — it opened over the playlist's header as a state of the page, which a reload or a shared link lost, while a dump's editor is /@name/dumps/<slug>/edit. It is now /@name/playlists/<slug>/edit: Edit goes there, and Cancel or Save come back to the page in its place. Both paths render PlaylistDetail at the same place, so the playlist stays loaded, and a new slug keeps the view. Anyone who may not edit it gets the page, a guest the sign-in first. playlistPath takes the view, and parsePagePath reads it, so its link has no preview, as a dump editor's has none. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:22:24 +00:00
khannurien
21f4db268e v4: every request about a user, and the dump editor's path, come from one builder each — fourteen places wrote /api/users/<key>/… by hand, some encoding a handle (@alice%40music.example) and some not, a slug never; userApiUrl (src/utils/shared/urls.ts) now writes them all, by the user's id or by @ and the handle a page's path gives, each segment encoded. The editor's path was the dump's page's with /edit added in two places; dumpPath takes the view, as userPath and feedPath do. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:18:13 +00:00
khannurien
669e65a3a2 v4: a user from another instance can't be renamed from their profile's manage tab — it offered the name field to every user, and the API refuses a remote one's rename with a 400, since their home names them. The tab now only shows their name, as the settings do for one's own. 
The profile's and the settings' tests of a remote account's links to its home still expected the home's host itself, which 8bcc005 replaced with GET /api/users/:key/home; they now expect that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:15:54 +00:00
khannurien
f1bc23a65e v4: an instance can be named by the address of its pages — sign-in and the trust list took only an instance's issuer, its API's origin, so where the pages live elsewhere the address people see in their browser named no instance: in development, localhost:3000 answered its /.well-known with index.html, and the sign-in failed as "doesn't answer as a gerbeur". resolveIssuer (remote/gerbeur-instance.ts) now reads the metadata at the address given and, when it names another issuer, takes it only if that issuer's own metadata names those pages back as where its /authorize is, so an impostor can't claim one; sign-in, the trust list and the check against this instance go through it. The Vite dev server passes /.well-known on to its API, as whatever serves the pages must in production (README). A handle still carries the API's host. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:14:21 +00:00
khannurien
9608520675 v4: a handle with this instance's own host names the local user — another instance writes bob from here as bob@gerbeur.example, so a mention or a link copied from there named no one here: /@bob@gerbeur.example was a 404, and the mention notified nobody. localHandle (shared/validation.ts) drops this instance's host, its API's or its pages', from a handle; resolveUsername reads names by it, so the page, the API and the Open Graph 301 reach bob at /@bob, and so do the mention notifications and the chat's highlight and chime. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:11:39 +00:00
khannurien
6dfba07341 v4: the followed feed's two views are pages — "From people" and "From playlists" on a feed's "follow" tab were a state of the page, which a reload, Back or a shared link lost, while a profile's same split is /@name/follow/playlists. They are now /follow and /follow/playlists, and /music/follow/playlists in a category, feedPath taking the view as userPath does (FOLLOW_VIEWS); picking a category keeps it. Their dumps come from GET /api/dumps?follow=users|playlists, signed in, which the feed's word for it names, instead of GET /api/me/feed?from=; an unknown category is a 404 there, as on the rest of the feed, rather than an empty list. 
A feed's tab, as a profile's, a search's or the community page's, now takes the place of the current history entry, since it is a view of the same page; picking a category, another page, still adds one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:09:12 +00:00
khannurien
53ceb6bcbc v4: signing in comes back to the page that asked for it — a guest on a signed-in page, the dump editor, /notifications, /community, /settings or /manage, went to /login and, once signed in, to /, so a digest's link opened signed out lost its way; and since the redirect pushed, Back led to it again. RestrictedLoggedIn now replaces the page with /login?next=<the page>, as /authorize already did. A guest on a feed's "follow" tab, which only a link leads to, is sent there the same way, rather than shown the default tab. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:05:39 +00:00
khannurien
683944ebe2 v4: a remote user's pages get their link previews and redirects — the Open Graph middleware passed on every path with a dot in it, taking it for a file's, and the handle of a user from a real host has dots (/@bob@music.example), so their profile, dumps and playlists had no tags, and a former name or another case of theirs no 301. Only the dev instances' localhost:8000 escaped it. parsePagePath already passes on any path that doesn't start with @, which no file's does, so the dot check is gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:04:35 +00:00
khannurien
53515ce46f v4: playlists have categories of their own — a set apart from the dumps', which admins manage on /manage under "Playlist categories", next to "Dump categories". Each category now has a kind, dump or playlist, its slug and name unique within its kind only; migration 0025 rebuilds categories with the column, every existing one becoming a dump category with its dumps, and adds playlist_categories. A playlist's owner files it under any of them in the create and edit forms, and it shows them as chips on its card and its page, each leading to /playlists/<slug>. 
/playlists is the new page of every public playlist, newest first (GET /api/playlists), and /playlists/<slug> a playlist category's, with a menu of the categories, linked from the user menu as "All playlists"; `playlists` is reserved as a dump category's slug, and one that had it is renamed. A profile's Playlists tab narrows to a playlist category with the same menu as its Dumps tab (?category=), and so do the Dumps and Playlists tabs of /search, each with its own kind (category and playlistCategory for GET /api/search), another tab dropping it.

The category service takes the kind throughout, so a dump can't be filed under a playlist category nor the other way round, and categoryJoin is the one filter join the dump, follow and playlist services share. useCategories(kind) gives one kind's set from the one fetch, CategoryChips replaces the chips the dump card and page each drew, useCategoryFilter keeps a list to its category as the socket brings more, and useCategorySelection holds a form's picked categories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:53:16 +00:00
khannurien
681170a517 v4: in French, an upvote is a "validation" of the reco rather than a vote for it — the button reads "Valider", then "Validé", the notification says "<who> a validé <reco>", and the digests' top recos are "les plus validées". The noun stays "vote": the counts, "Retirer le vote" and the live updates warning. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:10:22 +00:00
khannurien
93565cff0c v4: deno lint passes again — a Markdown test wrapped its child text in curly braces, which jsx-curly-braces refuses; the text is now written as is. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:06:14 +00:00
khannurien
8bcc005140 v4: a remote account's links to its home lead to the home's pages — the settings' "Change it on <host>" button and the profile's "Account on <host>" link went to the host in the handle, which is the home's issuer, its API, not necessarily where its pages are: in development musique's button led to localhost:8000/settings, gerbeur's API, which served a stale build, instead of localhost:3000/settings. GET /api/users/:key/home?path= now redirects to a page of a remote user's home, their profile there by default, at the origin of the home's authorization_endpoint, which its metadata gives (homePageUrl in federation-service.ts), the issuer standing in while the home doesn't answer; a local user 404s, and a path that isn't one 400s. homeUrl links to it. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:06:14 +00:00
khannurien
5f2fb4479a v4: the settings' controls line up down the page — each card of /settings was a grid of its own, its label column as wide as its longest label, so the controls began at a different place in every card, and the other instances, a grid inside their card, at yet another. Every card of settings now has a label column of one width, 10rem, a longer label wrapping, and a list such as GrantList, the card's one child, takes the card's rows: their padding, their rules, their columns. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 20:53:29 +00:00
khannurien
2c3ba6ad11 v4: link previews are read once, in one place, and no longer take an error page, a relative image or a mask icon for the page's — each provider fetched its page and scanned it with regular expressions of its own, the generic one some fifteen times, and Bandcamp and SoundCloud read only OG tags. fetchPage (remote/fetch.ts) now fetches every page, follows redirects to the address it was served from, and refuses an error status, so a 404 or a bot challenge is unreachable rather than a preview titled "Just a moment...", and a refresh no longer overwrites good metadata with it. parsePage (remote/html.ts) reads it in one linear pass with a real attribute reader: unquoted values, a > inside quotes, og: tags under name as well as property, and nothing from comments or scripts. Every URL a page names is resolved against where it was served from, so a relative og:image works; entities are decoded in one pass, &eacute; and &rsquo; included, and one that names no character no longer fails the preview (bandcamp-tralbum.ts shares the decoder). The mask icon is no longer the favicon, JSON-LD prefers the page's node over the site's, tracking pixels and lazy placeholders aren't thumbnails, an og:url with a slug keeps the OG tags, and a <title> loses its site suffix. genericPreview is the base of the Bandcamp, SoundCloud and YouTube channel previews; SoundCloud takes www., m. and on.soundcloud.com short links, embedding the page they lead to, and Bandcamp keeps only its own player. The service tidies every preview: text on one line, titles and descriptions capped, only http(s) images and players. RichContent.type is now RichContentType, and the unused videoId is gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 20:46:32 +00:00
khannurien
b4936cccf7 devcontainer: update lock file 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:53 +00:00
khannurien
141e526de7 v4: the instance fields ask for a URL — the login page's field read "Its address, like music.example" and /manage's "An instance's address, like music.example", an example that named no instance of the reader's. They now read "Your gerbeur's URL" and "An instance's URL". 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:53 +00:00
khannurien
63803c552f v4: a profile's lists of dumps can be narrowed to a category — /@name/dumps and /@name/upvoted showed every dump, with no way to see only those filed under one category, as the index can. The title row of each list now has the index's category menu, and the category picked goes in the URL as ?category=<slug>, which the other list keeps and every other tab drops (a path segment would collide with /@name/dumps/<slug>, a dump's own page). GET /api/users/:key/dumps and /upvoted take ?category= as the followed feed does, an unknown slug answering an empty list; resolveCategoryFilter moves to category-service.ts for both. useUserDumpFeed passes everything the socket changes through the category, so a dump edited out of it leaves and an upvote outside it doesn't come in. CategorySwitcher now reports the category picked to its owner (onSelect) instead of navigating to the index's path itself, and the divider after it in the index's tab row is drawn by CSS, so the profile gets none. useTabParam's setter takes the query string to go to, the current one by default. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:03 +00:00
khannurien
52a23db504 v4: an instance that takes no users of others still blocks instances its own users may not sign in to — with GERBEUR_FEDERATION_INCOMING=0, /manage hid its list of instances, though a blocked one still kept the instance's users from signing in there, and no admin could block another. The list now stays while federated sign-in goes either way: without incoming sign-ins it shows and adds only blocked instances, the trusted ones kept for when they're back, and says what blocking does there. Where both ways are on, its note now says that users here can't sign in to a blocked instance either. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:33:08 +00:00
khannurien
83dc44050e v4: a deployment chooses which ways federated sign-in goes — every instance took users of the others and let its own sign in to them, with no way to turn either off. GERBEUR_FEDERATION_INCOMING=0 keeps users of other instances out: the login and register pages offer no other instance, /manage no list of instances, their sign-ins are refused, and those already in are signed out until it is on again, their homes' events still applied meanwhile. GERBEUR_FEDERATION_OUTGOING=0 keeps the instance's users from signing in elsewhere: its /authorize page lets no instance sign them in, its metadata says so (gerbeur_outgoing), which other instances check before sending anyone there, the settings page lists no instances, and at startup every grant is withdrawn, which signs them out there. Both are on by default and reach the client through index.html. The dev instance gerbeur (:3000) takes no users of the others, while its own still sign in to them. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:18:21 +00:00
khannurien
66e402bd91 v4: federated sign-in closes the holes a review found — a callback link made from someone else's sign-in signed the visitor in to that account, a password reset left the user's sessions on other instances open, and events could be lost, reordered or delayed. FederatedSignIn keeps the sign-in's state in sessionStorage (utils/auth/federation.ts) and FederationCallback finishes only a sign-in this tab started. A password reset tells the instances the user signed in to, as a change does. The outbox delivers each instance's events in the order they were made, the first that fails holding back the rest, so a late older update no longer undoes a newer one, and delivers to every instance at once, so a dead one delays none of the others; an event queued during a run is sent by it. 
Receiving, a home's keys that can't be fetched answer 502, which the sender retries, instead of 401, which it dropped; a bad update answers 400 rather than 502; a blocked instance's events answer 403; events are limited to a burst of 120 per instance, and a token naming an unknown key refetches the keys at most once a minute. An avatar must be on its issuer's origin, and its URL is versioned by its file's write time instead of updatedAt, so a profile edit no longer makes clients download it again. An instance that doesn't answer can be blocked, keeping its name, though not trusted.

303 backend tests and 703 frontend tests pass, the federation tests 8 times in a row, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:05:11 +00:00
Vincent Lannurien
c77e173767 v4: the devcontainer builds and starts again — the docker-outside-of-docker feature failed to install, since the base image is now Debian trixie, where its default moby option isn't supported; it now installs Docker's own packages ("moby": false). Every service of the compose project was also built under one image name, gerbeur-dev, the workspace with the features and the others without, so whichever finished last took the name and the workspace could lose its docker CLI; each service now gets its own image, built from the same cache. 
Checked with the devcontainer CLI's up: the image builds, install exits 0, gerbeur and musique answer on :3000, :3001, :8000 and :8001 (openid-configuration included), and docker ps works as node in the workspace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:39:58 +02:00
khannurien
a25500d833 v4: the devcontainer starts several gerbeur instances side by side, so that signing in from one to another can be tried on one machine — the dev environment ran a single instance. dev/instances.ts lists them (gerbeur on :3000 and :8000, musique on :3001 and :8001), and dev/start.ts starts an API server, restarted on change, and a Vite server for each instance it's given, by one rule: instance <name> keeps api/sql/<name>.db and api/uploads/<name>/, reads .env, and takes its ports and what sets it apart from its entry, with federation allowed over http on localhost. It prefixes each line of output with the instance's name, and stops every server when one stops, such as a Vite server whose port is taken. The devcontainer is now a compose project (.devcontainer/compose.yaml): the workspace, a one-shot install that runs deno install and draws the icons once, and a service per instance running dev/start.ts for it, all on the host's network as before and sharing Deno's cache through a volume. ffmpeg moves from a feature into the Dockerfile, since the instances' services get no features, and docker-outside-of-docker gives the workspace's terminal their logs and restarts. Outside the devcontainer, deno task dev starts them all, or those named. Each Vite server keeps its own dependency cache. The existing dev uploads move to api/uploads/gerbeur/. 
dev/start.ts was checked by hand: musique starting alone on a fresh database, an unknown name refused, a taken port stopping both servers, and Ctrl+C or SIGTERM leaving no server behind. The compose project couldn't be run here (no Docker in the workspace): its YAML parses and resolves as intended, but it is untested; tsc, eslint, deno check, deno fmt, deno lint and doc:lint, which now covers dev/, are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 14:31:25 +00:00
khannurien
c0d03e9aaa v4: users sign in to one gerbeur instance with their account on another — topic instances, each with its own admins, domain and data, had no way to share accounts. Sign-in is federated over a subset of OpenID Connect, the authorization code flow with PKCE: each instance is an identity provider for its own users and a client for the others', and signs what it tells them with an Ed25519 key kept in federation_keys and published at /.well-known/jwks.json, beside its metadata at /.well-known/openid-configuration. Only sign-in is federated: an account from another instance is a local users row linked to (issuer, subject) in user_identities, with no password (password_hash ''), so sessions, roles, permissions, the socket and every foreign key work as before. 
A remote user is name@host here, in users.username, in paths (/@alice@music.example) and in mentions, through one handle rule in shared/validation.ts that paths, MENTION_RE and Markdown all build on; Markdown undoes GFM's email autolink of a handle. Their name, email and avatar come from their home at each sign-in and by its signed events (federation_outbox, delivered by the federation job with backoff): a rename at home leaves the old handle an alias here, and a password change or a withdrawn grant ends their sessions here. Their password, email, avatar and name can't change here (assertLocalUser), nor can they sign in elsewhere from here.

An admin keeps the trusted and blocked instances on /manage (federation:manage): a trusted instance's users sign in from a button on the login page, a blocked one's can't and are signed out at once, and any other's join with an invite from the register page. The /authorize page asks the user once per instance, and /settings lists the instances their account signs in to, each disconnectable; /login?next= brings a signed-out visitor back to /authorize. Emails stay unique among local accounts only, and migration 0024 adds the tables, makes that index partial and reserves authorize and federation as category slugs. remote/gerbeur-instance.ts is the only way to other instances: https and public addresses only, no redirects, no certificate fallback, unless GERBEUR_FEDERATION_ALLOW_PRIVATE lets two instances on one machine talk.

298 backend tests and 702 frontend tests pass, and deno check, deno fmt, deno lint, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked end to end with two built instances on one machine: sign-in through the other's login and consent pages, consent skipped once granted and asked again after disconnecting, a rename and a password change at home reaching the other, and blocking; screenshots of the login, consent, profile, settings and manage pages at 1280 and 390px.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 14:23:00 +00:00
khannurien
2d26d0aada v4: the late-digest test no longer races the clock — it posted its only dump at the current time and sent the digests right after, and a digest's window ends before the run's now: when both fell in the same millisecond, which createUser's cached password hash made common, the dump was outside the week, the digest empty, and the test failed about one run in fifteen. Reproduced with a dump stamped at now (no digest) and one a millisecond earlier (one); the dump is now dated a day back, as "recent" meant. 60 runs of the digest tests, 6 of the backend suite (277 tests) and 4 of the frontend suite (684 tests) pass. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:47:56 +00:00
khannurien
fac0e2db28 v4: an audio file's waveform is made once by the server, and the player seeks on one accessible scrubber — every player downloaded the whole file a second time and decoded all of it in the browser to draw 150 bars, so a long FLAC cost tens of megabytes of memory per page, and the waveform could only be clicked: no drag, no keyboard, no slider for a screen reader. GET /api/dumps/:id/waveform now answers an audio dump's peaks, which lib/ffmpeg.ts decodes with ffmpeg on first request, mono at 8 kHz and streamed, keeps beside the thumbnails as the response itself, and serves with an ETag (a ten-minute MP3 takes about a second, then a few hundred bytes). The peaks are taken by shared/waveform.ts's peaksOf, scaled to the loudest so a quiet recording fills the track, and the dump form's preview of a file not yet uploaded decodes it with the same function, at 8 kHz in an OfflineAudioContext. Scrubber replaces Waveform and SeekBar: a role="slider" that seeks where a pointer lets go, follows a drag (the clock shows where it is held), takes arrows, Home and End, and draws the bars as one path with the played part clipped by a nested svg, so it moves smoothly and a timeupdate redraws two attributes instead of 150 rects; audio without a waveform, and video, get a plain bar. MediaPlayer reads whether it plays from the element's events alone, so a click no longer reports the change twice, its handles for PlayerProvider are stable callbacks instead of refs to refs, the drag no longer reattaches every listener, and the <source> children that src made dead are gone, with the mime prop. The file, thumbnail and waveform routes resolve their dump, token and caching through lib/dump-media.ts, and what is generated from an upload is named and removed in lib/ffmpeg.ts, the custom thumbnail by dump-service's customThumbnailPath, instead of paths spelled out in each. The test setup's play() and pause() fire their events as a browser does. 277 backend tests and 684 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in the app with a real FLAC on its dump page and in the global player at 1280 and 390px, seeking by keyboard. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:34:22 +00:00
khannurien
c709b56911 v4: every message named by an id carries its English in the code — the emails and the four pages that share their words called Lingui with a bare id (i18n._("digest.community"), <Trans id="digest.community" />), so their English lived only in a hand-kept en.po, extract couldn't run with --overwrite, and a page reading the code couldn't tell what it said. The emails now pass descriptors, i18n._({ id, message, values }), the form Lingui reads without a macro, as the server needs. The fifteen messages the digest shares with the community page, Settings, the unsubscribe page and the avatar menu are declared once, as /*i18n*/ descriptors in shared/digest-messages.ts, which those pages translate with useLingui()'s i18n instead of the global one. lingui.config.ts scans shared/ too, and i18n:extract runs with --overwrite, so en.po is written from the code. Every other message already used the macros. The compiled catalogs are unchanged, key for key, in both languages. 272 backend tests and 677 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:19:01 +00:00
khannurien
c3594e4c27 v4: what the server and the client both state is declared once — several rules were written out in two or more files, some already drifting apart. A link's site name (its host without www.) was computed in five places, now shared/url.ts's siteHost; dumpFileVersion, which the frontend and the server had to keep equal by hand, is one function in shared/dump.ts; the Markdown flattening of the chat's reply previews and the digest's quotes is shared/markdown.ts's markdownToPlainText, so the digest also keeps an image's alt text; an attachment's URL and how to find one in a text are shared/attachment.ts, which the editor, Markdown and attachment-service read, its id pattern the new UUID_RE that the file and thumbnail routes check ids with. On the server, the Bandcamp, YouTube playlist and link preview caches use one TtlCache (api/lib/ttl-cache.ts) instead of three copies of the same expiry and eviction; the link previews' theme-color and the emails' theme tokens read colours with one parser (api/lib/css-color.ts), and a transparent theme-color now gives no tint; the WebSocket route sends through ws-service's send, the welcome included, so it is type-checked; fetch.ts strips an IPv6 literal's brackets with public-address's bareHostname; and a category's slug is checked by the same validateSlug as a dump's, with its own length. The migrations keep their frozen copies, as they document. 272 backend tests and 677 frontend tests pass, and deno check, tsc, eslint and doc:lint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:16:52 +00:00
khannurien
134d1a0b97 v4: a moderator's or an admin's @everyone in the chat notifies every user — there was no way to reach everyone at once. The new chat:everyone permission, held by moderators and admins, lets a chat message saying @everyone notify every user but its author and those reading the chat, through notifyEveryone, after notifyMentions, so a user also named is told once, by name; the notification is a user_mentioned with everyone set, which the notifications page and the digest word as "mentioned everyone in the chat". chat_messages.mentions_everyone, sent as ChatMessage.mentionsEveryone, is set from the author's role each time the body is written, so a user's @everyone reaches no one, a moderator's edit of a user's message can't make it reach everyone, and a demoted author's next edit loses it. Only an edit that adds @everyone notifies, so fixing an announcement's typo reaches no one new. The client highlights and chimes by the flag, never by the text; Markdown shows @everyone as a mention only in a flagged message, and never links it to a profile. The chat composer offers @everyone first to those who may say it. No account may be named everyone, in any case: migration 0023 adds the column, flags the moderators' and admins' past @everyone messages, renames a user named everyone to everyone_N and drops an alias everyone. 265 backend tests and 681 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in screenshots of the chat at 1280 and 390px, with a moderator's and a user's @everyone and the suggestions open. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 12:54:09 +00:00
khannurien
5f239a144d v4: the community page and the digests' section are titled Communauté in French — "La communauté" read awkwardly as a title. i18n:compile --strict is clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 11:52:54 +00:00
khannurien
708d113013 v4: a /community page shows the site's week and month, from the same figures as the digests, and every recurring server task is a job of one scheduler — the digests' top dumps and community section (who joined, the new public playlists, the chat and the totals) existed only in emails. community-service.ts now gathers them for a period from listTopDumps, listNewUsers and listNewPublicPlaylists, each in its own service and returning the API's Dump, PublicUser and Playlist; the digest builds its sections from the same call, its email-only DumpRow and DUMP_COLS replaced by one Dump mapper that networkOf uses too, and links its community section to the page. The page, /community for the last week and /community/monthly for the month, signed in only and linked from the avatar menu, reads GET /api/community?period=, which serves figures kept in memory and recomputed every GERBEUR_COMMUNITY_REFRESH_MINUTES (15 by default). It shows the four figures, the top dumps and the new playlists as live cards, and the new members as UserCards, the follow tab's user row made shared. Its words are the emails' own: the messages both use are renamed from email.digest.* to digest.*, as digest.weekly.off was. api/lib/scheduler.ts's startJobs runs the digests, the community refresh, the WebSocket ping, the typing expiry and the rate-limit sweep, which main.ts starts, so none of them runs on import any more. PERIOD_MS moves to shared/settings.ts. community is reserved as a category slug, and migration 0022 renames a category that had it, with renameReservedCategories moved out of 0021 into reserved-slugs.ts. A group's dump or playlist cards now fill its column. 262 backend tests and 678 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth light at 1280 and 390px, Smooth dark with mocked playlists, Brutalist dark at 390px and Kanagawa dark at 1280px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 11:42:01 +00:00
khannurien
7713c5a1b5 v4: a dump or playlist going public notifies its followers — a private dump notified no one when posted, and nothing told its followers when it was made public: the poster's followers never learnt of it, nor did the followers of the public playlists holding it. Likewise, the dumps added to a playlist while it was private were never notified to its followers. updateDump and replaceFileDump now call notifyIfWentPublic, which notifies the poster's followers (user_dump_posted) and, through the new notifyPlaylistFollowersNowPublic, the followers of each public playlist holding the dump (playlist_dump_added). updatePlaylist calls the same function when a playlist goes public, for each public dump in it, from the bottom up. A follower hears of a dump once both it and the playlist are public; the source keys stop a dump made private and public again from notifying twice. 256 backend tests pass, and deno check, deno fmt and doc:lint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 10:23:08 +00:00
khannurien
256857f72e v4: a chat mention notification opens the chat on the mentioned message, so the newest one clears the chat badge — it linked to ?chat=open, and the chat opened on the "new" divider whenever the unread messages didn't fit. That left the list above the bottom, so the read marker never moved and the badge stayed, while the mention itself was out of sight. It now links to ?chat=<message id>: ChatProvider opens on any ?chat= (open, or a message id passed to ChatModal as focusId), and ChatModal opens centred on that message and flashes it. When the message is within a screen of the bottom, the list stays at the bottom and reads everything, as it always has. An older mention still leaves the messages after it unread. A message outside the first page of history falls back to the divider. ChatModal's flash is now its own callback, shared with jumping to a message. 675 frontend tests pass, and tsc, eslint, deno fmt and doc:lint are clean; reproduced and checked in the app with 36 unread mentions from a second account: the newest clears the badge, an older one opens on it with the badge kept. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 08:56:59 +00:00
khannurien
4ea192e0b1 v4: an uploaded audio file plays from its cover everywhere, and a card's play badge sits in the middle of its thumbnail — a link played from its thumbnail in every view, but an uploaded file only did on a list card: a journal tile opened the dump, an audio file's page showed a round button by its waveform and never its uploaded cover, and an uploaded video's page was a black box until played. The list card's badge sat in the thumbnail's bottom-left corner, where it looked misplaced, and clicking a playing file restarted it. The new usePlayFile hook builds the player's item and the file's artwork (fileArtworkUrl) for the card, the journal tile and the dump's page, and plays, or pauses once the file is the current one. A card's badge, ▶ or ⏸, is now centred, as is the ↗ on a link out; over a placeholder it takes the place of the glyph. An audio file without a cover shows a placeholder tinted from its file name, not a bare 🎵. In the journal, audio counts as having art (hasThumbnail), so it is an image tile, its glyph raised above the title, and a file tile plays and pauses the file. On the dump's page, the cover (96px, the card's badge on it) plays the file beside the shared Waveform, which replaces FilePreview's copy of it; a video takes the server's still as its poster. 674 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth light at 1280 and 390px, Brutalist dark on the journal at 390px and Kanagawa dark at 390px, playing. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:33:33 +00:00
khannurien
2c031b0102 v4: the header drops the username link — on wide screens it stood beside the avatar, which leads to the same profile, so only Playlists stays there. The avatar's menu still opens with @username first. 670 frontend tests pass, and tsc, eslint, deno fmt and i18n:compile --strict are clean; checked in a screenshot in Smooth dark at 1400px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:23:46 +00:00
khannurien
886216749e v4: the avatar menu shows at every width and holds Log out, so a profile keeps only Invite; the profile's dump cards lay out again — your own profile had four buttons in a row, Invite someone, Settings, Manage and Log out, Invite taller than the rest, and on wide screens it was the only way to reach /settings or /manage. The avatar menu now leads the header's icon row at every width, after the name and Playlists text links on wide screens, and ends with Log out under a separator, muted until hovered. The profile keeps Invite someone alone. The menu's button outweighs .app-header-nav button and the themes' button rules (Brutalist and GeoCities list it among their flat buttons), so it reads as one of the menu's links. Since b9ff9c4, a card under 472px is laid out by the narrow-card container query, but the profile columns' dense-card rules still placed the thumb, body and vote themselves, more specifically: the thumb lost its row span, so the title fell under it and every card was a tall, mostly empty box. Those rules now only make the card denser (a smaller thumb, a 2-line title, smaller type) and leave placement to the container query. 670 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth dark at 1400 and 390px, Brutalist light at 390px, GeoCities light and Kanagawa dark at 1400px, in French. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:22:15 +00:00
khannurien
10c9cf4269 v4: a playlist's cover is a banner over its header, and the edit form picks it in the same shape — the cover was a 72px square beside the title, which left the title and description a narrow column under a mostly empty band: on a phone, in Kanagawa's wide mono, a long title broke into six lines. The cover now spans the header's width above the title, cropped to 3:1 and no taller than 14rem, inside the header's padding so outlined themes keep their ring around it; the title, buttons, description and meta take the whole width under it. The edit form's Cover picker takes the same shape (.playlist-cover-banner, shared with .playlist-detail-img), and with no cover is a dashed 3:1 placeholder. ImagePicker's size is now optional, so a class can size it, and its radius takes any CSS length, here --radius-lg; a picker sized by aspect-ratio sets min-height: 0, or its image would hold it at its own height. 669 frontend tests pass, and tsc, eslint, deno fmt and doc:lint are clean; checked in screenshots in Kanagawa dark (French, 385px), Brutalist at 390px with and without a cover, and Smooth at 1100px, the page and the form. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:10:06 +00:00
khannurien
5acdf79c8d v4: the playlist and dump edit forms end in one bar of actions, a panel of its own — the playlist editor drew rules between its fields and its actions, and on a phone its Delete, Cancel and Save wrapped into a staircase, then stacked (a0c8b1d), neither of which held together. Both forms are now built as DumpEdit was: surface panels 3px apart (.form-panel, from .dump-edit-panel), the last a bar (.form-panel--bar, from .dump-edit-footer) holding Delete on the left and Cancel and Save on the right, compact (btn-sm), on one line. The new DeleteButton says "Delete" and names what goes to assistive tech ("Delete playlist", "Delete dump"); in a bar too narrow for the three labels in every face and language (a container query, under 18rem), it shows 🗑 instead. While edited, the playlist's header drops its own card for the form's panels. The phone stacking rule is gone. An auto-sized editor also re-fits when its width changes with its text unchanged: opening the form makes the page tall enough for a scrollbar, which narrowed the description after it was measured, so it overflowed; and it hides its overflow below its max-height, where a scrollbar could only come of rounding, and drops the resize grip, which the next keystroke would undo. 669 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Kanagawa dark (French, 385 and 360px), Smooth light at 1000px and DumpEdit in Smooth at 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:05:43 +00:00
khannurien
b9281da9c9 v4: the avatar menu opens from the avatar's left edge — since 7531be3 the avatar leads the header's icon row, but its menu still hung from its right edge, so on a phone it ran off the left of the screen. It now hangs from the left edge and opens over the icons. Checked in screenshots in Smooth at 390px and Brutalist at 320px, in French, and Kanagawa at 1100px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:58:15 +00:00
khannurien
7531be300e v4: the header's controls are line icons, and the avatar leads them — 🔍, ✕, 💬 and 🔔 were four emoji in four palettes beside the avatar and the "+", none of them the theme's. They are now stroked icons (NavIcons.tsx, which the search bar's submit and ChatFab share) in the text colour, their bodies washed with the accent; the wash deepens on hover, and fills while something is unread, with a small dot in place of the red count (the count stays in the button's label and the tab's title, and ChatFab keeps its own). Themes set the line through --nav-icon-stroke, -linecap, -linejoin and -tint-strength: Brutalist heavy and square-cut, Times hairline in ink (its grayscale filter no longer needed on them), GeoCities fat and soaked. The search toggle was a ghost button with its own hover; it now shares the chat button's and the bell's rule, background and borders in every theme. The avatar sat among the icons, its hover box larger than theirs and 1.5px higher (a block .user-menu left a line-box gap under the inline-flex trigger): it now leads the row, where the name and Playlists links stand on wide screens, so the icons run unbroken to the "+"; it is 32px, and hover rings it with the accent instead of a box. 669 frontend tests pass, and tsc, eslint and deno fmt are clean; checked in screenshots in all six styles, light and dark, at 390px, at rest and unread, and Smooth and Brutalist at 1280px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:56:19 +00:00
khannurien
a0c8b1d509 v4: auto-sized text editors no longer scroll, and on a phone an action row with a destructive button stacks — an autoResize editor set its height to scrollHeight, which leaves out the borders that a border-box height counts, so the text always overflowed by a couple of pixels: a playlist's description, a .form textarea since cf89c18, showed a scrollbar that ran under the emoji button. The height now adds the borders. On a phone, Delete, Cancel and Save didn't fit one line in French or in a wide face, and wrapped into a staircase: Delete alone on the left, Cancel and Save under it on the right. A .form-actions row with a .btn-danger now stacks under 512px, with Cancel and Save across the top (Save taking the room left) and the destructive button full width under them; the playlist editor and DumpEdit's footer both take it. 669 frontend tests pass, and tsc and deno fmt are clean; checked in screenshots in Kanagawa (French, 385px, with a long description) and DumpEdit's footer in Smooth at 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:51:38 +00:00
khannurien
cf89c18eb9 v4: a playlist's edit form is a .form like the dump's, and its dumps keep the header's edges — the editor laid the cover beside its fields with display: contents, which on a phone left them a squeezed column, and drew its own inputs: the title and the slug both in bold large type, the description in another size, only the slug labelled, and a segmented control styled apart. It now stacks the shared fields, each under its label (Cover, Title with its counter, Link, Description, the visibility toggle), over the action row, as DumpEdit does, and takes the header's whole card. The dumps' drag handle and remove button took width from the cards, which sat inset from the header on both sides; they now sit in the page's side padding (--page-gutter, which .page-content now names), so the cards line up with the header at any width. Undo shows as a small btn-primary over the card it would bring back, instead of a third control beside it. CountedInput, VisibilitySegmented and the .playlist-edit-* rules had no other users and are gone. 669 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in all six styles at 390px, Kanagawa and Smooth at desktop widths too, Undo in Smooth and GeoCities. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:48:12 +00:00
khannurien
45f39a959b v4: /settings and /manage take after /notifications — the three pages the user menu and the bell lead to share one frame: a 680px column (PageShell's new narrow), a title with its icon (PageHeader: ⚙️, 🛠️, 🔔), then groups under a small capitalized label ruled to the column's end (PageGroup, the notifications' day label), their content on surface cards (.page-card, the notification items' surface). Settings drew big section headings over controls laid on the bare page, the full width wide, and Manage the same around the category manager; each section is now a card whose setting rows are ruled apart and keep their labels lined up (a subgrid of the card's columns). The notifications' column, centered by auto margins in a flex column, shrank to its content (380px with short ones); it fills its width now. Its own header, group and bell rules gave way to the shared ones, and Times, Brutalist and GeoCities dress .page-card as they dress the notification items. In the narrower column, Kanagawa's monospace no longer fit the six styles on one row, and French labels leave even less room: a segmented control of more than four options now makes two even rows at any width, as it did on phones. 670 frontend tests pass, and tsc, eslint, deno lint, deno fmt and doc:lint are clean; checked in screenshots in all six styles at 1280 and 390px, Smooth in light too, with the account's style and the notifications mocked so the shared test account was left as it was. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:40:10 +00:00
khannurien
1fda26a399 v4: /settings, /manage and /notifications title themselves alike, with a bare heading above their sections — Settings and Manage borrowed the edit form's .form-page-header, a panel rounded only on top to cap the panels under it, so on a page without them it ended in a square edge, and its title was the size of the section headings below. Notifications had a heading of its own. All three now take .page-header and .page-title: a bare h1 a size above the sections' h2, with what goes with it (the unread count) at the row's end; the edit forms keep their panel header. Times and Melange set their display type on .page-title, and the .profile-subpage-* rules, unused since the profile's pages became its tabs, are gone. (The CSS went in with d683ac0, which a concurrent session committed from the shared tree.) 21 Settings, Manage and Notifications tests pass, and deno fmt is clean; checked in screenshots in Smooth at 1280 and 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:28:28 +00:00
khannurien
45e004b465 v4: the slug and rename fields lose their hints about old links — "Links to the old address keep working." under the dump and playlist link field, and "Their old name keeps leading to them…" under the moderator's rename field, described a server guarantee nobody filling the form needs to read. Only the fields' errors show under them now; UsernameField loses its hint prop. 670 frontend tests pass, tsc and i18n:compile --strict are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:28:01 +00:00
khannurien
437eea0df3 v4: every inline editor ends in a btn-primary Save and a btn-secondary Cancel — comments drew their own .comment-submit-btn and .comment-cancel-btn, the profile's email its own .profile-email-btn pair, the bio a plain-text form-cancel, and the chat's edit box shrank the shared buttons with a rule of its own; they were four sizes, two orders of chrome, and the comment button's white label sat on Smooth's yellow accent. They all take the shared classes now, and the new .btn-sm is the one compact size, for the comment forms, the chat edit box and the email editor; the bio and the dump title stay full size. .btn-secondary gets a disabled state, as btn-primary has. The bespoke rules are gone, with GeoCities' bevel for the comment buttons, which btn-primary and btn-secondary already have. 670 frontend tests pass, and tsc, eslint and deno fmt are clean; the comment, reply, chat edit, email and bio editors were checked in screenshots in Smooth, Kanagawa, Brutalist and GeoCities at 1100 and 360px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:27:26 +00:00
khannurien
d683ac0548 v4: the dump title editor's Cancel is a button, beside Save — it was the grey text link of the full forms' action rows, next to a filled Save, where every other inline editor pairs Save with a bordered Cancel; it now takes btn-secondary, the same height as btn-primary. Save and Cancel sit in .dump-title-actions and wrap together, so on a phone they go under the input as a pair instead of Cancel dropping to a line of its own. 30 Dump page tests pass, and tsc, eslint and deno fmt are clean; checked in screenshots in Smooth, Kanagawa, Brutalist and GeoCities at 1200 and 360px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:23:28 +00:00
khannurien
b9ff9c417e v4: phone layouts hold down to 300px in every style, Kanagawa's wide monospace included, and the lint is clean — the floating chat and "+" buttons hid the end of every page (Save on the dump edit page, the last setting), since the page kept room for one button and they stack two; it keeps room for both now, read from one --fab-size. A 1fr page grid grew to its widest unbreakable content, and the edit forms' action bars didn't wrap, so at 320px the dump edit page scrolled sideways and the playlist edit's Save stuck out of its card; the grids take minmax(0, 1fr) and .form-actions wraps. The profile header's info column was a hidden horizontal scroller under a fade, which cut the username, the inviter, the email and the Log out and Invite buttons; it wraps instead (long names and emails break anywhere), and the avatar shrinks to 70% on phones. TabBar pins its leading content (the feed's category switcher) at the row's start while the tabs scroll under it, where scrolling to the active tab used to push it out of view; its edge fades show only while tabs are hidden past that edge, so the last tab is no longer dimmed once scrolled to, and a left fade marks a scrolled row. Tabs are narrower on phones, which lets Kanagawa's search tabs fit at 375px. The global player puts its track controls on a row of their own on phones, where they left the title three letters, and an audio bar narrower than 420px drops its volume slider for the waveform. The profile bio lost the margins it stacked on the page's gap. A dump card is now a container and lays out by its own width rather than the viewport's, since a playlist's is narrowed by its drag handle and remove button and titles broke every few letters: under 472px it is a grid where the vote ends the date line, as on the journal's cards, instead of hanging under the thumb, with a spacer row keeping it off the title; under 300px the thumb takes the top row and the text the full width. The lint errors that predated this are fixed: a useless fragment and two async callbacks without await in tests (they now await advanceTimersByTimeAsync), EmailKit's renderer gets the kit as an argument instead of aliasing this, its figure regex drops the literal no-break spaces \s already matches, digest-service's highlightOf takes NotificationData narrowed per case instead of any, and eslint ignores coverage/. 254 backend and 670 frontend tests pass, and tsc, deno check, deno lint, deno fmt, eslint and doc:lint are clean; every page was checked in screenshots at 300 to 480px in Kanagawa and Smooth. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:19:43 +00:00
khannurien
58c95dc4f3 v4: the notifications page restores its list and scroll on Back, and shows what it missed — it now saves its loaded pages and scroll position as it is left and restores them on Back, through useFeedCache, useScrollSave and useScrollRestore like the feeds and the profile's tabs; the saved list keeps the ones the server confirmed read as read, so coming back doesn't highlight them again. The page never refetched after a reconnect, so notifications sent while the socket was down never showed, and the next read-all marked them read unseen; a reconnect, and every restored list (after a reload the socket hasn't said yet whether anything is new), now fetch the first page again, and what is new goes on top with the socket's arrivals, those alone sorted newest first so the server's order stays. Marking read now follows the unread notifications shown, each sent once, instead of each socket message, and a list with none unread sends nothing. The arrivals shift the server's offset pages, so a next page could repeat the last one's tail: each notification shows once. The next page loads as the end scrolls into view, with FeedFooter, as every other page list does, and the loading and error states go through ListState. The "N new" pill was a number before a separately translated "new", which French made "3 nouveau"; it is one plural message, "3 nouvelles". Dates past a week and the tooltip use the app's locale rather than the browser's. The mention, comment and like icons had no colored circle; they have one now, and the decorative emoji are hidden from screen readers. 670 frontend tests pass (new: loading on scroll, the save on leaving, the restore with what came meanwhile, the reconnect backfill), and tsc, deno fmt, deno lint, doc:lint and i18n:compile --strict are clean; the restore was checked in the running app at phone width. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 19:54:31 +00:00
khannurien
0a16d3c96c v4: users can't rename themselves, /settings lays out again, and its selectors fit a phone — PATCH /api/me refuses a username with a 400 and changes nothing else in that request; a name changes only through PATCH /api/users/:key/username, which needs user:manage, from /@name/manage. The old name still stays an alias for good. renameUser lost its 30-day limit and its exempt flag, and RENAME_LIMITS holds the slug limit alone. /settings shows the username as plain text, and the rename hint left the catalogs. The settings grid sized its control column to its widest content, the rename hint, which stretched every control under it; that column now takes the remaining width and each control keeps its own size. A segmented control wraps instead of overflowing; in a settings row on a phone it takes the full width, and one of more than 4 options (the styles) lays out in two even rows. .btn-border turned red on hover, so Settings, Manage, Retry and Back looked like Log out; it is now neutral (Smooth's accent is red too), and only Log out takes the new btn-border--danger, muted until hovered red. A <button> and a <Link> .btn-border are now the same height. API.md and the module docs say who renames; the backend tests rename through the moderator route and check a self-rename is refused, and the malformed-name check moved to the manage tab's test. 254 backend and 667 frontend tests pass, and tsc, deno check and doc:lint are clean; Smooth, Brutalist, GeoCities and Kanagawa were checked in screenshots at desktop and phone widths. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-07 06:50:43 +00:00
khannurien
4468438ba7 v4: every page and API URL follows one of two rules, written down in API.md — pages are a hierarchy where @name is a user and any other root word a category or an app page, tabs and views are plain segments and a default view is the bare path; API resources are /api/<plural>/<key>, a user's key being an id or @name, the caller's things under /api/me, media a sub-resource, lookups by another key a query filter. ~ is gone: the feed is /, /hot, /new, /journal, /follow and /music/hot, a profile /@bob with its overview and the tabs dumps, upvoted, playlists (her own only), follow (users, and /follow/playlists), invitees and manage (role and name, for user:manage), which replace the separate UserDumps, UserUpvoted and UserPlaylists pages; /settings holds the account (now with the username) and /manage the category manager, both linked from the user menu and one's profile. Dumps and playlists live under their owner, /@bob/dumps/<slug> and /@bob/playlists/<slug>, named by slug alone: migration 0021 re-slugs them to the bare slugify(title), unique per owner and kind (the oldest keeps it, clashes get -2), renames the categories whose slug v4 reserves (hot, follow, settings…), and rewrites this site's v3 links in dump notes, comments, chat, and playlist and profile descriptions. A slug changes only when edited, through the edit forms' new link field, which proposes slugify(title) as the title changes; a username changes from /settings or, for user:manage, from /@name/manage. Every slug or name left behind stays an alias of its item or user for good (dump_slug_aliases, playlist_slug_aliases, user_aliases), so old links redirect without chaining and nobody else can take one; slugs change 3 times a day per item, names once in 30 days, moderators exempt. An alias, or a name in another case, answers the item with its current path: the pages put it in their URL (useCanonicalPath, without fetching again), and the OG middleware answers a 301. shared/paths.ts builds and parses every page path, for the client's links, the emails' (digest-service lost its own builders and its id fallbacks, and links /follow and /settings) and the server's matchers (OG tags, self previews, backlinks, self-url); shared/slug.ts holds slugify and the format categories share. Dumps and playlists carry ownerUsername and a required slug, and notifications join their users' current names and their items' paths each time they are sent or read, so renames reach them and a deleted item's has no link. The API was renamed per the plan's table: POST /api/users and /api/sessions, /api/password-resets, /api/me (with settings, password, avatar, media-token, playlists?contains=, follow, feed?from=, notifications), /api/users/:key/{dumps,upvoted,playlists,follow/users,follow/playlists,invitees,avatar} and by path /api/users/:key/{dumps,playlists}/:slug, /api/dumps?url= and ?category=, /api/dumps/:id/{file,thumbnail,metadata}, /api/playlists/:id/cover, /api/remote/{preview,image,bandcamp,youtube} (one router), and /api/unsubscribe, the v3 POST /api/users/unsubscribe staying for the digests already sent. A stored default feed tab "followed" reads as "follow" once. dumps.slug and playlists.slug stay nullable, since SQLite can't add NOT NULL without rebuilding tables others reference; every writer sets them and unique (user_id, slug) indexes hold them. 254 backend tests (new: migration 0021 on a v3 fixture, slug aliases, swaps and rate limits, username renames, aliases and limits, reads by path and their privacy, the OG 301s, both unsubscribe paths, v4-shaped digest links with no ids, notification joins, shared paths and slugs) and 668 frontend tests (new: the profile's tabs and follow views, Settings, Manage, the link field, canonical URLs on the dump, playlist and profile pages, bare default feed paths) pass, and tsc, eslint, deno check, deno lint, deno fmt and doc:lint are clean; the migration was also run on a copy of the development database. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 16:08:59 +00:00
khannurien
fd0d81fe66 v4: the emails' words live in the catalogs alone and their look is read from the site's stylesheets — every email message was written as i18n._({ id, message }), so its English sat in the template and again in en.po, and api/mail/styles.ts held a copy of each theme's palette, fonts and radii that a test chased. The templates now name each message by its id only (i18n._("email.reset.title", values)) and its English is written in locales/en.po, its French in fr.po; the four labels the settings tab and the unsubscribe page share with the emails read the same ids through @lingui/react's Trans instead of restating the English. i18n:extract runs without --overwrite, which would write the id where the English is, and i18n:compile runs --strict, so a language lacking a message fails the build; it caught email.digest.hl.mentionIn, never translated into French. The new api/mail/theme-tokens.ts reads src/tokens.css and src/themes/<style>.css and resolves a token for one scheme (var(), light-dark(), color-mix(), translucent colors laid over the card), and styles.ts names tokens for every color, GeoCities' rainbow, Melange's spice and shadow, Kanagawa's syntax hues, the font stack and the corner radii, keeping only what is the email's own (type scale, title, frame, divider, avatar rounding). The copies had drifted: GeoCities' stack had lost Marker Felt and its light emails drew the dark rainbow, which now also recolors its dividers on the "auto" scheme. The pills take the site's badge color, now the --color-badge-bg token that the private badge reads too. The Dockerfile copies src/tokens.css and src/themes/ into the runtime image, which reads them at load. The catalogs were re-extracted, having been stale against the code. 240 backend tests (new: the token reader, the dark rainbow; the palette-copy test is gone) and 661 frontend tests pass, and tsc, deno check and doc:lint are clean; sendDueDigests' empty-digest test fails now and then, as it did before. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:21:45 +00:00
khannurien
7960d74c55 v4: weekly digests and monthly recaps go out by email, every email is written once as Markdown in the app's one catalog, and a review of the last two commits removes what they stated twice — users get a weekly digest (what happened to them, what the people and playlists they follow posted, the week's top dumps, the community) and a monthly recap (the month's top dumps and the community), both on by default, in their own style and scheme. digest-service sends what is due every quarter of an hour from digest_schedule (migration 0020), which holds each user's rolling 7- or 30-day period; existing users' first periods are spread over the coming week or month, a digest more than a period late covers the last full one, an empty one isn't sent, and an address like admin@localhost is skipped. Each digest carries a one-click unsubscribe (List-Unsubscribe and List-Unsubscribe-Post, POST /api/users/unsubscribe) and a link to the new /unsubscribe page, which asks before turning that cadence off, since mail scanners open every link; the token (lib/jwt) names the user and the cadence and never expires. The welcome email now says the digests are on. The emails' words moved out of a hand-written api/mail/strings.ts into the Lingui catalogs the interface uses: lingui extracts api/mail/ too, the catalogs moved to a top-level locales/ that both halves read, compile emits ES modules the server imports (api/mail/i18n.ts), messages take ICU plurals, and the labels the settings tab, the unsubscribe page and the emails share are one message each (their French had already drifted apart). Each template writes its email once as Markdown, which is the plain-text part, and EmailKit.render draws its blocks in the style (a lone link is a button, a list item starting with an image a thumbnail row, a numbered list a ranking, --- the footer), so no string is declared for an HTML and a text version; user text is escaped for Markdown, which the welcome body didn't do for usernames, whose _ read as emphasis. paint() set colors before a border-top shorthand that reset them, so every divider was black. The review also gave one home to what was written twice: the locale list (shared/locale.ts, read by lingui.config, the client, the server and its catalog loader), the base style and default scheme (shared/theme.ts), an email's look (lookOf, which digest-service, welcome and reset share), whether email is on (EMAIL_ENABLED, read by email-service and index.html), a setting's column and its default in the digests' SQL (SETTINGS_COLUMNS, DEFAULT_SETTINGS), the unsubscribe token (made once, not parsed back from a URL), AuthProvider's compare-and-store, ThemeProvider's twin style and scheme code (useThemeChoice), the test JWT encoder, and duplicate digest messages. The email palettes stay a flattened copy of the themes' tokens, now under a test that caught Times' secondary and link colors drifting. The Dockerfile copies api/mail and locales so the build extracts the emails' messages and the server finds its catalogs, and deno.json maps @lingui/core for the runtime image. 236 backend tests (new: digest building, sending, catch-up, scheduling and unsubscribing; the Markdown renderer; French from the catalog; palettes against themes; the stagger migration; unsubscribe tokens) and 661 frontend tests (new: the unsubscribe page) pass, and tsc, eslint, doc:lint, deno check, deno lint and deno fmt are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:14:46 +00:00
khannurien
89014639cb v4: settings are stored on the server, starting with the style, the color scheme and the digest switches — every setting lived in the browser's localStorage, so it didn't follow an account to another browser and the server couldn't read it, which the digests need (whether to send each, and in which style). user_settings (migration 0019) holds one row per user who changed anything, the defaults otherwise, and GET/PATCH /api/users/me/settings read and change it (settings-service, model/settings, validators/settings, shared/settings). AuthProvider reads the settings into the session alongside the user, on mount and on focus, so a change made elsewhere arrives; useSettings changes them at once and puts them back if the server refuses. ThemeProvider now sits inside AuthProvider: a signed-in user's choice comes from their settings and is mirrored to localStorage for index.html's first-paint script, a visitor's stays in the browser, and an account with no choice yet takes the one this browser had, once. Until anyone picks a style, the instance's GERBEUR_DEFAULT_STYLE applies, written into index.html's default-style tag along with email-enabled (src/config/instance.ts), and a browser no longer stores the default as if it had been chosen, so changing the instance's style reaches everyone who never picked one. The settings tab gets an Email section with the weekly digest and monthly recap switches, disabled with a note when the server has no SMTP. The style and scheme names moved to shared/theme.ts in the previous commit; the default feed tab and sounds stay in the browser for now. 223 backend tests (new: settings defaults, patches, isolation, refusals and cascade; index.html's new tags) and 658 frontend tests (new: useSettings, the account's theme winning and a browser's carried over, settings read into the session, the digest switches) pass, and tsc, eslint, doc:lint, deno check, deno lint and deno fmt are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 21:20:57 +00:00
khannurien
f20ca83f0c v4: every email is rendered by its template in api/mail/ on one layout that takes the instance's style, and sent one way — the welcome email was built and sent inside the register route and the reset link in its service, each writing Markdown, running marked, and calling a nodemailer wrapper whose cc, bcc, replyTo and send result nothing used. Now api/mail/ holds one template per email (welcome.ts, password-reset.ts) on layout.ts's EmailKit, which draws each block with inline styles in a style from styles.ts: the six site styles' palettes, copied from src/themes/, and what makes each recognisable in a mail client (Brutalist's hard border and shadow, GeoCities' ridge and rainbow rules, Times' underlined ink links, Kanagawa's colored quotes, dates and wordmark, Melange's spice bar). On the "auto" scheme the light palette is inlined and the dark one swapped in by a prefers-color-scheme block; a chosen scheme sends only its own. User-written text is escaped where it lands. strings.ts holds every email's words in English and French. GERBEUR_EMAIL_LOCALE picks the language and GERBEUR_DEFAULT_STYLE the instance's style, which the welcome and reset emails are drawn in; GERBEUR_WELCOME_EMAIL_BODY still replaces the welcome's Markdown, whose blocks now take the style too. email-service sends a rendered email with its recipient and headers, refuses to load when GERBEUR_SMTPS_URL is set without GERBEUR_FROM_EMAIL (sends used to fail one by one), and takes a test transport, through which the tests now read what was sent. The welcome send moves to user-service's sendWelcomeEmail. The style and scheme names move to shared/theme.ts, which the server now needs. README, mod.ts and .env.example describe mail/ and the new variables, and drop the "notifications, invites" email that never existed. 221 backend tests pass (new: the layout's schemes, styles and escaping, both templates, the welcome on sign-up, a reset link from the email that works), and deno check, deno lint, doc:lint, deno fmt, tsc and eslint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 21:11:54 +00:00
457 changed files with 32629 additions and 12332 deletions
Expand all files Collapse all files

8
.devcontainer/Dockerfile
View File

@@ -1,3 +1,11 @@
FROM denoland/deno:bin-2.9.0 AS deno
FROM mcr.microsoft.com/devcontainers/typescript-node:20
COPY --from=deno /deno /usr/local/bin/deno
# The API draws thumbnails and waveforms with ffmpeg, in the workspace and in
# the instances' services alike (compose.yaml), which don't get the features.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# The services share Deno's cache through a volume (compose.yaml), which Docker
# creates owned as this directory is, so it must exist and be node's.
RUN mkdir -p /home/node/.cache/deno && chown -R node:node /home/node/.cache

57
.devcontainer/compose.yaml Normal file
View File

@@ -0,0 +1,57 @@
# The dev environment: the workspace VS Code attaches to (`dev`), and one
# service per dev instance of gerbeur (dev/instances.ts), which start with it.
# Each instance service runs dev/start.ts for its own instance, which derives
# its ports, URLs, database and uploads from its entry there; a new instance is
# an entry there and a service here.
#
# Every service shares the host's network, as the workspace always did: an
# instance's URLs are localhost ones, which the browser and the other
# instances' servers must both reach.
#
# From the workspace's terminal (docker-outside-of-docker):
# docker compose -p gerbeur-dev logs -f musique
# docker compose -p gerbeur-dev restart gerbeur
# docker compose -p gerbeur-dev stop gerbeur musique # then `deno task dev`
name: gerbeur-dev
x-workspace: &workspace
build:
context: .
dockerfile: Dockerfile
network_mode: host
user: node
working_dir: /workspaces/gerbeur
volumes:
- ..:/workspaces/gerbeur:cached
# Shared, so that each service doesn't download the jsr and npm packages
# again.
- deno-cache:/home/node/.cache/deno
x-instance: &instance
<<: *workspace
restart: unless-stopped
depends_on:
install:
condition: service_completed_successfully
services:
dev:
<<: *workspace
command: sleep infinity
# Installs the packages and draws the icons once, before any instance
# starts, so that the instances don't do it at the same time.
install:
<<: *workspace
command: sh -c "deno install && deno task icons public"
gerbeur:
<<: *instance
command: deno run --env-file -A dev/start.ts gerbeur
musique:
<<: *instance
command: deno run --env-file -A dev/start.ts musique
volumes:
deno-cache:

13
.devcontainer/devcontainer-lock.json
View File

@@ -1,15 +1,20 @@
{
"features": {
"ghcr.io/devcontainers-extra/features/ffmpeg-apt-get:1": {
"version": "1.0.17",
"resolved": "ghcr.io/devcontainers-extra/features/ffmpeg-apt-get@sha256:735491c8e9b2236cae727d6d4904663f6a58eb131d8c4b6b18fd0eea3d8c5b71",
"integrity": "sha256:735491c8e9b2236cae727d6d4904663f6a58eb131d8c4b6b18fd0eea3d8c5b71"
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"version": "1.10.1",
"resolved": "ghcr.io/devcontainers/features/docker-outside-of-docker@sha256:698f5900fac471e046c6e9d4333fec88c0e8d952c713494fa0712dce80baeca0",
"integrity": "sha256:698f5900fac471e046c6e9d4333fec88c0e8d952c713494fa0712dce80baeca0"
},
"ghcr.io/khannurien/devcontainer-features/claude-code:1": {
"version": "1.2.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/claude-code@sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9",
"integrity": "sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9"
},
"ghcr.io/khannurien/devcontainer-features/opencode:1": {
"version": "1.1.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/opencode@sha256:8df3f8c98a104ce7e089ba3e8df39e0d3890753c8dc92e03deb54527d30d3574",
"integrity": "sha256:8df3f8c98a104ce7e089ba3e8df39e0d3890753c8dc92e03deb54527d30d3574"
},
"ghcr.io/khannurien/devcontainer-features/rtk:1": {
"version": "1.1.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/rtk@sha256:4a60b3a1d1b64904de3d856c61cdf7e5943e3c74d114609787dc88d9db7c788f",

31
.devcontainer/devcontainer.json
View File

@@ -1,19 +1,12 @@
{
"name": "Deno",
"build": {
"dockerfile": "Dockerfile"
},
"name": "gerbeur",
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],
"runArgs": ["--network=host"],
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "yarn install",
// The workspace and the dev instances of gerbeur, one service each: see
// compose.yaml. The instances start with the container.
"dockerComposeFile": "compose.yaml",
"service": "dev",
"workspaceFolder": "/workspaces/gerbeur",
"shutdownAction": "stopCompose",
// Configure tool-specific properties.
"customizations": {
@@ -26,9 +19,11 @@
},
"features": {
"ghcr.io/warrenbuckley/codespace-features/sqlite:1": {},
"ghcr.io/devcontainers-extra/features/ffmpeg-apt-get:1": {}
// The docker CLI, on the host's daemon, for the instances' logs and
// restarts (compose.yaml). Docker's own packages: the base image is Debian
// trixie, which has no moby-cli.
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"moby": false
}
}
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
}

42
.env.example
View File

@@ -48,19 +48,45 @@ GERBEUR_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
# Secret key used to sign JWTs. Generate with: openssl rand -hex 32
GERBEUR_JWT_SECRET=
# SMTP connection URL used by the email service (notifications, invites, etc.)
# SMTP connection URL for outgoing email: the welcome email, password reset
# links and the weekly/monthly digests. Leave empty to send no email at all.
# Example: smtps://username:password@smtp.example.com:465
GERBEUR_SMTPS_URL=
# Sender address used in outgoing emails (e.g. no-reply@example.com)
# Required when GERBEUR_SMTPS_URL is set.
# Sender of every email, as an address or "Name <address>"
# (e.g. gerbeur <no-reply@example.com>).
# Required when GERBEUR_SMTPS_URL is set: the server refuses to start without it.
GERBEUR_FROM_EMAIL=
# Markdown body for the account creation welcome email.
# Supports {{username}} and {{site_name}} placeholders.
# Defaults to a built-in template when unset.
# Use \n for line breaks in single-line .env values, or use a quoted multiline block.
GERBEUR_WELCOME_EMAIL_BODY="# Welcome to {{site_name}}!\n\nHi **{{username}}**,\n\nYour account has been created successfully. Welcome aboard!"
# Language of every email: en (default) or fr. The interface follows each
# visitor's browser, which the server never sees, so emails use the instance's.
GERBEUR_EMAIL_LOCALE=en
# The instance's style: smooth (default), brutalist, geocities, nyt, kanagawa
# or melange. Visitors and accounts see it until they pick one, and the welcome
# and password reset emails are drawn in it; a digest uses its reader's own
# style. Applied at runtime; changing it needs a restart of both the API and Vite.
GERBEUR_DEFAULT_STYLE=smooth
# How often, in minutes, the /community page's figures are recomputed.
GERBEUR_COMMUNITY_REFRESH_MINUTES=15
# Set to 0 to keep users of other gerbeur instances from signing in here, which
# also signs out those already in until it is set back, or to keep this
# instance's users from signing in to others, which signs them out there.
# GERBEUR_FEDERATION_INCOMING=1
# GERBEUR_FEDERATION_OUTGOING=1
# Set to 1 to let federated sign-in reach other instances on private addresses
# (localhost included) and over plain http, so that two instances on one
# machine can trust each other in development. Never in production.
# GERBEUR_FEDERATION_ALLOW_PRIVATE=1
# Markdown body of the welcome email sent on account creation, replacing the
# built-in one (written in GERBEUR_EMAIL_LOCALE). Supports {{username}} and
# {{site_name}}. Use \n for line breaks in single-line .env values, or a
# quoted multiline block.
# GERBEUR_WELCOME_EMAIL_BODY="# Welcome to {{site_name}}!\n\nHi **{{username}}**,\n\nYour account has been created successfully. Welcome aboard!"
# ── Frontend (Vite) ───────────────────────────────────────────────────────────
# These must be prefixed with VITE_ to be exposed to the client bundle.

283
API.md Normal file
View File

@@ -0,0 +1,283 @@
# gerbeur's URLs
Every page and every API endpoint follows one of two rules. This file summarizes
them. Each router's module doc in `api/routes/` lists its own endpoints in full,
and a route change updates both. `shared/paths.ts` builds and parses the page
paths.
## The two rules
**Pages.** The path is a hierarchy. `@name` is a user, and `@name@host` a user
whose account lives on another instance. Any other word under the root is a
category or an app page. Under a user, each segment is a fixed tab name, and
under `dumps` or `playlists`, a slug. A view of a page is one more segment. A
default view is the bare path, with no redirect. A category is a segment where
the page is the category's own, a dump category's feed at the root (`/music`)
and a playlist category's under `/playlists` (`/playlists/mixes`); where it
narrows a list on another page, a profile's or a search's, it is `?category=`,
of the kind that list holds. Switching tabs, or a tab's views, takes the place
of the current history entry; picking a category, or opening another page or an
editor, adds one.
**API.** A resource is `/api/<plural>/<key>[/<noun>]`. Its key is its id, except
a user's, which may also be `@name`. The `@` keeps the two apart, so no regex
sniffing is needed. A read by page path mirrors the page:
`/api/users/@bob/dumps/<slug>`. The caller's own things live under `/api/me`.
Media is a sub-resource. A lookup by another key is a query filter. Each concept
has one word, the same as in pages.
## Pages
| Path | Views | Access | Page |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------------------------- |
| `/` | `/hot`, `/new`, `/journal`, `/follow`, `/follow/playlists`; the bare `/` is the viewer's default tab, the bare `/follow` the users followed | anyone, `/follow` signed in | `Index` |
| `/<category>` | `/<category>/hot`, `/new`, `/journal`, `/follow`, `/follow/playlists` | anyone, `/follow` signed in | `Index` |
| `/@<name>` | the overview: dumps beside upvoted | anyone | `UserPublicProfile` |
| `/@<name>/dumps`, `/upvoted`, `/playlists`, `/invitees` | `?category=` narrows the dumps, the upvoted or the playlists | anyone | `UserPublicProfile` |
| `/@<name>/follow` | `/@<name>/follow/playlists`; the bare tab lists users | anyone | `UserPublicProfile` |
| `/@<name>/manage` | | `user:manage`, on others' profiles | `UserPublicProfile` |
| `/@<name>/dumps/<slug>` | | anyone; a private dump, its poster | `Dump` |
| `/@<name>/dumps/<slug>/edit` | | its poster, `dump:moderate` | `DumpEdit` |
| `/@<name>/playlists/<slug>` | | anyone; a private playlist, its owner | `PlaylistDetail` |
| `/@<name>/playlists/<slug>/edit` | | its owner, `playlist:moderate` | `PlaylistDetail` |
| `/playlists` | `/playlists/<category>`, a playlist category's | anyone | `Playlists` |
| `/search?q=` | `/search/users?q=`, `/search/playlists?q=`; `?category=` narrows the dumps or the playlists | anyone | `Search` |
| `/notifications` | | signed in | `Notifications` |
| `/community` | `/community/monthly`; the bare path is the week | signed in | `Community` |
| `/settings` | | signed in | `Settings` |
| `/manage` | | `category:manage`, `federation:manage` | `Manage` |
| `/login`, `/register?token=` | `/login?next=<path>` comes back to a page | signed out | `UserLogin`, `UserRegister` |
| `/authorize?client_id=…` | | signed in, else via `/login?next=` | `Authorize` |
| `/federation/callback?code=&state=` | | anyone | `FederationCallback` |
| `/reset-password` | `/reset-password?token=`, from the emailed link, sets the new password | anyone | `ResetPassword` |
| `/unsubscribe?token=` | | anyone | `Unsubscribe` |
`/?share_url=` opens the create form on a link shared from Android, `?chat=open`
opens the chat over any page, `?chat=<message id>` on that message, and
`#comment-<id>` scrolls a dump's page to a comment. A signed-in page opened
signed out replaces itself with `/login?next=<the page>`.
## Names
- **Slugs** of categories, dumps and playlists are lowercase letters and digits
in words joined by single hyphens (`shared/slug.ts`). A category's is at most
50 characters, a dump's or a playlist's at most 60.
- **A dump's or a playlist's slug** is made from its title when it is created,
unique among its owner's items of that kind: a clash gets `-2`, `-3`. It
changes only when someone edits it: the edit forms propose `slugify(newTitle)`
when the title changes. Its owner may change it 3 times a day; moderators
(`dump:moderate`, `playlist:moderate`) may edit any slug, at any time.
- **A username** is 1 to 32 letters, digits and underscores, unique in any case.
A page writes it in its stored case, and another case still resolves. Users
can't rename themselves: those with `user:manage` rename anyone, from
`/@<name>/manage`.
- **A user from another instance** is `name@host` here: their name at home and
their home's host, with its port if it has one (`alice@music.example`). Their
name is their home's to change, which leaves the old handle an alias here too.
Mentions write the whole handle. A handle with this instance's own host, as
another instance writes a user from here (`bob@gerbeur.example`), names the
local `bob`, in a path, an API key or a mention.
- **Aliases.** A slug or a name left behind stays an alias for good, which
always names the item or user itself, so redirects never chain. Nobody else
can take one. An item can take back one of its own. Deleting an item frees its
aliases.
- **Redirects.** A page reached by an alias, or by a name in another case, gets
the item from the API under its current path and replaces its URL. The server
answers the same path with a 301 to crawlers and full page loads.
- **Reserved words.** A category can't take a word the root already uses:
`hot new journal follow search login register notifications reset-password
unsubscribe settings manage community playlists authorize federation api ws
assets emojibase`
(`api/lib/reserved-slugs.ts`). Dump and playlist slugs reserve nothing, and no
slug has an `@`.
## API
Every JSON endpoint answers the envelope of `shared/api.ts`:
`{ "success": true, "data": … }`, or
`{ "success": false, "error": { "code",
"message" } }` with the matching HTTP
status. A list answers `{ "items", "hasMore" }` and takes `?page=` (from 1) and
`?limit=` (20 by default, 100 at most). "Signed in" means a session token as
`Authorization: Bearer …`. Media can't send a header, so a private file,
thumbnail, waveform, cover or attachment takes the owner's media token as
`?token=`.
### me
- `GET /api/me` — signed in. The caller's own user.
- `PATCH /api/me` — signed in. Changes the description and email.
- `GET /api/me/media-token` — signed in. A media token.
- `GET`, `PATCH /api/me/settings` — signed in. The caller's settings.
- `PUT /api/me/password` — signed in, with the current one. A new password, and
fresh tokens.
- `PUT /api/me/avatar` — signed in. A new avatar (multipart).
- `GET /api/me/playlists?contains=:dumpId` — signed in. The caller's playlists,
and whether each holds the dump.
- `GET /api/me/follow` — signed in. The ids of the users and playlists followed.
- `PUT`, `DELETE /api/me/follow/users/:id`, `/api/me/follow/playlists/:id` —
signed in. Follows or stops following.
- `GET /api/me/notifications` — signed in. The caller's notifications.
- `POST /api/me/notifications/read` — signed in. Marks them all read.
- `GET /api/me/grants` — signed in. The other instances the caller's account
signs in to.
- `DELETE /api/me/grants/:host` — signed in. Withdraws one, ending the caller's
sessions there.
An account from another instance changes its password, email, avatar and name
there: those endpoints answer it 400.
### users
- `POST /api/users` — with an invite. Signs up.
- `GET /api/users?q=` — anyone. Up to 8 users whose name starts with `q`.
- `GET /api/users/:key` — anyone. A profile.
- `GET /api/users/:key/avatar` — anyone. The avatar.
- `GET /api/users/:key/dumps`, `/upvoted`, `/playlists` — anyone; private items
to their owner only. Their lists; `?category=` narrows one to a category of
its kind.
- `GET /api/users/:key/dumps/:slug`, `/playlists/:slug` — anyone; private items
to their owner only. An item by its page path, by an alias too.
- `GET /api/users/:key/follow/users`, `/follow/playlists` — anyone. What they
follow.
- `GET /api/users/:key/invitees` — anyone. The tree of users they invited.
- `GET /api/users/:key/home?path=` — anyone. Redirects to a page of a remote
user's home instance, their profile there by default.
- `PATCH /api/users/:key/role`, `/username` — `user:manage`. Another user's
role, or anyone's name, theirs included.
### dumps
- `POST /api/dumps` — signed in. Posts a link (JSON) or a file (multipart).
- `GET /api/dumps` — anyone. The feed; `?category=<slug>` narrows it, and
`?url=` answers the dumps of the same link instead.
- `GET /api/dumps?follow=users|playlists` — signed in. The feed's `follow` tab:
the dumps of the users, or of the playlists, the caller follows; `?category=`
narrows it too.
- `GET /api/dumps/:id` — anyone who may see it. A dump.
- `GET /api/dumps/:id/voters`, `/related`, `/playlists` — the same. Who upvoted
it, the dumps linked to or from it, the playlists holding it.
- `PUT /api/dumps/:id` — its poster, `dump:moderate`. Title, slug, note,
visibility, categories, link.
- `POST /api/dumps/:id/metadata` — the same. Fetches a link's preview again.
- `DELETE /api/dumps/:id` — the same. Deletes it.
- `GET`, `PUT /api/dumps/:id/file` — the file; a new one, by its poster or
`dump:moderate`.
- `GET`, `PUT`, `DELETE /api/dumps/:id/thumbnail` — the thumbnail; a custom one,
set or removed by its poster or `dump:moderate`.
- `GET /api/dumps/:id/waveform` — anyone who may see it. An audio file's
waveform, `{ "peaks" }`, for the player to seek on.
- `GET`, `POST /api/dumps/:id/comments` — anyone who may see the dump; posting
signed in. Its comments.
### playlists
- `POST /api/playlists` — signed in. Creates a playlist, filed under the
playlist categories it names.
- `GET /api/playlists` — anyone. Every public playlist, newest first;
`?category=<slug>` narrows it to a playlist category.
- `GET /api/playlists/:id` — anyone who may see it. A playlist and its dumps.
- `PUT`, `DELETE /api/playlists/:id` — its owner, `playlist:moderate`. Title,
slug, description, visibility, categories; deletion.
- `POST`, `DELETE /api/playlists/:id/dumps/:dumpId` — the same. Adds or removes
a dump.
- `PUT /api/playlists/:id/order` — the same. Reorders its dumps.
- `GET`, `PUT /api/playlists/:id/cover` — the cover; a new one, by its owner or
`playlist:moderate`.
### comments
- `PATCH`, `DELETE /api/comments/:id` — its author, `comment:moderate`. Edits or
deletes a comment.
- `GET /api/comments/:id/likers` — anyone. Who likes it.
### categories
- `GET /api/categories` — anyone. Every category, dump and playlist ones apart:
each has a `kind`, and its slug is unique within its kind.
- `POST /api/categories`, `PUT`, `DELETE /api/categories/:id` —
`category:manage`. Changes them.
### community
- `GET /api/community?period=weekly|monthly` — signed in. The site's top dumps,
new members, new public playlists and totals over the last week (by default)
or month, as of the server's last refresh.
### chat
- `GET /api/chat?before=&after=` — signed in. A page of messages. The chat
itself goes over `/ws`.
- A message from a moderator or an admin (`chat:everyone`) that says `@everyone`
notifies every user who doesn't have the chat open, and comes flagged
`mentionsEveryone`. No account may be named `everyone`.
### invites
- `POST /api/invites` — signed in. A new invite.
- `GET /api/invites/:token` — anyone. Whether it still works.
### sessions, password-resets, unsubscribe
- `POST /api/sessions` — anyone. Signs in.
- `POST /api/password-resets` — anyone. Emails a reset link.
- `PUT /api/password-resets/:token` — with the link's token. A new password.
- `POST /api/unsubscribe?token=` — with a digest's token. Turns it off. Its v3
path, `POST /api/users/unsubscribe`, stays for good, since sent emails'
`List-Unsubscribe` headers name it.
### federation, oauth, .well-known
Federated sign-in (`shared/federation.ts`): a user signs in here with their
account on another gerbeur instance, their home, by OpenID Connect's
authorization code flow with PKCE, with tokens signed by each instance's Ed25519
key. With `GERBEUR_FEDERATION_INCOMING=0`, `POST /api/federation/logins` and
`/sessions` answer 403; with `GERBEUR_FEDERATION_OUTGOING=0`,
`GET /api/oauth/clients` and `POST /api/oauth/codes` do. An instance is named by
its issuer, its API's origin, whose host is the one in its users' handles. Where
`instance` is asked for, the origin of its pages works too: their
`/.well-known/openid-configuration` names the issuer, whose own metadata must
name those pages back as where its `/authorize` is.
- `GET /.well-known/openid-configuration`, `/.well-known/jwks.json` — anyone.
The instance's metadata and public keys, bare JSON: the one exception to the
`/api/` rule, at the paths the standard fixes.
- `POST /api/federation/logins` — anyone. Starts a sign-in with an account on
another instance; one neither trusted nor already home to someone here needs
an invite.
- `POST /api/federation/sessions` — anyone. Finishes it, as a sign-in.
- `POST /api/federation/events` — another instance. Its signed news of one of
its users here: a new name, email or avatar, or the end of their sessions. A
4xx tells it to give up on the event, a 429 or a 5xx to try again.
- `GET /api/federation/peers` — anyone the trusted instances,
`federation:manage` every listed one.
- `PUT /api/federation/peers`, `DELETE /api/federation/peers/:host` —
`federation:manage`. Trusts or blocks an instance, or takes it off the lists.
- `GET /api/oauth/clients?client_id=` — signed in. The instance asking to sign
the caller in.
- `POST /api/oauth/codes` — signed in. The caller's consent, and a code.
- `POST /api/oauth/tokens` — another instance, form-encoded. Trades a code for
an identity token, bare JSON.
### search
- `GET /api/search?q=` — anyone. A page of dumps, and up to 20 users and
playlists. It finds both kinds of item at once, so `?category=` narrows the
dumps and `?playlistCategory=` the playlists.
### attachments
- `POST /api/attachments` — signed in. Stores an image.
- `GET /api/attachments/:id` — anyone who may see the text holding it.
### remote
- `GET /api/remote/preview?url=` — signed in. A link's preview.
- `GET /api/remote/image?url=` — anyone. A remote image through this site.
- `GET /api/remote/bandcamp?url=` — anyone. A Bandcamp page's tracks.
- `GET /api/remote/youtube?list=` — anyone. A YouTube playlist's tracklist.
### /ws
- `GET /ws?token=` — anyone; a session token signs it in. The WebSocket of live
updates and the chat (`shared/ws.ts`).

7
Dockerfile
View File

@@ -11,6 +11,9 @@ COPY index.html ./
COPY public/ ./public/
COPY src/ ./src/
COPY shared/ ./shared/
# The build extracts and compiles the catalogs, the emails' messages included.
COPY locales/ ./locales/
COPY api/mail/ ./api/mail/
# VITE_API_* are only needed when the API lives on a different host than the frontend.
ARG VITE_API_PROTOCOL
@@ -35,6 +38,10 @@ RUN deno install
COPY api/ ./api/
COPY shared/ ./shared/
COPY --from=builder /app/dist/ ./dist/
COPY --from=builder /app/locales/ ./locales/
# The emails read their colors, fonts and radii from the site's stylesheets.
COPY src/tokens.css ./src/
COPY src/themes/ ./src/themes/
# Persistent data: database and uploaded/generated files must be mounted as volumes.
VOLUME ["/app/api/sql", "/app/api/uploads"]

120
README.md
View File

@@ -3,8 +3,9 @@
A small invite-only social platform for sharing links and files. Users can post
URLs and media (YouTube, SoundCloud, Bandcamp, images, audio, video, …), vote,
comment, follow each other, build playlists, and search content. A real-time
WebSocket layer handles live presence, vote counts, and notifications. The UI is
localized (English and French) and ships multiple visual themes.
WebSocket layer handles live presence, vote counts, and notifications, and
weekly and monthly email digests sum up what happened. The UI is localized
(English and French) and ships multiple visual themes.
## Stack
@@ -30,15 +31,32 @@ cp .env.example .env
### Run
In the devcontainer, the dev instances listed in
[`dev/instances.ts`](dev/instances.ts) start with the container, each a service
of [`.devcontainer/compose.yaml`](.devcontainer/compose.yaml): `gerbeur` at
[http://localhost:3000](http://localhost:3000), `musique` at
[http://localhost:3001](http://localhost:3001). They run side by side so that
signing in from one instance to another can be tried on one machine, and from
the workspace's terminal:
```sh
deno task dev
docker compose -p gerbeur-dev logs -f musique # follow an instance
docker compose -p gerbeur-dev restart gerbeur # restart one
```
This starts both the API server (port 8000, with file watching) and the Vite dev
server (port 3000) concurrently.
Outside the devcontainer, or with those services stopped, `deno task dev` starts
them all, and `deno task dev gerbeur` only those named.
Open [http://localhost:3000](http://localhost:3000). On first run a default
`admin` / `admin` account is created — change the password immediately.
Each instance has an API server, restarted on change, and a Vite server, and its
output is prefixed with its name. They follow one rule
([`dev/start.ts`](dev/start.ts)): instance `<name>` keeps its database in
`api/sql/<name>.db` and its uploads in `api/uploads/<name>/`, reads `.env`, and
takes what sets it apart (ports, site name, emoji, style) from its entry. A new
instance is an entry in `dev/instances.ts` and a service in `compose.yaml`.
On first run each instance creates a default `admin` / `admin` account — change
the password immediately. To let one instance's users sign in to another, trust
it on the other's `/manage`.
### Environment variables
@@ -46,7 +64,7 @@ See [`.env.example`](.env.example) for the full list with descriptions. Key
variables:
| Variable | Description | Default |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| `GERBEUR_JWT_SECRET` | JWT signing secret — **required**, generate with `openssl rand -hex 32` | — |
| `GERBEUR_PUBLIC_URL` | Public-facing URL of the server (no trailing slash) — used for CORS, WebSocket origin, email links, OG URLs | `http://localhost:GERBEUR_PORT` |
| `GERBEUR_PORT` | Internal port Oak listens on | `8000` |
@@ -54,9 +72,15 @@ variables:
| `GERBEUR_FRONTEND_URL` | Frontend base URL for email links and CORS; auto-added to allowed origins — the only variable needed when the frontend runs on a separate host | `GERBEUR_PUBLIC_URL` |
| `GERBEUR_ALLOWED_ORIGINS` | Comma-separated extra origins for CORS/WebSocket; `PUBLIC_URL` and `FRONTEND_URL` are always included — typically only needed in dev for the Vite server | `""` (empty) |
| `GERBEUR_SITE_NAME` | Site name used in the browser tab, app header, OG meta tags, and emails | `gerbeur` |
| `GERBEUR_SMTPS_URL` | SMTPS connection URL for outgoing email (`smtps://user:pass@host:465`) | unset |
| `GERBEUR_FROM_EMAIL` | Sender address for outgoing emails — required when `GERBEUR_SMTPS_URL` is set | unset |
| `GERBEUR_SMTPS_URL` | SMTPS connection URL for outgoing email (`smtps://user:pass@host:465`); unset sends no email | unset |
| `GERBEUR_FROM_EMAIL` | Sender of every email — required when `GERBEUR_SMTPS_URL` is set, or the server won't start | unset |
| `GERBEUR_EMAIL_LOCALE` | Language of every email: `en` or `fr` | `en` |
| `GERBEUR_DEFAULT_STYLE` | The instance's style (`smooth`, `brutalist`, `geocities`, `nyt`, `kanagawa`, `melange`): the site's until one is picked, and the welcome and reset emails' | `smooth` |
| `GERBEUR_WELCOME_EMAIL_BODY` | Markdown body for the account-creation welcome email; supports `{{username}}` and `{{site_name}}` | built-in template |
| `GERBEUR_FEDERATION_INCOMING` | `0` keeps users of other instances from signing in here, and signs out those already in | `1` |
| `GERBEUR_FEDERATION_OUTGOING` | `0` keeps this instance's users from signing in to other instances, and signs them out there | `1` |
| `GERBEUR_FEDERATION_ALLOW_PRIVATE` | `1` lets federated sign-in reach instances on private addresses and over http — for two instances on one machine, never in production | unset |
| `GERBEUR_COMMUNITY_REFRESH_MINUTES` | How often, in minutes, the `/community` page's figures are recomputed | `15` |
| `VITE_API_PROTOCOL` | API protocol baked into the frontend bundle (see [Production](#production)) | `http` |
| `VITE_API_HOSTNAME` | API hostname baked into the frontend bundle | `localhost` |
| `VITE_API_PORT` | API port baked into the frontend bundle | `8000` |
@@ -118,18 +142,50 @@ example.com {
}
```
### Several instances
Each instance is its own deployment, with its own database, uploads, admins and
domain, but its users can sign in to the others with their account, and the
other way round. A user from `music.example` is `@alice@music.example` on the
instances they sign in to. Their password, email, avatar and name stay on their
home instance, and the others follow its changes.
Every instance is ready for it: nothing to configure, as long as
`GERBEUR_PUBLIC_URL` is the instance's https origin (no path) and the instances
can reach each other. Users and admins may name an instance by the address of
its API or of its pages: when the frontend runs on a host of its own
(`GERBEUR_FRONTEND_URL`), that host must pass `/.well-known/` on to the API, as
the Vite dev server does. A handle carries the API's host. Either way can be
turned off: `GERBEUR_FEDERATION_INCOMING=0` takes no users of other instances,
and `GERBEUR_FEDERATION_OUTGOING=0` lets no user sign in elsewhere. An admin
decides who comes in, on `/manage`:
- users of a **trusted** instance sign in directly, from a button on the login
page;
- users of a **blocked** instance can't, and those already in are signed out;
- users of **any other** instance join with an invite, from the register page,
and sign in without one afterwards.
`deno task dev` runs several instances on one machine (see [Run](#run)), with
`GERBEUR_FEDERATION_ALLOW_PRIVATE=1`, which lets them reach each other over
plain http on localhost. Never set it in production. The first, `gerbeur`, takes
no users of the others, whose users sign in to each other.
## Project structure
```
dev/ # The dev instances (instances.ts) and their launcher (start.ts), for `deno task dev`
shared/ # Imported by both halves: API and socket types, limits, permissions
test/ # Tests of shared/ (Deno): test/<module>.test.ts
locales/ # Lingui catalogues (en, fr): the interface's and the emails' messages
api/
main.ts # Entry point — Oak application, middleware, routes
config.ts # Environment variables
middleware/ # Oak middleware: errors, auth, OG tags
routes/ # One router per URL prefix: HTTP + WebSocket, no SQL
routes/ # One router per API resource: HTTP + WebSocket, no SQL
services/ # Business logic and every SQL query
remote/ # Everything that fetches from other sites
mail/ # One template per email, and the layout they share
providers/ # Link metadata providers (YouTube, SoundCloud, Bandcamp, …)
model/ # Per table: row type, row guard, row ↔ API mappers
validators/ # Per domain: request body guards
@@ -142,22 +198,31 @@ api/
migrations/ # Versioned schema migrations
test/ # Backend tests (Deno): test/<layer>/<module>.test.ts
sql/
gerbeur.db # SQLite database (not committed)
uploads/ # User-uploaded files — avatars, dumps, thumbnails, … (not committed)
gerbeur.db # SQLite database (not committed); <name>.db per dev instance
uploads/ # User-uploaded files — avatars, dumps, thumbnails, … (not committed); <name>/ per dev instance
src/ # React frontend (Vite)
config/ # API base URL, feed tabs, player mode
config/ # API base URL, default feed tab, player mode, instance settings
pages/ # One component per route, nothing else
components/ # React components only (.tsx)
contexts/ # FooContext.ts + FooProvider.tsx pairs
hooks/ # Every hook (useFoo.ts)
utils/ # Every non-React module: pure helpers, storage, API glue
test/ # Frontend tests (Vitest): test/<layer>/<domain>/<Module>.test.tsx
locales/ # Lingui message catalogues (en, fr)
themes/ # Per-theme CSS files
i18n.ts # Lingui runtime setup
public/ # Static assets (favicon, manifest, service worker)
```
### URLs
Every page and API path follows one of two rules, which [API.md](API.md) states,
along with every page, the slug and username rules and every endpoint. Pages are
a hierarchy under `@name` for users and plain words for categories and app pages
(`/@bob/dumps/never-gonna`, `/music/hot`), built and parsed by
`shared/paths.ts`. API resources are `/api/<plural>/<key>`, the caller's own
under `/api/me`. A route change updates its router's module doc and API.md
together.
Whatever crosses the wire is defined once, in `shared/`: the API envelope
(`shared/api.ts`), each domain's types and request bodies (`shared/dump.ts`,
`shared/user.ts`, …), the socket protocol (`shared/ws.ts`), the validation
@@ -169,8 +234,9 @@ The backend is split by layer, one kind of module per folder, and each file is
named after its domain: the dumps are `routes/dumps.ts`,
`services/dump-service.ts`, `model/dump.ts` and `validators/dump.ts`. Routes
parse and answer, services hold the logic and the SQL, `remote/` is the only
place that fetches from other sites. Tests mirror the source one file per
module, so the tests of `services/chat-service.ts` are in
place that fetches from other sites, and `mail/` renders each email that
`services/email-service.ts` sends. Tests mirror the source one file per module,
so the tests of `services/chat-service.ts` are in
`test/services/chat-service.test.ts`.
### Comments
@@ -245,16 +311,16 @@ types are used through the global `React` namespace rather than a default import
of `react`, which `deno doc` can't follow.
`deno task doc` builds the HTML reference into `docs/` from every module of
`api/`, `shared/` and `src/` but the tests (and the generated `src/locales/`).
Its index page is the module doc of `mod.ts` at the root, which introduces the
repository's three parts and their layers in prose; `deno doc` builds the list
of every module itself, grouped by path. The `docs` workflow publishes it on
every push to `main` that touches them. `deno task doc:lint` checks the rules
the linter can see, and that no summary ends in a colon, through `doc-lint.ts`.
Two kinds of diagnostic are let through. A field without a doc passes, since a
field gets one only when it adds to its type's. `deno doc` doesn't document npm
packages, so it takes a type imported from one (react-hook-form's `FieldValues`,
say) for a private one, and those references pass too.
`api/`, `shared/` and `src/` but the tests. Its index page is the module doc of
`mod.ts` at the root, which introduces the repository's three parts and their
layers in prose; `deno doc` builds the list of every module itself, grouped by
path. The `docs` workflow publishes it on every push to `main` that touches
them. `deno task doc:lint` checks the rules the linter can see, and that no
summary ends in a colon, through `doc-lint.ts`. Two kinds of diagnostic are let
through. A field without a doc passes, since a field gets one only when it adds
to its type's. `deno doc` doesn't document npm packages, so it takes a type
imported from one (react-hook-form's `FieldValues`, say) for a private one, and
those references pass too.
Each of the frontend's `components/`, `contexts/`, `hooks/` and `utils/` is
split into the same domain folders — `auth`, `category`, `chat`, `comment`,

99
api/config.ts
View File

@@ -7,33 +7,59 @@
* @module
*/
import { BASE_STYLE, isStyleName, type StyleName } from "../shared/theme.ts";
import { isLocale, type Locale, SOURCE_LOCALE } from "../shared/locale.ts";
/** The port the server listens on (`GERBEUR_PORT`, default 8000). */
export const PORT: number = Number(Deno.env.get("GERBEUR_PORT")) || 8000;
/**
* The `smtps://` URL of the server email-service sends mail through
* (`GERBEUR_SMTPS_URL`). When it is empty, the welcome and password reset
* emails are silently skipped.
* The `smtps://` URL of the server `email-service.ts` sends mail through
* (`GERBEUR_SMTPS_URL`). When it is empty, no email is sent: sign-up sends no
* welcome, a password reset request does nothing, and there are no digests.
*/
export const SMTPS_URL: string = Deno.env.get("GERBEUR_SMTPS_URL")?.trim() ||
"";
/** The `From` address of the welcome and password reset emails. */
/**
* Whether the instance sends email, which is whether {@link SMTPS_URL} is
* set. `email-service.ts` opens its transport by it, and index.html tells the
* client, whose settings show the digest switches disabled without it.
*/
export const EMAIL_ENABLED: boolean = SMTPS_URL.length > 0;
/**
* The `From` address of every email (`GERBEUR_FROM_EMAIL`), as an address or
* as `Name <address>`. `email-service.ts` refuses to load without it when
* {@link SMTPS_URL} is set.
*/
export const FROM_EMAIL: string = Deno.env.get("GERBEUR_FROM_EMAIL")?.trim() ||
"";
const DEFAULT_WELCOME_EMAIL_BODY = `# Welcome to {{site_name}}!
Hi **{{username}}**,
Your account has been created successfully. Welcome aboard!`;
/**
* The Markdown of the welcome email sent on registration
* (`GERBEUR_WELCOME_EMAIL_BODY`). The users route replaces `{{site_name}}` and
* `{{username}}` in it and sends it as both text and HTML.
* (`GERBEUR_WELCOME_EMAIL_BODY`), or null for the built-in one in the
* {@link EMAIL_LOCALE}. `api/mail/welcome.ts` replaces `{{site_name}}` and
* `{{username}}` in it.
*/
export const WELCOME_EMAIL_BODY: string =
Deno.env.get("GERBEUR_WELCOME_EMAIL_BODY")?.trim() ||
DEFAULT_WELCOME_EMAIL_BODY;
export const WELCOME_EMAIL_BODY: string | null =
Deno.env.get("GERBEUR_WELCOME_EMAIL_BODY")?.trim() || null;
/**
* The language of every email (`GERBEUR_EMAIL_LOCALE`). The interface picks
* each visitor's language in their browser, which the server never learns, so
* the emails take the instance's. A value that isn't a locale means the
* source one, English.
*/
export const EMAIL_LOCALE: Locale = (() => {
const locale = Deno.env.get("GERBEUR_EMAIL_LOCALE")?.trim();
return isLocale(locale) ? locale : SOURCE_LOCALE;
})();
/**
* The instance's style (`GERBEUR_DEFAULT_STYLE`). The site shows it until a
* visitor or their account picks one, through index.html, and the emails that
* aren't a user's own (the welcome and the password reset) are drawn in it. A
* value that isn't a style means {@link BASE_STYLE}.
*/
export const DEFAULT_STYLE: StyleName = (() => {
const style = Deno.env.get("GERBEUR_DEFAULT_STYLE")?.trim();
return isStyleName(style) ? style : BASE_STYLE;
})();
/**
* The key lib/jwt.ts signs and verifies tokens with (`GERBEUR_JWT_SECRET`).
* The server refuses to start without it.
@@ -138,6 +164,13 @@ export const ORPHANED_ATTACHMENTS_RETENTION_HOURS = 1;
*/
export const UNUSED_INVITES_RETENTION_DAYS = 7;
/**
* How often, in minutes, the `/community` page's figures are recomputed
* (`GERBEUR_COMMUNITY_REFRESH_MINUTES`), 15 by default.
*/
export const COMMUNITY_REFRESH_MINUTES: number =
Number(Deno.env.get("GERBEUR_COMMUNITY_REFRESH_MINUTES")) || 15;
/**
* The site's name (`GERBEUR_SITE_NAME`). It is filled into index.html, the OG
* tags, the icons' manifest and the emails.
@@ -189,3 +222,37 @@ export const ALLOWED_ORIGINS: string[] = Array.from(
...rawOrigins.split(",").map((o) => o.trim()).filter(Boolean),
]),
);
/**
* Lets federated sign-in reach instances on private addresses, `localhost`
* among them, and over plain http (`GERBEUR_FEDERATION_ALLOW_PRIVATE=1`), so
* that two instances on one machine can trust each other in development.
* Off, `remote/gerbeur-instance.ts` reaches only public https instances, since
* an identity is only as safe as the connection it came over.
*/
export const FEDERATION_ALLOW_PRIVATE: boolean =
Deno.env.get("GERBEUR_FEDERATION_ALLOW_PRIVATE")?.trim() === "1";
/**
* Which ways federated sign-in goes, each on unless set to `0`.
*
* - `incoming` (`GERBEUR_FEDERATION_INCOMING`): users of other instances sign
* in here. Off, the login and register pages offer no other instance,
* `/manage` lists only the blocked ones, which this instance's users can't
* sign in to (and none with `outgoing` off too), `federation-service.ts`
* refuses their sign-ins, and the accounts that came from one are signed
* out (`getSessionState`) until it is on again. Their homes' events are
* still applied, so that none is lost meanwhile.
* - `outgoing` (`GERBEUR_FEDERATION_OUTGOING`): this instance's users sign in
* to others. Off, `identity-provider-service.ts` lets no instance sign them
* in, and its metadata says so, which the others check before sending
* anyone here to sign in. The settings page lists no instances, since at
* startup `main.ts` withdraws every grant they gave, which signs them out
* there.
*
* index.html tells the client both. An object, so that the tests can turn one
* off for a while.
*/
export const FEDERATION: { incoming: boolean; outgoing: boolean } = {
incoming: Deno.env.get("GERBEUR_FEDERATION_INCOMING")?.trim() !== "0",
outgoing: Deno.env.get("GERBEUR_FEDERATION_OUTGOING")?.trim() !== "0",
};

14
api/db/migrate.ts
View File

@@ -25,6 +25,13 @@ import { up as up0015TokenRevocation } from "./migrations/0015_token_revocation.
import { up as up0016PerformanceIndexes } from "./migrations/0016_performance_indexes.ts";
import { up as up0017CaseInsensitiveUsers } from "./migrations/0017_case_insensitive_users.ts";
import { up as up0018AttachmentResourceType } from "./migrations/0018_attachment_resource_type.ts";
import { up as up0019UserSettings } from "./migrations/0019_user_settings.ts";
import { up as up0020DigestSchedule } from "./migrations/0020_digest_schedule.ts";
import { up as up0021PageUrls } from "./migrations/0021_page_urls.ts";
import { up as up0022ReserveCommunity } from "./migrations/0022_reserve_community.ts";
import { up as up0023MentionEveryone } from "./migrations/0023_mention_everyone.ts";
import { up as up0024Federation } from "./migrations/0024_federation.ts";
import { up as up0025PlaylistCategories } from "./migrations/0025_playlist_categories.ts";
interface Migration {
name: string;
@@ -62,6 +69,13 @@ const MIGRATIONS: Migration[] = [
name: "0018_attachment_resource_type",
up: up0018AttachmentResourceType,
},
{ name: "0019_user_settings", up: up0019UserSettings },
{ name: "0020_digest_schedule", up: up0020DigestSchedule },
{ name: "0021_page_urls", up: up0021PageUrls },
{ name: "0022_reserve_community", up: up0022ReserveCommunity },
{ name: "0023_mention_everyone", up: up0023MentionEveryone },
{ name: "0024_federation", up: up0024Federation },
{ name: "0025_playlist_categories", up: up0025PlaylistCategories },
];
/**

28
api/db/migrations/0019_user_settings.ts Normal file
View File

@@ -0,0 +1,28 @@
/**
* Gives each user settings stored on the server, so that they follow the
* account across browsers and the server can read them: the digests need the
* reader's choice to receive them and their style. `user_settings` holds one
* row per user who changed anything; a user without one has the defaults.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
/**
* Run once by `runMigrations`. Existing users get no row, so they start with
* the defaults, digests on, like a new account.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS user_settings (
user_id TEXT PRIMARY KEY,
style TEXT,
color_scheme TEXT,
digest_weekly INTEGER NOT NULL DEFAULT 1,
digest_monthly INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
`);
}

52
api/db/migrations/0020_digest_schedule.ts Normal file
View File

@@ -0,0 +1,52 @@
/**
* Lets `digest-service.ts` know when each user's next digest is due.
* `digest_schedule` holds, for each user and cadence they receive, the start
* of the period their next digest covers; it is due once a full period has
* passed.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
const DAY_MS = 86_400_000;
// A frozen copy of digest-service's periods: a migration must keep doing
// what it did when it shipped, whatever the service later becomes.
const PERIOD_DAYS = { weekly: 7, monthly: 30 } as const;
/** Spreads `userId` over `[0, days)` days, the same way each run. */
function offsetOf(userId: string, days: number): number {
let hash = 2166136261;
for (const ch of userId) hash = Math.imul(hash ^ ch.charCodeAt(0), 16777619);
return ((hash >>> 0) % (days * 24)) * 3_600_000;
}
/**
* Run once by `runMigrations`. Every user present gets both cadences, with a
* first period that ends at some hour of the coming week or month, picked
* from their id: the digests start trickling out after the deploy instead of
* all leaving a week later. Users created later are scheduled by the service
* when it first sees them.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS digest_schedule (
user_id TEXT NOT NULL,
cadence TEXT NOT NULL,
window_start TEXT NOT NULL,
PRIMARY KEY (user_id, cadence),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
`);
const users = db.prepare(`SELECT id FROM users;`).all() as { id: string }[];
const insert = db.prepare(
`INSERT OR IGNORE INTO digest_schedule (user_id, cadence, window_start) VALUES (?, ?, ?);`,
);
const now = Date.now();
for (const { id } of users) {
for (const [cadence, days] of Object.entries(PERIOD_DAYS)) {
const start = now - days * DAY_MS + offsetOf(id, days);
insert.run(id, cadence, new Date(start).toISOString());
}
}
}

239
api/db/migrations/0021_page_urls.ts Normal file
View File

@@ -0,0 +1,239 @@
/**
* Moves the site to v4's page URLs, where dumps and playlists live under their
* owner (`/@bob/dumps/<slug>`) and are named by an editable slug alone. A
* slug was `slugify(title)-<8 hex of the id>`, unique site-wide; it is now the
* bare `slugify(title)`, unique in its owner's space for its kind, and every
* slug an item or a user leaves behind stays an alias that redirects
* (`dump_slug_aliases`, `playlist_slug_aliases`, `user_aliases`).
*
* v3 URLs aren't redirected. Instead this rewrites the links to this site's
* pages in what users wrote: dump notes, comments, chat messages, and
* playlist and profile descriptions.
*
* SQLite can't make `slug` NOT NULL without rebuilding `dumps` and
* `playlists`, which other tables reference, so the column stays nullable
* and every writer sets it; the unique index is what the services rely on.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { slugify } from "../../../shared/slug.ts";
import {
dumpPath,
feedPath,
playlistPath,
type ProfileTab,
searchPath,
userPath,
} from "../../../shared/paths.ts";
import { type SlugKind, uniqueSlug } from "../../lib/slugify.ts";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
import { isSelfHost } from "../../lib/self-url.ts";
/** An item's new path, by its v3 slug and by its id. */
type PathsByKey = Map<string, string>;
/**
* Gives every dump or playlist of `kind` the bare slug of its title, unique
* per owner, the oldest first so it keeps the bare one. Returns each one's new
* path under both keys a v3 link may have used: its old slug and its id.
*/
function reslug(db: DatabaseSync, kind: SlugKind): PathsByKey {
const table = kind === "dump" ? "dumps" : "playlists";
const rows = db.prepare(
`SELECT t.id, t.user_id, t.title, t.slug, u.username
FROM ${table} t JOIN users u ON u.id = t.user_id
ORDER BY t.created_at, t.id;`,
).all() as {
id: string;
user_id: string;
title: string;
slug: string | null;
username: string;
}[];
// Cleared first, so that an old slug doesn't take a new one's place.
db.exec(`UPDATE ${table} SET slug = NULL;`);
const set = db.prepare(`UPDATE ${table} SET slug = ? WHERE id = ?;`);
const paths: PathsByKey = new Map();
const build = kind === "dump" ? dumpPath : playlistPath;
for (const row of rows) {
const slug = uniqueSlug(db, kind, row.user_id, slugify(row.title));
set.run(slug, row.id);
const path = build({ ownerUsername: row.username, slug });
paths.set(row.id, path);
if (row.slug) paths.set(row.slug, path);
}
return paths;
}
/** Where each v3 profile tab went: a tab, the overview, or a page of its own. */
const PROFILE_TAB_V4: Record<string, ProfileTab | "" | "/settings"> = {
dumps: "",
playlists: "playlists",
followed: "follow",
invitees: "invitees",
settings: "/settings",
manage: "manage",
};
/** The v3 full lists under a profile, which are v4 tabs. */
const PROFILE_LISTS = new Set<string>(["dumps", "upvoted", "playlists"]);
/**
* Matches a link to a v3 page, its origin being optional: an absolute one is
* this site's only when its host is, a relative one only as a Markdown link's
* target. The path stops before a query or a fragment, which stay as they are.
*/
const V3_LINK_RE =
/(https?:\/\/([^/\s)\]<>"']+))?(\/(?:dumps|playlists|users|~|search\/~)\/[^\s)\]<>"'?#]+)/g;
/**
* Returns the v4 path of the v3 page at `path`, or null when it names nothing
* known, which then stays as written: a deleted dump's link is dead either way.
*/
function v4PathOf(
path: string,
dumps: PathsByKey,
playlists: PathsByKey,
username: (name: string) => string | undefined,
): string | null {
let segments: string[];
try {
segments = path.split("/").slice(1).filter(Boolean).map(
decodeURIComponent,
);
} catch {
return null;
}
const [root, key, ...rest] = segments;
switch (root) {
case "dumps": {
const target = dumps.get(key);
if (!target) return null;
if (rest.length === 0) return target;
return rest.length === 1 && rest[0] === "edit" ? `${target}/edit` : null;
}
case "playlists":
return rest.length === 0 ? playlists.get(key) ?? null : null;
case "users": {
const name = username(key) ?? key;
if (rest.length === 0) return userPath(name);
if (rest.length === 1 && PROFILE_LISTS.has(rest[0])) {
return userPath(name, rest[0] as ProfileTab);
}
if (rest.length === 2 && rest[0] === "~" && rest[1] in PROFILE_TAB_V4) {
const tab = PROFILE_TAB_V4[rest[1]];
if (tab === "/settings") return tab;
return tab ? userPath(name, tab) : userPath(name);
}
return null;
}
case "~":
if (rest.length > 0) return null;
return key === "followed"
? feedPath(undefined, "follow")
: ["hot", "new", "journal"].includes(key)
? feedPath(undefined, key as "hot")
: null;
case "search":
// `/search/~/users`, split as "search", "~", "users".
return key === "~" && (rest[0] === "users" || rest[0] === "playlists") &&
rest.length === 1
? searchPath(rest[0])
: null;
}
return null;
}
/**
* Rewrites the links to v3 pages of this site in the text column `column` of
* `table`, in every row that has one.
*/
function rewriteLinks(
db: DatabaseSync,
table: string,
column: string,
toV4: (path: string) => string | null,
): void {
const rows = db.prepare(
`SELECT id, ${column} AS text FROM ${table} WHERE ${column} IS NOT NULL;`,
).all() as { id: string; text: string }[];
const update = db.prepare(`UPDATE ${table} SET ${column} = ? WHERE id = ?;`);
for (const { id, text } of rows) {
const rewritten = text.replace(
V3_LINK_RE,
(match, origin: string | undefined, host: string, path: string, at) => {
if (origin ? !isSelfHost(host) : text[at - 1] !== "(") return match;
const v4 = toV4(path);
return v4 ? `${origin ?? ""}${v4}` : match;
},
);
if (rewritten !== text) update.run(rewritten, id);
}
}
/**
* Run once by `runMigrations`. On a database made from schema.sql the tables
* and indexes exist and there is nothing to re-slug or rewrite.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS dump_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
dump_id TEXT NOT NULL REFERENCES dumps(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX IF NOT EXISTS idx_dump_slug_aliases_dump
ON dump_slug_aliases(dump_id, created_at);
CREATE TABLE IF NOT EXISTS playlist_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
playlist_id TEXT NOT NULL REFERENCES playlists(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX IF NOT EXISTS idx_playlist_slug_aliases_playlist
ON playlist_slug_aliases(playlist_id, created_at);
CREATE TABLE IF NOT EXISTS user_aliases (
username TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (username)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_aliases_lower
ON user_aliases(lower(username));
CREATE INDEX IF NOT EXISTS idx_user_aliases_user
ON user_aliases(user_id, created_at);
DROP INDEX IF EXISTS idx_dumps_slug;
DROP INDEX IF EXISTS idx_playlists_slug;
`);
const dumps = reslug(db, "dump");
const playlists = reslug(db, "playlist");
db.exec(`
CREATE UNIQUE INDEX IF NOT EXISTS idx_dumps_user_slug
ON dumps(user_id, slug);
CREATE UNIQUE INDEX IF NOT EXISTS idx_playlists_user_slug
ON playlists(user_id, slug);
`);
renameReservedCategories(db);
const findUser = db.prepare(
`SELECT username FROM users WHERE lower(username) = lower(?);`,
);
const username = (name: string) =>
(findUser.get(name) as { username: string } | undefined)?.username;
const toV4 = (path: string) => v4PathOf(path, dumps, playlists, username);
for (
const [table, column] of [
["dumps", "comment"],
["comments", "body"],
["chat_messages", "body"],
["playlists", "description"],
["users", "description"],
]
) rewriteLinks(db, table, column, toV4);
}

14
api/db/migrations/0022_reserve_community.ts Normal file
View File

@@ -0,0 +1,14 @@
/**
* Reserves `community` at the URL root for the community page, renaming a
* category that had it as its slug.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/** Run once by `runMigrations`. */
export function up(db: DatabaseSync): void {
renameReservedCategories(db);
}

74
api/db/migrations/0023_mention_everyone.ts Normal file
View File

@@ -0,0 +1,74 @@
/**
* Prepares the chat's `@everyone`: it adds `chat_messages.mentions_everyone`,
* and frees the name `everyone`, which no account may have any more.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { EVERYONE, mentionsEveryone } from "../../../shared/mention.ts";
import { ROLE_PERMISSIONS } from "../../../shared/permissions.ts";
/**
* Run once by `runMigrations`. The messages that a moderator or an admin
* already sent with `@everyone` get the flag, so that editing one later
* doesn't notify everyone of an old message; none is notified now. A user
* named `everyone`, in any case, is renamed to the first free `everyone_2`,
* `everyone_3`…, with no alias left behind, and an alias `everyone` is
* dropped. Both are logged.
*/
export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(chat_messages);`).all() as {
name: string;
}[];
if (!cols.some((c) => c.name === "mentions_everyone")) {
db.exec(
`ALTER TABLE chat_messages ADD COLUMN mentions_everyone INTEGER NOT NULL DEFAULT 0;`,
);
}
const roles = Object.entries(ROLE_PERMISSIONS)
.filter(([, perms]) => perms.includes("chat:everyone"))
.map(([role]) => role);
const messages = db.prepare(
`SELECT m.id, m.body FROM chat_messages m JOIN users u ON u.id = m.user_id
WHERE m.mentions_everyone = 0
AND u.role IN (${roles.map(() => "?").join(", ")})
AND m.body LIKE '%@%';`,
).all(...roles) as { id: string; body: string }[];
const flag = db.prepare(
`UPDATE chat_messages SET mentions_everyone = 1 WHERE id = ?;`,
);
for (const m of messages) {
if (mentionsEveryone(m.body)) flag.run(m.id);
}
const taken = (name: string) =>
!!db.prepare(
`SELECT 1 FROM users WHERE lower(username) = lower(?)
UNION ALL SELECT 1 FROM user_aliases WHERE lower(username) = lower(?);`,
).get(name, name);
const user = db.prepare(
`SELECT id, username FROM users WHERE lower(username) = ?;`,
).get(EVERYONE) as { id: string; username: string } | undefined;
if (user) {
let n = 2;
while (taken(`${EVERYONE}_${n}`)) n++;
const renamed = `${EVERYONE}_${n}`;
db.prepare(`UPDATE users SET username = ? WHERE id = ?;`).run(
renamed,
user.id,
);
console.log(
`[migrate] user "${user.username}" is now "${renamed}": @${EVERYONE} is reserved`,
);
}
const alias = db.prepare(
`DELETE FROM user_aliases WHERE lower(username) = ? RETURNING user_id;`,
).get(EVERYONE) as { user_id: string } | undefined;
if (alias) {
console.log(
`[migrate] the former name "${EVERYONE}" of user ${alias.user_id} is dropped: @${EVERYONE} is reserved`,
);
}
}

105
api/db/migrations/0024_federation.ts Normal file
View File

@@ -0,0 +1,105 @@
/**
* Prepares federated sign-in, by which a user signs in here with their
* account on another gerbeur instance, and the other way round. It adds the
* instance's signing key, the link from a local account to the one it stands
* for elsewhere, the instances an admin trusts or blocks, the state of the
* sign-ins in progress on both sides, the grants users gave other instances,
* and the events owed to them.
*
* `/authorize` and `/federation/callback` become pages, so their first
* segments are reserved.
*
* An account from another instance has no password here: its `password_hash`
* is `''`, which no password verifies. Emails stay unique among local accounts
* only, since the same person may also have a local account, or come from two
* instances, with one address.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/**
* Run once by `runMigrations`. Nothing exists to convert, since every account
* so far is local, but `authorize` and `federation` become the pages of a
* sign-in, so a category that had one as its slug is renamed.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS federation_keys (
kid TEXT PRIMARY KEY,
private_jwk TEXT NOT NULL,
public_jwk TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS user_identities (
user_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
issuer TEXT NOT NULL,
subject TEXT NOT NULL,
picture TEXT,
synced_at TEXT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_identities_subject
ON user_identities(issuer, subject);
CREATE TABLE IF NOT EXISTS federation_peers (
issuer TEXT PRIMARY KEY,
status TEXT NOT NULL,
name TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS federation_logins (
state TEXT PRIMARY KEY,
issuer TEXT NOT NULL,
code_verifier TEXT NOT NULL,
nonce TEXT NOT NULL,
invite_token TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS oauth_codes (
code_hash TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
redirect_uri TEXT NOT NULL,
code_challenge TEXT NOT NULL,
nonce TEXT NOT NULL,
expires_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS federation_grants (
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
client_name TEXT,
granted_at TEXT NOT NULL,
last_used_at TEXT NOT NULL,
PRIMARY KEY (user_id, client_id)
);
CREATE TABLE IF NOT EXISTS federation_outbox (
id TEXT PRIMARY KEY,
client_id TEXT NOT NULL,
event TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_federation_outbox_next
ON federation_outbox(next_attempt_at);
`);
const emailIndex = db.prepare(
`SELECT sql FROM sqlite_master WHERE name = 'idx_users_email_lower';`,
).get() as { sql: string } | undefined;
if (!emailIndex?.sql.includes("WHERE")) {
db.exec(`
DROP INDEX IF EXISTS idx_users_email_lower;
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email))
WHERE password_hash != '';
`);
}
renameReservedCategories(db);
}

110
api/db/migrations/0025_playlist_categories.ts Normal file
View File

@@ -0,0 +1,110 @@
/**
* Gives playlists categories of their own, a set apart from the dumps': each
* category gets a `kind`, `dump` or `playlist`, its slug and name now unique
* within its kind only, and `playlist_categories` files playlists under them.
*
* `/playlists` becomes the page of every public playlist, so its first segment
* is reserved.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/**
* Run once by `runMigrations`. Every existing category becomes a dump
* category, filing the same dumps.
*
* SQLite can't change a table's UNIQUE constraints in place, so `categories`
* is rebuilt. `foreign_keys` can't be switched off inside the runner's
* transaction, and dropping `categories` would then empty `dump_categories`
* through its cascade, so both are read, dropped child first, made anew as in
* schema.sql, and refilled. A database made from schema.sql already has the
* `kind` column and is left as it is.
*/
export function up(db: DatabaseSync): void {
const columns = new Set(
(db.prepare(`PRAGMA table_info(categories);`).all() as { name: string }[])
.map((c) => c.name),
);
if (!columns.has("kind")) {
const categories = db.prepare(
`SELECT id, slug, name, position, created_at, updated_at FROM categories;`,
).all() as {
id: string;
slug: string;
name: string;
position: number;
created_at: string;
updated_at: string | null;
}[];
const links = db.prepare(
`SELECT dump_id, category_id FROM dump_categories;`,
).all() as { dump_id: string; category_id: string }[];
db.exec(`DROP TABLE dump_categories;`);
db.exec(`DROP TABLE categories;`);
db.exec(
`CREATE TABLE categories (
id TEXT PRIMARY KEY,
kind TEXT NOT NULL DEFAULT 'dump' CHECK (kind IN ('dump', 'playlist')),
slug TEXT NOT NULL,
name TEXT NOT NULL,
position INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
updated_at TEXT,
UNIQUE (kind, slug),
UNIQUE (kind, name)
);`,
);
db.exec(
`CREATE TABLE dump_categories (
dump_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (dump_id, category_id),
FOREIGN KEY (dump_id) REFERENCES dumps(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);`,
);
db.exec(
`CREATE INDEX idx_dump_categories_category ON dump_categories(category_id);`,
);
const insertCategory = db.prepare(
`INSERT INTO categories (id, kind, slug, name, position, created_at, updated_at)
VALUES (?, 'dump', ?, ?, ?, ?, ?);`,
);
for (const c of categories) {
insertCategory.run(
c.id,
c.slug,
c.name,
c.position,
c.created_at,
c.updated_at,
);
}
const insertLink = db.prepare(
`INSERT INTO dump_categories (dump_id, category_id) VALUES (?, ?);`,
);
for (const l of links) insertLink.run(l.dump_id, l.category_id);
}
db.exec(
`CREATE TABLE IF NOT EXISTS playlist_categories (
playlist_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (playlist_id, category_id),
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);`,
);
db.exec(
`CREATE INDEX IF NOT EXISTS idx_playlist_categories_category ON playlist_categories(category_id);`,
);
renameReservedCategories(db);
}

135
api/db/schema.sql
View File

@@ -69,11 +69,14 @@ CREATE TABLE playlist_dumps (
CREATE TABLE categories (
id TEXT PRIMARY KEY,
slug TEXT NOT NULL UNIQUE,
name TEXT NOT NULL UNIQUE,
kind TEXT NOT NULL DEFAULT 'dump' CHECK (kind IN ('dump', 'playlist')),
slug TEXT NOT NULL,
name TEXT NOT NULL,
position INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
updated_at TEXT
updated_at TEXT,
UNIQUE (kind, slug),
UNIQUE (kind, name)
);
CREATE TABLE dump_categories (
@@ -84,6 +87,14 @@ CREATE TABLE dump_categories (
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);
CREATE TABLE playlist_categories (
playlist_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (playlist_id, category_id),
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);
CREATE TABLE comments (
id TEXT PRIMARY KEY,
dump_id TEXT NOT NULL,
@@ -121,12 +132,12 @@ CREATE TABLE dump_backlinks (
CREATE INDEX idx_dumps_created ON dumps(created_at);
CREATE INDEX idx_dumps_user_created ON dumps(user_id, created_at);
CREATE INDEX idx_dumps_slug ON dumps(slug);
CREATE UNIQUE INDEX idx_dumps_user_slug ON dumps(user_id, slug);
CREATE INDEX idx_dumps_url ON dumps(url);
CREATE INDEX idx_dumps_url_canonical ON dumps(url_canonical);
CREATE INDEX idx_votes_user_created ON votes(user_id, created_at);
CREATE INDEX idx_playlists_user ON playlists(user_id);
CREATE INDEX idx_playlists_slug ON playlists(slug);
CREATE UNIQUE INDEX idx_playlists_user_slug ON playlists(user_id, slug);
CREATE INDEX idx_playlist_dumps_order ON playlist_dumps(playlist_id, position);
CREATE INDEX idx_playlist_dumps_dump ON playlist_dumps(dump_id);
CREATE INDEX idx_comments_dump ON comments(dump_id, created_at);
@@ -136,6 +147,7 @@ CREATE INDEX idx_comment_likes_user ON comment_likes(user_id);
CREATE INDEX idx_dump_backlinks_to ON dump_backlinks(to_dump_id);
CREATE INDEX idx_dump_backlinks_from ON dump_backlinks(from_dump_id);
CREATE INDEX idx_dump_categories_category ON dump_categories(category_id);
CREATE INDEX idx_playlist_categories_category ON playlist_categories(category_id);
CREATE TABLE follows (
id TEXT PRIMARY KEY,
@@ -176,7 +188,10 @@ CREATE TABLE invites (
CREATE INDEX idx_invites_inviter ON invites(inviter_id);
CREATE INDEX idx_users_invited_by ON users(invited_by);
CREATE UNIQUE INDEX idx_users_username_lower ON users(lower(username));
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email));
-- An account from another instance has no password (''), and its email may
-- also be a local account's.
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email))
WHERE password_hash != '';
CREATE TABLE attachments (
id TEXT PRIMARY KEY,
@@ -213,6 +228,51 @@ CREATE UNIQUE INDEX idx_notifications_dedup
ON notifications(user_id, source_key)
WHERE source_key IS NOT NULL;
CREATE TABLE user_settings (
user_id TEXT PRIMARY KEY,
style TEXT,
color_scheme TEXT,
digest_weekly INTEGER NOT NULL DEFAULT 1,
digest_monthly INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE TABLE digest_schedule (
user_id TEXT NOT NULL,
cadence TEXT NOT NULL,
window_start TEXT NOT NULL,
PRIMARY KEY (user_id, cadence),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE TABLE dump_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
dump_id TEXT NOT NULL REFERENCES dumps(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX idx_dump_slug_aliases_dump ON dump_slug_aliases(dump_id, created_at);
CREATE TABLE playlist_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
playlist_id TEXT NOT NULL REFERENCES playlists(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX idx_playlist_slug_aliases_playlist ON playlist_slug_aliases(playlist_id, created_at);
CREATE TABLE user_aliases (
username TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (username)
);
CREATE UNIQUE INDEX idx_user_aliases_lower ON user_aliases(lower(username));
CREATE INDEX idx_user_aliases_user ON user_aliases(user_id, created_at);
CREATE TABLE chat_messages (
id TEXT NOT NULL PRIMARY KEY,
user_id TEXT NOT NULL,
@@ -220,7 +280,70 @@ CREATE TABLE chat_messages (
created_at TEXT NOT NULL,
updated_at TEXT,
reply_to_id TEXT,
mentions_everyone INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE INDEX idx_chat_messages_created_id ON chat_messages(created_at, id);
CREATE INDEX idx_chat_messages_user ON chat_messages(user_id);
CREATE TABLE federation_keys (
kid TEXT PRIMARY KEY,
private_jwk TEXT NOT NULL,
public_jwk TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE user_identities (
user_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
issuer TEXT NOT NULL,
subject TEXT NOT NULL,
picture TEXT,
synced_at TEXT NOT NULL
);
CREATE UNIQUE INDEX idx_user_identities_subject
ON user_identities(issuer, subject);
CREATE TABLE federation_peers (
issuer TEXT PRIMARY KEY,
status TEXT NOT NULL,
name TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE federation_logins (
state TEXT PRIMARY KEY,
issuer TEXT NOT NULL,
code_verifier TEXT NOT NULL,
nonce TEXT NOT NULL,
invite_token TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE oauth_codes (
code_hash TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
redirect_uri TEXT NOT NULL,
code_challenge TEXT NOT NULL,
nonce TEXT NOT NULL,
expires_at TEXT NOT NULL
);
CREATE TABLE federation_grants (
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
client_name TEXT,
granted_at TEXT NOT NULL,
last_used_at TEXT NOT NULL,
PRIMARY KEY (user_id, client_id)
);
CREATE TABLE federation_outbox (
id TEXT PRIMARY KEY,
client_id TEXT NOT NULL,
event TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE INDEX idx_federation_outbox_next ON federation_outbox(next_attempt_at);

12
api/lib/canonical-url.ts
View File

@@ -1,9 +1,9 @@
/**
* Spots reposts and links to known dumps, by reducing a URL to a key shared by
* every link to the same page or video. `dump-service` stores it in
* `dumps.url_canonical` when a dump is posted or edited, `GET
* /api/dumps/by-url` matches on it to warn the create form of a repost, and
* backlink-service compares links by it. The key is never fetched or shown, so
* `dumps.url_canonical` when a dump is posted or edited, `GET /api/dumps?url=`
* matches on it to warn the create form of a repost, and backlink-service
* compares links by it. The key is never fetched or shown, so
* it may be lossy: it forces `https`, drops `www.`, tracking parameters and
* timestamps.
*
@@ -13,6 +13,8 @@
* @module
*/
import { siteHostOf } from "../../shared/url.ts";
const TRACKING_PARAMS = new Set([
"fbclid",
"gclid",
@@ -52,7 +54,7 @@ const YOUTUBE_HOSTS = new Set([
* uses it too, so both agree on what a YouTube link is.
*/
export function youtubeHost(u: URL): string | null {
const host = u.hostname.toLowerCase().replace(/^www\./, "");
const host = siteHostOf(u);
return YOUTUBE_HOSTS.has(host) ? host : null;
}
@@ -90,7 +92,7 @@ export function canonicalizeUrl(raw: string): string | null {
}
if (u.protocol !== "http:" && u.protocol !== "https:") return null;
const host = u.hostname.toLowerCase().replace(/^www\./, "");
const host = siteHostOf(u);
if (!host) return null;
const videoId = youtubeVideoId(host, u.pathname, u.searchParams);

91
api/lib/css-color.ts Normal file
View File

@@ -0,0 +1,91 @@
/**
* Reads the CSS colors the server meets: a dumped page's `theme-color`, which
* the generic link preview keeps as its tint (`normalizeCssColor`), and the
* theme stylesheets' tokens, which the emails resolve (`theme-tokens.ts`).
*
* @module
*/
/** An sRGB color with its alpha, each channel 0–255 and alpha 0–1. */
export interface Rgba {
r: number;
g: number;
b: number;
a: number;
}
/**
* The named colors sites actually put in `theme-color`, and `transparent`,
* which the themes write; the rest of CSS's names aren't worth carrying.
*/
const NAMED: Record<string, string> = {
transparent: "#00000000",
black: "#000000",
white: "#ffffff",
red: "#ff0000",
green: "#008000",
blue: "#0000ff",
yellow: "#ffff00",
orange: "#ffa500",
purple: "#800080",
gray: "#808080",
grey: "#808080",
silver: "#c0c0c0",
maroon: "#800000",
navy: "#000080",
teal: "#008080",
olive: "#808000",
lime: "#00ff00",
aqua: "#00ffff",
cyan: "#00ffff",
fuchsia: "#ff00ff",
magenta: "#ff00ff",
};
/** Reads a channel of `rgb()`, a number or a percentage of 255. */
function channel(text: string): number {
const n = parseFloat(text);
const value = text.endsWith("%") ? (n / 100) * 255 : n;
return Math.max(0, Math.min(255, value));
}
/** Reads the alpha of `rgba()`, a number or a percentage of 1. */
function alpha(text: string): number {
const n = parseFloat(text);
return Math.max(0, Math.min(1, text.endsWith("%") ? n / 100 : n));
}
/**
* Returns the color `text` writes as a hex (3, 4, 6 or 8 digits), an `rgb()`
* or `rgba()` of numbers or percentages, or one of the common names; undefined
* for any other form. Channels out of range are clamped.
*/
export function parseCssColor(text: string): Rgba | undefined {
const value = text.trim().toLowerCase();
const hex = /^#([0-9a-f]{3,8})$/.exec(NAMED[value] ?? value)?.[1];
if (hex) {
if (hex.length === 5 || hex.length === 7) return undefined;
const full = hex.length <= 4 ? [...hex].map((c) => c + c).join("") : hex;
const byte = (i: number) => parseInt(full.slice(i * 2, i * 2 + 2), 16);
return {
r: byte(0),
g: byte(1),
b: byte(2),
a: full.length === 8 ? byte(3) / 255 : 1,
};
}
const args = /^rgba?\(([^)]+)\)$/.exec(value)?.[1];
if (!args) return undefined;
const parts = args.split(/[\s,/]+/).filter(Boolean);
if (parts.length !== 3 && parts.length !== 4) return undefined;
const [r, g, b] = parts.slice(0, 3).map(channel);
const a = parts[3] === undefined ? 1 : alpha(parts[3]);
return [r, g, b, a].some(Number.isNaN) ? undefined : { r, g, b, a };
}
/** Writes `color` as a lowercase `#rrggbb`, its alpha dropped. */
export function hexOf({ r, g, b }: Rgba): string {
return `#${
[r, g, b].map((c) => Math.round(c).toString(16).padStart(2, "0")).join("")
}`;
}

46
api/lib/dump-media.ts Normal file
View File

@@ -0,0 +1,46 @@
/**
* Resolves the dump whose media a route under `/api/dumps/:dumpId/` serves:
* its file (`routes/files.ts`), its thumbnail (`routes/thumbnails.ts`) or its
* waveform (`routes/waveforms.ts`). Media elements and `<img>`s can't send an
* Authorization header, so these routes take the viewer's media token from
* the query string, and are cached by the file's version.
*
* @module
*/
import type { Context } from "@oak/oak";
import { APIException } from "./api-exception.ts";
import { authenticateMedia } from "./auth.ts";
import { type Dump, dumpFileVersion } from "../../shared/dump.ts";
import { UUID_RE } from "../../shared/validation.ts";
import { getDump } from "../services/dump-service.ts";
/**
* Returns dump `dumpId` as the viewer of `ctx`'s `?token=` may see it, and the
* Cache-Control its media get: kept for good under a URL naming the file's
* current version (`?v=`, as `dumpFileVersion` gives), revalidated under any
* other, and never in a shared cache for a private dump. It also keeps the
* token out of the Referer of whatever the response links to.
*
* @throws {APIException} 400 when `dumpId` isn't an id, which names files on
* disk, and 404 when the viewer can't see the dump.
*/
export async function resolveMediaDump(
ctx: Context,
dumpId: string,
): Promise<{ dump: Dump; cacheControl: string }> {
if (!UUID_RE.test(dumpId)) throw new APIException(400, "Invalid dump ID");
const token = ctx.request.url.searchParams.get("token");
const viewerId = token ? await authenticateMedia(token) : null;
const dump = getDump(dumpId, viewerId ?? undefined);
ctx.response.headers.set("Referrer-Policy", "no-referrer");
const scope = dump.isPrivate ? "private" : "public";
const current = ctx.request.url.searchParams.get("v") ===
dumpFileVersion(dump);
return {
dump,
cacheControl: current
? `${scope}, max-age=31536000, immutable`
: `${scope}, no-cache`,
};
}

288
api/lib/ffmpeg.ts Normal file
View File

@@ -0,0 +1,288 @@
/**
* Makes what the server derives from a file dump's upload with ffmpeg: a
* video's still and an image's scaled-down copy, which the thumbnails route
* serves, and an audio file's waveform, which the waveform route serves. Each
* is made on its first request and kept in `THUMBNAILS_DIR` under a name this
* module gives, until dump-service replaces or deletes the upload and
* {@link removeGeneratedMedia} drops them.
*
* @module
*/
import { THUMBNAILS_DIR } from "../config.ts";
import { type DumpWaveform, peaksOf } from "../../shared/waveform.ts";
import type { APIResponse } from "../../shared/api.ts";
/**
* Spares a process per uncached request: whether ffmpeg runs is probed once
* per process, and kept here.
*/
let ffmpegProbe: Promise<boolean> | undefined;
function ffmpegAvailable(): Promise<boolean> {
ffmpegProbe ??= (async () => {
try {
const { success } = await new Deno.Command("ffmpeg", {
args: ["-version"],
stderr: "null",
stdout: "null",
}).output();
return success;
} catch {
return false;
}
})();
return ffmpegProbe;
}
/**
* Lists the container formats ffmpeg may read a video as. The upload's type is
* only what the uploader claimed, and ffmpeg guesses the format from the bytes,
* so a "video" could be an HLS or concat playlist pointing it at local files or
* internal URLs. With this list, and `-protocol_whitelist file`, ffmpeg reads
* nothing but the file, and only as a video.
*/
const VIDEO_FORMATS =
"mov,mp4,m4a,3gp,3g2,mj2,matroska,webm,avi,flv,mpegts,mpeg,ogg,asf,h264,hevc";
/**
* Lists the formats ffmpeg may read an image as, for the reason given at
* {@link VIDEO_FORMATS}.
*/
const IMAGE_FORMATS = "jpeg_pipe,png_pipe,webp_pipe,bmp_pipe";
/**
* Lists the formats ffmpeg may read an audio file as, for the reason given at
* {@link VIDEO_FORMATS}; an `.m4a` or a `.webm` is read by its container's.
*/
const AUDIO_FORMATS =
"mp3,ogg,flac,wav,aiff,aac,asf,caf,w64,wv,ape,mov,mp4,m4a,matroska,webm";
/**
* Maps the image types that get a scaled-down preview to the preview's format.
* A GIF is served as it is, to keep its animation.
*/
export const PREVIEWABLE_IMAGES: Record<string, "jpg" | "png"> = {
"image/jpeg": "jpg",
"image/bmp": "jpg",
// May carry transparency, which JPEG would fill with black.
"image/png": "png",
"image/webp": "png",
};
/**
* Caps an image preview's width. It covers the widest feed card on a
* high-density screen, at a fraction of a phone photo's bytes.
*/
const PREVIEW_WIDTH = 960;
/** Returns where the media generated from dump `dumpId`'s upload are kept. */
export function generatedPaths(dumpId: string): {
still: string;
preview: Record<"jpg" | "png", string>;
waveform: string;
} {
const base = `${THUMBNAILS_DIR}/${dumpId}`;
return {
still: `${base}.jpg`,
preview: { jpg: `${base}.preview.jpg`, png: `${base}.preview.png` },
waveform: `${base}.waveform.json`,
};
}
/**
* Deletes what was generated from dump `dumpId`'s upload, once dump-service
* has replaced or deleted it. The routes make them again from a new one.
*/
export async function removeGeneratedMedia(dumpId: string): Promise<void> {
const { still, preview, waveform } = generatedPaths(dumpId);
await Promise.all(
[still, preview.jpg, preview.png, waveform].map((path) =>
Deno.remove(path).catch(() => {})
),
);
}
/**
* Writes `outPath` with `write`, which gets a path beside it to fill, renamed
* over `outPath` once done, so a concurrent reader never gets half a file.
*/
async function writeAtomically(
outPath: string,
write: (tmpPath: string) => Promise<void>,
): Promise<void> {
await Deno.mkdir(THUMBNAILS_DIR, { recursive: true });
const tmpPath = `${outPath}.${crypto.randomUUID()}.tmp`;
try {
await write(tmpPath);
await Deno.rename(tmpPath, outPath);
} catch (err) {
await Deno.remove(tmpPath).catch(() => {});
throw err;
}
}
/**
* Runs ffmpeg with `args`, writing to `outPath`.
*
* @throws {Error} When ffmpeg fails.
*/
function runFfmpeg(args: string[], outPath: string): Promise<void> {
return writeAtomically(outPath, async (tmpPath) => {
const { success } = await new Deno.Command("ffmpeg", {
args: ["-protocol_whitelist", "file", ...args, "-y", tmpPath],
stdout: "null",
stderr: "null",
}).output();
if (!success) throw new Error("ffmpeg failed");
});
}
/** Writes a 320-pixel-wide JPEG still of a video, one second in. */
export function videoStill(srcPath: string, outPath: string): Promise<void> {
return runFfmpeg([
"-format_whitelist",
VIDEO_FORMATS,
"-ss",
"00:00:01",
"-i",
srcPath,
"-frames:v",
"1",
"-vf",
"scale=320:-1",
"-f",
"image2",
"-c:v",
"mjpeg",
], outPath);
}
/** Writes a copy of an image at most {@link PREVIEW_WIDTH} wide. */
export function imagePreview(srcPath: string, outPath: string): Promise<void> {
return runFfmpeg([
"-format_whitelist",
IMAGE_FORMATS,
"-i",
srcPath,
"-frames:v",
"1",
"-vf",
`scale='min(${PREVIEW_WIDTH},iw)':-2`,
"-f",
"image2",
...(outPath.endsWith(".png")
? ["-c:v", "png"]
: ["-c:v", "mjpeg", "-q:v", "4"]),
], outPath);
}
/**
* The sample rate the audio is decoded at for its waveform. A bar stands for
* a second or more of a song, so 8 kHz keeps every peak worth drawing at a
* sixth of the samples.
*/
const PEAKS_SAMPLE_RATE = 8000;
/** How many samples each running peak covers before `peaksOf` buckets them. */
const PEAKS_WINDOW = 256;
/**
* Decodes the audio of `srcPath` to its waveform's peaks. ffmpeg streams the
* samples, mono at {@link PEAKS_SAMPLE_RATE}, and only the peak of each
* {@link PEAKS_WINDOW} is kept, so an hour of audio holds a few hundred
* kilobytes at most.
*
* @throws {Error} When ffmpeg fails or the file holds no audio.
*/
async function audioPeaks(srcPath: string): Promise<number[]> {
const child = new Deno.Command("ffmpeg", {
args: [
"-protocol_whitelist",
"file",
"-format_whitelist",
AUDIO_FORMATS,
"-i",
srcPath,
"-vn",
"-ac",
"1",
"-ar",
String(PEAKS_SAMPLE_RATE),
"-f",
"s16le",
"-",
],
stdin: "null",
stdout: "piped",
stderr: "null",
}).spawn();
const windows: number[] = [];
let peak = 0;
let count = 0;
// A chunk may end halfway through a sample, whose first byte waits here.
let pending: number | null = null;
const take = (sample: number) => {
peak = Math.max(peak, Math.abs(sample) / 32768);
if (++count === PEAKS_WINDOW) {
windows.push(peak);
peak = 0;
count = 0;
}
};
for await (const chunk of child.stdout) {
let i = 0;
if (pending !== null) {
take(((chunk[0] << 24) >> 16) | pending);
pending = null;
i = 1;
}
for (; i + 1 < chunk.length; i += 2) {
// Little-endian, signed: the high byte is shifted into the sign bit
// and back.
take(((chunk[i + 1] << 24) >> 16) | chunk[i]);
}
if (i < chunk.length) pending = chunk[i];
}
if (count > 0) windows.push(peak);
const { success } = await child.status;
if (!success || windows.length === 0) throw new Error("ffmpeg failed");
return peaksOf(windows);
}
/**
* Writes the waveform of the audio file `srcPath` as the waveform route's
* whole response, so the route serves it as a file, with its ETag.
*/
export function audioWaveform(srcPath: string, outPath: string): Promise<void> {
return writeAtomically(outPath, async (tmpPath) => {
const body: APIResponse<DumpWaveform> = {
success: true,
data: { peaks: await audioPeaks(srcPath) },
};
await Deno.writeTextFile(tmpPath, JSON.stringify(body));
});
}
/**
* Lets a feed full of cards asking for the same new still at once wait on a
* single ffmpeg run. It holds the generations under way, by output path.
*/
const inFlight = new Map<string, Promise<void>>();
/**
* Makes `outPath` with `generate` unless it exists. Resolves false when it
* can't be made, because ffmpeg is missing or failed.
*/
export async function ensureGenerated(
outPath: string,
generate: () => Promise<void>,
): Promise<boolean> {
if (await Deno.stat(outPath).then(() => true, () => false)) return true;
if (!await ffmpegAvailable()) return false;
let pending = inFlight.get(outPath);
if (!pending) {
pending = generate().finally(() => inFlight.delete(outPath));
inFlight.set(outPath, pending);
}
return await pending.then(() => true, () => false);
}

20
api/lib/index-html.ts
View File

@@ -6,7 +6,14 @@
* @module
*/
import { BANDCAMP_PLAYER, OG_SITE_NAME, SITE_EMOJI } from "../config.ts";
import {
BANDCAMP_PLAYER,
DEFAULT_STYLE,
EMAIL_ENABLED,
FEDERATION,
OG_SITE_NAME,
SITE_EMOJI,
} from "../config.ts";
import { emojiToCodepoint } from "./site-icons.ts";
/**
@@ -21,8 +28,9 @@ const ICON_VERSION = emojiToCodepoint(SITE_EMOJI);
const indexHtml = new Map<string, { mtime: number; html: string }>();
/**
* Returns `${root}/index.html` with the site name, emoji, icon version and
* Bandcamp player mode filled in.
* Returns `${root}/index.html` with the site name, emoji, icon version,
* Bandcamp player mode, default style, and whether email and each way of
* federated sign-in are on filled in.
*
* @throws {Error} When the file can't be read.
*/
@@ -35,7 +43,11 @@ export async function loadIndexHtml(root: string): Promise<string> {
.replaceAll("__SITE_NAME__", OG_SITE_NAME)
.replaceAll("__SITE_EMOJI__", SITE_EMOJI)
.replaceAll("__ICON_VERSION__", ICON_VERSION)
.replaceAll("__BANDCAMP_PLAYER__", BANDCAMP_PLAYER);
.replaceAll("__BANDCAMP_PLAYER__", BANDCAMP_PLAYER)
.replaceAll("__DEFAULT_STYLE__", DEFAULT_STYLE)
.replaceAll("__EMAIL_ENABLED__", EMAIL_ENABLED ? "on" : "off")
.replaceAll("__FEDERATION_INCOMING__", FEDERATION.incoming ? "on" : "off")
.replaceAll("__FEDERATION_OUTGOING__", FEDERATION.outgoing ? "on" : "off");
indexHtml.set(root, { mtime, html });
return html;
}

179
api/lib/jwt.ts
View File

@@ -1,15 +1,30 @@
/**
* Signs and verifies the server's three kinds of token (session, media and
* password reset), all HS256 with the one key. Only a token's signature and
* Signs and verifies the server's four kinds of token (session, media,
* password reset and unsubscribe), all HS256 with the one key. Only a token's signature and
* expiry are checked here: `lib/auth.ts` checks a session or media token
* against its user.
*
* It also signs and verifies the tokens instances exchange for federated
* sign-in (identity tokens and events), EdDSA with the instance's Ed25519 key,
* whose public half other instances fetch: `instance-key-service.ts` keeps
* the key, and the federation services pass it in.
*
* Importing this module throws when `GERBEUR_JWT_SECRET` is unset.
*
* @module
*/
import { jwtVerify, SignJWT } from "@panva/jose";
import {
createLocalJWKSet,
exportJWK,
generateKeyPair,
importJWK,
type JSONWebKeySet,
type JWK,
type JWTPayload,
jwtVerify,
SignJWT,
} from "@panva/jose";
import {
type AuthPayload,
@@ -18,6 +33,8 @@ import {
type PasswordResetPayload,
} from "../model/auth.ts";
import { JWT_SECRET } from "../config.ts";
import { type DigestCadence, isDigestCadence } from "../../shared/settings.ts";
import type { AuthTokens, Role } from "../../shared/user.ts";
if (!JWT_SECRET) {
throw new Error(
@@ -60,9 +77,9 @@ export async function verifyPasswordResetToken(
}
/**
* Signs a session token carrying `payload`, valid for 7 days. The users routes
* issue it on sign-up, sign-in and password change, and the client sends it as
* a bearer token.
* Signs a session token carrying `payload`, valid for 7 days.
* {@link issueTokens} issues it on sign-up, sign-in and password change, and
* the client sends it as a bearer token.
*/
export async function createJWT(
payload: Omit<AuthPayload, "exp">,
@@ -76,6 +93,24 @@ export async function createJWT(
.sign(JWT_KEY);
}
/**
* Returns a new session token and media token for `user`, as sign-up
* (`POST /api/users`), sign-in (`POST /api/sessions`) and a password change
* (`PUT /api/me/password`) answer them.
*/
export async function issueTokens(
user: { id: string; username: string; role: Role },
): Promise<AuthTokens> {
return {
token: await createJWT({
userId: user.id,
username: user.username,
role: user.role,
}),
mediaToken: await createMediaToken(user.id),
};
}
/**
* Signs a media token for `userId`, valid for 7 days. Media elements can't send
* an `Authorization` header, so the URLs of private files, thumbnails and
@@ -128,3 +163,135 @@ export async function verifyJWT(token: string): Promise<AuthPayload | null> {
return null;
}
}
/**
* Signs the token of a digest's unsubscribe link, which turns off `cadence`
* for `userId` without signing in. It never expires, so the link in an old
* email still works; turning the digest back on doesn't revoke it, which is
* harmless since all it can do is turn it off again.
*/
export async function createUnsubscribeToken(
userId: string,
cadence: DigestCadence,
): Promise<string> {
return await new SignJWT({ purpose: "unsubscribe", userId, cadence })
.setProtectedHeader({ alg: "HS256" })
.sign(JWT_KEY);
}
/**
* Returns the user and cadence of an unsubscribe token, or null when it is
* invalid or another kind of token.
*/
export async function verifyUnsubscribeToken(
token: string,
): Promise<{ userId: string; cadence: DigestCadence } | null> {
try {
const { payload } = await jwtVerify(token, JWT_KEY);
if (
payload.purpose !== "unsubscribe" || typeof payload.userId !== "string" ||
!isDigestCadence(payload.cadence)
) {
return null;
}
return { userId: payload.userId, cadence: payload.cadence };
} catch {
return null;
}
}
/**
* The algorithm of the tokens instances exchange: Ed25519 signatures, short
* and fast, with no parameters to get wrong.
*/
const INSTANCE_ALG = "EdDSA";
/**
* An instance's key, ready to sign, from {@link importInstanceKey}. Its
* `kid` names the public half in the instance's JWKS.
*/
export interface InstanceSigningKey {
kid: string;
privateKey: CryptoKey;
}
/**
* Returns a new Ed25519 key pair as JWKs, with a random `kid`, for
* `instance-key-service.ts` to store when the instance has none. The public
* JWK carries the `kid`, `alg` and `use` a JWKS entry needs.
*/
export async function generateInstanceKey(): Promise<
{ kid: string; privateJwk: JWK; publicJwk: JWK }
> {
const { privateKey, publicKey } = await generateKeyPair(INSTANCE_ALG, {
crv: "Ed25519",
extractable: true,
});
const kid = crypto.randomUUID();
return {
kid,
privateJwk: await exportJWK(privateKey),
publicJwk: {
...await exportJWK(publicKey),
kid,
alg: INSTANCE_ALG,
use: "sig",
},
};
}
/** Returns the stored private JWK `privateJwk` as a key to sign with. */
export async function importInstanceKey(
kid: string,
privateJwk: JWK,
): Promise<InstanceSigningKey> {
const privateKey = await importJWK(privateJwk, INSTANCE_ALG);
if (!(privateKey instanceof CryptoKey)) {
throw new Error("The instance key is not an asymmetric key");
}
return { kid, privateKey };
}
/**
* Signs `payload` with the instance's key, from `issuer` (this instance) to
* `audience` (the instance it is for), valid for `expiresIn`, with a random
* `jti`. The identity tokens of `POST /api/oauth/tokens` and the events of
* the outbox are signed with it.
*/
export async function signInstanceToken(
payload: JWTPayload,
key: InstanceSigningKey,
claims: { issuer: string; audience: string; subject: string },
expiresIn: string,
): Promise<string> {
return await new SignJWT(payload)
.setProtectedHeader({ alg: INSTANCE_ALG, kid: key.kid })
.setIssuer(claims.issuer)
.setAudience(claims.audience)
.setSubject(claims.subject)
.setIssuedAt()
.setJti(crypto.randomUUID())
.setExpirationTime(expiresIn)
.sign(key.privateKey);
}
/**
* Returns the payload of `token` once its signature checks against `jwks`,
* the keys of `issuer`, and it is addressed to `audience` and unexpired.
*
* @throws {Error} jose's error when it doesn't verify; its `code` is
* `ERR_JWKS_NO_MATCHING_KEY` when no key of `jwks` has its `kid`, which
* tells the caller to refetch the keys, the issuer having rotated them.
*/
export async function verifyInstanceToken(
token: string,
jwks: JSONWebKeySet,
claims: { issuer: string; audience: string },
): Promise<JWTPayload> {
const { payload } = await jwtVerify(token, createLocalJWKSet(jwks), {
issuer: claims.issuer,
audience: claims.audience,
algorithms: [INSTANCE_ALG],
});
return payload;
}

17
api/lib/rate-limit.ts
View File

@@ -6,6 +6,8 @@
* @module
*/
import type { Job } from "./scheduler.ts";
interface Bucket {
tokens: number;
at: number;
@@ -42,11 +44,18 @@ export function takeToken(key: string, limit: RateLimit): boolean {
return true;
}
// Every limit in use refills within the hour, so an older bucket is full and
// dropping it changes nothing.
setInterval(() => {
/**
* Drops the buckets untouched for an hour, every minute. Every limit in use
* refills within the hour, so such a bucket is full and dropping it changes
* nothing. `main.ts` starts it.
*/
export const rateLimitSweepJob: Job = {
name: "rate-limit",
everyMs: 60_000,
run() {
const cutoff = Date.now() - 60 * 60_000;
for (const [key, bucket] of buckets) {
if (bucket.at < cutoff) buckets.delete(key);
}
}, 60_000);
},
};

77
api/lib/reserved-slugs.ts
View File

@@ -1,47 +1,78 @@
/**
* Checks which slugs a category may take. A category's feed lives at the URL
* root (`/music`), so its slug must not shadow another top-level route. The
* category service runs these checks when a category is created or renamed.
* Checks which slugs a dump category may take. A dump category's feed lives at
* the URL root (`/music`), so its slug must not shadow another top-level page.
* The category service runs this check when a dump category is created or
* renamed, and the migrations that reserve more words rename the categories
* that had one with {@link renameReservedCategories}. A playlist category's
* page is `/playlists/<slug>`, under a reserved word, so it reserves nothing.
*
* Dump and playlist slugs reserve nothing: they live under their owner's
* `dumps/` and `playlists/`, where nothing else is named by a user. A user
* can't clash with a category either, since `@` is never in a slug.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { FEED_TABS } from "../../shared/paths.ts";
/**
* Lists the top-level path segments a category slug may not take. They are
* the SPA's own routes (`src/App.tsx`); `api`, `ws`, `assets` and
* `emojibase`, which the server answers before the SPA; and `manage`, a
* former route whose old links may still be about. Keep it in step with the
* route table and the build's top-level folders.
* Lists the top-level path segments a category slug may not take: the feed
* tabs (`/hot`), the SPA's own pages (`src/App.tsx`), and `api`, `ws`,
* `assets` and `emojibase`, which the server answers before the SPA. Keep it
* in step with the route table and the build's top-level folders.
*/
export const RESERVED_SLUGS: readonly string[] = [
"~",
"dumps",
"users",
"playlists",
...FEED_TABS,
"search",
"login",
"register",
"notifications",
"reset-password",
"unsubscribe",
"settings",
"manage",
"community",
"playlists",
"authorize",
"federation",
"api",
"ws",
"assets",
"emojibase",
];
/**
* Matches a well-formed category slug, made of lowercase letters and digits
* joined by single hyphens.
*/
export const CATEGORY_SLUG_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
/** Checks whether `slug` matches {@link CATEGORY_SLUG_RE}. */
export function isValidSlugFormat(slug: string): boolean {
return CATEGORY_SLUG_RE.test(slug);
}
/** Checks whether `slug` is one of the {@link RESERVED_SLUGS}. */
export function isReservedSlug(slug: string): boolean {
return RESERVED_SLUGS.includes(slug);
}
/**
* Renames each category whose slug is now reserved, such as `hot`, to the
* first of `<slug>-2`, `<slug>-3`… that is free, and logs it. Migration 0021
* runs it for v4's words, 0022 for `community`, 0024 for `authorize` and
* `federation`, and 0025 for `playlists`.
*
* It reads every category: until 0025 gave categories a `kind`, each was a
* dump category, and 0025 makes no other. A later migration that reserves
* more words must leave the playlist categories alone, and limit it to
* `kind = 'dump'`, which the earlier ones can't, having no such column.
*/
export function renameReservedCategories(db: DatabaseSync): void {
const rows = db.prepare(`SELECT id, slug FROM categories;`).all() as {
id: string;
slug: string;
}[];
const taken = new Set(rows.map((r) => r.slug));
for (const { id, slug } of rows) {
if (!isReservedSlug(slug)) continue;
let n = 2;
while (taken.has(`${slug}-${n}`) || isReservedSlug(`${slug}-${n}`)) n++;
const renamed = `${slug}-${n}`;
taken.add(renamed);
db.prepare(`UPDATE categories SET slug = ? WHERE id = ?;`).run(renamed, id);
console.log(
`[migrate] category "${slug}" is now "${renamed}": its slug is reserved`,
);
}
}

46
api/lib/scheduler.ts Normal file
View File

@@ -0,0 +1,46 @@
/**
* Runs the server's recurring tasks. Each module that needs one exports a
* {@link Job}, and `main.ts` passes them all to {@link startJobs} when it runs
* directly, so nothing recurs in the tests, which call a job's `run` when
* they need it.
*
* @module
*/
/** A recurring task, exported by its module for `main.ts` to start. */
export interface Job {
/** Names it in the log, as `[digest]`. */
name: string;
/** How long from one run's start to the next. */
everyMs: number;
/** How long after the start the first run waits, `everyMs` by default. */
firstAfterMs?: number;
/** Logs what it did itself, if anything; its errors are logged here. */
run(): void | Promise<void>;
}
/**
* Starts running each of `jobs` on its own timer. A run still going when the
* next is due makes that one skip, and a run that throws or rejects is logged
* and the next one runs anyway.
*/
export function startJobs(jobs: readonly Job[]): void {
for (const job of jobs) {
let running = false;
const tick = async () => {
if (running) return;
running = true;
try {
await job.run();
} catch (err) {
console.error(`[${job.name}] run failed:`, err);
} finally {
running = false;
}
};
setTimeout(() => {
tick();
setInterval(tick, job.everyMs);
}, job.firstAfterMs ?? job.everyMs);
}
}

24
api/lib/self-url.ts
View File

@@ -1,12 +1,14 @@
/**
* Recognises links to this site's own pages. The self provider previews them
* from the database, and remote/fetch.ts accepts them as dump links even on a
* private host such as localhost.
* from the database, remote/fetch.ts accepts them as dump links even on a
* private host such as localhost, and migration 0021 rewrote the v3 ones in
* stored text.
*
* @module
*/
import { FRONTEND_URL, PUBLIC_URL } from "../config.ts";
import { parsePagePath } from "../../shared/paths.ts";
const SELF_HOSTS = new Set(
[PUBLIC_URL, FRONTEND_URL].flatMap((u) => {
@@ -18,14 +20,22 @@ const SELF_HOSTS = new Set(
}),
);
const SELF_PATH_RE = /^\/(dumps|users|playlists)\/[^/]+$/;
/**
* Checks whether `host` (with its port, as `URL.host` gives it) is where this
* site is served, at its public URL or its frontend's.
*/
export function isSelfHost(host: string): boolean {
return SELF_HOSTS.has(host.toLowerCase());
}
/**
* Checks whether `url` links to one of this site's own dump, user or playlist
* pages. Matched on the host as well as the path: `/users/bob` on some other
* site is someone else's page, not ours.
* pages, by `parsePagePath`; an editor doesn't count. Matched on the
* host as well as the path: `/@bob` on some other site is someone else's
* page, not ours.
*/
export function isSelfUrl(url: URL): boolean {
return SELF_HOSTS.has(url.host.toLowerCase()) &&
SELF_PATH_RE.test(url.pathname);
if (!isSelfHost(url.host)) return false;
const page = parsePagePath(url.pathname);
return !!page && !(page.kind !== "user" && page.edit);
}

98
api/lib/slugify.ts
View File

@@ -1,38 +1,86 @@
/**
* Builds the URL slugs of dumps and playlists from their titles. The dump and
* playlist services store the slug when the record is created or retitled.
* Picks and checks the slugs of dumps and playlists, which name them in their
* page URLs under their owner: `/@bob/dumps/<slug>`, `/@bob/playlists/<slug>`.
* Each kind has its own slug space per owner, shared by the items' current
* slugs and their aliases (the slugs they had before). `services/slug-service.ts`
* changes and resolves slugs with these, and migration 0021 gave every item
* one; both pass the database, since this module doesn't open it.
*
* @module
*/
/**
* Matches a UUID. A dump or playlist route takes either an id or a slug, and
* the services test the parameter with this to know which column to look up.
*/
export const UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
import type { DatabaseSync } from "node:sqlite";
import { APIException } from "./api-exception.ts";
import { isValidSlug, SLUG_MAX } from "../../shared/slug.ts";
/** A kind of item that has a slug under its owner. */
export type SlugKind = "dump" | "playlist";
/** Where each {@link SlugKind}'s slugs live. */
export const SLUG_TABLES: Record<
SlugKind,
{ items: string; aliases: string; itemId: string }
> = {
dump: { items: "dumps", aliases: "dump_slug_aliases", itemId: "dump_id" },
playlist: {
items: "playlists",
aliases: "playlist_slug_aliases",
itemId: "playlist_id",
},
};
/**
* Turns `title` into lowercase ASCII words joined by dashes, at most 60
* characters long. Accents are dropped, and a title with no ASCII letter or
* digit gives `untitled`.
* Returns the id of the item whose current slug or alias is `slug` in
* `userId`'s space for `kind`, or undefined when it is free.
*/
export function slugify(title: string): string {
const slug = title
.toLowerCase()
.normalize("NFD")
// NFD splits an accented letter into the letter and a combining mark.
.replace(/[\u0300-\u036f]/g, "")
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.substring(0, 60);
return slug || "untitled";
export function slugHolder(
db: DatabaseSync,
kind: SlugKind,
userId: string,
slug: string,
): string | undefined {
const { items, aliases, itemId } = SLUG_TABLES[kind];
const row = db.prepare(
`SELECT id FROM ${items} WHERE user_id = ? AND slug = ?
UNION ALL
SELECT ${itemId} FROM ${aliases} WHERE user_id = ? AND slug = ?
LIMIT 1;`,
).get(userId, slug, userId, slug) as { id: string } | undefined;
return row?.id;
}
/**
* Returns a record's slug, its title's {@link slugify} followed by the first 8
* characters of `id`. The suffix keeps two records with the same title apart.
* Returns `base`, or `base-2`, `base-3`… cut to fit, whichever is free in
* `userId`'s space for `kind`. `base` must be a valid slug, as `slugify`
* makes. The services take it on creating an item, the migration on
* re-slugging one, inside the transaction that writes it.
*/
export function makeSlug(title: string, id: string): string {
return `${slugify(title)}-${id.substring(0, 8)}`;
export function uniqueSlug(
db: DatabaseSync,
kind: SlugKind,
userId: string,
base: string,
): string {
for (let n = 1;; n++) {
const suffix = n === 1 ? "" : `-${n}`;
const candidate = `${
base.substring(0, SLUG_MAX - suffix.length).replace(/-+$/, "")
}${suffix}`;
if (!slugHolder(db, kind, userId, candidate)) return candidate;
}
}
/**
* Throws a 400 unless `slug` is a valid slug, lowercase words of letters and
* digits joined by hyphens, at most `max` long. slug-service checks a dump's
* or a playlist's with it, and category-service a category's, with that
* kind's own limit.
*/
export function validateSlug(slug: string, max: number = SLUG_MAX): void {
if (!isValidSlug(slug, max)) {
throw new APIException(
400,
`Slug must be lowercase letters, numbers and hyphens, at most ${max} characters`,
);
}
}

60
api/lib/ttl-cache.ts Normal file
View File

@@ -0,0 +1,60 @@
/**
* Keeps what the server fetched from another site for a while, by key, so a
* page or a feed asking again doesn't refetch it: Bandcamp tracklists, YouTube
* playlists, and link previews. Each holder sets how long an entry lives and
* how many are kept; past that count, the oldest written goes first.
*
* @module
*/
/**
* A map whose entries expire `ttlMs` after they are written, holding
* `maxEntries` at most.
*
* @example
* const playlists = new TtlCache<YouTubePlaylist>(6 * 3_600_000, 200);
* const hit = playlists.get(listId) ?? playlists.set(listId, await fetch());
*/
export class TtlCache<V> {
#entries = new Map<string, { value: V; at: number }>();
constructor(readonly ttlMs: number, readonly maxEntries: number) {}
/** Returns the value under `key` unless it is missing or expired. */
get(key: string): V | undefined {
const hit = this.#entries.get(key);
if (!hit) return undefined;
if (Date.now() - hit.at >= this.ttlMs) {
this.#entries.delete(key);
return undefined;
}
return hit.value;
}
/** Stores `value` under `key` from now on, and returns it. */
set(key: string, value: V): V {
this.#entries.delete(key);
// A Map keeps insertion order, so the first key is the oldest write.
if (this.#entries.size >= this.maxEntries) {
const oldest = this.#entries.keys().next().value;
if (oldest !== undefined) this.#entries.delete(oldest);
}
this.#entries.set(key, { value, at: Date.now() });
return value;
}
/**
* Removes `key`, or only when `value` is still what it holds, so a late
* failure can't remove the entry that replaced its own.
*/
delete(key: string, value?: V): void {
if (value === undefined || this.#entries.get(key)?.value === value) {
this.#entries.delete(key);
}
}
/** Empties the cache, for the tests' reset between cases. */
clear(): void {
this.#entries.clear();
}
}

626
api/mail/digest.ts Normal file
View File

@@ -0,0 +1,626 @@
/**
* Renders the weekly digest and the monthly recap, as the drafts approved
* them: the weekly one has four sections (what happened to the reader, what
* the people and playlists they follow posted, the period's top dumps, and
* the community), the monthly one only the last two. `digest-service.ts`
* gathers a {@link DigestContent} and sends what this returns, drawn in the
* reader's own style and scheme.
*
* @module
*/
import { EMAIL_LOCALE, OG_SITE_NAME } from "../config.ts";
import type { DigestCadence } from "../../shared/settings.ts";
import { i18n } from "./i18n.ts";
import { DIGEST_MESSAGES } from "../../shared/digest-messages.ts";
import {
EmailKit,
type EmailLook,
escMd,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/** The kinds of activity a weekly digest counts. */
export type ActivityKind =
| "upvotes"
| "followers"
| "comments"
| "mentions"
| "likes"
| "playlistFollows";
/** What a highlight of the weekly digest says happened. */
export type HighlightKind =
| "mentionComment"
| "mentionDump"
| "mentionPlaylist"
| "mentionChat"
| "mentionEveryone"
| "comment"
| "playlistFollow"
| "userFollow"
| "commentLike";
/** A dump in a digest's lists, with its links made absolute. */
export interface DigestDump {
title: string;
url: string;
poster: string;
/** The link's host, or null for an upload. */
domain: string | null;
/** An absolute image URL, or null to show no image. */
thumbnail: string | null;
votes: number;
comments: number;
/** Posted by the reader, which the lists mark. */
mine: boolean;
/** The followed playlist it was added to, when that's why it's listed. */
addedTo?: string;
}
/** One line of the weekly digest's "For you", made from a notification. */
export interface DigestHighlight {
kind: HighlightKind;
/** Who did it. */
who: string;
/**
* What they did it on, absent for a follow of the reader or the chat. Its
* `url` is null once it is deleted, and the title then shows unlinked.
*/
target?: { title: string; url: string | null };
/** The comment or message, flattened and shortened. */
quote?: string;
at: Date;
}
/**
* Passed to {@link renderDigestEmail} by `digest-service.ts`. Each section is
* null when it has nothing to show, and is left out; the service sends
* nothing when all of them are.
*/
export interface DigestContent {
cadence: DigestCadence;
/** The period covered, from `from` to `to`. */
from: Date;
to: Date;
reader: string;
/** Weekly only. */
activity: {
counts: Partial<Record<ActivityKind, number>>;
highlights: DigestHighlight[];
/** Every notification of the period, which the link to them counts. */
total: number;
} | null;
/** Weekly only. */
network: { dumps: DigestDump[]; more: number } | null;
top: DigestDump[];
community: {
members: { username: string; invitedBy: string | null }[];
membersMore: number;
playlists: { title: string; url: string; owner: string; dumps: number }[];
playlistsMore: number;
chatMessages: number;
totals: { dumps: number; upvotes: number; comments: number };
} | null;
links: {
notifications: string;
followed: string;
chat: string;
/** The community page's view of the cadence's period. */
community: string;
settings: string;
/** The one-click page that turns this cadence off. */
unsubscribe: string;
};
}
const ACTIVITY_ORDER: ActivityKind[] = [
"upvotes",
"followers",
"comments",
"mentions",
"likes",
"playlistFollows",
];
/** Returns the label of `n` of a kind of activity, as in "12 upvotes". */
function countLabel(kind: ActivityKind, n: number): string {
switch (kind) {
case "upvotes":
return i18n._({ ...DIGEST_MESSAGES.countUpvotes, values: { n } });
case "followers":
return i18n._({
id: "email.digest.count.followers",
message: "{n, plural, one {new follower} other {new followers}}",
values: { n },
});
case "comments":
return i18n._({ ...DIGEST_MESSAGES.countComments, values: { n } });
case "mentions":
return i18n._({
id: "email.digest.count.mentions",
message: "{n, plural, one {mention} other {mentions}}",
values: { n },
});
case "likes":
return i18n._({
id: "email.digest.count.likes",
message: "{n, plural, one {comment like} other {comment likes}}",
values: { n },
});
case "playlistFollows":
return i18n._({
id: "email.digest.count.playlistFollows",
message: "{n, plural, one {playlist follow} other {playlist follows}}",
values: { n },
});
}
}
/**
* Returns the sentence of a highlight, `who` and `target` already formatted
* for the part it goes in (HTML or Markdown).
*/
function highlightSentence(
kind: HighlightKind,
who: string,
target: string,
): string {
const values = { who, target };
switch (kind) {
case "mentionComment":
return i18n._({
id: "email.digest.hl.mentionComment",
message: "{who} mentioned you in a comment on {target}",
values,
});
case "mentionDump":
case "mentionChat":
return i18n._({
id: "email.digest.hl.mentionIn",
message: "{who} mentioned you in {target}",
values,
});
case "mentionPlaylist":
return i18n._({
id: "email.digest.hl.mentionPlaylist",
message: "{who} mentioned you in the playlist {target}",
values,
});
case "mentionEveryone":
return i18n._({
id: "email.digest.hl.mentionEveryone",
message: "{who} mentioned everyone in {target}",
values,
});
case "comment":
return i18n._({
id: "email.digest.hl.comment",
message: "{who} commented on your dump {target}",
values,
});
case "playlistFollow":
return i18n._({
id: "email.digest.hl.playlistFollow",
message: "{who} followed your playlist {target}",
values,
});
case "userFollow":
return i18n._({
id: "email.digest.hl.userFollow",
message: "{who} started following you",
values,
});
case "commentLike":
return i18n._({
id: "email.digest.hl.commentLike",
message: "{who} liked your comment on {target}",
values,
});
}
}
const byLine = (username: string) =>
i18n._({
id: "email.digest.by",
message: "by @{username}",
values: { username },
});
const addedToLine = (playlist: string) =>
i18n._({
id: "email.digest.addedTo",
message: "added to {playlist}",
values: { playlist },
});
const uploadLabel = () =>
i18n._({ id: "email.digest.upload", message: "upload" });
const invitedLine = (inviter: string, reader: string) =>
inviter === reader
? i18n._(DIGEST_MESSAGES.invitedByYou)
: i18n._({ ...DIGEST_MESSAGES.invitedBy, values: { username: inviter } });
const andMoreLine = (n: number) =>
i18n._({
id: "email.digest.andMore",
message: "and {n} more",
values: { n },
});
const playlistMeta = (owner: string, n: number) =>
i18n._({
id: "email.digest.playlistMeta",
message: "by @{username} · {n, plural, one {# dump} other {# dumps}}",
values: { username: owner, n },
});
/**
* Returns the digest `content` describes, written as Markdown and drawn in
* `look`, with its subject and hidden preheader.
*/
export function renderDigestEmail(
content: DigestContent,
look: EmailLook,
): RenderedEmail {
const monthly = content.cadence === "monthly";
const nf = new Intl.NumberFormat(EMAIL_LOCALE);
const range = (year: boolean) =>
new Intl.DateTimeFormat(
EMAIL_LOCALE,
year
? { month: "short", day: "numeric", year: "numeric" }
: { month: "short", day: "numeric" },
).formatRange(content.from, content.to);
const day = (d: Date) =>
new Intl.DateTimeFormat(EMAIL_LOCALE, { weekday: "short" }).format(d);
const figure = (n: number, label: string) =>
`**${nf.format(n)}** ${escMd(label)}`;
const yours = i18n._({ id: "email.digest.yours", message: "yours" });
/** A dump as a list item: its thumbnail, title, pill and detail. */
const dumpItem = (d: DigestDump, bullet: string, image = true) =>
`${bullet} ${image && d.thumbnail ? `![](${d.thumbnail}) ` : ""}${
mdLink(d.title, d.url)
}${d.mine ? ` *${escMd(yours)}*` : ""} · ${
escMd(
`${d.addedTo ? addedToLine(d.addedTo) : byLine(d.poster)}${
d.addedTo ? "" : ` · ${d.domain ?? uploadLabel()}`
} · ${nf.format(d.votes)} ${countLabel("upvotes", d.votes)} · ${
nf.format(d.comments)
} ${countLabel("comments", d.comments)}`,
)
}`;
const md: string[] = [
`*${
escMd(
`${
monthly
? i18n._(DIGEST_MESSAGES.monthly)
: i18n._(DIGEST_MESSAGES.weekly)
} · ${range(true)}`,
)
}*`,
`# ${
escMd(
monthly
? i18n._({
id: "email.digest.monthly.title",
message: "The month on {site}",
values: { site: OG_SITE_NAME },
})
: i18n._({
id: "email.digest.weekly.title",
message: "Your week, {username}",
values: { username: content.reader },
}),
)
}`,
];
const { activity, network, community } = content;
if (activity) {
md.push(
`## ${escMd(i18n._({ id: "email.digest.forYou", message: "For you" }))}`,
ACTIVITY_ORDER.filter((k) => (activity.counts[k] ?? 0) > 0)
.map((k) =>
figure(activity.counts[k]!, countLabel(k, activity.counts[k]!))
)
.join(" · "),
activity.highlights.map((h) => {
const target = h.kind === "mentionChat" || h.kind === "mentionEveryone"
? mdLink(
i18n._({ id: "email.digest.theChat", message: "the chat" }),
content.links.chat,
)
: h.target?.url
? mdLink(h.target.title, h.target.url)
: h.target
? escMd(h.target.title)
: "";
const line = `- ${
highlightSentence(h.kind, `**@${escMd(h.who)}**`, target)
} *${escMd(day(h.at))}*`;
return h.quote ? `${line}\n > ${escMd(h.quote)}` : line;
}).join("\n"),
`${
mdLink(
i18n._({
id: "email.digest.seeAll",
message:
"{n, plural, one {See the notification} other {See all # notifications}}",
values: { n: activity.total },
}),
content.links.notifications,
)
} →`,
);
}
if (network) {
md.push(
`## ${
escMd(
i18n._({
id: "email.digest.network",
message: "From people you follow",
}),
)
}`,
network.dumps.map((d) => dumpItem(d, "-")).join("\n"),
);
if (network.more > 0) {
md.push(`${
mdLink(
i18n._({
id: "email.digest.more",
message: "{n} more on your Followed feed",
values: { n: network.more },
}),
content.links.followed,
)
} →`);
}
}
if (content.top.length > 0) {
const [first] = content.top;
md.push(
`## ${
escMd(
monthly
? i18n._(DIGEST_MESSAGES.topMonth)
: i18n._(DIGEST_MESSAGES.topWeek),
)
}`,
);
// The recap gives its first dump a wide image and the next two a thumbnail.
if (monthly && first.thumbnail) md.push(`![](${first.thumbnail})`);
md.push(
content.top.map((d, i) =>
dumpItem(d, `${i + 1}.`, monthly ? i === 1 || i === 2 : true)
).join("\n"),
);
}
if (community) {
const { dumps, upvotes, comments } = community.totals;
md.push(
`## ${escMd(i18n._(DIGEST_MESSAGES.community))}`,
);
if (monthly) {
md.push(
[
figure(
dumps,
i18n._({ ...DIGEST_MESSAGES.statDumps, values: { n: dumps } }),
),
figure(upvotes, countLabel("upvotes", upvotes)),
figure(comments, countLabel("comments", comments)),
figure(
community.chatMessages,
i18n._({
...DIGEST_MESSAGES.statChat,
values: { n: community.chatMessages },
}),
),
].join(" · "),
);
}
if (community.members.length > 0) {
md.push(
`### ${
escMd(
i18n._(DIGEST_MESSAGES.newMembers),
)
}`,
[
...community.members.map((m) =>
`- **@${escMd(m.username)}**${
m.invitedBy
? ` ${escMd(invitedLine(m.invitedBy, content.reader))}`
: ""
}`
),
...(community.membersMore > 0
? [`- ${escMd(andMoreLine(community.membersMore))}`]
: []),
].join("\n"),
);
}
if (community.playlists.length > 0) {
md.push(
`### ${escMd(i18n._(DIGEST_MESSAGES.newPlaylists))}`,
[
...community.playlists.map((p) =>
`- ${mdLink(p.title, p.url)} · ${
escMd(playlistMeta(p.owner, p.dumps))
}`
),
...(community.playlistsMore > 0
? [`- ${escMd(andMoreLine(community.playlistsMore))}`]
: []),
].join("\n"),
);
}
if (!monthly && community.chatMessages > 0) {
md.push(
`### ${
escMd(i18n._({ id: "email.digest.inChat", message: "In the chat" }))
}`,
`${
escMd(
i18n._({
id: "email.digest.chatWeek",
message:
"{n, plural, one {# message this week} other {# messages this week}}",
values: { n: community.chatMessages },
}),
)
} · ${
mdLink(
i18n._({ id: "email.digest.openChat", message: "Open the chat" }),
content.links.chat,
)
}`,
);
}
if (!monthly && dumps + upvotes + comments > 0) {
md.push(
`*${
escMd(i18n._({
id: "email.digest.totals",
message:
"{dumps, plural, one {# dump} other {# dumps}}, {upvotes, plural, one {# upvote} other {# upvotes}} and {comments, plural, one {# comment} other {# comments}}.",
values: { dumps, upvotes, comments },
}))
}*`,
);
}
md.push(`${
mdLink(
i18n._({
id: "email.digest.communityPage",
message: "See it all on the community page",
}),
content.links.community,
)
} →`);
}
md.push(
"---",
escMd(
monthly
? i18n._({
id: "email.digest.monthly.reason",
message:
"You're getting the monthly recap because it's on in your settings.",
})
: i18n._({
id: "email.digest.weekly.reason",
message:
"You're getting the weekly digest because it's on in your settings.",
}),
),
`${
mdLink(
monthly
? i18n._(DIGEST_MESSAGES.monthlyOff)
: i18n._(DIGEST_MESSAGES.weeklyOff),
content.links.unsubscribe,
)
} · ${
mdLink(
i18n._({ id: "email.settings", message: "Settings" }),
content.links.settings,
)
}`,
);
return new EmailKit(look).render({
subject: monthly
? i18n._({
id: "email.digest.monthly.subject",
message: "The month on {site}, {range}",
values: { site: OG_SITE_NAME, range: range(false) },
})
: i18n._({
id: "email.digest.weekly.subject",
message: "Your week on {site}, {range}",
values: { site: OG_SITE_NAME, range: range(false) },
}),
preheader: preheader(content),
markdown: md.join("\n\n"),
});
}
/**
* Returns the line an inbox shows beside the subject: the counts that make
* the digest worth opening, at most three of them.
*/
function preheader(content: DigestContent): string {
const parts: string[] = [];
if (content.cadence === "weekly") {
for (const k of ["upvotes", "followers", "comments"] as const) {
const n = content.activity?.counts[k] ?? 0;
if (n > 0) parts.push(`${n} ${countLabel(k, n)}`);
}
const n = (content.network?.dumps.length ?? 0) +
(content.network?.more ?? 0);
if (n > 0) {
parts.push(
i18n._({
id: "email.digest.pre.network",
message:
"{n, plural, one {# new dump from people you follow} other {# new dumps from people you follow}}",
values: { n },
}),
);
}
} else {
const members = (content.community?.members.length ?? 0) +
(content.community?.membersMore ?? 0);
const playlists = (content.community?.playlists.length ?? 0) +
(content.community?.playlistsMore ?? 0);
if (content.top.length > 0) {
parts.push(
i18n._({
id: "email.digest.pre.topMonth",
message:
"{n, plural, one {the most upvoted dump of the month} other {the # most upvoted dumps of the month}}",
values: { n: content.top.length },
}),
);
}
if (members > 0) {
parts.push(
i18n._({
id: "email.digest.pre.members",
message: "{n, plural, one {# new member} other {# new members}}",
values: { n: members },
}),
);
}
if (playlists > 0) {
parts.push(
i18n._({
id: "email.digest.pre.playlists",
message: "{n, plural, one {# new playlist} other {# new playlists}}",
values: { n: playlists },
}),
);
}
}
if (parts.length === 0 && content.top.length > 0) {
parts.push(
i18n._({
id: "email.digest.pre.topWeek",
message:
"{n, plural, one {the most upvoted dump of the week} other {the # most upvoted dumps of the week}}",
values: { n: content.top.length },
}),
);
}
if (parts.length === 0) return "";
const line = new Intl.ListFormat(EMAIL_LOCALE, { type: "conjunction" })
.format(parts.slice(0, 3));
return `${line.charAt(0).toUpperCase()}${line.slice(1)}.`;
}

36
api/mail/i18n.ts Normal file
View File

@@ -0,0 +1,36 @@
/**
* Lets the emails speak the instance's language from the same catalogs as
* the interface. A template passes each message as a descriptor holding its
* id and its English, as `i18n._({ id: "email.reset.title", message: "Reset
* your password" })`, or one of `shared/digest-messages.ts`, which a page
* says too. `lingui extract` reads them into `locales/<lang>.po` beside the
* interface's messages, and `lingui compile` turns the catalogs into the
* `locales/<lang>.mjs` imported here, failing when a language lacks one. The
* ids are named rather than hashed from the English, since the server runs
* no Lingui macro.
*
* @module
*/
import { type I18n, type Messages, setupI18n } from "@lingui/core";
import { EMAIL_LOCALE } from "../config.ts";
import { LOCALES } from "../../shared/locale.ts";
const catalogs: [string, Messages][] = await Promise.all(
LOCALES.map(async (locale) =>
[
locale,
(await import(`../../locales/${locale}.mjs`)).messages as Messages,
] as [string, Messages]
),
);
/**
* Translates the emails' messages, in {@link EMAIL_LOCALE}. Every catalog is
* loaded, so a test can activate another language.
*/
export const i18n: I18n = setupI18n({
locale: EMAIL_LOCALE,
messages: Object.fromEntries(catalogs),
});

822
api/mail/layout.ts Normal file
View File

@@ -0,0 +1,822 @@
/**
* Every email the site sends shares one layout: the site's emoji and name,
* then a card holding the content, then a footer. The templates beside this
* module (welcome, password reset, digest) build their content with an
* {@link EmailKit}, which draws each block in a style from `styles.ts`, and
* `email-service.ts` sends what they return.
*
* Mail clients drop stylesheets, so every block carries inline styles. A
* reader on the "auto" scheme gets the light palette inline and the dark one
* in a `prefers-color-scheme` block that Apple's mail apps honour; elsewhere
* the light one shows.
*
* @module
*/
import { Marked, Parser, type Token, type Tokens } from "marked";
import {
DEFAULT_STYLE,
EMAIL_LOCALE,
OG_SITE_NAME,
SITE_EMOJI,
} from "../config.ts";
import {
type ColorScheme,
DEFAULT_COLOR_SCHEME,
type StyleName,
} from "../../shared/theme.ts";
import type { UserSettings } from "../../shared/settings.ts";
import { EMAIL_STYLES, type EmailPalette, type EmailStyle } from "./styles.ts";
/**
* Returned by each template to its caller, which hands it to `sendEmail`
* with a recipient.
*/
export interface RenderedEmail {
subject: string;
html: string;
/**
* The plain-text part, for clients that show no HTML. It is Markdown, so it
* reads as text and renders as the email's outline.
*/
text: string;
}
/**
* Passed to {@link EmailKit}, made by {@link lookOf}: the instance's style for
* the emails that aren't a user's own, their settings for a digest.
*/
export interface EmailLook {
style: StyleName;
scheme: ColorScheme;
}
/**
* Returns the look of a reader's settings. An unset style is the instance's,
* as on the site, and an unset scheme the system's; with no settings at all,
* as for the welcome and reset emails, both are.
*/
export function lookOf(
settings: Pick<UserSettings, "style" | "colorScheme"> | null = null,
): EmailLook {
return {
style: settings?.style ?? DEFAULT_STYLE,
scheme: settings?.colorScheme ?? DEFAULT_COLOR_SCHEME,
};
}
/**
* Names a color of an {@link EmailPalette}, as {@link EmailKit.paint} takes
* it, so that the dark block can recolor the element by the same name.
*/
export type PaletteRole = Exclude<keyof EmailPalette, "gradient">;
const ROLES: PaletteRole[] = [
"outer",
"card",
"text",
"sec",
"muted",
"accent",
"link",
"onAccent",
"rule",
"soft",
"quote",
"date",
"brand",
"border",
];
/**
* Escapes `value` for HTML text and attributes. Everything users wrote (names,
* titles, quotes) goes through it, since none of it may become markup.
*/
export function esc(value: string): string {
return value.replace(
/[&<>"']/g,
(c) =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[
c
]!,
);
}
/**
* Escapes `value` for Markdown, so that user text in a plain-text part (a
* title with `*` or `[`, say) stays text.
*/
export function escMd(value: string): string {
return value.replace(/[\\`*_[\]<>#!|~]/g, "\\$&");
}
/** Returns a Markdown link to `url` labelled with `text`, escaped. */
export function mdLink(text: string, url: string): string {
const safeUrl = url.replace(
/[()\s]/g,
(c) => `%${c.charCodeAt(0).toString(16).toUpperCase().padStart(2, "0")}`,
);
return `[${escMd(text)}](${safeUrl})`;
}
const gradient = (colors: string[]) =>
`linear-gradient(90deg,${colors.join(",")})`;
/**
* Draws an email in one {@link EmailLook}. A template writes the email as
* Markdown and hands it to {@link EmailKit.render}, which draws each block in
* the style. The block methods return HTML: text passed to them must already
* be escaped with {@link esc}.
*/
export class EmailKit {
/** The style being drawn. */
readonly style: EmailStyle;
/** The palette inlined into the HTML. */
readonly palette: EmailPalette;
/** The palette that replaces it in dark mode, on the "auto" scheme only. */
readonly darkPalette: EmailPalette | null;
/** The font stack, for templates that set their own blocks. */
readonly font: string;
constructor(look: EmailLook) {
this.style = EMAIL_STYLES[look.style];
this.palette = look.scheme === "dark" ? this.style.dark : this.style.light;
this.darkPalette = look.scheme === DEFAULT_COLOR_SCHEME
? this.style.dark
: null;
this.font = this.style.font;
}
/** Returns `px`, scaled for the style, as a CSS length. */
fs(px: number): string {
return `${Math.round(px * this.style.scale * 2) / 2}px`;
}
/**
* Returns the `class` and `style` attributes of an element colored by
* roles: `fg` for its text, `bg` for its fill, `bd` for its border. The
* classes let the dark block recolor it; `css` is the rest of its style.
*/
paint(
roles: {
fg?: PaletteRole;
bg?: PaletteRole;
bd?: PaletteRole;
extra?: string[];
},
css = "",
): string {
const p = this.palette;
const classes: string[] = [...(roles.extra ?? [])];
let style = "";
if (roles.fg) {
classes.push(`f-${roles.fg}`);
style += `color:${p[roles.fg] ?? p.text};`;
}
if (roles.bg) {
classes.push(`b-${roles.bg}`);
style += `background-color:${p[roles.bg] ?? p.card};`;
}
if (roles.bd) {
classes.push(`d-${roles.bd}`);
style += `border-color:${p[roles.bd] ?? p.rule};`;
}
const cls = classes.length ? ` class="${classes.join(" ")}"` : "";
// The colors come last: a shorthand such as `border-top` in `css` would
// otherwise reset them.
return `${cls} style="${css}${style}"`;
}
/** Returns a link to `href` around `html`, in the link color. */
link(html: string, href: string, strong = false): string {
const s = this.style;
return `<a href="${esc(href)}"${
this.paint(
{ fg: "link" },
`text-decoration:${s.linkUnderline ? "underline" : "none"};${
strong ? `font-weight:${s.heavy};` : ""
}`,
)
}>${html}</a>`;
}
/** Returns the email's heading, in the style's title type. */
heading(html: string, marginBottom = 24): string {
const t = this.style.title;
return `<h1${
this.paint(
{ fg: t.accent ? "accent" : "text" },
`font-family:${this.font};font-size:${
this.fs(t.size)
};line-height:1.15;font-weight:${t.weight};${
t.upper ? "text-transform:uppercase;" : ""
}${
t.spacing ? `letter-spacing:${t.spacing};` : ""
}margin:0 0 ${marginBottom}px;`,
)
}>${html}</h1>`;
}
/** Returns a paragraph. `css` adds to or overrides its style. */
para(html: string, css = ""): string {
return `<p${
this.paint(
{ fg: "text" },
`font-family:${this.font};font-size:${
this.fs(15)
};line-height:1.6;margin:0 0 16px;${css}`,
)
}>${html}</p>`;
}
/** Returns a smaller paragraph in the muted color. */
note(html: string, css = ""): string {
return `<p${
this.paint(
{ fg: "muted" },
`font-family:${this.font};font-size:${
this.fs(13.5)
};line-height:1.6;margin:0 0 16px;${css}`,
)
}>${html}</p>`;
}
/** Returns a button linking to `href`. */
button(label: string, href: string): string {
const s = this.style;
const frame = s.frame === "hard"
? `border:2px solid ${this.palette.text};box-shadow:2px 2px 0 ${this.palette.text};`
: "";
return `<table role="presentation" cellpadding="0" cellspacing="0" style="margin:6px 0 22px;"><tr><td${
this.paint(
{
bg: "accent",
bd: s.frame === "hard" ? "text" : undefined,
extra: s.frame === "hard" ? ["sh-btn"] : [],
},
`border-radius:${s.radius.btn}px;${frame}`,
)
}><a href="${esc(href)}"${
this.paint(
{ fg: "onAccent" },
`display:inline-block;padding:13px 22px;font-family:${this.font};font-size:${
this.fs(13.5)
};font-weight:${
Math.max(600, s.heavy)
};letter-spacing:0.08em;text-transform:uppercase;line-height:1;text-decoration:none;`,
)
}>${label}</a></td></tr></table>`;
}
/** Returns a small uppercase pill, as on one's own dump in a list. */
pill(label: string): string {
return `<span${
this.paint(
{ fg: "sec", bg: "soft" },
`display:inline-block;margin-left:6px;padding:1px 7px;border-radius:${this.style.radius.pill}px;font-size:${
this.fs(11)
};font-weight:600;letter-spacing:0.08em;text-transform:uppercase;vertical-align:2px;`,
)
}>${label}</span>`;
}
/** Returns an image block, rounded as the style rounds images. */
image(
src: string,
alt: string,
width: number,
height: number,
href?: string,
): string {
const img = `<img src="${
esc(src)
}" width="${width}" height="${height}" alt="${
esc(alt)
}" style="display:block;max-width:100%;height:auto;border:0;border-radius:${
Math.min(this.style.radius.img, width / 8)
}px;">`;
return href ? `<a href="${esc(href)}">${img}</a>` : img;
}
/** Returns a small bold heading inside a section. */
subhead(html: string): string {
return `<div${
this.paint(
{ fg: "text" },
`font-family:${this.font};font-size:${this.fs(14)};font-weight:${
Math.max(600, this.style.heavy)
};margin:0 0 8px;`,
)
}>${html}</div>`;
}
/**
* Returns a section of the card: a divider in the style's rule (none before
* the first), an uppercase title in the accent color, and `html`.
*/
section(title: string, html: string, first = false): string {
const s = this.style;
let divider = "";
let border = "";
if (!first) {
const stops = this.palette.gradient;
if (s.rule === "gradient" && stops) {
divider =
`<div class="grad" style="height:3px;line-height:3px;font-size:0;background-color:${
stops[0]
};background-image:${gradient(stops)};margin:0 0 26px;">&nbsp;</div>`;
} else {
border = `border-top:${
s.rule === "thick" ? 2 : 1
}px solid;padding-top:26px;`;
}
}
return `<div${
this.paint(
{ bd: border ? (s.rule === "thick" ? "text" : "rule") : undefined },
`${border}padding-bottom:26px;`,
)
}>${divider}<div${
this.paint(
{ fg: "accent" },
`font-family:${this.font};font-size:${this.fs(12)};font-weight:${
Math.max(600, s.heavy)
};letter-spacing:0.12em;text-transform:uppercase;margin:0 0 16px;`,
)
}>${title}</div>${html}</div>`;
}
/**
* Returns the email written as `markdown`: the Markdown itself is the
* plain-text part, and its blocks, drawn in this kit's style, are the HTML.
* Each template writes its email once this way. The blocks map as follows.
*
* - `# Title` is the heading; an `*emphasised*` paragraph above it, the
* eyebrow. Elsewhere an emphasised paragraph is a note.
* - `## Section` starts a section, with its divider; `### Name` a subhead.
* - A link alone in a paragraph is a button, an image alone a wide image,
* and a fenced block a box for a raw link.
* - A list item whose text starts with an image, then a link, is a row
* with that thumbnail and the link as its title; what follows is its
* detail, and an emphasised word right after the link a pill. Other items
* are sentences, where emphasis marks the date. A numbered list ranks its
* rows, and a quote under an item is set as one.
* - A bold number is a figure.
* - What follows the last `---` is the footer.
*/
render(
{ subject, preheader, markdown }: {
subject: string;
preheader: string;
markdown: string;
},
): RenderedEmail {
const cut = markdown.lastIndexOf("\n---\n");
const body = cut === -1 ? markdown : markdown.slice(0, cut);
const foot = cut === -1 ? "" : markdown.slice(cut + 5);
return {
subject,
html: this.document({
subject,
preheader,
body: this.blocks(this.marked.lexer(body)),
footer: this.marked.lexer(foot)
.filter((t): t is Tokens.Paragraph => t.type === "paragraph")
.map((t) => this.inline(t.tokens, "footer")).join("<br>"),
}),
text: markdown,
};
}
/** How {@link inline} draws emphasis and links, by where they sit. */
private mode: "text" | "footer" | "row" | "sentence" = "text";
/** Draws Markdown's inline tokens with the kit's colors. */
// The kit comes in as an argument: the renderer's methods need marked's
// `this` (its parser) as well as the kit's colors.
private readonly marked: Marked = ((kit: EmailKit) => {
return new Marked({
renderer: {
link({ href, tokens }) {
const html = this.parser.parseInline(tokens);
return kit.mode === "footer"
? `<a href="${esc(href)}"${
kit.paint({ fg: "muted" }, "text-decoration:underline;")
}>${html}</a>`
: kit.link(html, href);
},
strong({ tokens }) {
const html = this.parser.parseInline(tokens);
// A figure, its digits grouped by any space: `\s` takes the no-break
// ones locales group thousands with.
const figure = /^[\d\s.,]+$/.test(html);
return `<b${
kit.paint(
{ fg: "text" },
figure
? `font-size:${kit.fs(22)};font-weight:${kit.style.heavy};`
: `font-weight:${Math.max(600, kit.style.heavy)};`,
)
}>${html}</b>`;
},
em({ tokens }) {
const html = this.parser.parseInline(tokens);
if (kit.mode === "row") return kit.pill(html);
if (kit.mode === "sentence") {
return `<span${
kit.paint(
{ fg: kit.palette.date ? "date" : "muted" },
`font-size:${kit.fs(12.5)};white-space:nowrap;${
kit.palette.date ? "font-style:italic;" : ""
}`,
)
}>${html}</span>`;
}
return `<i>${html}</i>`;
},
codespan({ text }) {
return `<code style="font-family:Menlo,Consolas,'Courier New',monospace;">${text}</code>`;
},
image({ href, text }) {
return kit.image(href, text, 104, 78);
},
},
});
})(this);
private inline(tokens: Token[], mode: EmailKit["mode"]): string {
this.mode = mode;
try {
return Parser.parseInline(tokens, this.marked.defaults);
} finally {
this.mode = "text";
}
}
/** Draws the body's blocks, gathering each `##` section with its content. */
private blocks(tokens: Token[]): string {
const out: string[] = [];
let section: { title: string; parts: string[] } | null = null;
let first = true;
let headed = false;
const push = (html: string) =>
section ? section.parts.push(html) : out.push(html);
const close = () => {
if (!section) return;
out.push(this.section(section.title, section.parts.join(""), first));
first = false;
section = null;
};
for (const token of tokens) {
switch (token.type) {
case "heading": {
const html = this.inline(token.tokens ?? [], "text");
if (token.depth === 1) {
headed = true;
push(this.heading(html));
} else if (token.depth === 2) {
close();
section = { title: html, parts: [] };
} else {
push(this.subhead(html));
}
break;
}
case "paragraph": {
const kids = (token.tokens ?? []).filter((t) =>
!(t.type === "text" && !t.raw.trim())
);
const only = kids.length === 1 ? kids[0] : null;
if (only?.type === "link") {
push(
this.button(this.inline(only.tokens ?? [], "text"), only.href),
);
} else if (only?.type === "image") {
push(
`${
this.image(only.href, only.text, 528, 264)
}<div style="height:14px;line-height:14px;font-size:0;">&nbsp;</div>`,
);
} else if (only?.type === "em") {
const html = this.inline(only.tokens ?? [], "text");
push(headed ? this.note(html) : this.eyebrow(html));
} else {
push(this.para(this.inline(token.tokens ?? [], "text")));
}
break;
}
case "list":
push(this.list(token as Tokens.List));
break;
case "blockquote":
push(this.quote(this.inline(
(token.tokens ?? []).flatMap((t) =>
"tokens" in t ? t.tokens ?? [] : []
),
"text",
)));
break;
case "code":
push(this.linkBox(esc(token.text)));
break;
case "html":
push(token.text);
break;
}
}
close();
return out.join("");
}
/** Draws a list as rows, ranked when it is numbered. */
private list(list: Tokens.List): string {
const start = typeof list.start === "number" ? list.start : 1;
return list.items.map((item, i) => {
const text = item.tokens.find((t) =>
t.type === "text" || t.type === "paragraph"
) as Tokens.Text | Tokens.Paragraph | undefined;
const quote = item.tokens.find((t) => t.type === "blockquote") as
| Tokens.Blockquote
| undefined;
let rest: Token[] = [...(text?.tokens ?? [])];
const skipSpace = () => {
while (rest[0]?.type === "text" && !rest[0].raw.trim()) {
rest = rest.slice(1);
}
};
skipSpace();
const image = rest[0]?.type === "image"
? rest.shift() as Tokens.Image
: null;
skipSpace();
const link = rest[0]?.type === "link"
? rest.shift() as Tokens.Link
: null;
let content: string;
if (link) {
const pills: Token[] = [];
skipSpace();
while (rest[0]?.type === "em") {
pills.push(rest.shift()!);
skipSpace();
}
// The detail follows the title after a " · ".
if (rest[0]?.type === "text") {
const t = rest[0] as Tokens.Text;
const trimmed = t.text.replace(/^\s*·\s*/, "");
rest[0] = { ...t, text: trimmed, raw: trimmed, tokens: undefined };
}
content = `<a href="${esc(link.href)}"${
this.paint(
{ fg: "text" },
`font-size:${
this.fs(15)
};line-height:1.35;font-weight:${this.style.heavy};text-decoration:none;`,
)
}>${this.inline(link.tokens, "text")}</a>${this.inline(pills, "row")}${
rest.length
? `<div${
this.paint(
{ fg: "muted" },
`font-size:${this.fs(13)};line-height:1.5;margin-top:3px;`,
)
}>${this.inline(rest, "text")}</div>`
: ""
}`;
} else {
content = `<div${
this.paint(
{ fg: "sec" },
`font-size:${this.fs(14.5)};line-height:1.5;`,
)
}>${this.inline(rest, "sentence")}</div>`;
}
if (quote) {
content += this.quote(this.inline(
quote.tokens.flatMap((t) => "tokens" in t ? t.tokens ?? [] : []),
"text",
));
}
const size = list.ordered ? [64, 48] : [104, 78];
return `<table role="presentation" width="100%" cellpadding="0" cellspacing="0" style="margin:0 0 ${
link || image ? 16 : 10
}px;"><tr>${
list.ordered
? `<td width="30" valign="top"${
this.paint(
{ fg: "accent" },
`font-family:${this.font};font-size:${this.fs(20)};font-weight:${
Math.max(600, this.style.heavy)
};line-height:1.2;`,
)
}>${start + i}</td>`
: ""
}${
image
? `<td width="${size[0]}" valign="top" style="padding-right:${
list.ordered ? 12 : 14
}px;">${
this.image(image.href, image.text, size[0], size[1], link?.href)
}</td>`
: ""
}<td valign="top" style="font-family:${this.font};">${content}</td></tr></table>`;
}).join("");
}
/** Returns the small uppercase line above the heading. */
private eyebrow(html: string): string {
return `<div${
this.paint(
{ fg: "muted" },
`font-family:${this.font};font-size:${
this.fs(12)
};font-weight:600;letter-spacing:0.12em;text-transform:uppercase;margin:0 0 8px;`,
)
}>${html}</div>`;
}
/** Returns a quoted comment or message. */
private quote(html: string): string {
return `<div${
this.paint(
{ fg: "quote" },
`font-family:${this.font};font-size:${
this.fs(14.5)
};line-height:1.5;font-style:italic;margin:4px 0 0;`,
)
}>“${html}”</div>`;
}
/** Returns a raw link set in a box, for clients that break buttons. */
private linkBox(html: string): string {
return `<div${
this.paint(
{ fg: "sec", bg: "soft" },
`font-family:Menlo,Consolas,'Courier New',monospace;font-size:12px;line-height:1.5;border-radius:${
Math.min(this.style.radius.img, 6)
}px;padding:10px 12px;margin:0 0 22px;word-break:break-all;`,
)
}>${html}</div>`;
}
/**
* Returns the whole HTML document: the brand, the card holding `body`, and
* the `footer`. `preheader` is the hidden first line inboxes show beside the
* subject.
*/
private document(
{ subject, preheader, body, footer }: {
subject: string;
preheader: string;
body: string;
footer: string;
},
): string {
const s = this.style;
const p = this.palette;
const R = s.radius.card;
const bar = s.bar ? p.gradient ?? null : null;
const frame = s.frame === "hard"
? `border:2px solid ${p.text};box-shadow:5px 5px 0 ${p.text};`
: s.frame === "ridge"
? `border:3px ridge ${p.border};`
: s.frame === "hairline"
? `border:1px solid ${p.rule};`
: s.frame === "soft"
? `box-shadow:0 14px 34px -10px ${p.shadow};`
: "";
const frameRoles = s.frame === "hard"
? { bd: "text" as const, extra: ["sh-card"] }
: s.frame === "ridge"
? { bd: "border" as const }
: s.frame === "hairline"
? { bd: "rule" as const }
: s.frame === "soft"
? { extra: ["sh-soft"] }
: {};
const scheme = this.darkPalette
? "light dark"
: p === s.dark
? "dark"
: "light";
return `<!doctype html>
<html lang="${EMAIL_LOCALE}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="color-scheme" content="${scheme}">
<meta name="supported-color-schemes" content="${scheme}">
<title>${esc(subject)}</title>
${
s.webFont
? `<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=${s.webFont}&display=swap">`
: ""
}
<style>
body{margin:0;padding:0;}
@media (max-width:620px){.wrap{padding:20px 10px 28px!important}.card{padding:24px 20px 8px!important}}
${this.darkCss()}
</style>
</head>
<body${this.paint({ bg: "outer" }, "margin:0;padding:0;")}>
<div style="display:none;max-height:0;overflow:hidden;opacity:0;">${
esc(preheader)
}</div>
<table role="presentation" width="100%" cellpadding="0" cellspacing="0"${
this.paint({ bg: "outer" })
}>
<tr><td align="center" class="wrap" style="padding:32px 16px 40px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" style="max-width:600px;">
<tr><td style="padding:0 6px 16px;">
<table role="presentation" cellpadding="0" cellspacing="0"><tr>
<td style="padding-right:10px;vertical-align:middle;font-size:24px;line-height:1;">${
esc(SITE_EMOJI)
}</td>
<td${
this.paint(
{ fg: p.brand ? "brand" : "text" },
`vertical-align:middle;font-family:${this.font};font-weight:700;font-size:${
this.fs(21)
};line-height:1;letter-spacing:-0.01em;`,
)
}>${esc(OG_SITE_NAME)}</td>
</tr></table>
</td></tr>
${
bar
? `<tr><td class="grad" style="height:5px;line-height:5px;font-size:0;background-color:${
bar[0]
};background-image:${
gradient(bar)
};border-radius:${R}px ${R}px 0 0;">&nbsp;</td></tr>`
: ""
}
<tr><td${
this.paint(
{
bg: "card",
...frameRoles,
extra: ["card", ...(frameRoles.extra ?? [])],
},
`border-radius:${
bar ? `0 0 ${R}px ${R}px` : `${R}px`
};${frame}padding:34px 36px 12px;`,
)
}>
${body}
</td></tr>
<tr><td${
this.paint(
{ fg: "muted" },
`padding:22px 6px 0;font-family:${this.font};font-size:${
this.fs(12.5)
};line-height:1.6;`,
)
}>${footer}</td></tr>
</table>
</td></tr>
</table>
</body>
</html>`;
}
/**
* Returns the CSS that recolors every painted element in dark mode, or
* nothing off the "auto" scheme.
*/
private darkCss(): string {
const d = this.darkPalette;
if (!d) return "";
const rules: string[] = [];
for (const role of ROLES) {
const color = d[role];
if (!color) continue;
rules.push(
`.f-${role}{color:${color}!important}`,
`.b-${role}{background-color:${color}!important}`,
`.d-${role}{border-color:${color}!important}`,
);
}
rules.push(
`.sh-card{box-shadow:5px 5px 0 ${d.text}!important}`,
`.sh-btn{box-shadow:2px 2px 0 ${d.text}!important}`,
);
if (d.shadow) {
rules.push(
`.sh-soft{box-shadow:0 14px 34px -10px ${d.shadow}!important}`,
);
}
if (d.gradient) {
rules.push(
`.grad{background-color:${d.gradient[0]}!important;background-image:${
gradient(d.gradient)
}!important}`,
);
}
return `@media (prefers-color-scheme:dark){${rules.join("")}}`;
}
}

92
api/mail/password-reset.ts Normal file
View File

@@ -0,0 +1,92 @@
/**
* Renders the email carrying a password reset link. `requestPasswordReset`
* (`password-reset-service.ts`) sends it, in the instance's style and
* language: whoever asks isn't signed in, so their own settings don't apply.
*
* @module
*/
import { OG_SITE_NAME } from "../config.ts";
import { i18n } from "./i18n.ts";
import {
EmailKit,
escMd,
lookOf,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/**
* Returns the reset email of `username`, whose button opens `resetUrl`. The
* link stays usable for `hours`, which the email states.
*/
export function renderPasswordResetEmail(
username: string,
resetUrl: string,
hours: number,
): RenderedEmail {
const markdown = [
`# ${
escMd(i18n._({ id: "email.reset.title", message: "Reset your password" }))
}`,
escMd(
i18n._({
id: "email.reset.hi",
message: "Hi {username},",
values: { username },
}),
),
escMd(i18n._({
id: "email.reset.body",
message:
"Someone asked to reset the password of your {site} account. The button below lets you choose a new one. It works once, for {hours, plural, one {# hour} other {# hours}}.",
values: { site: OG_SITE_NAME, hours },
})),
mdLink(
i18n._({ id: "email.reset.button", message: "Reset password" }),
resetUrl,
),
`*${
escMd(
i18n._({
id: "email.reset.fallback",
message:
"If the button doesn't work, paste this link into your browser:",
}),
)
}*`,
"```\n" + resetUrl + "\n```",
`*${
escMd(
i18n._({
id: "email.reset.ignore",
message:
"If you didn't ask for this, ignore this email. Your password stays the same.",
}),
)
}*`,
"---",
escMd(
i18n._({
id: "email.reset.reason",
message:
"You're getting this email because a password reset was requested for this address.",
}),
),
].join("\n\n");
return new EmailKit(lookOf()).render({
subject: i18n._({
id: "email.reset.subject",
message: "Reset your {site} password",
values: { site: OG_SITE_NAME },
}),
preheader: i18n._({
id: "email.reset.preheader",
message:
"The link works once, for {hours, plural, one {# hour} other {# hours}}.",
values: { hours },
}),
markdown,
});
}

253
api/mail/styles.ts Normal file
View File

@@ -0,0 +1,253 @@
/**
* Carries each site style into email, so a digest looks like the site its
* reader chose and the other emails like the instance's style. A mail client
* takes no stylesheet of ours, so each style is resolved here into plain
* values, which `layout.ts` inlines.
*
* What a theme defines is read from it through `theme-tokens.ts`, never
* copied: the colors, the gradient, the font and the corner radii. What is
* the email's own is stated here: how large it sets its type, how its card
* stands out, which divider it draws, and which of the theme's own tokens
* carry the touches that make it recognisable (Kanagawa's syntax hues,
* GeoCities' rainbow, Melange's spice and warm shadow).
*
* @module
*/
import { STYLE_NAMES, type StyleName } from "../../shared/theme.ts";
import { readTokens, type Scheme } from "./theme-tokens.ts";
/**
* The colors of one style in one scheme, by role, each read from a token of
* its theme. The optional ones carry a style's particular touches.
*/
export interface EmailPalette {
/** The page around the card (`--color-bg`). */
outer: string;
/** The card holding the content (`--color-surface`). */
card: string;
/** `--color-text`. */
text: string;
/** `--color-text-secondary`. */
sec: string;
/** `--color-text-muted`. */
muted: string;
/** Section titles, ranks and button fills (`--color-accent`). */
accent: string;
/** `--color-link`. */
link: string;
/** Text on an accent fill (`--color-on-accent`). */
onAccent: string;
/** Dividers and hairlines (`--color-border`). */
rule: string;
/** Pills and the reset link's box, as the site's badges (`--color-badge-bg`). */
soft: string;
/** Quoted comments and messages (`--color-text-secondary` unless set). */
quote: string;
/** Kanagawa's fujiGray dates; others use `muted`. */
date?: string;
/** Kanagawa's oniViolet wordmark; others use `text`. */
brand?: string;
/** GeoCities' ridged border. */
border?: string;
/** Melange's warm card shadow, translucent. */
shadow?: string;
/** The colors of GeoCities' rainbow or Melange's spice, left to right. */
gradient?: string[];
}
/**
* How a style draws an email. `layout.ts` builds every block from it, so
* adding a style to `shared/theme.ts` means adding its {@link Recipe} here.
*/
export interface EmailStyle {
/**
* The theme's font stack (`--font-body`). Its first family is a web font
* only Apple's mail apps load, through {@link webFont}; other clients fall
* back along the stack.
*/
font: string;
/** The Google Fonts `family` parameter of the web font, if it is one. */
webFont?: string;
/** Multiplies every font size: Times sets its serif larger, Kanagawa its monospace smaller. */
scale: number;
/**
* Corner radii, in pixels: the card's as the site's cards
* (`--radius-xl`), the images' as its thumbnails (`--radius-md`), the
* buttons' as its buttons (`--radius-lg`), the pills' as its badges
* (`--radius-sm`). The avatars' is the email's own, since the site rounds
* them with a literal 50% that some styles square.
*/
radius: {
card: number;
img: number;
btn: number;
pill: number;
avatar: number;
};
/** The heading: its size in pixels, weight, and Brutalist's uppercase and spacing. */
title: {
size: number;
weight: number;
upper?: boolean;
spacing?: string;
accent?: boolean;
};
/** The weight of titles of items, figures and names. */
heavy: number;
/** How the card stands out from the page. */
frame: "none" | "hard" | "ridge" | "hairline" | "soft";
/** The divider between sections; "gradient" draws the palette's. */
rule: "hairline" | "thick" | "gradient";
/** Draws the palette's gradient as a bar along the card's top. */
bar?: boolean;
/** Underlines links, for styles whose links are the color of the text. */
linkUnderline?: boolean;
light: EmailPalette;
dark: EmailPalette;
}
/** The roles every style reads, and the token each is read from. */
const PALETTE_TOKENS = {
outer: "--color-bg",
card: "--color-surface",
text: "--color-text",
sec: "--color-text-secondary",
muted: "--color-text-muted",
accent: "--color-accent",
link: "--color-link",
onAccent: "--color-on-accent",
rule: "--color-border",
soft: "--color-badge-bg",
quote: "--color-text-secondary",
} satisfies Partial<Record<keyof EmailPalette, string>>;
/**
* What a style states about its email, beside what its theme defines: the
* {@link EmailStyle} less the values read from the theme, the axes of its
* web font, its avatars' radius, and the tokens of its own touches.
*/
interface Recipe
extends Omit<EmailStyle, "font" | "webFont" | "radius" | "light" | "dark"> {
/** The Google Fonts axes of the theme's first family; none for a system font. */
webFontAxes?: string;
avatarRadius: number;
/** The tokens of the palette's optional roles, or of a base role it reads elsewhere. */
tokens?: Partial<Record<keyof EmailPalette, string>>;
}
const RECIPES: Record<StyleName, Recipe> = {
smooth: {
webFontAxes: "wght@400;600;700",
scale: 1,
avatarRadius: 16,
title: { size: 30, weight: 600 },
heavy: 600,
frame: "none",
rule: "hairline",
},
brutalist: {
webFontAxes: "wght@400;700",
scale: 1,
avatarRadius: 0,
title: { size: 30, weight: 900, upper: true, spacing: "0.03em" },
heavy: 700,
frame: "hard",
rule: "thick",
},
geocities: {
scale: 1,
avatarRadius: 0,
title: { size: 30, weight: 700, accent: true },
heavy: 700,
frame: "ridge",
rule: "gradient",
bar: true,
linkUnderline: true,
tokens: { border: "--color-border", gradient: "--geo-rainbow" },
},
nyt: {
webFontAxes: "ital@0;1",
scale: 1.12,
avatarRadius: 16,
title: { size: 38, weight: 400 },
heavy: 400,
frame: "hairline",
rule: "hairline",
linkUnderline: true,
},
kanagawa: {
webFontAxes: "ital,wght@0,400;0,600;1,400",
scale: 0.9,
avatarRadius: 4,
title: { size: 26, weight: 600 },
heavy: 600,
frame: "hairline",
rule: "hairline",
tokens: {
quote: "--kngw-string",
date: "--kngw-comment",
brand: "--kngw-keyword",
},
},
melange: {
webFontAxes: "opsz,wght@9..144,400;9..144,600;9..144,700",
scale: 1,
avatarRadius: 16,
title: { size: 32, weight: 600 },
heavy: 600,
frame: "soft",
rule: "hairline",
bar: true,
tokens: { shadow: "--shadow-color", gradient: "--mel-spice" },
},
};
/**
* Returns the palette of `style` in `scheme` from its theme's tokens. Every
* color is flattened onto the card, as a translucent one shows on the site,
* but the shadow, which falls on the page.
*/
function readPalette(
style: StyleName,
scheme: Scheme,
own: Recipe["tokens"] = {},
): EmailPalette {
const tokens = readTokens(style, scheme);
const roles = { ...PALETTE_TOKENS, ...own };
const palette = Object.fromEntries(
Object.entries(roles)
.filter(([role]) => role !== "shadow" && role !== "gradient")
.map(([role, token]) => [role, tokens.color(token, "--color-surface")]),
) as unknown as EmailPalette;
if (own.shadow) palette.shadow = tokens.color(own.shadow);
if (own.gradient) palette.gradient = tokens.stops(own.gradient);
return palette;
}
/** Returns the email look of `style`, its theme read in both schemes. */
function readStyle(style: StyleName): EmailStyle {
const { webFontAxes, avatarRadius, tokens: own, ...recipe } = RECIPES[style];
const tokens = readTokens(style, "light");
const font = tokens.font("--font-body");
const family = font.split(",")[0].replace(/'/g, "").replaceAll(" ", "+");
return {
...recipe,
font,
webFont: webFontAxes ? `${family}:${webFontAxes}` : undefined,
radius: {
card: tokens.px("--radius-xl"),
img: tokens.px("--radius-md"),
btn: tokens.px("--radius-lg"),
pill: tokens.px("--radius-sm"),
avatar: avatarRadius,
},
light: readPalette(style, "light", own),
dark: readPalette(style, "dark", own),
};
}
/** Every style's email look, keyed by its name, read once at load. */
export const EMAIL_STYLES = Object.fromEntries(
STYLE_NAMES.map((style) => [style, readStyle(style)]),
) as Record<StyleName, EmailStyle>;

276
api/mail/theme-tokens.ts Normal file
View File

@@ -0,0 +1,276 @@
/**
* Lets the emails draw with the site's own values. It reads a style's design
* tokens out of the stylesheets the client loads, `src/tokens.css` and then
* `src/themes/<style>.css`, so `styles.ts` names a token wherever it needs a
* color, a gradient, a font or a radius, and a change to a theme reaches its
* emails with nothing copied to keep in step.
*
* Mail clients know no custom properties, so a token comes back resolved for
* one scheme: its `var()`s followed, its `light-dark()` picked, its
* `color-mix()` computed. Only the rules that set tokens on the root element
* are read (`:root`, the style's `[data-style]`, and either of those with the
* scheme's `[data-color-scheme]`); a theme states its forced schemes that way
* beside each `prefers-color-scheme` block, so the media blocks are skipped.
*
* @module
*/
import { BASE_STYLE, type StyleName } from "../../shared/theme.ts";
import { hexOf, parseCssColor, type Rgba } from "../lib/css-color.ts";
/** A scheme a token is resolved for. "auto" is drawn from both. */
export type Scheme = "light" | "dark";
/**
* Returned by {@link readTokens}: the tokens of one style in one scheme, read
* as the kind of value each is.
*/
export interface ThemeTokens {
/** Returns the resolved value of `token`, or throws when it isn't set. */
value(token: string): string;
/**
* Returns the color of `token` as `#rrggbb`. A translucent one is laid over
* `over` (another token) first, as it shows on the site; without `over` it
* stays `rgba()`, as a shadow's must.
*/
color(token: string, over?: string): string;
/** Returns the colors of the gradient `token`, in order, without positions. */
stops(token: string): string[];
/**
* Returns the font stack of `token`, single-quoted for a `style` attribute.
* A family the site bundles as "X Variable" is named "X", as a mail client
* or Google Fonts knows it.
*/
font(token: string): string;
/** Returns the length of `token` in pixels. */
px(token: string): number;
}
const SOURCES = ["../../src/tokens.css", "../../src/themes/{style}.css"];
/** Returns `css` without its comments. */
function stripComments(css: string): string {
return css.replace(/\/\*[\s\S]*?\*\//g, "");
}
/** Returns the top-level rules of `css` as their selector and body. */
function topRules(css: string): { selector: string; body: string }[] {
const rules = [];
let i = 0;
while (i < css.length) {
const open = css.indexOf("{", i);
if (open < 0) break;
let depth = 1;
let j = open + 1;
while (depth && j < css.length) {
if (css[j] === "{") depth++;
else if (css[j] === "}") depth--;
j++;
}
rules.push({
selector: css.slice(i, open).trim(),
body: css.slice(open + 1, j - 1),
});
i = j;
}
return rules;
}
/**
* Splits `text` at each `sep` outside parentheses and quotes, trimming the
* parts: the arguments of a CSS function, or the declarations of a rule.
*/
function splitTop(text: string, sep: string): string[] {
const parts: string[] = [];
let depth = 0;
let quote = "";
let start = 0;
for (let i = 0; i < text.length; i++) {
const c = text[i];
if (quote) {
if (c === quote) quote = "";
} else if (c === '"' || c === "'") quote = c;
else if (c === "(") depth++;
else if (c === ")") depth--;
else if (c === sep && depth === 0) {
parts.push(text.slice(start, i).trim());
start = i + 1;
}
}
parts.push(text.slice(start).trim());
return parts.filter((part) => part !== "");
}
/**
* Replaces each call of the CSS function `name` in `value`, innermost last
* first, with what `replace` returns for its arguments.
*/
function replaceCalls(
value: string,
name: string,
replace: (args: string[]) => string,
): string {
for (;;) {
const at = value.lastIndexOf(`${name}(`);
if (at < 0) return value;
let depth = 0;
let end = at + name.length;
do {
if (value[end] === "(") depth++;
else if (value[end] === ")") depth--;
end++;
} while (depth && end < value.length);
const args = splitTop(value.slice(at + name.length + 1, end - 1), ",");
value = value.slice(0, at) + replace(args) + value.slice(end);
}
}
/** Parses a CSS color in the forms the themes write, or throws. */
function parseColor(text: string): Rgba {
const color = parseCssColor(text);
if (!color) throw new Error(`Unreadable color: ${text}`);
return color;
}
/** Writes `color` as `#rrggbb`, or as `rgba()` when it is translucent. */
function formatColor(color: Rgba): string {
if (color.a >= 1) return hexOf(color);
const [r, g, b] = [color.r, color.g, color.b].map(Math.round);
return `rgba(${r},${g},${b},${+color.a.toFixed(3)})`;
}
/** Returns `top` laid over `under`, both straight (not premultiplied) alpha. */
function over(top: Rgba, under: Rgba): Rgba {
const a = top.a + under.a * (1 - top.a);
if (a === 0) return { r: 0, g: 0, b: 0, a: 0 };
const mix = (t: number, u: number) =>
(t * top.a + u * under.a * (1 - top.a)) / a;
return {
r: mix(top.r, under.r),
g: mix(top.g, under.g),
b: mix(top.b, under.b),
a,
};
}
/**
* Computes `color-mix(in srgb, …)` from its arguments, the two colors with
* their optional percentages, as CSS does: premultiplied, and scaled down
* when the percentages sum under 100%.
*/
function colorMix(args: string[]): string {
const [space, ...colors] = args;
if (space !== "in srgb" || colors.length !== 2) {
throw new Error(`Unsupported color-mix: ${args.join(", ")}`);
}
const parsed = colors.map((arg) => {
const pct = arg.match(/\s([\d.]+)%$/);
return {
color: parseColor(pct ? arg.slice(0, pct.index) : arg),
pct: pct ? parseFloat(pct[1]) / 100 : null,
};
});
const [p1, p2] = [
parsed[0].pct ?? 1 - (parsed[1].pct ?? 0.5),
parsed[1].pct ?? 1 - (parsed[0].pct ?? 0.5),
];
const sum = p1 + p2;
const [w1, w2] = [p1 / sum, p2 / sum];
const [c1, c2] = [parsed[0].color, parsed[1].color];
const a = c1.a * w1 + c2.a * w2;
const mix = (x: number, y: number) =>
a === 0 ? 0 : (x * c1.a * w1 + y * c2.a * w2) / a;
return formatColor({
r: mix(c1.r, c2.r),
g: mix(c1.g, c2.g),
b: mix(c1.b, c2.b),
a: a * Math.min(sum, 1),
});
}
/**
* Returns the tokens of `style` in `scheme`, read from the site's stylesheets
* at the time of the call. `styles.ts` reads each style once, at load.
*/
export function readTokens(style: StyleName, scheme: Scheme): ThemeTokens {
const styleSel = style === BASE_STYLE
? `(?:\\[data-style="${style}"\\]|:not\\(\\[data-style\\]\\))`
: `\\[data-style="${style}"\\]`;
const rootRule = new RegExp(
`^(?::root)?(?:${styleSel})?(?:\\[data-color-scheme="${scheme}"\\])?$`,
);
// Later declarations win, as in the cascade: the base tokens, then the
// theme's, then its forced scheme's.
const declared = new Map<string, string>();
for (const source of SOURCES) {
const css = stripComments(Deno.readTextFileSync(
new URL(source.replace("{style}", style), import.meta.url),
));
for (const { selector, body } of topRules(css)) {
if (selector.startsWith("@")) continue;
if (!selector.split(",").some((part) => rootRule.test(part.trim()))) {
continue;
}
for (const declaration of splitTop(body, ";")) {
const colon = declaration.indexOf(":");
const name = declaration.slice(0, colon).trim();
if (!name.startsWith("--")) continue;
declared.set(
name,
declaration.slice(colon + 1).replace(/\s+/g, " ").trim(),
);
}
}
}
const resolved = new Map<string, string>();
const value = (token: string): string => {
const known = resolved.get(token);
if (known !== undefined) return known;
const raw = declared.get(token);
if (raw === undefined) throw new Error(`${style} sets no ${token}`);
let result = replaceCalls(
raw,
"var",
([name, fallback]) =>
declared.has(name) ? value(name) : fallback ?? value(name),
);
result = replaceCalls(
result,
"light-dark",
([light, dark]) => scheme === "light" ? light : dark,
);
result = replaceCalls(result, "color-mix", colorMix);
resolved.set(token, result);
return result;
};
return {
value,
color(token, under) {
const color = parseColor(value(token));
return formatColor(under ? over(color, parseColor(value(under))) : color);
},
stops(token) {
const gradient = value(token).match(/^linear-gradient\((.*)\)$/)?.[1];
if (!gradient) throw new Error(`${style} ${token} is no linear gradient`);
return splitTop(gradient, ",")
.filter((stop) => !/^(?:to\s|[\d.-]+(?:deg|turn|rad)$)/.test(stop))
.map((stop) => formatColor(parseColor(splitTop(stop, " ")[0])));
},
font(token) {
return splitTop(value(token), ",")
.map((family) => family.replace(/["']/g, "").replace(/ Variable$/, ""))
.map((family) => /^[\w-]+$/.test(family) ? family : `'${family}'`)
.join(", ");
},
px(token) {
const length = value(token).match(/^([\d.]+)(?:px)?$/)?.[1];
if (length === undefined) {
throw new Error(`${style} ${token} is no pixel length`);
}
return parseFloat(length);
},
};
}

80
api/mail/welcome.ts Normal file
View File

@@ -0,0 +1,80 @@
/**
* Renders the email that greets a new account. `sendWelcomeEmail`
* (`user-service.ts`) sends it after sign-up, in the instance's style and
* language. It opens with the instance's `GERBEUR_WELCOME_EMAIL_BODY`, or the
* built-in body, then says the digests are on, since they are by default, and
* where to turn them off.
*
* @module
*/
import { FRONTEND_URL, OG_SITE_NAME, WELCOME_EMAIL_BODY } from "../config.ts";
import { i18n } from "./i18n.ts";
import {
EmailKit,
escMd,
lookOf,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/**
* Returns the welcome email of `username`. The instance's body has
* `{{username}}` and `{{site_name}}` filled in; raw HTML in it passes
* through, since it comes from whoever runs the instance, never from a user.
*/
export function renderWelcomeEmail(username: string): RenderedEmail {
// A username may hold a `_`, which Markdown would read as emphasis.
const body = WELCOME_EMAIL_BODY
? WELCOME_EMAIL_BODY
.replaceAll("{{username}}", escMd(username))
.replaceAll("{{site_name}}", escMd(OG_SITE_NAME))
: i18n._({
id: "email.welcome.body",
message:
"# Welcome to {site}!\n\nHi **{username}**,\n\nYour account has been created successfully. Welcome aboard!",
values: { site: escMd(OG_SITE_NAME), username: escMd(username) },
});
const markdown = [
body,
mdLink(
i18n._({
id: "email.welcome.button",
message: "Open {site}",
values: { site: OG_SITE_NAME },
}),
FRONTEND_URL,
),
`*${
escMd(
i18n._({
id: "email.welcome.digests",
message:
"Once a week you'll get a digest of what happened here, and once a month a recap of the most upvoted dumps. You can turn either off in your settings.",
}),
)
}*`,
"---",
escMd(
i18n._({
id: "email.welcome.reason",
message:
"You're getting this email because an account was created on {site} with this address.",
values: { site: OG_SITE_NAME },
}),
),
].join("\n\n");
return new EmailKit(lookOf()).render({
subject: i18n._({
id: "email.welcome.subject",
message: "Welcome to {site}",
values: { site: OG_SITE_NAME },
}),
preheader: i18n._({
id: "email.welcome.preheader",
message: "Your account is ready.",
}),
markdown,
});
}

176
api/main.ts
View File

@@ -1,8 +1,11 @@
/**
* Builds the Oak application and, when run directly, listens on
* {@link LISTEN_HOST}:{@link PORT}. A request goes through the error
* middleware, the OG tags and CORS, then the routers, and last the built
* frontend, which answers every path no router did.
* Builds the Oak application and, when run directly, starts the recurring
* jobs and listens on {@link LISTEN_HOST}:{@link PORT}, having first signed
* the users out of other instances if they may no longer sign in there
* (`FEDERATION.outgoing`). A request goes
* through the error middleware, the OG tags and CORS, then the routers, one
* per resource of the API (`API.md` lists them), and last the built frontend,
* which answers every path no router did.
*
* @module
*/
@@ -11,28 +14,50 @@ import { Application } from "@oak/oak";
import { oakCors } from "@tajpouria/cors";
import attachmentsRouter from "./routes/attachments.ts";
import avatarsRouter from "./routes/avatars.ts";
import categoriesRouter from "./routes/categories.ts";
import chatRouter from "./routes/chat.ts";
import communityRouter from "./routes/community.ts";
import commentsRouter from "./routes/comments.ts";
import dumpsRouter from "./routes/dumps.ts";
import filesRouter from "./routes/files.ts";
import thumbnailsRouter from "./routes/thumbnails.ts";
import usersRouter from "./routes/users.ts";
import avatarsRouter from "./routes/avatars.ts";
import wsRouter from "./routes/ws.ts";
import previewRouter from "./routes/preview.ts";
import bandcampRouter from "./routes/bandcamp.ts";
import youtubeRouter from "./routes/youtube.ts";
import playlistsRouter from "./routes/playlists.ts";
import commentsRouter from "./routes/comments.ts";
import chatRouter from "./routes/chat.ts";
import federationRouter from "./routes/federation.ts";
import followsRouter from "./routes/follows.ts";
import notificationsRouter from "./routes/notifications.ts";
import invitesRouter from "./routes/invites.ts";
import meRouter from "./routes/me.ts";
import notificationsRouter from "./routes/notifications.ts";
import oauthRouter from "./routes/oauth.ts";
import passwordResetsRouter from "./routes/password-resets.ts";
import playlistsRouter from "./routes/playlists.ts";
import remoteRouter from "./routes/remote.ts";
import searchRouter from "./routes/search.ts";
import categoriesRouter from "./routes/categories.ts";
import sessionsRouter from "./routes/sessions.ts";
import thumbnailsRouter from "./routes/thumbnails.ts";
import waveformsRouter from "./routes/waveforms.ts";
import unsubscribeRouter from "./routes/unsubscribe.ts";
import usersRouter from "./routes/users.ts";
import wellKnownRouter from "./routes/well-known.ts";
import wsRouter from "./routes/ws.ts";
import { ALLOWED_ORIGINS, LISTEN_HOST, PORT, PUBLIC_URL } from "./config.ts";
import {
ALLOWED_ORIGINS,
FEDERATION,
LISTEN_HOST,
PORT,
PUBLIC_URL,
} from "./config.ts";
import { errorMiddleware } from "./middleware/error.ts";
import { ogMiddleware } from "./middleware/og.ts";
import { routeStaticFilesFrom } from "./lib/static.ts";
import { startJobs } from "./lib/scheduler.ts";
import { rateLimitSweepJob } from "./lib/rate-limit.ts";
import { digestJob } from "./services/digest-service.ts";
import { communityJob } from "./services/community-service.ts";
import { pingJob, typingExpiryJob } from "./services/ws-service.ts";
import {
outboxJob,
withdrawAllGrants,
} from "./services/identity-provider-service.ts";
/**
* The whole server, exported so the tests' helpers can send it requests
@@ -54,78 +79,37 @@ app.use(async (ctx, next) => {
}
await next();
});
app.use(
dumpsRouter.routes(),
dumpsRouter.allowedMethods(),
);
app.use(
filesRouter.routes(),
filesRouter.allowedMethods(),
);
app.use(
thumbnailsRouter.routes(),
thumbnailsRouter.allowedMethods(),
);
app.use(
attachmentsRouter.routes(),
attachmentsRouter.allowedMethods(),
);
app.use(
usersRouter.routes(),
usersRouter.allowedMethods(),
);
app.use(
avatarsRouter.routes(),
avatarsRouter.allowedMethods(),
);
app.use(
wsRouter.routes(),
wsRouter.allowedMethods(),
);
app.use(
previewRouter.routes(),
previewRouter.allowedMethods(),
);
app.use(
bandcampRouter.routes(),
bandcampRouter.allowedMethods(),
);
app.use(
youtubeRouter.routes(),
youtubeRouter.allowedMethods(),
);
app.use(
playlistsRouter.routes(),
playlistsRouter.allowedMethods(),
);
app.use(
commentsRouter.routes(),
commentsRouter.allowedMethods(),
);
app.use(
chatRouter.routes(),
chatRouter.allowedMethods(),
);
app.use(
followsRouter.routes(),
followsRouter.allowedMethods(),
);
app.use(
notificationsRouter.routes(),
notificationsRouter.allowedMethods(),
);
app.use(
invitesRouter.routes(),
invitesRouter.allowedMethods(),
);
app.use(
searchRouter.routes(),
searchRouter.allowedMethods(),
);
app.use(
categoriesRouter.routes(),
categoriesRouter.allowedMethods(),
);
for (
const router of [
meRouter,
followsRouter,
notificationsRouter,
avatarsRouter,
usersRouter,
sessionsRouter,
passwordResetsRouter,
unsubscribeRouter,
dumpsRouter,
filesRouter,
thumbnailsRouter,
waveformsRouter,
commentsRouter,
playlistsRouter,
categoriesRouter,
searchRouter,
chatRouter,
communityRouter,
invitesRouter,
attachmentsRouter,
remoteRouter,
federationRouter,
oauthRouter,
wellKnownRouter,
wsRouter,
]
) {
app.use(router.routes(), router.allowedMethods());
}
app.use(routeStaticFilesFrom([
`${Deno.cwd()}/dist`,
`${Deno.cwd()}/public`,
@@ -142,6 +126,20 @@ app.addEventListener(
);
if (import.meta.main) {
if (!FEDERATION.outgoing) {
const withdrawn = await withdrawAllGrants();
if (withdrawn > 0) {
console.log(`Signed users out of other instances (${withdrawn} grants)`);
}
}
startJobs([
digestJob,
communityJob,
pingJob,
typingExpiryJob,
rateLimitSweepJob,
outboxJob,
]);
await app.listen({ hostname: LISTEN_HOST, port: PORT });
}

2
api/middleware/auth.ts
View File

@@ -28,7 +28,7 @@ export interface AuthState extends State {
/**
* The context type of a handler behind {@link authMiddleware}. Pass the route
* path, as in `AuthContext<"/:userId/role">`, to keep `ctx.params` typed.
* path, as in `AuthContext<"/:key/role">`, to keep `ctx.params` typed.
*/
export type AuthContext<
R extends string = string,

93
api/middleware/og.ts
View File

@@ -2,15 +2,22 @@
* Gives links to the site rich previews in chat apps and social sites, which
* don't run the app. It answers a link to a dump, a profile or a public
* playlist with `index.html` carrying that page's Open Graph and Twitter card
* tags. main.ts installs it before the routers.
* tags, and a link by a former name with a 301 to the current one. main.ts
* installs it before the routers.
*
* @module
*/
import { Context, Next } from "@oak/oak";
import { getDump } from "../services/dump-service.ts";
import { getUserByUsername } from "../services/user-service.ts";
import { getPlaylistById } from "../services/playlist-service.ts";
import { getDumpByPath } from "../services/dump-service.ts";
import { resolveUsername } from "../services/user-service.ts";
import { getPlaylistByPath } from "../services/playlist-service.ts";
import {
dumpPath,
parsePagePath,
playlistPath,
userPath,
} from "../../shared/paths.ts";
import { FRONTEND_URL, OG_SITE_NAME, PUBLIC_URL } from "../config.ts";
import { loadIndexHtml } from "../lib/index-html.ts";
@@ -83,17 +90,23 @@ async function findIndexHtml(): Promise<string | null> {
return null;
}
const DUMP_RE = /^\/dumps\/([^/]+)$/;
/** Matches a profile, and its tabs under `/~/<tab>`. */
const USER_RE = /^\/users\/([^/]+)(?:\/~\/[^/]+)?$/;
const PLAYLIST_RE = /^\/playlists\/([^/]+)$/;
/** Answers `canonical` with a 301, keeping the request's query string. */
function redirect(ctx: Context, canonical: string): void {
ctx.response.status = 301;
// Revalidated each time: a slug taken back would otherwise loop through a
// redirect the browser kept.
ctx.response.headers.set("Cache-Control", "no-cache");
ctx.response.redirect(`${FRONTEND_URL}${canonical}${ctx.request.url.search}`);
}
/**
* Serves `index.html` with the tags of the dump, user or public playlist a
* path names, and passes the request on when it names none that exists or
* the visitor may see. The page's URL is on `FRONTEND_URL` and the images' on
* `PUBLIC_URL`, not on the request's origin, which a reverse proxy may have
* rewritten to an internal host or plain http.
* path names (`parsePagePath`), and passes the request on when it names none
* that exists or the visitor may see. A path naming one by a former name, a
* user's or a slug, or in another case, gets a 301 to its current path. The
* page's URL is on `FRONTEND_URL` and the images' on `PUBLIC_URL`, not on the
* request's origin, which a reverse proxy may have rewritten to an internal
* host or plain http.
*/
export async function ogMiddleware(
ctx: Context,
@@ -101,25 +114,25 @@ export async function ogMiddleware(
): Promise<unknown> {
const { pathname } = ctx.request.url;
if (pathname.startsWith("/api/") || pathname.includes(".")) {
return await next();
}
// A file's path never starts with `@`, so parsePagePath passes it on, while
// a remote user's has dots in it: `/@bob@music.example`.
if (pathname.startsWith("/api/")) return await next();
const page = parsePagePath(pathname);
if (!page || (page.kind !== "user" && page.edit)) return await next();
const origin = PUBLIC_URL;
const pageUrl = `${FRONTEND_URL}${pathname}`;
let meta: OGMeta | null = null;
let canonical: string | null = null;
const dumpMatch = pathname.match(DUMP_RE);
const userMatch = pathname.match(USER_RE);
const playlistMatch = pathname.match(PLAYLIST_RE);
if (dumpMatch) {
try {
const dump = getDump(dumpMatch[1]);
const user = resolveUsername(page.username);
if (page.kind === "dump") {
const dump = getDumpByPath(user.id, page.slug);
canonical = dumpPath(dump);
let imageUrl: string | undefined;
if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) {
imageUrl = `${origin}/api/files/${dump.id}`;
imageUrl = `${origin}/api/dumps/${dump.id}/file`;
} else if (dump.richContent?.thumbnailUrl) {
imageUrl = dump.richContent.thumbnailUrl;
}
@@ -130,41 +143,41 @@ export async function ogMiddleware(
description: dump.comment,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
imageIsWide: !!imageUrl,
url: pageUrl,
url: `${FRONTEND_URL}${canonical}`,
};
} catch { /* Unknown, or private. */ }
} else if (userMatch) {
try {
const user = getUserByUsername(userMatch[1]);
} else if (page.kind === "user") {
canonical = [userPath(user.username), ...page.rest].join("/");
const imageUrl = user.avatarMime
? `${origin}/api/avatars/${user.id}`
? `${origin}/api/users/${user.id}/avatar`
: undefined;
meta = {
title: user.username,
description: user.description,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
url: pageUrl,
url: `${FRONTEND_URL}${canonical}`,
};
} catch { /* Unknown. */ }
} else if (playlistMatch) {
try {
const playlist = getPlaylistById(playlistMatch[1]);
// getPlaylistById doesn't check visibility, unlike getDump.
} else {
const playlist = getPlaylistByPath(user.id, page.slug);
// getPlaylistByPath doesn't check visibility, unlike getDumpByPath.
if (playlist.isPublic) {
canonical = playlistPath(playlist);
const imageUrl = playlist.imageMime
? `${origin}/api/playlists/${playlist.id}/image`
? `${origin}/api/playlists/${playlist.id}/cover`
: undefined;
meta = {
title: playlist.title,
description: playlist.description,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
url: pageUrl,
url: `${FRONTEND_URL}${canonical}`,
};
}
} catch { /* Unknown. */ }
}
} catch { /* Unknown, or private. */ }
if (!meta) return await next();
if (!meta || !canonical) return await next();
if (canonical !== pathname.replace(/\/+$/, "")) {
return redirect(ctx, canonical);
}
const html = await findIndexHtml();
if (!html) return await next();

9
api/model/category.ts
View File

@@ -6,11 +6,16 @@
*/
import type { SQLOutputValue } from "node:sqlite";
import type { Category } from "../../shared/category.ts";
import {
type Category,
type CategoryKind,
isCategoryKind,
} from "../../shared/category.ts";
/** Returned by the category service's `SELECT`s on `categories`. */
export interface CategoryRow {
id: string;
kind: CategoryKind;
slug: string;
name: string;
/** Sorts the categories, lowest first. */
@@ -24,6 +29,7 @@ export interface CategoryRow {
export function isCategoryRow(obj: unknown): obj is CategoryRow {
return !!obj && typeof obj === "object" &&
"id" in obj && typeof obj.id === "string" &&
"kind" in obj && isCategoryKind(obj.kind) &&
"slug" in obj && typeof obj.slug === "string" &&
"name" in obj && typeof obj.name === "string" &&
"position" in obj && typeof obj.position === "number" &&
@@ -36,6 +42,7 @@ export function isCategoryRow(obj: unknown): obj is CategoryRow {
export function categoryRowToApi(row: CategoryRow): Category {
return {
id: row.id,
kind: row.kind,
slug: row.slug,
name: row.name,
position: row.position,

6
api/model/chat.ts
View File

@@ -26,6 +26,8 @@ export interface ChatMessageRow {
reply_to_author: string | null;
/** Cut to chat-service's `REPLY_SNIPPET_LEN` characters. */
reply_to_body: string | null;
/** 1 or 0. */
mentions_everyone: number;
[key: string]: SQLOutputValue;
}
@@ -50,7 +52,8 @@ export function isChatMessageRow(obj: unknown): obj is ChatMessageRow {
"reply_to_author" in obj &&
(typeof obj.reply_to_author === "string" || obj.reply_to_author === null) &&
"reply_to_body" in obj &&
(typeof obj.reply_to_body === "string" || obj.reply_to_body === null);
(typeof obj.reply_to_body === "string" || obj.reply_to_body === null) &&
"mentions_everyone" in obj && typeof obj.mentions_everyone === "number";
}
/**
@@ -69,5 +72,6 @@ export function chatMessageRowToApi(row: ChatMessageRow): ChatMessage {
replyToId: row.reply_to_id ?? undefined,
replyToAuthor: row.reply_to_author ?? undefined,
replyToBody: row.reply_to_body ?? undefined,
mentionsEveryone: row.mentions_everyone === 1 || undefined,
};
}

22
api/model/dump.ts
View File

@@ -23,7 +23,8 @@ export interface DumpRow {
user_id: string;
created_at: string;
updated_at: string | null;
slug: string | null;
/** Unique among its poster's dumps, and set on every row since 0021. */
slug: string;
/** The link as the poster submitted it. */
url: string | null;
/** {@link RichContent} as JSON. */
@@ -34,6 +35,8 @@ export interface DumpRow {
vote_count: number;
/** Computed by a subquery, which counts the comments that aren't deleted. */
comment_count: number;
/** The poster's current name, computed by a subquery. */
owner_username: string;
is_private: number;
custom_thumbnail_mime: string | null;
/** Lets a query select more columns, such as a join's. */
@@ -53,17 +56,19 @@ const DUMP_COLUMNS_ALIASED = DUMP_COLUMNS.split(", ").map((c) => `d.${c}`)
/**
* Selects a whole {@link DumpRow}, the stored columns and the computed
* `comment_count`.
* `comment_count` and `owner_username`.
*/
export const DUMP_SELECT_COLUMNS: string = `${DUMP_COLUMNS},
(SELECT COUNT(*) FROM comments WHERE dump_id = dumps.id AND deleted = 0) as comment_count`;
(SELECT COUNT(*) FROM comments WHERE dump_id = dumps.id AND deleted = 0) as comment_count,
(SELECT username FROM users WHERE id = dumps.user_id) as owner_username`;
/**
* Selects a whole {@link DumpRow} in a query that aliases `dumps` as `d` to
* join it with other tables.
*/
export const DUMP_SELECT_COLUMNS_ALIASED: string = `${DUMP_COLUMNS_ALIASED},
(SELECT COUNT(*) FROM comments WHERE dump_id = d.id AND deleted = 0) as comment_count`;
(SELECT COUNT(*) FROM comments WHERE dump_id = d.id AND deleted = 0) as comment_count,
(SELECT username FROM users WHERE id = d.user_id) as owner_username`;
/**
* Checks whether `obj`, a row from a query, is a {@link DumpRow}. It checks
@@ -81,7 +86,7 @@ export function isDumpRow(obj: unknown): obj is DumpRow {
"created_at" in obj && typeof obj.created_at === "string" &&
"updated_at" in obj &&
(typeof obj.updated_at === "string" || obj.updated_at === null) &&
"slug" in obj && (typeof obj.slug === "string" || obj.slug === null) &&
"slug" in obj && typeof obj.slug === "string" &&
"url" in obj && (typeof obj.url === "string" || obj.url === null) &&
"rich_content" in obj &&
(typeof obj.rich_content === "string" || obj.rich_content === null) &&
@@ -93,6 +98,7 @@ export function isDumpRow(obj: unknown): obj is DumpRow {
(typeof obj.file_size === "number" || obj.file_size === null) &&
"vote_count" in obj && typeof obj.vote_count === "number" &&
"comment_count" in obj && typeof obj.comment_count === "number" &&
"owner_username" in obj && typeof obj.owner_username === "string" &&
"is_private" in obj && typeof obj.is_private === "number" &&
"custom_thumbnail_mime" in obj &&
(typeof obj.custom_thumbnail_mime === "string" ||
@@ -108,10 +114,11 @@ export function dumpRowToApi(row: DumpRow): Dump {
id: row.id,
kind: row.kind as "url" | "file",
title: row.title,
slug: row.slug ?? undefined,
slug: row.slug,
url: row.url ?? undefined,
comment: row.comment ?? undefined,
userId: row.user_id,
ownerUsername: row.owner_username,
createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
richContent: row.rich_content
@@ -136,7 +143,7 @@ export function dumpApiToRow(dump: Dump): DumpRow {
id: dump.id,
kind: dump.kind,
title: dump.title,
slug: dump.slug ?? null,
slug: dump.slug,
comment: dump.comment ?? null,
user_id: dump.userId,
created_at: dump.createdAt.toISOString(),
@@ -148,6 +155,7 @@ export function dumpApiToRow(dump: Dump): DumpRow {
file_size: dump.fileSize ?? null,
vote_count: dump.voteCount,
comment_count: dump.commentCount,
owner_username: dump.ownerUsername,
is_private: dump.isPrivate ? 1 : 0,
custom_thumbnail_mime: dump.thumbnailMime ?? null,
};

2
api/model/notification.ts
View File

@@ -1,6 +1,6 @@
/**
* Turns the rows of `notifications`, once checked, into the {@link
* Notification} that `GET /api/notifications` answers.
* Notification} that `GET /api/me/notifications` answers.
*
* @module
*/

20
api/model/playlist.ts
View File

@@ -9,15 +9,18 @@ import type { SQLOutputValue } from "node:sqlite";
import type { Playlist } from "../../shared/playlist.ts";
/**
* Returned by the playlist and follow services' `SELECT`s on `playlists`. A
* query may add `dump_count` and `owner_username` columns, which
* {@link playlistRowToApi} carries over when present.
* Returned by the playlist and follow services' `SELECT`s on `playlists`,
* which join the owner's name as `owner_username`. A query may add a
* `dump_count` column, which {@link playlistRowToApi} carries over when
* present.
*/
export interface PlaylistRow {
id: string;
user_id: string;
title: string;
slug: string | null;
/** Unique among its owner's playlists, and set on every row since 0021. */
slug: string;
owner_username: string;
description: string | null;
is_public: number;
created_at: string;
@@ -34,7 +37,8 @@ export function isPlaylistRow(
"id" in obj && typeof obj.id === "string" &&
"user_id" in obj && typeof obj.user_id === "string" &&
"title" in obj && typeof obj.title === "string" &&
"slug" in obj && (typeof obj.slug === "string" || obj.slug === null) &&
"slug" in obj && typeof obj.slug === "string" &&
"owner_username" in obj && typeof obj.owner_username === "string" &&
"description" in obj &&
(typeof obj.description === "string" || obj.description === null) &&
"is_public" in obj && typeof obj.is_public === "number" &&
@@ -51,15 +55,13 @@ export function playlistRowToApi(row: PlaylistRow): Playlist {
id: row.id,
userId: row.user_id,
title: row.title,
slug: row.slug ?? undefined,
slug: row.slug,
description: row.description ?? undefined,
isPublic: Boolean(row.is_public),
createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
imageMime: row.image_mime ?? undefined,
dumpCount: typeof row.dump_count === "number" ? row.dump_count : undefined,
ownerUsername: typeof row.owner_username === "string"
? row.owner_username
: undefined,
ownerUsername: row.owner_username,
};
}

66
api/model/settings.ts Normal file
View File

@@ -0,0 +1,66 @@
/**
* Checks the rows `settings-service.ts` reads from `user_settings`, and turns
* them into the {@link UserSettings} the settings endpoints answer.
*
* @module
*/
import type { SQLOutputValue } from "node:sqlite";
import type { UserSettings } from "../../shared/settings.ts";
import { isColorScheme, isStyleName } from "../../shared/theme.ts";
/**
* Stored by `updateSettings` and read back by `getSettings`. A user without a
* row has the defaults.
*/
export interface UserSettingsRow {
user_id: string;
/** A style name, or null for the instance's. */
style: string | null;
color_scheme: string | null;
/** 0 or 1. */
digest_weekly: number;
/** 0 or 1. */
digest_monthly: number;
updated_at: string;
[key: string]: SQLOutputValue;
}
/**
* Names the column of each setting, for the queries that pick users by one,
* as `digest-service.ts` does by its digest's switch.
*/
export const SETTINGS_COLUMNS: Record<keyof UserSettings, string> = {
style: "style",
colorScheme: "color_scheme",
digestWeekly: "digest_weekly",
digestMonthly: "digest_monthly",
};
/**
* Checks whether `obj`, a row `getSettings` selected, is a
* {@link UserSettingsRow}.
*/
export function isUserSettingsRow(obj: unknown): obj is UserSettingsRow {
return !!obj && typeof obj === "object" &&
"user_id" in obj && typeof obj.user_id === "string" &&
"style" in obj && (obj.style === null || typeof obj.style === "string") &&
"color_scheme" in obj &&
(obj.color_scheme === null || typeof obj.color_scheme === "string") &&
"digest_weekly" in obj && typeof obj.digest_weekly === "number" &&
"digest_monthly" in obj && typeof obj.digest_monthly === "number" &&
"updated_at" in obj && typeof obj.updated_at === "string";
}
/**
* Converts a row to the {@link UserSettings} the client gets. A style or
* scheme the code no longer knows, as after one is removed, reads as unset.
*/
export function settingsRowToApi(row: UserSettingsRow): UserSettings {
return {
style: isStyleName(row.style) ? row.style : null,
colorScheme: isColorScheme(row.color_scheme) ? row.color_scheme : null,
digestWeekly: row.digest_weekly === 1,
digestMonthly: row.digest_monthly === 1,
};
}

23
api/model/user.ts
View File

@@ -19,8 +19,22 @@ import {
* it carries the password hash.
*/
export interface User extends SelfUser {
/** From `hashPassword`, as `hash.salt`. */
passwordHash: string;
/**
* From `hashPassword`, as `hash.salt`, or null for an account from another
* instance ({@link isRemoteUser}), which signs in there and never with a
* password here. The row stores that as `''`.
*/
passwordHash: string | null;
}
/**
* Checks whether `user`'s account lives on another instance, whose user signs
* in here through federation (`federation-service.ts`). Their password,
* email, avatar and name are that instance's to change, so the routes that
* change them here refuse.
*/
export function isRemoteUser(user: User): boolean {
return user.passwordHash === null;
}
/**
@@ -49,6 +63,7 @@ export function toSelfUser({ passwordHash: _, ...user }: User): SelfUser {
export interface UserRow {
id: string;
username: string;
/** `''` for an account from another instance, which has no password. */
password_hash: string;
role: Role;
created_at: string;
@@ -97,7 +112,7 @@ export function userRowToApi(row: UserRow): User {
return {
id: row.id,
username: row.username,
passwordHash: row.password_hash,
passwordHash: row.password_hash || null,
role: row.role,
createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
@@ -118,7 +133,7 @@ export function userApiToRow(user: User): UserRow {
return {
id: user.id,
username: user.username,
password_hash: user.passwordHash,
password_hash: user.passwordHash ?? "",
role: user.role,
created_at: user.createdAt.toISOString(),
updated_at: user.updatedAt?.toISOString() ?? null,

37
api/remote/bandcamp-streams.ts
View File

@@ -1,6 +1,6 @@
/**
* Resolves a Bandcamp page to its streamable tracks when the native player
* starts it, for `GET /api/bandcamp/tracks`. The stream URLs are signed and
* starts it, for `GET /api/remote/bandcamp`. The stream URLs are signed and
* expire 24 hours after the page is fetched, so unlike the embed URL they
* can't be stored in the dump's preview. The browser plays them from bcbits
* directly.
@@ -9,19 +9,16 @@
*/
import { APIException } from "../lib/api-exception.ts";
import { TtlCache } from "../lib/ttl-cache.ts";
import { fetchWithTimeout, isValidHttpUrl, readTextCapped } from "./fetch.ts";
import { BANDCAMP_HOST_RE, parseTralbum } from "./bandcamp-tralbum.ts";
import type { Tralbum } from "../../shared/player.ts";
/**
* Half the signatures' life, so a cached URL still has at least 12 hours left
* when it is handed out.
* Lives half the signatures' life, so a cached URL still has at least 12
* hours left when it is handed out.
*/
const TTL_MS = 12 * 60 * 60 * 1000;
/** The oldest write goes first. */
const MAX_ENTRIES = 200;
const cache = new Map<string, Tralbum>();
const cache = new TtlCache<Tralbum>(12 * 3_600_000, 200);
/**
* Returns `url` without its query, fragment and trailing slashes, and with its
@@ -38,25 +35,6 @@ function cacheKey(url: string): string {
}
}
function readCache(key: string): Tralbum | null {
const hit = cache.get(key);
if (!hit) return null;
if (Date.now() - hit.resolvedAt > TTL_MS) {
cache.delete(key);
return null;
}
return hit;
}
function writeCache(key: string, value: Tralbum): void {
// Map preserves insertion order, so the first key is the oldest write.
if (cache.size >= MAX_ENTRIES) {
const oldest = cache.keys().next().value;
if (oldest !== undefined) cache.delete(oldest);
}
cache.set(key, value);
}
/**
* Returns the tracklist of the Bandcamp page `pageUrl`, from the cache when it
* was resolved in the last 12 hours. `force` skips the cache: the player sets
@@ -87,7 +65,7 @@ export async function resolveBandcamp(
const key = cacheKey(pageUrl);
if (!force) {
const hit = readCache(key);
const hit = cache.get(key);
if (hit) return hit;
}
@@ -115,6 +93,5 @@ export async function resolveBandcamp(
throw new APIException(404, "No streamable tracks on this page");
}
writeCache(key, parsed);
return parsed;
return cache.set(key, parsed);
}

27
api/remote/bandcamp-tralbum.ts
View File

@@ -8,6 +8,7 @@
*/
import type { Tralbum, TralbumTrack } from "../../shared/player.ts";
import { decodeHtmlEntities } from "./html.ts";
/**
* Matches the hostnames `bandcamp-streams.ts` accepts: bandcamp.com and its
@@ -16,30 +17,6 @@ import type { Tralbum, TralbumTrack } from "../../shared/player.ts";
*/
export const BANDCAMP_HOST_RE = /(?:^|\.)bandcamp\.com$/;
/**
* Decodes HTML entities in a single pass. `html.ts` decodes `&amp;` before
* `&quot;`, which turns a title's literal `&quot;` (escaped as `&amp;quot;`)
* into a bare `"` that breaks the JSON.
*/
function decodeEntitiesOnce(input: string): string {
const named: Record<string, string> = {
quot: '"',
apos: "'",
amp: "&",
lt: "<",
gt: ">",
nbsp: " ",
};
return input.replace(
/&(?:#(\d+)|#[xX]([0-9a-fA-F]+)|([a-zA-Z]+));/g,
(match, dec: string, hex: string, name: string) => {
if (dec) return String.fromCodePoint(Number(dec));
if (hex) return String.fromCodePoint(parseInt(hex, 16));
return named[name.toLowerCase()] ?? match;
},
);
}
/** Returns the URL of the artwork `artId`; undefined when it isn't numeric. */
function artworkUrl(artId: unknown): string | undefined {
if (typeof artId !== "number" && typeof artId !== "string") return undefined;
@@ -60,7 +37,7 @@ export function parseTralbum(html: string, sourceUrl: string): Tralbum | null {
let raw: Record<string, unknown>;
try {
raw = JSON.parse(decodeEntitiesOnce(match[2]));
raw = JSON.parse(decodeHtmlEntities(match[2]));
} catch {
return null;
}

80
api/remote/fetch.ts
View File

@@ -7,7 +7,11 @@
* @module
*/
import { isPrivateHostname, resolvePublicAddress } from "./public-address.ts";
import {
bareHostname,
isPrivateHostname,
resolvePublicAddress,
} from "./public-address.ts";
import { isSelfUrl } from "../lib/self-url.ts";
const FETCH_HEADERS = {
@@ -43,7 +47,7 @@ async function fetchViaCurl(
"--proto",
"=http,https",
"--resolve",
`${url.hostname.replace(/^\[(.*)\]$/, "$1")}:${port}:${pinned}`,
`${bareHostname(url)}:${port}:${pinned}`,
"--max-time",
String(Math.ceil(timeoutMs / 1000)),
// curl fails on a larger file rather than stopping at a cap, so this is
@@ -171,7 +175,7 @@ function declaredCharset(contentType: string, bytes: Uint8Array): string {
/**
* Reads the body of `res` as text, up to `maxBytes`, in the charset the page
* declares; the rest is cancelled, never downloaded. The providers and the
* declares; the rest is cancelled, never downloaded. {@link fetchPage} and the
* Bandcamp and YouTube resolvers read pages through it.
*/
export async function readTextCapped(
@@ -231,6 +235,33 @@ async function fetchOnce(
});
}
/**
* Fetches `url` as {@link fetchWithTimeout} does, and also returns the URL the
* last hop answered from, which {@link fetchPage} resolves the page's links
* against.
*/
async function fetchFollowing(
url: string,
timeoutMs: number,
): Promise<{ res: Response; url: URL }> {
let current = new URL(url);
for (let hop = 0;; hop++) {
if (current.protocol !== "http:" && current.protocol !== "https:") {
throw new TypeError(`Unsupported protocol ${current.protocol}`);
}
const address = await resolvePublicAddress(current);
const res = await fetchOnce(current, address, timeoutMs);
const location = res.headers.get("location");
if (res.status < 300 || res.status >= 400 || !location) {
return { res, url: current };
}
await res.body?.cancel();
if (hop >= MAX_REDIRECTS) throw new TypeError("Too many redirects");
current = new URL(location, current);
}
}
/**
* Fetches a user-supplied URL, following up to five redirects, each of which
* must resolve to a public address too. `timeoutMs` bounds each hop, body
@@ -245,20 +276,41 @@ export async function fetchWithTimeout(
url: string,
timeoutMs = 5000,
): Promise<Response> {
let current = new URL(url);
for (let hop = 0;; hop++) {
if (current.protocol !== "http:" && current.protocol !== "https:") {
throw new TypeError(`Unsupported protocol ${current.protocol}`);
}
const address = await resolvePublicAddress(current);
const res = await fetchOnce(current, address, timeoutMs);
const location = res.headers.get("location");
if (res.status < 300 || res.status >= 400 || !location) return res;
return (await fetchFollowing(url, timeoutMs)).res;
}
/**
* Returned by {@link fetchPage} to the link preview providers.
*/
export interface FetchedPage {
/**
* The URL the page was served from, after redirects: the base its relative
* links resolve against.
*/
url: string;
/** The page's text, up to {@link MAX_PAGE_BYTES}; missing when it isn't HTML. */
html?: string;
}
/**
* Fetches a page for its link preview, through {@link fetchWithTimeout}. A
* response that isn't HTML is cancelled unread.
*
* @throws {Error} When the page answers with an error status, so that an error
* page never becomes a preview; and as {@link fetchWithTimeout} does.
*/
export async function fetchPage(url: string): Promise<FetchedPage> {
const { res, url: finalUrl } = await fetchFollowing(url, 5000);
if (!res.ok) {
await res.body?.cancel();
if (hop >= MAX_REDIRECTS) throw new TypeError("Too many redirects");
current = new URL(location, current);
throw new Error(`${finalUrl} answered ${res.status}`);
}
const contentType = res.headers.get("content-type") ?? "";
if (!/^(text\/html|application\/xhtml\+xml)\b/i.test(contentType)) {
await res.body?.cancel();
return { url: finalUrl.href };
}
return { url: finalUrl.href, html: await readTextCapped(res) };
}
/**

271
api/remote/gerbeur-instance.ts Normal file
View File

@@ -0,0 +1,271 @@
/**
* Talks to other gerbeur instances for federated sign-in: it reads their
* metadata and public keys, trades a code for an identity token, delivers
* events, and fetches a user's avatar. `federation-service.ts`,
* `identity-provider-service.ts` and `federation-peer-service.ts` reach other
* instances only through it.
*
* An identity is only as safe as the connection it came over, so unlike
* `fetch.ts` it never falls back to skipping certificate checks, and follows
* no redirect. It reaches only https instances on public addresses, unless
* `GERBEUR_FEDERATION_ALLOW_PRIVATE` lets two instances on one machine talk.
*
* @module
*/
import type { JSONWebKeySet } from "@panva/jose";
import {
type InstanceMetadata,
isInstanceMetadata,
} from "../../shared/federation.ts";
import {
FEDERATION_ALLOW_PRIVATE,
MAX_IMAGE_SIZE_BYTES,
OG_SITE_NAME,
} from "../config.ts";
import { resolvePublicAddress } from "./public-address.ts";
import { readBytesCapped, readTextCapped } from "./fetch.ts";
import { TtlCache } from "../lib/ttl-cache.ts";
/** How long a request to another instance may take, body included. */
const TIMEOUT_MS = 8000;
/** Caps a metadata document, a key set or a token response. */
const MAX_JSON_BYTES = 64 * 1024;
const USER_AGENT = `gerbeur (${OG_SITE_NAME}; federation)`;
/**
* Keeps each instance's metadata and keys for 10 minutes, so that a sign-in
* or an event doesn't fetch them each time. A key set is refetched sooner
* when a token names a key it lacks ({@link fetchKeys} with `fresh`), at most
* once a minute: anyone can send a token naming a key that doesn't exist.
*/
const metadataCache = new TtlCache<InstanceMetadata>(10 * 60_000, 500);
const keysCache = new TtlCache<JSONWebKeySet>(10 * 60_000, 500);
const keysRefetched = new TtlCache<true>(60_000, 500);
/** Keeps which issuer an instance's pages belong to ({@link resolveIssuer}). */
const issuerCache = new TtlCache<string>(10 * 60_000, 500);
/**
* Checks that `url` may be reached: https on a public address, or anything
* under `GERBEUR_FEDERATION_ALLOW_PRIVATE`.
*
* @throws {Error} When it may not.
*/
export async function assertReachable(url: URL): Promise<void> {
if (FEDERATION_ALLOW_PRIVATE) {
if (url.protocol !== "https:" && url.protocol !== "http:") {
throw new Error(`Unsupported protocol ${url.protocol}`);
}
return;
}
if (url.protocol !== "https:") {
throw new Error(`Refusing to reach ${url.origin} without https`);
}
await resolvePublicAddress(url);
}
/**
* Makes one request to another instance, after {@link assertReachable}, with
* the deadline covering the body. The caller reads or cancels the body.
*/
async function request(url: string, init: RequestInit = {}): Promise<Response> {
const target = new URL(url);
await assertReachable(target);
return await fetch(target, {
...init,
redirect: "error",
signal: AbortSignal.timeout(TIMEOUT_MS),
headers: {
"User-Agent": USER_AGENT,
"Accept": "application/json",
...init.headers,
},
});
}
/**
* Returns the JSON body of `res`, up to {@link MAX_JSON_BYTES}.
*
* @throws {Error} When the status isn't a success or the body isn't JSON.
*/
async function readJson(res: Response, what: string): Promise<unknown> {
if (!res.ok) {
await res.body?.cancel();
throw new Error(`${what} answered ${res.status}`);
}
return JSON.parse(await readTextCapped(res, MAX_JSON_BYTES));
}
/**
* Returns the metadata `origin` serves at `/.well-known/openid-configuration`,
* whichever issuer it names.
*
* @throws {Error} When it can't be reached or isn't a gerbeur instance's.
*/
async function metadataAt(origin: string): Promise<InstanceMetadata> {
const body = await readJson(
await request(`${origin}/.well-known/openid-configuration`),
origin,
);
if (!isInstanceMetadata(body)) {
throw new Error(`${origin} is not a gerbeur instance this one can trust`);
}
return body;
}
/**
* Returns the metadata of the instance at `origin`, its issuer, cached. It
* must name `origin` as its issuer and speak this protocol's version.
*
* @throws {Error} When it can't be reached or isn't a gerbeur instance.
*/
export async function fetchMetadata(origin: string): Promise<InstanceMetadata> {
const hit = metadataCache.get(origin);
if (hit) return hit;
const body = await metadataAt(origin);
if (body.issuer !== origin) {
throw new Error(`${origin} names itself ${body.issuer}`);
}
return metadataCache.set(origin, body);
}
/**
* Returns the issuer of the instance at `origin`, cached: `origin` itself, or,
* when `origin` is where the instance's pages are and its API lives at
* another, the issuer the pages' metadata names, whose own metadata must name
* those pages back as where its `/authorize` is. People know an instance by
* the address in their browser, which a sign-in form or the trust list may be
* given; a handle and the stored identities name the issuer.
*
* @throws {Error} When `origin` can't be reached, isn't a gerbeur instance,
* or names an issuer that doesn't name it back.
*/
export async function resolveIssuer(origin: string): Promise<string> {
const hit = issuerCache.get(origin);
if (hit) return hit;
if (metadataCache.get(origin)) return origin;
const body = await metadataAt(origin);
if (body.issuer === origin) {
metadataCache.set(origin, body);
return issuerCache.set(origin, origin);
}
const pages =
new URL((await fetchMetadata(body.issuer)).authorization_endpoint)
.origin;
if (pages !== origin) {
throw new Error(`${origin} names ${body.issuer}, whose pages are ${pages}`);
}
return issuerCache.set(origin, body.issuer);
}
/**
* Returns the public keys of the instance at `origin`, cached unless `fresh`
* asks to refetch them and they weren't in the last minute.
*
* @throws {Error} When they can't be fetched.
*/
export async function fetchKeys(
origin: string,
fresh = false,
): Promise<JSONWebKeySet> {
const refetch = fresh && !keysRefetched.get(origin);
const hit = refetch ? undefined : keysCache.get(origin);
if (hit) return hit;
if (refetch) keysRefetched.set(origin, true);
const { jwks_uri } = await fetchMetadata(origin);
const body = await readJson(await request(jwks_uri), `${origin}'s keys`);
if (
!body || typeof body !== "object" ||
!Array.isArray((body as { keys?: unknown }).keys)
) {
throw new Error(`${origin}'s keys are not a key set`);
}
return keysCache.set(origin, body as JSONWebKeySet);
}
/**
* Trades `code` for an identity token at the token endpoint of the instance
* at `origin`, and returns the token, unverified.
*
* @throws {Error} When the instance refuses or answers no token.
*/
export async function exchangeCode(
origin: string,
params: {
code: string;
clientId: string;
redirectUri: string;
codeVerifier: string;
},
): Promise<string> {
const { token_endpoint } = await fetchMetadata(origin);
const body = await readJson(
await request(token_endpoint, {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "authorization_code",
code: params.code,
client_id: params.clientId,
redirect_uri: params.redirectUri,
code_verifier: params.codeVerifier,
}),
}),
`${origin}'s token endpoint`,
);
const idToken = (body as { id_token?: unknown } | null)?.id_token;
if (typeof idToken !== "string") {
throw new Error(`${origin} answered no identity token`);
}
return idToken;
}
/**
* Sends the signed event `event` to the events endpoint of the instance at
* `origin`, and returns whether it took it. A 4xx other than 429 means it never
* will, which the caller treats as delivered.
*
* @throws {Error} When it can't be reached, or answers 429 or a 5xx.
*/
export async function postEvent(
origin: string,
event: string,
): Promise<boolean> {
const { gerbeur_events_endpoint } = await fetchMetadata(origin);
const res = await request(gerbeur_events_endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ event }),
});
await res.body?.cancel();
if (res.ok) return true;
if (res.status === 429 || res.status >= 500) {
throw new Error(`${origin}'s events endpoint answered ${res.status}`);
}
return false;
}
/**
* Returns the image at `url`, a user's avatar on their home instance, up to
* the size an avatar may be.
*
* @throws {Error} When it can't be fetched.
*/
export async function fetchAvatar(url: string): Promise<Uint8Array> {
const res = await request(url, { headers: { "Accept": "image/*" } });
if (!res.ok) {
await res.body?.cancel();
throw new Error(`The avatar at ${url} answered ${res.status}`);
}
return await readBytesCapped(res, MAX_IMAGE_SIZE_BYTES + 1);
}
/** Empties the caches, for the tests' reset between cases. */
export function clearInstanceCaches(): void {
metadataCache.clear();
issuerCache.clear();
keysCache.clear();
keysRefetched.clear();
}

734
api/remote/html.ts
View File

@@ -1,78 +1,230 @@
/**
* Reads a link preview's metadata out of a page's HTML with regular
* expressions, without parsing the document. The generic provider uses most of
* it; the Bandcamp, SoundCloud and YouTube providers read only OG tags. Each
* function returns undefined when the page doesn't say.
* Reads a link preview's metadata out of a page's HTML. {@link parsePage}
* scans the page once for the tags that carry it (`<meta>`, `<link>`, `<img>`,
* `<title>` and JSON-LD scripts) without building a document, and returns a
* {@link Page} that the providers' `genericPreview` picks from. Every field is
* undefined when the page doesn't say.
*
* @module
*/
import { hexOf, parseCssColor } from "../lib/css-color.ts";
/**
* Decodes the five XML entities and numeric ones in `str`. `&amp;` goes first,
* so `&amp;quot;` decodes twice; `bandcamp-tralbum.ts`, which decodes JSON,
* has its own single-pass decoder.
* The named entities decoded besides numeric ones: the XML five, and those
* page titles and descriptions actually use, Latin-1 letters and typography.
*/
function decodeHtmlEntities(str: string): string {
return str
.replace(/&amp;/gi, "&")
.replace(/&lt;/gi, "<")
.replace(/&gt;/gi, ">")
.replace(/&quot;/gi, '"')
.replace(/&apos;/gi, "'")
.replace(/&#(\d+);/g, (_, dec) => String.fromCodePoint(Number(dec)))
.replace(
/&#x([0-9a-f]+);/gi,
(_, hex) => String.fromCodePoint(parseInt(hex, 16)),
const NAMED_ENTITIES: Record<string, string> = {
amp: "&",
lt: "<",
gt: ">",
quot: '"',
apos: "'",
nbsp: " ",
aelig: "æ",
AElig: "Æ",
oelig: "œ",
OElig: "Œ",
szlig: "ß",
oslash: "ø",
Oslash: "Ø",
eth: "ð",
ETH: "Ð",
thorn: "þ",
THORN: "Þ",
hellip: "…",
lsquo: "‘",
rsquo: "’",
sbquo: "‚",
ldquo: "“",
rdquo: "”",
bdquo: "„",
laquo: "«",
raquo: "»",
lsaquo: "‹",
rsaquo: "›",
ndash: "–",
mdash: "—",
bull: "•",
middot: "·",
dagger: "†",
Dagger: "‡",
prime: "′",
Prime: "″",
euro: "€",
pound: "£",
yen: "¥",
cent: "¢",
copy: "©",
reg: "®",
trade: "™",
deg: "°",
plusmn: "±",
times: "×",
divide: "÷",
micro: "µ",
para: "¶",
sect: "§",
iexcl: "¡",
iquest: "¿",
ordf: "ª",
ordm: "º",
sup2: "²",
sup3: "³",
frac14: "¼",
frac12: "½",
frac34: "¾",
shy: "­",
ensp: " ",
emsp: " ",
thinsp: " ",
zwnj: "‌",
zwj: "‍",
};
// The accented letters, `&eacute;` and the like, composed from their letter
// and accent rather than listed one by one.
for (
const [accent, mark, letters] of [
["grave", "̀", "aeiouAEIOU"],
["acute", "́", "aeiouyAEIOUY"],
["circ", "̂", "aeiouAEIOU"],
["tilde", "̃", "anoANO"],
["uml", "̈", "aeiouyAEIOUY"],
["ring", "̊", "aA"],
["cedil", "̧", "cC"],
]
) {
for (const letter of letters) {
NAMED_ENTITIES[letter + accent] = (letter + mark).normalize("NFC");
}
}
/** Returns the character of code point `n`, or undefined when there is none. */
function fromCodePoint(n: number): string | undefined {
if (n <= 0 || n > 0x10ffff || (n >= 0xd800 && n <= 0xdfff)) {
return undefined;
}
return String.fromCodePoint(n);
}
/**
* Decodes the entities in `str` in a single pass, so `&amp;quot;` reads
* `&quot;`. An entity it doesn't know, or a code point that doesn't exist, is
* left as written. Also used by `bandcamp-tralbum.ts` on the JSON a page
* carries in an attribute.
*/
export function decodeHtmlEntities(str: string): string {
return str.replace(
/&(?:#(\d{1,8})|#[xX]([0-9a-fA-F]{1,6})|([a-zA-Z][a-zA-Z0-9]{1,31}));/g,
(match, dec?: string, hex?: string, name?: string) => {
if (dec) return fromCodePoint(Number(dec)) ?? match;
if (hex) return fromCodePoint(parseInt(hex, 16)) ?? match;
return NAMED_ENTITIES[name!] ?? NAMED_ENTITIES[name!.toLowerCase()] ??
match;
},
);
}
/**
* Returns the `content` of the first `<meta>` tag whose `attr` equals `value`,
* in whichever order the attributes come. Each tag is matched on its own, so
* the capture can't run into the next tag.
* Returns `raw` resolved against `base` as an absolute http(s) URL; undefined
* for a `data:` URI, another scheme or a URL that doesn't parse. Every URL a
* page names goes through it, so `/og.png` and `//cdn/og.png` work.
*/
function extractMetaContent(
html: string,
attr: string,
value: string,
export function absoluteHttpUrl(
raw: string | undefined,
base: string,
): string | undefined {
const escaped = value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const keyRe = new RegExp(`\\b${attr}=["']${escaped}["']`, "i");
const contentRe = /\bcontent=(["'])([\s\S]*?)\1/i;
const tagRe = /<meta\b[^>]*>/gi;
let m: RegExpExecArray | null;
while ((m = tagRe.exec(html)) !== null) {
const tag = m[0];
if (!keyRe.test(tag)) continue;
const content = contentRe.exec(tag);
if (content) return decodeHtmlEntities(content[2]);
}
if (!raw?.trim()) return undefined;
try {
const url = new URL(raw.trim(), base);
return url.protocol === "http:" || url.protocol === "https:"
? url.href
: undefined;
} catch {
return undefined;
}
}
/** Returns the content of `<meta property="og:<tag>">`, entities decoded. */
export function extractOgTag(
html: string,
tag: string,
): string | undefined {
return extractMetaContent(html, "property", `og:${tag}`);
/** A tag's attributes by lowercase name, values decoded; the first one wins. */
type Attrs = Map<string, string>;
/**
* Matches the start of a tag the scan reads, or of a comment. Quoted values
* may hold `>`, but no tag spans a `<`, so each attempt stops at the next one
* and the scan stays linear in the page's size.
*/
const TAG_RE =
/<(meta|link|img|title|script)\b((?:[^<>"']|"[^"<]*"|'[^'<]*')*)>|<!--/gi;
/** Matches one attribute: unquoted, single- or double-quoted, or bare. */
const ATTR_RE =
/([^\s"'<>/=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+)))?/g;
function parseAttrs(source: string): Attrs {
const attrs: Attrs = new Map();
for (const m of source.matchAll(ATTR_RE)) {
const name = m[1].toLowerCase();
if (attrs.has(name)) continue;
attrs.set(name, decodeHtmlEntities(m[2] ?? m[3] ?? m[4] ?? ""));
}
return attrs;
}
/** What {@link scanTags} found, in page order. */
interface Scan {
metas: Attrs[];
links: Attrs[];
imgs: Attrs[];
/** The raw text of the first `<title>`. */
title?: string;
/** The bodies of the `application/ld+json` scripts. */
jsonLd: string[];
}
/**
* Returns the content of `<meta name="<name>">`, entities decoded. Twitter
* card tags use `name`, not `property`.
* Collects the tags {@link parsePage} reads in one pass. Comments and the
* bodies of scripts and the title are skipped over, so a tag written inside
* them isn't read; one left open runs to the end of the page, as in a browser.
*/
export function extractMetaName(
html: string,
name: string,
): string | undefined {
return extractMetaContent(html, "name", name);
function scanTags(html: string): Scan {
const scan: Scan = { metas: [], links: [], imgs: [], jsonLd: [] };
const re = new RegExp(TAG_RE);
let m: RegExpExecArray | null;
while ((m = re.exec(html)) !== null) {
const tag = m[1]?.toLowerCase();
if (!tag) {
const end = html.indexOf("-->", re.lastIndex);
if (end < 0) break;
re.lastIndex = end + 3;
continue;
}
const attrs = parseAttrs(m[2]);
if (tag === "meta") scan.metas.push(attrs);
else if (tag === "link") scan.links.push(attrs);
else if (tag === "img") scan.imgs.push(attrs);
else {
const close = new RegExp(`</${tag}\\s*>`, "gi");
close.lastIndex = re.lastIndex;
const end = close.exec(html);
if (!end) break;
const body = html.slice(re.lastIndex, end.index);
if (tag === "title") scan.title ??= body;
else if (
attrs.get("type")?.trim().toLowerCase() === "application/ld+json"
) {
scan.jsonLd.push(body);
}
re.lastIndex = close.lastIndex;
}
}
return scan;
}
/** Returns the text of the `<title>` element, trimmed and entities decoded. */
export function extractPageTitle(html: string): string | undefined {
const match = html.match(/<title[^>]*>([^<]+)<\/title>/i);
return match ? decodeHtmlEntities(match[1].trim()) : undefined;
/** Returns `value` trimmed, or undefined when nothing is left. */
function nonBlank(value: string | undefined): string | undefined {
const trimmed = value?.trim();
return trimmed ? trimmed : undefined;
}
/**
@@ -80,21 +232,15 @@ export function extractPageTitle(html: string): string | undefined {
* {@link normalizeCssColor}. Sites often ship one per color scheme with the
* dark one first, so the unscoped tag wins, then the light one, then any.
*/
export function extractThemeColor(html: string): string | undefined {
const tagRe = /<meta\b[^>]*>/gi;
const contentRe = /\bcontent=(["'])([\s\S]*?)\1/i;
const mediaRe = /\bmedia=(["'])([\s\S]*?)\1/i;
function themeColorOf(metas: Attrs[]): string | undefined {
let unscoped: string | undefined;
let light: string | undefined;
let any: string | undefined;
let m: RegExpExecArray | null;
while ((m = tagRe.exec(html)) !== null) {
const tag = m[0];
if (!/\bname=["']theme-color["']/i.test(tag)) continue;
const content = contentRe.exec(tag)?.[2];
for (const meta of metas) {
if (meta.get("name")?.toLowerCase() !== "theme-color") continue;
const content = nonBlank(meta.get("content"));
if (!content) continue;
const media = mediaRe.exec(tag)?.[2];
const media = meta.get("media");
if (!media) unscoped ??= content;
else if (/light/i.test(media)) light ??= content;
any ??= content;
@@ -102,239 +248,179 @@ export function extractThemeColor(html: string): string | undefined {
return unscoped ?? light ?? any;
}
/** Returns the lowercase tokens of a `<link>`'s `rel`. */
function relTokens(link: Attrs): string[] {
return (link.get("rel") ?? "").toLowerCase().split(/\s+/);
}
/**
* Returns the `color` of `<link rel="mask-icon">`, Safari's pinned-tab icon,
* as written. The generic provider's last resort for a brand color.
* as written. The generic preview's last resort for a brand color.
*/
export function extractMaskIconColor(html: string): string | undefined {
const linkRe = /<link[^>]+>/gi;
let m: RegExpExecArray | null;
while ((m = linkRe.exec(html)) !== null) {
const tag = m[0];
if (!/\brel=["'][^"']*mask-icon[^"']*["']/i.test(tag)) continue;
const color = /\bcolor=(["'])([\s\S]*?)\1/i.exec(tag)?.[2];
function maskIconColorOf(links: Attrs[]): string | undefined {
for (const link of links) {
if (!relTokens(link).includes("mask-icon")) continue;
const color = nonBlank(link.get("color"));
if (color) return color;
}
return undefined;
}
/**
* The named colors sites actually put in `theme-color`; the rest of CSS's names
* aren't worth carrying.
*/
const NAMED_COLORS: Record<string, string> = {
black: "#000000",
white: "#ffffff",
red: "#ff0000",
green: "#008000",
blue: "#0000ff",
yellow: "#ffff00",
orange: "#ffa500",
purple: "#800080",
gray: "#808080",
grey: "#808080",
silver: "#c0c0c0",
maroon: "#800000",
navy: "#000080",
teal: "#008080",
olive: "#808000",
lime: "#00ff00",
aqua: "#00ffff",
cyan: "#00ffff",
fuchsia: "#ff00ff",
magenta: "#ff00ff",
};
function clampByte(n: number): number {
return Math.max(0, Math.min(255, Math.round(n)));
}
/**
* Returns `raw` as a lowercase `#rrggbb`, from a hex, `rgb()`, `rgba()` or
* common named color; undefined for any other form. The generic provider
* stores it as the preview's `accentColor`, a tint over the app's own surface,
* so alpha is dropped.
*/
export function normalizeCssColor(raw: string | undefined): string | undefined {
if (!raw) return undefined;
const value = raw.trim().toLowerCase();
const named = NAMED_COLORS[value];
if (named) return named;
const hex = /^#([0-9a-f]{3,8})$/.exec(value)?.[1];
if (hex) {
if (hex.length === 3 || hex.length === 4) {
const [r, g, b] = [...hex.slice(0, 3)];
return `#${r}${r}${g}${g}${b}${b}`;
}
if (hex.length === 6 || hex.length === 8) return `#${hex.slice(0, 6)}`;
return undefined;
}
const rgb = /^rgba?\(([^)]+)\)$/.exec(value)?.[1];
if (rgb) {
const parts = rgb.split(/[\s,/]+/).filter(Boolean).slice(0, 3);
if (parts.length !== 3) return undefined;
const bytes = parts.map((p) => {
const n = parseFloat(p);
if (!Number.isFinite(n)) return NaN;
return clampByte(p.endsWith("%") ? (n / 100) * 255 : n);
});
if (bytes.some(Number.isNaN)) return undefined;
return `#${bytes.map((b) => b.toString(16).padStart(2, "0")).join("")}`;
}
return undefined;
}
/**
* Returned to the generic provider by {@link extractJsonLd}, which ranks each
* field below the page's OG and Twitter tags.
*/
export interface JsonLdResult {
/** From `name`, or else `headline`. */
title?: string;
description?: string;
/**
* From `image`, or else `thumbnailUrl` or `thumbnail`, and only when it is
* an absolute URL.
*/
thumbnailUrl?: string;
}
function ldString(v: unknown): string | undefined {
if (typeof v === "string" && v.trim()) return v.trim();
if (Array.isArray(v) && typeof v[0] === "string" && v[0].trim()) {
return v[0].trim();
}
return undefined;
}
function ldImage(v: unknown): string | undefined {
if (
typeof v === "string" &&
(v.startsWith("http://") || v.startsWith("https://"))
) return v;
if (Array.isArray(v)) return ldImage(v[0]);
if (v && typeof v === "object") {
const o = v as Record<string, unknown>;
return ldImage(o.url ?? o.contentUrl);
}
return undefined;
}
function ldExtractNode(data: unknown): JsonLdResult {
if (Array.isArray(data)) {
for (const item of data) {
const r = ldExtractNode(item);
if (r.title || r.thumbnailUrl) return r;
}
return {};
}
if (!data || typeof data !== "object") return {};
const o = data as Record<string, unknown>;
if (o["@graph"]) return ldExtractNode(o["@graph"]);
return {
title: ldString(o.name ?? o.headline),
description: ldString(o.description),
thumbnailUrl: ldImage(o.image ?? o.thumbnailUrl ?? o.thumbnail),
};
}
/**
* Returns the first JSON-LD node in `html` that has a title or an image,
* looking through `@graph` and arrays; empty when there is none. An image may
* be a URL, a list of them or an `ImageObject`. Blocks that aren't valid JSON
* are skipped.
*/
export function extractJsonLd(html: string): JsonLdResult {
const pattern =
/<script[^>]+type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi;
let match: RegExpExecArray | null;
while ((match = pattern.exec(html)) !== null) {
try {
const result = ldExtractNode(JSON.parse(match[1]));
if (result.title || result.thumbnailUrl) return result;
} catch { /* invalid JSON: try the next block */ }
}
return {};
}
/**
* Returns the absolute URL of the first `<img>` whose declared width and height
* are both at least `minSize` pixels; an image without them doesn't count.
* Data URIs are skipped.
*/
export function extractLargeImage(
html: string,
baseUrl: string,
minSize = 200,
): string | undefined {
const imgPattern = /<img[^>]+>/gi;
let match: RegExpExecArray | null;
while ((match = imgPattern.exec(html)) !== null) {
const tag = match[0];
const src = /\bsrc=["']([^"']+)["']/i.exec(tag)?.[1];
if (!src || src.startsWith("data:")) continue;
const w = parseInt(/\bwidth=["']?(\d+)/i.exec(tag)?.[1] ?? "0");
const h = parseInt(/\bheight=["']?(\d+)/i.exec(tag)?.[1] ?? "0");
if (w >= minSize && h >= minSize) {
try {
return new URL(src, baseUrl).toString();
} catch {
continue;
}
}
}
return undefined;
}
/**
* Returns the absolute URL of the page's largest icon, among its `icon` and
* `apple-touch-icon` links, by declared `sizes`; the first one when none
* declares a size. The generic provider stores it as the preview's
* `faviconUrl`, which the thumbnail placeholder draws.
* declares a size. A `mask-icon` is no candidate: it is a one-color shape for
* Safari to tint, black when drawn as is.
*
* An SVG icon wins over any size: it declares none, yet scales to any.
*/
export function extractBestIcon(
html: string,
baseUrl: string,
): string | undefined {
const linkRe = /<link[^>]+>/gi;
const relRe = /\brel=["']([^"']+)["']/i;
const hrefRe = /\bhref=["']([^"']+)["']/i;
const sizesRe = /\bsizes=["']([^"']+)["']/i;
const candidates: { href: string; area: number }[] = [];
let m: RegExpExecArray | null;
while ((m = linkRe.exec(html)) !== null) {
const tag = m[0];
const rel = relRe.exec(tag)?.[1] ?? "";
if (!/\bicon\b/i.test(rel) && !/apple-touch-icon/i.test(rel)) continue;
const href = hrefRe.exec(tag)?.[1];
function bestIconOf(links: Attrs[], baseUrl: string): string | undefined {
let best: { href: string; area: number } | undefined;
for (const link of links) {
const rel = relTokens(link);
if (
!rel.includes("icon") && !rel.includes("apple-touch-icon") &&
!rel.includes("apple-touch-icon-precomposed")
) continue;
const href = absoluteHttpUrl(link.get("href"), baseUrl);
if (!href) continue;
const sizesStr = sizesRe.exec(tag)?.[1] ?? "";
const sm = sizesStr.match(/(\d+)x(\d+)/i);
const sizes = (link.get("sizes") ?? "").toLowerCase();
const isSvg = /\.svg(\?|$)/i.test(href) ||
/\btype=["']image\/svg\+xml["']/i.test(tag);
const area = isSvg
? Number.MAX_SAFE_INTEGER
: sm
? parseInt(sm[1]) * parseInt(sm[2])
: 0;
link.get("type")?.toLowerCase() === "image/svg+xml" ||
sizes === "any";
const area = isSvg ? Number.MAX_SAFE_INTEGER : Math.max(
0,
...[...sizes.matchAll(/(\d+)x(\d+)/g)].map(([, w, h]) => +w * +h),
);
if (!best || area > best.area) best = { href, area };
}
return best?.href;
}
/**
* Returned by the JSON-LD reading of {@link parsePage}, which the generic
* preview ranks below the page's OG and Twitter tags.
*/
export interface JsonLdResult {
/** From `name`, or else `headline`. */
title?: string;
description?: string;
/** From `image`, or else `thumbnailUrl` or `thumbnail`. */
thumbnailUrl?: string;
}
/**
* The JSON-LD types that describe the site around a page rather than the
* page itself. Their name is the site's or the company's, so a node of one of
* them is read only when no other node has a title or an image.
*/
const SITE_LD_TYPES = new Set([
"WebSite",
"Organization",
"Corporation",
"NewsMediaOrganization",
"Person",
"BreadcrumbList",
"SearchAction",
"ImageObject",
"SiteNavigationElement",
"WPHeader",
"WPFooter",
]);
function ldString(v: unknown): string | undefined {
if (typeof v === "string") return nonBlank(v);
if (Array.isArray(v) && typeof v[0] === "string") return nonBlank(v[0]);
return undefined;
}
function ldImage(v: unknown, baseUrl: string): string | undefined {
if (typeof v === "string") return absoluteHttpUrl(v, baseUrl);
if (Array.isArray(v)) return ldImage(v[0], baseUrl);
if (v && typeof v === "object") {
const o = v as Record<string, unknown>;
return ldImage(o.url ?? o.contentUrl, baseUrl);
}
return undefined;
}
/** Lists every node of `data`, through arrays and `@graph`, in order. */
function ldNodes(data: unknown): Record<string, unknown>[] {
if (Array.isArray(data)) return data.flatMap(ldNodes);
if (!data || typeof data !== "object") return [];
const o = data as Record<string, unknown>;
return o["@graph"] ? ldNodes(o["@graph"]) : [o];
}
/**
* Returns the first JSON-LD node about the page that has a title or an image,
* else the first about its site ({@link SITE_LD_TYPES}); empty when there is
* none. An image may be a URL, a list of them or an `ImageObject`. Blocks that
* aren't valid JSON are skipped.
*/
function jsonLdOf(blocks: string[], baseUrl: string): JsonLdResult {
let fallback: JsonLdResult | undefined;
for (const block of blocks) {
let data: unknown;
try {
candidates.push({ href: new URL(href, baseUrl).toString(), area });
data = JSON.parse(block);
} catch {
continue;
}
for (const node of ldNodes(data)) {
const result: JsonLdResult = {
title: ldString(node.name ?? node.headline),
description: ldString(node.description),
thumbnailUrl: ldImage(
node.image ?? node.thumbnailUrl ?? node.thumbnail,
baseUrl,
),
};
if (!result.title && !result.thumbnailUrl) continue;
const types = [node["@type"]].flat();
if (!types.some((t) => typeof t === "string" && SITE_LD_TYPES.has(t))) {
return result;
}
fallback ??= result;
}
}
return fallback ?? {};
}
if (candidates.length === 0) return undefined;
candidates.sort((a, b) => b.area - a.area);
return candidates[0].href;
/**
* Returns the source of an `<img>`: its `src`, or for a lazy-loaded image
* whose `src` is a `data:` placeholder, its `data-src` or `data-lazy-src`.
*/
function imageSource(img: Attrs, baseUrl: string): string | undefined {
const src = img.get("src")?.trim();
return src && !src.startsWith("data:")
? absoluteHttpUrl(src, baseUrl)
: absoluteHttpUrl(img.get("data-src") ?? img.get("data-lazy-src"), baseUrl);
}
/** Returns an `<img>`'s declared `name` dimension in pixels; 0 when none. */
function declaredSize(img: Attrs, name: "width" | "height"): number {
return parseInt(img.get(name) ?? "") || 0;
}
/**
* Returns the absolute URL of the first `<img>` whose declared width and height
* are both at least `minSize` pixels; an image without them doesn't count.
*/
function largeImageOf(
imgs: Attrs[],
baseUrl: string,
minSize = 200,
): string | undefined {
for (const img of imgs) {
if (
declaredSize(img, "width") < minSize ||
declaredSize(img, "height") < minSize
) continue;
const src = imageSource(img, baseUrl);
if (src) return src;
}
return undefined;
}
/** Filename words that mark an image as page furniture rather than content. */
@@ -354,6 +440,11 @@ const UI_IMAGE_KEYWORDS = new Set([
"pixel",
"bg",
"background",
"placeholder",
"spacer",
"blank",
"lazy",
"loading",
]);
/**
@@ -371,25 +462,86 @@ function isUiImage(src: string): boolean {
/**
* Returns the absolute URL of the first `<img>` that looks like content rather
* than a logo, icon or other page furniture. Data URIs are skipped. The generic
* provider's last resort for a thumbnail.
* than a logo, icon, tracking pixel or other page furniture: one declared
* smaller than 50 pixels either way doesn't count. The generic preview's last
* resort for a thumbnail.
*/
export function extractFirstContentImage(
html: string,
function firstContentImageOf(
imgs: Attrs[],
baseUrl: string,
): string | undefined {
const imgPattern = /<img[^>]+>/gi;
let match: RegExpExecArray | null;
while ((match = imgPattern.exec(html)) !== null) {
const tag = match[0];
const src = /\bsrc=["']([^"']+)["']/i.exec(tag)?.[1];
if (!src || src.startsWith("data:")) continue;
if (isUiImage(src)) continue;
try {
return new URL(src, baseUrl).toString();
} catch {
continue;
}
for (const img of imgs) {
const w = declaredSize(img, "width");
const h = declaredSize(img, "height");
if ((w && w < 50) || (h && h < 50)) continue;
const src = imageSource(img, baseUrl);
if (src && !isUiImage(src)) return src;
}
return undefined;
}
/**
* Returned by {@link parsePage}: what a page says about itself, entities
* decoded, values trimmed and URLs absolute.
*/
export interface Page {
/**
* Returns the `content` of the first `<meta>` whose `property` or `name` is
* `key`, in any case: sites write `og:` tags under either attribute, and
* Twitter's too.
*/
meta(key: string): string | undefined;
/** The text of the `<title>` element. */
title?: string;
jsonLd: JsonLdResult;
/** The `theme-color`, as written: pass it to {@link normalizeCssColor}. */
themeColor?: string;
/** The `color` of the Safari pinned-tab icon, as written. */
maskIconColor?: string;
/** The largest icon the page links, which the thumbnail placeholder draws. */
bestIcon?: string;
/** The first image declared at least 200 pixels both ways. */
largeImage?: string;
/** The first image that doesn't look like page furniture. */
firstContentImage?: string;
}
/**
* Reads `html`, served from `pageUrl`, in one pass. The generic preview and
* the providers built on it read a page through it.
*/
export function parsePage(html: string, pageUrl: string): Page {
const scan = scanTags(html);
const metas = new Map<string, string>();
for (const meta of scan.metas) {
const content = nonBlank(meta.get("content"));
if (!content) continue;
for (const attr of ["property", "name"]) {
const key = meta.get(attr)?.trim().toLowerCase();
if (key && !metas.has(key)) metas.set(key, content);
}
}
return {
meta: (key) => metas.get(key.toLowerCase()),
title: nonBlank(scan.title && decodeHtmlEntities(scan.title)),
jsonLd: jsonLdOf(scan.jsonLd, pageUrl),
themeColor: themeColorOf(scan.metas),
maskIconColor: maskIconColorOf(scan.links),
bestIcon: bestIconOf(scan.links, pageUrl),
largeImage: largeImageOf(scan.imgs, pageUrl),
firstContentImage: firstContentImageOf(scan.imgs, pageUrl),
};
}
/**
* Returns `raw` as a lowercase `#rrggbb`, from a hex, `rgb()`, `rgba()` or
* common named color; undefined for any other form. The generic preview
* stores it as the preview's `accentColor`, a tint over the app's own surface,
* so alpha is dropped, and a fully transparent one is no color at all.
*/
export function normalizeCssColor(raw: string | undefined): string | undefined {
const color = raw ? parseCssColor(raw) : undefined;
return color && color.a > 0 ? hexOf(color) : undefined;
}

4
api/remote/image-proxy.ts
View File

@@ -1,7 +1,7 @@
/**
* Shows the thumbnails a browser can't load from their own site: those on plain
* http, which an https page would block, and those behind hotlink protection.
* It fetches and caches the images `GET /api/proxy-image` serves, which the
* It fetches and caches the images `GET /api/remote/image` serves, which the
* frontend routes a thumbnail through when it is on http or when loading it
* directly failed.
*
@@ -11,7 +11,7 @@
import { fetchWithTimeout, readBytesCapped } from "./fetch.ts";
/**
* Returned to `GET /api/proxy-image` by {@link fetchProxiedImage}, which sends
* Returned to `GET /api/remote/image` by {@link fetchProxiedImage}, which sends
* it back under `contentType`.
*/
export interface ProxiedImage {

48
api/remote/providers/bandcamp.ts
View File

@@ -1,21 +1,41 @@
/**
* Previews Bandcamp links from their page's OG tags, for
* `rich-content-service.ts`.
* Previews Bandcamp links from their page, for `rich-content-service.ts`.
*
* @module
*/
import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts";
import { parsePage } from "../html.ts";
import { fetchPage } from "../fetch.ts";
import { BANDCAMP_HOST_RE } from "../bandcamp-tralbum.ts";
import { genericPreview } from "./generic.ts";
/**
* Returns the embed player's URL a page's `og:video` names, over https; only
* a player on bandcamp.com counts, since the frontend frames it.
*/
function embeddedPlayerUrl(raw: string | undefined): string | undefined {
if (!raw) return undefined;
try {
const url = new URL(raw);
if (
(url.protocol !== "https:" && url.protocol !== "http:") ||
url.hostname !== "bandcamp.com" ||
!url.pathname.startsWith("/EmbeddedPlayer/")
) return undefined;
url.protocol = "https:";
return url.href;
} catch {
return undefined;
}
}
/**
* Claims bandcamp.com and its subdomains, where artists have their pages, and
* takes the embed player's URL from the page's `og:video`. The native player
* fetches the page's streams itself at play time, through
* `GET /api/bandcamp/tracks`.
* `GET /api/remote/bandcamp`.
*/
export const bandcampProvider: RichContentProvider = {
name: "bandcamp",
@@ -29,24 +49,16 @@ export const bandcampProvider: RichContentProvider = {
},
async fetch(url: string): Promise<RichContent> {
const res = await fetchWithTimeout(url);
const contentType = res.headers.get("content-type") ?? "";
if (!contentType.startsWith("text/html")) {
await res.body?.cancel();
const fetched = await fetchPage(url);
if (fetched.html === undefined) {
return { type: "bandcamp", siteName: "Bandcamp", url };
}
const html = await readTextCapped(res);
const page = parsePage(fetched.html, fetched.url);
return {
...genericPreview(page, url, fetched.url),
type: "bandcamp",
siteName: "Bandcamp",
url,
title: extractOgTag(html, "title"),
description: extractOgTag(html, "description"),
thumbnailUrl: extractOgTag(html, "image"),
embedUrl: extractOgTag(html, "video") ?? undefined,
embedUrl: embeddedPlayerUrl(page.meta("og:video")),
};
},
};

151
api/remote/providers/generic.ts
View File

@@ -1,95 +1,92 @@
/**
* Previews any page no other provider claims, for `rich-content-service.ts`,
* from whatever metadata its HTML carries.
* from whatever metadata its HTML carries. Its reading of a page,
* {@link genericPreview}, is also the base of the Bandcamp, SoundCloud and
* YouTube channel previews.
*
* @module
*/
import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts";
import { siteHost } from "../../../shared/url.ts";
import {
extractBestIcon,
extractFirstContentImage,
extractJsonLd,
extractLargeImage,
extractMaskIconColor,
extractMetaName,
extractOgTag,
extractPageTitle,
extractThemeColor,
absoluteHttpUrl,
normalizeCssColor,
type Page,
parsePage,
} from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts";
import { fetchPage } from "../fetch.ts";
/** Returns the path of `url` without its trailing slashes, `/` for the root. */
function pathOf(url: string): string {
return new URL(url).pathname.replace(/\/+$/, "") || "/";
}
/**
* Matches every URL, so it comes last in the service's list. Each field falls
* back from OG tags to Twitter tags, JSON-LD and the markup itself. The icon
* and brand color feed the placeholder the frontend draws when there is no
* thumbnail.
* Strips a trailing ` | Site`, ` – Site` or the like from a `<title>`, which
* names the page and then its site; the site is in the preview's badge.
*/
export const genericProvider: RichContentProvider = {
name: "generic",
function withoutSiteSuffix(title: string, siteName: string): string {
const escaped = siteName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const suffix = new RegExp(`\\s+[|–—\\-·•]\\s+${escaped}\\s*$`, "i");
return title.replace(suffix, "") || title;
}
matches(_url: string): boolean {
return true;
},
async fetch(url: string): Promise<RichContent> {
const res = await fetchWithTimeout(url);
const contentType = res.headers.get("content-type") ?? "";
if (!contentType.startsWith("text/html")) {
await res.body?.cancel();
return { type: "generic", url };
}
const html = await readTextCapped(res);
const ld = extractJsonLd(html);
// An og:url naming another page (often the home page) means the OG tags
// are the site's defaults, not this page's: its title and image are
// skipped.
const ogUrl = extractOgTag(html, "url");
const useOg = !ogUrl || (() => {
/**
* Returns the preview of the page `url` that `page` read, served from
* `finalUrl` after redirects. Each field falls back from OG tags to Twitter
* tags, JSON-LD and the markup itself. The icon and brand color feed the
* placeholder the frontend draws when there is no thumbnail.
*/
export function genericPreview(
page: Page,
url: string,
finalUrl: string,
): RichContent {
// An og:url naming the home page from another page means the OG tags are
// the site's defaults, not this page's: their title and image are skipped.
// Any other og:url is the page's canonical address, often with a slug the
// link left out, and its tags are the page's.
const ogUrl = page.meta("og:url");
let useOg = true;
try {
const ogPath = new URL(ogUrl).pathname.replace(/\/+$/, "") || "/";
const pagePath = new URL(url).pathname.replace(/\/+$/, "") || "/";
return ogPath === pagePath;
} catch {
return true;
}
})();
useOg = !ogUrl || pathOf(new URL(ogUrl, finalUrl).href) !== "/" ||
pathOf(finalUrl) === "/";
} catch { /* an og:url that doesn't parse names no other page */ }
const title = (useOg ? extractOgTag(html, "title") : undefined) ??
extractMetaName(html, "twitter:title") ??
ld.title ??
extractPageTitle(html);
const ogSiteName = page.meta("og:site_name");
const pageTitle = page.title && ogSiteName
? withoutSiteSuffix(page.title, ogSiteName)
: page.title;
const title = (useOg ? page.meta("og:title") : undefined) ??
page.meta("twitter:title") ??
page.jsonLd.title ??
pageTitle;
const siteName = extractOgTag(html, "site_name") ??
new URL(url).hostname.replace(/^www\./, "");
const description = extractOgTag(html, "description") ??
extractMetaName(html, "twitter:description") ??
ld.description ??
extractMetaName(html, "description");
const description = page.meta("og:description") ??
page.meta("twitter:description") ??
page.jsonLd.description ??
page.meta("description");
// Never the favicon: a 16×16 icon cropped into a thumbnail looks broken,
// and the placeholder drawn instead shows the icon properly.
const thumbnailUrl = (useOg ? extractOgTag(html, "image") : undefined) ??
extractMetaName(html, "twitter:image") ??
ld.thumbnailUrl ??
extractLargeImage(html, url) ??
extractFirstContentImage(html, url);
// and the placeholder drawn instead shows the icon properly. A tag naming
// no usable URL falls through to the next.
const thumbnailUrl =
(useOg ? absoluteHttpUrl(page.meta("og:image"), finalUrl) : undefined) ??
absoluteHttpUrl(page.meta("twitter:image"), finalUrl) ??
page.jsonLd.thumbnailUrl ??
page.largeImage ??
page.firstContentImage;
// `/favicon.ico` is a guess. When it doesn't load, the placeholder shows
// the site's initial.
const faviconUrl = extractBestIcon(html, url) ??
`${new URL(url).origin}/favicon.ico`;
const faviconUrl = page.bestIcon ?? `${new URL(finalUrl).origin}/favicon.ico`;
const accentColor = normalizeCssColor(
extractThemeColor(html) ??
extractMetaName(html, "msapplication-TileColor") ??
extractMaskIconColor(html),
page.themeColor ??
page.meta("msapplication-TileColor") ??
page.maskIconColor,
);
return {
@@ -100,7 +97,25 @@ export const genericProvider: RichContentProvider = {
thumbnailUrl,
faviconUrl,
accentColor,
siteName,
siteName: ogSiteName ?? siteHost(url),
};
}
/** Matches every URL, so it comes last in the service's list. */
export const genericProvider: RichContentProvider = {
name: "generic",
matches(_url: string): boolean {
return true;
},
async fetch(url: string): Promise<RichContent> {
const fetched = await fetchPage(url);
if (fetched.html === undefined) return { type: "generic", url };
return genericPreview(
parsePage(fetched.html, fetched.url),
url,
fetched.url,
);
},
};

7
api/remote/providers/provider.ts
View File

@@ -18,9 +18,10 @@ export interface RichContentProvider {
matches(url: string): boolean;
/**
* Returns the preview of `url`, which ends up in the dump's `rich_content` or
* the create form's preview. It rejects when the page can't be fetched, and
* the service then falls back to a stub. A page that isn't HTML yields a
* preview without metadata.
* the create form's preview. It rejects when the page can't be fetched or
* answers with an error status, and the service then falls back to a stub.
* A page that isn't HTML yields a preview without metadata. The service
* tidies what it returns, so a provider leaves text as the page wrote it.
*/
fetch(url: string): Promise<RichContent>;
}

73
api/remote/providers/self.ts
View File

@@ -1,23 +1,21 @@
/**
* Previews links to this site's own dump, user and playlist pages from the
* database, for `rich-content-service.ts`. No request goes out, so this works
* even when the site runs on a private host that `fetchWithTimeout` refuses.
* Previews links to this site's own dump, user and playlist pages, as
* `parsePagePath` reads them, from the database, for
* `rich-content-service.ts`. No request goes out, so this works even when the
* site runs on a private host that `fetchWithTimeout` refuses.
*
* @module
*/
import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts";
import { getDump } from "../../services/dump-service.ts";
import { getUserByUsername } from "../../services/user-service.ts";
import { getPlaylistById } from "../../services/playlist-service.ts";
import { getDumpByPath } from "../../services/dump-service.ts";
import { resolveUsername } from "../../services/user-service.ts";
import { getPlaylistByPath } from "../../services/playlist-service.ts";
import { APIException } from "../../lib/api-exception.ts";
import { PUBLIC_URL, THEME_COLOR } from "../../config.ts";
import { isSelfUrl } from "../../lib/self-url.ts";
const DUMP_RE = /^\/dumps\/([^/]+)$/;
const USER_RE = /^\/users\/([^/]+)$/;
const PLAYLIST_RE = /^\/playlists\/([^/]+)$/;
import { parsePagePath } from "../../../shared/paths.ts";
/**
* Gives a link to this site without a thumbnail a placeholder in the site's
@@ -30,10 +28,10 @@ const SELF_BRANDING = {
} as const;
/**
* Comes before the generic provider, and claims what `isSelfUrl` accepts. Its
* `fetch` rejects with a 404 `APIException` for a missing page, a private dump
* or a private playlist: the preview is stored in the linking dump, which
* others may see.
* Comes before the generic provider, and claims what `isSelfUrl` accepts. A
* profile's tab previews as the profile. Its `fetch` rejects with a 404
* `APIException` for a missing page, a private dump or a private playlist:
* the preview is stored in the linking dump, which others may see.
*/
export const selfProvider: RichContentProvider = {
name: "self",
@@ -47,14 +45,15 @@ export const selfProvider: RichContentProvider = {
},
fetch(url: string): Promise<RichContent> {
const { pathname } = new URL(url);
// matches() let only these paths through.
const page = parsePagePath(new URL(url).pathname)!;
const user = resolveUsername(page.username);
const dumpMatch = pathname.match(DUMP_RE);
if (dumpMatch) {
const dump = getDump(dumpMatch[1]);
if (page.kind === "dump") {
const dump = getDumpByPath(user.id, page.slug);
let thumbnailUrl: string | undefined;
if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) {
thumbnailUrl = `${PUBLIC_URL}/api/files/${dump.id}`;
thumbnailUrl = `${PUBLIC_URL}/api/dumps/${dump.id}/file`;
} else if (dump.richContent?.thumbnailUrl) {
thumbnailUrl = dump.richContent.thumbnailUrl;
}
@@ -68,30 +67,13 @@ export const selfProvider: RichContentProvider = {
});
}
const userMatch = pathname.match(USER_RE);
if (userMatch) {
const user = getUserByUsername(userMatch[1]);
const thumbnailUrl = user.avatarMime
? `${PUBLIC_URL}/api/avatars/${user.id}`
: undefined;
return Promise.resolve({
type: "generic",
url,
...SELF_BRANDING,
title: user.username,
description: user.description,
thumbnailUrl,
});
}
const playlistMatch = pathname.match(PLAYLIST_RE);
if (playlistMatch) {
const playlist = getPlaylistById(playlistMatch[1]);
if (page.kind === "playlist") {
const playlist = getPlaylistByPath(user.id, page.slug);
if (!playlist.isPublic) {
throw new APIException(404, "Playlist not found");
}
const thumbnailUrl = playlist.imageMime
? `${PUBLIC_URL}/api/playlists/${playlist.id}/image`
? `${PUBLIC_URL}/api/playlists/${playlist.id}/cover`
: undefined;
return Promise.resolve({
type: "generic",
@@ -103,7 +85,16 @@ export const selfProvider: RichContentProvider = {
});
}
// Unreachable: isSelfUrl accepts only the three paths above.
return Promise.resolve({ type: "generic", url });
const thumbnailUrl = user.avatarMime
? `${PUBLIC_URL}/api/users/${user.id}/avatar`
: undefined;
return Promise.resolve({
type: "generic",
url,
...SELF_BRANDING,
title: user.username,
description: user.description,
thumbnailUrl,
});
},
};

66
api/remote/providers/soundcloud.ts
View File

@@ -1,50 +1,72 @@
/**
* Previews SoundCloud links from their page's OG tags, for
* `rich-content-service.ts`.
* Previews SoundCloud links from their page, for `rich-content-service.ts`.
*
* @module
*/
import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts";
import { parsePage } from "../html.ts";
import { fetchPage } from "../fetch.ts";
import { genericPreview } from "./generic.ts";
/** SoundCloud's own hosts, the short links' `on.` included. */
const SOUNDCLOUD_HOSTS = new Set([
"soundcloud.com",
"www.soundcloud.com",
"m.soundcloud.com",
"on.soundcloud.com",
]);
/**
* Claims soundcloud.com only, not www. or m.soundcloud.com, and builds the
* embed player's URL from the link itself.
* Returns the soundcloud.com address of the page, which the embed player
* takes: its `og:url`, or else the URL it was served from. A short link only
* redirects, so it is never the answer; tracking parameters are dropped.
*/
function pageAddress(ogUrl: string | undefined, finalUrl: string): string {
for (const raw of [ogUrl, finalUrl]) {
if (!raw) continue;
try {
const url = new URL(raw);
if (
!SOUNDCLOUD_HOSTS.has(url.hostname) ||
url.hostname === "on.soundcloud.com"
) continue;
return `https://soundcloud.com${url.pathname}`;
} catch {
continue;
}
}
return finalUrl;
}
/**
* Claims SoundCloud's hosts, short links included, and builds the embed
* player's URL from the page's address.
*/
export const soundcloudProvider: RichContentProvider = {
name: "soundcloud",
matches(url: string): boolean {
try {
return new URL(url).hostname === "soundcloud.com";
return SOUNDCLOUD_HOSTS.has(new URL(url).hostname);
} catch {
return false;
}
},
async fetch(url: string): Promise<RichContent> {
const res = await fetchWithTimeout(url);
const contentType = res.headers.get("content-type") ?? "";
if (!contentType.startsWith("text/html")) {
await res.body?.cancel();
return { type: "soundcloud", siteName: "SoundCloud", url };
}
const html = await readTextCapped(res);
const fetched = await fetchPage(url);
const page = fetched.html === undefined
? undefined
: parsePage(fetched.html, fetched.url);
const address = pageAddress(page?.meta("og:url"), fetched.url);
return {
...(page ? genericPreview(page, url, fetched.url) : { url }),
type: "soundcloud",
siteName: "SoundCloud",
url,
title: extractOgTag(html, "title"),
description: extractOgTag(html, "description"),
thumbnailUrl: extractOgTag(html, "image"),
embedUrl: `https://w.soundcloud.com/player/?url=${
encodeURIComponent(url)
encodeURIComponent(address)
}&visual=true&auto_play=false`,
};
},

33
api/remote/providers/youtube.ts
View File

@@ -1,22 +1,24 @@
/**
* Previews YouTube videos, playlists and channels, for
* `rich-content-service.ts`. Videos and playlists are read from YouTube's
* oEmbed endpoint, channels from their page's OG tags.
* oEmbed endpoint, channels from their page.
*
* @module
*/
import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts";
import { parsePage } from "../html.ts";
import { fetchPage, fetchWithTimeout } from "../fetch.ts";
import { youtubeHost, youtubeVideoId } from "../../lib/canonical-url.ts";
import { siteHostOf } from "../../../shared/url.ts";
import { genericPreview } from "./generic.ts";
/** Returns the id of the video `url` points at, in any shape YouTube uses. */
function extractVideoId(url: string): string | null {
try {
const u = new URL(url);
const host = u.hostname.toLowerCase().replace(/^www\./, "");
const host = siteHostOf(u);
return youtubeVideoId(host, u.pathname, u.searchParams);
} catch {
return null;
@@ -122,27 +124,17 @@ export const youtubeProvider: RichContentProvider = {
if (channelPath && !videoId) {
// oEmbed doesn't take channel URLs.
let title: string | undefined;
let thumbnailUrl: string | undefined;
let channel: RichContent | undefined;
try {
const res = await fetchWithTimeout(url);
if (res.ok) {
const html = await readTextCapped(res);
title = extractOgTag(html, "title");
thumbnailUrl = extractOgTag(html, "image");
} else {
await res.body?.cancel();
const fetched = await fetchPage(url);
if (fetched.html !== undefined) {
const page = parsePage(fetched.html, fetched.url);
channel = genericPreview(page, url, fetched.url);
}
} catch {
// YouTube unreachable: the preview goes without.
}
return {
type: "youtube",
siteName: "YouTube",
url,
title,
thumbnailUrl,
};
return { ...channel, type: "youtube", siteName: "YouTube", url };
}
if (listId && !videoId) {
@@ -171,7 +163,6 @@ export const youtubeProvider: RichContentProvider = {
type: "youtube",
siteName: "YouTube",
url,
videoId: videoId!,
title,
thumbnailUrl,
embedUrl: `https://www.youtube.com/embed/${videoId}?${embedParams}`,

5
api/remote/public-address.ts
View File

@@ -100,9 +100,10 @@ function isIpLiteral(host: string): boolean {
/**
* Returns `url`'s hostname without the brackets `URL.hostname` keeps around
* an IPv6 literal.
* an IPv6 literal: the address checks here read it, and `fetch.ts` hands it
* to curl's `--resolve`.
*/
function bareHostname(url: URL): string {
export function bareHostname(url: URL): string {
return url.hostname.replace(/^\[(.*)\]$/, "$1");
}

19
api/remote/youtube-playlist.ts
View File

@@ -1,7 +1,7 @@
/**
* Gives the global player a tracklist for a YouTube playlist, which the embed
* plays but doesn't list. It resolves the playlist to its entries' titles and
* lengths, for `GET /api/youtube/playlist`.
* lengths, for `GET /api/remote/youtube`.
*
* They are read from the `ytInitialData` JSON of the public playlist page,
* whose undocumented shape changes, so both the current `lockupViewModel` rows
@@ -13,6 +13,7 @@
*/
import { APIException } from "../lib/api-exception.ts";
import { TtlCache } from "../lib/ttl-cache.ts";
import { readTextCapped } from "./fetch.ts";
import type {
YouTubePlaylist,
@@ -20,11 +21,7 @@ import type {
} from "../../shared/player.ts";
/** Titles and lengths rarely change, and a stale one only mislabels a row. */
const TTL_MS = 6 * 60 * 60 * 1000;
/** The oldest write goes first. */
const MAX_ENTRIES = 200;
const cache = new Map<string, { value: YouTubePlaylist; at: number }>();
const cache = new TtlCache<YouTubePlaylist>(6 * 3_600_000, 200);
/**
* Admits every kind of list id (`PL…`, `OLAK5uy_…`, `RD…`) and nothing else.
@@ -167,8 +164,7 @@ export async function resolveYouTubePlaylist(
}
const hit = cache.get(listId);
if (hit && Date.now() - hit.at < TTL_MS) return hit.value;
if (hit) cache.delete(listId);
if (hit) return hit;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 8000);
@@ -198,10 +194,5 @@ export async function resolveYouTubePlaylist(
throw new APIException(404, "No tracks on this playlist");
}
if (cache.size >= MAX_ENTRIES) {
const oldest = cache.keys().next().value;
if (oldest !== undefined) cache.delete(oldest);
}
cache.set(listId, { value: parsed, at: Date.now() });
return parsed;
return cache.set(listId, parsed);
}

2
api/routes/attachments.ts
View File

@@ -20,7 +20,7 @@ import {
import { authenticateMedia } from "../lib/auth.ts";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import type { AttachmentResponse } from "../../shared/dump.ts";
import type { AttachmentResponse } from "../../shared/attachment.ts";
import { serveUploadedFile, validateImageUpload } from "../lib/upload.ts";
import { ATTACHMENTS_DIR } from "../config.ts";

54
api/routes/avatars.ts
View File

@@ -3,10 +3,11 @@
* and the client's `Avatar` component loads it, with the user's `updatedAt` as
* a cache-busting version.
*
* - `POST /api/avatars/me` — signed in. Replaces the caller's avatar (multipart
* - `PUT /api/me/avatar` — signed in. Replaces the caller's avatar (multipart
* `file`, at most 5 MB), sends `user_updated` to everyone, and answers the
* caller's user.
* - `GET /api/avatars/:userId` — public. Serves a user's avatar; 404 without
* caller's user. 400 for an account from another instance, whose avatar
* comes from there. The instances the caller signed in to are told.
* - `GET /api/users/:key/avatar` — public. Serves a user's avatar; 404 without
* one.
*
* @module
@@ -14,20 +15,18 @@
import { Router } from "@oak/oak";
import { authMiddleware } from "../middleware/auth.ts";
import { getUserById, updateUserAvatar } from "../services/user-service.ts";
import {
broadcastUserUpdated,
updateClientAvatar,
} from "../services/ws-service.ts";
assertLocalUser,
getUserById,
resolveUserKey,
setUserAvatar,
} from "../services/user-service.ts";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import type { SelfUser } from "../../shared/user.ts";
import { toPublicUser, toSelfUser } from "../model/user.ts";
import {
replaceUploadedFile,
serveUploadedFile,
validateImageUpload,
} from "../lib/upload.ts";
import { toSelfUser } from "../model/user.ts";
import { serveUploadedFile } from "../lib/upload.ts";
import { notifyClients } from "../services/identity-provider-service.ts";
import { AVATARS_DIR } from "../config.ts";
/**
@@ -36,7 +35,7 @@ import { AVATARS_DIR } from "../config.ts";
*/
const router: Router = new Router();
router.post("/api/avatars/me", authMiddleware, async (ctx) => {
router.put("/api/me/avatar", authMiddleware, async (ctx) => {
const authPayload = ctx.state.user;
const contentType = ctx.request.headers.get("content-type") ?? "";
if (!contentType.includes("multipart/form-data")) {
@@ -50,19 +49,12 @@ router.post("/api/avatars/me", authMiddleware, async (ctx) => {
throw new APIException(400, "Missing file field");
}
const data = new Uint8Array(await file.arrayBuffer());
const mime = validateImageUpload(data);
// Written before the record announces it, or clients would fetch it too
// early.
await replaceUploadedFile(`${AVATARS_DIR}/${authPayload.userId}`, data);
updateUserAvatar(authPayload.userId, mime);
updateClientAvatar(authPayload.userId, mime);
const user = getUserById(authPayload.userId);
// Presence alone would update only the online list; `user_updated` reaches
// every avatar on screen, and its bumped `updatedAt` busts their cache.
broadcastUserUpdated(toPublicUser(user));
assertLocalUser(getUserById(authPayload.userId));
const user = await setUserAvatar(
authPayload.userId,
new Uint8Array(await file.arrayBuffer()),
);
await notifyClients(user.id, { type: "update" });
const responseBody: APIResponse<SelfUser> = {
success: true,
data: toSelfUser(user),
@@ -70,12 +62,10 @@ router.post("/api/avatars/me", authMiddleware, async (ctx) => {
ctx.response.body = responseBody;
});
router.get("/api/avatars/:userId", async (ctx) => {
const { userId } = ctx.params;
router.get("/api/users/:key/avatar", async (ctx) => {
let user;
try {
user = getUserById(userId);
user = resolveUserKey(ctx.params.key);
} catch {
ctx.response.status = 404;
return;
@@ -86,7 +76,7 @@ router.get("/api/avatars/:userId", async (ctx) => {
return;
}
await serveUploadedFile(ctx, `${AVATARS_DIR}/${userId}`, user.avatarMime);
await serveUploadedFile(ctx, `${AVATARS_DIR}/${user.id}`, user.avatarMime);
});
export default router;

39
api/routes/bandcamp.ts
View File

@@ -1,39 +0,0 @@
/**
* Serves the Bandcamp release the native player plays, resolved from its page
* when playback starts. On any failure the player falls back to the embed.
*
* - `GET /api/bandcamp/tracks?url=&force=` — public. Answers the page's
* tracks and their stream URLs; `force=1` skips the server's cache.
*
* @module
*/
import { Router } from "@oak/oak";
import type { APIResponse } from "../../shared/api.ts";
import { resolveBandcamp } from "../remote/bandcamp-streams.ts";
import type { Tralbum } from "../../shared/player.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const bandcampRouter: Router = new Router();
// Public: it reads a public page and answers the URLs that page gives any
// visitor.
bandcampRouter.get("/api/bandcamp/tracks", async (ctx) => {
const url = ctx.request.url.searchParams.get("url") ?? "";
const force = ctx.request.url.searchParams.get("force") === "1";
const responseBody: APIResponse<Tralbum> = {
success: true,
data: await resolveBandcamp(url, { force }),
};
// Not in a shared cache: the signed URLs belong to one page fetch. An hour
// is well inside their 24 hours.
ctx.response.headers.set("Cache-Control", "private, max-age=3600");
ctx.response.body = responseBody;
});
export default bandcampRouter;

49
api/routes/categories.ts
View File

@@ -1,40 +1,36 @@
/**
* Serves the categories dumps are filed under and their feeds, and changes them
* for those with `category:manage`.
* Serves the categories dumps and playlists are filed under, a set for each
* kind, and changes them for those with `category:manage`. A dump category's
* feed is `GET /api/dumps?category=:slug`, a playlist category's
* `GET /api/playlists?category=:slug`.
*
* - `GET /api/categories` — public. Answers every category, in order.
* - `GET /api/categories/:slug` — public. Answers one category.
* - `GET /api/categories/:slug/dumps?page=&limit=` — optional sign-in. Answers
* a page of its dumps, newest first, with the caller's private ones.
* - `POST /api/categories` — permission `category:manage`. Creates a category.
* - `GET /api/categories` — public. Answers every category of both kinds, by
* kind, then in order.
* - `POST /api/categories` — permission `category:manage`. Creates a category
* of the kind it names.
* - `PUT /api/categories/:id` — permission `category:manage`. Updates a
* category.
* - `DELETE /api/categories/:id` — permission `category:manage`. Deletes a
* category; its dumps stay.
* category; its dumps or playlists stay.
*
* @module
*/
import { Router } from "@oak/oak";
import { type APIResponse, type PaginatedData } from "../../shared/api.ts";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import type { Category } from "../../shared/category.ts";
import type { Dump } from "../../shared/dump.ts";
import {
isCreateCategoryRequest,
isUpdateCategoryRequest,
} from "../validators/category.ts";
import { authMiddleware, requirePermission } from "../middleware/auth.ts";
import { parseOptionalAuth } from "../lib/auth.ts";
import { parsePagination } from "../lib/pagination.ts";
import {
createCategory,
deleteCategory,
getCategoryBySlug,
listCategories,
listDumpsByCategory,
updateCategory,
} from "../services/category-service.ts";
@@ -52,31 +48,6 @@ router.get("/", (ctx) => {
ctx.response.body = responseBody;
});
router.get("/:slug", (ctx) => {
const category = getCategoryBySlug(ctx.params.slug);
const responseBody: APIResponse<Category> = {
success: true,
data: category,
};
ctx.response.body = responseBody;
});
router.get("/:slug/dumps", async (ctx) => {
const requestingUserId = await parseOptionalAuth(ctx) ?? undefined;
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = listDumpsByCategory(
ctx.params.slug,
page,
limit,
requestingUserId,
);
const responseBody: APIResponse<PaginatedData<Dump>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
router.post(
"/",
authMiddleware,

16
api/routes/comments.ts
View File

@@ -1,7 +1,7 @@
/**
* Serves the comments on dumps. The comments of a private dump 404 for anyone
* but its poster, and changes to them aren't broadcast. Comments are otherwise
* broadcast over the WebSocket as they are posted, edited and deleted.
* Serves the comments on dumps. Comments are broadcast over the WebSocket as
* they are posted, edited and deleted, to everyone, or to its poster alone for
* a private dump, whose comments 404 for anyone else.
*
* - `GET /api/dumps/:dumpId/comments` — optional sign-in. Answers every comment
* on a dump, oldest first.
@@ -76,7 +76,7 @@ router.post("/dumps/:dumpId/comments", authMiddleware, async (ctx) => {
body.body,
body.parentId ?? undefined,
);
if (!dump.isPrivate) broadcastCommentCreated(comment);
broadcastCommentCreated(comment, dump);
const responseBody: APIResponse<Comment> = { success: true, data: comment };
ctx.response.status = 201;
ctx.response.body = responseBody;
@@ -89,13 +89,13 @@ router.patch("/comments/:commentId", authMiddleware, async (ctx) => {
if (!isUpdateCommentRequest(body)) {
throw new APIException(400, "Invalid comment data");
}
const { comment, isPrivate } = updateComment(
const { comment, dump } = updateComment(
ctx.params.commentId,
body.body,
userId,
canModerate,
);
if (!isPrivate) broadcastCommentUpdated(comment);
broadcastCommentUpdated(comment, dump);
const responseBody: APIResponse<Comment> = { success: true, data: comment };
ctx.response.body = responseBody;
});
@@ -123,12 +123,12 @@ router.get("/comments/:commentId/likers", async (ctx) => {
router.delete("/comments/:commentId", authMiddleware, (ctx) => {
const userId = ctx.state.user.userId;
const canModerate = can(ctx.state.user, "comment:moderate");
const { dumpId, isPrivate } = deleteComment(
const dump = deleteComment(
ctx.params.commentId,
userId,
canModerate,
);
if (!isPrivate) broadcastCommentDeleted(ctx.params.commentId, dumpId);
broadcastCommentDeleted(ctx.params.commentId, dump);
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
});

37
api/routes/community.ts Normal file
View File

@@ -0,0 +1,37 @@
/**
* Serves the figures of the client's `/community` page, which
* `community-service.ts` recomputes on a timer.
*
* - `GET /api/community?period=weekly|monthly` — signed in. Answers the
* site's figures over the last week (by default) or month.
*
* @module
*/
import { Router } from "@oak/oak";
import type { APIResponse } from "../../shared/api.ts";
import type { CommunityStats } from "../../shared/community.ts";
import { isDigestCadence } from "../../shared/settings.ts";
import { APIException } from "../lib/api-exception.ts";
import { type AuthContext, authMiddleware } from "../middleware/auth.ts";
import { getCommunityStats } from "../services/community-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/community" });
router.get("/", authMiddleware, (ctx: AuthContext) => {
const period = ctx.request.url.searchParams.get("period") ?? "weekly";
if (!isDigestCadence(period)) {
throw new APIException(400, "period must be weekly or monthly");
}
const responseBody: APIResponse<CommunityStats> = {
success: true,
data: getCommunityStats(period),
};
ctx.response.body = responseBody;
});
export default router;

205
api/routes/dumps.ts
View File

@@ -1,11 +1,24 @@
/**
* Posts, reads, edits and deletes dumps, for the index feed, the dump page, its
* edit page and the create form. `:dumpId` takes an id or a slug. A private
* dump answers 404 to anyone but its poster, moderators included.
* Posts, reads, edits and deletes dumps, for the feeds, the dump page, its
* edit page and the create form. `:dumpId` is a dump's id; the dump page reads
* a dump by its path, through `GET /api/users/:key/dumps/:slug`. A private
* dump answers 404 to anyone but its poster, moderators included. Its file and
* thumbnail are served by `routes/files.ts` and `routes/thumbnails.ts`, its
* comments by `routes/comments.ts`.
*
* - `POST /api/dumps` — signed in. Posts a link (JSON body) or a file
* (multipart `file`, with `title`, `comment`, `isPrivate` and `categoryIds`).
* - `GET /api/dumps/by-url?url=` — optional sign-in. Answers the dumps already
* - `GET /api/dumps?page=&limit=` — optional sign-in. Answers a page of the
* dumps the caller may see, newest first.
* - `GET /api/dumps?category=&page=&limit=` — optional sign-in. The same, in
* the category with that slug; 404 when there is none.
* - `GET /api/dumps?follow=users&category=&page=&limit=` — signed in. The
* index's "follow" tab: a page of the public dumps of the users the caller
* follows, newest first, optionally in one category.
* - `GET /api/dumps?follow=playlists&category=&page=&limit=` — signed in. Its
* `/follow/playlists` view: a page of the public dumps in the public
* playlists the caller follows, most recently added first.
* - `GET /api/dumps?url=` — optional sign-in. Answers the dumps already
* pointing at the same thing as `url`, for the create form's duplicate hint.
* - `GET /api/dumps/:dumpId` — optional sign-in. Answers a dump.
* - `GET /api/dumps/:dumpId/voters?page=&limit=` — optional sign-in. Answers a
@@ -14,21 +27,12 @@
* linked to or from it.
* - `GET /api/dumps/:dumpId/playlists` — optional sign-in. Answers the
* playlists it appears in.
* - `GET /api/dumps?page=&limit=` — optional sign-in. Answers a page of the
* dumps the caller may see, newest first.
* - `PUT /api/dumps/:dumpId/file` — signed in, as its poster or with
* `dump:moderate`. Replaces a file dump's upload and comment, and any of its
* title, categories and visibility (multipart).
* - `PUT /api/dumps/:dumpId` — signed in, as its poster or with
* `dump:moderate`. Updates its title, comment, visibility, categories, and a
* link's URL.
* - `PUT /api/dumps/:dumpId/thumbnail` — signed in, as its poster or with
* `dump:moderate`. Replaces its custom thumbnail (multipart `file`, at most 5
* MB).
* - `DELETE /api/dumps/:dumpId/thumbnail` — signed in, as its poster or with
* `dump:moderate`. Removes its custom thumbnail.
* - `POST /api/dumps/:dumpId/refresh-metadata` — signed in, as its poster or
* with `dump:moderate`. Fetches a link's preview again.
* `dump:moderate`. Updates its title, slug, comment, visibility, categories,
* and a link's URL. A new slug keeps the old one as an alias, 3 times a day
* at most but for those with `dump:moderate`.
* - `POST /api/dumps/:dumpId/metadata` — signed in, as its poster or with
* `dump:moderate`. Fetches a link's preview again.
* - `DELETE /api/dumps/:dumpId` — signed in, as its poster or with
* `dump:moderate`. Deletes it.
*
@@ -43,6 +47,7 @@ import type { Dump, DumpUrlMatch } from "../../shared/dump.ts";
import {
isCreateUrlDumpRequest,
isUpdateDumpRequest,
parseCategoryIds,
} from "../validators/dump.ts";
import type { Playlist } from "../../shared/playlist.ts";
import type { PublicUser } from "../../shared/user.ts";
@@ -51,9 +56,7 @@ import { toPublicUser } from "../model/user.ts";
import { authMiddleware } from "../middleware/auth.ts";
import { can } from "../lib/permissions.ts";
import { parseOptionalAuth } from "../lib/auth.ts";
import { parsePagination } from "../lib/pagination.ts";
import { replaceUploadedFile, validateImageUpload } from "../lib/upload.ts";
import { THUMBNAILS_DIR } from "../config.ts";
import { type Page, parsePagination } from "../lib/pagination.ts";
import {
createFileDump,
createUrlDump,
@@ -62,11 +65,17 @@ import {
getDump,
listDumps,
refreshDumpMetadata,
removeDumpThumbnail,
replaceFileDump,
setDumpThumbnail,
updateDump,
} from "../services/dump-service.ts";
import {
getCategoryBySlug,
listDumpsByCategory,
} from "../services/category-service.ts";
import {
getFollowedPlaylistsDumpFeed,
getFollowedUsersDumpFeed,
} from "../services/follow-service.ts";
import { FOLLOW_VIEWS, type FollowView } from "../../shared/paths.ts";
import { getDumpVoters } from "../services/vote-service.ts";
import { getRelatedDumps } from "../services/backlink-service.ts";
import { getPlaylistsForDump } from "../services/playlist-service.ts";
@@ -77,24 +86,8 @@ import { getPlaylistsForDump } from "../services/playlist-service.ts";
*/
const router: Router = new Router({ prefix: "/api/dumps" });
// Each handler resolves `:dumpId` with getDump, which checks privacy and takes
// a slug, and passes `dump.id` on: the services' writes match on the id alone.
/**
* Parses a multipart form's `categoryIds`, a JSON array sent as a string. A
* missing or malformed value counts as no category.
*/
function parseCategoryIds(value: FormDataEntryValue | null): string[] {
if (typeof value !== "string" || !value) return [];
try {
const parsed: unknown = JSON.parse(value);
return Array.isArray(parsed)
? parsed.filter((v): v is string => typeof v === "string")
: [];
} catch {
return [];
}
}
// Each handler resolves `:dumpId` with getDump, which checks privacy, before
// acting on the dump.
router.post(
"/",
@@ -140,17 +133,6 @@ router.post(
},
);
// Registered ahead of "/:dumpId" so the literal path wins over the parameter.
router.get("/by-url", async (ctx) => {
const requestingUserId = await parseOptionalAuth(ctx) ?? undefined;
const url = ctx.request.url.searchParams.get("url") ?? "";
const responseBody: APIResponse<DumpUrlMatch[]> = {
success: true,
data: findDumpsByUrl(url, requestingUserId),
};
ctx.response.body = responseBody;
});
router.get("/:dumpId", async (ctx) => {
const requestingUserId = await parseOptionalAuth(ctx) ?? undefined;
const dump = getDump(ctx.params.dumpId, requestingUserId);
@@ -194,8 +176,37 @@ router.get("/:dumpId/playlists", async (ctx) => {
router.get("/", async (ctx) => {
const requestingUserId = await parseOptionalAuth(ctx) ?? undefined;
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = listDumps(page, limit, requestingUserId);
const params = ctx.request.url.searchParams;
const url = params.get("url");
if (url !== null) {
const responseBody: APIResponse<DumpUrlMatch[]> = {
success: true,
data: findDumpsByUrl(url, requestingUserId),
};
ctx.response.body = responseBody;
return;
}
const { page, limit } = parsePagination(params);
const category = params.get("category");
const follow = params.get("follow");
let feed: Page<Dump>;
if (follow !== null) {
if (!requestingUserId) throw new APIException(401, "Missing token");
if (!(FOLLOW_VIEWS as readonly string[]).includes(follow)) {
throw new APIException(400, "follow must be users or playlists");
}
const categoryId = category
? getCategoryBySlug("dump", category).id
: undefined;
feed = (follow as FollowView) === "users"
? getFollowedUsersDumpFeed(requestingUserId, page, limit, categoryId)
: getFollowedPlaylistsDumpFeed(requestingUserId, page, limit, categoryId);
} else {
feed = category
? listDumpsByCategory(category, page, limit, requestingUserId)
: listDumps(page, limit, requestingUserId);
}
const { items, hasMore } = feed;
const responseBody: APIResponse<PaginatedData<Dump>> = {
success: true,
data: { items, hasMore },
@@ -203,40 +214,6 @@ router.get("/", async (ctx) => {
ctx.response.body = responseBody;
});
router.put("/:dumpId/file", authMiddleware, async (ctx) => {
const dumpId = ctx.params.dumpId;
const userId = ctx.state.user.userId;
const dump = getDump(dumpId, userId);
if (userId !== dump.userId && !can(ctx.state.user, "dump:moderate")) {
throw new APIException(403, "Not authorized to update dump");
}
const formData = await ctx.request.body.formData();
const file = formData.get("file");
const comment = formData.get("comment");
const title = formData.get("title");
if (!(file instanceof File)) {
throw new APIException(400, "A file is required");
}
const updatedDump = await replaceFileDump(
dump.id,
file,
typeof comment === "string" && comment ? comment : undefined,
typeof title === "string" && title ? title : undefined,
formData.has("categoryIds")
? parseCategoryIds(formData.get("categoryIds"))
: undefined,
formData.has("isPrivate")
? formData.get("isPrivate") === "true"
: undefined,
);
const responseBody: APIResponse<Dump> = { success: true, data: updatedDump };
ctx.response.body = responseBody;
});
router.put("/:dumpId", authMiddleware, async (ctx) => {
const dumpId = ctx.params.dumpId;
const userId = ctx.state.user.userId;
@@ -252,54 +229,16 @@ router.put("/:dumpId", authMiddleware, async (ctx) => {
throw new APIException(403, "Not authorized to update dump");
}
const updatedDump = await updateDump(dump.id, body);
const updatedDump = await updateDump(
dump.id,
body,
can(ctx.state.user, "dump:moderate"),
);
const responseBody: APIResponse<Dump> = { success: true, data: updatedDump };
ctx.response.body = responseBody;
});
router.put("/:dumpId/thumbnail", authMiddleware, async (ctx) => {
const dumpId = ctx.params.dumpId;
const userId = ctx.state.user.userId;
const dump = getDump(dumpId, userId);
if (userId !== dump.userId && !can(ctx.state.user, "dump:moderate")) {
throw new APIException(403, "Not authorized to update dump");
}
const formData = await ctx.request.body.formData();
const file = formData.get("file");
if (!(file instanceof File)) {
throw new APIException(400, "Missing file field");
}
const data = new Uint8Array(await file.arrayBuffer());
const mime = validateImageUpload(data);
// Written before setDumpThumbnail announces it, or clients would fetch it
// too early.
await replaceUploadedFile(`${THUMBNAILS_DIR}/${dump.id}-custom`, data);
const updatedDump = setDumpThumbnail(dump.id, mime);
const responseBody: APIResponse<Dump> = { success: true, data: updatedDump };
ctx.response.body = responseBody;
});
router.delete("/:dumpId/thumbnail", authMiddleware, async (ctx) => {
const dumpId = ctx.params.dumpId;
const userId = ctx.state.user.userId;
const dump = getDump(dumpId, userId);
if (userId !== dump.userId && !can(ctx.state.user, "dump:moderate")) {
throw new APIException(403, "Not authorized to update dump");
}
const updatedDump = await removeDumpThumbnail(dump.id);
const responseBody: APIResponse<Dump> = { success: true, data: updatedDump };
ctx.response.body = responseBody;
});
router.post("/:dumpId/refresh-metadata", authMiddleware, async (ctx) => {
router.post("/:dumpId/metadata", authMiddleware, async (ctx) => {
const dumpId = ctx.params.dumpId;
const userId = ctx.state.user.userId;
const dump = getDump(dumpId, userId);

152
api/routes/federation.ts Normal file
View File

@@ -0,0 +1,152 @@
/**
* Lets users of other gerbeur instances sign in here with their account
* there, as `federation-service.ts` describes, and lets admins pick the
* instances to trust or block. The login and register pages start a sign-in,
* the `/federation/callback` page finishes it, `/manage` edits the peers, and
* the other instances' servers send their events.
*
* - `POST /api/federation/logins` — public. Starts a sign-in with an account
* on another instance (`FederationLoginRequest`, with an invite unless it
* is trusted), and answers its `/authorize` page to go to. Rate-limited per
* instance.
* - `POST /api/federation/sessions` — public. Finishes it
* (`FederationSessionRequest`) and answers the user and their tokens, as a
* sign-in does. Each sign-in started finishes once, so the limit on
* `/logins` bounds it.
* - `POST /api/federation/events` — public, server to server. Applies a
* home instance's signed event about one of its users (`{ event }`).
* Rate-limited per instance; a 4xx tells the sender to give up on the
* event, a 429 or a 5xx to try again.
* - `GET /api/federation/peers` — optional sign-in. Answers the trusted
* instances, or every listed one to `federation:manage`.
* - `PUT /api/federation/peers` — permission `federation:manage`. Trusts or
* blocks an instance (`SetPeerRequest`) and answers it; 502 when one to
* trust doesn't answer as a gerbeur.
* - `DELETE /api/federation/peers/:host` — permission `federation:manage`.
* Takes an instance off the lists.
*
* @module
*/
import { Router } from "@oak/oak";
import type { APIResponse } from "../../shared/api.ts";
import {
type FederationLoginResponse,
type FederationPeer,
parseInstance,
} from "../../shared/federation.ts";
import type { AuthResponse } from "../../shared/user.ts";
import { APIException } from "../lib/api-exception.ts";
import { parseOptionalAuth } from "../lib/auth.ts";
import { issueTokens } from "../lib/jwt.ts";
import { hasPermission } from "../../shared/permissions.ts";
import { type RateLimit, takeToken } from "../lib/rate-limit.ts";
import {
type AuthContext,
authMiddleware,
requirePermission,
} from "../middleware/auth.ts";
import { toSelfUser } from "../model/user.ts";
import {
isFederationLoginRequest,
isFederationSessionRequest,
isSetPeerRequest,
} from "../validators/federation.ts";
import {
completeLogin,
receiveEvent,
startLogin,
} from "../services/federation-service.ts";
import {
listPeers,
removePeer,
setPeer,
} from "../services/federation-peer-service.ts";
import { getUserById } from "../services/user-service.ts";
import { fieldOf } from "../validators/user.ts";
/**
* Allows a burst of 30 sign-ins per instance, then one every 2 seconds, so
* that nobody makes this instance hammer another.
*/
const LOGIN_LIMIT: RateLimit = { capacity: 30, refill: 1 / 2 };
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/federation" });
router.post("/logins", async (ctx) => {
const body: unknown = await ctx.request.body.json();
if (!isFederationLoginRequest(body)) {
throw new APIException(400, "Invalid request");
}
if (!takeToken(`federation:${parseInstance(body.instance)}`, LOGIN_LIMIT)) {
throw new APIException(429, "Too many sign-ins — try again in a minute");
}
const responseBody: APIResponse<FederationLoginResponse> = {
success: true,
data: { authorizeUrl: await startLogin(body) },
};
ctx.response.body = responseBody;
});
router.post("/sessions", async (ctx) => {
const body: unknown = await ctx.request.body.json();
if (!isFederationSessionRequest(body)) {
throw new APIException(400, "Invalid request");
}
const user = await completeLogin(body);
const responseBody: APIResponse<AuthResponse> = {
success: true,
data: { ...await issueTokens(user), user: toSelfUser(user) },
};
ctx.response.body = responseBody;
});
router.post("/events", async (ctx) => {
const event = fieldOf(await ctx.request.body.json(), "event");
if (typeof event !== "string") throw new APIException(400, "Invalid event");
await receiveEvent(event);
ctx.response.status = 204;
});
router.get("/peers", async (ctx) => {
const userId = await parseOptionalAuth(ctx);
const all = !!userId &&
hasPermission(getUserById(userId).role, "federation:manage");
const responseBody: APIResponse<FederationPeer[]> = {
success: true,
data: listPeers(!all),
};
ctx.response.body = responseBody;
});
router.put(
"/peers",
authMiddleware,
requirePermission("federation:manage"),
async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
if (!isSetPeerRequest(body)) throw new APIException(400, "Invalid request");
const responseBody: APIResponse<FederationPeer> = {
success: true,
data: await setPeer(body.instance, body.status),
};
ctx.response.body = responseBody;
},
);
router.delete(
"/peers/:host",
authMiddleware,
requirePermission("federation:manage"),
(ctx: AuthContext<"/peers/:host">) => {
removePeer(ctx.params.host);
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
},
);
export default router;

89
api/routes/files.ts
View File

@@ -1,26 +1,35 @@
/**
* Serves file dumps' uploads to the players, viewers and download links of the
* frontend, which builds the URL with `dumpFileUrl`.
* Serves and replaces file dumps' uploads, a sub-resource of their dump. The
* players, viewers and download links of the frontend build the URL with
* `dumpFileUrl`, and its edit page replaces the file.
*
* - `GET /api/files/:dumpId?v=&token=` — public; a private dump's file needs
* its poster's media `token`. Serves the file, inline when that is safe and
* as a download otherwise, with byte ranges.
* - `GET /api/dumps/:dumpId/file?v=&token=` — public; a private dump's file
* needs its poster's media `token`. Serves the file, inline when that is
* safe and as a download otherwise, with byte ranges.
* - `PUT /api/dumps/:dumpId/file` — signed in, as its poster or with
* `dump:moderate`. Replaces a file dump's upload and comment, and any of its
* title, slug, categories and visibility (multipart).
*
* @module
*/
import { Router } from "@oak/oak";
import { APIException } from "../lib/api-exception.ts";
import type { APIResponse } from "../../shared/api.ts";
import type { Dump } from "../../shared/dump.ts";
import { DUMPS_DIR } from "../config.ts";
import { authenticateMedia } from "../lib/auth.ts";
import { resolveMediaDump } from "../lib/dump-media.ts";
import { can } from "../lib/permissions.ts";
import { serveFile } from "../lib/serve-file.ts";
import { dumpFileVersion, getDump } from "../services/dump-service.ts";
import { authMiddleware } from "../middleware/auth.ts";
import { parseCategoryIds } from "../validators/dump.ts";
import { getDump, replaceFileDump } from "../services/dump-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/files" });
const router: Router = new Router({ prefix: "/api/dumps" });
/**
* Builds a `Content-Disposition` value for a file called `name`. `Headers.set`
@@ -54,19 +63,9 @@ function isInlineSafe(mime: string): boolean {
return INLINE_MIME_RE.test(mime.split(";")[0].trim().toLowerCase());
}
router.get("/:dumpId", async (ctx) => {
router.get("/:dumpId/file", async (ctx) => {
const { dumpId } = ctx.params;
// The id names a file on disk: nothing but an id's characters gets through.
if (!/^[0-9a-f-]{36}$/.test(dumpId)) {
throw new APIException(400, "Invalid dump ID");
}
// Media elements can't send an Authorization header, so the viewer's media
// token comes in the query string.
const token = ctx.request.url.searchParams.get("token");
const viewerId = token ? await authenticateMedia(token) : null;
const dump = getDump(dumpId, viewerId ?? undefined);
const { dump, cacheControl } = await resolveMediaDump(ctx, dumpId);
if (dump.kind !== "file" || !dump.fileMime || !dump.fileName) {
throw new APIException(404, "No file for this dump");
@@ -88,21 +87,9 @@ router.get("/:dumpId", async (ctx) => {
"sandbox; default-src 'none'; img-src 'self'; media-src 'self'; style-src 'unsafe-inline'",
);
}
// The URL may carry a media token; don't pass it on to wherever the file
// links.
ctx.response.headers.set("Referrer-Policy", "no-referrer");
// A URL naming the file's current version (dumpFileVersion) always serves the
// same bytes, so it is cached for good; any other is revalidated against the
// ETag. A private dump's bytes must not sit in a shared cache.
const scope = dump.isPrivate ? "private" : "public";
const current = ctx.request.url.searchParams.get("v") ===
dumpFileVersion(dump);
const served = await serveFile(ctx, path, {
mime: dump.fileMime,
cacheControl: current
? `${scope}, max-age=31536000, immutable`
: `${scope}, no-cache`,
cacheControl,
ranges: true,
});
if (!served) {
@@ -110,4 +97,40 @@ router.get("/:dumpId", async (ctx) => {
}
});
router.put("/:dumpId/file", authMiddleware, async (ctx) => {
const userId = ctx.state.user.userId;
const dump = getDump(ctx.params.dumpId, userId);
const canModerate = can(ctx.state.user, "dump:moderate");
if (userId !== dump.userId && !canModerate) {
throw new APIException(403, "Not authorized to update dump");
}
const formData = await ctx.request.body.formData();
const file = formData.get("file");
const comment = formData.get("comment");
const title = formData.get("title");
const slug = formData.get("slug");
if (!(file instanceof File)) {
throw new APIException(400, "A file is required");
}
const updatedDump = await replaceFileDump(
dump.id,
file,
typeof comment === "string" && comment ? comment : undefined,
typeof title === "string" && title ? title : undefined,
formData.has("categoryIds")
? parseCategoryIds(formData.get("categoryIds"))
: undefined,
formData.has("isPrivate")
? formData.get("isPrivate") === "true"
: undefined,
typeof slug === "string" && slug ? slug : undefined,
canModerate,
);
const responseBody: APIResponse<Dump> = { success: true, data: updatedDump };
ctx.response.body = responseBody;
});
export default router;

91
api/routes/follows.ts
View File

@@ -1,118 +1,61 @@
/**
* Serves the endpoints for what the caller follows, and the index's "followed"
* tab built from it. What another user follows is served by `routes/users.ts`.
* Serves what the caller follows, and follows or stops following for them.
* The dumps of what they follow, the index's "follow" tab, are
* `GET /api/dumps?follow=`'s (`routes/dumps.ts`), and what another user
* follows is `routes/users.ts`'s.
*
* - `GET /api/follows/status` — signed in. Answers the ids of the users and
* - `GET /api/me/follow` — signed in. Answers the ids of the users and
* playlists they follow.
* - `GET /api/follows/feed/users?page=&limit=&category=` — signed in. Answers a
* page of the public dumps of the users they follow, newest first, optionally
* in one category.
* - `GET /api/follows/feed/playlists?page=&limit=&category=` — signed in.
* Answers a page of the public dumps in the public playlists they follow,
* most recently added first, optionally in one category.
* - `POST /api/follows/users/:userId` — signed in. Follows a user.
* - `DELETE /api/follows/users/:userId` — signed in. Stops following a user.
* - `POST /api/follows/playlists/:playlistId` — signed in. Follows a public
* playlist.
* - `DELETE /api/follows/playlists/:playlistId` — signed in. Stops following a
* - `PUT /api/me/follow/users/:userId` — signed in. Follows a user.
* - `DELETE /api/me/follow/users/:userId` — signed in. Stops following a user.
* - `PUT /api/me/follow/playlists/:playlistId` — signed in. Follows a public
* playlist.
* - `DELETE /api/me/follow/playlists/:playlistId` — signed in. Stops following
* a playlist.
*
* @module
*/
import { Router } from "@oak/oak";
import { authMiddleware } from "../middleware/auth.ts";
import type { APIResponse, PaginatedData } from "../../shared/api.ts";
import type { Dump } from "../../shared/dump.ts";
import type { APIResponse } from "../../shared/api.ts";
import type { FollowStatus } from "../../shared/follow.ts";
import { type Page, parsePagination } from "../lib/pagination.ts";
import {
followPlaylist,
followUser,
getFollowedPlaylistsDumpFeed,
getFollowedUsersDumpFeed,
getFollowStatus,
unfollowPlaylist,
unfollowUser,
} from "../services/follow-service.ts";
import { getCategoryIdBySlug } from "../services/category-service.ts";
/**
* Returns the id of the category the `?category=` slug names, undefined without
* one, or "missing" when it names none, so that the feed comes back empty
* rather than unfiltered.
*/
function resolveCategoryFilter(
params: URLSearchParams,
): string | "missing" | undefined {
const slug = params.get("category");
if (!slug) return undefined;
return getCategoryIdBySlug(slug) ?? "missing";
}
const EMPTY_FEED: Page<Dump> = { items: [], hasMore: false };
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/follows" });
const router: Router = new Router({ prefix: "/api/me" });
router.get("/status", authMiddleware, (ctx) => {
router.get("/follow", authMiddleware, (ctx) => {
const status = getFollowStatus(ctx.state.user.userId);
const body: APIResponse<FollowStatus> = { success: true, data: status };
ctx.response.body = body;
});
router.get("/feed/users", authMiddleware, (ctx) => {
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const categoryId = resolveCategoryFilter(ctx.request.url.searchParams);
const { items, hasMore } = categoryId === "missing"
? EMPTY_FEED
: getFollowedUsersDumpFeed(ctx.state.user.userId, page, limit, categoryId);
const data: PaginatedData<Dump> = {
items,
hasMore,
};
const body: APIResponse<PaginatedData<Dump>> = { success: true, data };
ctx.response.body = body;
});
router.get("/feed/playlists", authMiddleware, (ctx) => {
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const categoryId = resolveCategoryFilter(ctx.request.url.searchParams);
const { items, hasMore } = categoryId === "missing"
? EMPTY_FEED
: getFollowedPlaylistsDumpFeed(
ctx.state.user.userId,
page,
limit,
categoryId,
);
const data: PaginatedData<Dump> = {
items,
hasMore,
};
const body: APIResponse<PaginatedData<Dump>> = { success: true, data };
ctx.response.body = body;
});
router.post("/users/:userId", authMiddleware, (ctx) => {
router.put("/follow/users/:userId", authMiddleware, (ctx) => {
followUser(ctx.state.user.userId, ctx.params.userId);
ctx.response.status = 204;
});
router.delete("/users/:userId", authMiddleware, (ctx) => {
router.delete("/follow/users/:userId", authMiddleware, (ctx) => {
unfollowUser(ctx.state.user.userId, ctx.params.userId);
ctx.response.status = 204;
});
router.post("/playlists/:playlistId", authMiddleware, (ctx) => {
router.put("/follow/playlists/:playlistId", authMiddleware, (ctx) => {
followPlaylist(ctx.state.user.userId, ctx.params.playlistId);
ctx.response.status = 204;
});
router.delete("/playlists/:playlistId", authMiddleware, (ctx) => {
router.delete("/follow/playlists/:playlistId", authMiddleware, (ctx) => {
unfollowPlaylist(ctx.state.user.userId, ctx.params.playlistId);
ctx.response.status = 204;
});

198
api/routes/me.ts Normal file
View File

@@ -0,0 +1,198 @@
/**
* Serves the caller's own account: their user, settings, password and media
* token, and which of their playlists hold a dump. What they follow, their
* feed and their notifications are under `/api/me` too, served by
* `routes/follows.ts` and `routes/notifications.ts`, and their avatar by
* `routes/avatars.ts`.
*
* - `GET /api/me` — signed in. Answers the caller's own user.
* - `PATCH /api/me` — signed in. Changes the caller's description and email,
* and answers their user. The email of an account from another instance is
* that instance's: 400. A username changes only by `user:manage`, through
* `PATCH /api/users/:key/username`.
* - `GET /api/me/media-token` — signed in. Answers a media token, for a
* session that predates them.
* - `GET /api/me/settings` — signed in. Answers the caller's settings.
* - `PATCH /api/me/settings` — signed in. Changes some of the caller's
* settings, and answers them all.
* - `PUT /api/me/password` — signed in, with the current password. Sets a new
* one, ending every other session, and answers fresh tokens. 400 for an
* account from another instance, which has no password here.
* - `GET /api/me/playlists?contains=:dumpId` — signed in. Answers each of the
* caller's playlists and whether it holds the dump.
* - `GET /api/me/grants` — signed in. Answers the other instances the caller
* let their account sign in to.
* - `DELETE /api/me/grants/:host` — signed in. Withdraws that grant, which
* ends the caller's sessions there.
*
* A change to the caller's email or password is told to those instances
* (`notifyClients`).
*
* @module
*/
import { Router } from "@oak/oak";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import type {
AuthTokens,
MediaTokenResponse,
SelfUser,
} from "../../shared/user.ts";
import type { PlaylistMembership } from "../../shared/playlist.ts";
import type { UserSettings } from "../../shared/settings.ts";
import { fieldOf, isUpdateUserRequest } from "../validators/user.ts";
import { isUpdateSettingsRequest } from "../validators/settings.ts";
import { toPublicUser, toSelfUser } from "../model/user.ts";
import { createMediaToken, issueTokens } from "../lib/jwt.ts";
import { verifyPassword } from "../lib/password.ts";
import { type AuthContext, authMiddleware } from "../middleware/auth.ts";
import {
assertLocalUser,
getUserById,
updateUser,
} from "../services/user-service.ts";
import { getSettings, updateSettings } from "../services/settings-service.ts";
import { getPlaylistMembershipsForDump } from "../services/playlist-service.ts";
import { broadcastUserUpdated } from "../services/ws-service.ts";
import { VALIDATION } from "../../shared/validation.ts";
import type { FederationGrant } from "../../shared/federation.ts";
import {
listGrants,
notifyClients,
revokeGrant,
} from "../services/identity-provider-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/me" });
router.get("/", authMiddleware, (ctx: AuthContext) => {
const responseBody: APIResponse<SelfUser> = {
success: true,
data: toSelfUser(getUserById(ctx.state.user.userId)),
};
ctx.response.body = responseBody;
});
router.patch("/", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
if (!isUpdateUserRequest(body)) {
throw new APIException(400, "Invalid request");
}
const userId = ctx.state.user.userId;
if (body.email !== undefined) assertLocalUser(getUserById(userId));
// Only these fields: a stolen token alone must not change the role or the
// password, which needs the current one.
const updated = await updateUser(userId, {
description: body.description,
email: body.email,
});
broadcastUserUpdated(toPublicUser(updated));
if (body.email !== undefined) await notifyClients(userId, { type: "update" });
const responseBody: APIResponse<SelfUser> = {
success: true,
data: toSelfUser(updated),
};
ctx.response.body = responseBody;
});
router.get("/media-token", authMiddleware, async (ctx: AuthContext) => {
const responseBody: APIResponse<MediaTokenResponse> = {
success: true,
data: { mediaToken: await createMediaToken(ctx.state.user.userId) },
};
ctx.response.body = responseBody;
});
router.get("/settings", authMiddleware, (ctx: AuthContext) => {
const responseBody: APIResponse<UserSettings> = {
success: true,
data: getSettings(ctx.state.user.userId),
};
ctx.response.body = responseBody;
});
router.patch("/settings", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
if (!isUpdateSettingsRequest(body)) {
throw new APIException(400, "Invalid settings");
}
const responseBody: APIResponse<UserSettings> = {
success: true,
data: updateSettings(ctx.state.user.userId, body),
};
ctx.response.body = responseBody;
});
router.put("/password", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
const currentPassword = fieldOf(body, "currentPassword");
const newPassword = fieldOf(body, "newPassword");
if (typeof currentPassword !== "string" || typeof newPassword !== "string") {
throw new APIException(400, "Invalid request");
}
if (
newPassword.length < VALIDATION.PASSWORD_MIN ||
newPassword.length > VALIDATION.PASSWORD_MAX
) {
throw new APIException(
400,
`Password must be ${VALIDATION.PASSWORD_MIN}–${VALIDATION.PASSWORD_MAX} characters`,
);
}
const user = getUserById(ctx.state.user.userId);
assertLocalUser(user);
const valid = await verifyPassword(currentPassword, user.passwordHash ?? "");
if (!valid) {
throw new APIException(403, "Current password is incorrect");
}
// Changing the password revokes every token issued before it, this
// session's included, so hand back fresh ones for the client to keep.
const updated = await updateUser(ctx.state.user.userId, {
password: newPassword,
});
await notifyClients(updated.id, { type: "revoke" });
const responseBody: APIResponse<AuthTokens> = {
success: true,
data: await issueTokens(updated),
};
ctx.response.body = responseBody;
});
router.get("/playlists", authMiddleware, (ctx: AuthContext) => {
const dumpId = ctx.request.url.searchParams.get("contains");
if (!dumpId) throw new APIException(400, "contains is required");
const responseBody: APIResponse<PlaylistMembership[]> = {
success: true,
data: getPlaylistMembershipsForDump(dumpId, ctx.state.user.userId),
};
ctx.response.body = responseBody;
});
router.get("/grants", authMiddleware, (ctx: AuthContext) => {
const responseBody: APIResponse<FederationGrant[]> = {
success: true,
data: listGrants(ctx.state.user.userId),
};
ctx.response.body = responseBody;
});
router.delete(
"/grants/:host",
authMiddleware,
async (ctx: AuthContext<"/grants/:host">) => {
await revokeGrant(ctx.state.user.userId, ctx.params.host);
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
},
);
export default router;

10
api/routes/notifications.ts
View File

@@ -3,9 +3,9 @@
* page, which marks them all read when it opens. New ones arrive over the
* WebSocket instead, as `notification_created`.
*
* - `GET /api/notifications?page=&limit=` — signed in. Answers a page of them,
* newest first.
* - `POST /api/notifications/read-all` — signed in. Marks them all read.
* - `GET /api/me/notifications?page=&limit=` — signed in. Answers a page of
* them, newest first.
* - `POST /api/me/notifications/read` — signed in. Marks them all read.
*
* @module
*/
@@ -24,7 +24,7 @@ import {
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/notifications" });
const router: Router = new Router({ prefix: "/api/me/notifications" });
router.get("/", authMiddleware, (ctx: AuthContext) => {
const { page, limit } = parsePagination(ctx.request.url.searchParams);
@@ -40,7 +40,7 @@ router.get("/", authMiddleware, (ctx: AuthContext) => {
ctx.response.body = responseBody;
});
router.post("/read-all", authMiddleware, (ctx: AuthContext) => {
router.post("/read", authMiddleware, (ctx: AuthContext) => {
markAllRead(ctx.state.user.userId);
ctx.response.status = 204;
});

78
api/routes/oauth.ts Normal file
View File

@@ -0,0 +1,78 @@
/**
* Lets this instance's users sign in to other gerbeur instances with their
* account here, as `identity-provider-service.ts` describes. The client's
* `/authorize` page calls the first two; the other instance's server calls the
* last.
*
* - `GET /api/oauth/clients?client_id=` — signed in. Answers the instance
* asking to sign the caller in (`OAuthClient`), and whether they already
* let it; 400 for an account from another instance, 403 when that instance
* is blocked here, 502 when it doesn't answer as a gerbeur.
* - `POST /api/oauth/codes` — signed in. Records the caller's consent to the
* request (`AuthorizeRequest`) and answers where to send them, the
* instance's callback page with a code.
* - `POST /api/oauth/tokens` — public, form-encoded, server to server. Trades
* a code and its PKCE verifier for an identity token (`id_token`), bare
* JSON as OAuth has it; 400 when anything doesn't match. Rate-limited per
* client.
*
* @module
*/
import { Router } from "@oak/oak";
import type { APIResponse } from "../../shared/api.ts";
import type {
AuthorizeResponse,
OAuthClient,
} from "../../shared/federation.ts";
import { APIException } from "../lib/api-exception.ts";
import { type RateLimit, takeToken } from "../lib/rate-limit.ts";
import { type AuthContext, authMiddleware } from "../middleware/auth.ts";
import { isAuthorizeRequest } from "../validators/federation.ts";
import {
createAuthorizationCode,
describeClient,
exchangeCode,
} from "../services/identity-provider-service.ts";
/** Allows a burst of 30 code trades per client instance, then one a second. */
const TOKEN_LIMIT: RateLimit = { capacity: 30, refill: 1 };
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/oauth" });
router.get("/clients", authMiddleware, async (ctx: AuthContext) => {
const clientId = ctx.request.url.searchParams.get("client_id");
if (!clientId) throw new APIException(400, "client_id is required");
const responseBody: APIResponse<OAuthClient> = {
success: true,
data: await describeClient(ctx.state.user.userId, clientId),
};
ctx.response.body = responseBody;
});
router.post("/codes", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
if (!isAuthorizeRequest(body)) throw new APIException(400, "Invalid request");
const responseBody: APIResponse<AuthorizeResponse> = {
success: true,
data: {
redirectTo: await createAuthorizationCode(ctx.state.user.userId, body),
},
};
ctx.response.body = responseBody;
});
router.post("/tokens", async (ctx) => {
const params = new URLSearchParams(await ctx.request.body.text());
if (!takeToken(`oauth-token:${params.get("client_id")}`, TOKEN_LIMIT)) {
throw new APIException(429, "Too many requests");
}
ctx.response.headers.set("Cache-Control", "no-store");
ctx.response.body = await exchangeCode(params);
});
export default router;

59
api/routes/password-resets.ts Normal file
View File

@@ -0,0 +1,59 @@
/**
* Resets forgotten passwords. The client's `/login` page asks for a reset
* link, which the email takes to the `/reset-password` page, whose token
* names the reset.
*
* - `POST /api/password-resets` — public. Emails a reset link to the account
* with that `email`, if any; always answers 200.
* - `PUT /api/password-resets/:token` — public, with a reset token. Sets the
* `newPassword`, which ends the user's sessions here and on the instances
* they signed in to.
*
* @module
*/
import { Router } from "@oak/oak";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import { fieldOf } from "../validators/user.ts";
import {
requestPasswordReset,
resetPassword,
} from "../services/password-reset-service.ts";
import { notifyClients } from "../services/identity-provider-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/password-resets" });
// Answers the same whatever happens, so it can't tell which addresses have an
// account.
router.post("/", async (ctx) => {
const body: unknown = await ctx.request.body.json();
const email = fieldOf(body, "email");
const trimmed = typeof email === "string" ? email.trim() : "";
if (trimmed) {
await requestPasswordReset(trimmed).catch((err) =>
console.error("[password-resets]", err)
);
}
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
});
router.put("/:token", async (ctx) => {
const body: unknown = await ctx.request.body.json();
const newPassword = fieldOf(body, "newPassword");
if (typeof newPassword !== "string") {
throw new APIException(400, "Invalid request");
}
const userId = await resetPassword(ctx.params.token, newPassword);
await notifyClients(userId, { type: "revoke" });
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
});
export default router;

54
api/routes/playlists.ts
View File

@@ -1,46 +1,50 @@
/**
* Serves playlists, their dumps and their covers. `:playlistId` takes an id or
* a slug.
* Serves playlists, their dumps and their covers. `:playlistId` is a
* playlist's id; its page reads it by its path, through
* `GET /api/users/:key/playlists/:slug`.
*
* - `GET /api/playlists/by-dump/:dumpId/memberships` — signed in. Answers each
* of the caller's playlists and whether it holds the dump.
* - `POST /api/playlists` — signed in. Creates a playlist.
* - `GET /api/playlists?category=&page=&limit=` — public. Answers a page of
* every public playlist, newest first, or only those in the playlist
* category at `category`, which 404s when there is none.
* - `POST /api/playlists` — signed in. Creates a playlist, filed under the
* playlist categories it names.
* - `GET /api/playlists/:playlistId` — optional sign-in. Answers a playlist
* with the dumps the caller may see in it. A private one 404s but for its
* owner.
* - `PUT /api/playlists/:playlistId` — signed in, as its owner or with
* `playlist:moderate`. Updates its title, description and visibility.
* `playlist:moderate`. Updates its title, slug, description, visibility and
* categories.
* A new slug keeps the old one as an alias, 3 times a day at most but for
* those with `playlist:moderate`.
* - `DELETE /api/playlists/:playlistId` — signed in, as its owner or with
* `playlist:moderate`. Deletes it.
* - `POST /api/playlists/:playlistId/dumps/:dumpId` — signed in, as its owner
* or with `playlist:moderate`. Adds a dump at the top.
* - `DELETE /api/playlists/:playlistId/dumps/:dumpId` — signed in, as its owner
* or with `playlist:moderate`. Removes a dump.
* - `POST /api/playlists/:playlistId/image` — signed in, as its owner or with
* - `PUT /api/playlists/:playlistId/cover` — signed in, as its owner or with
* `playlist:moderate`. Replaces its cover (multipart `file`, at most 5 MB).
* - `GET /api/playlists/:playlistId/image?token=` — public, but a private
* - `GET /api/playlists/:playlistId/cover?token=` — public, but a private
* playlist's cover needs its owner's media `token`. Serves its cover, or 404s
* without one.
* - `PUT /api/playlists/:playlistId/order` — signed in, as its owner or with
* `playlist:moderate`. Reorders the dumps the caller can see in it.
*
* Which of the caller's playlists hold a dump is `GET /api/me/playlists`'s.
*
* @module
*/
import { Router } from "@oak/oak";
import { authenticateMedia, parseOptionalAuth } from "../lib/auth.ts";
import { type APIResponse } from "../../shared/api.ts";
import { type APIResponse, type PaginatedData } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import {
isCreatePlaylistRequest,
isReorderPlaylistRequest,
isUpdatePlaylistRequest,
} from "../validators/playlist.ts";
import type {
Playlist,
PlaylistMembership,
PlaylistWithDumps,
} from "../../shared/playlist.ts";
import type { Playlist, PlaylistWithDumps } from "../../shared/playlist.ts";
import { authMiddleware, type AuthState } from "../middleware/auth.ts";
import { can } from "../lib/permissions.ts";
import {
@@ -50,7 +54,7 @@ import {
getEditablePlaylist,
getPlaylist,
getPlaylistImageInfo,
getPlaylistMembershipsForDump,
listPublicPlaylists,
removeDumpFromPlaylist,
reorderPlaylist,
setPlaylistImage,
@@ -62,6 +66,8 @@ import {
validateImageUpload,
} from "../lib/upload.ts";
import { PLAYLIST_IMAGES_DIR } from "../config.ts";
import { parsePagination } from "../lib/pagination.ts";
import { getCategoryBySlug } from "../services/category-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
@@ -71,13 +77,15 @@ const router: Router<AuthState> = new Router<AuthState>({
prefix: "/api/playlists",
});
// Before /:playlistId, which would otherwise match "by-dump".
router.get("/by-dump/:dumpId/memberships", authMiddleware, (ctx) => {
const { dumpId } = ctx.params;
const userId = ctx.state.user.userId;
const responseBody: APIResponse<PlaylistMembership[]> = {
router.get("/", (ctx) => {
const params = ctx.request.url.searchParams;
const { page, limit } = parsePagination(params);
const slug = params.get("category");
const categoryId = slug ? getCategoryBySlug("playlist", slug).id : undefined;
const { items, hasMore } = listPublicPlaylists(page, limit, categoryId);
const responseBody: APIResponse<PaginatedData<Playlist>> = {
success: true,
data: getPlaylistMembershipsForDump(dumpId, userId),
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
@@ -148,7 +156,7 @@ router.delete("/:playlistId/dumps/:dumpId", authMiddleware, (ctx) => {
ctx.response.status = 204;
});
router.post("/:playlistId/image", authMiddleware, async (ctx) => {
router.put("/:playlistId/cover", authMiddleware, async (ctx) => {
const contentType = ctx.request.headers.get("content-type") ?? "";
if (!contentType.includes("multipart/form-data")) {
throw new APIException(400, "Expected multipart/form-data");
@@ -182,7 +190,7 @@ router.post("/:playlistId/image", authMiddleware, async (ctx) => {
ctx.response.body = responseBody;
});
router.get("/:playlistId/image", async (ctx) => {
router.get("/:playlistId/cover", async (ctx) => {
// An <img> can't send an Authorization header, so a private playlist's owner
// is known by the media token in the query string, as for dump files.
const token = ctx.request.url.searchParams.get("token");

60
api/routes/preview.ts → api/routes/remote.ts
View File

@@ -1,11 +1,19 @@
/**
* Fetches link previews for the create form, and proxies the remote images
* that the frontend's thumbnails can't load directly.
* Reads other sites for the client, as `api/remote/` does for the server: the
* previews of links on the create form, the images its thumbnails can't load
* directly, and the tracks the players list. On any failure of the last two,
* the player falls back to the embed.
*
* - `GET /api/preview?url=` — signed in. Answers the preview of a link, and
* whether its page could be reached.
* - `GET /api/proxy-image?url=` — public. Serves a remote raster image through
* this site; 400 when it isn't one, 502 when its host can't be reached.
* - `GET /api/remote/preview?url=` — signed in. Answers the preview of a link,
* and whether its page could be reached.
* - `GET /api/remote/image?url=` — public. Serves a remote raster image
* through this site; 400 when it isn't one, 502 when its host can't be
* reached.
* - `GET /api/remote/bandcamp?url=&force=` — public. Answers a Bandcamp
* page's tracks and their stream URLs, for the native player; `force=1`
* skips the server's cache.
* - `GET /api/remote/youtube?list=` — public. Answers a YouTube playlist's
* tracklist, which the global player lists beside the embed.
*
* @module
*/
@@ -13,6 +21,8 @@
import { Router } from "@oak/oak";
import { isValidHttpUrl } from "../remote/fetch.ts";
import { fetchProxiedImage } from "../remote/image-proxy.ts";
import { resolveBandcamp } from "../remote/bandcamp-streams.ts";
import { resolveYouTubePlaylist } from "../remote/youtube-playlist.ts";
import { tryFetchRichContent } from "../services/rich-content-service.ts";
import {
APIErrorCode,
@@ -20,16 +30,17 @@ import {
type APIResponse,
} from "../../shared/api.ts";
import type { LinkPreview } from "../../shared/dump.ts";
import type { Tralbum, YouTubePlaylist } from "../../shared/player.ts";
import { authMiddleware } from "../middleware/auth.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const previewRouter: Router = new Router();
const router: Router = new Router({ prefix: "/api/remote" });
// Signed-in only: an open preview would let anyone make the server fetch pages.
previewRouter.get("/api/preview", authMiddleware, async (ctx) => {
router.get("/preview", authMiddleware, async (ctx) => {
const url = ctx.request.url.searchParams.get("url") ?? "";
if (!isValidHttpUrl(url)) {
const responseBody: APIFailure = {
@@ -69,7 +80,7 @@ function sendImage(
// The frontend sends a thumbnail here when it is plain HTTP, which an HTTPS
// page won't load, or when its host's hotlink protection refused the browser.
previewRouter.get("/api/proxy-image", async (ctx) => {
router.get("/image", async (ctx) => {
const url = ctx.request.url.searchParams.get("url") ?? "";
if (!isValidHttpUrl(url)) {
ctx.response.status = 400;
@@ -88,4 +99,33 @@ previewRouter.get("/api/proxy-image", async (ctx) => {
}
});
export default previewRouter;
// Public: it reads a public page and answers the URLs that page gives any
// visitor.
router.get("/bandcamp", async (ctx) => {
const url = ctx.request.url.searchParams.get("url") ?? "";
const force = ctx.request.url.searchParams.get("force") === "1";
const responseBody: APIResponse<Tralbum> = {
success: true,
data: await resolveBandcamp(url, { force }),
};
// Not in a shared cache: the signed URLs belong to one page fetch. An hour
// is well inside their 24 hours.
ctx.response.headers.set("Cache-Control", "private, max-age=3600");
ctx.response.body = responseBody;
});
// Public, like /bandcamp: it only reads a public page.
router.get("/youtube", async (ctx) => {
const listId = ctx.request.url.searchParams.get("list") ?? "";
const responseBody: APIResponse<YouTubePlaylist> = {
success: true,
data: await resolveYouTubePlaylist(listId),
};
ctx.response.headers.set("Cache-Control", "public, max-age=3600");
ctx.response.body = responseBody;
});
export default router;

26
api/routes/search.ts
View File

@@ -3,9 +3,11 @@
* Only the dumps are paged. Private dumps and playlists are found only by
* their owner.
*
* - `GET /api/search?q=&page=&limit=` — optional sign-in. Answers a page of the
* matching dumps with their total, and up to 20 matching users and
* playlists. A blank `q` finds nothing.
* - `GET /api/search?q=&page=&limit=&category=&playlistCategory=` — optional
* sign-in. Answers a page of the matching dumps with their total, and up to
* 20 matching users and playlists. A blank `q` finds nothing. `category`
* keeps the dumps in that dump category, `playlistCategory` the playlists in
* that playlist category; a slug that names none finds nothing.
*
* @module
*/
@@ -16,6 +18,7 @@ import { parsePagination } from "../lib/pagination.ts";
import { searchDumps } from "../services/dump-service.ts";
import { searchUsersGlobal } from "../services/user-service.ts";
import { searchPlaylists } from "../services/playlist-service.ts";
import { resolveCategoryFilter } from "../services/category-service.ts";
import type { APIResponse } from "../../shared/api.ts";
import type { SearchResults } from "../../shared/search.ts";
@@ -44,14 +47,19 @@ router.get("/", async (ctx) => {
return;
}
const { items: dumpItems, total: dumpTotal } = searchDumps(
q,
page,
limit,
requestingUserId,
const dumpCategoryId = resolveCategoryFilter("dump", params);
const playlistCategoryId = resolveCategoryFilter(
"playlist",
params,
"playlistCategory",
);
const { items: dumpItems, total: dumpTotal } = dumpCategoryId === "missing"
? { items: [], total: 0 }
: searchDumps(q, page, limit, requestingUserId, dumpCategoryId);
const users = searchUsersGlobal(q, 20);
const playlists = searchPlaylists(q, 20, requestingUserId);
const playlists = playlistCategoryId === "missing"
? []
: searchPlaylists(q, 20, requestingUserId, playlistCategoryId);
const responseBody: APIResponse<SearchResults> = {
success: true,

74
api/routes/sessions.ts Normal file
View File

@@ -0,0 +1,74 @@
/**
* Signs users in, from the client's `/login` page. A session is a token the
* client keeps; signing out forgets it, with no request.
*
* - `POST /api/sessions` — public. Answers the user and their tokens;
* rate-limited per account.
*
* @module
*/
import { Router } from "@oak/oak";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import type { AuthResponse } from "../../shared/user.ts";
import { isLoginUserRequest } from "../validators/user.ts";
import { toSelfUser } from "../model/user.ts";
import { issueTokens } from "../lib/jwt.ts";
import { hashPassword, verifyPassword } from "../lib/password.ts";
import { type RateLimit, takeToken } from "../lib/rate-limit.ts";
import { getUserByUsername } from "../services/user-service.ts";
// Hashed once, at startup, for the sign-ins of usernames that don't exist.
const UNKNOWN_USER_HASH: Promise<string> = hashPassword(crypto.randomUUID());
/** Allows a burst of 10 login attempts per account, then one every 30s. */
const LOGIN_LIMIT: RateLimit = { capacity: 10, refill: 1 / 30 };
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/sessions" });
router.post("/", async (ctx) => {
const body: unknown = await ctx.request.body.json();
if (!isLoginUserRequest(body)) {
throw new APIException(400, "Invalid request");
}
// Per account, so guessing one user's password is slow however many
// connections it comes from.
if (!takeToken(`login:${body.username.toLowerCase()}`, LOGIN_LIMIT)) {
throw new APIException(
429,
"Too many login attempts — try again in a minute",
);
}
let user;
try {
user = getUserByUsername(body.username);
} catch {
user = undefined;
}
// An unknown username gets the same answer as a wrong password, after the
// same work, so neither the message nor the timing tells which exist.
const valid = await verifyPassword(
body.password,
user?.passwordHash ?? await UNKNOWN_USER_HASH,
);
if (!user || !valid) {
throw new APIException(401, "Invalid username or password");
}
const responseBody: APIResponse<AuthResponse> = {
success: true,
data: { ...await issueTokens(user), user: toSelfUser(user) },
};
ctx.response.body = responseBody;
});
export default router;

277
api/routes/thumbnails.ts
View File

@@ -1,207 +1,72 @@
/**
* Serves dumps' thumbnails to the cards and pages that show a file dump, which
* build the URL with `dumpThumbnailUrl`. Thumbnails made from an upload are
* generated with ffmpeg on first request and kept in `THUMBNAILS_DIR`.
* Serves and changes dumps' thumbnails, a sub-resource of their dump. The
* cards and pages that show a file dump build the URL with
* `dumpThumbnailUrl`, and the edit page uploads a custom one. Thumbnails made
* from an upload are generated on first request by lib/ffmpeg.ts, which keeps
* them.
*
* - `GET /api/thumbnails/:dumpId?v=&token=` — public; a private dump's
* - `GET /api/dumps/:dumpId/thumbnail?v=&token=` — public; a private dump's
* thumbnail needs its poster's media `token`. Serves the dump's custom
* thumbnail, else one made from its upload: a still of a video, or a
* scaled-down copy of an image (the image itself when none can be made); 404
* when there is neither.
* - `PUT /api/dumps/:dumpId/thumbnail` — signed in, as its poster or with
* `dump:moderate`. Replaces its custom thumbnail (multipart `file`, at most 5
* MB).
* - `DELETE /api/dumps/:dumpId/thumbnail` — signed in, as its poster or with
* `dump:moderate`. Removes its custom thumbnail.
*
* @module
*/
import { Router } from "@oak/oak";
import { APIException } from "../lib/api-exception.ts";
import type { APIResponse } from "../../shared/api.ts";
import type { Dump } from "../../shared/dump.ts";
import {
dumpFileVersion,
customThumbnailPath,
getDump,
getDumpThumbnailInfo,
removeDumpThumbnail,
setDumpThumbnail,
} from "../services/dump-service.ts";
import { serveFile } from "../lib/serve-file.ts";
import { serveUploadedFile } from "../lib/upload.ts";
import { DUMPS_DIR, THUMBNAILS_DIR } from "../config.ts";
import { authenticateMedia } from "../lib/auth.ts";
import {
replaceUploadedFile,
serveUploadedFile,
validateImageUpload,
} from "../lib/upload.ts";
import { DUMPS_DIR } from "../config.ts";
import { resolveMediaDump } from "../lib/dump-media.ts";
import {
ensureGenerated,
generatedPaths,
imagePreview,
PREVIEWABLE_IMAGES,
videoStill,
} from "../lib/ffmpeg.ts";
import { can } from "../lib/permissions.ts";
import { type AuthContext, authMiddleware } from "../middleware/auth.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/thumbnails" });
const router: Router = new Router({ prefix: "/api/dumps" });
/**
* Spares a process per uncached thumbnail request: whether ffmpeg runs is
* probed once per process, and kept here.
*/
let ffmpegProbe: Promise<boolean> | undefined;
function ffmpegAvailable(): Promise<boolean> {
ffmpegProbe ??= (async () => {
try {
const { success } = await new Deno.Command("ffmpeg", {
args: ["-version"],
stderr: "null",
stdout: "null",
}).output();
return success;
} catch {
return false;
}
})();
return ffmpegProbe;
}
/**
* Lists the container formats ffmpeg may read a video as. The upload's type is
* only what the uploader claimed, and ffmpeg guesses the format from the bytes,
* so a "video" could be an HLS or concat playlist pointing it at local files or
* internal URLs. With this list, and `-protocol_whitelist file`, ffmpeg reads
* nothing but the file, and only as a video.
*/
const VIDEO_FORMATS =
"mov,mp4,m4a,3gp,3g2,mj2,matroska,webm,avi,flv,mpegts,mpeg,ogg,asf,h264,hevc";
/**
* Lists the formats ffmpeg may read an image as, for the reason given at
* {@link VIDEO_FORMATS}.
*/
const IMAGE_FORMATS = "jpeg_pipe,png_pipe,webp_pipe,bmp_pipe";
/**
* Maps the image types that get a scaled-down preview to the preview's format.
* A GIF is served as it is, to keep its animation.
*/
const PREVIEWABLE_IMAGES: Record<string, "jpg" | "png"> = {
"image/jpeg": "jpg",
"image/bmp": "jpg",
// May carry transparency, which JPEG would fill with black.
"image/png": "png",
"image/webp": "png",
};
/**
* Caps an image preview's width. It covers the widest feed card on a
* high-density screen, at a fraction of a phone photo's bytes.
*/
const PREVIEW_WIDTH = 960;
/**
* Runs ffmpeg with `args`, writing to `outPath`.
*
* @throws {Error} When ffmpeg fails.
*/
async function runFfmpeg(args: string[], outPath: string): Promise<void> {
await Deno.mkdir(THUMBNAILS_DIR, { recursive: true });
// Written beside and renamed over, so a concurrent reader never gets half.
const tmpPath = `${outPath}.${crypto.randomUUID()}.tmp`;
const { success } = await new Deno.Command("ffmpeg", {
args: ["-protocol_whitelist", "file", ...args, "-y", tmpPath],
stdout: "null",
stderr: "null",
}).output();
if (!success) {
await Deno.remove(tmpPath).catch(() => {});
throw new Error("ffmpeg failed");
}
await Deno.rename(tmpPath, outPath);
}
/** Writes a 320-pixel-wide JPEG still of a video, one second in. */
function videoStill(srcPath: string, outPath: string): Promise<void> {
return runFfmpeg([
"-format_whitelist",
VIDEO_FORMATS,
"-ss",
"00:00:01",
"-i",
srcPath,
"-frames:v",
"1",
"-vf",
"scale=320:-1",
"-f",
"image2",
"-c:v",
"mjpeg",
], outPath);
}
/** Writes a copy of an image at most {@link PREVIEW_WIDTH} wide. */
function imagePreview(srcPath: string, outPath: string): Promise<void> {
return runFfmpeg([
"-format_whitelist",
IMAGE_FORMATS,
"-i",
srcPath,
"-frames:v",
"1",
"-vf",
`scale='min(${PREVIEW_WIDTH},iw)':-2`,
"-f",
"image2",
...(outPath.endsWith(".png")
? ["-c:v", "png"]
: ["-c:v", "mjpeg", "-q:v", "4"]),
], outPath);
}
/**
* Lets a feed full of cards asking for the same new still at once wait on a
* single ffmpeg run. It holds the generations under way, by output path.
*/
const inFlight = new Map<string, Promise<void>>();
/**
* Makes `outPath` with `generate` unless it exists. Resolves false when it
* can't be made, because ffmpeg is missing or failed.
*/
async function ensureGenerated(
outPath: string,
generate: () => Promise<void>,
): Promise<boolean> {
if (await Deno.stat(outPath).then(() => true, () => false)) return true;
if (!await ffmpegAvailable()) return false;
let pending = inFlight.get(outPath);
if (!pending) {
pending = generate().finally(() => inFlight.delete(outPath));
inFlight.set(outPath, pending);
}
return await pending.then(() => true, () => false);
}
router.get("/:dumpId", async (ctx) => {
router.get("/:dumpId/thumbnail", async (ctx) => {
const { dumpId } = ctx.params;
const { dump, cacheControl: generatedCache } = await resolveMediaDump(
ctx,
dumpId,
);
// The id names files on disk: nothing but an id's characters gets through.
if (!/^[0-9a-f-]{36}$/.test(dumpId)) {
throw new APIException(400, "Invalid dump ID");
}
// Media elements can't send an Authorization header, so the viewer's media
// token comes in the query string. This privacy check covers the custom
// thumbnail too, since getDumpThumbnailInfo makes none.
const token = ctx.request.url.searchParams.get("token");
const viewerId = token ? await authenticateMedia(token) : null;
const dump = getDump(dumpId, viewerId ?? undefined);
// The URL may carry a media token, which must not leak through a Referer.
ctx.response.headers.set("Referrer-Policy", "no-referrer");
// What is generated from the file is cached for good under a URL naming its
// current version (dumpFileVersion), and revalidated under any other. A
// private dump's thumbnail must not sit in a shared cache.
const scope = dump.isPrivate ? "private" : "public";
const generatedCache =
ctx.request.url.searchParams.get("v") === dumpFileVersion(dump)
? `${scope}, max-age=31536000, immutable`
: `${scope}, no-cache`;
const customThumb = getDumpThumbnailInfo(dumpId);
if (customThumb) {
if (dump.thumbnailMime) {
// Replacing it doesn't change the file's version, so it is always
// revalidated.
const served = await serveUploadedFile(
ctx,
`${THUMBNAILS_DIR}/${customThumb.id}-custom`,
customThumb.mime,
customThumbnailPath(dump.id),
dump.thumbnailMime,
0,
dump.isPrivate,
);
@@ -210,9 +75,10 @@ router.get("/:dumpId", async (ctx) => {
const mime = dump.kind === "file" ? dump.fileMime ?? "" : "";
const srcPath = `${DUMPS_DIR}/${dump.id}`;
const generated = generatedPaths(dump.id);
if (mime.startsWith("video/")) {
const thumbPath = `${THUMBNAILS_DIR}/${dump.id}.jpg`;
const thumbPath = generated.still;
if (
!await ensureGenerated(thumbPath, () => videoStill(srcPath, thumbPath))
) {
@@ -232,7 +98,7 @@ router.get("/:dumpId", async (ctx) => {
ctx.response.headers.set("X-Content-Type-Options", "nosniff");
const ext = PREVIEWABLE_IMAGES[mime];
if (ext) {
const previewPath = `${THUMBNAILS_DIR}/${dump.id}.preview.${ext}`;
const previewPath = generated.preview[ext];
if (
await ensureGenerated(
previewPath,
@@ -253,4 +119,59 @@ router.get("/:dumpId", async (ctx) => {
throw new APIException(404, "No thumbnail for this dump");
});
/**
* Returns the dump `ctx` names if the caller may change it, as its poster or
* with `dump:moderate`.
*
* @throws {APIException} 404 when the caller can't see it, 403 when they may
* not change it.
*/
function editableDump(ctx: AuthContext<"/:dumpId/thumbnail">): Dump {
const userId = ctx.state.user.userId;
const dump = getDump(ctx.params.dumpId, userId);
if (userId !== dump.userId && !can(ctx.state.user, "dump:moderate")) {
throw new APIException(403, "Not authorized to update dump");
}
return dump;
}
router.put(
"/:dumpId/thumbnail",
authMiddleware,
async (ctx: AuthContext<"/:dumpId/thumbnail">) => {
const dump = editableDump(ctx);
const formData = await ctx.request.body.formData();
const file = formData.get("file");
if (!(file instanceof File)) {
throw new APIException(400, "Missing file field");
}
const data = new Uint8Array(await file.arrayBuffer());
const mime = validateImageUpload(data);
// Written before setDumpThumbnail announces it, or clients would fetch it
// too early.
await replaceUploadedFile(customThumbnailPath(dump.id), data);
const responseBody: APIResponse<Dump> = {
success: true,
data: setDumpThumbnail(dump.id, mime),
};
ctx.response.body = responseBody;
},
);
router.delete(
"/:dumpId/thumbnail",
authMiddleware,
async (ctx: AuthContext<"/:dumpId/thumbnail">) => {
const dump = editableDump(ctx);
const responseBody: APIResponse<Dump> = {
success: true,
data: await removeDumpThumbnail(dump.id),
};
ctx.response.body = responseBody;
},
);
export default router;

49
api/routes/unsubscribe.ts Normal file
View File

@@ -0,0 +1,49 @@
/**
* Turns a digest off from the link in its email, without signing in. Mail
* clients post here for a one-click unsubscribe (RFC 8058), and so does the
* client's `/unsubscribe` page.
*
* - `POST /api/unsubscribe?token=` — public, with a digest's unsubscribe
* token. Turns that digest off and answers its cadence.
* - `POST /api/users/unsubscribe?token=` — the same, at its v3 path, which the
* `List-Unsubscribe` header of every digest sent before v4 names. Those
* tokens never expire, so this path stays for good.
*
* @module
*/
import { Router, type RouterMiddleware } from "@oak/oak";
import { type APIResponse } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import { DIGEST_SETTING, type DigestCadence } from "../../shared/settings.ts";
import { verifyUnsubscribeToken } from "../lib/jwt.ts";
import { getUserById } from "../services/user-service.ts";
import { updateSettings } from "../services/settings-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router();
// The body is ignored: a mail client's one-click post sends a form saying
// "List-Unsubscribe=One-Click", the client's page sends nothing.
const unsubscribe: RouterMiddleware<string> = async (ctx) => {
const token = ctx.request.url.searchParams.get("token") ?? "";
const payload = await verifyUnsubscribeToken(token);
if (!payload) throw new APIException(400, "Invalid unsubscribe link");
// Throws a 404 once the account is gone.
getUserById(payload.userId);
updateSettings(payload.userId, { [DIGEST_SETTING[payload.cadence]]: false });
const responseBody: APIResponse<{ cadence: DigestCadence }> = {
success: true,
data: { cadence: payload.cadence },
};
ctx.response.body = responseBody;
};
router.post("/api/unsubscribe", unsubscribe);
router.post("/api/users/unsubscribe", unsubscribe);
export default router;

507
api/routes/users.ts
View File

@@ -1,42 +1,45 @@
/**
* Serves accounts (sign-up, sign-in, tokens, passwords) and user profiles,
* with what each user posted, upvoted, follows and invited. A profile sent to
* anyone but its user goes through `toPublicUser`, which drops the email.
* Serves sign-up and user profiles, with what each user posted, upvoted, made,
* follows and invited, and the dumps and playlists under them. A user's `:key`
* is their id or `@name`, a current or former name in any case
* (`resolveUserKey`). A profile sent to anyone goes through `toPublicUser`,
* which drops the email. The caller's own account is `routes/me.ts`'s.
*
* - `POST /api/users/register` — public, with a valid invite. Creates an
* account, sends the welcome email, and answers the user and their tokens.
* - `POST /api/users/login` — public. Answers the user and their tokens;
* rate-limited per account.
* - `GET /api/users/me` — signed in. Answers the caller's own user.
* - `GET /api/users/me/media-token` — signed in. Answers a media token, for a
* session that predates them.
* - `POST /api/users/request-password-reset` — public. Emails a reset link to
* the account with that address, if any; always answers 200.
* - `POST /api/users/reset-password` — public, with a reset token. Sets a new
* password.
* - `POST /api/users/me/change-password` — signed in, with the current
* password. Sets a new one, ending every other session, and answers fresh
* tokens.
* - `PATCH /api/users/me` — signed in. Updates the caller's profile fields:
* description and email.
* - `PATCH /api/users/:userId/role` — permission `user:manage`. Sets another
* - `POST /api/users` — public, with a valid invite. Creates an account, sends
* the welcome email, and answers the user and their tokens.
* - `GET /api/users?q=` — public. Answers up to 8 users whose name starts
* with `q`, for @mention suggestions.
* - `GET /api/users/:key` — public. Answers a user's profile.
* - `GET /api/users/:key/dumps?page=&limit=&category=` — optional sign-in.
* Answers a page of their dumps, the private ones only to themselves,
* optionally in one category.
* - `GET /api/users/:key/dumps/:slug` — optional sign-in. Answers their dump
* whose slug is or was `slug`, with its current slug; a private one 404s
* but for its poster.
* - `GET /api/users/:key/upvoted?page=&limit=&category=` — optional sign-in.
* Answers a page of the dumps they upvoted, private ones only to
* themselves, optionally in one category.
* - `GET /api/users/:key/playlists?page=&limit=&category=` — optional
* sign-in. Answers a page of the playlists they made, the private ones only
* to themselves, optionally in one playlist category.
* - `GET /api/users/:key/playlists/:slug` — optional sign-in. Answers their
* playlist whose slug is or was `slug`, with its current slug and the dumps
* the caller may see; a private one 404s but for its owner.
* - `GET /api/users/:key/follow/users?page=&limit=` — public. Answers a page of
* the users they follow.
* - `GET /api/users/:key/follow/playlists?page=&limit=` — public. Answers a
* page of the public playlists they follow.
* - `GET /api/users/:key/invitees` — public. Answers the tree of users they
* invited.
* - `GET /api/users/:key/home?path=` — public. Redirects to a page of the
* instance a user from another instance comes from, their profile there
* unless `path` names another; a local user 404s.
* - `PATCH /api/users/:key/role` — permission `user:manage`. Sets another
* user's role.
* - `GET /api/users/search?q=` — public. Answers up to 8 users whose name
* starts with `q`, for @mention suggestions.
* - `GET /api/users/by-id/:userId` — public. Answers a user's profile by id.
* - `GET /api/users/:username/followed-users?page=&limit=` — public. Answers a
* page of the users they follow.
* - `GET /api/users/:username/followed-playlists?page=&limit=` — public.
* Answers a page of the public playlists they follow.
* - `GET /api/users/:username/playlists?page=&limit=` — optional sign-in.
* Answers a page of their playlists, the private ones only to themselves.
* - `GET /api/users/:username/invitees` — public. Answers the tree of users
* they invited.
* - `GET /api/users/:username` — public. Answers a user's profile.
* - `GET /api/users/:username/dumps?page=&limit=` — optional sign-in. Answers a
* page of their dumps, the private ones only to themselves.
* - `GET /api/users/:username/votes?page=&limit=` — optional sign-in. Answers a
* page of the dumps they upvoted, private ones only to themselves.
* - `PATCH /api/users/:key/username` — permission `user:manage`. Renames a
* user, the caller included. It is the only way a local name changes; a
* user from another instance is named there (400). The instances the user
* signed in to are told.
*
* @module
*/
@@ -47,28 +50,20 @@ import { type APIResponse, type PaginatedData } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts";
import {
type AuthResponse,
type AuthTokens,
type Invitee,
isRole,
type MediaTokenResponse,
type PublicUser,
type SelfUser,
type UserSuggestion,
} from "../../shared/user.ts";
import type { Dump } from "../../shared/dump.ts";
import type { Playlist, PlaylistWithDumps } from "../../shared/playlist.ts";
import {
fieldOf,
isLoginUserRequest,
isRegisterUserRequest,
isUpdateUserRequest,
validateRegisterUserRequest,
} from "../validators/user.ts";
import type { Playlist } from "../../shared/playlist.ts";
import { toPublicUser, toSelfUser, type User } from "../model/user.ts";
import { createJWT, createMediaToken } from "../lib/jwt.ts";
import { hashPassword, verifyPassword } from "../lib/password.ts";
import { type RateLimit, takeToken } from "../lib/rate-limit.ts";
import { toPublicUser, toSelfUser } from "../model/user.ts";
import { issueTokens } from "../lib/jwt.ts";
import {
type AuthContext,
authMiddleware,
@@ -77,35 +72,37 @@ import {
import { parseOptionalAuth } from "../lib/auth.ts";
import { parsePagination } from "../lib/pagination.ts";
import {
assertLocalUser,
createUser,
getInviteTree,
getUserById,
getUserByUsername,
renameUser,
resolveUserKey,
searchUsers,
sendWelcomeEmail,
updateUser,
} from "../services/user-service.ts";
import { claimInvite, releaseInvite } from "../services/invite-service.ts";
import { notifyClients } from "../services/identity-provider-service.ts";
import {
requestPasswordReset,
resetPassword,
} from "../services/password-reset-service.ts";
import { isEmailEnabled, sendEmail } from "../services/email-service.ts";
import { FROM_EMAIL, OG_SITE_NAME, WELCOME_EMAIL_BODY } from "../config.ts";
import { VALIDATION } from "../../shared/validation.ts";
import { marked } from "marked";
import { broadcastUserUpdated } from "../services/ws-service.ts";
broadcastUserUpdated,
updateClientUsername,
} from "../services/ws-service.ts";
import {
getDumpByPath,
getDumpsByUser,
getVotedDumpsByUser,
} from "../services/dump-service.ts";
import { listPlaylistsByUser } from "../services/playlist-service.ts";
import {
getPlaylist,
getPlaylistByPath,
listPlaylistsByUser,
} from "../services/playlist-service.ts";
import {
getFollowedPlaylistsByUser,
getFollowedUsersByUser,
} from "../services/follow-service.ts";
// Hashed once, at startup, for the sign-ins of usernames that don't exist.
const UNKNOWN_USER_HASH: Promise<string> = hashPassword(crypto.randomUUID());
import { resolveCategoryFilter } from "../services/category-service.ts";
import { homePageUrl } from "../services/federation-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
@@ -113,25 +110,7 @@ const UNKNOWN_USER_HASH: Promise<string> = hashPassword(crypto.randomUUID());
*/
const router: Router = new Router({ prefix: "/api/users" });
/** Allows a burst of 10 login attempts per account, then one every 30s. */
const LOGIN_LIMIT: RateLimit = { capacity: 10, refill: 1 / 30 };
/**
* Returns a new session token and media token for `user`, as sign-up, sign-in
* and a password change answer them.
*/
async function issueTokens(user: User): Promise<AuthTokens> {
return {
token: await createJWT({
userId: user.id,
username: user.username,
role: user.role,
}),
mediaToken: await createMediaToken(user.id),
};
}
router.post("/register", async (ctx) => {
router.post("/", async (ctx) => {
const body: unknown = await ctx.request.body.json();
if (!isRegisterUserRequest(body)) {
@@ -153,19 +132,7 @@ router.post("/register", async (ctx) => {
throw err;
}
// Not awaited: a slow or failing SMTP server mustn't hold up the sign-up.
if (isEmailEnabled()) {
const emailMarkdown = WELCOME_EMAIL_BODY
.replaceAll("{{username}}", user.username)
.replaceAll("{{site_name}}", OG_SITE_NAME);
sendEmail({
from: FROM_EMAIL,
to: user.email,
subject: `Welcome to ${OG_SITE_NAME}`,
text: emailMarkdown,
html: await marked(emailMarkdown),
}).catch((err) => console.error("[register] welcome email failed:", err));
}
sendWelcomeEmail(user);
const responseBody: APIResponse<AuthResponse> = {
success: true,
@@ -175,172 +142,7 @@ router.post("/register", async (ctx) => {
ctx.response.body = responseBody;
});
router.post("/login", async (ctx) => {
const body: unknown = await ctx.request.body.json();
if (!isLoginUserRequest(body)) {
throw new APIException(400, "Invalid request");
}
// Per account, so guessing one user's password is slow however many
// connections it comes from.
if (!takeToken(`login:${body.username.toLowerCase()}`, LOGIN_LIMIT)) {
throw new APIException(
429,
"Too many login attempts — try again in a minute",
);
}
let user;
try {
user = getUserByUsername(body.username);
} catch {
user = undefined;
}
// An unknown username gets the same answer as a wrong password, after the
// same work, so neither the message nor the timing tells which exist.
const valid = await verifyPassword(
body.password,
user?.passwordHash ?? await UNKNOWN_USER_HASH,
);
if (!user || !valid) {
throw new APIException(401, "Invalid username or password");
}
const responseBody: APIResponse<AuthResponse> = {
success: true,
data: { ...await issueTokens(user), user: toSelfUser(user) },
};
ctx.response.body = responseBody;
});
router.get("/me", authMiddleware, (ctx: AuthContext) => {
const responseBody: APIResponse<SelfUser> = {
success: true,
data: toSelfUser(getUserById(ctx.state.user.userId)),
};
ctx.response.body = responseBody;
});
router.get("/me/media-token", authMiddleware, async (ctx: AuthContext) => {
const responseBody: APIResponse<MediaTokenResponse> = {
success: true,
data: { mediaToken: await createMediaToken(ctx.state.user.userId) },
};
ctx.response.body = responseBody;
});
// Answers the same whatever happens, so it can't tell which addresses have an
// account.
router.post("/request-password-reset", async (ctx) => {
const body: unknown = await ctx.request.body.json();
const email = fieldOf(body, "email");
const trimmed = typeof email === "string" ? email.trim() : "";
if (trimmed) {
await requestPasswordReset(trimmed).catch((err) =>
console.error("[request-password-reset]", err)
);
}
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
});
router.post("/reset-password", async (ctx) => {
const body: unknown = await ctx.request.body.json();
const token = fieldOf(body, "token");
const newPassword = fieldOf(body, "newPassword");
if (typeof token !== "string" || typeof newPassword !== "string") {
throw new APIException(400, "Invalid request");
}
await resetPassword(token, newPassword);
const responseBody: APIResponse<null> = { success: true, data: null };
ctx.response.body = responseBody;
});
router.post("/me/change-password", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
const currentPassword = fieldOf(body, "currentPassword");
const newPassword = fieldOf(body, "newPassword");
if (typeof currentPassword !== "string" || typeof newPassword !== "string") {
throw new APIException(400, "Invalid request");
}
if (
newPassword.length < VALIDATION.PASSWORD_MIN ||
newPassword.length > VALIDATION.PASSWORD_MAX
) {
throw new APIException(
400,
`Password must be ${VALIDATION.PASSWORD_MIN}–${VALIDATION.PASSWORD_MAX} characters`,
);
}
const user = getUserById(ctx.state.user.userId);
const valid = await verifyPassword(currentPassword, user.passwordHash);
if (!valid) {
throw new APIException(403, "Current password is incorrect");
}
// Changing the password revokes every token issued before it, this
// session's included, so hand back fresh ones for the client to keep.
const updated = await updateUser(ctx.state.user.userId, {
password: newPassword,
});
const responseBody: APIResponse<AuthTokens> = {
success: true,
data: await issueTokens(updated),
};
ctx.response.body = responseBody;
});
router.patch("/me", authMiddleware, async (ctx: AuthContext) => {
const body: unknown = await ctx.request.body.json();
if (!isUpdateUserRequest(body)) {
throw new APIException(400, "Invalid request");
}
// Only the profile fields: a stolen token alone must not change the role,
// the password (which needs the current one) or the username (which is
// fixed).
const updated = await updateUser(ctx.state.user.userId, {
description: body.description,
email: body.email,
});
const publicUser = toPublicUser(updated);
broadcastUserUpdated(publicUser);
const responseBody: APIResponse<PublicUser> = {
success: true,
data: publicUser,
};
ctx.response.body = responseBody;
});
router.patch(
"/:userId/role",
authMiddleware,
requirePermission("user:manage"),
async (ctx: AuthContext<"/:userId/role">) => {
const body: unknown = await ctx.request.body.json();
const role = fieldOf(body, "role");
if (!isRole(role)) {
throw new APIException(400, "Invalid role");
}
// So that an admin can't demote themselves by mistake and leave no admin.
if (ctx.params.userId === ctx.state.user.userId) {
throw new APIException(400, "Cannot change your own role");
}
const updated = await updateUser(ctx.params.userId, { role });
const publicUser = toPublicUser(updated);
broadcastUserUpdated(publicUser);
const responseBody: APIResponse<PublicUser> = {
success: true,
data: publicUser,
};
ctx.response.body = responseBody;
},
);
router.get("/search", (ctx) => {
router.get("/", (ctx) => {
const q = (ctx.request.url.searchParams.get("q") ?? "").trim();
const responseBody: APIResponse<UserSuggestion[]> = {
success: true,
@@ -349,16 +151,88 @@ router.get("/search", (ctx) => {
ctx.response.body = responseBody;
});
router.get("/by-id/:userId", (ctx) => {
router.get("/:key", (ctx) => {
const responseBody: APIResponse<PublicUser> = {
success: true,
data: toPublicUser(getUserById(ctx.params.userId)),
data: toPublicUser(resolveUserKey(ctx.params.key)),
};
ctx.response.body = responseBody;
});
router.get("/:username/followed-users", (ctx) => {
const user = getUserByUsername(ctx.params.username);
router.get("/:key/dumps", async (ctx) => {
const user = resolveUserKey(ctx.params.key);
const requestingUserId = await parseOptionalAuth(ctx);
const params = ctx.request.url.searchParams;
const { page, limit } = parsePagination(params);
const categoryId = resolveCategoryFilter("dump", params);
const includePrivate = requestingUserId === user.id;
const { items, hasMore } = categoryId === "missing"
? { items: [], hasMore: false }
: getDumpsByUser(user.id, page, limit, includePrivate, categoryId);
const responseBody: APIResponse<PaginatedData<Dump>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
router.get("/:key/dumps/:slug", async (ctx) => {
const user = resolveUserKey(ctx.params.key);
const requestingUserId = await parseOptionalAuth(ctx) ?? undefined;
const responseBody: APIResponse<Dump> = {
success: true,
data: getDumpByPath(user.id, ctx.params.slug, requestingUserId),
};
ctx.response.body = responseBody;
});
router.get("/:key/upvoted", async (ctx) => {
const user = resolveUserKey(ctx.params.key);
const requestingUserId = await parseOptionalAuth(ctx);
const params = ctx.request.url.searchParams;
const { page, limit } = parsePagination(params);
const categoryId = resolveCategoryFilter("dump", params);
const { items, hasMore } = categoryId === "missing"
? { items: [], hasMore: false }
: getVotedDumpsByUser(user.id, page, limit, requestingUserId, categoryId);
const responseBody: APIResponse<PaginatedData<Dump>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
router.get("/:key/playlists", async (ctx) => {
const user = resolveUserKey(ctx.params.key);
const requestingUserId = await parseOptionalAuth(ctx);
const params = ctx.request.url.searchParams;
const { page, limit } = parsePagination(params);
const categoryId = resolveCategoryFilter("playlist", params);
const { items, hasMore } = categoryId === "missing"
? { items: [], hasMore: false }
: listPlaylistsByUser(user.id, requestingUserId, page, limit, categoryId);
const responseBody: APIResponse<PaginatedData<Playlist>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
router.get("/:key/playlists/:slug", async (ctx) => {
const user = resolveUserKey(ctx.params.key);
const requestingUserId = await parseOptionalAuth(ctx);
const responseBody: APIResponse<PlaylistWithDumps> = {
success: true,
data: getPlaylist(
getPlaylistByPath(user.id, ctx.params.slug),
requestingUserId,
),
};
ctx.response.body = responseBody;
});
router.get("/:key/follow/users", (ctx) => {
const user = resolveUserKey(ctx.params.key);
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = getFollowedUsersByUser(user.id, page, limit);
const responseBody: APIResponse<PaginatedData<PublicUser>> = {
@@ -371,8 +245,8 @@ router.get("/:username/followed-users", (ctx) => {
ctx.response.body = responseBody;
});
router.get("/:username/followed-playlists", (ctx) => {
const user = getUserByUsername(ctx.params.username);
router.get("/:key/follow/playlists", (ctx) => {
const user = resolveUserKey(ctx.params.key);
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = getFollowedPlaylistsByUser(user.id, page, limit);
const responseBody: APIResponse<PaginatedData<Playlist>> = {
@@ -382,25 +256,8 @@ router.get("/:username/followed-playlists", (ctx) => {
ctx.response.body = responseBody;
});
router.get("/:username/playlists", async (ctx) => {
const user = getUserByUsername(ctx.params.username);
const requestingUserId = await parseOptionalAuth(ctx);
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = listPlaylistsByUser(
user.id,
requestingUserId,
page,
limit,
);
const responseBody: APIResponse<PaginatedData<Playlist>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
router.get("/:username/invitees", (ctx) => {
const user = getUserByUsername(ctx.params.username);
router.get("/:key/invitees", (ctx) => {
const user = resolveUserKey(ctx.params.key);
const responseBody: APIResponse<Invitee[]> = {
success: true,
data: getInviteTree(user.id),
@@ -408,47 +265,61 @@ router.get("/:username/invitees", (ctx) => {
ctx.response.body = responseBody;
});
router.get("/:username", (ctx) => {
router.get("/:key/home", async (ctx) => {
const path = ctx.request.url.searchParams.get("path") ?? undefined;
ctx.response.redirect(
await homePageUrl(resolveUserKey(ctx.params.key), path),
);
});
router.patch(
"/:key/role",
authMiddleware,
requirePermission("user:manage"),
async (ctx: AuthContext<"/:key/role">) => {
const body: unknown = await ctx.request.body.json();
const role = fieldOf(body, "role");
if (!isRole(role)) {
throw new APIException(400, "Invalid role");
}
const user = resolveUserKey(ctx.params.key);
// So that an admin can't demote themselves by mistake and leave no admin.
if (user.id === ctx.state.user.userId) {
throw new APIException(400, "Cannot change your own role");
}
const updated = await updateUser(user.id, { role });
const publicUser = toPublicUser(updated);
broadcastUserUpdated(publicUser);
const responseBody: APIResponse<PublicUser> = {
success: true,
data: toPublicUser(getUserByUsername(ctx.params.username)),
data: publicUser,
};
ctx.response.body = responseBody;
});
},
);
router.get("/:username/dumps", async (ctx) => {
const user = getUserByUsername(ctx.params.username);
const requestingUserId = await parseOptionalAuth(ctx);
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const includePrivate = requestingUserId === user.id;
const { items, hasMore } = getDumpsByUser(
user.id,
page,
limit,
includePrivate,
);
const responseBody: APIResponse<PaginatedData<Dump>> = {
router.patch(
"/:key/username",
authMiddleware,
requirePermission("user:manage"),
async (ctx: AuthContext<"/:key/username">) => {
const body: unknown = await ctx.request.body.json();
const username = fieldOf(body, "username");
if (typeof username !== "string") {
throw new APIException(400, "Invalid request");
}
const user = resolveUserKey(ctx.params.key);
assertLocalUser(user);
const publicUser = toPublicUser(renameUser(user.id, username));
updateClientUsername(publicUser.id, publicUser.username);
broadcastUserUpdated(publicUser);
await notifyClients(publicUser.id, { type: "update" });
const responseBody: APIResponse<PublicUser> = {
success: true,
data: { items, hasMore },
data: publicUser,
};
ctx.response.body = responseBody;
});
router.get("/:username/votes", async (ctx) => {
const user = getUserByUsername(ctx.params.username);
const requestingUserId = await parseOptionalAuth(ctx);
const { page, limit } = parsePagination(ctx.request.url.searchParams);
const { items, hasMore } = getVotedDumpsByUser(
user.id,
page,
limit,
requestingUserId,
);
const responseBody: APIResponse<PaginatedData<Dump>> = {
success: true,
data: { items, hasMore },
};
ctx.response.body = responseBody;
});
},
);
export default router;

48
api/routes/waveforms.ts Normal file
View File

@@ -0,0 +1,48 @@
/**
* Serves an audio file dump's waveform, a sub-resource of its dump, for the
* player to seek on. The frontend builds the URL with `dumpWaveformUrl`.
* lib/ffmpeg.ts makes it from the upload on first request and keeps it, so a
* listener downloads a few hundred bytes of peaks rather than the file a
* second time to decode it.
*
* - `GET /api/dumps/:dumpId/waveform?v=&token=` — public; a private dump's
* waveform needs its poster's media `token`. Answers a `DumpWaveform`; 404
* when the dump isn't an audio file or its audio can't be read.
*
* @module
*/
import { Router } from "@oak/oak";
import { APIException } from "../lib/api-exception.ts";
import { DUMPS_DIR } from "../config.ts";
import { resolveMediaDump } from "../lib/dump-media.ts";
import {
audioWaveform,
ensureGenerated,
generatedPaths,
} from "../lib/ffmpeg.ts";
import { serveFile } from "../lib/serve-file.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/api/dumps" });
router.get("/:dumpId/waveform", async (ctx) => {
const { dump, cacheControl } = await resolveMediaDump(ctx, ctx.params.dumpId);
if (dump.kind !== "file" || !dump.fileMime?.startsWith("audio/")) {
throw new APIException(404, "No waveform for this dump");
}
const outPath = generatedPaths(dump.id).waveform;
const srcPath = `${DUMPS_DIR}/${dump.id}`;
if (
!await ensureGenerated(outPath, () => audioWaveform(srcPath, outPath)) ||
!await serveFile(ctx, outPath, { mime: "application/json", cacheControl })
) {
throw new APIException(404, "Could not generate waveform");
}
});
export default router;

36
api/routes/well-known.ts Normal file
View File

@@ -0,0 +1,36 @@
/**
* Serves what other gerbeur instances read about this one before federated
* sign-in, at the root paths OpenID Connect fixes, which are the one
* exception to the API's `/api/` rule. `remote/gerbeur-instance.ts` reads
* them on the other side.
*
* - `GET /.well-known/openid-configuration` — public. Answers the instance's
* metadata (`InstanceMetadata`): its endpoints, its name and the protocol's
* version. Bare JSON, not the API's envelope.
* - `GET /.well-known/jwks.json` — public. Answers the public keys its tokens
* are signed with. Bare JSON.
*
* @module
*/
import { Router } from "@oak/oak";
import { getInstanceMetadata } from "../services/identity-provider-service.ts";
import { getPublicKeys } from "../services/instance-key-service.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const router: Router = new Router({ prefix: "/.well-known" });
router.get("/openid-configuration", (ctx) => {
ctx.response.headers.set("Cache-Control", "public, max-age=600");
ctx.response.body = getInstanceMetadata();
});
router.get("/jwks.json", async (ctx) => {
ctx.response.headers.set("Cache-Control", "public, max-age=600");
ctx.response.body = await getPublicKeys();
});
export default router;

33
api/routes/ws.ts
View File

@@ -27,6 +27,7 @@ import {
isUserChatOpen,
type RateLimit,
register,
send,
sendToUser,
setClientTyping,
takeToken,
@@ -34,6 +35,7 @@ import {
type WsClient,
} from "../services/ws-service.ts";
import {
chatMessageMentionsEveryone,
createChatMessage,
deleteChatMessage,
getChatReadState,
@@ -42,6 +44,7 @@ import {
} from "../services/chat-service.ts";
import {
getUnreadCount,
notifyEveryone,
notifyMentions,
} from "../services/notification-service.ts";
import { hasPermission } from "../../shared/permissions.ts";
@@ -57,10 +60,7 @@ import {
} from "../services/comment-like-service.ts";
import { getUserById } from "../services/user-service.ts";
import { APIException } from "../lib/api-exception.ts";
import type {
ClientToServerMessage,
ServerToClientMessage,
} from "../../shared/ws.ts";
import type { ClientToServerMessage } from "../../shared/ws.ts";
import type { User } from "../model/user.ts";
/**
@@ -123,7 +123,7 @@ router.get("/ws", async (ctx) => {
const myCommentLikes = user ? getUserCommentLikes(user.id) : [];
const unreadNotificationCount = user ? getUnreadCount(user.id) : 0;
const chatRead = user ? getChatReadState(user.id) : undefined;
socket.send(JSON.stringify({
send(socket, {
type: "welcome",
users: getOnlineUsers(),
myVotes,
@@ -131,7 +131,7 @@ router.get("/ws", async (ctx) => {
unreadNotificationCount,
chatReadAt: chatRead?.readAt,
unreadChatCount: chatRead?.unread ?? 0,
}));
});
} catch (err) {
console.error("[ws] welcome send failed:", err);
}
@@ -221,11 +221,6 @@ function handleMessage(client: WsClient, data: unknown): void {
}
}
/** Sends `data` unless the socket is closing, where `send` would throw. */
function send(socket: WebSocket, data: ServerToClientMessage): void {
if (socket.readyState === WebSocket.OPEN) socket.send(JSON.stringify(data));
}
/**
* Handles `vote_cast` and `vote_remove`. It acks the voter with `vote_ack` and
* broadcasts `votes_update`. A failure goes back as an `error` naming the dump
@@ -369,7 +364,8 @@ function optionalString(value: unknown, maxLength: number): string | undefined {
/**
* Handles `chat_send`. It stores the message, broadcasts it, and notifies the
* @mentioned users who aren't reading the chat.
* @mentioned users who aren't reading the chat, and everyone who isn't when
* the message is flagged `mentionsEveryone`.
*/
function handleChatSend(
client: WsClient,
@@ -404,6 +400,9 @@ function handleChatSend(
undefined,
isUserChatOpen,
);
if (message.mentionsEveryone) {
notifyEveryone(userId, message.id, isUserChatOpen);
}
} catch (err) {
sendChatError(
client,
@@ -430,7 +429,8 @@ function canModerateChat(client: WsClient): boolean {
/**
* Handles `chat_edit`. It stores the new body, broadcasts it, and notifies, in
* the author's name, the users the edit newly mentions.
* the author's name, the users the edit newly mentions, and everyone when the
* edit adds a `@everyone` the author may say.
*/
function handleChatEdit(client: WsClient, id: unknown, body: unknown): void {
const userId = chatWriter(client);
@@ -441,6 +441,10 @@ function handleChatEdit(client: WsClient, id: unknown, body: unknown): void {
}
try {
// Only an edit that adds `@everyone` notifies everyone: fixing a typo in
// an announcement mustn't reach those who had the chat open when it was
// sent, nor those who joined since.
const wasEveryone = chatMessageMentionsEveryone(id);
const message = updateChatMessage(
id,
body,
@@ -460,6 +464,9 @@ function handleChatEdit(client: WsClient, id: unknown, body: unknown): void {
undefined,
isUserChatOpen,
);
if (message.mentionsEveryone && !wasEveryone) {
notifyEveryone(message.userId, message.id, isUserChatOpen);
}
} catch (err) {
sendChatError(
client,

35
api/routes/youtube.ts
View File

@@ -1,35 +0,0 @@
/**
* Serves the titles and lengths of a YouTube playlist's entries, which the
* global player lists beside the embed. On any failure it shows the embed
* without a tracklist.
*
* - `GET /api/youtube/playlist?list=` — public. Answers the playlist's
* tracklist.
*
* @module
*/
import { Router } from "@oak/oak";
import type { APIResponse } from "../../shared/api.ts";
import { resolveYouTubePlaylist } from "../remote/youtube-playlist.ts";
import type { YouTubePlaylist } from "../../shared/player.ts";
/**
* Mounted by `api/main.ts` behind the error, Open Graph and CORS middleware.
* An `APIException` thrown by a handler becomes the JSON error response there.
*/
const youtubeRouter: Router = new Router();
// Public, like /api/bandcamp/tracks: it only reads a public page.
youtubeRouter.get("/api/youtube/playlist", async (ctx) => {
const listId = ctx.request.url.searchParams.get("list") ?? "";
const responseBody: APIResponse<YouTubePlaylist> = {
success: true,
data: await resolveYouTubePlaylist(listId),
};
ctx.response.headers.set("Cache-Control", "public, max-age=3600");
ctx.response.body = responseBody;
});
export default youtubeRouter;

6
api/services/attachment-service.ts
View File

@@ -9,6 +9,7 @@
import { db, transaction } from "../db/connection.ts";
import { APIException } from "../lib/api-exception.ts";
import { attachmentIdsIn } from "../../shared/attachment.ts";
/**
* Records attachment `id` once `POST /api/attachments` has written its file.
@@ -108,9 +109,6 @@ function resourceOwner(
}
}
const ATTACHMENT_RE =
/\/api\/attachments\/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/g;
/**
* Binds the attachments whose URLs appear in `text` to the resource that saved
* it, so the startup purge keeps them as long as that resource exists, and
@@ -124,7 +122,7 @@ export function linkAttachments(
resourceType: AttachmentResourceType,
resourceId: string,
): void {
const ids = [...text.matchAll(ATTACHMENT_RE)].map((m) => m[1]);
const ids = attachmentIdsIn(text);
if (ids.length === 0) return;
const update = db.prepare(
"UPDATE attachments SET resource_type = ?, resource_id = ? WHERE id = ? AND resource_id IS NULL",

46
api/services/backlink-service.ts
View File

@@ -13,7 +13,7 @@ import {
dumpRowToApi,
isDumpRow,
} from "../model/dump.ts";
import { UUID_RE } from "../lib/slugify.ts";
import { parsePagePath } from "../../shared/paths.ts";
import { canonicalizeUrl } from "../lib/canonical-url.ts";
import { parseRows } from "../model/rows.ts";
@@ -24,25 +24,34 @@ import { parseRows } from "../model/rows.ts";
const URL_RE = /https?:\/\/[^\s)\]<>"']+/g;
/**
* Matches the path of a dump's page, `/dumps/{slug-or-id}`, and ignores what
* follows it, so an `/edit` or `#comment-x` link counts too. The host isn't
* checked, since the site's public URL differs between setups.
* Returns the id of the dump whose page, or its editor, `pathname` is, by its
* owner's name and its slug, current or former; null when it is no dump's.
* The host isn't checked, since the site's public URL differs between setups;
* a query or a fragment (`#comment-x`) isn't in a pathname anyway.
*/
const DUMP_PATH_RE = /^\/dumps\/([^/?#]+)/;
/** Returns the id of the dump with id or slug `idOrSlug`, or null. */
function resolveDumpPathId(idOrSlug: string): string | null {
const row = UUID_RE.test(idOrSlug)
? db.prepare(`SELECT id FROM dumps WHERE id = ?;`).get(idOrSlug)
: db.prepare(`SELECT id FROM dumps WHERE slug = ?;`).get(idOrSlug);
return (row as { id: string } | undefined)?.id ?? null;
function resolveDumpPathId(pathname: string): string | null {
const page = parsePagePath(pathname);
if (page?.kind !== "dump") return null;
const row = db.prepare(
`SELECT d.id FROM dumps d
JOIN users u ON u.id = d.user_id
LEFT JOIN user_aliases ua ON ua.user_id = u.id
WHERE (lower(u.username) = lower(:name) OR lower(ua.username) = lower(:name))
AND (d.slug = :slug OR d.id IN (SELECT dump_id FROM dump_slug_aliases
WHERE user_id = u.id AND slug = :slug))
LIMIT 1;`,
).get({ name: page.username, slug: page.slug }) as
| { id: string }
| undefined;
return row?.id ?? null;
}
/**
* Replaces the backlinks from one text, a dump's note (`dump_comment`) or one
* of its comments (`comment`), with the links its current `text` holds. A link
* counts when it is a dump's page on this site, or when it is the URL a dump
* was posted with, compared by `canonicalizeUrl` as reposts are. The dump and
* counts when it is a dump's page on this site (`/@owner/dumps/<slug>`), or
* when it is the URL a dump was posted with, compared by `canonicalizeUrl` as
* reposts are. The dump and
* comment services call it on every save, and with an empty `text` when a
* comment is deleted.
*/
@@ -62,11 +71,10 @@ export function relinkBacklinks(
} catch {
continue;
}
const pathMatch = DUMP_PATH_RE.exec(parsed.pathname);
if (pathMatch) {
const resolved = resolveDumpPathId(decodeURIComponent(pathMatch[1]));
if (resolved) targetIds.add(resolved);
} else {
const resolved = resolveDumpPathId(parsed.pathname);
if (resolved) {
targetIds.add(resolved);
} else if (parsePagePath(parsed.pathname)?.kind !== "dump") {
const canonical = canonicalizeUrl(match[0]);
if (canonical) externalUrls.add(canonical);
}

222
api/services/category-service.ts
View File

@@ -1,7 +1,10 @@
/**
* Stores the categories and which dumps belong to them. The category routes
* manage and list them, the dump service files each dump it writes under its
* categories, and every service that lists dumps fills in their `categoryIds`.
* Stores the categories and which dumps and playlists belong to them, a set
* of categories for each {@link CategoryKind}. The category routes manage and
* list them, the dump and playlist services file each dump or playlist they
* write under its categories, and every service that lists dumps or playlists
* fills in their `categoryIds` and filters them by category with
* {@link categoryJoin}.
*
* @module
*/
@@ -9,6 +12,7 @@
import { APIException } from "../lib/api-exception.ts";
import type {
Category,
CategoryKind,
CreateCategoryRequest,
UpdateCategoryRequest,
} from "../../shared/category.ts";
@@ -21,28 +25,30 @@ import {
isDumpRow,
} from "../model/dump.ts";
import { type Page, pageOf } from "../lib/pagination.ts";
import { isReservedSlug, isValidSlugFormat } from "../lib/reserved-slugs.ts";
import { isReservedSlug } from "../lib/reserved-slugs.ts";
import { validateSlug } from "../lib/slugify.ts";
import { VALIDATION } from "../../shared/validation.ts";
import { broadcastCategoriesChanged } from "./ws-service.ts";
import { parseRows } from "../model/rows.ts";
/**
* Throws a 400 unless `slug` is well-formed, short enough and not reserved,
* with a message for each case.
* Names, for each kind, the table that files things under its categories and
* the column there of the thing filed.
*/
function validateSlug(slug: string): void {
if (!isValidSlugFormat(slug)) {
throw new APIException(
400,
"Slug must be lowercase letters, numbers and hyphens",
);
}
if (slug.length > VALIDATION.CATEGORY_SLUG_MAX) {
throw new APIException(
400,
`Slug must be at most ${VALIDATION.CATEGORY_SLUG_MAX} characters`,
);
}
if (isReservedSlug(slug)) {
const LINKS: Record<CategoryKind, { table: string; column: string }> = {
dump: { table: "dump_categories", column: "dump_id" },
playlist: { table: "playlist_categories", column: "playlist_id" },
};
/**
* Throws a 400 unless `slug` is a valid slug for a category of `kind`: a slug
* within a category's length, and for a dump category none of the top-level
* paths other pages hold (reserved-slugs.ts), since its feed lives at the URL
* root. A playlist category's lives under `/playlists/`.
*/
function validateCategorySlug(slug: string, kind: CategoryKind): void {
validateSlug(slug, VALIDATION.CATEGORY_SLUG_MAX);
if (kind === "dump" && isReservedSlug(slug)) {
throw new APIException(400, `Slug "${slug}" is reserved`);
}
}
@@ -91,12 +97,15 @@ export function getCategoryById(id: string): Category {
}
/**
* Returns the category at `slug`, for `GET /api/categories/:slug`.
* Returns the category of `kind` at `slug`, for {@link listDumpsByCategory}
* and the list of every public playlist.
*
* @throws {APIException} 404 when there is no such category.
*/
export function getCategoryBySlug(slug: string): Category {
const row = db.prepare(`SELECT * FROM categories WHERE slug = ?;`).get(slug);
export function getCategoryBySlug(kind: CategoryKind, slug: string): Category {
const row = db.prepare(
`SELECT * FROM categories WHERE kind = ? AND slug = ?;`,
).get(kind, slug);
if (!row || !isCategoryRow(row)) {
throw new APIException(404, "Category not found");
}
@@ -104,45 +113,89 @@ export function getCategoryBySlug(slug: string): Category {
}
/**
* Returns the id of the category at `slug`, or undefined when there is none.
* The follows routes use it for their `?category=` feed filter.
* Returns the id of the category of `kind` at `slug`, or undefined when there
* is none.
*/
export function getCategoryIdBySlug(slug: string): string | undefined {
const row = db.prepare(`SELECT id FROM categories WHERE slug = ?;`).get(slug);
export function getCategoryIdBySlug(
kind: CategoryKind,
slug: string,
): string | undefined {
const row = db.prepare(
`SELECT id FROM categories WHERE kind = ? AND slug = ?;`,
).get(kind, slug);
return row && typeof (row as { id?: unknown }).id === "string"
? (row as { id: string }).id
: undefined;
}
/**
* Returns every category, by position then slug, for `GET /api/categories`.
* Returns the id of the category of `kind` whose slug the `name` parameter of
* `params`, `?category=` by default, gives, undefined without one, or
* "missing" when it names none, so that the list comes back empty rather than
* unfiltered. The users routes filter a profile's lists with it, and the
* search route its results.
*/
export function resolveCategoryFilter(
kind: CategoryKind,
params: URLSearchParams,
name = "category",
): string | "missing" | undefined {
const slug = params.get(name);
if (!slug) return undefined;
return getCategoryIdBySlug(kind, slug) ?? "missing";
}
/**
* Returns the join that keeps only the rows of `alias`, a dump or playlist as
* `kind` says, filed under `categoryId`, with its parameters, or nothing
* without one. A query puts the join after its `FROM` and the parameters
* before its own.
*/
export function categoryJoin(
kind: CategoryKind,
alias: string,
categoryId?: string,
): [string, string[]] {
if (!categoryId) return ["", []];
const { table, column } = LINKS[kind];
return [
`INNER JOIN ${table} cat ON cat.${column} = ${alias}.id AND cat.category_id = ?`,
[categoryId],
];
}
/**
* Returns every category, of both kinds, by kind, position then slug, for
* `GET /api/categories`.
*/
export function listCategories(): Category[] {
const rows = db.prepare(
`SELECT * FROM categories ORDER BY position ASC, slug ASC;`,
`SELECT * FROM categories ORDER BY kind ASC, position ASC, slug ASC;`,
).all();
return parseRows(rows, isCategoryRow, categoryRowToApi);
}
/**
* Creates a category, for `POST /api/categories`. The name is stored trimmed,
* and the position defaults to 0.
* and the position defaults to 0. Every client is told to fetch the
* categories again, as after the two writes below.
*
* @throws {APIException} 400 when the slug or name is invalid, 409 when
* either is taken.
* either is taken within its kind.
*/
export function createCategory(req: CreateCategoryRequest): Category {
validateSlug(req.slug);
validateCategorySlug(req.slug, req.kind);
const name = requireName(req.name);
const id = crypto.randomUUID();
const createdAt = new Date();
try {
db.prepare(
`INSERT INTO categories (id, slug, name, position, created_at)
VALUES (?, ?, ?, ?, ?);`,
`INSERT INTO categories (id, kind, slug, name, position, created_at)
VALUES (?, ?, ?, ?, ?, ?);`,
).run(
id,
req.kind,
req.slug,
name,
req.position ?? 0,
@@ -151,9 +204,11 @@ export function createCategory(req: CreateCategoryRequest): Category {
} catch (err) {
uniqueConflict(err, req.slug, name);
}
broadcastCategoriesChanged();
return {
id,
kind: req.kind,
slug: req.slug,
name,
position: req.position ?? 0,
@@ -166,7 +221,7 @@ export function createCategory(req: CreateCategoryRequest): Category {
* the category as it now is. Fields `req` leaves out keep their value.
*
* @throws {APIException} 404 when there is no such category, 400 when the new
* slug or name is invalid, 409 when either is taken.
* slug or name is invalid, 409 when either is taken within its kind.
*/
export function updateCategory(
id: string,
@@ -176,7 +231,7 @@ export function updateCategory(
const newSlug = req.slug ?? existing.slug;
if (req.slug !== undefined && req.slug !== existing.slug) {
validateSlug(req.slug);
validateCategorySlug(req.slug, existing.kind);
}
const newName = req.name !== undefined
? requireName(req.name)
@@ -197,6 +252,7 @@ export function updateCategory(
} catch (err) {
uniqueConflict(err, newSlug, newName);
}
broadcastCategoriesChanged();
return {
...existing,
@@ -208,8 +264,8 @@ export function updateCategory(
}
/**
* Deletes category `id`, for `DELETE /api/categories/:id`. Its dumps stay, and
* lose it from their `categoryIds`.
* Deletes category `id`, for `DELETE /api/categories/:id`. Its dumps or
* playlists stay, and lose it from their `categoryIds`.
*
* @throws {APIException} 404 when there is no such category.
*/
@@ -218,23 +274,29 @@ export function deleteCategory(id: string): void {
if (result.changes === 0) {
throw new APIException(404, "Category not found");
}
broadcastCategoriesChanged();
}
/**
* Checks that every id in `categoryIds` names a category. The dump service
* runs it before it writes a dump row or stores an uploaded file, so a bad id
* fails the request before anything is written.
* Checks that every id in `categoryIds` names a category of `kind`, so that a
* dump is never filed under a playlist category, nor the other way round. The
* dump and playlist services run it before they write a row or store an
* uploaded file, so a bad id fails the request before anything is written.
*
* @throws {APIException} 400 when an id names no category.
* @throws {APIException} 400 when an id names no category of `kind`.
*/
export function assertCategoriesExist(categoryIds: string[]): void {
export function assertCategoriesExist(
kind: CategoryKind,
categoryIds: string[],
): void {
const unique = [...new Set(categoryIds)];
if (unique.length > 0) {
const placeholders = unique.map(() => "?").join(", ");
const found = db.prepare(
`SELECT COUNT(*) as count FROM categories WHERE id IN (${placeholders});`,
).get(...unique) as { count: number } | undefined;
`SELECT COUNT(*) as count FROM categories
WHERE kind = ? AND id IN (${placeholders});`,
).get(kind, ...unique) as { count: number } | undefined;
if ((found?.count ?? 0) !== unique.length) {
throw new APIException(400, "One or more categories do not exist");
}
@@ -242,56 +304,64 @@ export function assertCategoriesExist(categoryIds: string[]): void {
}
/**
* Replaces the categories of dump `dumpId` with `categoryIds`, duplicates
* dropped. An empty list takes the dump out of every category. Called inside
* the dump service's transaction, it joins that one.
* Replaces the categories of the dump or playlist `id`, as `kind` says, with
* `categoryIds`, duplicates dropped. An empty list takes it out of every
* category. Called inside the dump or playlist service's transaction, it
* joins that one.
*
* @throws {APIException} 400 when an id names no category.
* @throws {APIException} 400 when an id names no category of `kind`.
*/
export function setDumpCategories(
dumpId: string,
export function setCategories(
kind: CategoryKind,
id: string,
categoryIds: string[],
): void {
assertCategoriesExist(categoryIds);
assertCategoriesExist(kind, categoryIds);
const unique = [...new Set(categoryIds)];
const { table, column } = LINKS[kind];
transaction(() => {
db.prepare(`DELETE FROM dump_categories WHERE dump_id = ?;`).run(dumpId);
db.prepare(`DELETE FROM ${table} WHERE ${column} = ?;`).run(id);
const insert = db.prepare(
`INSERT INTO dump_categories (dump_id, category_id) VALUES (?, ?);`,
`INSERT INTO ${table} (${column}, category_id) VALUES (?, ?);`,
);
for (const categoryId of unique) {
insert.run(dumpId, categoryId);
insert.run(id, categoryId);
}
});
}
/**
* Sets `categoryIds` on each of `dumps` in place, in one query. The dump row
* doesn't carry them, so every service that returns dumps calls this first.
* Sets `categoryIds` on each of `items`, dumps or playlists as `kind` says, in
* place, in one query. Their rows don't carry them, so every service that
* returns dumps or playlists calls this first.
*/
export function attachCategoryIds(dumps: Dump[]): void {
if (dumps.length === 0) return;
const ids = dumps.map((d) => d.id);
export function attachCategoryIds(
kind: CategoryKind,
items: { id: string; categoryIds?: string[] }[],
): void {
if (items.length === 0) return;
const { table, column } = LINKS[kind];
const ids = items.map((d) => d.id);
const placeholders = ids.map(() => "?").join(", ");
const rows = db.prepare(
`SELECT dump_id, category_id FROM dump_categories WHERE dump_id IN (${placeholders});`,
).all(...ids) as Array<{ dump_id: string; category_id: string }>;
`SELECT ${column} as id, category_id FROM ${table} WHERE ${column} IN (${placeholders});`,
).all(...ids) as Array<{ id: string; category_id: string }>;
const byDump = new Map<string, string[]>();
for (const { dump_id, category_id } of rows) {
const list = byDump.get(dump_id) ?? [];
const byItem = new Map<string, string[]>();
for (const { id, category_id } of rows) {
const list = byItem.get(id) ?? [];
list.push(category_id);
byDump.set(dump_id, list);
byItem.set(id, list);
}
for (const dump of dumps) {
dump.categoryIds = byDump.get(dump.id) ?? [];
for (const item of items) {
item.categoryIds = byItem.get(item.id) ?? [];
}
}
/**
* Returns a page of the dumps in the category at `slug`, newest first, for
* `GET /api/categories/:slug/dumps`. Private dumps are left out, except
* Returns a page of the dumps in the dump category at `slug`, newest first,
* for `GET /api/dumps?category=`. Private dumps are left out, except
* `requestingUserId`'s own.
*
* @throws {APIException} 404 when there is no such category.
@@ -302,7 +372,7 @@ export function listDumpsByCategory(
limit: number,
requestingUserId?: string,
): Page<Dump> {
const category = getCategoryBySlug(slug);
const category = getCategoryBySlug("dump", slug);
const offset = (page - 1) * limit;
const privacyClause = requestingUserId
@@ -310,15 +380,15 @@ export function listDumpsByCategory(
: "d.is_private = 0";
const privacyParams = requestingUserId ? [requestingUserId] : [];
const [catJoin, catParams] = categoryJoin("dump", "d", category.id);
const rows = db.prepare(
`SELECT ${SELECT_COLS_ALIASED}
FROM dumps d
INNER JOIN dump_categories dc ON dc.dump_id = d.id
WHERE dc.category_id = ? AND ${privacyClause}
FROM dumps d ${catJoin}
WHERE ${privacyClause}
ORDER BY d.created_at DESC LIMIT ? OFFSET ?;`,
).all(category.id, ...privacyParams, limit + 1, offset);
).all(...catParams, ...privacyParams, limit + 1, offset);
const items = parseRows(rows, isDumpRow, dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}

53
api/services/chat-service.ts
View File

@@ -14,6 +14,9 @@ import { chatMessageRowToApi, isChatMessageRow } from "../model/chat.ts";
import { db } from "../db/connection.ts";
import { linkAttachments } from "./attachment-service.ts";
import { VALIDATION } from "../../shared/validation.ts";
import { mentionsEveryone } from "../../shared/mention.ts";
import { hasPermission } from "../../shared/permissions.ts";
import { isRole } from "../../shared/user.ts";
const DEFAULT_LIMIT = 50;
const MAX_LIMIT = 100;
@@ -28,7 +31,7 @@ const REPLY_SNIPPET_LEN = 300;
/** The columns of a `ChatMessageRow`, to select over {@link FROM_JOINS}. */
const SELECT_COLS =
`m.id, m.user_id, m.body, m.created_at, m.updated_at, m.reply_to_id,
u.username as author_username, u.avatar_mime as author_avatar_mime,
m.mentions_everyone, u.username as author_username, u.avatar_mime as author_avatar_mime,
ru.username as reply_to_author, substr(rm.body, 1, ${REPLY_SNIPPET_LEN}) as reply_to_body`;
/**
@@ -51,10 +54,27 @@ function fetchMessage(id: string): ChatMessage {
return chatMessageRowToApi(row);
}
/**
* Returns 1 when `body`, written by `authorId`, reaches everyone: it says
* `@everyone` and the author's role, read now, has `chat:everyone`. Else 0,
* for `chat_messages.mentions_everyone`.
*/
function everyoneFlag(authorId: string, body: string): number {
if (!mentionsEveryone(body)) return 0;
const row = db.prepare(`SELECT role FROM users WHERE id = ?;`).get(
authorId,
) as { role: unknown } | undefined;
return row && isRole(row.role) && hasPermission(row.role, "chat:everyone")
? 1
: 0;
}
/**
* Stores a message from `userId` on a `chat_send`, and claims the pasted images
* it links to. A `replyToId` naming no message is dropped, and the message is
* posted as no reply. The caller broadcasts the result.
* posted as no reply. The message `mentionsEveryone` when it says `@everyone`
* and `userId` may. The caller broadcasts the result, and notifies everyone
* when it is flagged.
*
* @throws {APIException} 400 when the body is blank or longer than
* `VALIDATION.CHAT_MESSAGE_MAX`.
@@ -84,8 +104,9 @@ export function createChatMessage(
const id = crypto.randomUUID();
const createdAt = new Date().toISOString();
db.prepare(
`INSERT INTO chat_messages (id, user_id, body, created_at, reply_to_id) VALUES (?, ?, ?, ?, ?);`,
).run(id, userId, trimmed, createdAt, replyTo);
`INSERT INTO chat_messages (id, user_id, body, created_at, reply_to_id, mentions_everyone)
VALUES (?, ?, ?, ?, ?, ?);`,
).run(id, userId, trimmed, createdAt, replyTo, everyoneFlag(userId, trimmed));
// Unclaimed attachments are deleted as orphans at startup.
linkAttachments(trimmed, "chat", id);
return fetchMessage(id);
@@ -102,10 +123,24 @@ function getMessageOwner(id: string): string {
return row.user_id;
}
/**
* Checks whether message `id` is flagged `mentionsEveryone`, false when there
* is no such message. The WebSocket route reads it before an edit, to notify
* everyone only when the edit adds `@everyone`.
*/
export function chatMessageMentionsEveryone(id: string): boolean {
const row = db.prepare(
`SELECT mentions_everyone FROM chat_messages WHERE id = ?;`,
).get(id) as { mentions_everyone: number } | undefined;
return row?.mentions_everyone === 1;
}
/**
* Replaces a message's body on a `chat_edit`, and claims any image the new body
* links to. `canModerate` lets a moderator edit anyone's message. The caller
* broadcasts the result.
* links to. `canModerate` lets a moderator edit anyone's message. The flag
* `mentionsEveryone` is set again from the new body and the author's role, not
* the editor's: a moderator's edit can't make a user's message reach everyone.
* The caller broadcasts the result.
*
* @throws {APIException} 404 when there is no such message, 403 when the user
* is neither its author nor a moderator, 400 when the body is blank or too
@@ -131,8 +166,10 @@ export function updateChatMessage(
`Message exceeds ${VALIDATION.CHAT_MESSAGE_MAX} characters`,
);
}
db.prepare(`UPDATE chat_messages SET body = ?, updated_at = ? WHERE id = ?;`)
.run(trimmed, new Date().toISOString(), id);
db.prepare(
`UPDATE chat_messages SET body = ?, updated_at = ?, mentions_everyone = ?
WHERE id = ?;`,
).run(trimmed, new Date().toISOString(), everyoneFlag(ownerId, trimmed), id);
linkAttachments(trimmed, "chat", id);
return fetchMessage(id);
}

72
api/services/comment-service.ts
View File

@@ -114,12 +114,44 @@ export function createComment(
return comment;
}
/**
* The dump a comment is on, as {@link updateComment} and {@link deleteComment}
* return it: what the route needs to choose the broadcast's audience.
*/
export interface CommentDump {
id: string;
isPrivate: boolean;
/** The dump's poster, the only one who sees a private dump's comments. */
userId: string;
}
/**
* Returns the dump comment `commentId` is on.
*
* @throws {APIException} 404 when there is no such comment.
*/
function commentDump(commentId: string): CommentDump {
const row = db.prepare(
`SELECT d.id, d.is_private, d.user_id FROM comments c JOIN dumps d ON c.dump_id = d.id WHERE c.id = ?;`,
).get(commentId) as
| { id: string; is_private: number; user_id: string }
| undefined;
if (!row) {
throw new APIException(404, "Comment not found");
}
return {
id: row.id,
isPrivate: Boolean(row.is_private),
userId: row.user_id,
};
}
/**
* Replaces a comment's body, for `PATCH /api/comments/:commentId`.
* `canModerate` lets someone other than the author do it. It notifies, in the
* author's name, the users it mentions for the first time, unless the dump is
* private, and relinks the attachments and backlinks. The route broadcasts the
* comment unless `isPrivate`.
* comment to those who may see its `dump`.
*
* @throws {APIException} 404 when there is no such comment, 400 when it is
* deleted, 403 when the user is neither its author nor a moderator.
@@ -129,13 +161,8 @@ export function updateComment(
body: string,
requestingUserId: string,
canModerate: boolean,
): { comment: Comment; dumpId: string; isPrivate: boolean } {
const row = db.prepare(
`SELECT c.dump_id, d.is_private FROM comments c JOIN dumps d ON c.dump_id = d.id WHERE c.id = ?;`,
).get(commentId) as { dump_id: string; is_private: number } | undefined;
if (!row) {
throw new APIException(404, "Comment not found");
}
): { comment: Comment; dump: CommentDump } {
const dump = commentDump(commentId);
const existing = fetchComment(commentId);
if (existing.deleted) {
throw new APIException(400, "Cannot edit a deleted comment");
@@ -149,9 +176,9 @@ export function updateComment(
now,
commentId,
);
if (!row.is_private) {
if (!dump.isPrivate) {
const dumpRow = db.prepare(`SELECT title FROM dumps WHERE id = ?;`).get(
row.dump_id,
dump.id,
) as { title: string } | undefined;
// A moderator's edit keeps the author's words, so the author mentions.
notifyMentions(
@@ -160,23 +187,19 @@ export function updateComment(
"comment",
commentId,
dumpRow?.title ?? "",
row.dump_id,
dump.id,
);
}
linkAttachments(body, "comment", commentId);
relinkBacklinks("comment", commentId, row.dump_id, body);
return {
comment: fetchComment(commentId),
dumpId: row.dump_id,
isPrivate: Boolean(row.is_private),
};
relinkBacklinks("comment", commentId, dump.id, body);
return { comment: fetchComment(commentId), dump };
}
/**
* Deletes a comment, for `DELETE /api/comments/:commentId`. `canModerate` lets
* someone other than the author do it. The body is emptied and the backlinks
* dropped, but the row stays so its replies keep their place. The route
* broadcasts the deletion unless `isPrivate`.
* broadcasts the deletion to those who may see the returned dump.
*
* @throws {APIException} 404 when there is no such comment, 403 when the user
* is neither its author nor a moderator.
@@ -185,13 +208,8 @@ export function deleteComment(
commentId: string,
requestingUserId: string,
canModerate: boolean,
): { dumpId: string; isPrivate: boolean } {
const row = db.prepare(
`SELECT c.dump_id, d.is_private FROM comments c JOIN dumps d ON c.dump_id = d.id WHERE c.id = ?;`,
).get(commentId) as { dump_id: string; is_private: number } | undefined;
if (!row) {
throw new APIException(404, "Comment not found");
}
): CommentDump {
const dump = commentDump(commentId);
const comment = fetchComment(commentId);
if (comment.userId !== requestingUserId && !canModerate) {
throw new APIException(403, "Not authorized to delete this comment");
@@ -199,6 +217,6 @@ export function deleteComment(
db.prepare(`UPDATE comments SET deleted = 1, body = '' WHERE id = ?;`).run(
commentId,
);
relinkBacklinks("comment", commentId, row.dump_id, "");
return { dumpId: row.dump_id, isPrivate: Boolean(row.is_private) };
relinkBacklinks("comment", commentId, dump.id, "");
return dump;
}

93
api/services/community-service.ts Normal file
View File

@@ -0,0 +1,93 @@
/**
* Gathers what the whole site did over a period, for the `/community` page
* and the digests' shared sections. The digests ask {@link communityStatsOf}
* for each reader's own period. The page reads the figures of the week and
* the month ending at the last refresh, which {@link communityJob} redoes
* every {@link COMMUNITY_REFRESH_MINUTES}, so a visit runs no query.
*
* @module
*/
import { db } from "../db/connection.ts";
import { COMMUNITY_REFRESH_MINUTES } from "../config.ts";
import type { CommunityStats } from "../../shared/community.ts";
import {
DIGEST_CADENCES,
type DigestCadence,
PERIOD_MS,
} from "../../shared/settings.ts";
import type { Job } from "../lib/scheduler.ts";
import { listTopDumps } from "./dump-service.ts";
import { listNewPublicPlaylists } from "./playlist-service.ts";
import { listNewUsers } from "./user-service.ts";
/** How many top dumps a period keeps, as many as the monthly recap shows. */
const TOP_MAX = 10;
/** Returns how many rows of `table` were created from `from` to `to`. */
function countOf(
table: string,
from: string,
to: string,
where = "1",
): number {
return (db.prepare(
`SELECT COUNT(*) AS n FROM ${table}
WHERE ${where} AND created_at >= ? AND created_at < ?;`,
).get(from, to) as { n: number }).n;
}
/**
* Returns what the site did from `from` to `to`: its most upvoted public
* dumps, who joined, the new public playlists, the chat's messages and the
* totals.
*/
export function communityStatsOf(from: Date, to: Date): CommunityStats {
const [f, t] = [from.toISOString(), to.toISOString()];
return {
from,
to,
top: listTopDumps(from, to, TOP_MAX),
members: listNewUsers(from, to),
playlists: listNewPublicPlaylists(from, to),
chatMessages: countOf("chat_messages", f, t),
totals: {
dumps: countOf("dumps", f, t, "is_private = 0"),
upvotes: countOf("votes", f, t),
comments: countOf("comments", f, t, "deleted = 0"),
},
};
}
/** The figures {@link getCommunityStats} serves, by period. */
const latest = new Map<DigestCadence, CommunityStats>();
/** Recomputes the figures of each period, ending at `now`. */
export function refreshCommunityStats(now: Date = new Date()): void {
for (const period of DIGEST_CADENCES) {
latest.set(
period,
communityStatsOf(new Date(now.getTime() - PERIOD_MS[period]), now),
);
}
}
/**
* Returns the figures of `period` as of the last refresh, for
* `GET /api/community`, refreshing them first if there are none yet.
*/
export function getCommunityStats(period: DigestCadence): CommunityStats {
if (!latest.has(period)) refreshCommunityStats();
return latest.get(period)!;
}
/**
* Refreshes the figures at startup, then every
* {@link COMMUNITY_REFRESH_MINUTES}. `main.ts` starts it.
*/
export const communityJob: Job = {
name: "community",
everyMs: COMMUNITY_REFRESH_MINUTES * 60_000,
firstAfterMs: 0,
run: () => refreshCommunityStats(),
};

536
api/services/digest-service.ts Normal file
View File

@@ -0,0 +1,536 @@
/**
* Sends each user their weekly digest and monthly recap, unless they turned
* it off. `main.ts` starts {@link digestJob}, which every quarter of an hour
* sends the digests that are due; `digest_schedule` remembers, for
* each user and cadence, the start of the period their next one covers.
*
* A period is 7 days or 30, rolling from when the user's schedule began, so
* the digests spread over the week instead of leaving together. A digest
* covers the period just ended, or the last full period when the server was
* down longer than one; a digest with nothing to show isn't sent, and either
* way the next period begins. The top dumps and the community section come
* from `community-service.ts`, which the `/community` page reads too.
*
* @module
*/
import { db, transaction } from "../db/connection.ts";
import { FRONTEND_URL, PUBLIC_URL } from "../config.ts";
import {
DEFAULT_SETTINGS,
DIGEST_CADENCES,
DIGEST_SETTING,
type DigestCadence,
PERIOD_MS,
} from "../../shared/settings.ts";
import type { CommunityStats } from "../../shared/community.ts";
import { type Dump, dumpFileVersion } from "../../shared/dump.ts";
import { siteHost } from "../../shared/url.ts";
import { markdownToPlainText } from "../../shared/markdown.ts";
import {
DUMP_SELECT_COLUMNS_ALIASED,
dumpRowToApi,
isDumpRow,
} from "../model/dump.ts";
import { parseRows } from "../model/rows.ts";
import type { Job } from "../lib/scheduler.ts";
import { SETTINGS_COLUMNS } from "../model/settings.ts";
import type {
CommentLikedData,
DumpCommentedData,
DumpRef,
NotificationData,
NotificationType,
PlaylistFollowedData,
PlaylistRef,
UserFollowedData,
UserMentionedData,
} from "../../shared/notification.ts";
import {
communityPath,
dumpPath,
feedPath,
playlistPath,
} from "../../shared/paths.ts";
import {
type ActivityKind,
type DigestContent,
type DigestDump,
type DigestHighlight,
renderDigestEmail,
} from "../mail/digest.ts";
import { lookOf } from "../mail/layout.ts";
import { createUnsubscribeToken } from "../lib/jwt.ts";
import { isEmailEnabled, sendEmail } from "./email-service.ts";
import { getSettings } from "./settings-service.ts";
import { withCurrentRefs } from "./notification-service.ts";
import { communityStatsOf } from "./community-service.ts";
const TICK_MS = 15 * 60_000;
// What one digest shows at most, as the drafts settled it.
const HIGHLIGHTS_MAX = 5;
const NETWORK_MAX = 6;
const TOP_MAX: Record<DigestCadence, number> = { weekly: 5, monthly: 10 };
const MEMBERS_MAX = 10;
const PLAYLISTS_MAX = 5;
const QUOTE_MAX = 140;
const COUNTED: Partial<Record<NotificationType, ActivityKind>> = {
dump_upvoted: "upvotes",
user_followed: "followers",
dump_commented: "comments",
user_mentioned: "mentions",
comment_liked: "likes",
playlist_followed: "playlistFollows",
};
/** A period as the ISO strings dates are stored in, for comparing as text. */
interface Window {
from: string;
to: string;
}
/** Flattens Markdown into a short line of text for a quote. */
function quoteOf(markdown: string | undefined | null): string | undefined {
if (!markdown) return undefined;
const flat = markdownToPlainText(markdown, "");
if (!flat) return undefined;
return flat.length > QUOTE_MAX
? `${flat.slice(0, QUOTE_MAX - 1).trimEnd()}…`
: flat;
}
/**
* Returns the image a mail client can load for `dump`, which must be public:
* the thumbnails route for a custom thumbnail or an image or video upload,
* else the link preview's image, through the image proxy when it is plain
* http, which a mail client may refuse.
*/
function thumbnailOf(dump: Dump): string | null {
const isMedia = dump.kind === "file" &&
/^(image|video)\//.test(dump.fileMime ?? "");
if (dump.thumbnailMime || isMedia) {
return `${PUBLIC_URL}/api/dumps/${dump.id}/thumbnail?v=${
dumpFileVersion(dump)
}`;
}
const thumbnailUrl = dump.richContent?.thumbnailUrl;
if (typeof thumbnailUrl !== "string") return null;
if (thumbnailUrl.startsWith("https://")) return thumbnailUrl;
if (thumbnailUrl.startsWith("http://")) {
return `${PUBLIC_URL}/api/remote/image?url=${
encodeURIComponent(thumbnailUrl)
}`;
}
return null;
}
function toDigestDump(dump: Dump, readerId: string): DigestDump {
const domain = (dump.kind === "url" && dump.url && siteHost(dump.url)) ||
null;
return {
title: dump.title,
url: `${FRONTEND_URL}${dumpPath(dump)}`,
poster: dump.ownerUsername,
domain,
thumbnail: thumbnailOf(dump),
votes: dump.voteCount,
comments: dump.commentCount,
mine: dump.userId === readerId,
};
}
/**
* Returns what happened to the reader: their notifications of the period,
* counted by kind, and the few worth reading, mentions and comments first.
*/
function activityOf(
readerId: string,
w: Window,
): DigestContent["activity"] {
const byType = db.prepare(
`SELECT type, COUNT(*) AS n FROM notifications
WHERE user_id = ? AND created_at >= ? AND created_at < ?
GROUP BY type;`,
).all(readerId, w.from, w.to) as { type: NotificationType; n: number }[];
const counts: Partial<Record<ActivityKind, number>> = {};
let total = 0;
for (const { type, n } of byType) {
total += n;
const kind = COUNTED[type];
if (kind) counts[kind] = n;
}
const rows = db.prepare(
`SELECT type, data, created_at FROM notifications
WHERE user_id = ? AND created_at >= ? AND created_at < ?
AND type IN ('user_mentioned', 'dump_commented', 'playlist_followed',
'user_followed', 'comment_liked')
ORDER BY CASE type
WHEN 'user_mentioned' THEN 0 WHEN 'dump_commented' THEN 0
WHEN 'comment_liked' THEN 2 ELSE 1 END,
created_at DESC
LIMIT ?;`,
).all(readerId, w.from, w.to, HIGHLIGHTS_MAX) as {
type: NotificationType;
data: string;
created_at: string;
}[];
const highlights = rows.flatMap((row) => {
try {
const h = highlightOf(row.type, withCurrentRefs(JSON.parse(row.data)));
return h ? [{ ...h, at: new Date(row.created_at) }] : [];
} catch {
return [];
}
});
if (Object.keys(counts).length === 0 && highlights.length === 0) return null;
return { counts, highlights, total };
}
const commentBody = (id: string) =>
(db.prepare(`SELECT body FROM comments WHERE id = ? AND deleted = 0;`)
.get(id) as { body: string } | undefined)?.body;
/**
* Returns a highlight's dump, titled `title`, linked where the notification's
* joined ref says it is now, or unlinked once it is deleted.
*/
const dumpTarget = (ref: DumpRef, title: string) => ({
title,
url: ref.dumpSlug && ref.dumpOwnerUsername
? `${FRONTEND_URL}${
dumpPath({ ownerUsername: ref.dumpOwnerUsername, slug: ref.dumpSlug })
}`
: null,
});
/** Returns a highlight's playlist as {@link dumpTarget} does its dump. */
const playlistTarget = (ref: PlaylistRef, title: string) => ({
title,
url: ref.playlistSlug && ref.playlistOwnerUsername
? `${FRONTEND_URL}${
playlistPath({
ownerUsername: ref.playlistOwnerUsername,
slug: ref.playlistSlug,
})
}`
: null,
});
/**
* Returns the highlight a notification makes, from its data with the refs
* joined, or null for another kind.
*/
function highlightOf(
type: NotificationType,
data: NotificationData,
): Omit<DigestHighlight, "at"> | null {
// `type` says which member `data` is, which TypeScript can't follow from a
// separate argument, so each case names it.
switch (type) {
case "user_mentioned": {
const d = data as UserMentionedData;
switch (d.contextType) {
case "comment":
return {
kind: "mentionComment",
who: d.mentionerUsername,
target: dumpTarget(d, d.contextTitle),
quote: quoteOf(commentBody(d.contextId)),
};
case "dump":
return {
kind: "mentionDump",
who: d.mentionerUsername,
target: dumpTarget(d, d.contextTitle),
};
case "playlist":
return {
kind: "mentionPlaylist",
who: d.mentionerUsername,
target: playlistTarget(d, d.contextTitle),
};
case "chat":
return {
kind: d.everyone ? "mentionEveryone" : "mentionChat",
who: d.mentionerUsername,
quote: quoteOf(
(db.prepare(`SELECT body FROM chat_messages WHERE id = ?;`)
.get(d.contextId) as { body: string } | undefined)?.body,
),
};
}
return null;
}
case "dump_commented": {
const d = data as DumpCommentedData;
return {
kind: "comment",
who: d.commenterUsername,
target: dumpTarget(d, d.dumpTitle),
quote: quoteOf(commentBody(d.commentId)),
};
}
case "playlist_followed": {
const d = data as PlaylistFollowedData;
return {
kind: "playlistFollow",
who: d.followerUsername,
target: playlistTarget(d, d.playlistTitle),
};
}
case "user_followed":
return {
kind: "userFollow",
who: (data as UserFollowedData).followerUsername,
};
case "comment_liked": {
const d = data as CommentLikedData;
return {
kind: "commentLike",
who: d.likerUsername,
target: dumpTarget(d, d.dumpTitle),
quote: quoteOf(commentBody(d.commentId)),
};
}
}
return null;
}
/**
* Returns the public dumps of the period from the users and playlists the
* reader follows, the most upvoted first, without their own.
*/
function networkOf(readerId: string, w: Window): DigestContent["network"] {
const rows = db.prepare(
`SELECT ${DUMP_SELECT_COLUMNS_ALIASED}, NULL AS playlist_title, d.created_at AS at
FROM dumps d
WHERE d.is_private = 0 AND d.user_id != :reader
AND d.created_at >= :from AND d.created_at < :to
AND d.user_id IN (SELECT followed_user_id FROM follows
WHERE follower_id = :reader AND followed_user_id IS NOT NULL)
UNION ALL
SELECT ${DUMP_SELECT_COLUMNS_ALIASED}, p.title AS playlist_title, pd.added_at AS at
FROM playlist_dumps pd
JOIN playlists p ON p.id = pd.playlist_id
JOIN dumps d ON d.id = pd.dump_id
WHERE p.is_public = 1 AND d.is_private = 0 AND d.user_id != :reader
AND pd.added_at >= :from AND pd.added_at < :to
AND pd.playlist_id IN (SELECT followed_playlist_id FROM follows
WHERE follower_id = :reader AND followed_playlist_id IS NOT NULL);`,
).all({ reader: readerId, from: w.from, to: w.to });
const found = parseRows(rows, isDumpRow, (row) => ({
dump: dumpRowToApi(row),
playlistTitle: row.playlist_title as string | null,
at: row.at as string,
}));
// A dump both posted by someone followed and added to a followed playlist
// is listed once, as the post.
const byId = new Map<string, (typeof found)[number]>();
for (const item of found) {
const seen = byId.get(item.dump.id);
if (!seen || (seen.playlistTitle && !item.playlistTitle)) {
byId.set(item.dump.id, item);
}
}
const sorted = [...byId.values()].sort((a, b) =>
b.dump.voteCount - a.dump.voteCount || b.at.localeCompare(a.at)
);
if (sorted.length === 0) return null;
return {
dumps: sorted.slice(0, NETWORK_MAX).map(({ dump, playlistTitle }) => ({
...toDigestDump(dump, readerId),
...(playlistTitle ? { addedTo: playlistTitle } : {}),
})),
more: Math.max(0, sorted.length - NETWORK_MAX),
};
}
/**
* Returns the community section from `stats`, the site's figures for the
* period, leaving the reader out of who joined.
*/
function communityOf(
readerId: string,
stats: CommunityStats,
): DigestContent["community"] {
const members = stats.members.filter((m) => m.id !== readerId);
const { playlists, chatMessages, totals } = stats;
if (
members.length + playlists.length + chatMessages + totals.dumps +
totals.upvotes + totals.comments === 0
) return null;
return {
members: members.slice(0, MEMBERS_MAX).map((m) => ({
username: m.username,
invitedBy: m.invitedByUsername ?? null,
})),
membersMore: Math.max(0, members.length - MEMBERS_MAX),
playlists: playlists.slice(0, PLAYLISTS_MAX).map((p) => ({
title: p.title,
url: `${FRONTEND_URL}${playlistPath(p)}`,
owner: p.ownerUsername,
dumps: p.dumpCount ?? 0,
})),
playlistsMore: Math.max(0, playlists.length - PLAYLISTS_MAX),
chatMessages,
totals,
};
}
/**
* Returns the digest of user `readerId` for the period from `from` to `to`,
* or null when it would show nothing. The links are absolute; the
* unsubscribe one carries `token`, which turns off `cadence`.
*/
export function buildDigest(
readerId: string,
reader: string,
cadence: DigestCadence,
from: Date,
to: Date,
token: string,
): DigestContent | null {
const w = { from: from.toISOString(), to: to.toISOString() };
const weekly = cadence === "weekly";
const activity = weekly ? activityOf(readerId, w) : null;
const network = weekly ? networkOf(readerId, w) : null;
const stats = communityStatsOf(from, to);
const top = stats.top.slice(0, TOP_MAX[cadence]).map((d) =>
toDigestDump(d, readerId)
);
const community = communityOf(readerId, stats);
if (!activity && !network && top.length === 0 && !community) return null;
return {
cadence,
from,
to,
reader,
activity,
network,
top,
community,
links: {
notifications: `${FRONTEND_URL}/notifications`,
followed: `${FRONTEND_URL}${feedPath(undefined, "follow")}`,
chat: `${FRONTEND_URL}/?chat=open`,
community: `${FRONTEND_URL}${communityPath(cadence)}`,
settings: `${FRONTEND_URL}/settings`,
unsubscribe: `${FRONTEND_URL}/unsubscribe?token=${
encodeURIComponent(token)
}`,
},
};
}
/**
* Brings `digest_schedule` in line with the settings: a user who turned a
* cadence off loses its row, and one who receives it without a row, being
* new or having turned it back on, starts a period at `now`.
*/
export function syncDigestSchedules(now: Date): void {
transaction(() => {
for (const cadence of DIGEST_CADENCES) {
const setting = DIGEST_SETTING[cadence];
const column = SETTINGS_COLUMNS[setting];
const byDefault = DEFAULT_SETTINGS[setting] ? 1 : 0;
db.prepare(
`DELETE FROM digest_schedule WHERE cadence = ? AND user_id IN
(SELECT user_id FROM user_settings WHERE ${column} = 0);`,
).run(cadence);
db.prepare(
`INSERT INTO digest_schedule (user_id, cadence, window_start)
SELECT u.id, ?, ? FROM users u
LEFT JOIN user_settings s ON s.user_id = u.id
WHERE COALESCE(s.${column}, ${byDefault}) = 1
AND NOT EXISTS (SELECT 1 FROM digest_schedule ds
WHERE ds.user_id = u.id AND ds.cadence = ?);`,
).run(cadence, now.toISOString(), cadence);
}
});
}
/**
* Sends every digest due at `now`, one after the other, and returns how many
* went out. A digest that fails to send is logged and not retried: its
* period is over either way, and the next one starts.
*/
export async function sendDueDigests(now: Date = new Date()): Promise<number> {
if (!isEmailEnabled()) return 0;
syncDigestSchedules(now);
let sent = 0;
for (const cadence of DIGEST_CADENCES) {
const period = PERIOD_MS[cadence];
const due = db.prepare(
`SELECT ds.user_id, ds.window_start, u.username, u.email
FROM digest_schedule ds JOIN users u ON u.id = ds.user_id
WHERE ds.cadence = ? AND ds.window_start <= ?;`,
).all(cadence, new Date(now.getTime() - period).toISOString()) as {
user_id: string;
window_start: string;
username: string;
email: string;
}[];
for (const row of due) {
let from = new Date(row.window_start);
let to = new Date(from.getTime() + period);
// More than a period late (the server was down): cover the last full
// period rather than one long gone.
if (to.getTime() + period <= now.getTime()) {
from = new Date(now.getTime() - period);
to = now;
}
db.prepare(
`UPDATE digest_schedule SET window_start = ? WHERE user_id = ? AND cadence = ?;`,
).run(to.toISOString(), row.user_id, cadence);
// An address with no dot in its domain, such as the first admin's
// admin@localhost, can't receive mail.
if (!/@[^@]+\.[^@]+$/.test(row.email)) continue;
const token = await createUnsubscribeToken(row.user_id, cadence);
const content = buildDigest(
row.user_id,
row.username,
cadence,
from,
to,
token,
);
if (!content) continue;
const email = renderDigestEmail(
content,
lookOf(getSettings(row.user_id)),
);
try {
await sendEmail({
to: row.email,
...email,
headers: {
"List-Unsubscribe": `<${PUBLIC_URL}/api/unsubscribe?token=${
encodeURIComponent(token)
}>`,
"List-Unsubscribe-Post": "List-Unsubscribe=One-Click",
},
});
sent++;
} catch (err) {
console.error(`[digest] ${cadence} to ${row.username} failed:`, err);
}
}
}
return sent;
}
/**
* Sends the due digests a minute after the server starts, then every quarter
* of an hour. `main.ts` starts it.
*/
export const digestJob: Job = {
name: "digest",
everyMs: TICK_MS,
firstAfterMs: 60_000,
async run() {
const n = await sendDueDigests();
if (n > 0) console.log(`[digest] sent ${n}`);
},
};

417
api/services/dump-service.ts
View File

@@ -1,7 +1,8 @@
/**
* Posts, edits and deletes dumps, and reads them as each viewer may see them.
* `routes/dumps.ts` calls it, and so do the search, user, file and thumbnail
* routes to read dumps. It broadcasts the changes to public dumps over the
* routes to read dumps. A dump's slug is made from its title when it is
* posted, and changes only when an edit sends a new one (`slug-service.ts`). It broadcasts the changes to public dumps over the
* WebSocket.
*
* The functions that change a dump don't check who asks: the routes do first.
@@ -16,6 +17,7 @@ import type {
DumpUrlMatch,
UpdateDumpRequest,
} from "../../shared/dump.ts";
import { siteHost } from "../../shared/url.ts";
import { db, transaction } from "../db/connection.ts";
import {
DUMP_SELECT_COLUMNS as SELECT_COLS,
@@ -31,12 +33,19 @@ import {
} from "./rich-content-service.ts";
import { isValidHttpUrl } from "../remote/fetch.ts";
import { broadcastDumpChange } from "./ws-service.ts";
import {
broadcastHoldersDumps,
type PlaylistHolder,
playlistsHolding,
} from "./playlist-service.ts";
import {
notifyMentions,
notifyPlaylistFollowersNowPublic,
notifyUserFollowersNewDump,
} from "./notification-service.ts";
import { makeSlug, UUID_RE } from "../lib/slugify.ts";
import { changeSlug, newSlug, resolveSlug } from "./slug-service.ts";
import { canonicalizeUrl } from "../lib/canonical-url.ts";
import { removeGeneratedMedia } from "../lib/ffmpeg.ts";
import {
DUMP_ALLOWED_MIME_PREFIXES,
DUMP_ALLOWED_MIME_TYPES,
@@ -52,7 +61,8 @@ import { relinkBacklinks } from "./backlink-service.ts";
import {
assertCategoriesExist,
attachCategoryIds,
setDumpCategories,
categoryJoin,
setCategories,
} from "./category-service.ts";
import { LIKE, likePattern } from "../lib/like.ts";
import { parseRows } from "../model/rows.ts";
@@ -81,13 +91,34 @@ function assertFileDumpText(title?: string, comment?: string): void {
}
}
/** Returns a link's host name, the title it gets when its page has none. */
function titleFromUrl(url: string): string {
try {
return new URL(url).hostname.replace(/^www\./, "");
} catch {
return url;
}
/**
* Tells clients how a dump changed, for every write of this service:
* `broadcastDumpChange` for the dump itself, and, when it was deleted or
* changed visibility, the dumps of each playlist holding it, which moved for
* their viewers. A deletion passes the `holders` it read before the cascade
* took the dump out of them.
*/
function announceDumpChange(
before: Dump | null,
after: Dump | null,
holders?: PlaylistHolder[],
): void {
broadcastDumpChange(before, after);
if (!before || (after && after.isPrivate === before.isPrivate)) return;
broadcastHoldersDumps(holders ?? playlistsHolding(before.id));
}
/**
* Notifies the followers of a dump that went public, `before` being private
* and `after` public: the poster's, as if it were just posted, and those of
* each public playlist holding it, as if it were just added. A private dump's
* followers were never told of it, so an edit that makes it public is when
* they learn of it.
*/
function notifyIfWentPublic(before: Dump, after: Dump): void {
if (!before.isPrivate || after.isPrivate) return;
notifyUserFollowersNewDump(after.userId, after.id, after.title);
notifyPlaylistFollowersNowPublic({ dumpId: after.id });
}
/**
@@ -106,7 +137,7 @@ export async function createUrlDump(
if (!isValidHttpUrl(request.url)) {
throw new APIException(400, "Invalid URL");
}
assertCategoriesExist(request.categoryIds ?? []);
assertCategoriesExist("dump", request.categoryIds ?? []);
const dumpId = crypto.randomUUID();
const createdAt = new Date();
@@ -114,12 +145,12 @@ export async function createUrlDump(
// A title typed on the create form wins over the one scraped from the page:
// the poster has seen the preview and is correcting it.
const title = request.title?.trim() || richContent?.title ||
titleFromUrl(request.url);
(siteHost(request.url) ?? request.url);
const isPrivate = request.isPrivate ?? false;
const slug = makeSlug(title, dumpId);
const categoryIds = [...new Set(request.categoryIds ?? [])];
transaction(() => {
const slug = newSlug("dump", userId, title);
db.prepare(
`INSERT INTO dumps (id, kind, title, slug, comment, user_id, created_at, url, url_canonical, rich_content, is_private)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);`,
@@ -136,25 +167,11 @@ export async function createUrlDump(
richContent ? JSON.stringify(richContent) : null,
isPrivate ? 1 : 0,
);
setDumpCategories(dumpId, categoryIds);
setCategories("dump", dumpId, categoryIds);
});
const dump: Dump = {
id: dumpId,
kind: "url",
title,
slug,
comment: request.comment,
userId,
createdAt,
url: request.url,
richContent,
voteCount: 0,
commentCount: 0,
isPrivate,
categoryIds,
};
broadcastDumpChange(null, dump);
const dump: Dump = { ...fetchDump(dumpId), categoryIds };
announceDumpChange(null, dump);
if (!isPrivate) notifyUserFollowersNewDump(userId, dumpId, title);
if (request.comment) {
// A private dump's mentions are notified if it goes public.
@@ -190,12 +207,11 @@ export async function createFileDump(
if (file.size > DUMP_MAX_FILE_SIZE_BYTES) {
throw new APIException(400, "File too large (max 50 MB)");
}
assertCategoriesExist(categoryIds);
assertCategoriesExist("dump", categoryIds);
const dumpId = crypto.randomUUID();
const createdAt = new Date();
const finalTitle = title?.trim() || file.name;
const slug = makeSlug(finalTitle, dumpId);
const dedupedCategoryIds = [...new Set(categoryIds)];
await Deno.mkdir(DUMPS_DIR, { recursive: true });
@@ -205,6 +221,7 @@ export async function createFileDump(
await Deno.writeFile(`${DUMPS_DIR}/${dumpId}`, file.stream());
transaction(() => {
const slug = newSlug("dump", userId, finalTitle);
db.prepare(
`INSERT INTO dumps (id, kind, title, slug, comment, user_id, created_at, file_name, file_mime, file_size, is_private)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);`,
@@ -221,7 +238,7 @@ export async function createFileDump(
file.size,
isPrivate ? 1 : 0,
);
setDumpCategories(dumpId, dedupedCategoryIds);
setCategories("dump", dumpId, dedupedCategoryIds);
});
} catch (err) {
// Leave no file behind when the write or the insert fails.
@@ -229,23 +246,8 @@ export async function createFileDump(
throw err;
}
const dump: Dump = {
id: dumpId,
kind: "file",
title: finalTitle,
slug,
comment,
userId,
createdAt,
fileName: file.name,
fileMime: file.type,
fileSize: file.size,
voteCount: 0,
commentCount: 0,
isPrivate,
categoryIds: dedupedCategoryIds,
};
broadcastDumpChange(null, dump);
const dump: Dump = { ...fetchDump(dumpId), categoryIds: dedupedCategoryIds };
announceDumpChange(null, dump);
if (!isPrivate) notifyUserFollowersNewDump(userId, dumpId, finalTitle);
if (comment) {
// A private dump's mentions are notified if it goes public.
@@ -259,42 +261,12 @@ export async function createFileDump(
}
/**
* Returns a file dump's version, which the app's file and thumbnail URLs carry
* as `?v=`. It changes whenever the file may have, so the file and thumbnail
* routes cache a URL naming the current one for good. The frontend's
* `dumpFileVersion` must compute the same.
* Returns dump `dumpId`, without its categories, or throws a 404. It doesn't
* check privacy: the routes do, with {@link getDump}.
*/
export function dumpFileVersion(
dump: Pick<Dump, "fileSize" | "createdAt" | "updatedAt">,
): string {
return `${dump.fileSize ?? 0}-${
(dump.updatedAt ?? dump.createdAt).getTime()
}`;
}
/**
* Deletes the thumbnails that `routes/thumbnails.ts` generated from a dump's
* upload, once the upload is replaced or deleted. The route makes them again
* from a new one.
*/
async function removeGeneratedThumbnails(dumpId: string): Promise<void> {
await Promise.all(
[`${dumpId}.jpg`, `${dumpId}.preview.jpg`, `${dumpId}.preview.png`].map(
(name) => Deno.remove(`${THUMBNAILS_DIR}/${name}`).catch(() => {}),
),
);
}
/**
* Returns the dump with id or slug `idOrSlug`, without its categories, or
* throws a 404. It doesn't check privacy: the routes do, with {@link getDump}.
*/
function fetchDump(idOrSlug: string): Dump {
const row = UUID_RE.test(idOrSlug)
? db.prepare(`SELECT ${SELECT_COLS} FROM dumps WHERE id = ?;`).get(idOrSlug)
: db.prepare(`SELECT ${SELECT_COLS} FROM dumps WHERE slug = ?;`).get(
idOrSlug,
);
function fetchDump(dumpId: string): Dump {
const row = db.prepare(`SELECT ${SELECT_COLS} FROM dumps WHERE id = ?;`)
.get(dumpId);
if (!row || !isDumpRow(row)) {
throw new APIException(404, "Dump not found");
}
@@ -302,8 +274,8 @@ function fetchDump(idOrSlug: string): Dump {
}
/**
* Returns a dump, by id or slug, as `requestingUserId` may see it. The routes
* resolve every `:dumpId` with it before acting on the dump.
* Returns dump `dumpId` as `requestingUserId` may see it. The routes resolve
* every `:dumpId` with it before acting on the dump.
*
* @throws {APIException} 404 when there is no such dump, or it is private and
* not theirs.
@@ -313,20 +285,41 @@ export function getDump(dumpId: string, requestingUserId?: string): Dump {
if (dump.isPrivate && dump.userId !== requestingUserId) {
throw new APIException(404, "Dump not found");
}
attachCategoryIds([dump]);
attachCategoryIds("dump", [dump]);
return dump;
}
/**
* Returns the dump of `userId`'s whose slug is or was `slug`, as
* `requestingUserId` may see it, for `GET /api/users/:key/dumps/:slug`, which
* reads a dump by its page's path. Reached by a former slug, it comes with
* its current one, which the client puts in its URL.
*
* @throws {APIException} 404 when there is no such dump, or it is private and
* not theirs.
*/
export function getDumpByPath(
userId: string,
slug: string,
requestingUserId?: string,
): Dump {
const dumpId = resolveSlug("dump", userId, slug);
if (!dumpId) throw new APIException(404, "Dump not found");
return getDump(dumpId, requestingUserId);
}
/**
* Returns a page of the dumps `requestingUserId` may see whose title, note, or
* previewed title or description contains `query`, newest first, with how many
* match in all. `GET /api/search` calls it.
* match in all, optionally only those in the dump category `categoryId`.
* `GET /api/search` calls it.
*/
export function searchDumps(
query: string,
page: number,
limit: number,
requestingUserId?: string,
categoryId?: string,
): { items: Dump[]; total: number } {
if (!query.trim()) return { items: [], total: 0 };
const pattern = likePattern(query);
@@ -334,29 +327,28 @@ export function searchDumps(
const searchClause =
`(title ${LIKE} OR comment ${LIKE} OR json_extract(rich_content,'$.title') ${LIKE} OR json_extract(rich_content,'$.description') ${LIKE})`;
const searchParams = [pattern, pattern, pattern, pattern] as const;
const privacyClause = requestingUserId
? "(is_private = 0 OR user_id = ?)"
: "is_private = 0";
const [catJoin, catParams] = categoryJoin("dump", "dumps", categoryId);
const params = [
...catParams,
...(requestingUserId ? [requestingUserId] : []),
pattern,
pattern,
pattern,
pattern,
];
if (requestingUserId) {
const rows = db.prepare(
`SELECT ${SELECT_COLS} FROM dumps WHERE (is_private = 0 OR user_id = ?) AND ${searchClause} ORDER BY created_at DESC LIMIT ? OFFSET ?;`,
).all(requestingUserId, ...searchParams, limit, offset);
`SELECT ${SELECT_COLS} FROM dumps ${catJoin} WHERE ${privacyClause} AND ${searchClause} ORDER BY created_at DESC LIMIT ? OFFSET ?;`,
).all(...params, limit, offset);
const totalRow = db.prepare(
`SELECT COUNT(*) as count FROM dumps WHERE (is_private = 0 OR user_id = ?) AND ${searchClause};`,
).get(requestingUserId, ...searchParams) as { count: number } | undefined;
`SELECT COUNT(*) as count FROM dumps ${catJoin} WHERE ${privacyClause} AND ${searchClause};`,
).get(...params) as { count: number } | undefined;
const items = parseRows(rows, isDumpRow, dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return { items, total: totalRow?.count ?? 0 };
} else {
const rows = db.prepare(
`SELECT ${SELECT_COLS} FROM dumps WHERE is_private = 0 AND ${searchClause} ORDER BY created_at DESC LIMIT ? OFFSET ?;`,
).all(...searchParams, limit, offset);
const totalRow = db.prepare(
`SELECT COUNT(*) as count FROM dumps WHERE is_private = 0 AND ${searchClause};`,
).get(...searchParams) as { count: number } | undefined;
const items = parseRows(rows, isDumpRow, dumpRowToApi);
attachCategoryIds(items);
return { items, total: totalRow?.count ?? 0 };
}
}
/**
@@ -384,14 +376,34 @@ export function listDumps(
}
const items = rows.map(dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}
/**
* Returns the `limit` most upvoted public dumps posted from `from` to `to`,
* the newer first on a tie, leaving out those no one upvoted. The community
* service gathers them for the `/community` page and the digests.
*
* @throws {APIException} 500 when a row doesn't parse as a dump.
*/
export function listTopDumps(from: Date, to: Date, limit: number): Dump[] {
const rows = db.prepare(
`SELECT ${SELECT_COLS} FROM dumps
WHERE is_private = 0 AND vote_count > 0
AND created_at >= ? AND created_at < ?
ORDER BY vote_count DESC, created_at DESC
LIMIT ?;`,
).all(from.toISOString(), to.toISOString(), limit);
const items = parseRows(rows, isDumpRow, dumpRowToApi);
attachCategoryIds("dump", items);
return items;
}
/**
* Returns the dumps `requestingUserId` may see that were posted with the same
* link as `url`, newest first, for the create form's duplicate notice
* (`GET /api/dumps/by-url`). Links are compared in their canonical form, so
* (`GET /api/dumps?url=`). Links are compared in their canonical form, so
* tracking parameters, the scheme, `www.` or a YouTube share format don't hide
* a repost.
*/
@@ -413,7 +425,7 @@ export function findDumpsByUrl(
LIMIT ?;`,
).all(canonical, requestingUserId ?? null, limit) as {
id: string;
slug: string | null;
slug: string;
title: string;
created_at: string;
vote_count: number;
@@ -423,9 +435,9 @@ export function findDumpsByUrl(
return rows.map((row) => ({
id: row.id,
slug: row.slug ?? undefined,
slug: row.slug,
title: row.title,
username: row.username,
ownerUsername: row.username,
createdAt: new Date(row.created_at),
voteCount: row.vote_count,
commentCount: row.comment_count,
@@ -435,27 +447,39 @@ export function findDumpsByUrl(
/**
* Applies `request` to a dump, for `PUT /api/dumps/:dumpId`. A new link has its
* preview fetched and gives the dump its page's title, unless `request` sets
* one. A file dump ignores `url`. Clients get the dump as `dump_updated`, or a
* one. A file dump ignores `url`. A new `slug` goes through `changeSlug`,
* which `exempt` (a moderator) frees from its rate limit; a new title leaves
* the slug as it is. Clients get the dump as `dump_updated`, or a
* `dump_deleted` when it turns private and a `dump_created` when it turns
* public. Anyone @mentioned in the note is notified.
* public. Anyone @mentioned in the note is notified, and when it turns public,
* its followers are as when it is posted ({@link notifyIfWentPublic}).
*
* @throws {APIException} 404 when there is no such dump, 400 when a new URL
* isn't a public http(s) URL or a category doesn't exist.
* isn't a public http(s) URL, a category doesn't exist or the slug isn't
* valid, 409 when the slug is taken, 429 when it changed too often lately.
*/
export async function updateDump(
dumpId: string,
request: UpdateDumpRequest,
exempt = false,
): Promise<Dump> {
const dump = fetchDump(dumpId);
if (request.categoryIds !== undefined) {
assertCategoriesExist(request.categoryIds);
assertCategoriesExist("dump", request.categoryIds);
}
if (dump.kind === "url" && !isValidHttpUrl(request.url ?? dump.url!)) {
throw new APIException(400, "Invalid URL");
}
// First, so that a refused slug leaves the rest of the dump unchanged.
if (request.slug !== undefined) {
changeSlug("dump", dump, request.slug, exempt);
}
const slug = request.slug ?? dump.slug;
const now = new Date();
if (dump.kind === "file") {
const title = request.title?.trim() || dump.title;
const slug = title !== dump.title ? makeSlug(title, dumpId) : dump.slug;
const updatedDump: Dump = {
...dump,
title,
@@ -469,10 +493,9 @@ export async function updateDump(
updatedAt: now,
};
db.prepare(
`UPDATE dumps SET title = ?, slug = ?, comment = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
`UPDATE dumps SET title = ?, comment = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
).run(
updatedDump.title,
updatedDump.slug ?? null,
updatedDump.comment ?? null,
updatedDump.isPrivate ? 1 : 0,
now.toISOString(),
@@ -480,12 +503,13 @@ export async function updateDump(
);
if (request.categoryIds !== undefined) {
const ids = [...new Set(request.categoryIds)];
setDumpCategories(dumpId, ids);
setCategories("dump", dumpId, ids);
updatedDump.categoryIds = ids;
} else {
attachCategoryIds([updatedDump]);
attachCategoryIds("dump", [updatedDump]);
}
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
notifyIfWentPublic(dump, updatedDump);
if (updatedDump.comment) {
if (!updatedDump.isPrivate) {
notifyMentions(
@@ -504,26 +528,21 @@ export async function updateDump(
const newUrl = request.url ?? dump.url!;
if (!isValidHttpUrl(newUrl)) {
throw new APIException(400, "Invalid URL");
}
let { richContent, title } = dump;
if (newUrl !== dump.url) {
richContent = await fetchRichContent(newUrl);
title = richContent?.title ?? titleFromUrl(newUrl);
title = richContent?.title ?? siteHost(newUrl) ?? newUrl;
}
if (request.title?.trim()) {
title = request.title.trim();
}
const newSlug = title !== dump.title ? makeSlug(title, dumpId) : dump.slug;
const updatedDump: Dump = {
...dump,
title,
slug: newSlug,
slug,
comment: "comment" in request
? (request.comment ?? undefined)
: dump.comment,
@@ -537,10 +556,9 @@ export async function updateDump(
const row = dumpApiToRow(updatedDump);
const result = db.prepare(
`UPDATE dumps SET title = ?, slug = ?, comment = ?, url = ?, url_canonical = ?, rich_content = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
`UPDATE dumps SET title = ?, comment = ?, url = ?, url_canonical = ?, rich_content = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
).run(
row.title,
row.slug,
row.comment,
row.url,
row.url ? canonicalizeUrl(row.url) : null,
@@ -556,13 +574,14 @@ export async function updateDump(
if (request.categoryIds !== undefined) {
const ids = [...new Set(request.categoryIds)];
setDumpCategories(dumpId, ids);
setCategories("dump", dumpId, ids);
updatedDump.categoryIds = ids;
} else {
attachCategoryIds([updatedDump]);
attachCategoryIds("dump", [updatedDump]);
}
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
notifyIfWentPublic(dump, updatedDump);
if (updatedDump.comment) {
if (!updatedDump.isPrivate) {
notifyMentions(
@@ -581,14 +600,17 @@ export async function updateDump(
/**
* Replaces a file dump's upload and its comment, for
* `PUT /api/dumps/:dumpId/file`. A `title`, `categoryIds` or `isPrivate` left
* out stays as it is. Visibility is taken here so that a new file and a change
* of visibility land together. The old file is put back if the database update
* fails, and clients are told as {@link updateDump} tells them.
* `PUT /api/dumps/:dumpId/file`. A `title`, `slug`, `categoryIds` or
* `isPrivate` left out stays as it is. Visibility is taken here so that a new
* file and a change of visibility land together. The slug changes as in
* {@link updateDump}, before the file is written. The old file is put back if
* the database update fails, and clients and followers are told as
* {@link updateDump} tells them.
*
* @throws {APIException} 404 when there is no such dump, 400 when it isn't a
* file dump, the title or comment is too long, the file's type isn't allowed,
* it exceeds 50 MB, or a category doesn't exist.
* it exceeds 50 MB, a category doesn't exist or the slug isn't valid, 409 when
* the slug is taken, 429 when it changed too often lately.
*/
export async function replaceFileDump(
dumpId: string,
@@ -597,6 +619,8 @@ export async function replaceFileDump(
title?: string,
categoryIds?: string[],
isPrivate?: boolean,
slug?: string,
exempt = false,
): Promise<Dump> {
assertFileDumpText(title, comment);
if (!isAllowedMime(file.type)) {
@@ -610,7 +634,8 @@ export async function replaceFileDump(
if (dump.kind !== "file") {
throw new APIException(400, "Not a file dump");
}
if (categoryIds !== undefined) assertCategoriesExist(categoryIds);
if (categoryIds !== undefined) assertCategoriesExist("dump", categoryIds);
if (slug !== undefined) changeSlug("dump", dump, slug, exempt);
// The new file is written beside the old one and renamed over it (atomic: a
// reader sees one or the other). A hard link keeps the old one reachable so
@@ -633,17 +658,13 @@ export async function replaceFileDump(
const now = new Date();
const finalTitle = title?.trim() || dump.title;
const newSlug = finalTitle !== dump.title
? makeSlug(finalTitle, dumpId)
: dump.slug;
const finalPrivate = isPrivate ?? dump.isPrivate;
try {
transaction(() => {
db.prepare(
`UPDATE dumps SET title = ?, slug = ?, file_name = ?, file_mime = ?, file_size = ?, comment = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
`UPDATE dumps SET title = ?, file_name = ?, file_mime = ?, file_size = ?, comment = ?, is_private = ?, updated_at = ? WHERE id = ?;`,
).run(
finalTitle,
newSlug ?? null,
file.name,
file.type,
file.size,
@@ -653,7 +674,7 @@ export async function replaceFileDump(
dumpId,
);
if (categoryIds !== undefined) {
setDumpCategories(dumpId, [...new Set(categoryIds)]);
setCategories("dump", dumpId, [...new Set(categoryIds)]);
}
});
} catch (err) {
@@ -662,12 +683,12 @@ export async function replaceFileDump(
throw err;
}
if (hadOld) await Deno.remove(backupPath).catch(() => {});
await removeGeneratedThumbnails(dumpId);
await removeGeneratedMedia(dumpId);
if (categoryIds !== undefined) {
dump.categoryIds = [...new Set(categoryIds)];
} else {
attachCategoryIds([dump]);
attachCategoryIds("dump", [dump]);
}
if (comment) {
@@ -680,7 +701,7 @@ export async function replaceFileDump(
const updatedDump: Dump = {
...dump,
title: finalTitle,
slug: newSlug,
slug: slug ?? dump.slug,
fileName: file.name,
fileMime: file.type,
fileSize: file.size,
@@ -688,14 +709,15 @@ export async function replaceFileDump(
isPrivate: finalPrivate,
updatedAt: now,
};
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
notifyIfWentPublic(dump, updatedDump);
return updatedDump;
}
/**
* Returns a page of `userId`'s dumps, newest first, for
* `GET /api/users/:username/dumps`. The route sets `includePrivate` when users
* ask for their own.
* `GET /api/users/:key/dumps`, optionally only those in `categoryId`. The
* route sets `includePrivate` when users ask for their own.
*
* @throws {APIException} 500 when a row doesn't parse as a dump.
*/
@@ -704,24 +726,28 @@ export function getDumpsByUser(
page: number,
limit: number,
includePrivate: boolean,
categoryId?: string,
): Page<Dump> {
const offset = (page - 1) * limit;
const privacyFilter = includePrivate ? "" : " AND is_private = 0";
const privacyFilter = includePrivate ? "" : " AND d.is_private = 0";
const [catJoin, catParams] = categoryJoin("dump", "d", categoryId);
const rows = db.prepare(
`SELECT ${SELECT_COLS} FROM dumps WHERE user_id = ?${privacyFilter} ORDER BY created_at DESC LIMIT ? OFFSET ?;`,
).all(userId, limit + 1, offset);
`SELECT ${SELECT_COLS_ALIASED} FROM dumps d ${catJoin}
WHERE d.user_id = ?${privacyFilter}
ORDER BY d.created_at DESC LIMIT ? OFFSET ?;`,
).all(...catParams, userId, limit + 1, offset);
if (!rows.every(isDumpRow)) {
throw new APIException(500, "Malformed dump data");
}
const items = rows.map(dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}
/**
* Returns a page of the dumps `userId` upvoted, latest vote first, for
* `GET /api/users/:username/votes`. Their own private dumps are included only
* when they are the one asking.
* `GET /api/users/:key/upvoted`, optionally only those in `categoryId`.
* Their own private dumps are included only when they are the one asking.
*
* @throws {APIException} 500 when a row doesn't parse as a dump.
*/
@@ -730,40 +756,40 @@ export function getVotedDumpsByUser(
page: number,
limit: number,
requestingUserId: string | null,
categoryId?: string,
): Page<Dump> {
const offset = (page - 1) * limit;
const dumpCols = SELECT_COLS_ALIASED;
let rows: unknown[];
if (requestingUserId === userId) {
rows = db.prepare(
`SELECT ${dumpCols}
const isOwn = requestingUserId === userId;
const privacyFilter = isOwn
? "(d.is_private = 0 OR d.user_id = ?)"
: "d.is_private = 0";
const [catJoin, catParams] = categoryJoin("dump", "d", categoryId);
const rows = db.prepare(
`SELECT ${SELECT_COLS_ALIASED}
FROM dumps d
INNER JOIN votes v ON d.id = v.dump_id
WHERE v.user_id = ? AND (d.is_private = 0 OR d.user_id = ?)
${catJoin}
WHERE v.user_id = ? AND ${privacyFilter}
ORDER BY v.created_at DESC LIMIT ? OFFSET ?;`,
).all(userId, userId, limit + 1, offset);
} else {
rows = db.prepare(
`SELECT ${dumpCols}
FROM dumps d
INNER JOIN votes v ON d.id = v.dump_id
WHERE v.user_id = ? AND d.is_private = 0
ORDER BY v.created_at DESC LIMIT ? OFFSET ?;`,
).all(userId, limit + 1, offset);
}
).all(
...catParams,
userId,
...(isOwn ? [userId] : []),
limit + 1,
offset,
);
if (!rows.every(isDumpRow)) {
throw new APIException(500, "Malformed dump data");
}
const items = rows.map(dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}
/**
* Fetches a URL dump's preview again, bypassing the cache, and replaces its
* title and preview, for `POST /api/dumps/:dumpId/refresh-metadata`.
* title and preview, for `POST /api/dumps/:dumpId/metadata`. The slug stays.
*
* @throws {APIException} 404 when there is no such dump, 400 when it isn't a
* URL dump, 502 when the page can't be reached.
@@ -787,7 +813,7 @@ export async function refreshDumpMetadata(dumpId: string): Promise<Dump> {
);
}
const title = richContent?.title ?? titleFromUrl(dump.url);
const title = richContent?.title ?? siteHost(dump.url) ?? dump.url;
const updatedDump: Dump = { ...dump, title, richContent };
const row = dumpApiToRow(updatedDump);
@@ -795,25 +821,18 @@ export async function refreshDumpMetadata(dumpId: string): Promise<Dump> {
`UPDATE dumps SET title = ?, rich_content = ? WHERE id = ?;`,
).run(row.title, row.rich_content, row.id);
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
return updatedDump;
}
/**
* Returns a dump's id and custom thumbnail type, or undefined without one, for
* the thumbnail route. It doesn't check privacy: the route calls
* {@link getDump} first.
*
* @throws {APIException} 404 when there is no such dump.
* Returns where dump `dumpId`'s custom thumbnail is kept: the thumbnail route
* writes and serves it, and this service deletes it with the thumbnail or the
* dump.
*/
export function getDumpThumbnailInfo(
idOrSlug: string,
): { id: string; mime: string } | undefined {
const dump = fetchDump(idOrSlug);
return dump.thumbnailMime
? { id: dump.id, mime: dump.thumbnailMime }
: undefined;
export function customThumbnailPath(dumpId: string): string {
return `${THUMBNAILS_DIR}/${dumpId}-custom`;
}
/**
@@ -829,7 +848,7 @@ export function setDumpThumbnail(dumpId: string, mime: string): Dump {
dump.id,
);
const updatedDump: Dump = { ...dump, thumbnailMime: mime };
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
return updatedDump;
}
@@ -843,20 +862,22 @@ export async function removeDumpThumbnail(dumpId: string): Promise<Dump> {
const dump = fetchDump(dumpId);
db.prepare(`UPDATE dumps SET custom_thumbnail_mime = NULL WHERE id = ?;`)
.run(dump.id);
await Deno.remove(`${THUMBNAILS_DIR}/${dump.id}-custom`).catch(() => {});
await Deno.remove(customThumbnailPath(dump.id)).catch(() => {});
const updatedDump: Dump = { ...dump, thumbnailMime: undefined };
broadcastDumpChange(dump, updatedDump);
announceDumpChange(dump, updatedDump);
return updatedDump;
}
/**
* Deletes a dump with its files and thumbnails, for
* `DELETE /api/dumps/:dumpId`, and broadcasts `dump_deleted`.
* `DELETE /api/dumps/:dumpId`, and broadcasts `dump_deleted` and the new
* dumps of the playlists that held it.
*
* @throws {APIException} 404 when there is no such dump.
*/
export async function deleteDump(dumpId: string): Promise<void> {
const dump = fetchDump(dumpId);
const holders = playlistsHolding(dumpId);
const result = db.prepare(`DELETE FROM dumps WHERE id = ?;`).run(dumpId);
@@ -866,11 +887,11 @@ export async function deleteDump(dumpId: string): Promise<void> {
if (dump.kind === "file") {
await Deno.remove(`${DUMPS_DIR}/${dump.id}`).catch(() => {});
await removeGeneratedThumbnails(dump.id);
await removeGeneratedMedia(dump.id);
}
if (dump.thumbnailMime) {
await Deno.remove(`${THUMBNAILS_DIR}/${dump.id}-custom`).catch(() => {});
await Deno.remove(customThumbnailPath(dump.id)).catch(() => {});
}
broadcastDumpChange(dump, null);
announceDumpChange(dump, null, holders);
}

153
api/services/email-service.ts
View File

@@ -1,121 +1,76 @@
/**
* Sends mail through the SMTP server of {@link SMTPS_URL}, which may be unset.
* Its two callers, the welcome email at sign-up and the password reset link,
* check {@link isEmailEnabled} first and don't wait for delivery.
* Sends the site's email through the SMTP server of {@link SMTPS_URL}, which
* may be unset. The welcome email, the password reset link and the digests go
* through it, each rendered first by its module in `api/mail/`. They check
* {@link isEmailEnabled} before rendering.
*
* Importing this module throws when {@link SMTPS_URL} is set without
* {@link FROM_EMAIL}, so a misconfigured server doesn't start rather than fail
* at its first send.
*
* @module
*/
import nodemailer, { type SendMailOptions, type Transporter } from "nodemailer";
import nodemailer from "nodemailer";
import { SMTPS_URL } from "../config.ts";
import { EMAIL_ENABLED, FROM_EMAIL, SMTPS_URL } from "../config.ts";
import type { RenderedEmail } from "../mail/layout.ts";
/**
* One address, a comma-separated list of them, or an array, as nodemailer
* takes them.
*/
export type EmailRecipient = string | string[];
/**
* Passed to {@link sendEmail}. It needs a recipient, a subject, and a text or
* HTML body; the callers send the same Markdown as both, rendered for `html`.
*/
export interface EmailMessage {
from: string;
to: EmailRecipient;
subject: string;
text?: string;
html?: string;
cc?: EmailRecipient;
bcc?: EmailRecipient;
replyTo?: string;
if (EMAIL_ENABLED && !FROM_EMAIL) {
throw new Error(
"GERBEUR_FROM_EMAIL environment variable is required when GERBEUR_SMTPS_URL is set.",
);
}
/**
* Returned by {@link sendEmail} once the SMTP server has taken the message.
* No caller reads it yet.
* Passed to {@link sendEmail}: a rendered email, its recipient, and the
* headers some emails add (the digests' `List-Unsubscribe`).
*/
export interface EmailSendResult {
messageId: string;
accepted: string[];
rejected: string[];
export interface OutgoingEmail extends RenderedEmail {
to: string;
headers?: Record<string, string>;
}
let transporter: Transporter | null = null;
/**
* Hands a message to a mail server. nodemailer's SMTP transport is one, and
* the tests put a recording one in its place with {@link setEmailTransport}.
*/
export interface EmailTransport {
sendMail(message: OutgoingEmail & { from: string }): Promise<unknown>;
}
let transport: EmailTransport | null = EMAIL_ENABLED
? nodemailer.createTransport(SMTPS_URL)
: null;
/**
* Checks whether an SMTP server is configured. Without one, sign-up sends no
* welcome email and a password reset request does nothing.
* Checks whether email can be sent. Without an SMTP server, sign-up sends no
* welcome email, a password reset request does nothing, and no digest goes
* out.
*/
export function isEmailEnabled(): boolean {
return SMTPS_URL.length > 0;
}
/** Flattens the recipient forms nodemailer takes and returns into addresses. */
function normalizeRecipients(
recipient: string | readonly (string | { address?: string })[] | undefined,
): string[] {
if (!recipient) return [];
if (typeof recipient === "string") {
return recipient.split(",").map((s) => s.trim()).filter(Boolean);
}
return recipient.flatMap((entry) =>
typeof entry === "string" ? entry.trim() : (entry.address?.trim() ?? "")
).filter(Boolean);
}
function getTransporter(): Transporter {
if (!isEmailEnabled()) {
throw new Error(
"GERBEUR_SMTPS_URL environment variable is required to send emails.",
);
}
if (!transporter) {
transporter = nodemailer.createTransport(SMTPS_URL);
}
return transporter;
return transport !== null;
}
/**
* Sends `message` and resolves once the SMTP server has accepted it. The
* transport is created on the first send.
*
* @throws {Error} When email is disabled, the message lacks a recipient,
* subject or body, or the server refuses it.
* Replaces the transport, so that email is enabled and every message goes to
* `next`; null disables email again. The tests use it to read what was sent.
*/
export async function sendEmail(
message: EmailMessage,
): Promise<EmailSendResult> {
if (normalizeRecipients(message.to).length === 0) {
throw new Error("Email recipient is required.");
}
if (!message.subject.trim()) {
throw new Error("Email subject is required.");
}
if (!message.text?.trim() && !message.html?.trim()) {
throw new Error("Email body is required (text and/or html).");
}
const options: SendMailOptions = {
from: message.from,
to: message.to,
cc: message.cc,
bcc: message.bcc,
subject: message.subject,
text: message.text,
html: message.html,
replyTo: message.replyTo,
};
const info = await getTransporter().sendMail(options);
return {
messageId: info.messageId,
accepted: normalizeRecipients(info.accepted),
rejected: normalizeRecipients(info.rejected),
};
export function setEmailTransport(next: EmailTransport | null): void {
transport = next;
}
/**
* Sends `email` from {@link FROM_EMAIL} and resolves once the SMTP server has
* accepted it. The welcome and reset emails don't wait for it: a slow server
* mustn't hold up the request.
*
* @throws {Error} When email is disabled, or when the server refuses the
* message or can't be reached.
*/
export async function sendEmail(email: OutgoingEmail): Promise<void> {
if (!transport) {
throw new Error("Email is disabled: GERBEUR_SMTPS_URL is unset.");
}
await transport.sendMail({ ...email, from: FROM_EMAIL });
}

138
api/services/federation-peer-service.ts Normal file
View File

@@ -0,0 +1,138 @@
/**
* Keeps the instances an admin trusts or blocks for federated sign-in
* (`federation_peers`). Users of a trusted instance sign in here directly,
* those of a blocked one never do, and an instance on neither list is invite
* only. `federation-service.ts` reads it at each sign-in, `/manage` edits it
* through `routes/federation.ts`, and the login page offers the trusted ones.
*
* @module
*/
import { APIException } from "../lib/api-exception.ts";
import { db } from "../db/connection.ts";
import { PUBLIC_URL } from "../config.ts";
import {
type FederationPeer,
instanceHost,
isPeerStatus,
parseInstance,
type PeerStatus,
} from "../../shared/federation.ts";
import { fetchMetadata, resolveIssuer } from "../remote/gerbeur-instance.ts";
import { isSelfHost } from "../lib/self-url.ts";
import { closeUserSockets } from "./ws-service.ts";
interface PeerRow {
issuer: string;
status: string;
name: string | null;
created_at: string;
}
/** Converts a row to a {@link FederationPeer}. */
function rowToPeer(row: PeerRow): FederationPeer {
return {
issuer: row.issuer,
host: instanceHost(row.issuer),
name: row.name,
status: isPeerStatus(row.status) ? row.status : "blocked",
createdAt: new Date(row.created_at),
};
}
/**
* Returns what an admin decided about the instance at `issuer`, or null when
* it is on neither list.
*/
export function getPeerStatus(issuer: string): PeerStatus | null {
const row = db.prepare(
`SELECT status FROM federation_peers WHERE issuer = ?;`,
).get(issuer) as { status: string } | undefined;
return row && isPeerStatus(row.status) ? row.status : null;
}
/**
* Returns the instances on the lists, by name: every one for `/manage`, or
* only the trusted ones, which the login page offers.
*/
export function listPeers(trustedOnly: boolean): FederationPeer[] {
const rows = db.prepare(
`SELECT issuer, status, name, created_at FROM federation_peers
${trustedOnly ? "WHERE status = 'trusted'" : ""}
ORDER BY lower(coalesce(name, issuer));`,
).all() as unknown as PeerRow[];
return rows.map(rowToPeer);
}
/**
* Puts the instance `instance` names, by its API's address or its pages'
* (`resolveIssuer`), on the list of `status`, for `/manage`, and returns it. A trusted one must answer as a gerbeur instance, whose name
* is kept; a blocked one is blocked whether it answers or not, keeping the
* name it had. Blocking one ends the sessions of its users here at once:
* their tokens stop counting by `getSessionState`, and this closes their
* sockets.
*
* @throws {APIException} 400 when `instance` names no instance, or this one,
* 502 when an instance to trust doesn't answer as a gerbeur instance.
*/
export async function setPeer(
instance: string,
status: PeerStatus,
): Promise<FederationPeer> {
const origin = parseInstance(instance);
if (!origin) throw new APIException(400, "Not an instance's address");
if (isSelfHost(new URL(origin).host)) {
throw new APIException(400, "That is this instance");
}
// An instance is listed by its issuer, which its pages' address leads to;
// one that doesn't answer may still be blocked, by the address given.
let issuer = origin;
let name: string | null = null;
try {
issuer = await resolveIssuer(origin);
name = (await fetchMetadata(issuer)).gerbeur_site_name;
} catch (err) {
console.error(`[federation] ${origin} doesn't answer:`, err);
if (status === "trusted") {
throw new APIException(502, "That address doesn't answer as a gerbeur");
}
}
if (issuer === PUBLIC_URL) {
throw new APIException(400, "That is this instance");
}
db.prepare(
`INSERT INTO federation_peers (issuer, status, name, created_at)
VALUES (?, ?, ?, ?)
ON CONFLICT (issuer) DO UPDATE SET status = excluded.status,
name = coalesce(excluded.name, name);`,
).run(issuer, status, name, new Date().toISOString());
if (status === "blocked") {
const users = db.prepare(
`SELECT user_id FROM user_identities WHERE issuer = ?;`,
).all(issuer) as { user_id: string }[];
for (const { user_id } of users) {
closeUserSockets(user_id, "Instance blocked");
}
}
const row = db.prepare(
`SELECT issuer, status, name, created_at FROM federation_peers WHERE issuer = ?;`,
).get(issuer) as unknown as PeerRow;
return rowToPeer(row);
}
/**
* Takes the instance whose host is `host` off the lists, for `/manage`: its
* users are invite only again, and those already here may sign in again if it
* was blocked.
*
* @throws {APIException} 404 when no listed instance has that host.
*/
export function removePeer(host: string): void {
const peer = listPeers(false).find((p) => p.host === host.toLowerCase());
if (!peer) throw new APIException(404, "Instance not found");
db.prepare(`DELETE FROM federation_peers WHERE issuer = ?;`).run(
peer.issuer,
);
}

501
api/services/federation-service.ts Normal file
View File

@@ -0,0 +1,501 @@
/**
* Plays the client in federated sign-in: it lets users of other gerbeur
* instances sign in here with their account there, their home.
*
* 1. The login or register page names the user's home, and
* {@link startLogin} answers the home's `/authorize` page with a request:
* a random state, the hash of a PKCE verifier, and a nonce, kept in
* `federation_logins` for 10 minutes.
* 2. The home sends the user back to the `/federation/callback` page with a
* code. {@link completeLogin} trades it for the home's identity token,
* checks its signature against the home's keys, and finds or creates the
* local account linked to it (`user_identities`), named `<name>@<host>`.
* 3. The home's events ({@link receiveEvent}) keep the account's name, email
* and avatar in step, and end its sessions when the home says so.
*
* Whether a user of another instance may join is the admin's call
* (`federation-peer-service.ts`): from a trusted instance they may, from a
* blocked one they may not, and from any other only with an invite, which the
* register page carries. None may while the deployment turns sign-ins from
* other instances off (`FEDERATION.incoming`). Once here, an account works like any other, but its
* password, email, avatar and name are its home's (`assertLocalUser`), and
* change on its pages there ({@link homePageUrl}).
*
* @module
*/
import { decodeJwt, type JSONWebKeySet } from "@panva/jose";
import { createHash, randomBytes } from "node:crypto";
import { APIException } from "../lib/api-exception.ts";
import { FEDERATION } from "../config.ts";
import { db, transaction } from "../db/connection.ts";
import { verifyInstanceToken } from "../lib/jwt.ts";
import { isSelfHost } from "../lib/self-url.ts";
import { type RateLimit, takeToken } from "../lib/rate-limit.ts";
import {
type FederationLoginRequest,
type FederationSessionRequest,
type IdentityClaims,
instanceHost,
type InstanceMetadata,
isFederationEvent,
isIdentityClaims,
parseInstance,
} from "../../shared/federation.ts";
import { userPath } from "../../shared/paths.ts";
import { EMAIL_RE, splitHandle, USERNAME_RE } from "../../shared/validation.ts";
import { toPublicUser, type User } from "../model/user.ts";
import {
exchangeCode,
fetchAvatar,
fetchKeys,
fetchMetadata,
resolveIssuer,
} from "../remote/gerbeur-instance.ts";
import { ISSUER, REDIRECT_URI } from "./identity-provider-service.ts";
import { getPeerStatus } from "./federation-peer-service.ts";
import {
claimInvite,
releaseInvite,
validateInvite,
} from "./invite-service.ts";
import {
createRemoteUser,
getUserById,
revokeSessions,
sendWelcomeEmail,
setUserAvatar,
setUsername,
updateUser,
} from "./user-service.ts";
import { broadcastUserUpdated, updateClientUsername } from "./ws-service.ts";
/** How long a sign-in may take, from the login page to the callback. */
const LOGIN_TTL_MS = 10 * 60_000;
/**
* Allows an instance a burst of 120 events, as when many of its users change
* at once, then one a second. Each may fetch its keys, so a stranger posting
* events in an instance's name can't make this one hammer it.
*/
const EVENT_LIMIT: RateLimit = { capacity: 120, refill: 1 };
/**
* Refuses a sign-in from another instance while the deployment turns them off
* (`FEDERATION.incoming`), as {@link startLogin} and {@link completeLogin}
* must first.
*
* @throws {APIException} 403 when they are off.
*/
function assertIncoming(): void {
if (!FEDERATION.incoming) {
throw new APIException(
403,
"This instance doesn't take accounts from other instances",
);
}
}
/**
* Returns the issuer of the instance `instance` names, by its API's address or
* its pages' (`resolveIssuer`), checked as a home: not this instance, nor a
* blocked one.
*
* @throws {APIException} 400 when it names no instance or this one, 403 when
* it is blocked, 502 when it doesn't answer as a gerbeur.
*/
async function homeOf(instance: string): Promise<string> {
const origin = parseInstance(instance);
if (!origin) throw new APIException(400, "Not an instance's address");
const blocked = (issuer: string) => getPeerStatus(issuer) === "blocked";
if (isSelfHost(new URL(origin).host)) {
throw new APIException(400, "That is this instance");
}
// Typed by its API's address, a blocked instance is refused unreached.
if (blocked(origin)) {
throw new APIException(403, "This instance is blocked here");
}
let issuer: string;
try {
issuer = await resolveIssuer(origin);
} catch (err) {
console.error(`[federation] ${origin} unreachable:`, err);
throw new APIException(502, "That instance doesn't answer as a gerbeur");
}
if (issuer === ISSUER) throw new APIException(400, "That is this instance");
if (blocked(issuer)) {
throw new APIException(403, "This instance is blocked here");
}
return issuer;
}
/**
* Starts a sign-in with an account on the instance `request.instance` names,
* and returns its `/authorize` page with the request, for
* `POST /api/federation/logins`. Unless that instance is trusted, or someone
* here already comes from it, the request must carry a valid invite, which is
* claimed only once the account is made; {@link completeLogin} makes no
* account without one.
*
* @throws {APIException} 400 when the instance isn't one; 403 when sign-ins
* from other instances are off, or it is blocked, neither trusted nor
* invited, or keeps its users from signing in elsewhere; 404 or 409 when the
* invite doesn't work; 502 when it doesn't answer as a gerbeur.
*/
export async function startLogin(
request: FederationLoginRequest,
): Promise<string> {
assertIncoming();
const issuer = await homeOf(request.instance);
if (request.inviteToken) {
validateInvite(request.inviteToken);
} else if (
getPeerStatus(issuer) !== "trusted" &&
// Someone who joined with an invite signs in again without one; whether
// this user is one of them is only known once their home answers.
!db.prepare(`SELECT 1 FROM user_identities WHERE issuer = ? LIMIT 1;`)
.get(issuer)
) {
throw new APIException(
403,
"Users of this instance need an invite to join",
);
}
let metadata: InstanceMetadata;
try {
metadata = await fetchMetadata(issuer);
} catch (err) {
console.error(`[federation] ${issuer} unreachable:`, err);
throw new APIException(502, "That instance doesn't answer as a gerbeur");
}
if (metadata.gerbeur_outgoing === false) {
throw new APIException(
403,
"Accounts on that instance don't sign in to other instances",
);
}
const state = randomBytes(24).toString("base64url");
const verifier = randomBytes(48).toString("base64url");
const nonce = randomBytes(24).toString("base64url");
const now = new Date();
transaction(() => {
db.prepare(`DELETE FROM federation_logins WHERE created_at < ?;`).run(
new Date(now.getTime() - LOGIN_TTL_MS).toISOString(),
);
db.prepare(
`INSERT INTO federation_logins (state, issuer, code_verifier, nonce, invite_token, created_at)
VALUES (?, ?, ?, ?, ?, ?);`,
).run(
state,
issuer,
verifier,
nonce,
request.inviteToken ?? null,
now.toISOString(),
);
});
const url = new URL(metadata.authorization_endpoint);
url.search = new URLSearchParams({
response_type: "code",
scope: "openid profile email",
client_id: ISSUER,
redirect_uri: REDIRECT_URI,
state,
code_challenge: createHash("sha256").update(verifier).digest("base64url"),
code_challenge_method: "S256",
nonce,
}).toString();
return url.href;
}
/**
* Returns the keys of the instance at `issuer`, refetched when `fresh` asks.
*
* @throws {APIException} 502 when they can't be fetched: the sender may try
* again, which it wouldn't after a 4xx.
*/
async function keysOf(issuer: string, fresh: boolean): Promise<JSONWebKeySet> {
try {
return await fetchKeys(issuer, fresh);
} catch (err) {
console.error(`[federation] ${issuer}'s keys failed:`, err);
throw new APIException(502, "That instance's keys can't be fetched");
}
}
/**
* Returns the payload of `token`, an identity token or an event from the
* instance at `issuer`, once its signature checks against that instance's
* keys and it is addressed to this one. The keys are refetched once when the
* token names one they lack.
*
* @throws {APIException} 401 when it doesn't verify, 502 when the keys can't
* be fetched.
*/
async function verifyFrom(
token: string,
issuer: string,
): Promise<Record<string, unknown>> {
const claims = { issuer, audience: ISSUER };
let failure: unknown;
for (const fresh of [false, true]) {
if (
fresh &&
(failure as { code?: string }).code !== "ERR_JWKS_NO_MATCHING_KEY"
) {
break;
}
const keys = await keysOf(issuer, fresh);
try {
return await verifyInstanceToken(token, keys, claims);
} catch (err) {
failure = err;
}
}
console.error(`[federation] a token from ${issuer} failed:`, failure);
throw new APIException(401, "Invalid token");
}
/**
* Returns `claims` from the instance at `issuer` once checked: a name a local
* user could have, an email, and an avatar on that instance, so that it can't
* make this server fetch from elsewhere. Returns null when they aren't.
*/
function checkClaims(claims: unknown, issuer: string): IdentityClaims | null {
if (
!isIdentityClaims(claims) ||
!USERNAME_RE.test(claims.preferred_username) ||
!EMAIL_RE.test(claims.email)
) {
return null;
}
if (claims.picture !== undefined) {
try {
if (new URL(claims.picture).origin !== issuer) return null;
} catch {
return null;
}
}
return claims;
}
/**
* Finishes a sign-in begun by {@link startLogin}, for
* `POST /api/federation/sessions`, and returns the account it signs in to:
* the one linked to the identity, brought up to date, or a new one. A login
* works once. A new account needs the home to be trusted, or the invite the
* login was started with, claimed now.
*
* @throws {APIException} 400 when the login is unknown or expired; 401 when
* the identity token doesn't verify; 403 when sign-ins from other instances
* are off, or the account may not be created; 409 when its handle is
* someone's; 502 when the home fails.
*/
export async function completeLogin(
request: FederationSessionRequest,
): Promise<User> {
assertIncoming();
const login = db.prepare(
`DELETE FROM federation_logins WHERE state = ?
RETURNING issuer, code_verifier, nonce, invite_token, created_at;`,
).get(request.state) as
| {
issuer: string;
code_verifier: string;
nonce: string;
invite_token: string | null;
created_at: string;
}
| undefined;
if (
!login ||
Date.now() - new Date(login.created_at).getTime() > LOGIN_TTL_MS
) {
throw new APIException(400, "This sign-in has expired, start again");
}
const issuer = login.issuer;
let idToken: string;
try {
idToken = await exchangeCode(issuer, {
code: request.code,
clientId: ISSUER,
redirectUri: REDIRECT_URI,
codeVerifier: login.code_verifier,
});
} catch (err) {
console.error(`[federation] ${issuer} refused a code:`, err);
throw new APIException(502, "The instance refused the sign-in");
}
const payload = await verifyFrom(idToken, issuer);
if (payload.nonce !== login.nonce || typeof payload.sub !== "string") {
throw new APIException(401, "Invalid token");
}
const claims = checkClaims(payload, issuer);
if (!claims) {
throw new APIException(502, "The instance sent an invalid identity");
}
const subject = payload.sub;
const identity = db.prepare(
`SELECT user_id FROM user_identities WHERE issuer = ? AND subject = ?;`,
).get(issuer, subject) as { user_id: string } | undefined;
if (getPeerStatus(issuer) === "blocked") {
throw new APIException(403, "This instance is blocked here");
}
if (identity) {
return await applyClaims(identity.user_id, issuer, claims);
}
const trusted = getPeerStatus(issuer) === "trusted";
if (!trusted && !login.invite_token) {
throw new APIException(
403,
"Users of this instance need an invite to join",
);
}
const inviterId = trusted ? null : claimInvite(login.invite_token!);
let user: User;
try {
user = transaction(() => {
const created = createRemoteUser({
username: `${claims.preferred_username}@${instanceHost(issuer)}`,
email: claims.email,
}, inviterId);
db.prepare(
`INSERT INTO user_identities (user_id, issuer, subject, synced_at) VALUES (?, ?, ?, ?);`,
).run(created.id, issuer, subject, new Date().toISOString());
return created;
});
} catch (err) {
if (!trusted) releaseInvite(login.invite_token!);
throw err;
}
sendWelcomeEmail(user);
return await applyClaims(user.id, issuer, claims);
}
/**
* Brings the account `userId`, from the instance at `issuer`, in step with
* `claims` and returns it: a new name at home renames it, the old handle
* staying its alias, and a new email or avatar replaces its own. A failed
* avatar fetch is logged, and tried again at the next change or sign-in.
*
* @throws {APIException} 409 when the new handle is someone else's.
*/
async function applyClaims(
userId: string,
issuer: string,
claims: IdentityClaims,
): Promise<User> {
let user = getUserById(userId);
const handle = `${claims.preferred_username}@${instanceHost(issuer)}`;
if (handle !== user.username) {
user = setUsername(userId, handle);
updateClientUsername(userId, handle);
broadcastUserUpdated(toPublicUser(user));
}
if (claims.email !== user.email) {
user = await updateUser(userId, { email: claims.email });
}
const { picture } = db.prepare(
`SELECT picture FROM user_identities WHERE user_id = ?;`,
).get(userId) as { picture: string | null };
let synced = claims.picture ?? null;
if (claims.picture && claims.picture !== picture) {
try {
user = await setUserAvatar(userId, await fetchAvatar(claims.picture));
} catch (err) {
console.error(`[federation] ${handle}'s avatar failed:`, err);
synced = picture;
}
}
db.prepare(
`UPDATE user_identities SET picture = ?, synced_at = ? WHERE user_id = ?;`,
).run(synced, new Date().toISOString(), userId);
return user;
}
/**
* Applies the signed event `token` from a user's home, for
* `POST /api/federation/events`: `update` brings their account in step,
* `revoke` ends their sessions here. Only an instance some account here comes
* from, and not blocked, may send one, about one of those accounts, and at
* most {@link EVENT_LIMIT}'s rate. The sender gives up on an event at a 4xx
* other than 429, and tries it again later otherwise.
*
* @throws {APIException} 400 when it isn't an event of a known instance about
* one of its users; 401 when it doesn't verify; 403 when the instance is
* blocked; 429 when it sends too many; 502 when its keys can't be fetched.
*/
export async function receiveEvent(token: string): Promise<void> {
let issuer: unknown;
try {
issuer = decodeJwt(token).iss;
} catch {
throw new APIException(400, "Invalid event");
}
if (
typeof issuer !== "string" ||
!db.prepare(`SELECT 1 FROM user_identities WHERE issuer = ? LIMIT 1;`)
.get(issuer)
) {
throw new APIException(400, "Unknown instance");
}
if (getPeerStatus(issuer) === "blocked") {
throw new APIException(403, "This instance is blocked here");
}
if (!takeToken(`federation-events:${issuer}`, EVENT_LIMIT)) {
throw new APIException(429, "Too many events");
}
const payload = await verifyFrom(token, issuer);
const identity = db.prepare(
`SELECT user_id FROM user_identities WHERE issuer = ? AND subject = ?;`,
).get(issuer, String(payload.sub)) as { user_id: string } | undefined;
if (!identity || !isFederationEvent(payload)) {
throw new APIException(400, "Invalid event");
}
if (payload.type === "revoke") {
revokeSessions(identity.user_id, new Date(), "Signed out by home");
return;
}
const claims = checkClaims(payload.claims, issuer);
if (!claims) throw new APIException(400, "Invalid event");
await applyClaims(identity.user_id, issuer, claims);
}
/**
* Returns the URL of `path` on the home of `user`, an account from another
* instance, for `GET /api/users/:key/home`: their profile there by default,
* or another of its pages, such as `/settings`, where they change their
* password. A handle's host is its home's issuer, the API's origin, which
* needn't be its pages': those are where its `/authorize` page is, and the
* home's metadata says. While it doesn't answer, the issuer stands in.
*
* @throws {APIException} 400 when `path` isn't a path; 404 when `user` is
* local.
*/
export async function homePageUrl(user: User, path?: string): Promise<string> {
const identity = db.prepare(
`SELECT issuer FROM user_identities WHERE user_id = ?;`,
).get(user.id) as { issuer: string } | undefined;
if (!identity) throw new APIException(404, "This account lives here");
if (path !== undefined && (!path.startsWith("/") || path.startsWith("//"))) {
throw new APIException(400, "Not a path");
}
let pages = identity.issuer;
try {
const { origin, protocol } = new URL(
(await fetchMetadata(identity.issuer)).authorization_endpoint,
);
if (protocol === "https:" || protocol === "http:") pages = origin;
} catch (err) {
console.error(`[federation] ${identity.issuer} unreachable:`, err);
}
return `${pages}${path ?? userPath(splitHandle(user.username).name)}`;
}

22
api/services/follow-service.ts
View File

@@ -24,7 +24,7 @@ import {
import { isFollowRow } from "../model/follow.ts";
import { isPlaylistRow, playlistRowToApi } from "../model/playlist.ts";
import { type Page, pageOf } from "../lib/pagination.ts";
import { attachCategoryIds } from "./category-service.ts";
import { attachCategoryIds, categoryJoin } from "./category-service.ts";
import { sendToUser } from "./ws-service.ts";
/**
@@ -127,7 +127,7 @@ export function unfollowPlaylist(followerId: string, playlistId: string): void {
/**
* Returns the ids of every user and playlist `followerId` follows, private
* playlists included, for `GET /api/follows/status`.
* playlists included, for `GET /api/me/follow`.
*
* @throws {APIException} 500 when a row doesn't parse.
*/
@@ -167,10 +167,7 @@ export function getFollowedUsersDumpFeed(
): Page<Dump> {
const offset = (page - 1) * limit;
const catJoin = categoryId
? "INNER JOIN dump_categories dc ON dc.dump_id = d.id AND dc.category_id = ?"
: "";
const catParams = categoryId ? [categoryId] : [];
const [catJoin, catParams] = categoryJoin("dump", "d", categoryId);
const rawRows = db.prepare(
`SELECT ${SELECT_COLS_ALIASED}
@@ -188,7 +185,7 @@ export function getFollowedUsersDumpFeed(
throw new APIException(500, "Malformed dump data");
}
const items = userFeedRows.map(dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}
@@ -208,10 +205,7 @@ export function getFollowedPlaylistsDumpFeed(
): Page<Dump> {
const offset = (page - 1) * limit;
const catJoin = categoryId
? "INNER JOIN dump_categories dc ON dc.dump_id = d.id AND dc.category_id = ?"
: "";
const catParams = categoryId ? [categoryId] : [];
const [catJoin, catParams] = categoryJoin("dump", "d", categoryId);
const rawRows = db.prepare(
`SELECT ${SELECT_COLS_ALIASED}
@@ -232,7 +226,7 @@ export function getFollowedPlaylistsDumpFeed(
throw new APIException(500, "Malformed dump data");
}
const items = rawRows.map(dumpRowToApi);
attachCategoryIds(items);
attachCategoryIds("dump", items);
return pageOf(items, limit);
}
@@ -264,7 +258,9 @@ export function getFollowedPlaylistsByUser(
if (!rawRows.every(isPlaylistRow)) {
throw new APIException(500, "Malformed playlist data");
}
return pageOf(rawRows.map(playlistRowToApi), limit);
const items = rawRows.map(playlistRowToApi);
attachCategoryIds("playlist", items);
return pageOf(items, limit);
}
/**

514
api/services/identity-provider-service.ts Normal file
View File

@@ -0,0 +1,514 @@
/**
* Plays the home instance in federated sign-in: it lets this instance's users
* sign in to other gerbeur instances with their account here.
*
* The flow is OpenID Connect's authorization code with PKCE, in three steps.
*
* 1. Another instance, the client, sends the user to the `/authorize` page
* with a request. The page shows who asks ({@link describeClient}) and,
* once the user consents, gets a code ({@link createAuthorizationCode}),
* which it carries back to the client's callback page.
* 2. The client trades the code for an identity token, server to server
* ({@link exchangeCode}): this user's name, email and avatar, signed with
* the instance key.
* 3. While the user's grant lasts, the client hears of their changes: a new
* name, email or avatar, and the end of their sessions after a password
* change or reset. {@link notifyClients} queues the signed events in
* `federation_outbox`, and {@link outboxJob} delivers them.
*
* Only a local account signs in elsewhere: a user who came from another
* instance signs in from there. `routes/oauth.ts`, `routes/well-known.ts` and
* the routes that change a user call it.
*
* @module
*/
import { createHash, randomBytes } from "node:crypto";
import { APIException } from "../lib/api-exception.ts";
import { db, transaction } from "../db/connection.ts";
import {
AVATARS_DIR,
FEDERATION,
FRONTEND_URL,
OG_SITE_NAME,
PUBLIC_URL,
} from "../config.ts";
import { signInstanceToken } from "../lib/jwt.ts";
import type { Job } from "../lib/scheduler.ts";
import {
type AuthorizeRequest,
FEDERATION_VERSION,
type FederationEvent,
type FederationGrant,
type IdentityClaims,
instanceHost,
type InstanceMetadata,
type OAuthClient,
parseInstance,
} from "../../shared/federation.ts";
import { fetchMetadata, postEvent } from "../remote/gerbeur-instance.ts";
import { isRemoteUser, type User } from "../model/user.ts";
import { getSigningKey } from "./instance-key-service.ts";
import { getUserById } from "./user-service.ts";
import { getPeerStatus } from "./federation-peer-service.ts";
/**
* The instance's name in every token it signs and every token addressed to
* it: its public URL, which must be an origin for other instances to find
* its metadata.
*/
export const ISSUER: string = PUBLIC_URL;
/**
* The client's page a home instance sends the user back to, on the frontend.
* Its metadata announces it, and a home accepts no other.
*/
export const REDIRECT_URI = `${FRONTEND_URL}/federation/callback`;
/** How long a code may wait to be traded, from the consent. */
const CODE_TTL_MS = 60_000;
/**
* How long an identity token is valid: enough to cross from one server to the
* other, never enough to be replayed later.
*/
const ID_TOKEN_TTL = "5m";
/**
* How long an event is valid, and so how long the outbox keeps trying to
* deliver it before giving up.
*/
const EVENT_TTL_MS = 3 * 86_400_000;
/** The longest wait between two tries of an event. */
const MAX_RETRY_MS = 6 * 3_600_000;
/**
* Returns the instance's metadata, served at
* `/.well-known/openid-configuration`.
*/
export function getInstanceMetadata(): InstanceMetadata {
return {
issuer: ISSUER,
authorization_endpoint: `${FRONTEND_URL}/authorize`,
token_endpoint: `${PUBLIC_URL}/api/oauth/tokens`,
jwks_uri: `${PUBLIC_URL}/.well-known/jwks.json`,
response_types_supported: ["code"],
grant_types_supported: ["authorization_code"],
subject_types_supported: ["public"],
id_token_signing_alg_values_supported: ["EdDSA"],
code_challenge_methods_supported: ["S256"],
scopes_supported: ["openid", "profile", "email"],
gerbeur_federation: FEDERATION_VERSION,
gerbeur_site_name: OG_SITE_NAME,
gerbeur_redirect_uri: REDIRECT_URI,
gerbeur_events_endpoint: `${PUBLIC_URL}/api/federation/events`,
gerbeur_outgoing: FEDERATION.outgoing,
};
}
/** Returns the SHA-256 of `value`, as base64url. */
function sha256(value: string): string {
return createHash("sha256").update(value).digest("base64url");
}
/**
* Returns the version of `user`'s avatar: when its file was last written,
* which only a new avatar does, unlike `updatedAt`, which every profile edit
* bumps.
*/
function avatarVersion(user: User): number {
try {
return Deno.statSync(`${AVATARS_DIR}/${user.id}`).mtime?.getTime() ?? 0;
} catch {
return user.updatedAt?.getTime() ?? 0;
}
}
/**
* Returns what this instance tells another of `user`: their name, email and
* avatar, whose URL changes with each new one, so that a client fetches it
* again only then.
*/
export function identityClaims(user: User): IdentityClaims {
return {
preferred_username: user.username,
email: user.email,
...(user.avatarMime && {
picture: `${PUBLIC_URL}/api/users/${user.id}/avatar?v=${
avatarVersion(user)
}`,
}),
};
}
/**
* Returns the client `clientId` names, checked: an instance other than this
* one, not blocked here, answering as a gerbeur. `user` must have a local
* account, which alone may sign in elsewhere, and this instance must let its
* users sign in elsewhere (`FEDERATION.outgoing`).
*
* @throws {APIException} 400 when `clientId` isn't an instance's address, is
* this instance, or `user` comes from another instance; 403 when this
* instance's users don't sign in elsewhere, or the client is blocked here;
* 502 when it doesn't answer as a gerbeur.
*/
async function checkClient(
user: User,
clientId: string,
): Promise<{ clientId: string; metadata: InstanceMetadata }> {
if (!FEDERATION.outgoing) {
throw new APIException(
403,
"Accounts on this instance don't sign in to other instances",
);
}
if (isRemoteUser(user)) {
throw new APIException(
400,
"This account signs in from the instance it comes from",
);
}
if (parseInstance(clientId) !== clientId || clientId === ISSUER) {
throw new APIException(400, "Invalid client");
}
if (getPeerStatus(clientId) === "blocked") {
throw new APIException(403, "This instance is blocked here");
}
try {
return { clientId, metadata: await fetchMetadata(clientId) };
} catch (err) {
console.error(`[federation] client ${clientId} unreachable:`, err);
throw new APIException(502, "That instance doesn't answer as a gerbeur");
}
}
/**
* Returns the instance `clientId` names, for the `/authorize` page's question
* to user `userId`, and whether they already let it sign them in.
*
* @throws {APIException} As {@link checkClient}.
*/
export async function describeClient(
userId: string,
clientId: string,
): Promise<OAuthClient> {
const { metadata } = await checkClient(getUserById(userId), clientId);
const granted = !!db.prepare(
`SELECT 1 FROM federation_grants WHERE user_id = ? AND client_id = ?;`,
).get(userId, clientId);
return {
clientId,
host: instanceHost(clientId),
name: metadata.gerbeur_site_name,
granted,
};
}
/**
* Records user `userId`'s consent to sign in to the client of `request`, and
* returns where to send them: the client's callback page with a code, which
* the client trades within a minute ({@link exchangeCode}). The grant lasts
* until they withdraw it, so the page asks no more.
*
* @throws {APIException} 400 when the request is malformed or its
* `redirectUri` isn't the client's own; otherwise as {@link checkClient}.
*/
export async function createAuthorizationCode(
userId: string,
request: AuthorizeRequest,
): Promise<string> {
const user = getUserById(userId);
const { clientId, metadata } = await checkClient(user, request.clientId);
if (request.redirectUri !== metadata.gerbeur_redirect_uri) {
throw new APIException(400, "Invalid redirect URI");
}
if (
request.codeChallengeMethod !== "S256" ||
!/^[A-Za-z0-9_-]{43}$/.test(request.codeChallenge) ||
!request.state || request.state.length > 512 ||
!request.nonce || request.nonce.length > 512
) {
throw new APIException(400, "Invalid request");
}
const code = randomBytes(32).toString("base64url");
const now = new Date();
transaction(() => {
db.prepare(`DELETE FROM oauth_codes WHERE expires_at < ?;`).run(
now.toISOString(),
);
db.prepare(
`INSERT INTO oauth_codes (code_hash, user_id, client_id, redirect_uri, code_challenge, nonce, expires_at)
VALUES (?, ?, ?, ?, ?, ?, ?);`,
).run(
sha256(code),
userId,
clientId,
request.redirectUri,
request.codeChallenge,
request.nonce,
new Date(now.getTime() + CODE_TTL_MS).toISOString(),
);
db.prepare(
`INSERT INTO federation_grants (user_id, client_id, client_name, granted_at, last_used_at)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT (user_id, client_id) DO UPDATE SET client_name = excluded.client_name;`,
).run(
userId,
clientId,
metadata.gerbeur_site_name,
now.toISOString(),
now.toISOString(),
);
});
const redirect = new URL(request.redirectUri);
redirect.searchParams.set("code", code);
redirect.searchParams.set("state", request.state);
// RFC 9207: tells the client which instance answered, against mix-ups.
redirect.searchParams.set("iss", ISSUER);
return redirect.href;
}
/**
* Trades a code for an identity token, for `POST /api/oauth/tokens`, which
* the client calls server to server with the form `params`. A code works once,
* within a minute, for the client and redirect it was made for, and with the
* verifier whose hash the request carried (PKCE). The token, signed with the
* instance key and addressed to the client, carries {@link identityClaims}
* and the request's nonce.
*
* @throws {APIException} 400 when anything doesn't match.
*/
export async function exchangeCode(
params: URLSearchParams,
): Promise<{ id_token: string; token_type: string }> {
const code = params.get("code");
const clientId = params.get("client_id");
const verifier = params.get("code_verifier");
if (
params.get("grant_type") !== "authorization_code" || !code || !clientId ||
!verifier || !/^[A-Za-z0-9._~-]{43,128}$/.test(verifier)
) {
throw new APIException(400, "invalid_request");
}
// Taken in the same statement that finds it, so it can't be traded twice.
const row = db.prepare(
`DELETE FROM oauth_codes WHERE code_hash = ?
RETURNING user_id, client_id, redirect_uri, code_challenge, nonce, expires_at;`,
).get(sha256(code)) as
| {
user_id: string;
client_id: string;
redirect_uri: string;
code_challenge: string;
nonce: string;
expires_at: string;
}
| undefined;
if (
!row || row.expires_at < new Date().toISOString() ||
row.client_id !== clientId ||
row.redirect_uri !== params.get("redirect_uri") ||
row.code_challenge !== sha256(verifier)
) {
throw new APIException(400, "invalid_grant");
}
const user = getUserById(row.user_id);
if (isRemoteUser(user)) throw new APIException(400, "invalid_grant");
db.prepare(
`UPDATE federation_grants SET last_used_at = ? WHERE user_id = ? AND client_id = ?;`,
).run(new Date().toISOString(), user.id, clientId);
const idToken = await signInstanceToken(
{ ...identityClaims(user), nonce: row.nonce },
await getSigningKey(),
{ issuer: ISSUER, audience: clientId, subject: user.id },
ID_TOKEN_TTL,
);
return { id_token: idToken, token_type: "Bearer" };
}
interface GrantRow {
client_id: string;
client_name: string | null;
granted_at: string;
last_used_at: string;
}
/**
* Returns the instances user `userId` let their account sign in to, the most
* recently used first, for the settings page.
*/
export function listGrants(userId: string): FederationGrant[] {
const rows = db.prepare(
`SELECT client_id, client_name, granted_at, last_used_at FROM federation_grants
WHERE user_id = ? ORDER BY last_used_at DESC;`,
).all(userId) as unknown as GrantRow[];
return rows.map((r) => ({
clientId: r.client_id,
host: instanceHost(r.client_id),
name: r.client_name,
grantedAt: new Date(r.granted_at),
lastUsedAt: new Date(r.last_used_at),
}));
}
/**
* Withdraws user `userId`'s grant to the instance whose host is `host`: it
* stops answering for them there, and is told to end their sessions.
*
* @throws {APIException} 404 when they gave that instance no grant.
*/
export async function revokeGrant(userId: string, host: string): Promise<void> {
const grant = listGrants(userId).find((g) => g.host === host.toLowerCase());
if (!grant) throw new APIException(404, "Grant not found");
await notifyClients(userId, { type: "revoke" }, [grant.clientId]);
db.prepare(
`DELETE FROM federation_grants WHERE user_id = ? AND client_id = ?;`,
).run(userId, grant.clientId);
}
/**
* Withdraws every grant, as each user would theirs, and drops the codes not
* yet traded, so that no account here stays signed in elsewhere: `main.ts`
* calls it at startup when `FEDERATION.outgoing` is off. Returns how many
* grants it withdrew.
*/
export async function withdrawAllGrants(): Promise<number> {
db.prepare(`DELETE FROM oauth_codes;`).run();
const grants = db.prepare(
`SELECT user_id, client_id FROM federation_grants;`,
).all() as { user_id: string; client_id: string }[];
for (const { user_id, client_id } of grants) {
await notifyClients(user_id, { type: "revoke" }, [client_id]);
}
db.prepare(`DELETE FROM federation_grants;`).run();
return grants.length;
}
/**
* Tells each instance user `userId` signed in to of `event`, or only those of
* `clientIds`: `update` when their name, email or avatar changed, `revoke`
* when their sessions there must end. The routes that change a user call it,
* and for an `update`, `event` may be just the type: the claims are read now.
* The events are queued in the outbox and their delivery starts at once,
* after those queued before for the same instance ({@link deliverTo}).
*/
export async function notifyClients(
userId: string,
event: FederationEvent | { type: "update" },
clientIds?: string[],
): Promise<void> {
const clients = clientIds ?? listGrants(userId).map((g) => g.clientId);
if (clients.length === 0) return;
const payload: FederationEvent = event.type === "update"
? { type: "update", claims: identityClaims(getUserById(userId)) }
: event;
const key = await getSigningKey();
const now = new Date();
for (const clientId of clients) {
const token = await signInstanceToken(
{ ...payload },
key,
{ issuer: ISSUER, audience: clientId, subject: userId },
`${EVENT_TTL_MS / 1000}s`,
);
db.prepare(
`INSERT INTO federation_outbox (id, client_id, event, next_attempt_at, created_at)
VALUES (?, ?, ?, ?, ?);`,
).run(
crypto.randomUUID(),
clientId,
token,
now.toISOString(),
now.toISOString(),
);
}
for (const clientId of clients) deliverTo(clientId);
}
/**
* The instances whose events {@link deliverTo} is sending, so that two runs
* don't send the same events, and those whose events came meanwhile, which
* the running one sends too once done.
*/
const delivering = new Set<string>();
const queuedMeanwhile = new Set<string>();
/**
* Sends the outbox's events to each instance that has some, every instance at
* once so that one that doesn't answer delays none of the others.
*/
export async function deliverOutbox(): Promise<void> {
db.prepare(`DELETE FROM federation_outbox WHERE created_at < ?;`).run(
new Date(Date.now() - EVENT_TTL_MS).toISOString(),
);
const clients = db.prepare(
`SELECT DISTINCT client_id FROM federation_outbox;`,
).all() as { client_id: string }[];
await Promise.all(clients.map((c) => deliverTo(c.client_id)));
}
/**
* Sends the events for the instance `clientId` in the order they were made,
* as long as it takes them, since each `update` carries all the claims and a
* late older one would undo a newer one. One it took, or refused for good,
* leaves the outbox. The first that fails stops the run, and waits twice as
* long as the time before, up to 6 hours, with the events behind it, until
* it expires after 3 days.
*/
async function deliverTo(clientId: string): Promise<void> {
if (delivering.has(clientId)) {
queuedMeanwhile.add(clientId);
return;
}
delivering.add(clientId);
try {
do {
queuedMeanwhile.delete(clientId);
await sendDue(clientId);
} while (queuedMeanwhile.has(clientId));
} catch (err) {
console.error(`[federation] delivery to ${clientId} failed:`, err);
} finally {
delivering.delete(clientId);
}
}
/** Runs one pass of {@link deliverTo}. */
async function sendDue(clientId: string): Promise<void> {
for (;;) {
const row = db.prepare(
`SELECT id, event, attempts, next_attempt_at FROM federation_outbox
WHERE client_id = ? ORDER BY created_at, rowid LIMIT 1;`,
).get(clientId) as
| { id: string; event: string; attempts: number; next_attempt_at: string }
| undefined;
if (!row || row.next_attempt_at > new Date().toISOString()) return;
try {
if (!await postEvent(clientId, row.event)) {
console.error(`[federation] ${clientId} refused an event`);
}
db.prepare(`DELETE FROM federation_outbox WHERE id = ?;`).run(row.id);
} catch (err) {
const wait = Math.min(60_000 * 2 ** row.attempts, MAX_RETRY_MS);
db.prepare(
`UPDATE federation_outbox SET attempts = attempts + 1, next_attempt_at = ? WHERE id = ?;`,
).run(new Date(Date.now() + wait).toISOString(), row.id);
console.error(`[federation] ${clientId} unreachable:`, err);
return;
}
}
}
/** Retries the outbox's failed events every minute. */
export const outboxJob: Job = {
name: "federation",
everyMs: 60_000,
run: deliverOutbox,
};

73
api/services/instance-key-service.ts Normal file
View File

@@ -0,0 +1,73 @@
/**
* Keeps the instance's Ed25519 key, which signs what it tells other instances
* in federated sign-in: the identity tokens of `identity-provider-service.ts`
* and the events of its outbox. The key is made the first time it's needed
* and stored in `federation_keys`, so it survives restarts; other instances
* fetch its public half from `GET /.well-known/jwks.json`.
*
* @module
*/
import type { JSONWebKeySet, JWK } from "@panva/jose";
import { db } from "../db/connection.ts";
import {
generateInstanceKey,
importInstanceKey,
type InstanceSigningKey,
} from "../lib/jwt.ts";
/**
* Holds the key once loaded. It is a promise so that two first requests in a
* row don't each make a key.
*/
let current: Promise<InstanceSigningKey> | null = null;
/**
* Returns the key the instance signs with now: the newest stored, or a new
* one, stored, when there is none.
*/
export function getSigningKey(): Promise<InstanceSigningKey> {
current ??= loadOrCreate().catch((err) => {
current = null;
throw err;
});
return current;
}
/** Reads the newest stored key, or makes and stores the first one. */
async function loadOrCreate(): Promise<InstanceSigningKey> {
const row = db.prepare(
`SELECT kid, private_jwk FROM federation_keys ORDER BY created_at DESC LIMIT 1;`,
).get() as { kid: string; private_jwk: string } | undefined;
if (row) {
return await importInstanceKey(row.kid, JSON.parse(row.private_jwk));
}
const { kid, privateJwk, publicJwk } = await generateInstanceKey();
db.prepare(
`INSERT INTO federation_keys (kid, private_jwk, public_jwk, created_at) VALUES (?, ?, ?, ?);`,
).run(
kid,
JSON.stringify(privateJwk),
JSON.stringify(publicJwk),
new Date().toISOString(),
);
return await importInstanceKey(kid, privateJwk);
}
/**
* Returns the public halves of every stored key, for
* `GET /.well-known/jwks.json`. It makes the first key when there is none, so
* that the set is never empty.
*/
export async function getPublicKeys(): Promise<JSONWebKeySet> {
await getSigningKey();
const rows = db.prepare(
`SELECT public_jwk FROM federation_keys ORDER BY created_at DESC;`,
).all() as { public_jwk: string }[];
return { keys: rows.map((r) => JSON.parse(r.public_jwk) as JWK) };
}
/** Forgets the loaded key, for the tests, which wipe the table between cases. */
export function resetSigningKey(): void {
current = null;
}

Some files were not shown because too many files have changed in this diff Show More