vincent/gerbeur
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vincent:563daa720cd5f31d91c3037e297c464978139ffe
Branches Tags
vincent:main vincent:gh-pages
..
compare: vincent:main
Branches Tags
vincent:gh-pages vincent:main

91 Commits
563daa720c ... main

Author SHA1 Message Date
khannurien
eec73e9749 v4: a comment and a chat message are edited in the same field as every text edited in place — each had an editor of its own: a comment's was its reply form, which showed a refused edit as a card and left the focus nowhere once closed, and a chat message's a box of its own with its own buttons and rules. InlineEdit gives its field, InlineEditField, to the components that open it with an action of their own, as these two do with their Edit and ✎: they keep those, a click on someone's message being no request to edit it. The field opens on the text with the caret at its end and as tall as the text, saves on Ctrl+Enter (and on Enter in the chat, where Shift+Enter breaks a line, as in its composer), leaves on Escape, sends nothing for a text left as it was, refuses an empty one, says why a save was refused in one line, and gives the focus back to the button that opened it, unless the caller has put it elsewhere, as the chat does to its composer. CommentForm only posts a comment or a reply now; ChatEditBox keeps only what is the chat's, scrolling itself into the list's view.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 3m20s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 44s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 18:16:43 +00:00
khannurien
3e03626cc1 v4: text is edited where it stands through one component, InlineEdit, and a dump's note and a playlist's title and description join its title, a profile's description and email — those three were written apart, each with its own state, keys, buttons, error and CSS, and the note and the playlist could only be changed in their full editors. InlineEdit (components/form) shows the text; a click on it, or the ✎ after it, opens a field holding it, with Save and Cancel under it. A one-line field takes the text's own type in every style, from the class it is handed (dump-title, playlist-detail-title), and sits on the faint plate the text showed under the pointer, so that nothing moves but the buttons that come; a Markdown text opens the editor comments use. Enter saves a line, Ctrl+Enter a Markdown text, Escape backs out, a text left as it was closes without a request, a required one can't be emptied, a refused save says why in one line beside the buttons, and the focus goes back to the ✎. A click on a link in the text, or one that ends a selection, follows the link or keeps the selection. The ✎ is now a button with the action's name, where a div pretended to be one, and the text keeps its own role, a heading or Markdown. The plate is drawn under the text by a pseudo-element, tinted from its colour: the surface colour it had was the card's own, and showed nothing on the dump page. On a playlist's page the title edits in place beside Follow and Edit, which go to the row's end, and the description too; an empty note or description says what goes there to whoever may write it. DescriptionEditor and EmailEditor go. 
The GeoCities marquee stops only for a link the keyboard focused: a link pressed with the pointer took the focus as well, which put the line back at its start and scrolled it, so the link slipped from under the pointer before the button came up and the click was lost. The next page's marquee also carries on from the very point the last one had reached, with the loop it showed: it worked its place out from when the loop began, which ran on while the crawl was paused under the pointer, and showed the feed as it was by then, of another width.

The chat's message field rests on one line, as tall as the Send button: it rested on two, 54 to 62 pixels against the button's 38, and the button sat low beside it. Both take --chat-composer-height.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 18:08:53 +00:00
khannurien
b92c674726 v4: the devcontainer has Playwright's browsers — they were installed by hand in the home directory, which the rebuild onto Node 26 left without them, and the screenshots of the dev instances stopped working. The Dockerfile installs Firefox and the Chromium headless shell of npm:playwright@1.63.0, with the system libraries they need, in /opt/ms-playwright (PLAYWRIGHT_BROWSERS_PATH), which the workspace and the instances' services share as part of the image. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-20 18:08:53 +00:00
khannurien
7d0a0e7534 v4: the devcontainer runs Deno 2.9.7 on Node 26 — its image took Deno 2.9.0 and the Node 20 base. The Dockerfile now copies the deno binary from denoland/deno:bin-2.9.7 onto mcr.microsoft.com/devcontainers/typescript-node:26.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 54s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-13 13:38:16 +00:00
khannurien
51439f475d v4: the GeoCities marquee scrolls the community feed, what the site's members are doing — it crawled three canned lines, a welcome, "best viewed in Netscape Navigator" and "under construction", which said nothing of the site. The feed is a list any style may show (CommunityFeed.tsx), which GeoCities lays out along its marquee: who is online, the week's dumps, upvotes, comments and chat messages, linking to /community, its most upvoted dump, then the 12 latest things done in public, newest first, each ending on how long ago it was: a public dump posted, a public playlist created, a comment or an upvote on a public dump, someone joining and who invited them. Every name and title links to its page. A dump has one comment line and one upvote line at most, its latest, so that a thread or a popular dump doesn't fill the feed. GET /api/community/feed serves it, signed in only (community-service.ts): the events are read from the tables on each request, the week's figures are those /community already keeps, as of its last refresh. A signed-out visitor has no marquee. CommunityFeedProvider holds the feed around every page, so that it outlives the header each page renders anew, fetches it only while a component shows it, and fetches it again two seconds after the socket last told of a dump, a comment, a vote, a playlist, a rename, a deletion or a reconnection: the socket's messages don't carry a feed's lines, and the server's query stays the one place that decides what the feed holds. Someone joining comes with no message, and shows at the next of those. The crawl goes at 50 pixels a second however much the feed holds, Marquee.tsx measuring one copy for the stylesheet to work the loop's length out from, where a fixed 36 seconds would have sped up with every line. A loop shows what the feed held when it began: text that changed mid-crawl would jump, so what came meanwhile waits for the next loop, which also brings the times up to date. The next page's marquee carries on from where the last one's had got to, rather than from the start. The marquee was hidden from assistive technology as decoration; its first copy is now a labelled list of links, the second, which the seamless loop needs, hidden and out of the tab order. A link that takes the focus stops the crawl and the line scrolls to it, the pointer pauses it as before, and with reduced motion the line shows once, still, follows the feed as it changes and scrolls by hand. The sentences a notification also says, "posted", "upvoted", "commented on", share its translations. The tests' DOM gets an AnimationEvent before React loads (src/test/before-react.ts): finding none, React listened for the animation events under their WebKit names, and a test's "animationiteration" reached no handler.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 51s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 07:38:44 +00:00
khannurien
62e51f1622 v4: the six styles are reworked, each around one idea of its own, and the tab row stops painting the page — a pass over every theme, then the faults it showed in what they share. Brutalist draws every box in ink with a solid offset shadow and sets the palette's colors as flat blocks (a teal tag, an apricot quote, an inked label); its dark scheme keeps that ink darker than the page, on Twilight Indigo, where eggshell lines glared across every frame. GeoCities goes to the desktop of its day: the header is the taskbar, the site's name its Start button, the controls grey bevels, and a marquee crawls under the header (Marquee.tsx, drawn in that style alone), with Comic Neue where the system has no Comic Sans. Melange leaves its browns and Fraunces for a sorbet blend in Bricolage Grotesque, run along the header, through the site's name, where it drifts, and between a journal's blocks; its two schemes are one print and its negative, trading ink and paper while the pink and the plum written on it stay, and they share one fine grain where the light one had a coarse tile that greyed the page. Kanagawa opens the comments and the related dumps on a status line and reads on Lotus's brightest paper; its page is flat. Times draws its heavy rules in a mid grey at night, where ink is near white, and sets a journal cell with no picture as text. In what the styles share: a row of tabs (TabBar) hid the tabs scrolled under its scope switcher, and faded its ends, by painting the page's color over them, which showed as a patch on any page that is a gradient or a picture. The switcher now sits before the row that scrolls, which clips the tabs at its edge, and the fades are a mask (--tab-fade-start and --tab-fade-end); useRoomBeside measures the row through that one more level. A page's title keeps a word whole unless it alone is wider than the line: Firefox sizes a title set with negative letter-spacing a hair narrower than it lays it out, which dropped its last letter to a second line in any style that tightened it, so the rule is the base's and the two styles' copies go. A card holding one paragraph, such as the instances a new account hasn't signed in to on /settings, is spaced by its padding and not by the paragraph's margins as well. The site's name answers the pointer as a card's title does in each style, where two of them left it still. The emails follow the styles' tokens, Melange's blend among them.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 55s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 19:30:57 +00:00
khannurien
eb4e64683c v4: the Times nameplate sits level with the header's buttons — the name was 7 pixels under their middle on a wide screen and 3 on a narrow one, with twice the paper above its letters as below. The blackletter sets its letters low in its line, an ascender stopping a fifth of an em under the line's top and a descender ending on its bottom, and the nameplate's padding was the heavier above besides. The name now carries 0.22em under its line, which leaves as much below the letters as above at either size, and the nameplate's padding is the same on both sides, 0.6rem and 0.7rem from 860 pixels up. The wide masthead's row is a pixel taller, the narrow header keeps its 60.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 53s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 16:30:02 +00:00
khannurien
d54ae554a4 v4: the Times style gets a masthead and a face for each job — one condensed serif did everything, buttons and tabs included, under a plain header bar. The site's name is now a blackletter nameplate (Manufacturing Consent), its emoji beside it on the same baseline at the size of a printer's mark. From 860 pixels up it takes the middle of a row of its own, the buttons to its right, and what a page puts in the header, a feed's tabs or the open search field, runs underneath between a hairline and a double rule; a page that puts nothing there keeps one row and a single ink rule. Instrument Serif stays for headlines, at its one weight and larger sizes, Newsreader sets what is read at length (summaries, notes, comments), and Libre Franklin everything the interface says: controls, tabs, datelines, bylines, labels (--font-ui, listed in themes/nyt.css). Tabs are sentence case with an ink underline on the open one. The comments, the related dumps, a profile's sections and a page's groups open on an ink rule with their label in bold capitals under it. A dump's categories are small capital kickers, where they were grey chips. Buttons and fields take a 3px radius, pictures, cards and sheets staying square. The header sits on the page's color in the dark scheme too. In the journal each cell draws its four rules as a 1px shadow in a 1px gap, where it drew its top and left edges: a last row the dumps don't fill was left open on its right, and the row above it unruled over the empty tracks. A page's title keeps a word whole unless it is wider than the line. deno.lock also takes the key format of the Deno that wrote it, with no version changed beyond the three font packages.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 53s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-12 09:26:09 +00:00
khannurien
3e15e9a4f2 v4: /follow/playlists opens the followed playlists' feed, and a page's test opens it through the app's routes — the path was a 404 since the two views of the followed feed became pages: the route table held each feed tab as an exact path, /follow, so /follow/playlists fell to /:handle/*, which read "follow" as a category and "playlists" as a tab it hasn't. /music/follow/playlists, which goes through that route, worked. The tabs' routes now take what follows them (/${tab}/*), which Feed already reads and refuses when it names no view.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 57s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 28s
Details 
No test saw it because each page's test mounted the page on a route pattern of its own, a copy of App's, with stand-in pages where a redirect lands: Index's parsed the path itself, and passed on one the app doesn't serve. renderPage (src/test/render.tsx) now opens a URL through AppRoutes, which App.tsx exports for it, inside the real providers, and every page's test uses it: the page is the one the app routes that URL to, behind its guard, and a redirect is checked by the path it lands on. An entry may be a location, to arrive with the state a link hands over. Every page being loaded with its route, the first query after opening one waits. src/test/App.test.tsx tests the table itself: the paths that are a feed, those that are not found, those that send a guest to sign in and back, and those that send a signed-in user home.

The profile's test held /@ann/dumps/x to be a path the profile answers as not found, which only its own pattern made true: in the app it is a dump's page. It now asks for /@ann/dumps/x/y. The tests of the two guards still mount them around a stand-in child, being tests of the guard alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 19:59:52 +00:00
khannurien
cbb770bff8 v4: a still is scaled from its first frame alone, a second image in its file made the copy flash — an avatar came out as a WebP of two frames looping at 40 ms each, the picture then a blank. A still's file may hold more than one image, a phone's JPEG carrying a gain map or a depth map after the photo, and ffmpeg reads each as a frame: only the JPEG and PNG copies were held to one, the WebP keeping them all as it does a GIF's. scalableImage (lib/upload.ts) now says whether the image is an animation, which a GIF alone is, and scaleImage (lib/ffmpeg.ts) writes one frame of anything else, in every format; the image proxy scales through the same call. The copies already made stay, a copy being current while it has its upload's date: uploads/scaled is to be deleted once on an instance that ran the previous version, everything in it being made again on first request.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 52s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 19:40:14 +00:00
khannurien
d2b7d21824 v4: the page a visit lands on asks for its content while its code loads — a first load went one step after the other: the entry script, then the interface's messages, then the page's chunk and its twenty-odd dependencies, and only then the request for the feed. The built page now carries a script in its head that asks for the messages' chunk at once, beside the entry, picking the language as detectLocale does (preloadCatalog in vite.config.ts; the two must agree). And main.tsx sends the landing page's first request as the app boots (utils/shared/earlyFetch.ts): the feed's first page on / and its tabs, the dump on a dump's page, as the stored session. authFetch hands that response to the page that asks for the same URL, once, to the session it was sent as and within ten seconds, in place of a request of its own; past that the page asks again. feedApiUrl (utils/shared/urls.ts) is the feed's URL for both. A category's feed, a profile, a playlist and the other pages wait for their code as before. On a local build with 150 ms of latency added, the feed's request leaves at 0.69 s where it left at 1.70 s, and the first card shows at 1.59 s where it showed at 1.92 s.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 51s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 25s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 19:01:36 +00:00
khannurien
90fc281e90 v4: every image the API serves comes scaled to the size it is shown at — a first visit to the feed loaded 4.8 MB of images out of 5.1 MB in all: an avatar shown at 32 pixels was the upload itself, a 1.4 MB GIF among them, and so were a cover, a custom thumbnail, an attachment and a proxied image, only an image dump having a preview, 960 pixels wide. Every route that answers an image now takes ?w= and sends a WebP copy at most that wide, of the widths shared/image.ts lists, 128, 320, 640 and 960, a width between two taking the one above, and never wider than the image; without it the upload goes out as before, a dump's thumbnail alone being scaled to the widest by default, the custom ones included. serveImage (lib/upload.ts) is where they all end, and the image proxy scales what it fetched the same way, keeping each width in its cache. lib/ffmpeg.ts makes the copies as it made the previews, which they replace: on first request, under uploads/scaled/ at the upload's own path, each given its upload's date and made again when the two differ, so a new avatar or cover needs nobody to delete the old ones. WebP keeps a PNG's transparency and a GIF's animation. A JPEG is turned by its EXIF orientation here, ffmpeg's own rotation of images depending on its version. An animated WebP, which ffmpeg can't read, and an image it fails on go out as they are, a failure being remembered until the upload changes. ?f=compat asks for the copy as a JPEG, or a PNG for what may be transparent: the digest's emails take their thumbnails that way, at 640 pixels, not every mail client reading WebP. On the client, imageProps (utils/shared/urls.ts) gives an image its srcset of those copies and the width it is laid out at, which config/imageSizes.ts states for each place from App.css: Avatar, Thumbnail, which scales an API image and leaves another site's as that site sends it, and the playlists' covers use it. The images of notes and comments still load as uploaded. The copies go with the dump, the custom thumbnail or the attachment they were made from, and generatedMediaSweepJob, a minute after the start then daily, takes those whose upload is gone, the previews under their former name and the temporary files a generation left unfinished. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 19:01:36 +00:00
khannurien
6ab398e687 v4: the API's JSON and the page are sent compressed — a page of the feed went out as it was, twenty kilobytes where a fifth does: only the built frontend's files were compressed, at build time, and Deno's server leaves alone a response whose length Oak has set. compressMiddleware (api/middleware/compress.ts), installed after the error middleware and around everything that sets a body, sends an object or array, and a string whose type its route set, in brotli or gzip as the request's Accept-Encoding takes, a coding refused with q=0 not being taken, from a kilobyte up, the threshold the build compresses from. Brotli is at quality 5, the frontend's files, compressed once, staying at 11. Files, streams and bytes, a response already encoded, a HEAD and one marked no-transform go out as they are, and a compressible answer says Vary: Accept-Encoding whether compressed or not. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 19:01:06 +00:00
khannurien
1bd5c924b0 v4: Dark Reader is kept off the pages, which lost every background under it — on a phone in dark mode with the extension on, the avatar menu, the cards and the tabs were transparent over Dark Reader's own page colour, in every style. The build gives each light-dark() a fallback for the browsers without it, two custom properties of which one is empty, and Dark Reader's rewrite of the stylesheet drops the empty one under a dark system theme: both halves of every colour then apply at once, which is no colour, so nothing paints a background or a border. index.html now carries <meta name="darkreader-lock">, which the extension obeys by generating nothing; the site has dark palettes of its own, following the system or the user's choice. The dev server never showed it, serving light-dark() as written. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 09:26:21 +00:00
khannurien
de223baaa4 v4: the playlists page is reachable at every width, and lists the user's own — below 1150px the header's text link was hidden and nothing replaced it: the avatar menu's "My playlists" led to the user's own, and a guest has no menu. The menu's entry is now "Playlists", to /playlists, at every width, and the page gains a tab after its orders, "Mine", for a signed-in user: their own playlists, private ones too, from GET /api/users/:id/playlists in the default order, narrowed by the category menu and kept live as the others are. It is /playlists?mine=1 (PLAYLISTS_TABS, parsePlaylistsTab, shared/paths.ts), a tab taking the order's place; a guest isn't offered it, and its link sends one to sign in. PlaylistSortTabs takes the tabs it shows, a profile's being the orders alone. The text link shows to a guest at every width, their header giving up its margins under 400px rather than the link, and to a signed-in user where the row has room beside the icons, as measured in every style and both languages: from 560px up to 860px, where the index page's tabs move into the header, and from 1150px as before. A guest in French in the kanagawa style still overflows the header under 340px, by up to 19px.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 51s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 26s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 09:14:55 +00:00
khannurien
561cdf98b0 v4: playlists sort by their last dump, their creation or their size, and /playlists is a grid of covers — every list of playlists came newest first, so one fed every week sank under those made since and left empty. GET /api/playlists and /api/users/:key/playlists take ?sort=updated|new|dumps (PLAYLIST_SORTS, shared/playlist.ts), updated being the default: the last to take a dump first, or its creation while it holds none; a new title, cover or description doesn't bring it back up. A playlist carries that date as lastDumpAt, read with its dumpCount by isMemberVisibleTo's rule (VISIBLE_DUMP_FIGURES, the viewer's id as ?1), so the owner's private dumps date and count their playlists for them alone, in the lists and over the socket: playlist_created, playlist_updated and playlist_dumps_updated carry each socket's own. The /playlists page and a profile's playlists tab show the orders as tabs beside the category menu and keep the one picked in ?sort=, the default being the bare path; a tab takes the current history entry's place, and picking a category keeps the order. Live, usePlaylistListSync takes the list's order: a new playlist goes where it puts it, one that rises moves up to its place, as in the updated order when it takes a dump, and one that falls stays until the list is fetched again, its place being perhaps on a page not loaded. The lists in an order of their own, the followed playlists and the search's, pass none, and take no new playlist as before (noNewEntries goes). comparePlaylists is the order on the client, which a test holds to the API's. 
/playlists lays its cards out as tiles, the cover across the top, four to a row and two on a phone, on the wide column: the same PlaylistCard, laid out by its list's class (.playlist-grid), so each theme's card skin carries over. It drops the "public" badge there, every playlist listed being public. On every card, a playlist without a cover shows a tile of its own colour with its initial (ThumbnailPlaceholder) in place of the 📋, and the date is "updated …" of its last dump, or "created …".

A middle, ctrl, cmd or shift click anywhere on a playlist card opens it in a new tab — only its title was a link, the rest navigating by a click handler. useCardLink (hooks/shared) is useDumpCardLink's handlers for any card, which the dump cards now get through it.

A profile tab's query stays with the tab — going from the playlists to the dumps kept the whole query, so a playlist category, and now ?sort=, landed on the dumps' URL. Only the dump lists pass their category to each other; any other change of tab drops the query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:51:37 +00:00
khannurien
350e945eb1 v4: a new link between two dumps shows among their related dumps at once — it showed on the next load of the page: the backlink service recorded the link without telling anyone, and the dump page fetched its related dumps once, the socket then only editing or removing the ones it had. relinkBacklinks now sends dump_related_changed when a note or a comment gains or loses a link, naming the dump and those at the other end, and the page showing one of them fetches /related again. A link shows to those who may see both its ends, so each socket is told only of the dumps its user may see, and not at all without one at each end. A private dump that turns public sends it too, for the links it already had, which are new to everyone but its poster. The page also fetches its related dumps again on a reconnection, as it does its comments and playlists. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:43:55 +00:00
khannurien
dde0232acb v4: a link from one dump's page to another's opens that dump — it reloaded the one on screen: the page stays mounted when its path changes, and for one render the old dump was still loaded under the new path, so useCanonicalPath took that path for a former one and put the old dump's back, which was then fetched again. A related dump's link did it, and so did any dump link in a note or a comment. The page now drops the dump on screen in the render that sees a path naming another, as usePlaylist and useProfile do. The dump handed over by DumpEdit or a title edit goes with it, so that coming back to its path fetches it instead of matching that snapshot and loading forever. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:34:32 +00:00
khannurien
7dd00068df v4: the header has two playlists links, each to its own page — it had three: the text button and the avatar menu's "Playlists" both led to the user's own playlists, and the menu's "All playlists" to /playlists. The text button now leads to /playlists, every public playlist, and shows to guests too, the page being open to them. The menu keeps one entry, "My playlists", to the user's own. Below 1150px, where the text button is hidden, the header no longer links to /playlists. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:30:45 +00:00
khannurien
28f910b00e v4: a playlist's dump count is of the dumps the viewer may see — it counted every dump in the playlist, so a card said "3 dumps" over a page listing two, and told everyone how many private dumps a playlist held. The lists (/playlists, a profile's playlists and followed ones, the search, a dump's playlists, the add-to-playlist panel) now count by isMemberVisibleTo's rule, in SQL as VISIBLE_DUMP_COUNT (lib/playlist-members.ts): the public dumps, and for the owner their own private ones. GET /api/playlists and /api/users/:key/follow/playlists read the optional sign-in for it. The community page and the digests count the public dumps, their figures being the same for every reader. Live, playlist_created and playlist_updated carry each socket's own count: broadcastChange builds the owner's message and the others' apart. playlist_dumps_updated loses its dumpCount, a client counting the dumps it is sent. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:25:24 +00:00
khannurien
ffc071af4c v4: a description shows as text where a list quotes it — a playlist's card printed its description as written, so one holding an image showed "![](/api/attachments/…)", and so did a user's line in the search results. Both now go through markdownToPlainText (shared/markdown.ts), as the chat's reply previews do: the markup goes, a link keeps its text, an image is left out, and a description of nothing but an image leaves no line. The playlist's page and the profile render the Markdown as before. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:12:44 +00:00
khannurien
1d56c323a4 v4: the community page's figures stay on one line when a label wraps — each figure is a column-reverse flex box stretched to the row's height, and the default justify-content packed it at the bottom, so a label on two lines ("messages dans la tribune") lifted its number above the others. justify-content: flex-end puts each figure at the top of its cell. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 08:10:27 +00:00
khannurien
a2ec163ee5 v4: ESLint and deno fmt pass on the whole tree again — html.ts's attribute pattern escaped a "/" its character class doesn't need escaped (no-useless-escape), and a line of journalLayout.test.ts ran past the width.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 57s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 25s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:43:28 +00:00
khannurien
7c00db2b84 v4: what comes into a live list goes where its date puts it — a dump edited into a category stayed out of the index's feed of it until a reload, and a dump or a playlist made public went on top of the others' lists as if just posted. The index's feed now takes a dump edited into its scope, and orders the dumps that arrive by date, as the server pages them; a created playlist goes by date in the profile's and /playlists' lists, a new one first as before. Both use withByDate (utils/feed/pagedList.ts), which leaves an item older than every one loaded to a later page. The search runs again after a reconnection, for every page of dumps loaded, with its users and playlists; it shares refetchPages with usePagedList's resync. injectDump's doc said a private dump is never broadcast, while its poster gets it. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:43:28 +00:00
khannurien
308a8299f3 v4: every live update reaches every page that shows what it changed — the client kept each kind of event in one "latest" slot, so two arriving before a render lost the first (two notifications of one dump going public showed one); they now go through a subscription, useWSEvent, and each reaches every subscriber. A playlist made private sent its new title and description to everyone, and kept them on a visitor's page; playlists now follow the dumps' rule, broadcastChange: others get playlist_deleted, and playlist_created when it goes public again. A dump deleted or turned private sends its playlists' dumps again, so their counts and pages follow. Comments on a private dump reach its poster's other tabs. Reading the notifications clears the bell in every tab (notifications_read), and a category change makes every client fetch the categories again (categories_changed). The dump page gives way to a notice when its dump is deleted or made private. A profile gains the user's new dumps, the followed feed those of the users followed and the playlists followed, the search results follow edits and deletions, the index's category feed drops a dump edited out of it, and a renamed user's name follows on cards and comments. After a reconnection, every live list fetches the pages it holds again (usePagedList's resync), and the playlist, the profile, the dump, the categories and the follows are fetched again. The position memory of usePositionAwareSync is now inside the list sync hooks, so a returning item is put back without depending on the order the hooks are called. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-10 07:34:49 +00:00
khannurien
1c5a449d84 v4: API.md says what the URLs are again — it lacked the /playlists pages and GET /api/playlists, a playlist's categories, the ?category= of a user's lists and of the search, and ?chat=<message id>, and its reserved words lacked playlists. It now also states the rules the pages kept without saying: a category is a segment where the page is the category's own and ?category= where it narrows a list on another page; switching tabs takes the place of the history entry, while picking a category or opening another page or an editor adds one; a signed-in page opened signed out comes back after /login. canonical-url.ts named GET /api/dumps/by-url, which is GET /api/dumps?url=. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:25:08 +00:00
khannurien
71df0268e3 v4: asking for a password reset link is a page, /reset-password — "Forgot password?" unfolded the request form on /login as a state of the page, which a reload or a link lost, while /reset-password, where the emailed link leads, answered without its token "Invalid link". The page now holds both steps: without a token it asks for the link, with one it sets the new password, and "Forgot password?" links to it. The login page's unfolding panel and its styles are gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:24:13 +00:00
khannurien
9a215b0cc5 v4: a playlist's editor has a URL, as a dump's has — it opened over the playlist's header as a state of the page, which a reload or a shared link lost, while a dump's editor is /@name/dumps/<slug>/edit. It is now /@name/playlists/<slug>/edit: Edit goes there, and Cancel or Save come back to the page in its place. Both paths render PlaylistDetail at the same place, so the playlist stays loaded, and a new slug keeps the view. Anyone who may not edit it gets the page, a guest the sign-in first. playlistPath takes the view, and parsePagePath reads it, so its link has no preview, as a dump editor's has none. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:22:24 +00:00
khannurien
21f4db268e v4: every request about a user, and the dump editor's path, come from one builder each — fourteen places wrote /api/users/<key>/… by hand, some encoding a handle (@alice%40music.example) and some not, a slug never; userApiUrl (src/utils/shared/urls.ts) now writes them all, by the user's id or by @ and the handle a page's path gives, each segment encoded. The editor's path was the dump's page's with /edit added in two places; dumpPath takes the view, as userPath and feedPath do. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:18:13 +00:00
khannurien
669e65a3a2 v4: a user from another instance can't be renamed from their profile's manage tab — it offered the name field to every user, and the API refuses a remote one's rename with a 400, since their home names them. The tab now only shows their name, as the settings do for one's own. 
The profile's and the settings' tests of a remote account's links to its home still expected the home's host itself, which 8bcc005 replaced with GET /api/users/:key/home; they now expect that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:15:54 +00:00
khannurien
f1bc23a65e v4: an instance can be named by the address of its pages — sign-in and the trust list took only an instance's issuer, its API's origin, so where the pages live elsewhere the address people see in their browser named no instance: in development, localhost:3000 answered its /.well-known with index.html, and the sign-in failed as "doesn't answer as a gerbeur". resolveIssuer (remote/gerbeur-instance.ts) now reads the metadata at the address given and, when it names another issuer, takes it only if that issuer's own metadata names those pages back as where its /authorize is, so an impostor can't claim one; sign-in, the trust list and the check against this instance go through it. The Vite dev server passes /.well-known on to its API, as whatever serves the pages must in production (README). A handle still carries the API's host. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:14:21 +00:00
khannurien
9608520675 v4: a handle with this instance's own host names the local user — another instance writes bob from here as bob@gerbeur.example, so a mention or a link copied from there named no one here: /@bob@gerbeur.example was a 404, and the mention notified nobody. localHandle (shared/validation.ts) drops this instance's host, its API's or its pages', from a handle; resolveUsername reads names by it, so the page, the API and the Open Graph 301 reach bob at /@bob, and so do the mention notifications and the chat's highlight and chime. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:11:39 +00:00
khannurien
6dfba07341 v4: the followed feed's two views are pages — "From people" and "From playlists" on a feed's "follow" tab were a state of the page, which a reload, Back or a shared link lost, while a profile's same split is /@name/follow/playlists. They are now /follow and /follow/playlists, and /music/follow/playlists in a category, feedPath taking the view as userPath does (FOLLOW_VIEWS); picking a category keeps it. Their dumps come from GET /api/dumps?follow=users|playlists, signed in, which the feed's word for it names, instead of GET /api/me/feed?from=; an unknown category is a 404 there, as on the rest of the feed, rather than an empty list. 
A feed's tab, as a profile's, a search's or the community page's, now takes the place of the current history entry, since it is a view of the same page; picking a category, another page, still adds one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:09:12 +00:00
khannurien
53ceb6bcbc v4: signing in comes back to the page that asked for it — a guest on a signed-in page, the dump editor, /notifications, /community, /settings or /manage, went to /login and, once signed in, to /, so a digest's link opened signed out lost its way; and since the redirect pushed, Back led to it again. RestrictedLoggedIn now replaces the page with /login?next=<the page>, as /authorize already did. A guest on a feed's "follow" tab, which only a link leads to, is sent there the same way, rather than shown the default tab. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:05:39 +00:00
khannurien
683944ebe2 v4: a remote user's pages get their link previews and redirects — the Open Graph middleware passed on every path with a dot in it, taking it for a file's, and the handle of a user from a real host has dots (/@bob@music.example), so their profile, dumps and playlists had no tags, and a former name or another case of theirs no 301. Only the dev instances' localhost:8000 escaped it. parsePagePath already passes on any path that doesn't start with @, which no file's does, so the dot check is gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 22:04:35 +00:00
khannurien
53515ce46f v4: playlists have categories of their own — a set apart from the dumps', which admins manage on /manage under "Playlist categories", next to "Dump categories". Each category now has a kind, dump or playlist, its slug and name unique within its kind only; migration 0025 rebuilds categories with the column, every existing one becoming a dump category with its dumps, and adds playlist_categories. A playlist's owner files it under any of them in the create and edit forms, and it shows them as chips on its card and its page, each leading to /playlists/<slug>. 
/playlists is the new page of every public playlist, newest first (GET /api/playlists), and /playlists/<slug> a playlist category's, with a menu of the categories, linked from the user menu as "All playlists"; `playlists` is reserved as a dump category's slug, and one that had it is renamed. A profile's Playlists tab narrows to a playlist category with the same menu as its Dumps tab (?category=), and so do the Dumps and Playlists tabs of /search, each with its own kind (category and playlistCategory for GET /api/search), another tab dropping it.

The category service takes the kind throughout, so a dump can't be filed under a playlist category nor the other way round, and categoryJoin is the one filter join the dump, follow and playlist services share. useCategories(kind) gives one kind's set from the one fetch, CategoryChips replaces the chips the dump card and page each drew, useCategoryFilter keeps a list to its category as the socket brings more, and useCategorySelection holds a form's picked categories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:53:16 +00:00
khannurien
681170a517 v4: in French, an upvote is a "validation" of the reco rather than a vote for it — the button reads "Valider", then "Validé", the notification says "<who> a validé <reco>", and the digests' top recos are "les plus validées". The noun stays "vote": the counts, "Retirer le vote" and the live updates warning. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:10:22 +00:00
khannurien
93565cff0c v4: deno lint passes again — a Markdown test wrapped its child text in curly braces, which jsx-curly-braces refuses; the text is now written as is. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:06:14 +00:00
khannurien
8bcc005140 v4: a remote account's links to its home lead to the home's pages — the settings' "Change it on <host>" button and the profile's "Account on <host>" link went to the host in the handle, which is the home's issuer, its API, not necessarily where its pages are: in development musique's button led to localhost:8000/settings, gerbeur's API, which served a stale build, instead of localhost:3000/settings. GET /api/users/:key/home?path= now redirects to a page of a remote user's home, their profile there by default, at the origin of the home's authorization_endpoint, which its metadata gives (homePageUrl in federation-service.ts), the issuer standing in while the home doesn't answer; a local user 404s, and a path that isn't one 400s. homeUrl links to it. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:06:14 +00:00
khannurien
5f2fb4479a v4: the settings' controls line up down the page — each card of /settings was a grid of its own, its label column as wide as its longest label, so the controls began at a different place in every card, and the other instances, a grid inside their card, at yet another. Every card of settings now has a label column of one width, 10rem, a longer label wrapping, and a list such as GrantList, the card's one child, takes the card's rows: their padding, their rules, their columns. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 20:53:29 +00:00
khannurien
2c3ba6ad11 v4: link previews are read once, in one place, and no longer take an error page, a relative image or a mask icon for the page's — each provider fetched its page and scanned it with regular expressions of its own, the generic one some fifteen times, and Bandcamp and SoundCloud read only OG tags. fetchPage (remote/fetch.ts) now fetches every page, follows redirects to the address it was served from, and refuses an error status, so a 404 or a bot challenge is unreachable rather than a preview titled "Just a moment...", and a refresh no longer overwrites good metadata with it. parsePage (remote/html.ts) reads it in one linear pass with a real attribute reader: unquoted values, a > inside quotes, og: tags under name as well as property, and nothing from comments or scripts. Every URL a page names is resolved against where it was served from, so a relative og:image works; entities are decoded in one pass, &eacute; and &rsquo; included, and one that names no character no longer fails the preview (bandcamp-tralbum.ts shares the decoder). The mask icon is no longer the favicon, JSON-LD prefers the page's node over the site's, tracking pixels and lazy placeholders aren't thumbnails, an og:url with a slug keeps the OG tags, and a <title> loses its site suffix. genericPreview is the base of the Bandcamp, SoundCloud and YouTube channel previews; SoundCloud takes www., m. and on.soundcloud.com short links, embedding the page they lead to, and Bandcamp keeps only its own player. The service tidies every preview: text on one line, titles and descriptions capped, only http(s) images and players. RichContent.type is now RichContentType, and the unused videoId is gone. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 20:46:32 +00:00
khannurien
b4936cccf7 devcontainer: update lock file 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:53 +00:00
khannurien
141e526de7 v4: the instance fields ask for a URL — the login page's field read "Its address, like music.example" and /manage's "An instance's address, like music.example", an example that named no instance of the reader's. They now read "Your gerbeur's URL" and "An instance's URL". 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:53 +00:00
khannurien
63803c552f v4: a profile's lists of dumps can be narrowed to a category — /@name/dumps and /@name/upvoted showed every dump, with no way to see only those filed under one category, as the index can. The title row of each list now has the index's category menu, and the category picked goes in the URL as ?category=<slug>, which the other list keeps and every other tab drops (a path segment would collide with /@name/dumps/<slug>, a dump's own page). GET /api/users/:key/dumps and /upvoted take ?category= as the followed feed does, an unknown slug answering an empty list; resolveCategoryFilter moves to category-service.ts for both. useUserDumpFeed passes everything the socket changes through the category, so a dump edited out of it leaves and an upvote outside it doesn't come in. CategorySwitcher now reports the category picked to its owner (onSelect) instead of navigating to the index's path itself, and the divider after it in the index's tab row is drawn by CSS, so the profile gets none. useTabParam's setter takes the query string to go to, the current one by default. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:23:03 +00:00
khannurien
52a23db504 v4: an instance that takes no users of others still blocks instances its own users may not sign in to — with GERBEUR_FEDERATION_INCOMING=0, /manage hid its list of instances, though a blocked one still kept the instance's users from signing in there, and no admin could block another. The list now stays while federated sign-in goes either way: without incoming sign-ins it shows and adds only blocked instances, the trusted ones kept for when they're back, and says what blocking does there. Where both ways are on, its note now says that users here can't sign in to a blocked instance either. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:33:08 +00:00
khannurien
83dc44050e v4: a deployment chooses which ways federated sign-in goes — every instance took users of the others and let its own sign in to them, with no way to turn either off. GERBEUR_FEDERATION_INCOMING=0 keeps users of other instances out: the login and register pages offer no other instance, /manage no list of instances, their sign-ins are refused, and those already in are signed out until it is on again, their homes' events still applied meanwhile. GERBEUR_FEDERATION_OUTGOING=0 keeps the instance's users from signing in elsewhere: its /authorize page lets no instance sign them in, its metadata says so (gerbeur_outgoing), which other instances check before sending anyone there, the settings page lists no instances, and at startup every grant is withdrawn, which signs them out there. Both are on by default and reach the client through index.html. The dev instance gerbeur (:3000) takes no users of the others, while its own still sign in to them. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:18:21 +00:00
khannurien
66e402bd91 v4: federated sign-in closes the holes a review found — a callback link made from someone else's sign-in signed the visitor in to that account, a password reset left the user's sessions on other instances open, and events could be lost, reordered or delayed. FederatedSignIn keeps the sign-in's state in sessionStorage (utils/auth/federation.ts) and FederationCallback finishes only a sign-in this tab started. A password reset tells the instances the user signed in to, as a change does. The outbox delivers each instance's events in the order they were made, the first that fails holding back the rest, so a late older update no longer undoes a newer one, and delivers to every instance at once, so a dead one delays none of the others; an event queued during a run is sent by it. 
Receiving, a home's keys that can't be fetched answer 502, which the sender retries, instead of 401, which it dropped; a bad update answers 400 rather than 502; a blocked instance's events answer 403; events are limited to a burst of 120 per instance, and a token naming an unknown key refetches the keys at most once a minute. An avatar must be on its issuer's origin, and its URL is versioned by its file's write time instead of updatedAt, so a profile edit no longer makes clients download it again. An instance that doesn't answer can be blocked, keeping its name, though not trusted.

303 backend tests and 703 frontend tests pass, the federation tests 8 times in a row, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:05:11 +00:00
Vincent Lannurien
c77e173767 v4: the devcontainer builds and starts again — the docker-outside-of-docker feature failed to install, since the base image is now Debian trixie, where its default moby option isn't supported; it now installs Docker's own packages ("moby": false). Every service of the compose project was also built under one image name, gerbeur-dev, the workspace with the features and the others without, so whichever finished last took the name and the workspace could lose its docker CLI; each service now gets its own image, built from the same cache. 
Checked with the devcontainer CLI's up: the image builds, install exits 0, gerbeur and musique answer on :3000, :3001, :8000 and :8001 (openid-configuration included), and docker ps works as node in the workspace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:39:58 +02:00
khannurien
a25500d833 v4: the devcontainer starts several gerbeur instances side by side, so that signing in from one to another can be tried on one machine — the dev environment ran a single instance. dev/instances.ts lists them (gerbeur on :3000 and :8000, musique on :3001 and :8001), and dev/start.ts starts an API server, restarted on change, and a Vite server for each instance it's given, by one rule: instance <name> keeps api/sql/<name>.db and api/uploads/<name>/, reads .env, and takes its ports and what sets it apart from its entry, with federation allowed over http on localhost. It prefixes each line of output with the instance's name, and stops every server when one stops, such as a Vite server whose port is taken. The devcontainer is now a compose project (.devcontainer/compose.yaml): the workspace, a one-shot install that runs deno install and draws the icons once, and a service per instance running dev/start.ts for it, all on the host's network as before and sharing Deno's cache through a volume. ffmpeg moves from a feature into the Dockerfile, since the instances' services get no features, and docker-outside-of-docker gives the workspace's terminal their logs and restarts. Outside the devcontainer, deno task dev starts them all, or those named. Each Vite server keeps its own dependency cache. The existing dev uploads move to api/uploads/gerbeur/. 
dev/start.ts was checked by hand: musique starting alone on a fresh database, an unknown name refused, a taken port stopping both servers, and Ctrl+C or SIGTERM leaving no server behind. The compose project couldn't be run here (no Docker in the workspace): its YAML parses and resolves as intended, but it is untested; tsc, eslint, deno check, deno fmt, deno lint and doc:lint, which now covers dev/, are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 14:31:25 +00:00
khannurien
c0d03e9aaa v4: users sign in to one gerbeur instance with their account on another — topic instances, each with its own admins, domain and data, had no way to share accounts. Sign-in is federated over a subset of OpenID Connect, the authorization code flow with PKCE: each instance is an identity provider for its own users and a client for the others', and signs what it tells them with an Ed25519 key kept in federation_keys and published at /.well-known/jwks.json, beside its metadata at /.well-known/openid-configuration. Only sign-in is federated: an account from another instance is a local users row linked to (issuer, subject) in user_identities, with no password (password_hash ''), so sessions, roles, permissions, the socket and every foreign key work as before. 
A remote user is name@host here, in users.username, in paths (/@alice@music.example) and in mentions, through one handle rule in shared/validation.ts that paths, MENTION_RE and Markdown all build on; Markdown undoes GFM's email autolink of a handle. Their name, email and avatar come from their home at each sign-in and by its signed events (federation_outbox, delivered by the federation job with backoff): a rename at home leaves the old handle an alias here, and a password change or a withdrawn grant ends their sessions here. Their password, email, avatar and name can't change here (assertLocalUser), nor can they sign in elsewhere from here.

An admin keeps the trusted and blocked instances on /manage (federation:manage): a trusted instance's users sign in from a button on the login page, a blocked one's can't and are signed out at once, and any other's join with an invite from the register page. The /authorize page asks the user once per instance, and /settings lists the instances their account signs in to, each disconnectable; /login?next= brings a signed-out visitor back to /authorize. Emails stay unique among local accounts only, and migration 0024 adds the tables, makes that index partial and reserves authorize and federation as category slugs. remote/gerbeur-instance.ts is the only way to other instances: https and public addresses only, no redirects, no certificate fallback, unless GERBEUR_FEDERATION_ALLOW_PRIVATE lets two instances on one machine talk.

298 backend tests and 702 frontend tests pass, and deno check, deno fmt, deno lint, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked end to end with two built instances on one machine: sign-in through the other's login and consent pages, consent skipped once granted and asked again after disconnecting, a rename and a password change at home reaching the other, and blocking; screenshots of the login, consent, profile, settings and manage pages at 1280 and 390px.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 14:23:00 +00:00
khannurien
2d26d0aada v4: the late-digest test no longer races the clock — it posted its only dump at the current time and sent the digests right after, and a digest's window ends before the run's now: when both fell in the same millisecond, which createUser's cached password hash made common, the dump was outside the week, the digest empty, and the test failed about one run in fifteen. Reproduced with a dump stamped at now (no digest) and one a millisecond earlier (one); the dump is now dated a day back, as "recent" meant. 60 runs of the digest tests, 6 of the backend suite (277 tests) and 4 of the frontend suite (684 tests) pass. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:47:56 +00:00
khannurien
fac0e2db28 v4: an audio file's waveform is made once by the server, and the player seeks on one accessible scrubber — every player downloaded the whole file a second time and decoded all of it in the browser to draw 150 bars, so a long FLAC cost tens of megabytes of memory per page, and the waveform could only be clicked: no drag, no keyboard, no slider for a screen reader. GET /api/dumps/:id/waveform now answers an audio dump's peaks, which lib/ffmpeg.ts decodes with ffmpeg on first request, mono at 8 kHz and streamed, keeps beside the thumbnails as the response itself, and serves with an ETag (a ten-minute MP3 takes about a second, then a few hundred bytes). The peaks are taken by shared/waveform.ts's peaksOf, scaled to the loudest so a quiet recording fills the track, and the dump form's preview of a file not yet uploaded decodes it with the same function, at 8 kHz in an OfflineAudioContext. Scrubber replaces Waveform and SeekBar: a role="slider" that seeks where a pointer lets go, follows a drag (the clock shows where it is held), takes arrows, Home and End, and draws the bars as one path with the played part clipped by a nested svg, so it moves smoothly and a timeupdate redraws two attributes instead of 150 rects; audio without a waveform, and video, get a plain bar. MediaPlayer reads whether it plays from the element's events alone, so a click no longer reports the change twice, its handles for PlayerProvider are stable callbacks instead of refs to refs, the drag no longer reattaches every listener, and the <source> children that src made dead are gone, with the mime prop. The file, thumbnail and waveform routes resolve their dump, token and caching through lib/dump-media.ts, and what is generated from an upload is named and removed in lib/ffmpeg.ts, the custom thumbnail by dump-service's customThumbnailPath, instead of paths spelled out in each. The test setup's play() and pause() fire their events as a browser does. 277 backend tests and 684 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in the app with a real FLAC on its dump page and in the global player at 1280 and 390px, seeking by keyboard. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:34:22 +00:00
khannurien
c709b56911 v4: every message named by an id carries its English in the code — the emails and the four pages that share their words called Lingui with a bare id (i18n._("digest.community"), <Trans id="digest.community" />), so their English lived only in a hand-kept en.po, extract couldn't run with --overwrite, and a page reading the code couldn't tell what it said. The emails now pass descriptors, i18n._({ id, message, values }), the form Lingui reads without a macro, as the server needs. The fifteen messages the digest shares with the community page, Settings, the unsubscribe page and the avatar menu are declared once, as /*i18n*/ descriptors in shared/digest-messages.ts, which those pages translate with useLingui()'s i18n instead of the global one. lingui.config.ts scans shared/ too, and i18n:extract runs with --overwrite, so en.po is written from the code. Every other message already used the macros. The compiled catalogs are unchanged, key for key, in both languages. 272 backend tests and 677 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:19:01 +00:00
khannurien
c3594e4c27 v4: what the server and the client both state is declared once — several rules were written out in two or more files, some already drifting apart. A link's site name (its host without www.) was computed in five places, now shared/url.ts's siteHost; dumpFileVersion, which the frontend and the server had to keep equal by hand, is one function in shared/dump.ts; the Markdown flattening of the chat's reply previews and the digest's quotes is shared/markdown.ts's markdownToPlainText, so the digest also keeps an image's alt text; an attachment's URL and how to find one in a text are shared/attachment.ts, which the editor, Markdown and attachment-service read, its id pattern the new UUID_RE that the file and thumbnail routes check ids with. On the server, the Bandcamp, YouTube playlist and link preview caches use one TtlCache (api/lib/ttl-cache.ts) instead of three copies of the same expiry and eviction; the link previews' theme-color and the emails' theme tokens read colours with one parser (api/lib/css-color.ts), and a transparent theme-color now gives no tint; the WebSocket route sends through ws-service's send, the welcome included, so it is type-checked; fetch.ts strips an IPv6 literal's brackets with public-address's bareHostname; and a category's slug is checked by the same validateSlug as a dump's, with its own length. The migrations keep their frozen copies, as they document. 272 backend tests and 677 frontend tests pass, and deno check, tsc, eslint and doc:lint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 13:16:52 +00:00
khannurien
134d1a0b97 v4: a moderator's or an admin's @everyone in the chat notifies every user — there was no way to reach everyone at once. The new chat:everyone permission, held by moderators and admins, lets a chat message saying @everyone notify every user but its author and those reading the chat, through notifyEveryone, after notifyMentions, so a user also named is told once, by name; the notification is a user_mentioned with everyone set, which the notifications page and the digest word as "mentioned everyone in the chat". chat_messages.mentions_everyone, sent as ChatMessage.mentionsEveryone, is set from the author's role each time the body is written, so a user's @everyone reaches no one, a moderator's edit of a user's message can't make it reach everyone, and a demoted author's next edit loses it. Only an edit that adds @everyone notifies, so fixing an announcement's typo reaches no one new. The client highlights and chimes by the flag, never by the text; Markdown shows @everyone as a mention only in a flagged message, and never links it to a profile. The chat composer offers @everyone first to those who may say it. No account may be named everyone, in any case: migration 0023 adds the column, flags the moderators' and admins' past @everyone messages, renames a user named everyone to everyone_N and drops an alias everyone. 265 backend tests and 681 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in screenshots of the chat at 1280 and 390px, with a moderator's and a user's @everyone and the suggestions open. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 12:54:09 +00:00
khannurien
5f239a144d v4: the community page and the digests' section are titled Communauté in French — "La communauté" read awkwardly as a title. i18n:compile --strict is clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 11:52:54 +00:00
khannurien
708d113013 v4: a /community page shows the site's week and month, from the same figures as the digests, and every recurring server task is a job of one scheduler — the digests' top dumps and community section (who joined, the new public playlists, the chat and the totals) existed only in emails. community-service.ts now gathers them for a period from listTopDumps, listNewUsers and listNewPublicPlaylists, each in its own service and returning the API's Dump, PublicUser and Playlist; the digest builds its sections from the same call, its email-only DumpRow and DUMP_COLS replaced by one Dump mapper that networkOf uses too, and links its community section to the page. The page, /community for the last week and /community/monthly for the month, signed in only and linked from the avatar menu, reads GET /api/community?period=, which serves figures kept in memory and recomputed every GERBEUR_COMMUNITY_REFRESH_MINUTES (15 by default). It shows the four figures, the top dumps and the new playlists as live cards, and the new members as UserCards, the follow tab's user row made shared. Its words are the emails' own: the messages both use are renamed from email.digest.* to digest.*, as digest.weekly.off was. api/lib/scheduler.ts's startJobs runs the digests, the community refresh, the WebSocket ping, the typing expiry and the rate-limit sweep, which main.ts starts, so none of them runs on import any more. PERIOD_MS moves to shared/settings.ts. community is reserved as a category slug, and migration 0022 renames a category that had it, with renameReservedCategories moved out of 0021 into reserved-slugs.ts. A group's dump or playlist cards now fill its column. 262 backend tests and 678 frontend tests pass, and deno check, tsc, eslint, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth light at 1280 and 390px, Smooth dark with mocked playlists, Brutalist dark at 390px and Kanagawa dark at 1280px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 11:42:01 +00:00
khannurien
7713c5a1b5 v4: a dump or playlist going public notifies its followers — a private dump notified no one when posted, and nothing told its followers when it was made public: the poster's followers never learnt of it, nor did the followers of the public playlists holding it. Likewise, the dumps added to a playlist while it was private were never notified to its followers. updateDump and replaceFileDump now call notifyIfWentPublic, which notifies the poster's followers (user_dump_posted) and, through the new notifyPlaylistFollowersNowPublic, the followers of each public playlist holding the dump (playlist_dump_added). updatePlaylist calls the same function when a playlist goes public, for each public dump in it, from the bottom up. A follower hears of a dump once both it and the playlist are public; the source keys stop a dump made private and public again from notifying twice. 256 backend tests pass, and deno check, deno fmt and doc:lint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 10:23:08 +00:00
khannurien
256857f72e v4: a chat mention notification opens the chat on the mentioned message, so the newest one clears the chat badge — it linked to ?chat=open, and the chat opened on the "new" divider whenever the unread messages didn't fit. That left the list above the bottom, so the read marker never moved and the badge stayed, while the mention itself was out of sight. It now links to ?chat=<message id>: ChatProvider opens on any ?chat= (open, or a message id passed to ChatModal as focusId), and ChatModal opens centred on that message and flashes it. When the message is within a screen of the bottom, the list stays at the bottom and reads everything, as it always has. An older mention still leaves the messages after it unread. A message outside the first page of history falls back to the divider. ChatModal's flash is now its own callback, shared with jumping to a message. 675 frontend tests pass, and tsc, eslint, deno fmt and doc:lint are clean; reproduced and checked in the app with 36 unread mentions from a second account: the newest clears the badge, an older one opens on it with the badge kept. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 08:56:59 +00:00
khannurien
4ea192e0b1 v4: an uploaded audio file plays from its cover everywhere, and a card's play badge sits in the middle of its thumbnail — a link played from its thumbnail in every view, but an uploaded file only did on a list card: a journal tile opened the dump, an audio file's page showed a round button by its waveform and never its uploaded cover, and an uploaded video's page was a black box until played. The list card's badge sat in the thumbnail's bottom-left corner, where it looked misplaced, and clicking a playing file restarted it. The new usePlayFile hook builds the player's item and the file's artwork (fileArtworkUrl) for the card, the journal tile and the dump's page, and plays, or pauses once the file is the current one. A card's badge, ▶ or ⏸, is now centred, as is the ↗ on a link out; over a placeholder it takes the place of the glyph. An audio file without a cover shows a placeholder tinted from its file name, not a bare 🎵. In the journal, audio counts as having art (hasThumbnail), so it is an image tile, its glyph raised above the title, and a file tile plays and pauses the file. On the dump's page, the cover (96px, the card's badge on it) plays the file beside the shared Waveform, which replaces FilePreview's copy of it; a video takes the server's still as its poster. 674 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth light at 1280 and 390px, Brutalist dark on the journal at 390px and Kanagawa dark at 390px, playing. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:33:33 +00:00
khannurien
2c031b0102 v4: the header drops the username link — on wide screens it stood beside the avatar, which leads to the same profile, so only Playlists stays there. The avatar's menu still opens with @username first. 670 frontend tests pass, and tsc, eslint, deno fmt and i18n:compile --strict are clean; checked in a screenshot in Smooth dark at 1400px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:23:46 +00:00
khannurien
886216749e v4: the avatar menu shows at every width and holds Log out, so a profile keeps only Invite; the profile's dump cards lay out again — your own profile had four buttons in a row, Invite someone, Settings, Manage and Log out, Invite taller than the rest, and on wide screens it was the only way to reach /settings or /manage. The avatar menu now leads the header's icon row at every width, after the name and Playlists text links on wide screens, and ends with Log out under a separator, muted until hovered. The profile keeps Invite someone alone. The menu's button outweighs .app-header-nav button and the themes' button rules (Brutalist and GeoCities list it among their flat buttons), so it reads as one of the menu's links. Since b9ff9c4, a card under 472px is laid out by the narrow-card container query, but the profile columns' dense-card rules still placed the thumb, body and vote themselves, more specifically: the thumb lost its row span, so the title fell under it and every card was a tall, mostly empty box. Those rules now only make the card denser (a smaller thumb, a 2-line title, smaller type) and leave placement to the container query. 670 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Smooth dark at 1400 and 390px, Brutalist light at 390px, GeoCities light and Kanagawa dark at 1400px, in French. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:22:15 +00:00
khannurien
10c9cf4269 v4: a playlist's cover is a banner over its header, and the edit form picks it in the same shape — the cover was a 72px square beside the title, which left the title and description a narrow column under a mostly empty band: on a phone, in Kanagawa's wide mono, a long title broke into six lines. The cover now spans the header's width above the title, cropped to 3:1 and no taller than 14rem, inside the header's padding so outlined themes keep their ring around it; the title, buttons, description and meta take the whole width under it. The edit form's Cover picker takes the same shape (.playlist-cover-banner, shared with .playlist-detail-img), and with no cover is a dashed 3:1 placeholder. ImagePicker's size is now optional, so a class can size it, and its radius takes any CSS length, here --radius-lg; a picker sized by aspect-ratio sets min-height: 0, or its image would hold it at its own height. 669 frontend tests pass, and tsc, eslint, deno fmt and doc:lint are clean; checked in screenshots in Kanagawa dark (French, 385px), Brutalist at 390px with and without a cover, and Smooth at 1100px, the page and the form. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:10:06 +00:00
khannurien
5acdf79c8d v4: the playlist and dump edit forms end in one bar of actions, a panel of its own — the playlist editor drew rules between its fields and its actions, and on a phone its Delete, Cancel and Save wrapped into a staircase, then stacked (a0c8b1d), neither of which held together. Both forms are now built as DumpEdit was: surface panels 3px apart (.form-panel, from .dump-edit-panel), the last a bar (.form-panel--bar, from .dump-edit-footer) holding Delete on the left and Cancel and Save on the right, compact (btn-sm), on one line. The new DeleteButton says "Delete" and names what goes to assistive tech ("Delete playlist", "Delete dump"); in a bar too narrow for the three labels in every face and language (a container query, under 18rem), it shows 🗑 instead. While edited, the playlist's header drops its own card for the form's panels. The phone stacking rule is gone. An auto-sized editor also re-fits when its width changes with its text unchanged: opening the form makes the page tall enough for a scrollbar, which narrowed the description after it was measured, so it overflowed; and it hides its overflow below its max-height, where a scrollbar could only come of rounding, and drops the resize grip, which the next keystroke would undo. 669 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in Kanagawa dark (French, 385 and 360px), Smooth light at 1000px and DumpEdit in Smooth at 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 22:05:43 +00:00
khannurien
b9281da9c9 v4: the avatar menu opens from the avatar's left edge — since 7531be3 the avatar leads the header's icon row, but its menu still hung from its right edge, so on a phone it ran off the left of the screen. It now hangs from the left edge and opens over the icons. Checked in screenshots in Smooth at 390px and Brutalist at 320px, in French, and Kanagawa at 1100px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:58:15 +00:00
khannurien
7531be300e v4: the header's controls are line icons, and the avatar leads them — 🔍, ✕, 💬 and 🔔 were four emoji in four palettes beside the avatar and the "+", none of them the theme's. They are now stroked icons (NavIcons.tsx, which the search bar's submit and ChatFab share) in the text colour, their bodies washed with the accent; the wash deepens on hover, and fills while something is unread, with a small dot in place of the red count (the count stays in the button's label and the tab's title, and ChatFab keeps its own). Themes set the line through --nav-icon-stroke, -linecap, -linejoin and -tint-strength: Brutalist heavy and square-cut, Times hairline in ink (its grayscale filter no longer needed on them), GeoCities fat and soaked. The search toggle was a ghost button with its own hover; it now shares the chat button's and the bell's rule, background and borders in every theme. The avatar sat among the icons, its hover box larger than theirs and 1.5px higher (a block .user-menu left a line-box gap under the inline-flex trigger): it now leads the row, where the name and Playlists links stand on wide screens, so the icons run unbroken to the "+"; it is 32px, and hover rings it with the accent instead of a box. 669 frontend tests pass, and tsc, eslint and deno fmt are clean; checked in screenshots in all six styles, light and dark, at 390px, at rest and unread, and Smooth and Brutalist at 1280px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:56:19 +00:00
khannurien
a0c8b1d509 v4: auto-sized text editors no longer scroll, and on a phone an action row with a destructive button stacks — an autoResize editor set its height to scrollHeight, which leaves out the borders that a border-box height counts, so the text always overflowed by a couple of pixels: a playlist's description, a .form textarea since cf89c18, showed a scrollbar that ran under the emoji button. The height now adds the borders. On a phone, Delete, Cancel and Save didn't fit one line in French or in a wide face, and wrapped into a staircase: Delete alone on the left, Cancel and Save under it on the right. A .form-actions row with a .btn-danger now stacks under 512px, with Cancel and Save across the top (Save taking the room left) and the destructive button full width under them; the playlist editor and DumpEdit's footer both take it. 669 frontend tests pass, and tsc and deno fmt are clean; checked in screenshots in Kanagawa (French, 385px, with a long description) and DumpEdit's footer in Smooth at 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:51:38 +00:00
khannurien
cf89c18eb9 v4: a playlist's edit form is a .form like the dump's, and its dumps keep the header's edges — the editor laid the cover beside its fields with display: contents, which on a phone left them a squeezed column, and drew its own inputs: the title and the slug both in bold large type, the description in another size, only the slug labelled, and a segmented control styled apart. It now stacks the shared fields, each under its label (Cover, Title with its counter, Link, Description, the visibility toggle), over the action row, as DumpEdit does, and takes the header's whole card. The dumps' drag handle and remove button took width from the cards, which sat inset from the header on both sides; they now sit in the page's side padding (--page-gutter, which .page-content now names), so the cards line up with the header at any width. Undo shows as a small btn-primary over the card it would bring back, instead of a third control beside it. CountedInput, VisibilitySegmented and the .playlist-edit-* rules had no other users and are gone. 669 frontend tests pass, and tsc, eslint, deno fmt, doc:lint and i18n:compile --strict are clean; checked in screenshots in all six styles at 390px, Kanagawa and Smooth at desktop widths too, Undo in Smooth and GeoCities. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:48:12 +00:00
khannurien
45f39a959b v4: /settings and /manage take after /notifications — the three pages the user menu and the bell lead to share one frame: a 680px column (PageShell's new narrow), a title with its icon (PageHeader: ⚙️, 🛠️, 🔔), then groups under a small capitalized label ruled to the column's end (PageGroup, the notifications' day label), their content on surface cards (.page-card, the notification items' surface). Settings drew big section headings over controls laid on the bare page, the full width wide, and Manage the same around the category manager; each section is now a card whose setting rows are ruled apart and keep their labels lined up (a subgrid of the card's columns). The notifications' column, centered by auto margins in a flex column, shrank to its content (380px with short ones); it fills its width now. Its own header, group and bell rules gave way to the shared ones, and Times, Brutalist and GeoCities dress .page-card as they dress the notification items. In the narrower column, Kanagawa's monospace no longer fit the six styles on one row, and French labels leave even less room: a segmented control of more than four options now makes two even rows at any width, as it did on phones. 670 frontend tests pass, and tsc, eslint, deno lint, deno fmt and doc:lint are clean; checked in screenshots in all six styles at 1280 and 390px, Smooth in light too, with the account's style and the notifications mocked so the shared test account was left as it was. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:40:10 +00:00
khannurien
1fda26a399 v4: /settings, /manage and /notifications title themselves alike, with a bare heading above their sections — Settings and Manage borrowed the edit form's .form-page-header, a panel rounded only on top to cap the panels under it, so on a page without them it ended in a square edge, and its title was the size of the section headings below. Notifications had a heading of its own. All three now take .page-header and .page-title: a bare h1 a size above the sections' h2, with what goes with it (the unread count) at the row's end; the edit forms keep their panel header. Times and Melange set their display type on .page-title, and the .profile-subpage-* rules, unused since the profile's pages became its tabs, are gone. (The CSS went in with d683ac0, which a concurrent session committed from the shared tree.) 21 Settings, Manage and Notifications tests pass, and deno fmt is clean; checked in screenshots in Smooth at 1280 and 390px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:28:28 +00:00
khannurien
45e004b465 v4: the slug and rename fields lose their hints about old links — "Links to the old address keep working." under the dump and playlist link field, and "Their old name keeps leading to them…" under the moderator's rename field, described a server guarantee nobody filling the form needs to read. Only the fields' errors show under them now; UsernameField loses its hint prop. 670 frontend tests pass, tsc and i18n:compile --strict are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:28:01 +00:00
khannurien
437eea0df3 v4: every inline editor ends in a btn-primary Save and a btn-secondary Cancel — comments drew their own .comment-submit-btn and .comment-cancel-btn, the profile's email its own .profile-email-btn pair, the bio a plain-text form-cancel, and the chat's edit box shrank the shared buttons with a rule of its own; they were four sizes, two orders of chrome, and the comment button's white label sat on Smooth's yellow accent. They all take the shared classes now, and the new .btn-sm is the one compact size, for the comment forms, the chat edit box and the email editor; the bio and the dump title stay full size. .btn-secondary gets a disabled state, as btn-primary has. The bespoke rules are gone, with GeoCities' bevel for the comment buttons, which btn-primary and btn-secondary already have. 670 frontend tests pass, and tsc, eslint and deno fmt are clean; the comment, reply, chat edit, email and bio editors were checked in screenshots in Smooth, Kanagawa, Brutalist and GeoCities at 1100 and 360px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:27:26 +00:00
khannurien
d683ac0548 v4: the dump title editor's Cancel is a button, beside Save — it was the grey text link of the full forms' action rows, next to a filled Save, where every other inline editor pairs Save with a bordered Cancel; it now takes btn-secondary, the same height as btn-primary. Save and Cancel sit in .dump-title-actions and wrap together, so on a phone they go under the input as a pair instead of Cancel dropping to a line of its own. 30 Dump page tests pass, and tsc, eslint and deno fmt are clean; checked in screenshots in Smooth, Kanagawa, Brutalist and GeoCities at 1200 and 360px. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:23:28 +00:00
khannurien
b9ff9c417e v4: phone layouts hold down to 300px in every style, Kanagawa's wide monospace included, and the lint is clean — the floating chat and "+" buttons hid the end of every page (Save on the dump edit page, the last setting), since the page kept room for one button and they stack two; it keeps room for both now, read from one --fab-size. A 1fr page grid grew to its widest unbreakable content, and the edit forms' action bars didn't wrap, so at 320px the dump edit page scrolled sideways and the playlist edit's Save stuck out of its card; the grids take minmax(0, 1fr) and .form-actions wraps. The profile header's info column was a hidden horizontal scroller under a fade, which cut the username, the inviter, the email and the Log out and Invite buttons; it wraps instead (long names and emails break anywhere), and the avatar shrinks to 70% on phones. TabBar pins its leading content (the feed's category switcher) at the row's start while the tabs scroll under it, where scrolling to the active tab used to push it out of view; its edge fades show only while tabs are hidden past that edge, so the last tab is no longer dimmed once scrolled to, and a left fade marks a scrolled row. Tabs are narrower on phones, which lets Kanagawa's search tabs fit at 375px. The global player puts its track controls on a row of their own on phones, where they left the title three letters, and an audio bar narrower than 420px drops its volume slider for the waveform. The profile bio lost the margins it stacked on the page's gap. A dump card is now a container and lays out by its own width rather than the viewport's, since a playlist's is narrowed by its drag handle and remove button and titles broke every few letters: under 472px it is a grid where the vote ends the date line, as on the journal's cards, instead of hanging under the thumb, with a spacer row keeping it off the title; under 300px the thumb takes the top row and the text the full width. The lint errors that predated this are fixed: a useless fragment and two async callbacks without await in tests (they now await advanceTimersByTimeAsync), EmailKit's renderer gets the kit as an argument instead of aliasing this, its figure regex drops the literal no-break spaces \s already matches, digest-service's highlightOf takes NotificationData narrowed per case instead of any, and eslint ignores coverage/. 254 backend and 670 frontend tests pass, and tsc, deno check, deno lint, deno fmt, eslint and doc:lint are clean; every page was checked in screenshots at 300 to 480px in Kanagawa and Smooth. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 21:19:43 +00:00
khannurien
58c95dc4f3 v4: the notifications page restores its list and scroll on Back, and shows what it missed — it now saves its loaded pages and scroll position as it is left and restores them on Back, through useFeedCache, useScrollSave and useScrollRestore like the feeds and the profile's tabs; the saved list keeps the ones the server confirmed read as read, so coming back doesn't highlight them again. The page never refetched after a reconnect, so notifications sent while the socket was down never showed, and the next read-all marked them read unseen; a reconnect, and every restored list (after a reload the socket hasn't said yet whether anything is new), now fetch the first page again, and what is new goes on top with the socket's arrivals, those alone sorted newest first so the server's order stays. Marking read now follows the unread notifications shown, each sent once, instead of each socket message, and a list with none unread sends nothing. The arrivals shift the server's offset pages, so a next page could repeat the last one's tail: each notification shows once. The next page loads as the end scrolls into view, with FeedFooter, as every other page list does, and the loading and error states go through ListState. The "N new" pill was a number before a separately translated "new", which French made "3 nouveau"; it is one plural message, "3 nouvelles". Dates past a week and the tooltip use the app's locale rather than the browser's. The mention, comment and like icons had no colored circle; they have one now, and the decorative emoji are hidden from screen readers. 670 frontend tests pass (new: loading on scroll, the save on leaving, the restore with what came meanwhile, the reconnect backfill), and tsc, deno fmt, deno lint, doc:lint and i18n:compile --strict are clean; the restore was checked in the running app at phone width. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 19:54:31 +00:00
khannurien
0a16d3c96c v4: users can't rename themselves, /settings lays out again, and its selectors fit a phone — PATCH /api/me refuses a username with a 400 and changes nothing else in that request; a name changes only through PATCH /api/users/:key/username, which needs user:manage, from /@name/manage. The old name still stays an alias for good. renameUser lost its 30-day limit and its exempt flag, and RENAME_LIMITS holds the slug limit alone. /settings shows the username as plain text, and the rename hint left the catalogs. The settings grid sized its control column to its widest content, the rename hint, which stretched every control under it; that column now takes the remaining width and each control keeps its own size. A segmented control wraps instead of overflowing; in a settings row on a phone it takes the full width, and one of more than 4 options (the styles) lays out in two even rows. .btn-border turned red on hover, so Settings, Manage, Retry and Back looked like Log out; it is now neutral (Smooth's accent is red too), and only Log out takes the new btn-border--danger, muted until hovered red. A <button> and a <Link> .btn-border are now the same height. API.md and the module docs say who renames; the backend tests rename through the moderator route and check a self-rename is refused, and the malformed-name check moved to the manage tab's test. 254 backend and 667 frontend tests pass, and tsc, deno check and doc:lint are clean; Smooth, Brutalist, GeoCities and Kanagawa were checked in screenshots at desktop and phone widths. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-07 06:50:43 +00:00
khannurien
4468438ba7 v4: every page and API URL follows one of two rules, written down in API.md — pages are a hierarchy where @name is a user and any other root word a category or an app page, tabs and views are plain segments and a default view is the bare path; API resources are /api/<plural>/<key>, a user's key being an id or @name, the caller's things under /api/me, media a sub-resource, lookups by another key a query filter. ~ is gone: the feed is /, /hot, /new, /journal, /follow and /music/hot, a profile /@bob with its overview and the tabs dumps, upvoted, playlists (her own only), follow (users, and /follow/playlists), invitees and manage (role and name, for user:manage), which replace the separate UserDumps, UserUpvoted and UserPlaylists pages; /settings holds the account (now with the username) and /manage the category manager, both linked from the user menu and one's profile. Dumps and playlists live under their owner, /@bob/dumps/<slug> and /@bob/playlists/<slug>, named by slug alone: migration 0021 re-slugs them to the bare slugify(title), unique per owner and kind (the oldest keeps it, clashes get -2), renames the categories whose slug v4 reserves (hot, follow, settings…), and rewrites this site's v3 links in dump notes, comments, chat, and playlist and profile descriptions. A slug changes only when edited, through the edit forms' new link field, which proposes slugify(title) as the title changes; a username changes from /settings or, for user:manage, from /@name/manage. Every slug or name left behind stays an alias of its item or user for good (dump_slug_aliases, playlist_slug_aliases, user_aliases), so old links redirect without chaining and nobody else can take one; slugs change 3 times a day per item, names once in 30 days, moderators exempt. An alias, or a name in another case, answers the item with its current path: the pages put it in their URL (useCanonicalPath, without fetching again), and the OG middleware answers a 301. shared/paths.ts builds and parses every page path, for the client's links, the emails' (digest-service lost its own builders and its id fallbacks, and links /follow and /settings) and the server's matchers (OG tags, self previews, backlinks, self-url); shared/slug.ts holds slugify and the format categories share. Dumps and playlists carry ownerUsername and a required slug, and notifications join their users' current names and their items' paths each time they are sent or read, so renames reach them and a deleted item's has no link. The API was renamed per the plan's table: POST /api/users and /api/sessions, /api/password-resets, /api/me (with settings, password, avatar, media-token, playlists?contains=, follow, feed?from=, notifications), /api/users/:key/{dumps,upvoted,playlists,follow/users,follow/playlists,invitees,avatar} and by path /api/users/:key/{dumps,playlists}/:slug, /api/dumps?url= and ?category=, /api/dumps/:id/{file,thumbnail,metadata}, /api/playlists/:id/cover, /api/remote/{preview,image,bandcamp,youtube} (one router), and /api/unsubscribe, the v3 POST /api/users/unsubscribe staying for the digests already sent. A stored default feed tab "followed" reads as "follow" once. dumps.slug and playlists.slug stay nullable, since SQLite can't add NOT NULL without rebuilding tables others reference; every writer sets them and unique (user_id, slug) indexes hold them. 254 backend tests (new: migration 0021 on a v3 fixture, slug aliases, swaps and rate limits, username renames, aliases and limits, reads by path and their privacy, the OG 301s, both unsubscribe paths, v4-shaped digest links with no ids, notification joins, shared paths and slugs) and 668 frontend tests (new: the profile's tabs and follow views, Settings, Manage, the link field, canonical URLs on the dump, playlist and profile pages, bare default feed paths) pass, and tsc, eslint, deno check, deno lint, deno fmt and doc:lint are clean; the migration was also run on a copy of the development database. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 16:08:59 +00:00
khannurien
fd0d81fe66 v4: the emails' words live in the catalogs alone and their look is read from the site's stylesheets — every email message was written as i18n._({ id, message }), so its English sat in the template and again in en.po, and api/mail/styles.ts held a copy of each theme's palette, fonts and radii that a test chased. The templates now name each message by its id only (i18n._("email.reset.title", values)) and its English is written in locales/en.po, its French in fr.po; the four labels the settings tab and the unsubscribe page share with the emails read the same ids through @lingui/react's Trans instead of restating the English. i18n:extract runs without --overwrite, which would write the id where the English is, and i18n:compile runs --strict, so a language lacking a message fails the build; it caught email.digest.hl.mentionIn, never translated into French. The new api/mail/theme-tokens.ts reads src/tokens.css and src/themes/<style>.css and resolves a token for one scheme (var(), light-dark(), color-mix(), translucent colors laid over the card), and styles.ts names tokens for every color, GeoCities' rainbow, Melange's spice and shadow, Kanagawa's syntax hues, the font stack and the corner radii, keeping only what is the email's own (type scale, title, frame, divider, avatar rounding). The copies had drifted: GeoCities' stack had lost Marker Felt and its light emails drew the dark rainbow, which now also recolors its dividers on the "auto" scheme. The pills take the site's badge color, now the --color-badge-bg token that the private badge reads too. The Dockerfile copies src/tokens.css and src/themes/ into the runtime image, which reads them at load. The catalogs were re-extracted, having been stale against the code. 240 backend tests (new: the token reader, the dark rainbow; the palette-copy test is gone) and 661 frontend tests pass, and tsc, deno check and doc:lint are clean; sendDueDigests' empty-digest test fails now and then, as it did before. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:21:45 +00:00
khannurien
7960d74c55 v4: weekly digests and monthly recaps go out by email, every email is written once as Markdown in the app's one catalog, and a review of the last two commits removes what they stated twice — users get a weekly digest (what happened to them, what the people and playlists they follow posted, the week's top dumps, the community) and a monthly recap (the month's top dumps and the community), both on by default, in their own style and scheme. digest-service sends what is due every quarter of an hour from digest_schedule (migration 0020), which holds each user's rolling 7- or 30-day period; existing users' first periods are spread over the coming week or month, a digest more than a period late covers the last full one, an empty one isn't sent, and an address like admin@localhost is skipped. Each digest carries a one-click unsubscribe (List-Unsubscribe and List-Unsubscribe-Post, POST /api/users/unsubscribe) and a link to the new /unsubscribe page, which asks before turning that cadence off, since mail scanners open every link; the token (lib/jwt) names the user and the cadence and never expires. The welcome email now says the digests are on. The emails' words moved out of a hand-written api/mail/strings.ts into the Lingui catalogs the interface uses: lingui extracts api/mail/ too, the catalogs moved to a top-level locales/ that both halves read, compile emits ES modules the server imports (api/mail/i18n.ts), messages take ICU plurals, and the labels the settings tab, the unsubscribe page and the emails share are one message each (their French had already drifted apart). Each template writes its email once as Markdown, which is the plain-text part, and EmailKit.render draws its blocks in the style (a lone link is a button, a list item starting with an image a thumbnail row, a numbered list a ranking, --- the footer), so no string is declared for an HTML and a text version; user text is escaped for Markdown, which the welcome body didn't do for usernames, whose _ read as emphasis. paint() set colors before a border-top shorthand that reset them, so every divider was black. The review also gave one home to what was written twice: the locale list (shared/locale.ts, read by lingui.config, the client, the server and its catalog loader), the base style and default scheme (shared/theme.ts), an email's look (lookOf, which digest-service, welcome and reset share), whether email is on (EMAIL_ENABLED, read by email-service and index.html), a setting's column and its default in the digests' SQL (SETTINGS_COLUMNS, DEFAULT_SETTINGS), the unsubscribe token (made once, not parsed back from a URL), AuthProvider's compare-and-store, ThemeProvider's twin style and scheme code (useThemeChoice), the test JWT encoder, and duplicate digest messages. The email palettes stay a flattened copy of the themes' tokens, now under a test that caught Times' secondary and link colors drifting. The Dockerfile copies api/mail and locales so the build extracts the emails' messages and the server finds its catalogs, and deno.json maps @lingui/core for the runtime image. 236 backend tests (new: digest building, sending, catch-up, scheduling and unsubscribing; the Markdown renderer; French from the catalog; palettes against themes; the stagger migration; unsubscribe tokens) and 661 frontend tests (new: the unsubscribe page) pass, and tsc, eslint, doc:lint, deno check, deno lint and deno fmt are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:14:46 +00:00
khannurien
89014639cb v4: settings are stored on the server, starting with the style, the color scheme and the digest switches — every setting lived in the browser's localStorage, so it didn't follow an account to another browser and the server couldn't read it, which the digests need (whether to send each, and in which style). user_settings (migration 0019) holds one row per user who changed anything, the defaults otherwise, and GET/PATCH /api/users/me/settings read and change it (settings-service, model/settings, validators/settings, shared/settings). AuthProvider reads the settings into the session alongside the user, on mount and on focus, so a change made elsewhere arrives; useSettings changes them at once and puts them back if the server refuses. ThemeProvider now sits inside AuthProvider: a signed-in user's choice comes from their settings and is mirrored to localStorage for index.html's first-paint script, a visitor's stays in the browser, and an account with no choice yet takes the one this browser had, once. Until anyone picks a style, the instance's GERBEUR_DEFAULT_STYLE applies, written into index.html's default-style tag along with email-enabled (src/config/instance.ts), and a browser no longer stores the default as if it had been chosen, so changing the instance's style reaches everyone who never picked one. The settings tab gets an Email section with the weekly digest and monthly recap switches, disabled with a note when the server has no SMTP. The style and scheme names moved to shared/theme.ts in the previous commit; the default feed tab and sounds stay in the browser for now. 223 backend tests (new: settings defaults, patches, isolation, refusals and cascade; index.html's new tags) and 658 frontend tests (new: useSettings, the account's theme winning and a browser's carried over, settings read into the session, the digest switches) pass, and tsc, eslint, doc:lint, deno check, deno lint and deno fmt are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 21:20:57 +00:00
khannurien
f20ca83f0c v4: every email is rendered by its template in api/mail/ on one layout that takes the instance's style, and sent one way — the welcome email was built and sent inside the register route and the reset link in its service, each writing Markdown, running marked, and calling a nodemailer wrapper whose cc, bcc, replyTo and send result nothing used. Now api/mail/ holds one template per email (welcome.ts, password-reset.ts) on layout.ts's EmailKit, which draws each block with inline styles in a style from styles.ts: the six site styles' palettes, copied from src/themes/, and what makes each recognisable in a mail client (Brutalist's hard border and shadow, GeoCities' ridge and rainbow rules, Times' underlined ink links, Kanagawa's colored quotes, dates and wordmark, Melange's spice bar). On the "auto" scheme the light palette is inlined and the dark one swapped in by a prefers-color-scheme block; a chosen scheme sends only its own. User-written text is escaped where it lands. strings.ts holds every email's words in English and French. GERBEUR_EMAIL_LOCALE picks the language and GERBEUR_DEFAULT_STYLE the instance's style, which the welcome and reset emails are drawn in; GERBEUR_WELCOME_EMAIL_BODY still replaces the welcome's Markdown, whose blocks now take the style too. email-service sends a rendered email with its recipient and headers, refuses to load when GERBEUR_SMTPS_URL is set without GERBEUR_FROM_EMAIL (sends used to fail one by one), and takes a test transport, through which the tests now read what was sent. The welcome send moves to user-service's sendWelcomeEmail. The style and scheme names move to shared/theme.ts, which the server now needs. README, mod.ts and .env.example describe mail/ and the new variables, and drop the "notifications, invites" email that never existed. 221 backend tests pass (new: the layout's schemes, styles and escaping, both templates, the welcome on sign-up, a reset link from the email that works), and deno check, deno lint, doc:lint, deno fmt, tsc and eslint are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 21:11:54 +00:00
khannurien
5f0cf86fe6 v3: the dump page's "In collections" list only shows the owner's private badge — every card there said "public", which says nothing on a list of playlists holding the dump. PlaylistCard takes privateBadgeOnly, passed through PlaylistCardList, which hides the visibility badge except on a private playlist of the signed-in user's; the other lists of playlists keep both badges. 5 PlaylistCard tests pass (new: privateBadgeOnly), and tsc, eslint, doc:lint and deno fmt are clean.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 53s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 22s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-30 21:19:06 +00:00
khannurien
f3f07ed1e3 v3: every doc opens with the role, then the mechanism, then the details — the docs were written bottom-up: what a thing does first ("Declares the three contexts WSProvider fills…"), details that only make sense once its purpose is known next, and the reason it exists last ("…so that a feed doesn't re-render"), so a reader had nothing to hang the first sentences on. The README's rule caused it, since a module doc "opened with what the module does"; it now opens with the module's role in the app, and the value-side phrasing of a type ("Returned to Y by X") comes once the role is said, with WSContext as the before and after. WSContext.ts is rewritten that way throughout, and so are the docs elsewhere that buried their purpose in a trailing "so that" (the migrations, og, image-proxy, public-address, useFading, useFeedCache, the drafts), opened on a generic verb that hid it ("Holds", "Declares", "Loads", "Defines", "Types": the contexts and providers, the shared types, the models, the hooks), or were label fragments ("The page of a dump…", "A row of tabs."). A function whose verb is its action ("Stores session"), a props or result type, and a protocol message ("Sent by X when Y") already say their role and are left as they are. Only comments changed, and the catalogs only move their references; deno fmt, deno lint and doc:lint are clean.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 49s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 22s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 21:42:08 +00:00
khannurien
2af64721c7 v3: the invite tree spaces every row the same — a child sat 0.2rem under its parent but siblings 0.4rem apart, since each row's spacing was split into top and bottom padding and a last child dropped its bottom half. Every row now has one gap above it (--invite-tree-gap), the connectors run through it from the same variable, and the user link is a block, so a theme's line height can't push the avatar off the connector's center. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 21:28:43 +00:00
khannurien
07ac4bc629 v3: every paged list loads its pages one way, PlaylistDetail is split like the profile, and the comments keep the README's rules — Index, FollowedFeed, useUserDumpFeed, UserPlaylists, UserListPopover, Notifications, Search and the profile's followed users each hand-wrote "fetch the next page, append it, clear loadingMore on failure". A list the API pages is now a PagedList whose next page useLoadMore appends, and a list that loads on its own goes through usePagedList, which is useFetchState (now taking an initial value, as a restored cache, that starts loaded) plus useLoadMore. Pages are read by pageUrl with readPage (a refusal throws) or readFirstPage (a refusal is empty, as fetchFirstPage was), signed-in reads go through authFetch, and useFeedCache's saveState takes the list. Notifications keeps the socket's arrivals as their own list merged at render, as Index does with recentDumps, and marks them read in one effect. PlaylistDetail (963 lines) only puts the pieces together (146): usePlaylist holds the playlist, its live sync and its edits, PlaylistHeader, PlaylistEditForm and PlaylistDumpList are one file each, and LiveDumpCard is the vote wiring DumpCardList and the playlist both use. What it re-invented is shared: the load is useFetchState, a dump going private and back is usePositionAwareSync, deletes and edits are useDumpListSync (which also keeps comment counts, which the page didn't). A removal is sent when its fade ends, or as the page is left, so an undo sends nothing: the removal/re-add/reorder dance, the pending-undo order and the 80 ms debounce of its echoes are gone. A refused reorder goes back to the server's last order and says so, instead of reloading the page. The comments written since the documentation pass broke its rules in places: module docs as a label and a colon ("The "dumps" tab of a profile: …"), noun-phrase fragments on types, fields and constants, stale lists of callers, over-long lines; they are sentences now, and the ones the refactor made wrong (PlaylistDetail as useFading's and FollowPlaylistButton's caller, "the profiles" as usePositionAwareSync's only users) say what is true. 650 frontend and 214 backend tests pass (new: usePagedList, useFetchState's initial, pageUrl/readPage, the deferred removal, a dump returning at its place in a playlist), and tsc, eslint, doc:lint, deno check, deno lint and deno fmt are clean.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 49s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 22s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 21:19:33 +00:00
khannurien
69d9bf899b v3: the profile page is split by one rule, and what it re-invented is shared across the app — UserPublicProfile was 1,949 lines because it held its data, six tabs, seven subcomponents and hand-rolled copies of things the app already had or repeated elsewhere. The page now only picks the tab and puts the pieces together (253 lines); useProfile holds the user and the three lists with their live sync, cache and scroll; each tab and header piece is one file under components/profile/; the followed and invitees tabs load on first open through the new useFetchState, keyed by username, which keeps them across tab switches. Every piece taken out replaces its copies: DumpCardList, which reads the votes from WSContext, is every list of dump cards (the index feeds, followed, search, related dumps, UserDumps, UserUpvoted, the profile), so Index no longer threads vote props into the feeds; PlaylistCardList every list of playlist cards; FeedFooter the eight sentinel/loading-more/end blocks; ListState the loading/error/empty chains; Segmented the settings toggles, RoleSwitcher and the playlist edit form, with SegmentedField built on it; LoadMoreButton the three load-more buttons; NewDumpButton UserDumps' and the profile's; useScrollRestore four copies of the scroll restore, and the profile uses useScrollSave instead of saving its whole cache on every scroll. useFading takes an onFaded callback, which replaces PlaylistDetail's copy of its timers; useUpvotedList is the vote tracking UserUpvoted and the profile each had, and a withdrawn upvote now leaves the list and the cache once faded (it was only hidden, and came back after a tab switch). usePlaylistMemberships is the membership load and toggle AddToPlaylistModal and DumpCreateModal each had. pagedList.ts (PagedList, firstPage, restoredList, fetchFirstPage, withFirst), fetchPublicUser and dataOr are shared by the profile, UserPlaylists and useUserDumpFeed; a refused side list still shows empty, as documented, and only the user fails the page. FollowedFeed's two sections are one local hook, and its readPage is expectOk. A card is its owner's when the item is the signed-in user's, everywhere: the profile's upvoted column marked every dump as the owner's on one's own profile. No message changed; the catalogs only move their references. 643 frontend tests pass (11 new for useFetchState, useFading's onFaded and pagedList), and tsc, eslint, doc:lint, deno check and deno fmt are clean.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 48s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 22s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 20:50:49 +00:00
khannurien
80f3292c8f v3: fix the frontend's findings, and the backend pieces they needed — each by a rule the code now keeps everywhere. Votes and likes go through one useOptimisticToggles in WSProvider: clicks on one item before any ack share one baseline, which a failure or the timeout restores (a second click used to cancel the first one's rollback), only the last ack's count is applied, a votes_update or comment_likes_update waits while this tab has toggles in flight, and a welcome reconciles them against the server's sets (a vote cast offline left its count one too high). lastLikeEvent, which nothing read, is gone. Follows are refetched when the new follows_changed message, sent to the follower's sockets on every follow write, says another tab changed them; a failed load keeps the Follow buttons disabled and is retried 10 s later instead of leaving them disabled for good (success: false) or enabled over empty sets (network error); follow and unfollow resolve to whether they worked, and the button says when they didn't. CategoriesProvider follows the same rule, so a network error no longer turns every category URL into a 404. Every list is kept by one rule: useDumpListSync and usePlaylistListSync never add an item, and usePositionAwareSync remembers the place of any item that leaves a list, whatever took it out, and puts it back when its next event shows it again; with dump_created now setting lastDumpEvent, a dump going private and back returns to its place on another user's profile, which never happened before, and an updated playlist missing from a list is no longer prepended. useFeedCache keys every cache by viewer (the profile pages restored the owner's private dumps to whoever went back) and reads it again when its key changes (another profile in the same page showed the previous one's list). Every response of a list's first page or next pages goes through expectOk (Index, UserPlaylists, useUserDumpFeed and Notifications read body.data unchecked).
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 55s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 22s
Details 
Smaller fixes: the redirects from / and /<slug> keep the navigation state, so the index hides the dump DumpEdit just deleted; Index's mainFetchDone ref, which blocked nothing, is gone; DumpEdit refuses someone else's dump up front, as the API does; the Dump page takes a handed-over dump once and clears it from the history entry, so a reload fetches; a newly upvoted dump is fetched with the viewer's token; the notifications page sends read-all once per notification; the profile's followed and invitees tabs finish loading when left mid-fetch; UserListPopover no longer closes when its own list scrolls; the vote and like counts open their lists for visitors too (a disabled button swallowed the click; they are aria-disabled now); the removal of a playlist's dump forgets it (an edit brought it back), a failed removal puts it back where it was, an undo waits for the removal in flight, and a refused undo says so; PlaylistCreateForm tells whether the dump made it into the new playlist; the cover preview's object URL is revoked; the muted icon has its cross; the bell no longer stays rung after two notifications within 700 ms; a stale invite check can't overwrite a newer one; a stored theme or color scheme is checked; the default feed tab is one store every reader follows, across tabs; the file fields enforce the size they state, derived from DUMP_MAX_FILE_SIZE_BYTES; formatBytes, friendlyFetchError and ImagePicker are translated (Intl names the units), and ImagePicker ignores keys while uploading; visited dumps and playlists keep a week of visits and are parsed once; a chime is claimed under a Web Lock, so two tabs can't both play it; a date in the future reads as "now". Dead code goes: PageShell's centerSlot, overLimit, the exports of isSafeHex and hueFromString, the comment tree's token prop and ChatContext's isChatOpen and closeChat. The French catalogue has the 11 new messages. 214 backend and 632 frontend tests pass; tsc, deno lint, ESLint and doc:lint are clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 20:08:00 +00:00
khannurien
a36a4cac6b v3: fix the backend's correctness findings — each by a rule shared across the code rather than one more special case. Link previews: readTextCapped decoded every page as UTF-8, so a Latin-1 page came out garbled unless it went through the curl fallback, which re-encoded it on its own; pages are now decoded in the charset their Content-Type or <meta> declares, and the curl fallback hands the bytes over as they are. The image proxy served and cached for a day an error answer that claimed an image type, and read a body without Content-Length whole before checking its size; it now refuses an error status and reads through the same capped reader. The icons task's Twemoji fetch gives up after 15 s instead of holding the start up, and extractLinkHref, which only its tests used, is gone. Backlinks matched a linked URL against the raw url column, so https://x.com/ missed a dump posted as https://x.com; they now compare canonical URLs, as reposts do. The four LIKE searches put the query in raw, so % and _ were wildcards and the mention search for "a_b" found "axb"; lib/like.ts escapes them. Every list query now treats a malformed row the same way, a 500 through model/rows.ts's parseRows, where ten of them silently dropped it; the playlist dumps query uses the model's column list rather than a hand-written copy, and the aliased list is derived from the plain one. The vote and like services commit through transaction() instead of their own BEGIN and ROLLBACK, which would have rolled back a caller's transaction. 
Validation: the dump validators refuse the nulls their shared types don't allow (a null title or note on creation, a null URL on edit); a playlist title of spaces is refused and titles are stored trimmed; usernames and emails are checked by the same USERNAME_RE and EMAIL_RE in shared/validation.ts on sign-up, on PATCH /api/users/me (which checked other limits) and in the sign-up form. Mentions are read by shared/mention.ts's mentionedUsernames, which skips code and URLs as Markdown shows them: https://mastodon.social/@bob or a `@bob` in code notified bob; the editor's suggestions derive their trigger from the same pattern instead of a copy. Following an unknown user is a 404 rather than a foreign-key 500, and following a private playlist a 404 rather than a 403 that confirmed it exists. `ws` and `emojibase` are reserved category slugs, since the server answers them before the app. GET /api/users/me no longer turns every error into a 500.

Dead code goes: adminOnlyMiddleware, verifyEmailTransport, listUsers, deleteUser with disconnectUser and the force_logout message only it sent, PATCH /api/notifications/:id/read (with markOneRead), isSlugAvailable, the unreachable 500 in fetchComment, and the `if (!ctx.state.user)` checks behind authMiddleware. 214 backend and 627 frontend tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 19:35:53 +00:00
khannurien
14e3ab125b v3: fix the privacy and security findings of the comment pass — private dumps' broadcasts, mentions and attachments reached people who can't see them, and a few smaller holes. A dump's every write now goes through ws-service's broadcastDumpChange(before, after), which decides the audience: a private dump's creation, edits and deletion reach its owner's sockets alone (its deletion id went to everyone before), going private is dump_deleted for everyone else and dump_updated for the owner (whose other tabs dropped it), going public dump_created for everyone else. Mentions in a private dump, a comment on one, or a private playlist's description notify no one until it goes public (mentions are deduplicated per context, so going public notifies them once), and a moderator's edit of a comment, chat message or playlist mentions in the author's name. A link preview of a private playlist is refused, as one of a private dump already was: the preview is stored in the linking dump, which others may see. Attachments record the type of the resource that links them (migration 0018 backfills it); one linked by a private dump or playlist, or a comment on a private dump, is served only with its owner's media token, which Markdown now adds to attachment URLs, and one whose resource is gone answers 404 and is purged at startup. The Bandcamp provider claimed bandcamp.com.evil.example (the pattern wasn't anchored) and now reuses bandcamp-tralbum's anchored one; the YouTube provider took any site's ?list= for a YouTube playlist and missed m., music., youtube-nocookie and /live/ links, and now shares canonical-url's rules. Sign-in does the same scrypt work for an unknown username as for a wrong password, so timing no longer tells which exist. Usernames and emails are unique regardless of case (migration 0017, which refuses to run over existing clashes and names them): "Bob" and "bob" could both sign up and a mention picked one, emails had no constraint, and a reset request in another case silently sent nothing. The Open Graph tags escape the site name and build their URLs on PUBLIC_URL and FRONTEND_URL instead of the request's origin, which a proxy may rewrite. 
Error statuses follow one rule: APIException takes a status and derives the code from it (BAD_REQUEST 400, UNAUTHORIZED 401, FORBIDDEN 403, NOT_FOUND 404, CONFLICT 409, RATE_LIMITED 429, SERVER_ERROR 500, BAD_GATEWAY 502), so a code can't contradict its status. That fixed real bugs: a wrong current password on change-password answered 401, on which authFetch signs the user out; editing or deleting someone else's comment or chat message answered 401 rather than 403; one invalid dump edit answered 422. A taken username is 409, like a taken email.

Along the way: the database is created from schema.sql by connection.ts when the file doesn't exist, so every task works on a fresh checkout (dev and serve never ran db/init.ts, which is gone); the startup purge compared ISO dates with SQLite's datetime() as text and kept anything from the current UTC day, and now compares ISO cutoffs; the seeded admin's hash comes from lib/password.ts (split out of jwt.ts, which needs the JWT secret) rather than a copy of its format; avatar, thumbnail and playlist-cover uploads go through replaceUploadedFile, which writes beside the old file and renames over it before the record announces it (a failed avatar write lost the old avatar; a thumbnail was announced before it existed); upload.ts derives its size message from MAX_IMAGE_SIZE_BYTES and stops re-exporting the upload folders; ALLOWED_IMAGE_MIMES, which duplicated detectImageMime, is gone; the SPA fallback no longer calls next() after answering; a reply must answer a comment on the same dump (a foreign or unknown parent was a 500 or a cross-dump thread). 211 backend and 628 frontend tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 15:19:07 +00:00
khannurien
0d4e344d77 v3: comments say what the code can't — the documentation pass wanted JSDoc on every export and every field of an exported interface, which filled the code with restatements ("The dump.", "Whether it is loading.", "The props of X.", "Renders the X.") and module docs that told a reader nothing ("The hot and new feeds, and the props every main feed of the index takes."); some were wrong as well as vague. Every comment in api/, shared/ and src/ is rewritten under one rule, now stated in the README's "Comments" section: every module and export still has a JSDoc, and it gives context (who uses it and where), usage (what a caller must know: a precondition, what it leaves to the caller, a side effect) or its place in the chain (where its input comes from, where its output goes); a field, parameter, private function or local gets one only when it adds to its parent's, and context about a whole type goes on the type; a type's doc is phrased from the value's side ("Returned to Y by X when Z", "Passed to Y by X"); full sentences, as short as they can be, no label-colon-list; // goes wherever the code isn't obvious and says why, and the section banners and narration go. Every claim was checked against the code, which turned up some forty comments that were wrong: DumpCard's claimed it was only used by the hot and new feeds, the WebSocket welcome claimed to be a socket's first message (presence comes first), UpdateUserRequest said null removes a description (it keeps it), deletedDumpIds leaves out that it also holds dumps made private, THEME_COLOR claimed to mirror index.html's, and the like. The routers' JSDoc now says where they are mounted and what becomes of a thrown APIException. doc-lint.ts lets a field without a doc through; a module or export without one still fails. No code changes; 204 backend and 628 frontend tests pass, and doc:lint, deno check, deno lint and tsc are clean. 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-29 14:31:14 +00:00
khannurien
42e0356d21 v3: the docs follow deno doc's grain — the index page was a hand-kept list of some 250 links into deno doc's output layout (./api/main.ts/index.html), policed by mod.test.ts; deno doc has no module index of its own beyond all_symbols.html, which lists every module by path with its summary. mod.ts is now a prose overview of the three parts and their layers, linking that list once, and mod.test.ts is gone. Re-exporting everything from mod.ts, as deno doc's tutorial does for a library, was tried and doesn't fit: export * drops the 18 routers (default exports) and makes 17 names ambiguous, and deno 2.9 renders export * as namespaces with the root's module doc and nested ones empty, losing every module doc. A router's page showed one line: its endpoint list sat on the default symbol, whose summary was cut at the colon introducing it; the list is now the module doc of all 18 routers, and the router's JSDoc says so. 22 summaries ended in a colon (the routers, BandcampPlayer, migration 0016, useDumpListSync, usePlaylistListSync) and read cut short wherever deno doc shows a first paragraph alone; they end in a full stop, doc-lint.ts now fails on a colon summary, and the README states the rule. No behaviour changes; 204 backend tests pass.
All checks were successful
Build and Publish Docker Image / build-and-push (push) Successful in 50s
Details
Build and Deploy Static Documentation Site / build-and-push (push) Successful in 21s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-28 19:28:24 +00:00
626 changed files with 49651 additions and 23118 deletions
Expand all files Collapse all files

21
.devcontainer/Dockerfile
View File

@@ -1,3 +1,20 @@
FROM denoland/deno:bin-2.9.0 AS deno FROM denoland/deno:bin-2.9.7 AS deno
FROM mcr.microsoft.com/devcontainers/typescript-node:20 FROM mcr.microsoft.com/devcontainers/typescript-node:26
COPY --from=deno /deno /usr/local/bin/deno COPY --from=deno /deno /usr/local/bin/deno
# The API draws thumbnails and waveforms with ffmpeg, in the workspace and in
# the instances' services alike (compose.yaml), which don't get the features.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& rm -rf /var/lib/apt/lists/*
# Playwright's Firefox and Chromium headless shell, with the system libraries
# they need, for screenshots of the dev instances (`npm:playwright@1.63.0`): a
# script importing another version looks for other builds. They live outside
# the home directory, which a rebuild would otherwise leave without them.
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright
RUN DENO_DIR=/tmp/deno deno run --no-lock -A npm:playwright@1.63.0 install \
--with-deps firefox chromium-headless-shell \
&& chmod -R a+rX /opt/ms-playwright \
&& rm -rf /tmp/deno /var/lib/apt/lists/*
# The services share Deno's cache through a volume (compose.yaml), which Docker
# creates owned as this directory is, so it must exist and be node's.
RUN mkdir -p /home/node/.cache/deno && chown -R node:node /home/node/.cache

57
.devcontainer/compose.yaml Normal file
View File

@@ -0,0 +1,57 @@
# The dev environment: the workspace VS Code attaches to (`dev`), and one
# service per dev instance of gerbeur (dev/instances.ts), which start with it.
# Each instance service runs dev/start.ts for its own instance, which derives
# its ports, URLs, database and uploads from its entry there; a new instance is
# an entry there and a service here.
#
# Every service shares the host's network, as the workspace always did: an
# instance's URLs are localhost ones, which the browser and the other
# instances' servers must both reach.
#
# From the workspace's terminal (docker-outside-of-docker):
# docker compose -p gerbeur-dev logs -f musique
# docker compose -p gerbeur-dev restart gerbeur
# docker compose -p gerbeur-dev stop gerbeur musique # then `deno task dev`
name: gerbeur-dev
x-workspace: &workspace
build:
context: .
dockerfile: Dockerfile
network_mode: host
user: node
working_dir: /workspaces/gerbeur
volumes:
- ..:/workspaces/gerbeur:cached
# Shared, so that each service doesn't download the jsr and npm packages
# again.
- deno-cache:/home/node/.cache/deno
x-instance: &instance
<<: *workspace
restart: unless-stopped
depends_on:
install:
condition: service_completed_successfully
services:
dev:
<<: *workspace
command: sleep infinity
# Installs the packages and draws the icons once, before any instance
# starts, so that the instances don't do it at the same time.
install:
<<: *workspace
command: sh -c "deno install && deno task icons public"
gerbeur:
<<: *instance
command: deno run --env-file -A dev/start.ts gerbeur
musique:
<<: *instance
command: deno run --env-file -A dev/start.ts musique
volumes:
deno-cache:

13
.devcontainer/devcontainer-lock.json
View File

@@ -1,15 +1,20 @@
{ {
"features": { "features": {
"ghcr.io/devcontainers-extra/features/ffmpeg-apt-get:1": { "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"version": "1.0.17", "version": "1.10.1",
"resolved": "ghcr.io/devcontainers-extra/features/ffmpeg-apt-get@sha256:735491c8e9b2236cae727d6d4904663f6a58eb131d8c4b6b18fd0eea3d8c5b71", "resolved": "ghcr.io/devcontainers/features/docker-outside-of-docker@sha256:698f5900fac471e046c6e9d4333fec88c0e8d952c713494fa0712dce80baeca0",
"integrity": "sha256:735491c8e9b2236cae727d6d4904663f6a58eb131d8c4b6b18fd0eea3d8c5b71" "integrity": "sha256:698f5900fac471e046c6e9d4333fec88c0e8d952c713494fa0712dce80baeca0"
}, },
"ghcr.io/khannurien/devcontainer-features/claude-code:1": { "ghcr.io/khannurien/devcontainer-features/claude-code:1": {
"version": "1.2.0", "version": "1.2.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/claude-code@sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9", "resolved": "ghcr.io/khannurien/devcontainer-features/claude-code@sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9",
"integrity": "sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9" "integrity": "sha256:17eb69bfd15ff5206c0bff2945935a4349f7423cabee461a0ba2681d52205db9"
}, },
"ghcr.io/khannurien/devcontainer-features/opencode:1": {
"version": "1.1.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/opencode@sha256:8df3f8c98a104ce7e089ba3e8df39e0d3890753c8dc92e03deb54527d30d3574",
"integrity": "sha256:8df3f8c98a104ce7e089ba3e8df39e0d3890753c8dc92e03deb54527d30d3574"
},
"ghcr.io/khannurien/devcontainer-features/rtk:1": { "ghcr.io/khannurien/devcontainer-features/rtk:1": {
"version": "1.1.0", "version": "1.1.0",
"resolved": "ghcr.io/khannurien/devcontainer-features/rtk@sha256:4a60b3a1d1b64904de3d856c61cdf7e5943e3c74d114609787dc88d9db7c788f", "resolved": "ghcr.io/khannurien/devcontainer-features/rtk@sha256:4a60b3a1d1b64904de3d856c61cdf7e5943e3c74d114609787dc88d9db7c788f",

31
.devcontainer/devcontainer.json
View File

@@ -1,19 +1,12 @@
{ {
"name": "Deno", "name": "gerbeur",
"build": {
"dockerfile": "Dockerfile"
},
// Features to add to the dev container. More info: https://containers.dev/features. // The workspace and the dev instances of gerbeur, one service each: see
// "features": {}, // compose.yaml. The instances start with the container.
"dockerComposeFile": "compose.yaml",
// Use 'forwardPorts' to make a list of ports inside the container available locally. "service": "dev",
// "forwardPorts": [], "workspaceFolder": "/workspaces/gerbeur",
"shutdownAction": "stopCompose",
"runArgs": ["--network=host"],
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "yarn install",
// Configure tool-specific properties. // Configure tool-specific properties.
"customizations": { "customizations": {
@@ -26,9 +19,11 @@
}, },
"features": { "features": {
"ghcr.io/warrenbuckley/codespace-features/sqlite:1": {}, "ghcr.io/warrenbuckley/codespace-features/sqlite:1": {},
"ghcr.io/devcontainers-extra/features/ffmpeg-apt-get:1": {} // The docker CLI, on the host's daemon, for the instances' logs and
// restarts (compose.yaml). Docker's own packages: the base image is Debian
// trixie, which has no moby-cli.
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
"moby": false
}
} }
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
} }

42
.env.example
View File

@@ -48,19 +48,45 @@ GERBEUR_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
# Secret key used to sign JWTs. Generate with: openssl rand -hex 32 # Secret key used to sign JWTs. Generate with: openssl rand -hex 32
GERBEUR_JWT_SECRET= GERBEUR_JWT_SECRET=
# SMTP connection URL used by the email service (notifications, invites, etc.) # SMTP connection URL for outgoing email: the welcome email, password reset
# links and the weekly/monthly digests. Leave empty to send no email at all.
# Example: smtps://username:password@smtp.example.com:465 # Example: smtps://username:password@smtp.example.com:465
GERBEUR_SMTPS_URL= GERBEUR_SMTPS_URL=
# Sender address used in outgoing emails (e.g. no-reply@example.com) # Sender of every email, as an address or "Name <address>"
# Required when GERBEUR_SMTPS_URL is set. # (e.g. gerbeur <no-reply@example.com>).
# Required when GERBEUR_SMTPS_URL is set: the server refuses to start without it.
GERBEUR_FROM_EMAIL= GERBEUR_FROM_EMAIL=
# Markdown body for the account creation welcome email. # Language of every email: en (default) or fr. The interface follows each
# Supports {{username}} and {{site_name}} placeholders. # visitor's browser, which the server never sees, so emails use the instance's.
# Defaults to a built-in template when unset. GERBEUR_EMAIL_LOCALE=en
# Use \n for line breaks in single-line .env values, or use a quoted multiline block.
GERBEUR_WELCOME_EMAIL_BODY="# Welcome to {{site_name}}!\n\nHi **{{username}}**,\n\nYour account has been created successfully. Welcome aboard!" # The instance's style: smooth (default), brutalist, geocities, nyt, kanagawa
# or melange. Visitors and accounts see it until they pick one, and the welcome
# and password reset emails are drawn in it; a digest uses its reader's own
# style. Applied at runtime; changing it needs a restart of both the API and Vite.
GERBEUR_DEFAULT_STYLE=smooth
# How often, in minutes, the /community page's figures are recomputed.
GERBEUR_COMMUNITY_REFRESH_MINUTES=15
# Set to 0 to keep users of other gerbeur instances from signing in here, which
# also signs out those already in until it is set back, or to keep this
# instance's users from signing in to others, which signs them out there.
# GERBEUR_FEDERATION_INCOMING=1
# GERBEUR_FEDERATION_OUTGOING=1
# Set to 1 to let federated sign-in reach other instances on private addresses
# (localhost included) and over plain http, so that two instances on one
# machine can trust each other in development. Never in production.
# GERBEUR_FEDERATION_ALLOW_PRIVATE=1
# Markdown body of the welcome email sent on account creation, replacing the
# built-in one (written in GERBEUR_EMAIL_LOCALE). Supports {{username}} and
# {{site_name}}. Use \n for line breaks in single-line .env values, or a
# quoted multiline block.
# GERBEUR_WELCOME_EMAIL_BODY="# Welcome to {{site_name}}!\n\nHi **{{username}}**,\n\nYour account has been created successfully. Welcome aboard!"
# ── Frontend (Vite) ─────────────────────────────────────────────────────────── # ── Frontend (Vite) ───────────────────────────────────────────────────────────
# These must be prefixed with VITE_ to be exposed to the client bundle. # These must be prefixed with VITE_ to be exposed to the client bundle.

302
API.md Normal file
View File

@@ -0,0 +1,302 @@
# gerbeur's URLs
Every page and every API endpoint follows one of two rules. This file summarizes
them. Each router's module doc in `api/routes/` lists its own endpoints in full,
and a route change updates both. `shared/paths.ts` builds and parses the page
paths.
## The two rules
**Pages.** The path is a hierarchy. `@name` is a user, and `@name@host` a user
whose account lives on another instance. Any other word under the root is a
category or an app page. Under a user, each segment is a fixed tab name, and
under `dumps` or `playlists`, a slug. A view of a page is one more segment. A
default view is the bare path, with no redirect. A category is a segment where
the page is the category's own, a dump category's feed at the root (`/music`)
and a playlist category's under `/playlists` (`/playlists/mixes`); where it
narrows a list on another page, a profile's or a search's, it is `?category=`,
of the kind that list holds. The order of a list of playlists is `?sort=`, the
default order being the bare path, and the playlists page's list of the viewer's
own is `?mine=1`. Switching tabs, a tab's views or a list's order takes the
place of the current history entry; picking a category, or opening another page
or an editor, adds one.
**API.** A resource is `/api/<plural>/<key>[/<noun>]`. Its key is its id, except
a user's, which may also be `@name`. The `@` keeps the two apart, so no regex
sniffing is needed. A read by page path mirrors the page:
`/api/users/@bob/dumps/<slug>`. The caller's own things live under `/api/me`.
Media is a sub-resource. A lookup by another key is a query filter. Each concept
has one word, the same as in pages.
## Pages
| Path | Views | Access | Page |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------------------------- |
| `/` | `/hot`, `/new`, `/journal`, `/follow`, `/follow/playlists`; the bare `/` is the viewer's default tab, the bare `/follow` the users followed | anyone, `/follow` signed in | `Index` |
| `/<category>` | `/<category>/hot`, `/new`, `/journal`, `/follow`, `/follow/playlists` | anyone, `/follow` signed in | `Index` |
| `/@<name>` | the overview: dumps beside upvoted | anyone | `UserPublicProfile` |
| `/@<name>/dumps`, `/upvoted`, `/playlists`, `/invitees` | `?category=` narrows the dumps, the upvoted or the playlists; `?sort=` orders the playlists | anyone | `UserPublicProfile` |
| `/@<name>/follow` | `/@<name>/follow/playlists`; the bare tab lists users | anyone | `UserPublicProfile` |
| `/@<name>/manage` | | `user:manage`, on others' profiles | `UserPublicProfile` |
| `/@<name>/dumps/<slug>` | | anyone; a private dump, its poster | `Dump` |
| `/@<name>/dumps/<slug>/edit` | | its poster, `dump:moderate` | `DumpEdit` |
| `/@<name>/playlists/<slug>` | | anyone; a private playlist, its owner | `PlaylistDetail` |
| `/@<name>/playlists/<slug>/edit` | | its owner, `playlist:moderate` | `PlaylistDetail` |
| `/playlists` | `/playlists/<category>`, a playlist category's; `?sort=new`, `?sort=dumps` order either; `?mine=1` lists the viewer's own, signed in only | anyone | `Playlists` |
| `/search?q=` | `/search/users?q=`, `/search/playlists?q=`; `?category=` narrows the dumps or the playlists | anyone | `Search` |
| `/notifications` | | signed in | `Notifications` |
| `/community` | `/community/monthly`; the bare path is the week | signed in | `Community` |
| `/settings` | | signed in | `Settings` |
| `/manage` | | `category:manage`, `federation:manage` | `Manage` |
| `/login`, `/register?token=` | `/login?next=<path>` comes back to a page | signed out | `UserLogin`, `UserRegister` |
| `/authorize?client_id=…` | | signed in, else via `/login?next=` | `Authorize` |
| `/federation/callback?code=&state=` | | anyone | `FederationCallback` |
| `/reset-password` | `/reset-password?token=`, from the emailed link, sets the new password | anyone | `ResetPassword` |
| `/unsubscribe?token=` | | anyone | `Unsubscribe` |
`/?share_url=` opens the create form on a link shared from Android, `?chat=open`
opens the chat over any page, `?chat=<message id>` on that message, and
`#comment-<id>` scrolls a dump's page to a comment. A signed-in page opened
signed out replaces itself with `/login?next=<the page>`.
## Names
- **Slugs** of categories, dumps and playlists are lowercase letters and digits
in words joined by single hyphens (`shared/slug.ts`). A category's is at most
50 characters, a dump's or a playlist's at most 60.
- **A dump's or a playlist's slug** is made from its title when it is created,
unique among its owner's items of that kind: a clash gets `-2`, `-3`. It
changes only when someone edits it: the edit forms propose `slugify(newTitle)`
when the title changes. Its owner may change it 3 times a day; moderators
(`dump:moderate`, `playlist:moderate`) may edit any slug, at any time.
- **A username** is 1 to 32 letters, digits and underscores, unique in any case.
A page writes it in its stored case, and another case still resolves. Users
can't rename themselves: those with `user:manage` rename anyone, from
`/@<name>/manage`.
- **A user from another instance** is `name@host` here: their name at home and
their home's host, with its port if it has one (`alice@music.example`). Their
name is their home's to change, which leaves the old handle an alias here too.
Mentions write the whole handle. A handle with this instance's own host, as
another instance writes a user from here (`bob@gerbeur.example`), names the
local `bob`, in a path, an API key or a mention.
- **Aliases.** A slug or a name left behind stays an alias for good, which
always names the item or user itself, so redirects never chain. Nobody else
can take one. An item can take back one of its own. Deleting an item frees its
aliases.
- **Redirects.** A page reached by an alias, or by a name in another case, gets
the item from the API under its current path and replaces its URL. The server
answers the same path with a 301 to crawlers and full page loads.
- **Reserved words.** A category can't take a word the root already uses:
`hot new journal follow search login register notifications reset-password
unsubscribe settings manage community playlists authorize federation api ws
assets emojibase`
(`api/lib/reserved-slugs.ts`). Dump and playlist slugs reserve nothing, and no
slug has an `@`.
## API
Every JSON endpoint answers the envelope of `shared/api.ts`:
`{ "success": true, "data": … }`, or
`{ "success": false, "error": { "code",
"message" } }` with the matching HTTP
status. A list answers `{ "items", "hasMore" }` and takes `?page=` (from 1) and
`?limit=` (20 by default, 100 at most). "Signed in" means a session token as
`Authorization: Bearer …`. Media can't send a header, so a private file,
thumbnail, waveform, cover or attachment takes the owner's media token as
`?token=`.
An image the API serves, an avatar, a thumbnail, a cover, an attachment or a
remote image, takes `?w=` and answers a WebP copy scaled down to that width,
snapped up to one of `shared/image.ts`'s (128, 320, 640, 960), and never wider
than the image; without it, the image as it was uploaded, a dump's thumbnail
alone being scaled to the widest by default. `?f=compat` asks for the copy as a
JPEG or PNG, for a mail client that reads no WebP. An image no copy can be made
of goes out as it is. JSON answers of a kilobyte or more are compressed, in
brotli or gzip, for a client that accepts it.
### me
- `GET /api/me` — signed in. The caller's own user.
- `PATCH /api/me` — signed in. Changes the description and email.
- `GET /api/me/media-token` — signed in. A media token.
- `GET`, `PATCH /api/me/settings` — signed in. The caller's settings.
- `PUT /api/me/password` — signed in, with the current one. A new password, and
fresh tokens.
- `PUT /api/me/avatar` — signed in. A new avatar (multipart).
- `GET /api/me/playlists?contains=:dumpId` — signed in. The caller's playlists,
and whether each holds the dump.
- `GET /api/me/follow` — signed in. The ids of the users and playlists followed.
- `PUT`, `DELETE /api/me/follow/users/:id`, `/api/me/follow/playlists/:id` —
signed in. Follows or stops following.
- `GET /api/me/notifications` — signed in. The caller's notifications.
- `POST /api/me/notifications/read` — signed in. Marks them all read.
- `GET /api/me/grants` — signed in. The other instances the caller's account
signs in to.
- `DELETE /api/me/grants/:host` — signed in. Withdraws one, ending the caller's
sessions there.
An account from another instance changes its password, email, avatar and name
there: those endpoints answer it 400.
### users
- `POST /api/users` — with an invite. Signs up.
- `GET /api/users?q=` — anyone. Up to 8 users whose name starts with `q`.
- `GET /api/users/:key` — anyone. A profile.
- `GET /api/users/:key/avatar` — anyone. The avatar.
- `GET /api/users/:key/dumps`, `/upvoted`, `/playlists` — anyone; private items
to their owner only. Their lists; `?category=` narrows one to a category of
its kind, and `?sort=` orders the playlists as `GET /api/playlists` does.
- `GET /api/users/:key/dumps/:slug`, `/playlists/:slug` — anyone; private items
to their owner only. An item by its page path, by an alias too.
- `GET /api/users/:key/follow/users`, `/follow/playlists` — anyone. What they
follow.
- `GET /api/users/:key/invitees` — anyone. The tree of users they invited.
- `GET /api/users/:key/home?path=` — anyone. Redirects to a page of a remote
user's home instance, their profile there by default.
- `PATCH /api/users/:key/role`, `/username` — `user:manage`. Another user's
role, or anyone's name, theirs included.
### dumps
- `POST /api/dumps` — signed in. Posts a link (JSON) or a file (multipart).
- `GET /api/dumps` — anyone. The feed; `?category=<slug>` narrows it, and
`?url=` answers the dumps of the same link instead.
- `GET /api/dumps?follow=users|playlists` — signed in. The feed's `follow` tab:
the dumps of the users, or of the playlists, the caller follows; `?category=`
narrows it too.
- `GET /api/dumps/:id` — anyone who may see it. A dump.
- `GET /api/dumps/:id/voters`, `/related`, `/playlists` — the same. Who upvoted
it, the dumps linked to or from it, the playlists holding it.
- `PUT /api/dumps/:id` — its poster, `dump:moderate`. Title, slug, note,
visibility, categories, link.
- `POST /api/dumps/:id/metadata` — the same. Fetches a link's preview again.
- `DELETE /api/dumps/:id` — the same. Deletes it.
- `GET`, `PUT /api/dumps/:id/file` — the file; a new one, by its poster or
`dump:moderate`.
- `GET`, `PUT`, `DELETE /api/dumps/:id/thumbnail` — the thumbnail; a custom one,
set or removed by its poster or `dump:moderate`.
- `GET /api/dumps/:id/waveform` — anyone who may see it. An audio file's
waveform, `{ "peaks" }`, for the player to seek on.
- `GET`, `POST /api/dumps/:id/comments` — anyone who may see the dump; posting
signed in. Its comments.
### playlists
- `POST /api/playlists` — signed in. Creates a playlist, filed under the
playlist categories it names.
- `GET /api/playlists` — anyone. Every public playlist; `?category=<slug>`
narrows it to a playlist category, and `?sort=updated|new|dumps` orders it:
the last to take a dump first (the default), the newest, or the fullest. The
caller's own private dumps count, and date their playlists, for them alone.
- `GET /api/playlists/:id` — anyone who may see it. A playlist and its dumps.
- `PUT`, `DELETE /api/playlists/:id` — its owner, `playlist:moderate`. Title,
slug, description, visibility, categories; deletion.
- `POST`, `DELETE /api/playlists/:id/dumps/:dumpId` — the same. Adds or removes
a dump.
- `PUT /api/playlists/:id/order` — the same. Reorders its dumps.
- `GET`, `PUT /api/playlists/:id/cover` — the cover; a new one, by its owner or
`playlist:moderate`.
### comments
- `PATCH`, `DELETE /api/comments/:id` — its author, `comment:moderate`. Edits or
deletes a comment.
- `GET /api/comments/:id/likers` — anyone. Who likes it.
### categories
- `GET /api/categories` — anyone. Every category, dump and playlist ones apart:
each has a `kind`, and its slug is unique within its kind.
- `POST /api/categories`, `PUT`, `DELETE /api/categories/:id` —
`category:manage`. Changes them.
### community
- `GET /api/community?period=weekly|monthly` — signed in. The site's top dumps,
new members, new public playlists and totals over the last week (by default)
or month, as of the server's last refresh.
- `GET /api/community/feed` — signed in. The community feed: the 12 latest
things done in public, newest first (a public dump posted, a public playlist
created, a comment or an upvote on a public dump, someone joining), and the
last week's totals and most upvoted dump, as of the server's last refresh. A
dump has one comment event and one upvote event at most, its latest. Every
reader gets the same feed.
### chat
- `GET /api/chat?before=&after=` — signed in. A page of messages. The chat
itself goes over `/ws`.
- A message from a moderator or an admin (`chat:everyone`) that says `@everyone`
notifies every user who doesn't have the chat open, and comes flagged
`mentionsEveryone`. No account may be named `everyone`.
### invites
- `POST /api/invites` — signed in. A new invite.
- `GET /api/invites/:token` — anyone. Whether it still works.
### sessions, password-resets, unsubscribe
- `POST /api/sessions` — anyone. Signs in.
- `POST /api/password-resets` — anyone. Emails a reset link.
- `PUT /api/password-resets/:token` — with the link's token. A new password.
- `POST /api/unsubscribe?token=` — with a digest's token. Turns it off. Its v3
path, `POST /api/users/unsubscribe`, stays for good, since sent emails'
`List-Unsubscribe` headers name it.
### federation, oauth, .well-known
Federated sign-in (`shared/federation.ts`): a user signs in here with their
account on another gerbeur instance, their home, by OpenID Connect's
authorization code flow with PKCE, with tokens signed by each instance's Ed25519
key. With `GERBEUR_FEDERATION_INCOMING=0`, `POST /api/federation/logins` and
`/sessions` answer 403; with `GERBEUR_FEDERATION_OUTGOING=0`,
`GET /api/oauth/clients` and `POST /api/oauth/codes` do. An instance is named by
its issuer, its API's origin, whose host is the one in its users' handles. Where
`instance` is asked for, the origin of its pages works too: their
`/.well-known/openid-configuration` names the issuer, whose own metadata must
name those pages back as where its `/authorize` is.
- `GET /.well-known/openid-configuration`, `/.well-known/jwks.json` — anyone.
The instance's metadata and public keys, bare JSON: the one exception to the
`/api/` rule, at the paths the standard fixes.
- `POST /api/federation/logins` — anyone. Starts a sign-in with an account on
another instance; one neither trusted nor already home to someone here needs
an invite.
- `POST /api/federation/sessions` — anyone. Finishes it, as a sign-in.
- `POST /api/federation/events` — another instance. Its signed news of one of
its users here: a new name, email or avatar, or the end of their sessions. A
4xx tells it to give up on the event, a 429 or a 5xx to try again.
- `GET /api/federation/peers` — anyone the trusted instances,
`federation:manage` every listed one.
- `PUT /api/federation/peers`, `DELETE /api/federation/peers/:host` —
`federation:manage`. Trusts or blocks an instance, or takes it off the lists.
- `GET /api/oauth/clients?client_id=` — signed in. The instance asking to sign
the caller in.
- `POST /api/oauth/codes` — signed in. The caller's consent, and a code.
- `POST /api/oauth/tokens` — another instance, form-encoded. Trades a code for
an identity token, bare JSON.
### search
- `GET /api/search?q=` — anyone. A page of dumps, and up to 20 users and
playlists. It finds both kinds of item at once, so `?category=` narrows the
dumps and `?playlistCategory=` the playlists.
### attachments
- `POST /api/attachments` — signed in. Stores an image.
- `GET /api/attachments/:id` — anyone who may see the text holding it.
### remote
- `GET /api/remote/preview?url=` — signed in. A link's preview.
- `GET /api/remote/image?url=` — anyone. A remote image through this site.
- `GET /api/remote/bandcamp?url=` — anyone. A Bandcamp page's tracks.
- `GET /api/remote/youtube?list=` — anyone. A YouTube playlist's tracklist.
### /ws
- `GET /ws?token=` — anyone; a session token signs it in. The WebSocket of live
updates and the chat (`shared/ws.ts`).

7
Dockerfile
View File

@@ -11,6 +11,9 @@ COPY index.html ./
COPY public/ ./public/ COPY public/ ./public/
COPY src/ ./src/ COPY src/ ./src/
COPY shared/ ./shared/ COPY shared/ ./shared/
# The build extracts and compiles the catalogs, the emails' messages included.
COPY locales/ ./locales/
COPY api/mail/ ./api/mail/
# VITE_API_* are only needed when the API lives on a different host than the frontend. # VITE_API_* are only needed when the API lives on a different host than the frontend.
ARG VITE_API_PROTOCOL ARG VITE_API_PROTOCOL
@@ -35,6 +38,10 @@ RUN deno install
COPY api/ ./api/ COPY api/ ./api/
COPY shared/ ./shared/ COPY shared/ ./shared/
COPY --from=builder /app/dist/ ./dist/ COPY --from=builder /app/dist/ ./dist/
COPY --from=builder /app/locales/ ./locales/
# The emails read their colors, fonts and radii from the site's stylesheets.
COPY src/tokens.css ./src/
COPY src/themes/ ./src/themes/
# Persistent data: database and uploaded/generated files must be mounted as volumes. # Persistent data: database and uploaded/generated files must be mounted as volumes.
VOLUME ["/app/api/sql", "/app/api/uploads"] VOLUME ["/app/api/sql", "/app/api/uploads"]

219
README.md
View File

@@ -1,6 +1,11 @@
# gerbeur # gerbeur
A small invite-only social platform for sharing links and files. Users can post URLs and media (YouTube, SoundCloud, Bandcamp, images, audio, video, …), vote, comment, follow each other, build playlists, and search content. A real-time WebSocket layer handles live presence, vote counts, and notifications. The UI is localized (English and French) and ships multiple visual themes. A small invite-only social platform for sharing links and files. Users can post
URLs and media (YouTube, SoundCloud, Bandcamp, images, audio, video, …), vote,
comment, follow each other, build playlists, and search content. A real-time
WebSocket layer handles live presence, vote counts, and notifications, and
weekly and monthly email digests sum up what happened. The UI is localized
(English and French) and ships multiple visual themes.
## Stack ## Stack
@@ -26,20 +31,40 @@ cp .env.example .env
### Run ### Run
In the devcontainer, the dev instances listed in
[`dev/instances.ts`](dev/instances.ts) start with the container, each a service
of [`.devcontainer/compose.yaml`](.devcontainer/compose.yaml): `gerbeur` at
[http://localhost:3000](http://localhost:3000), `musique` at
[http://localhost:3001](http://localhost:3001). They run side by side so that
signing in from one instance to another can be tried on one machine, and from
the workspace's terminal:
```sh ```sh
deno task dev docker compose -p gerbeur-dev logs -f musique # follow an instance
docker compose -p gerbeur-dev restart gerbeur # restart one
``` ```
This starts both the API server (port 8000, with file watching) and the Vite dev server (port 3000) concurrently. Outside the devcontainer, or with those services stopped, `deno task dev` starts
them all, and `deno task dev gerbeur` only those named.
Open [http://localhost:3000](http://localhost:3000). On first run a default `admin` / `admin` account is created — change the password immediately. Each instance has an API server, restarted on change, and a Vite server, and its
output is prefixed with its name. They follow one rule
([`dev/start.ts`](dev/start.ts)): instance `<name>` keeps its database in
`api/sql/<name>.db` and its uploads in `api/uploads/<name>/`, reads `.env`, and
takes what sets it apart (ports, site name, emoji, style) from its entry. A new
instance is an entry in `dev/instances.ts` and a service in `compose.yaml`.
On first run each instance creates a default `admin` / `admin` account — change
the password immediately. To let one instance's users sign in to another, trust
it on the other's `/manage`.
### Environment variables ### Environment variables
See [`.env.example`](.env.example) for the full list with descriptions. Key variables: See [`.env.example`](.env.example) for the full list with descriptions. Key
variables:
| Variable | Description | Default | | Variable | Description | Default |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| `GERBEUR_JWT_SECRET` | JWT signing secret — **required**, generate with `openssl rand -hex 32` | — | | `GERBEUR_JWT_SECRET` | JWT signing secret — **required**, generate with `openssl rand -hex 32` | — |
| `GERBEUR_PUBLIC_URL` | Public-facing URL of the server (no trailing slash) — used for CORS, WebSocket origin, email links, OG URLs | `http://localhost:GERBEUR_PORT` | | `GERBEUR_PUBLIC_URL` | Public-facing URL of the server (no trailing slash) — used for CORS, WebSocket origin, email links, OG URLs | `http://localhost:GERBEUR_PORT` |
| `GERBEUR_PORT` | Internal port Oak listens on | `8000` | | `GERBEUR_PORT` | Internal port Oak listens on | `8000` |
@@ -47,9 +72,15 @@ See [`.env.example`](.env.example) for the full list with descriptions. Key vari
| `GERBEUR_FRONTEND_URL` | Frontend base URL for email links and CORS; auto-added to allowed origins — the only variable needed when the frontend runs on a separate host | `GERBEUR_PUBLIC_URL` | | `GERBEUR_FRONTEND_URL` | Frontend base URL for email links and CORS; auto-added to allowed origins — the only variable needed when the frontend runs on a separate host | `GERBEUR_PUBLIC_URL` |
| `GERBEUR_ALLOWED_ORIGINS` | Comma-separated extra origins for CORS/WebSocket; `PUBLIC_URL` and `FRONTEND_URL` are always included — typically only needed in dev for the Vite server | `""` (empty) | | `GERBEUR_ALLOWED_ORIGINS` | Comma-separated extra origins for CORS/WebSocket; `PUBLIC_URL` and `FRONTEND_URL` are always included — typically only needed in dev for the Vite server | `""` (empty) |
| `GERBEUR_SITE_NAME` | Site name used in the browser tab, app header, OG meta tags, and emails | `gerbeur` | | `GERBEUR_SITE_NAME` | Site name used in the browser tab, app header, OG meta tags, and emails | `gerbeur` |
| `GERBEUR_SMTPS_URL` | SMTPS connection URL for outgoing email (`smtps://user:pass@host:465`) | unset | | `GERBEUR_SMTPS_URL` | SMTPS connection URL for outgoing email (`smtps://user:pass@host:465`); unset sends no email | unset |
| `GERBEUR_FROM_EMAIL` | Sender address for outgoing emails — required when `GERBEUR_SMTPS_URL` is set | unset | | `GERBEUR_FROM_EMAIL` | Sender of every email — required when `GERBEUR_SMTPS_URL` is set, or the server won't start | unset |
| `GERBEUR_EMAIL_LOCALE` | Language of every email: `en` or `fr` | `en` |
| `GERBEUR_DEFAULT_STYLE` | The instance's style (`smooth`, `brutalist`, `geocities`, `nyt`, `kanagawa`, `melange`): the site's until one is picked, and the welcome and reset emails' | `smooth` |
| `GERBEUR_WELCOME_EMAIL_BODY` | Markdown body for the account-creation welcome email; supports `{{username}}` and `{{site_name}}` | built-in template | | `GERBEUR_WELCOME_EMAIL_BODY` | Markdown body for the account-creation welcome email; supports `{{username}}` and `{{site_name}}` | built-in template |
| `GERBEUR_FEDERATION_INCOMING` | `0` keeps users of other instances from signing in here, and signs out those already in | `1` |
| `GERBEUR_FEDERATION_OUTGOING` | `0` keeps this instance's users from signing in to other instances, and signs them out there | `1` |
| `GERBEUR_FEDERATION_ALLOW_PRIVATE` | `1` lets federated sign-in reach instances on private addresses and over http — for two instances on one machine, never in production | unset |
| `GERBEUR_COMMUNITY_REFRESH_MINUTES` | How often, in minutes, the `/community` page's figures are recomputed | `15` |
| `VITE_API_PROTOCOL` | API protocol baked into the frontend bundle (see [Production](#production)) | `http` | | `VITE_API_PROTOCOL` | API protocol baked into the frontend bundle (see [Production](#production)) | `http` |
| `VITE_API_HOSTNAME` | API hostname baked into the frontend bundle | `localhost` | | `VITE_API_HOSTNAME` | API hostname baked into the frontend bundle | `localhost` |
| `VITE_API_PORT` | API port baked into the frontend bundle | `8000` | | `VITE_API_PORT` | API port baked into the frontend bundle | `8000` |
@@ -58,7 +89,10 @@ See [`.env.example`](.env.example) for the full list with descriptions. Key vari
### Docker (recommended) ### Docker (recommended)
The standard deployment runs API and frontend in a single container. The API server (Oak) serves the compiled frontend as static files, so both share the same origin — no `VITE_API_*` build args needed. Set `GERBEUR_PUBLIC_URL` to the externally-visible URL; it is automatically allowed for HTTP/WebSocket requests. The standard deployment runs API and frontend in a single container. The API
server (Oak) serves the compiled frontend as static files, so both share the
same origin — no `VITE_API_*` build args needed. Set `GERBEUR_PUBLIC_URL` to the
externally-visible URL; it is automatically allowed for HTTP/WebSocket requests.
```sh ```sh
docker build -t gerbeur . docker build -t gerbeur .
@@ -77,12 +111,17 @@ docker run -d \
``` ```
The two volumes are required for persistence: The two volumes are required for persistence:
- `gerbeur-db` — SQLite database (`api/sql/gerbeur.db`), initialized automatically on first run
- `gerbeur-db` — SQLite database (`api/sql/gerbeur.db`), initialized
automatically on first run
- `gerbeur-uploads` — user-uploaded files (`api/uploads/`) - `gerbeur-uploads` — user-uploaded files (`api/uploads/`)
#### Separate API and frontend (optional) #### Separate API and frontend (optional)
If you need to run the API on a different host than the frontend, pass the API location as build args so it gets baked into the frontend bundle. In that setup, add the frontend origin to `GERBEUR_ALLOWED_ORIGINS` so cross-origin HTTP/WebSocket requests are accepted: If you need to run the API on a different host than the frontend, pass the API
location as build args so it gets baked into the frontend bundle. In that setup,
add the frontend origin to `GERBEUR_ALLOWED_ORIGINS` so cross-origin
HTTP/WebSocket requests are accepted:
```sh ```sh
docker build \ docker build \
@@ -94,7 +133,8 @@ docker build \
### Reverse proxy ### Reverse proxy
Put a reverse proxy (nginx, Caddy, …) in front of the container to handle TLS. Forward everything to port 8000. Example Caddyfile: Put a reverse proxy (nginx, Caddy, …) in front of the container to handle TLS.
Forward everything to port 8000. Example Caddyfile:
``` ```
example.com { example.com {
@@ -102,47 +142,87 @@ example.com {
} }
``` ```
### Several instances
Each instance is its own deployment, with its own database, uploads, admins and
domain, but its users can sign in to the others with their account, and the
other way round. A user from `music.example` is `@alice@music.example` on the
instances they sign in to. Their password, email, avatar and name stay on their
home instance, and the others follow its changes.
Every instance is ready for it: nothing to configure, as long as
`GERBEUR_PUBLIC_URL` is the instance's https origin (no path) and the instances
can reach each other. Users and admins may name an instance by the address of
its API or of its pages: when the frontend runs on a host of its own
(`GERBEUR_FRONTEND_URL`), that host must pass `/.well-known/` on to the API, as
the Vite dev server does. A handle carries the API's host. Either way can be
turned off: `GERBEUR_FEDERATION_INCOMING=0` takes no users of other instances,
and `GERBEUR_FEDERATION_OUTGOING=0` lets no user sign in elsewhere. An admin
decides who comes in, on `/manage`:
- users of a **trusted** instance sign in directly, from a button on the login
page;
- users of a **blocked** instance can't, and those already in are signed out;
- users of **any other** instance join with an invite, from the register page,
and sign in without one afterwards.
`deno task dev` runs several instances on one machine (see [Run](#run)), with
`GERBEUR_FEDERATION_ALLOW_PRIVATE=1`, which lets them reach each other over
plain http on localhost. Never set it in production. The first, `gerbeur`, takes
no users of the others, whose users sign in to each other.
## Project structure ## Project structure
``` ```
dev/ # The dev instances (instances.ts) and their launcher (start.ts), for `deno task dev`
shared/ # Imported by both halves: API and socket types, limits, permissions shared/ # Imported by both halves: API and socket types, limits, permissions
test/ # Tests of shared/ (Deno): test/<module>.test.ts test/ # Tests of shared/ (Deno): test/<module>.test.ts
locales/ # Lingui catalogues (en, fr): the interface's and the emails' messages
api/ api/
main.ts # Entry point — Oak application, middleware, routes main.ts # Entry point — Oak application, middleware, routes
config.ts # Environment variables config.ts # Environment variables
middleware/ # Oak middleware: errors, auth, OG tags middleware/ # Oak middleware: errors, auth, OG tags
routes/ # One router per URL prefix: HTTP + WebSocket, no SQL routes/ # One router per API resource: HTTP + WebSocket, no SQL
services/ # Business logic and every SQL query services/ # Business logic and every SQL query
remote/ # Everything that fetches from other sites remote/ # Everything that fetches from other sites
mail/ # One template per email, and the layout they share
providers/ # Link metadata providers (YouTube, SoundCloud, Bandcamp, …) providers/ # Link metadata providers (YouTube, SoundCloud, Bandcamp, …)
model/ # Per table: row type, row guard, row ↔ API mappers model/ # Per table: row type, row guard, row ↔ API mappers
validators/ # Per domain: request body guards validators/ # Per domain: request body guards
lib/ # Helpers with no SQL and no outbound fetch: JWT, slugs, files, … lib/ # Helpers with no SQL and no outbound fetch: JWT, slugs, files, …
db/ db/
connection.ts # The database handle and transaction() connection.ts # The database handle and transaction(); creates a new database
startup.ts # Housekeeping run when the database opens startup.ts # Housekeeping run when the database opens
schema.sql # Database schema schema.sql # Database schema
init.ts # First-run database initialisation
migrate.ts # Migration runner migrate.ts # Migration runner
migrations/ # Versioned schema migrations migrations/ # Versioned schema migrations
test/ # Backend tests (Deno): test/<layer>/<module>.test.ts test/ # Backend tests (Deno): test/<layer>/<module>.test.ts
sql/ sql/
gerbeur.db # SQLite database (not committed) gerbeur.db # SQLite database (not committed); <name>.db per dev instance
uploads/ # User-uploaded files — avatars, dumps, thumbnails, … (not committed) uploads/ # User-uploaded files — avatars, dumps, thumbnails, … and scaled/, their copies made on request (not committed); <name>/ per dev instance
src/ # React frontend (Vite) src/ # React frontend (Vite)
config/ # API base URL, feed tabs, player mode config/ # API base URL, default feed tab, player mode, instance settings
pages/ # One component per route, nothing else pages/ # One component per route, nothing else
components/ # React components only (.tsx) components/ # React components only (.tsx)
contexts/ # FooContext.ts + FooProvider.tsx pairs contexts/ # FooContext.ts + FooProvider.tsx pairs
hooks/ # Every hook (useFoo.ts) hooks/ # Every hook (useFoo.ts)
utils/ # Every non-React module: pure helpers, storage, API glue utils/ # Every non-React module: pure helpers, storage, API glue
test/ # Frontend tests (Vitest): test/<layer>/<domain>/<Module>.test.tsx test/ # Frontend tests (Vitest): test/<layer>/<domain>/<Module>.test.tsx
locales/ # Lingui message catalogues (en, fr)
themes/ # Per-theme CSS files themes/ # Per-theme CSS files
i18n.ts # Lingui runtime setup i18n.ts # Lingui runtime setup
public/ # Static assets (favicon, manifest, service worker) public/ # Static assets (favicon, manifest, service worker)
``` ```
### URLs
Every page and API path follows one of two rules, which [API.md](API.md) states,
along with every page, the slug and username rules and every endpoint. Pages are
a hierarchy under `@name` for users and plain words for categories and app pages
(`/@bob/dumps/never-gonna`, `/music/hot`), built and parsed by
`shared/paths.ts`. API resources are `/api/<plural>/<key>`, the caller's own
under `/api/me`. A route change updates its router's module doc and API.md
together.
Whatever crosses the wire is defined once, in `shared/`: the API envelope Whatever crosses the wire is defined once, in `shared/`: the API envelope
(`shared/api.ts`), each domain's types and request bodies (`shared/dump.ts`, (`shared/api.ts`), each domain's types and request bodies (`shared/dump.ts`,
`shared/user.ts`, …), the socket protocol (`shared/ws.ts`), the validation `shared/user.ts`, …), the socket protocol (`shared/ws.ts`), the validation
@@ -154,38 +234,93 @@ The backend is split by layer, one kind of module per folder, and each file is
named after its domain: the dumps are `routes/dumps.ts`, named after its domain: the dumps are `routes/dumps.ts`,
`services/dump-service.ts`, `model/dump.ts` and `validators/dump.ts`. Routes `services/dump-service.ts`, `model/dump.ts` and `validators/dump.ts`. Routes
parse and answer, services hold the logic and the SQL, `remote/` is the only parse and answer, services hold the logic and the SQL, `remote/` is the only
place that fetches from other sites. Tests mirror the source one file per place that fetches from other sites, and `mail/` renders each email that
module, so the tests of `services/chat-service.ts` are in `services/email-service.ts` sends. Tests mirror the source one file per module,
so the tests of `services/chat-service.ts` are in
`test/services/chat-service.test.ts`. `test/services/chat-service.test.ts`.
All three parts — `api/`, `shared/` and `src/` — are documented for `deno doc`, ### Comments
the same way in every file. Each module opens with a `/** … @module */` block
saying what it is for. Every export has a JSDoc comment, and so does every field A comment says what the code can't. If deleting it would lose nothing, it is the
of an exported interface; a comment that describes a declaration is always `/** wrong comment: on an export, rewrite it; anywhere else, delete it.
*/`, and `//` is only for notes on the code inside. Functions open with a verb
("Returns…", "Checks…", "Renders…" for a component), everything else with a noun **Every module and every export has a JSDoc** (`/** */`). Each gives at least
phrase. Parameters are named in the prose, in backticks. `{@link}` points at one of:
symbols the file already has in scope. A function that can throw to its caller
says so with `@throws`: for an `APIException`, the statuses and when. A router's 1. **Context**: who uses it and where (a page and its route, a caller, a
JSDoc lists its endpoints, one line each: method and path, who may call it, what WebSocket message).
it does; a page's names its route and who may see it. 2. **Usage**: what a caller must know to use it correctly, such as a
precondition, an invariant, what it leaves to the caller, or a side effect.
Add an example only when the call isn't obvious.
3. **Place in the chain**: where its input comes from and where its output goes.
**A doc opens with the role, then the mechanism, then the details.** Its first
sentence says what part of the app the thing serves or what problem it solves,
so a reader who has never seen it knows why it exists. How it does that (what
feeds it, what reads it, how it's split) comes next, and the details last. Each
sentence makes sense given only the ones before it. Not "Declares the three
contexts `WSProvider` fills from the WebSocket. They are split so that a feed
doesn't re-render with every chat message", which leaves the reason for last,
but "Live updates from the server reach the pages through three contexts, split
so that a feed doesn't re-render with every chat message. `WSProvider` fills
them from its WebSocket."
Once the role is said, a type's doc is phrased from the value's side, naming the
producer and the consumer: "Returned to Y by X when Z", "Passed to Y by X". It
is never "What X returns to Y" or "What X passes to Y".
Don't pack a doc into a label followed by a colon and a list ("X: one thing,
plus some other thing"). Write sentences with a subject and a verb.
When the obvious comment is useless, write the useful one: the context is always
there. A **field** (of an interface, or a parameter) gets a doc only when it
adds something its parent's doc doesn't. Context that concerns the whole type
goes on the type, not on one of its fields. The same goes for private functions
and locals. None of these count as a doc:
- a restatement of the name or type (`/** The dump. */`, "Whether it is
loading.", "The props of X.", "Renders the X.");
- a description of what the body, the JSX or the SQL plainly does;
- a list that goes stale (every caller of a widely used component): name the
kind of caller instead.
When a comment is there, it is as short as it can be, often one sentence, and it
is written in full sentences. Labels and fragments ("Upvotes on dumps.") don't
count. A module doc is a few lines at most. It opens with the module's role in
the app, then says how it plays it, who calls it and what it hands back, and it
doesn't repeat what its exports' docs say. `@throws` stays on whatever throws to
its caller, with the statuses of an `APIException` and when each happens. A
router's module doc lists its endpoints (method, path, who may call it), since
the code states them nowhere in one place. A summary never ends in a colon: the
reference shows it alone.
**`//` comments** go wherever the code isn't obvious. They explain why, not
what: a workaround, an ordering constraint, a surprising limitation, a security
reason, an event handler that must not bubble.
**Every claim is checked against the code** (grep the callers). A wrong comment
is worse than none.
### Reference
In the frontend, every exported function states its return type, so the In the frontend, every exported function states its return type, so the
reference can show it: a component returns `React.JSX.Element` (or `| null`), a reference can show it: a component returns `React.JSX.Element` (or `| null`), a
hook returning several things returns a documented `Use<Name>Result` interface. hook returning several things returns a documented `Use<Name>Result` interface.
A component that takes props takes an exported `<Name>Props` interface, each A component that takes props takes an exported `<Name>Props` interface. React's
prop documented. React's types are used through the global `React` namespace types are used through the global `React` namespace rather than a default import
rather than a default import of `react`, which `deno doc` can't follow. of `react`, which `deno doc` can't follow.
`deno task doc` builds the HTML reference into `docs/` from every module of `deno task doc` builds the HTML reference into `docs/` from every module of
`api/`, `shared/` and `src/` but the tests (and the generated `src/locales/`). `api/`, `shared/` and `src/` but the tests. Its index page is the module doc of
Its index page is the module doc of `mod.ts` at the root, which introduces the `mod.ts` at the root, which introduces the repository's three parts and their
repository's three parts and links every documented module (`mod.test.ts` checks layers in prose; `deno doc` builds the list of every module itself, grouped by
it), and the `docs` workflow publishes it on every push to `main` that touches path. The `docs` workflow publishes it on every push to `main` that touches
them. `deno task doc:lint` checks the rules the linter can see, through them. `deno task doc:lint` checks the rules the linter can see, and that no
`doc-lint.ts`: `deno doc` doesn't document npm packages, so it takes a type summary ends in a colon, through `doc-lint.ts`. Two kinds of diagnostic are let
through. A field without a doc passes, since a field gets one only when it adds
to its type's. `deno doc` doesn't document npm packages, so it takes a type
imported from one (react-hook-form's `FieldValues`, say) for a private one, and imported from one (react-hook-form's `FieldValues`, say) for a private one, and
those references alone are let through. those references pass too.
Each of the frontend's `components/`, `contexts/`, `hooks/` and `utils/` is Each of the frontend's `components/`, `contexts/`, `hooks/` and `utils/` is
split into the same domain folders — `auth`, `category`, `chat`, `comment`, split into the same domain folders — `auth`, `category`, `chat`, `comment`,

230
api/config.ts
View File

@@ -1,106 +1,149 @@
/** /**
* The server's settings: those read from `GERBEUR_*` environment variables, * Lets a deployment configure the server through the `GERBEUR_*` environment
* with their defaults, and the fixed limits the backend applies. * variables, which it reads once at import, and holds the fixed limits the
* backend applies. The deno tasks load `.env` with `--env-file`, so changing a
* setting needs a restart.
* *
* @module * @module
*/ */
import { BASE_STYLE, isStyleName, type StyleName } from "../shared/theme.ts";
import { isLocale, type Locale, SOURCE_LOCALE } from "../shared/locale.ts";
/** The port the server listens on (`GERBEUR_PORT`, default 8000). */ /** The port the server listens on (`GERBEUR_PORT`, default 8000). */
export const PORT: number = Number(Deno.env.get("GERBEUR_PORT")) || 8000; export const PORT: number = Number(Deno.env.get("GERBEUR_PORT")) || 8000;
/** /**
* The SMTP server mail is sent through, as an `smtps://` URL * The `smtps://` URL of the server `email-service.ts` sends mail through
* (`GERBEUR_SMTPS_URL`). Empty disables email. * (`GERBEUR_SMTPS_URL`). When it is empty, no email is sent: sign-up sends no
* welcome, a password reset request does nothing, and there are no digests.
*/ */
export const SMTPS_URL: string = Deno.env.get("GERBEUR_SMTPS_URL")?.trim() || export const SMTPS_URL: string = Deno.env.get("GERBEUR_SMTPS_URL")?.trim() ||
""; "";
/** The sender address of outgoing mail (`GERBEUR_FROM_EMAIL`). */ /**
* Whether the instance sends email, which is whether {@link SMTPS_URL} is
* set. `email-service.ts` opens its transport by it, and index.html tells the
* client, whose settings show the digest switches disabled without it.
*/
export const EMAIL_ENABLED: boolean = SMTPS_URL.length > 0;
/**
* The `From` address of every email (`GERBEUR_FROM_EMAIL`), as an address or
* as `Name <address>`. `email-service.ts` refuses to load without it when
* {@link SMTPS_URL} is set.
*/
export const FROM_EMAIL: string = Deno.env.get("GERBEUR_FROM_EMAIL")?.trim() || export const FROM_EMAIL: string = Deno.env.get("GERBEUR_FROM_EMAIL")?.trim() ||
""; "";
const DEFAULT_WELCOME_EMAIL_BODY = `# Welcome to {{site_name}}!
Hi **{{username}}**,
Your account has been created successfully. Welcome aboard!`;
/** /**
* The Markdown body of the welcome email (`GERBEUR_WELCOME_EMAIL_BODY`), with * The Markdown of the welcome email sent on registration
* `{{site_name}}` and `{{username}}` placeholders. * (`GERBEUR_WELCOME_EMAIL_BODY`), or null for the built-in one in the
* {@link EMAIL_LOCALE}. `api/mail/welcome.ts` replaces `{{site_name}}` and
* `{{username}}` in it.
*/
export const WELCOME_EMAIL_BODY: string | null =
Deno.env.get("GERBEUR_WELCOME_EMAIL_BODY")?.trim() || null;
/**
* The language of every email (`GERBEUR_EMAIL_LOCALE`). The interface picks
* each visitor's language in their browser, which the server never learns, so
* the emails take the instance's. A value that isn't a locale means the
* source one, English.
*/
export const EMAIL_LOCALE: Locale = (() => {
const locale = Deno.env.get("GERBEUR_EMAIL_LOCALE")?.trim();
return isLocale(locale) ? locale : SOURCE_LOCALE;
})();
/**
* The instance's style (`GERBEUR_DEFAULT_STYLE`). The site shows it until a
* visitor or their account picks one, through index.html, and the emails that
* aren't a user's own (the welcome and the password reset) are drawn in it. A
* value that isn't a style means {@link BASE_STYLE}.
*/
export const DEFAULT_STYLE: StyleName = (() => {
const style = Deno.env.get("GERBEUR_DEFAULT_STYLE")?.trim();
return isStyleName(style) ? style : BASE_STYLE;
})();
/**
* The key lib/jwt.ts signs and verifies tokens with (`GERBEUR_JWT_SECRET`).
* The server refuses to start without it.
*/ */
export const WELCOME_EMAIL_BODY: string =
Deno.env.get("GERBEUR_WELCOME_EMAIL_BODY")?.trim() ||
DEFAULT_WELCOME_EMAIL_BODY;
/** The key tokens are signed with (`GERBEUR_JWT_SECRET`). Required. */
export const JWT_SECRET: string = Deno.env.get("GERBEUR_JWT_SECRET")?.trim() || export const JWT_SECRET: string = Deno.env.get("GERBEUR_JWT_SECRET")?.trim() ||
""; "";
/** /**
* The network interface the server binds to (`GERBEUR_LISTEN_HOST`). Defaults * The interface the server binds to (`GERBEUR_LISTEN_HOST`). The default,
* to 0.0.0.0 so Docker port-forwarding works out of the box; 127.0.0.1 * 0.0.0.0, lets Docker forward the port; 127.0.0.1 keeps it to loopback.
* restricts it to loopback.
*/ */
export const LISTEN_HOST: string = Deno.env.get("GERBEUR_LISTEN_HOST") || export const LISTEN_HOST: string = Deno.env.get("GERBEUR_LISTEN_HOST") ||
"0.0.0.0"; "0.0.0.0";
/** /**
* The server's public-facing URL (`GERBEUR_PUBLIC_URL`), without a trailing * The URL visitors reach the server at (`GERBEUR_PUBLIC_URL`), without a
* slash — used for CORS, WebSocket origin checks, email links, and OG image * trailing slash. Behind a reverse proxy it must be the outside URL: it is an
* URLs. Behind a reverse proxy, set it to the externally-visible URL (e.g. * allowed origin, it tells links to this site's own pages from other links,
* https://example.com). Defaults to http://localhost:PORT for local * and it prefixes the image URLs in their previews.
* development.
*/ */
export const PUBLIC_URL: string = export const PUBLIC_URL: string =
Deno.env.get("GERBEUR_PUBLIC_URL")?.trim().replace(/\/$/, "") || Deno.env.get("GERBEUR_PUBLIC_URL")?.trim().replace(/\/$/, "") ||
`http://localhost:${PORT}`; `http://localhost:${PORT}`;
/** /**
* The frontend's URL (`GERBEUR_FRONTEND_URL`). In single-container deployments * Points password reset links and the Open Graph tags' page URLs at the
* the API serves the frontend, so it defaults to {@link PUBLIC_URL}; set it * frontend when it isn't served from {@link PUBLIC_URL}, as with a separate
* when the frontend runs on a separate host or port (e.g. Vite dev server or a * Vite server, and allows it as an origin. It is `GERBEUR_FRONTEND_URL`,
* CDN origin). * without a trailing slash.
*/ */
export const FRONTEND_URL: string = export const FRONTEND_URL: string =
Deno.env.get("GERBEUR_FRONTEND_URL")?.trim() || PUBLIC_URL; Deno.env.get("GERBEUR_FRONTEND_URL")?.trim().replace(/\/$/, "") ||
PUBLIC_URL;
/** /**
* The SQLite database file (`GERBEUR_DB_PATH`). Overridable so the test suite * The SQLite database file (`GERBEUR_DB_PATH`). api/test/setup.ts points it at
* can run against a throwaway database (see api/test/setup.ts). * a throwaway file before anything opens the database.
*/ */
export const DB_PATH: string = Deno.env.get("GERBEUR_DB_PATH") || export const DB_PATH: string = Deno.env.get("GERBEUR_DB_PATH") ||
"api/sql/gerbeur.db"; "api/sql/gerbeur.db";
// Upload/files
/** /**
* The directory uploads are stored under (`GERBEUR_UPLOADS_DIR`), one * The directory uploads are stored under (`GERBEUR_UPLOADS_DIR`). Each kind has
* subdirectory per kind. * its subdirectory below, created by the first upload, and each file is named
* after the id of what it belongs to.
*/ */
export const UPLOADS_DIR: string = Deno.env.get("GERBEUR_UPLOADS_DIR") || export const UPLOADS_DIR: string = Deno.env.get("GERBEUR_UPLOADS_DIR") ||
"api/uploads"; "api/uploads";
/** Where file dumps are stored. */ /** Holds the file of each file dump. */
export const DUMPS_DIR = `${UPLOADS_DIR}/dumps`; export const DUMPS_DIR = `${UPLOADS_DIR}/dumps`;
/** Where avatars are stored. */ /** Holds each user's avatar. */
export const AVATARS_DIR = `${UPLOADS_DIR}/avatars`; export const AVATARS_DIR = `${UPLOADS_DIR}/avatars`;
/** Where playlist cover images are stored. */ /** Holds each playlist's cover. */
export const PLAYLIST_IMAGES_DIR = `${UPLOADS_DIR}/playlist-images`; export const PLAYLIST_IMAGES_DIR = `${UPLOADS_DIR}/playlist-images`;
/** Where images pasted into chat, comments and descriptions are stored. */ /**
* Holds the images pasted into chat messages, comments, dump notes and
* descriptions. The startup tasks purge those nothing links to.
*/
export const ATTACHMENTS_DIR = `${UPLOADS_DIR}/attachments`; export const ATTACHMENTS_DIR = `${UPLOADS_DIR}/attachments`;
/** Where custom dump thumbnails are stored. */ /**
* Holds the thumbnails a poster uploads (`<id>-custom`) and those the
* thumbnails route generates from file dumps.
*/
export const THUMBNAILS_DIR = `${UPLOADS_DIR}/thumbnails`; export const THUMBNAILS_DIR = `${UPLOADS_DIR}/thumbnails`;
/**
* Holds the scaled-down copies of the images above, which lib/ffmpeg.ts makes
* as they are first asked for, each at its upload's path with its width. It
* can be emptied at any time: the copies are made again.
*/
export const SCALED_DIR = `${UPLOADS_DIR}/scaled`;
/** The largest image upload accepted: 5 MB. */ /**
* Caps the size of an image upload, in bytes, for avatars, covers, thumbnails
* and attachments. lib/upload.ts states it in megabytes in its error.
*/
export const MAX_IMAGE_SIZE_BYTES = 5 * 1024 * 1024; export const MAX_IMAGE_SIZE_BYTES = 5 * 1024 * 1024;
/** The image types accepted for avatars, covers, thumbnails and attachments. */
export const ALLOWED_IMAGE_MIMES: Set<string> = new Set([
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
]);
/** The MIME type prefixes a file dump may have. */ /**
* The MIME type prefixes a file dump may have. dump-service also accepts the
* exact types of {@link DUMP_ALLOWED_MIME_TYPES}.
*/
export const DUMP_ALLOWED_MIME_PREFIXES: string[] = [ export const DUMP_ALLOWED_MIME_PREFIXES: string[] = [
"text/", "text/",
"image/", "image/",
"video/", "video/",
"audio/", "audio/",
]; ];
/** The exact MIME types a file dump may have, beyond the prefixes. */ /** The documents and archives a file dump may be, beyond the prefixes. */
export const DUMP_ALLOWED_MIME_TYPES: Set<string> = new Set([ export const DUMP_ALLOWED_MIME_TYPES: Set<string> = new Set([
"application/pdf", "application/pdf",
"application/json", "application/json",
@@ -114,40 +157,51 @@ export const DUMP_ALLOWED_MIME_TYPES: Set<string> = new Set([
"application/vnd.ms-powerpoint", "application/vnd.ms-powerpoint",
]); ]);
// Pagination
/** The page size when a request names none. */ /** The page size when a request names none. */
export const PAGINATION_DEFAULT_LIMIT = 20; export const PAGINATION_DEFAULT_LIMIT = 20;
/** The largest page size a request may ask for. */ /** The largest page size a request may ask for. */
export const PAGINATION_MAX_LIMIT = 100; export const PAGINATION_MAX_LIMIT = 100;
// Startup housekeeping
/** How long an attachment nothing links to is kept before startup purges it. */ /** How long an attachment nothing links to is kept before startup purges it. */
export const ORPHANED_ATTACHMENTS_RETENTION_HOURS = 1; export const ORPHANED_ATTACHMENTS_RETENTION_HOURS = 1;
/** How long an unused invite is kept before startup purges it. */ /**
* How long an invite stays usable. invite-service refuses an older one, and
* the startup tasks purge it.
*/
export const UNUSED_INVITES_RETENTION_DAYS = 7; export const UNUSED_INVITES_RETENTION_DAYS = 7;
// SEO/OG /**
/** The site's name (`GERBEUR_SITE_NAME`), in page titles, OG tags and mail. */ * How often, in minutes, the `/community` page's figures are recomputed
* (`GERBEUR_COMMUNITY_REFRESH_MINUTES`), 15 by default.
*/
export const COMMUNITY_REFRESH_MINUTES: number =
Number(Deno.env.get("GERBEUR_COMMUNITY_REFRESH_MINUTES")) || 15;
/**
* The site's name (`GERBEUR_SITE_NAME`). It is filled into index.html, the OG
* tags, the icons' manifest and the emails.
*/
export const OG_SITE_NAME: string = Deno.env.get("GERBEUR_SITE_NAME") || export const OG_SITE_NAME: string = Deno.env.get("GERBEUR_SITE_NAME") ||
"gerbeur"; "gerbeur";
/** /**
* The emoji the favicon set and app header logo are generated from * The emoji that serves as the site's logo (`GERBEUR_SITE_EMOJI`). The icons
* (`GERBEUR_SITE_EMOJI`). Applied at runtime (server startup and index.html * script draws the favicons from it, and index.html carries it to the app's
* injection), so changing it only needs a restart — no rebuild. * header, so changing it needs a restart but no rebuild.
*/ */
export const SITE_EMOJI: string = Deno.env.get("GERBEUR_SITE_EMOJI") || "🚚"; export const SITE_EMOJI: string = Deno.env.get("GERBEUR_SITE_EMOJI") || "🚚";
/** /**
* A Bandcamp playback path: * How the frontend plays Bandcamp. `"embed"` uses Bandcamp's iframe, which
* - `"embed"` — the bandcamp.com iframe (cannot autoplay) * can't autoplay. `"native"` resolves the page's streams through
* - `"native"` — resolve the page's mp3 streams and play them in the app's own * `GET /api/bandcamp/tracks` and plays them in the app's player, which
* player, which autoplays and supports albums as a queue * autoplays and queues an album's tracks.
*/ */
export type BandcampPlayer = "embed" | "native"; export type BandcampPlayer = "embed" | "native";
/** /**
* The Bandcamp playback path the frontend uses (`GERBEUR_BANDCAMP_PLAYER`). * How the frontend plays Bandcamp (`GERBEUR_BANDCAMP_PLAYER`), passed to it in
* Anything unrecognised falls back to `"embed"`, so a typo degrades safely. * index.html. Any value but `"native"` means `"embed"`, so a typo falls back to
* the player that always works.
*/ */
export const BANDCAMP_PLAYER: BandcampPlayer = export const BANDCAMP_PLAYER: BandcampPlayer =
Deno.env.get("GERBEUR_BANDCAMP_PLAYER")?.trim() === "native" Deno.env.get("GERBEUR_BANDCAMP_PLAYER")?.trim() === "native"
@@ -155,22 +209,56 @@ export const BANDCAMP_PLAYER: BandcampPlayer =
: "embed"; : "embed";
/** /**
* The background color of generated icons and the manifest. Mirrors the * The background of the generated Apple touch icon, the manifest's colors, and
* hard-coded theme-color in index.html. * the accent of previews of this site's own pages. index.html's `theme-color`
* tags don't follow it.
*/ */
export const THEME_COLOR = "#111827"; export const THEME_COLOR = "#111827";
const rawOrigins = Deno.env.get("GERBEUR_ALLOWED_ORIGINS") ?? ""; const rawOrigins = Deno.env.get("GERBEUR_ALLOWED_ORIGINS") ?? "";
/** /**
* The origins CORS and the WebSocket accept: {@link PUBLIC_URL}, * The origins that CORS in main.ts and the WebSocket route accept. They are
* {@link FRONTEND_URL} and the comma-separated `GERBEUR_ALLOWED_ORIGINS`. * {@link PUBLIC_URL}, {@link FRONTEND_URL} and the comma-separated
* `GERBEUR_ALLOWED_ORIGINS`.
*/ */
export const ALLOWED_ORIGINS: string[] = Array.from( export const ALLOWED_ORIGINS: string[] = Array.from(
new Set([ new Set([
PUBLIC_URL, PUBLIC_URL,
// FRONTEND_URL is auto-included so the separate-frontend case only needs
// GERBEUR_FRONTEND_URL — no need to repeat it in GERBEUR_ALLOWED_ORIGINS.
FRONTEND_URL, FRONTEND_URL,
...rawOrigins.split(",").map((o) => o.trim()).filter(Boolean), ...rawOrigins.split(",").map((o) => o.trim()).filter(Boolean),
]), ]),
); );
/**
* Lets federated sign-in reach instances on private addresses, `localhost`
* among them, and over plain http (`GERBEUR_FEDERATION_ALLOW_PRIVATE=1`), so
* that two instances on one machine can trust each other in development.
* Off, `remote/gerbeur-instance.ts` reaches only public https instances, since
* an identity is only as safe as the connection it came over.
*/
export const FEDERATION_ALLOW_PRIVATE: boolean =
Deno.env.get("GERBEUR_FEDERATION_ALLOW_PRIVATE")?.trim() === "1";
/**
* Which ways federated sign-in goes, each on unless set to `0`.
*
* - `incoming` (`GERBEUR_FEDERATION_INCOMING`): users of other instances sign
* in here. Off, the login and register pages offer no other instance,
* `/manage` lists only the blocked ones, which this instance's users can't
* sign in to (and none with `outgoing` off too), `federation-service.ts`
* refuses their sign-ins, and the accounts that came from one are signed
* out (`getSessionState`) until it is on again. Their homes' events are
* still applied, so that none is lost meanwhile.
* - `outgoing` (`GERBEUR_FEDERATION_OUTGOING`): this instance's users sign in
* to others. Off, `identity-provider-service.ts` lets no instance sign them
* in, and its metadata says so, which the others check before sending
* anyone here to sign in. The settings page lists no instances, since at
* startup `main.ts` withdraws every grant they gave, which signs them out
* there.
*
* index.html tells the client both. An object, so that the tests can turn one
* off for a while.
*/
export const FEDERATION: { incoming: boolean; outgoing: boolean } = {
incoming: Deno.env.get("GERBEUR_FEDERATION_INCOMING")?.trim() !== "0",
outgoing: Deno.env.get("GERBEUR_FEDERATION_OUTGOING")?.trim() !== "0",
};

34
api/db/connection.ts
View File

@@ -1,6 +1,8 @@
/** /**
* The database handle. Importing this module opens the database at * Opens the database at {@link DB_PATH} when first imported, creating it from
* {@link DB_PATH}, applies the pending migrations and runs the startup tasks. * schema.sql if the file doesn't exist, then applies the pending migrations
* and runs the startup tasks before any importer runs. Only the services
* import it.
* *
* @module * @module
*/ */
@@ -10,12 +12,25 @@ import { DB_PATH } from "../config.ts";
import { runMigrations } from "./migrate.ts"; import { runMigrations } from "./migrate.ts";
import { runStartupTasks } from "./startup.ts"; import { runStartupTasks } from "./startup.ts";
/** The one connection every service queries through. */ const isNew = await Deno.stat(DB_PATH).then(() => false, () => true);
/**
* Serves every query of the server, as its only connection. Its queries are
* synchronous, so each one blocks the event loop while it runs.
*/
export const db: DatabaseSync = new DatabaseSync(DB_PATH); export const db: DatabaseSync = new DatabaseSync(DB_PATH);
// schema.sql holds every migration's change, so on a new database each
// migration below finds its work done and is only recorded.
if (isNew) {
db.exec(
await Deno.readTextFile(new URL("./schema.sql", import.meta.url)),
);
}
// WAL lets readers run alongside a writer, and with synchronous = NORMAL a // WAL lets readers run alongside a writer, and with synchronous = NORMAL a
// commit no longer waits on an fsync (a crash can lose the last transactions, // commit doesn't wait on an fsync: a power loss can lose the last commits but
// never corrupt the file). busy_timeout covers the init/migration scripts and // never corrupts the file. busy_timeout makes another connection to the file
// the test harness opening the same file. // wait for the lock instead of failing. SQLite enforces foreign keys, and so
// the schema's ON DELETE cascades, only when each connection asks.
db.exec(` db.exec(`
PRAGMA journal_mode = WAL; PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL; PRAGMA synchronous = NORMAL;
@@ -26,9 +41,10 @@ runMigrations(db);
await runStartupTasks(db); await runStartupTasks(db);
/** /**
* Runs `fn` inside one transaction: several writes commit together (and cost * Runs `fn` in a transaction, so its writes commit together, in one commit
* one commit rather than one each), and a throw rolls them all back. Nested * rather than one each, and a throw rolls them all back and rethrows. A call
* calls join the outer transaction. * inside another joins the outer transaction. `fn` must be synchronous: an
* `await` would commit before the rest of it runs.
*/ */
export function transaction<T>(fn: () => T): T { export function transaction<T>(fn: () => T): T {
if (db.isTransaction) return fn(); if (db.isTransaction) return fn();

21
api/db/init.ts
View File

@@ -1,21 +0,0 @@
/**
* Creates the database from schema.sql when {@link DB_PATH} doesn't exist yet;
* run once before the server starts (`deno task start`).
*
* @module
*/
import { DatabaseSync } from "node:sqlite";
import { DB_PATH } from "../config.ts";
try {
await Deno.stat(DB_PATH);
console.log("Database already exists, skipping initialization.");
} catch {
console.log("Initializing database from schema...");
const schema = Deno.readTextFileSync("api/db/schema.sql");
const db = new DatabaseSync(DB_PATH);
db.exec(schema);
db.close();
console.log("Database initialized.");
}

40
api/db/migrate.ts
View File

@@ -1,6 +1,7 @@
/** /**
* Brings an existing database up to date with the migrations under * Brings the database up to date with the migrations under migrations/, and
* migrations/, recording each applied one in `schema_migrations`. * records each one it applies in `schema_migrations`. connection.ts runs it
* each time the server opens the database.
* *
* @module * @module
*/ */
@@ -22,6 +23,15 @@ import { up as up0013DumpUrlCanonical } from "./migrations/0013_dump_url_canonic
import { up as up0014ChatReadState } from "./migrations/0014_chat_read_state.ts"; import { up as up0014ChatReadState } from "./migrations/0014_chat_read_state.ts";
import { up as up0015TokenRevocation } from "./migrations/0015_token_revocation.ts"; import { up as up0015TokenRevocation } from "./migrations/0015_token_revocation.ts";
import { up as up0016PerformanceIndexes } from "./migrations/0016_performance_indexes.ts"; import { up as up0016PerformanceIndexes } from "./migrations/0016_performance_indexes.ts";
import { up as up0017CaseInsensitiveUsers } from "./migrations/0017_case_insensitive_users.ts";
import { up as up0018AttachmentResourceType } from "./migrations/0018_attachment_resource_type.ts";
import { up as up0019UserSettings } from "./migrations/0019_user_settings.ts";
import { up as up0020DigestSchedule } from "./migrations/0020_digest_schedule.ts";
import { up as up0021PageUrls } from "./migrations/0021_page_urls.ts";
import { up as up0022ReserveCommunity } from "./migrations/0022_reserve_community.ts";
import { up as up0023MentionEveryone } from "./migrations/0023_mention_everyone.ts";
import { up as up0024Federation } from "./migrations/0024_federation.ts";
import { up as up0025PlaylistCategories } from "./migrations/0025_playlist_categories.ts";
interface Migration { interface Migration {
name: string; name: string;
@@ -29,10 +39,10 @@ interface Migration {
} }
/** /**
* Every migration, in order. Append-only — never edit or reorder a migration * Lists every migration, in the order they run. A new one goes at the end,
* once it has shipped. Each `up` must be idempotent: it also runs against * along with the same change to schema.sql, and a shipped one is never edited
* fresh databases created straight from schema.sql, where the change it makes * or moved. A database that connection.ts made from schema.sql records none of
* already exists. * them, so each `up` must find its change already made and leave it be.
*/ */
const MIGRATIONS: Migration[] = [ const MIGRATIONS: Migration[] = [
{ name: "0001_comment_likes", up: up0001CommentLikes }, { name: "0001_comment_likes", up: up0001CommentLikes },
@@ -54,14 +64,26 @@ const MIGRATIONS: Migration[] = [
{ name: "0014_chat_read_state", up: up0014ChatReadState }, { name: "0014_chat_read_state", up: up0014ChatReadState },
{ name: "0015_token_revocation", up: up0015TokenRevocation }, { name: "0015_token_revocation", up: up0015TokenRevocation },
{ name: "0016_performance_indexes", up: up0016PerformanceIndexes }, { name: "0016_performance_indexes", up: up0016PerformanceIndexes },
{ name: "0017_case_insensitive_users", up: up0017CaseInsensitiveUsers },
{
name: "0018_attachment_resource_type",
up: up0018AttachmentResourceType,
},
{ name: "0019_user_settings", up: up0019UserSettings },
{ name: "0020_digest_schedule", up: up0020DigestSchedule },
{ name: "0021_page_urls", up: up0021PageUrls },
{ name: "0022_reserve_community", up: up0022ReserveCommunity },
{ name: "0023_mention_everyone", up: up0023MentionEveryone },
{ name: "0024_federation", up: up0024Federation },
{ name: "0025_playlist_categories", up: up0025PlaylistCategories },
]; ];
/** /**
* Applies each migration `db` hasn't recorded yet, each in its own * Applies each migration `db` hasn't recorded yet, in order, each in its own
* transaction. * transaction.
* *
* @throws {Error} When a migration fails; that migration is rolled back and * @throws {Error} When a migration fails. That one is rolled back, the later
* the later ones aren't run. * ones don't run, and the server doesn't start.
*/ */
export function runMigrations(db: DatabaseSync): void { export function runMigrations(db: DatabaseSync): void {
db.exec(`CREATE TABLE IF NOT EXISTS schema_migrations ( db.exec(`CREATE TABLE IF NOT EXISTS schema_migrations (

13
api/db/migrations/0001_comment_likes.ts
View File

@@ -1,16 +1,17 @@
/** /**
* Adds comment likes: the `comment_likes` table and the `comments.like_count` * Adds likes on comments, as the `comment_likes` table and a `like_count` on
* counter. * each comment that comment-like-service keeps in step, so that listing
* * comments doesn't count their likes.
* Idempotent: safe to run against a fresh db (already created from schema.sql
* with these columns/tables) or an existing one that predates them.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing comments start with no likes, and a
* database made from schema.sql already has both.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const commentCols = db.prepare(`PRAGMA table_info(comments);`).all() as { const commentCols = db.prepare(`PRAGMA table_info(comments);`).all() as {
name: string; name: string;

12
api/db/migrations/0002_dump_thumbnail.ts
View File

@@ -1,15 +1,17 @@
/** /**
* Adds `dumps.custom_thumbnail_mime`, for thumbnails uploaded by the poster. * Lets a poster upload a thumbnail that replaces the fetched one. It adds
* * `dumps.custom_thumbnail_mime`, which holds the uploaded image's type; NULL
* Idempotent: safe to run against a fresh db (already created from schema.sql * means the dump has none.
* with this column) or an existing one that predates it.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing dumps keep their fetched thumbnails,
* and a database made from schema.sql already has the column.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const dumpCols = db.prepare(`PRAGMA table_info(dumps);`).all() as { const dumpCols = db.prepare(`PRAGMA table_info(dumps);`).all() as {
name: string; name: string;

14
api/db/migrations/0003_dump_backlinks.ts
View File

@@ -1,16 +1,18 @@
/** /**
* Adds the `dump_backlinks` table, recording which dumps, comments and * Lets a dump page list the dumps related to it. It adds `dump_backlinks`,
* descriptions link to a dump, and indexes `dumps.url`. * which records the links from a dump's note or comments to other dumps and
* * which backlink-service fills, and an index on `dumps.url` to find the dumps
* Idempotent: safe to run against a fresh db (already created from schema.sql * an external link was posted as.
* with these) or an existing one that predates them.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. It doesn't scan the notes and comments already
* posted, so their links are recorded only once they are edited.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
db.exec(`CREATE TABLE IF NOT EXISTS dump_backlinks ( db.exec(`CREATE TABLE IF NOT EXISTS dump_backlinks (
source_type TEXT NOT NULL, source_type TEXT NOT NULL,

14
api/db/migrations/0004_user_roles.ts
View File

@@ -1,15 +1,16 @@
/** /**
* Adds `users.role` and makes every legacy `is_admin` user an admin. * Makes room for moderators between users and admins. It adds `users.role`,
* * which replaces the `is_admin` flag, and 0005 then drops the flag.
* Idempotent: safe to run against a fresh db (already created from schema.sql
* with the `role` column) or an existing one that predates it.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Every user becomes a `user`, except those with
* `is_admin` set, who become admins.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const userCols = db.prepare(`PRAGMA table_info(users);`).all() as { const userCols = db.prepare(`PRAGMA table_info(users);`).all() as {
name: string; name: string;
@@ -20,8 +21,7 @@ export function up(db: DatabaseSync): void {
); );
} }
// Backfill existing admins to role 'admin'. Guarded because the legacy // schema.sql never had is_admin.
// is_admin column is absent on fresh databases (and dropped by 0005).
if (userCols.some((c) => c.name === "is_admin")) { if (userCols.some((c) => c.name === "is_admin")) {
db.exec( db.exec(
`UPDATE users SET role = 'admin' WHERE is_admin = 1 AND role != 'admin';`, `UPDATE users SET role = 'admin' WHERE is_admin = 1 AND role != 'admin';`,

9
api/db/migrations/0005_drop_is_admin.ts
View File

@@ -1,14 +1,15 @@
/** /**
* Drops the legacy is_admin column now that `role` is the single source of * Drops `users.is_admin`, now that 0004 has copied it into `users.role`.
* truth. Idempotent: skips when the column is already gone (fresh databases
* created from schema.sql never had it). Must run after 0004 has backfilled.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`, after 0004. A database made from schema.sql
* never had the column.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const userCols = db.prepare(`PRAGMA table_info(users);`).all() as { const userCols = db.prepare(`PRAGMA table_info(users);`).all() as {
name: string; name: string;

12
api/db/migrations/0006_categories.ts
View File

@@ -1,16 +1,16 @@
/** /**
* Adds categories: the `categories` table and the `dump_categories` join * Adds categories, which admins create and posters file their dumps under, as
* table. * the `categories` table and the `dump_categories` join table.
*
* Idempotent: safe to run against a fresh db (already created from schema.sql
* with the categories tables) or an existing one that predates them.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing dumps start in no category, and a
* database made from schema.sql already has both tables.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const tables = new Set( const tables = new Set(
(db.prepare( (db.prepare(

11
api/db/migrations/0007_password_reset_tokens.ts
View File

@@ -1,15 +1,16 @@
/** /**
* Adds the `password_reset_tokens` table. * Adds `password_reset_tokens`, which holds the single-use tokens that
* * password-reset-service emails to users who forgot their password.
* Idempotent: safe to run against a fresh db (already created from schema.sql
* with password_reset_tokens) or an existing one that predates the table.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. A database made from schema.sql already has the
* table.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const tables = new Set( const tables = new Set(
(db.prepare( (db.prepare(

10
api/db/migrations/0008_chat_messages.ts
View File

@@ -1,15 +1,15 @@
/** /**
* Adds the `chat_messages` table. * Adds `chat_messages`, which holds the messages of the site-wide chat.
*
* Idempotent: safe to run against a fresh db (already created from schema.sql
* with this table) or an existing one that predates it.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. A database made from schema.sql already has the
* table.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
db.exec(`CREATE TABLE IF NOT EXISTS chat_messages ( db.exec(`CREATE TABLE IF NOT EXISTS chat_messages (
id TEXT NOT NULL PRIMARY KEY, id TEXT NOT NULL PRIMARY KEY,

16
api/db/migrations/0009_chat_reply.ts
View File

@@ -1,18 +1,18 @@
/** /**
* Adds an optional self-reference so a chat message can reply to another. * Lets a chat message reply to another, through `chat_messages.reply_to_id`.
* Kept as a plain column (no FK): when the replied-to message is later deleted * The column has no foreign key on purpose: when the replied-to message is
* the id is retained but the join yields null, letting the UI show a subtle * deleted, the id stays and chat-service's join reads null, so the chat shows
* "deleted message" reference instead of breaking the insert. * "deleted message" where the quote was.
*
* Idempotent: also runs against fresh databases created from schema.sql, where
* the column already exists.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing messages reply to nothing, and a
* database made from schema.sql already has the column.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(chat_messages);`).all() as { const cols = db.prepare(`PRAGMA table_info(chat_messages);`).all() as {
name: string; name: string;

32
api/db/migrations/0010_youtube_embed_start.ts
View File

@@ -1,25 +1,22 @@
/** /**
* Backfills the `start` parameter on stored YouTube embed URLs. * Makes the YouTube dumps posted with a start time, before the YouTube provider
* kept it, start there instead of at 0:00. `rich_content.embedUrl` is computed
* when a dump is posted, and it used to drop the source URL's `t` (`?t=1265`,
* `?t=1h2m3s`); this adds the `start` parameter back to the stored embed URLs.
* *
* `rich_content.embedUrl` is computed once, at dump creation, and previously * It reads the time from the stored source URL and fetches nothing. It keeps
* dropped the timestamp carried by the source URL (`?t=1265`, `?t=1h2m3s`), so * its own copy of the parsing, so that it migrates every database alike should
* those videos always opened at 0:00 in the global player. The provider now * the provider's change.
* translates `t`/`start` into the `start` param the iframe player honours; this
* migration applies the same translation to rows written before that fix.
*
* Purely local — it re-derives the param from the already-stored source URL and
* makes no network calls. Self-contained (no imports from api/services) so it
* stays reproducible even if the provider's parsing evolves later.
*
* Idempotent: rows whose embed URL already carries a `start` are skipped, so a
* fresh database built from schema.sql is simply a no-op.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Parses `t`/`start` — plain seconds, `90s`, or `1h2m3s` / `20m5s`. */ /**
* Returns the seconds in `url`'s `t` or `start`, which may be plain seconds,
* `90s`, `20m5s` or `1h2m3s`.
*/
function extractStartSeconds(url: string): number | null { function extractStartSeconds(url: string): number | null {
let raw: string | null; let raw: string | null;
try { try {
@@ -38,7 +35,10 @@ function extractStartSeconds(url: string): number | null {
Number(hms[3] ?? 0); Number(hms[3] ?? 0);
} }
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. It leaves alone an embed URL that already has a
* `start`, and a row whose `rich_content` doesn't parse.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const rows = db.prepare( const rows = db.prepare(
`SELECT id, url, rich_content FROM dumps `SELECT id, url, rich_content FROM dumps
@@ -55,7 +55,7 @@ export function up(db: DatabaseSync): void {
try { try {
rich = JSON.parse(row.rich_content); rich = JSON.parse(row.rich_content);
} catch { } catch {
continue; // malformed payload — leave it untouched continue;
} }
if (rich.type !== "youtube" || !rich.embedUrl) continue; if (rich.type !== "youtube" || !rich.embedUrl) continue;

43
api/db/migrations/0011_split_favicon_thumbnail.ts
View File

@@ -1,21 +1,15 @@
/** /**
* Moves favicon-shaped values out of `rich_content.thumbnailUrl` into the new * Stops older dumps from showing a site's icon as their thumbnail. The metadata
* `faviconUrl` field. * fetch used to fall back on the page's icon, and then on a guessed
* `/favicon.ico`, so most thumbnails were a 16×16 icon, or a 404, that the UI
* stretched over a 128×72 box. It now stops at real artwork, and a missing
* `thumbnailUrl` means none, which the frontend draws a placeholder for. This
* gives the older rows that meaning, by moving their icons out of
* `rich_content.thumbnailUrl` into the new `faviconUrl`.
* *
* The extraction cascade used to end at the page's icon and then at a guessed * It judges the stored URL alone and fetches nothing, so it doesn't fill
* `${origin}/favicon.ico`, so `thumbnailUrl` was almost never empty — it just * `accentColor`. The frontend derives a hue from the hostname when it is
* held a 16×16 icon (or a 404) that the UI then cover-cropped into a 128×72 * missing, and refreshing a dump's metadata fetches the real one.
* box. The cascade now stops at real artwork, and an absent `thumbnailUrl`
* means "no artwork", which is what lets the frontend draw a placeholder.
* This migration gives rows written before that change the same meaning.
*
* Purely local — it classifies the already-stored URL and makes no network
* calls, so `accentColor` is deliberately not backfilled: the frontend derives
* a stable hue from the hostname whenever one is missing, and
* `refreshDumpMetadata` fetches the real color on demand.
*
* Idempotent: rows that already carry a `faviconUrl` are skipped, so a fresh
* database built from schema.sql is a no-op.
* *
* @module * @module
*/ */
@@ -23,12 +17,9 @@
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** /**
* Checks whether a stored thumbnail URL is really a site icon. * Checks whether a stored thumbnail URL is a site icon. It is loose on
* * purpose, since artwork taken for an icon still shows whole on a tinted field,
* Deliberately loose. A false positive (a genuine cover image living under * only not cropped to fill the box. 0012 fixes the rows it got wrong.
* `/assets/icons/`) renders contained on a tinted field instead of
* cover-cropped — mildly wrong, never broken — so chasing them isn't worth the
* extra rules.
*/ */
function isIconUrl(raw: string): boolean { function isIconUrl(raw: string): boolean {
let pathname: string; let pathname: string;
@@ -41,7 +32,11 @@ function isIconUrl(raw: string): boolean {
/\.(ico|svg)$/i.test(pathname); /\.(ico|svg)$/i.test(pathname);
} }
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. It leaves alone a row that already has a
* `faviconUrl`, and one whose `rich_content` doesn't parse. 0012 undoes its
* mistakes.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const rows = db.prepare( const rows = db.prepare(
`SELECT id, rich_content FROM dumps `SELECT id, rich_content FROM dumps
@@ -58,7 +53,7 @@ export function up(db: DatabaseSync): void {
try { try {
rich = JSON.parse(row.rich_content); rich = JSON.parse(row.rich_content);
} catch { } catch {
continue; // malformed payload — leave it untouched continue;
} }
if (rich.faviconUrl || !rich.thumbnailUrl) continue; if (rich.faviconUrl || !rich.thumbnailUrl) continue;
if (!isIconUrl(rich.thumbnailUrl)) continue; if (!isIconUrl(rich.thumbnailUrl)) continue;

43
api/db/migrations/0012_fix_favicon_reclassification.ts
View File

@@ -1,30 +1,19 @@
/** /**
* Repairs 0011, which was too eager about what counts as a site icon. * Repairs the rows 0011 got wrong. 0011 took any path containing "favicon",
* or ending in `.svg`, for an icon, so artwork such as
* `/wp-content/uploads/2022/04/mir-logo-favicon.png` lost its `thumbnailUrl`,
* and its dump showed as a pull-quote instead of an image in the journal
* mosaic.
* *
* 0011 tested for "favicon" anywhere in the path, so a legitimate `og:image` * This rule judges the file name and its folder instead. It spares
* like mirtitles.org's `/wp-content/uploads/2022/04/mir-logo-favicon.png` was * `mir-logo-favicon.png` and catches `icon_SEARCH.png`, which 0011 missed.
* moved into `faviconUrl` and its `thumbnailUrl` cleared — which also demoted
* the dump from an image card to a pull-quote in the journal mosaic. A blanket
* `.svg` match had the same effect on real artwork.
*
* `isIconUrl` here looks at the *filename* rather than the whole path: an icon
* is named `favicon…`, `apple-touch-icon…` or `icon…`, lives in an `icons/`
* directory, or ends in `.ico`. That both spares `mir-logo-favicon.png` and
* catches `icon_SEARCH.png`, which 0011 missed.
*
* Applied in both directions, and deliberately narrow when reversing: a value
* is only promoted back to `thumbnailUrl` if it matches 0011's rule but not
* this one, so rows written by a normal fetch aren't disturbed.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** /** 0011's rule, copied as it shipped, to tell which rows it moved. */
* Applies 0011's rule, kept verbatim so its false positives can be
* identified.
*/
function wasIconUrl0011(raw: string): boolean { function wasIconUrl0011(raw: string): boolean {
let pathname: string; let pathname: string;
try { try {
@@ -37,8 +26,9 @@ function wasIconUrl0011(raw: string): boolean {
} }
/** /**
* Checks whether a URL names a site icon, judged on its filename and * Checks whether a URL is a site icon. It is one when the file ends in `.ico`,
* directory. * sits in a `favicon(s)` or `icon(s)` folder, or is named `favicon`,
* `apple-touch-icon` or `icon`, alone or followed by a digit, `-`, `_` or `.`.
*/ */
function isIconUrl(raw: string): boolean { function isIconUrl(raw: string): boolean {
let pathname: string; let pathname: string;
@@ -55,7 +45,12 @@ function isIconUrl(raw: string): boolean {
return /^(favicon|apple-touch-icon|icon)(?=$|[-_.\d])/i.test(basename); return /^(favicon|apple-touch-icon|icon)(?=$|[-_.\d])/i.test(basename);
} }
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. It moves a `faviconUrl` back to `thumbnailUrl`
* only when 0011's rule takes it for an icon and this one doesn't. That also
* catches the ones a later fetch stored that this rule doesn't recognise, such
* as a `logo.svg`. It then moves the icons 0011 missed.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const rows = db.prepare( const rows = db.prepare(
`SELECT id, rich_content FROM dumps `SELECT id, rich_content FROM dumps
@@ -73,10 +68,9 @@ export function up(db: DatabaseSync): void {
try { try {
rich = JSON.parse(row.rich_content); rich = JSON.parse(row.rich_content);
} catch { } catch {
continue; // malformed payload — leave it untouched continue;
} }
// Artwork 0011 mistook for an icon: put it back.
if ( if (
rich.faviconUrl && !rich.thumbnailUrl && rich.faviconUrl && !rich.thumbnailUrl &&
wasIconUrl0011(rich.faviconUrl) && !isIconUrl(rich.faviconUrl) wasIconUrl0011(rich.faviconUrl) && !isIconUrl(rich.faviconUrl)
@@ -90,7 +84,6 @@ export function up(db: DatabaseSync): void {
continue; continue;
} }
// An icon 0011's rule didn't recognize: move it now.
if ( if (
rich.thumbnailUrl && !rich.faviconUrl && isIconUrl(rich.thumbnailUrl) rich.thumbnailUrl && !rich.faviconUrl && isIconUrl(rich.thumbnailUrl)
) { ) {

27
api/db/migrations/0013_dump_url_canonical.ts
View File

@@ -1,19 +1,13 @@
/** /**
* Adds `dumps.url_canonical` — the lookup key behind the "already dumped?" * Lets the create form warn that a link was already dumped. It adds
* check on the create form — and backfills it for every existing URL dump. * `dumps.url_canonical`, the key the hint looks a URL up by, and fills it for
* every URL dump already posted.
* *
* Purely local: it re-derives the key from the URL already stored on each row * It fills the key from each stored URL and fetches nothing. Its helpers are a
* and makes no network calls. * copy of lib/canonical-url.ts as it stood when this shipped: importing the
* * live one would give two databases migrated at different times keys made by
* The helpers below are a deliberate frozen copy of `api/lib/canonical-url.ts` * different rules. A change to lib/canonical-url.ts therefore needs a new
* as it stood when this migration shipped, not an import of it. A migration * migration that fills the keys again.
* runs exactly once per database, so importing the live version would mean two
* databases migrating at different times end up with keys computed under
* different rules. Improving the shared canonicalizer therefore calls for a new
* backfill migration rather than an edit here.
*
* Idempotent: the column is only added when missing and only rows whose key is
* still NULL are touched, so a fresh database built from schema.sql is a no-op.
* *
* @module * @module
*/ */
@@ -98,7 +92,10 @@ function canonicalizeUrl(raw: string): string | null {
return `https://${host}${path}${query ? `?${query}` : ""}${hash}`; return `https://${host}${path}${query ? `?${query}` : ""}${hash}`;
} }
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. It fills only the keys still NULL, and leaves
* NULL a URL that isn't http(s).
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const columns = db.prepare(`PRAGMA table_info(dumps);`).all() as { const columns = db.prepare(`PRAGMA table_info(dumps);`).all() as {
name: string; name: string;

21
api/db/migrations/0014_chat_read_state.ts
View File

@@ -1,21 +1,22 @@
/** /**
* Chat unread state lives on the server so it survives reloads and agrees * Moves the chat's unread state to the server, so that it survives a reload
* across tabs and devices: `users.chat_read_at` is the created_at of the newest * and agrees across tabs and devices. `users.chat_read_at` holds the time of
* message the user has seen. Existing users start caught up (now) rather than * the newest message the user has seen.
* with the whole history unread; new users fall back to their own created_at.
* *
* The history cursor now pages on (created_at, id) so two messages sharing a * It also replaces the index on `created_at` with one on `(created_at, id)`,
* millisecond can't be skipped at a page boundary; the index follows suit. * which the history now pages on, so that two messages from the same
* * millisecond can't fall between two pages.
* Idempotent: also runs against fresh databases created from schema.sql, where
* the column and index already exist.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing users start with every message read,
* instead of the whole history unread. A user created later starts with a
* NULL `chat_read_at`, which chat-service reads as the user's creation time.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(users);`).all() as { const cols = db.prepare(`PRAGMA table_info(users);`).all() as {
name: string; name: string;

15
api/db/migrations/0015_token_revocation.ts
View File

@@ -1,18 +1,17 @@
/** /**
* Changing or resetting a password must end every session opened before it. * Lets a password change end every session opened before it. Tokens are
* Tokens are stateless JWTs, so each user carries a cut-off instead: * stateless JWTs, so this adds `users.tokens_valid_after`, and lib/auth.ts
* `users.tokens_valid_after` (ISO), and a token issued before it is refused. * refuses a token issued before that time.
* NULL — the state every existing user starts in — means no cut-off.
*
* Idempotent: also runs against fresh databases created from schema.sql, where
* the column already exists.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. Existing users start with NULL, which revokes no
* token.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(users);`).all() as { const cols = db.prepare(`PRAGMA table_info(users);`).all() as {
name: string; name: string;

24
api/db/migrations/0016_performance_indexes.ts
View File

@@ -1,23 +1,21 @@
/** /**
* Indexes for the queries that ran as full scans: * Adds indexes for the queries that scanned a whole table. The feeds page
* - feeds page by created_at (newest first), a user's dumps and upvotes by * dumps by `created_at`, and a user's dumps and upvotes by `(user_id,
* (user_id, created_at) — these replace the bare user_id indexes; * created_at)`, which replace the indexes on `user_id` alone. Dumps and
* - dump and playlist pages are looked up by slug; * playlists are looked up by slug. Each foreign key with ON DELETE gets an
* - the child side of every ON DELETE CASCADE / SET NULL foreign key, which * index on its child column, which SQLite otherwise scans on every delete of
* SQLite otherwise scans on each parent delete (a comment's replies, a user's * a parent row, and the followers of a user or playlist are read on every new
* comments, follows, chat messages, invites, reset tokens), and the * dump. Mentions look usernames up by `lower(username)`.
* followers of a user or playlist, read on every new dump;
* - @mentions resolve usernames case-insensitively, by lower(username).
*
* Idempotent: also runs against fresh databases created from schema.sql, where
* these indexes already exist.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
/** Applies this migration to `db`. */ /**
* Run once by `runMigrations`. A database made from schema.sql already has
* these indexes.
*/
export function up(db: DatabaseSync): void { export function up(db: DatabaseSync): void {
db.exec(` db.exec(`
DROP INDEX IF EXISTS idx_dumps_user; DROP INDEX IF EXISTS idx_dumps_user;

36
api/db/migrations/0017_case_insensitive_users.ts Normal file
View File

@@ -0,0 +1,36 @@
/**
* Makes usernames and emails unique regardless of case. Usernames were unique
* only as typed, so "Bob" and "bob" could both sign up, and a mention, which
* looks names up in lower case, then reached one of them at random. Emails
* had no constraint at all: two sign-ups racing past the service's check could
* share an address, and password resets are looked up by address.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
/**
* Run once by `runMigrations`. It refuses to run, naming the clashes, when two
* accounts already share a name or an address in some case: which one keeps
* it is for whoever runs the site to decide, by editing one of them.
*/
export function up(db: DatabaseSync): void {
for (const column of ["username", "email"]) {
const clashes = db.prepare(
`SELECT lower(${column}) AS value FROM users GROUP BY lower(${column}) HAVING COUNT(*) > 1;`,
).all() as { value: string }[];
if (clashes.length > 0) {
throw new Error(
`Several accounts share a ${column} in different cases: ${
clashes.map((c) => c.value).join(", ")
}. Change all but one of each, then restart.`,
);
}
}
db.exec(`
DROP INDEX IF EXISTS idx_users_username_lower;
CREATE UNIQUE INDEX IF NOT EXISTS idx_users_username_lower ON users(lower(username));
CREATE UNIQUE INDEX IF NOT EXISTS idx_users_email_lower ON users(lower(email));
`);
}

36
api/db/migrations/0018_attachment_resource_type.ts Normal file
View File

@@ -0,0 +1,36 @@
/**
* Serves an attachment only to those who may see what it was pasted into, and
* deletes it once that is gone. It adds `attachments.resource_type`, the kind
* of thing an attachment's `resource_id` names. Without it, an attachment
* couldn't tell whose it was: one pasted into a private dump was served to
* anyone with its URL, and one whose dump, comment or message was deleted was
* kept for good.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
/**
* Run once by `runMigrations`. Existing bound attachments get the type of the
* table that holds their resource; one whose resource is gone gets none, and
* the startup purge deletes it. A database made from schema.sql already has
* the column.
*/
export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(attachments);`).all() as {
name: string;
}[];
if (cols.some((c) => c.name === "resource_type")) return;
db.exec(`
ALTER TABLE attachments ADD COLUMN resource_type TEXT;
UPDATE attachments SET resource_type = CASE
WHEN resource_id IN (SELECT id FROM dumps) THEN 'dump'
WHEN resource_id IN (SELECT id FROM comments) THEN 'comment'
WHEN resource_id IN (SELECT id FROM playlists) THEN 'playlist'
WHEN resource_id IN (SELECT id FROM chat_messages) THEN 'chat'
WHEN resource_id IN (SELECT id FROM users) THEN 'user'
END
WHERE resource_id IS NOT NULL;
`);
}

28
api/db/migrations/0019_user_settings.ts Normal file
View File

@@ -0,0 +1,28 @@
/**
* Gives each user settings stored on the server, so that they follow the
* account across browsers and the server can read them: the digests need the
* reader's choice to receive them and their style. `user_settings` holds one
* row per user who changed anything; a user without one has the defaults.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
/**
* Run once by `runMigrations`. Existing users get no row, so they start with
* the defaults, digests on, like a new account.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS user_settings (
user_id TEXT PRIMARY KEY,
style TEXT,
color_scheme TEXT,
digest_weekly INTEGER NOT NULL DEFAULT 1,
digest_monthly INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
`);
}

52
api/db/migrations/0020_digest_schedule.ts Normal file
View File

@@ -0,0 +1,52 @@
/**
* Lets `digest-service.ts` know when each user's next digest is due.
* `digest_schedule` holds, for each user and cadence they receive, the start
* of the period their next digest covers; it is due once a full period has
* passed.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
const DAY_MS = 86_400_000;
// A frozen copy of digest-service's periods: a migration must keep doing
// what it did when it shipped, whatever the service later becomes.
const PERIOD_DAYS = { weekly: 7, monthly: 30 } as const;
/** Spreads `userId` over `[0, days)` days, the same way each run. */
function offsetOf(userId: string, days: number): number {
let hash = 2166136261;
for (const ch of userId) hash = Math.imul(hash ^ ch.charCodeAt(0), 16777619);
return ((hash >>> 0) % (days * 24)) * 3_600_000;
}
/**
* Run once by `runMigrations`. Every user present gets both cadences, with a
* first period that ends at some hour of the coming week or month, picked
* from their id: the digests start trickling out after the deploy instead of
* all leaving a week later. Users created later are scheduled by the service
* when it first sees them.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS digest_schedule (
user_id TEXT NOT NULL,
cadence TEXT NOT NULL,
window_start TEXT NOT NULL,
PRIMARY KEY (user_id, cadence),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
`);
const users = db.prepare(`SELECT id FROM users;`).all() as { id: string }[];
const insert = db.prepare(
`INSERT OR IGNORE INTO digest_schedule (user_id, cadence, window_start) VALUES (?, ?, ?);`,
);
const now = Date.now();
for (const { id } of users) {
for (const [cadence, days] of Object.entries(PERIOD_DAYS)) {
const start = now - days * DAY_MS + offsetOf(id, days);
insert.run(id, cadence, new Date(start).toISOString());
}
}
}

239
api/db/migrations/0021_page_urls.ts Normal file
View File

@@ -0,0 +1,239 @@
/**
* Moves the site to v4's page URLs, where dumps and playlists live under their
* owner (`/@bob/dumps/<slug>`) and are named by an editable slug alone. A
* slug was `slugify(title)-<8 hex of the id>`, unique site-wide; it is now the
* bare `slugify(title)`, unique in its owner's space for its kind, and every
* slug an item or a user leaves behind stays an alias that redirects
* (`dump_slug_aliases`, `playlist_slug_aliases`, `user_aliases`).
*
* v3 URLs aren't redirected. Instead this rewrites the links to this site's
* pages in what users wrote: dump notes, comments, chat messages, and
* playlist and profile descriptions.
*
* SQLite can't make `slug` NOT NULL without rebuilding `dumps` and
* `playlists`, which other tables reference, so the column stays nullable
* and every writer sets it; the unique index is what the services rely on.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { slugify } from "../../../shared/slug.ts";
import {
dumpPath,
feedPath,
playlistPath,
type ProfileTab,
searchPath,
userPath,
} from "../../../shared/paths.ts";
import { type SlugKind, uniqueSlug } from "../../lib/slugify.ts";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
import { isSelfHost } from "../../lib/self-url.ts";
/** An item's new path, by its v3 slug and by its id. */
type PathsByKey = Map<string, string>;
/**
* Gives every dump or playlist of `kind` the bare slug of its title, unique
* per owner, the oldest first so it keeps the bare one. Returns each one's new
* path under both keys a v3 link may have used: its old slug and its id.
*/
function reslug(db: DatabaseSync, kind: SlugKind): PathsByKey {
const table = kind === "dump" ? "dumps" : "playlists";
const rows = db.prepare(
`SELECT t.id, t.user_id, t.title, t.slug, u.username
FROM ${table} t JOIN users u ON u.id = t.user_id
ORDER BY t.created_at, t.id;`,
).all() as {
id: string;
user_id: string;
title: string;
slug: string | null;
username: string;
}[];
// Cleared first, so that an old slug doesn't take a new one's place.
db.exec(`UPDATE ${table} SET slug = NULL;`);
const set = db.prepare(`UPDATE ${table} SET slug = ? WHERE id = ?;`);
const paths: PathsByKey = new Map();
const build = kind === "dump" ? dumpPath : playlistPath;
for (const row of rows) {
const slug = uniqueSlug(db, kind, row.user_id, slugify(row.title));
set.run(slug, row.id);
const path = build({ ownerUsername: row.username, slug });
paths.set(row.id, path);
if (row.slug) paths.set(row.slug, path);
}
return paths;
}
/** Where each v3 profile tab went: a tab, the overview, or a page of its own. */
const PROFILE_TAB_V4: Record<string, ProfileTab | "" | "/settings"> = {
dumps: "",
playlists: "playlists",
followed: "follow",
invitees: "invitees",
settings: "/settings",
manage: "manage",
};
/** The v3 full lists under a profile, which are v4 tabs. */
const PROFILE_LISTS = new Set<string>(["dumps", "upvoted", "playlists"]);
/**
* Matches a link to a v3 page, its origin being optional: an absolute one is
* this site's only when its host is, a relative one only as a Markdown link's
* target. The path stops before a query or a fragment, which stay as they are.
*/
const V3_LINK_RE =
/(https?:\/\/([^/\s)\]<>"']+))?(\/(?:dumps|playlists|users|~|search\/~)\/[^\s)\]<>"'?#]+)/g;
/**
* Returns the v4 path of the v3 page at `path`, or null when it names nothing
* known, which then stays as written: a deleted dump's link is dead either way.
*/
function v4PathOf(
path: string,
dumps: PathsByKey,
playlists: PathsByKey,
username: (name: string) => string | undefined,
): string | null {
let segments: string[];
try {
segments = path.split("/").slice(1).filter(Boolean).map(
decodeURIComponent,
);
} catch {
return null;
}
const [root, key, ...rest] = segments;
switch (root) {
case "dumps": {
const target = dumps.get(key);
if (!target) return null;
if (rest.length === 0) return target;
return rest.length === 1 && rest[0] === "edit" ? `${target}/edit` : null;
}
case "playlists":
return rest.length === 0 ? playlists.get(key) ?? null : null;
case "users": {
const name = username(key) ?? key;
if (rest.length === 0) return userPath(name);
if (rest.length === 1 && PROFILE_LISTS.has(rest[0])) {
return userPath(name, rest[0] as ProfileTab);
}
if (rest.length === 2 && rest[0] === "~" && rest[1] in PROFILE_TAB_V4) {
const tab = PROFILE_TAB_V4[rest[1]];
if (tab === "/settings") return tab;
return tab ? userPath(name, tab) : userPath(name);
}
return null;
}
case "~":
if (rest.length > 0) return null;
return key === "followed"
? feedPath(undefined, "follow")
: ["hot", "new", "journal"].includes(key)
? feedPath(undefined, key as "hot")
: null;
case "search":
// `/search/~/users`, split as "search", "~", "users".
return key === "~" && (rest[0] === "users" || rest[0] === "playlists") &&
rest.length === 1
? searchPath(rest[0])
: null;
}
return null;
}
/**
* Rewrites the links to v3 pages of this site in the text column `column` of
* `table`, in every row that has one.
*/
function rewriteLinks(
db: DatabaseSync,
table: string,
column: string,
toV4: (path: string) => string | null,
): void {
const rows = db.prepare(
`SELECT id, ${column} AS text FROM ${table} WHERE ${column} IS NOT NULL;`,
).all() as { id: string; text: string }[];
const update = db.prepare(`UPDATE ${table} SET ${column} = ? WHERE id = ?;`);
for (const { id, text } of rows) {
const rewritten = text.replace(
V3_LINK_RE,
(match, origin: string | undefined, host: string, path: string, at) => {
if (origin ? !isSelfHost(host) : text[at - 1] !== "(") return match;
const v4 = toV4(path);
return v4 ? `${origin ?? ""}${v4}` : match;
},
);
if (rewritten !== text) update.run(rewritten, id);
}
}
/**
* Run once by `runMigrations`. On a database made from schema.sql the tables
* and indexes exist and there is nothing to re-slug or rewrite.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS dump_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
dump_id TEXT NOT NULL REFERENCES dumps(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX IF NOT EXISTS idx_dump_slug_aliases_dump
ON dump_slug_aliases(dump_id, created_at);
CREATE TABLE IF NOT EXISTS playlist_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
playlist_id TEXT NOT NULL REFERENCES playlists(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX IF NOT EXISTS idx_playlist_slug_aliases_playlist
ON playlist_slug_aliases(playlist_id, created_at);
CREATE TABLE IF NOT EXISTS user_aliases (
username TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (username)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_aliases_lower
ON user_aliases(lower(username));
CREATE INDEX IF NOT EXISTS idx_user_aliases_user
ON user_aliases(user_id, created_at);
DROP INDEX IF EXISTS idx_dumps_slug;
DROP INDEX IF EXISTS idx_playlists_slug;
`);
const dumps = reslug(db, "dump");
const playlists = reslug(db, "playlist");
db.exec(`
CREATE UNIQUE INDEX IF NOT EXISTS idx_dumps_user_slug
ON dumps(user_id, slug);
CREATE UNIQUE INDEX IF NOT EXISTS idx_playlists_user_slug
ON playlists(user_id, slug);
`);
renameReservedCategories(db);
const findUser = db.prepare(
`SELECT username FROM users WHERE lower(username) = lower(?);`,
);
const username = (name: string) =>
(findUser.get(name) as { username: string } | undefined)?.username;
const toV4 = (path: string) => v4PathOf(path, dumps, playlists, username);
for (
const [table, column] of [
["dumps", "comment"],
["comments", "body"],
["chat_messages", "body"],
["playlists", "description"],
["users", "description"],
]
) rewriteLinks(db, table, column, toV4);
}

14
api/db/migrations/0022_reserve_community.ts Normal file
View File

@@ -0,0 +1,14 @@
/**
* Reserves `community` at the URL root for the community page, renaming a
* category that had it as its slug.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/** Run once by `runMigrations`. */
export function up(db: DatabaseSync): void {
renameReservedCategories(db);
}

74
api/db/migrations/0023_mention_everyone.ts Normal file
View File

@@ -0,0 +1,74 @@
/**
* Prepares the chat's `@everyone`: it adds `chat_messages.mentions_everyone`,
* and frees the name `everyone`, which no account may have any more.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { EVERYONE, mentionsEveryone } from "../../../shared/mention.ts";
import { ROLE_PERMISSIONS } from "../../../shared/permissions.ts";
/**
* Run once by `runMigrations`. The messages that a moderator or an admin
* already sent with `@everyone` get the flag, so that editing one later
* doesn't notify everyone of an old message; none is notified now. A user
* named `everyone`, in any case, is renamed to the first free `everyone_2`,
* `everyone_3`…, with no alias left behind, and an alias `everyone` is
* dropped. Both are logged.
*/
export function up(db: DatabaseSync): void {
const cols = db.prepare(`PRAGMA table_info(chat_messages);`).all() as {
name: string;
}[];
if (!cols.some((c) => c.name === "mentions_everyone")) {
db.exec(
`ALTER TABLE chat_messages ADD COLUMN mentions_everyone INTEGER NOT NULL DEFAULT 0;`,
);
}
const roles = Object.entries(ROLE_PERMISSIONS)
.filter(([, perms]) => perms.includes("chat:everyone"))
.map(([role]) => role);
const messages = db.prepare(
`SELECT m.id, m.body FROM chat_messages m JOIN users u ON u.id = m.user_id
WHERE m.mentions_everyone = 0
AND u.role IN (${roles.map(() => "?").join(", ")})
AND m.body LIKE '%@%';`,
).all(...roles) as { id: string; body: string }[];
const flag = db.prepare(
`UPDATE chat_messages SET mentions_everyone = 1 WHERE id = ?;`,
);
for (const m of messages) {
if (mentionsEveryone(m.body)) flag.run(m.id);
}
const taken = (name: string) =>
!!db.prepare(
`SELECT 1 FROM users WHERE lower(username) = lower(?)
UNION ALL SELECT 1 FROM user_aliases WHERE lower(username) = lower(?);`,
).get(name, name);
const user = db.prepare(
`SELECT id, username FROM users WHERE lower(username) = ?;`,
).get(EVERYONE) as { id: string; username: string } | undefined;
if (user) {
let n = 2;
while (taken(`${EVERYONE}_${n}`)) n++;
const renamed = `${EVERYONE}_${n}`;
db.prepare(`UPDATE users SET username = ? WHERE id = ?;`).run(
renamed,
user.id,
);
console.log(
`[migrate] user "${user.username}" is now "${renamed}": @${EVERYONE} is reserved`,
);
}
const alias = db.prepare(
`DELETE FROM user_aliases WHERE lower(username) = ? RETURNING user_id;`,
).get(EVERYONE) as { user_id: string } | undefined;
if (alias) {
console.log(
`[migrate] the former name "${EVERYONE}" of user ${alias.user_id} is dropped: @${EVERYONE} is reserved`,
);
}
}

105
api/db/migrations/0024_federation.ts Normal file
View File

@@ -0,0 +1,105 @@
/**
* Prepares federated sign-in, by which a user signs in here with their
* account on another gerbeur instance, and the other way round. It adds the
* instance's signing key, the link from a local account to the one it stands
* for elsewhere, the instances an admin trusts or blocks, the state of the
* sign-ins in progress on both sides, the grants users gave other instances,
* and the events owed to them.
*
* `/authorize` and `/federation/callback` become pages, so their first
* segments are reserved.
*
* An account from another instance has no password here: its `password_hash`
* is `''`, which no password verifies. Emails stay unique among local accounts
* only, since the same person may also have a local account, or come from two
* instances, with one address.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/**
* Run once by `runMigrations`. Nothing exists to convert, since every account
* so far is local, but `authorize` and `federation` become the pages of a
* sign-in, so a category that had one as its slug is renamed.
*/
export function up(db: DatabaseSync): void {
db.exec(`
CREATE TABLE IF NOT EXISTS federation_keys (
kid TEXT PRIMARY KEY,
private_jwk TEXT NOT NULL,
public_jwk TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS user_identities (
user_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
issuer TEXT NOT NULL,
subject TEXT NOT NULL,
picture TEXT,
synced_at TEXT NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_identities_subject
ON user_identities(issuer, subject);
CREATE TABLE IF NOT EXISTS federation_peers (
issuer TEXT PRIMARY KEY,
status TEXT NOT NULL,
name TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS federation_logins (
state TEXT PRIMARY KEY,
issuer TEXT NOT NULL,
code_verifier TEXT NOT NULL,
nonce TEXT NOT NULL,
invite_token TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS oauth_codes (
code_hash TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
redirect_uri TEXT NOT NULL,
code_challenge TEXT NOT NULL,
nonce TEXT NOT NULL,
expires_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS federation_grants (
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
client_name TEXT,
granted_at TEXT NOT NULL,
last_used_at TEXT NOT NULL,
PRIMARY KEY (user_id, client_id)
);
CREATE TABLE IF NOT EXISTS federation_outbox (
id TEXT PRIMARY KEY,
client_id TEXT NOT NULL,
event TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_federation_outbox_next
ON federation_outbox(next_attempt_at);
`);
const emailIndex = db.prepare(
`SELECT sql FROM sqlite_master WHERE name = 'idx_users_email_lower';`,
).get() as { sql: string } | undefined;
if (!emailIndex?.sql.includes("WHERE")) {
db.exec(`
DROP INDEX IF EXISTS idx_users_email_lower;
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email))
WHERE password_hash != '';
`);
}
renameReservedCategories(db);
}

110
api/db/migrations/0025_playlist_categories.ts Normal file
View File

@@ -0,0 +1,110 @@
/**
* Gives playlists categories of their own, a set apart from the dumps': each
* category gets a `kind`, `dump` or `playlist`, its slug and name now unique
* within its kind only, and `playlist_categories` files playlists under them.
*
* `/playlists` becomes the page of every public playlist, so its first segment
* is reserved.
*
* @module
*/
import type { DatabaseSync } from "node:sqlite";
import { renameReservedCategories } from "../../lib/reserved-slugs.ts";
/**
* Run once by `runMigrations`. Every existing category becomes a dump
* category, filing the same dumps.
*
* SQLite can't change a table's UNIQUE constraints in place, so `categories`
* is rebuilt. `foreign_keys` can't be switched off inside the runner's
* transaction, and dropping `categories` would then empty `dump_categories`
* through its cascade, so both are read, dropped child first, made anew as in
* schema.sql, and refilled. A database made from schema.sql already has the
* `kind` column and is left as it is.
*/
export function up(db: DatabaseSync): void {
const columns = new Set(
(db.prepare(`PRAGMA table_info(categories);`).all() as { name: string }[])
.map((c) => c.name),
);
if (!columns.has("kind")) {
const categories = db.prepare(
`SELECT id, slug, name, position, created_at, updated_at FROM categories;`,
).all() as {
id: string;
slug: string;
name: string;
position: number;
created_at: string;
updated_at: string | null;
}[];
const links = db.prepare(
`SELECT dump_id, category_id FROM dump_categories;`,
).all() as { dump_id: string; category_id: string }[];
db.exec(`DROP TABLE dump_categories;`);
db.exec(`DROP TABLE categories;`);
db.exec(
`CREATE TABLE categories (
id TEXT PRIMARY KEY,
kind TEXT NOT NULL DEFAULT 'dump' CHECK (kind IN ('dump', 'playlist')),
slug TEXT NOT NULL,
name TEXT NOT NULL,
position INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
updated_at TEXT,
UNIQUE (kind, slug),
UNIQUE (kind, name)
);`,
);
db.exec(
`CREATE TABLE dump_categories (
dump_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (dump_id, category_id),
FOREIGN KEY (dump_id) REFERENCES dumps(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);`,
);
db.exec(
`CREATE INDEX idx_dump_categories_category ON dump_categories(category_id);`,
);
const insertCategory = db.prepare(
`INSERT INTO categories (id, kind, slug, name, position, created_at, updated_at)
VALUES (?, 'dump', ?, ?, ?, ?, ?);`,
);
for (const c of categories) {
insertCategory.run(
c.id,
c.slug,
c.name,
c.position,
c.created_at,
c.updated_at,
);
}
const insertLink = db.prepare(
`INSERT INTO dump_categories (dump_id, category_id) VALUES (?, ?);`,
);
for (const l of links) insertLink.run(l.dump_id, l.category_id);
}
db.exec(
`CREATE TABLE IF NOT EXISTS playlist_categories (
playlist_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (playlist_id, category_id),
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);`,
);
db.exec(
`CREATE INDEX IF NOT EXISTS idx_playlist_categories_category ON playlist_categories(category_id);`,
);
renameReservedCategories(db);
}

139
api/db/schema.sql
View File

@@ -69,11 +69,14 @@ CREATE TABLE playlist_dumps (
CREATE TABLE categories ( CREATE TABLE categories (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
slug TEXT NOT NULL UNIQUE, kind TEXT NOT NULL DEFAULT 'dump' CHECK (kind IN ('dump', 'playlist')),
name TEXT NOT NULL UNIQUE, slug TEXT NOT NULL,
name TEXT NOT NULL,
position INTEGER NOT NULL DEFAULT 0, position INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL, created_at TEXT NOT NULL,
updated_at TEXT updated_at TEXT,
UNIQUE (kind, slug),
UNIQUE (kind, name)
); );
CREATE TABLE dump_categories ( CREATE TABLE dump_categories (
@@ -84,6 +87,14 @@ CREATE TABLE dump_categories (
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
); );
CREATE TABLE playlist_categories (
playlist_id TEXT NOT NULL,
category_id TEXT NOT NULL,
PRIMARY KEY (playlist_id, category_id),
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
FOREIGN KEY (category_id) REFERENCES categories(id) ON DELETE CASCADE
);
CREATE TABLE comments ( CREATE TABLE comments (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
dump_id TEXT NOT NULL, dump_id TEXT NOT NULL,
@@ -121,12 +132,12 @@ CREATE TABLE dump_backlinks (
CREATE INDEX idx_dumps_created ON dumps(created_at); CREATE INDEX idx_dumps_created ON dumps(created_at);
CREATE INDEX idx_dumps_user_created ON dumps(user_id, created_at); CREATE INDEX idx_dumps_user_created ON dumps(user_id, created_at);
CREATE INDEX idx_dumps_slug ON dumps(slug); CREATE UNIQUE INDEX idx_dumps_user_slug ON dumps(user_id, slug);
CREATE INDEX idx_dumps_url ON dumps(url); CREATE INDEX idx_dumps_url ON dumps(url);
CREATE INDEX idx_dumps_url_canonical ON dumps(url_canonical); CREATE INDEX idx_dumps_url_canonical ON dumps(url_canonical);
CREATE INDEX idx_votes_user_created ON votes(user_id, created_at); CREATE INDEX idx_votes_user_created ON votes(user_id, created_at);
CREATE INDEX idx_playlists_user ON playlists(user_id); CREATE INDEX idx_playlists_user ON playlists(user_id);
CREATE INDEX idx_playlists_slug ON playlists(slug); CREATE UNIQUE INDEX idx_playlists_user_slug ON playlists(user_id, slug);
CREATE INDEX idx_playlist_dumps_order ON playlist_dumps(playlist_id, position); CREATE INDEX idx_playlist_dumps_order ON playlist_dumps(playlist_id, position);
CREATE INDEX idx_playlist_dumps_dump ON playlist_dumps(dump_id); CREATE INDEX idx_playlist_dumps_dump ON playlist_dumps(dump_id);
CREATE INDEX idx_comments_dump ON comments(dump_id, created_at); CREATE INDEX idx_comments_dump ON comments(dump_id, created_at);
@@ -136,6 +147,7 @@ CREATE INDEX idx_comment_likes_user ON comment_likes(user_id);
CREATE INDEX idx_dump_backlinks_to ON dump_backlinks(to_dump_id); CREATE INDEX idx_dump_backlinks_to ON dump_backlinks(to_dump_id);
CREATE INDEX idx_dump_backlinks_from ON dump_backlinks(from_dump_id); CREATE INDEX idx_dump_backlinks_from ON dump_backlinks(from_dump_id);
CREATE INDEX idx_dump_categories_category ON dump_categories(category_id); CREATE INDEX idx_dump_categories_category ON dump_categories(category_id);
CREATE INDEX idx_playlist_categories_category ON playlist_categories(category_id);
CREATE TABLE follows ( CREATE TABLE follows (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
@@ -175,13 +187,18 @@ CREATE TABLE invites (
CREATE INDEX idx_invites_inviter ON invites(inviter_id); CREATE INDEX idx_invites_inviter ON invites(inviter_id);
CREATE INDEX idx_users_invited_by ON users(invited_by); CREATE INDEX idx_users_invited_by ON users(invited_by);
CREATE INDEX idx_users_username_lower ON users(lower(username)); CREATE UNIQUE INDEX idx_users_username_lower ON users(lower(username));
-- An account from another instance has no password (''), and its email may
-- also be a local account's.
CREATE UNIQUE INDEX idx_users_email_lower ON users(lower(email))
WHERE password_hash != '';
CREATE TABLE attachments ( CREATE TABLE attachments (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
resource_id TEXT, resource_id TEXT,
mime TEXT NOT NULL, mime TEXT NOT NULL,
created_at TEXT NOT NULL created_at TEXT NOT NULL,
resource_type TEXT
); );
CREATE INDEX idx_attachments_resource ON attachments(resource_id); CREATE INDEX idx_attachments_resource ON attachments(resource_id);
@@ -211,6 +228,51 @@ CREATE UNIQUE INDEX idx_notifications_dedup
ON notifications(user_id, source_key) ON notifications(user_id, source_key)
WHERE source_key IS NOT NULL; WHERE source_key IS NOT NULL;
CREATE TABLE user_settings (
user_id TEXT PRIMARY KEY,
style TEXT,
color_scheme TEXT,
digest_weekly INTEGER NOT NULL DEFAULT 1,
digest_monthly INTEGER NOT NULL DEFAULT 1,
updated_at TEXT NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE TABLE digest_schedule (
user_id TEXT NOT NULL,
cadence TEXT NOT NULL,
window_start TEXT NOT NULL,
PRIMARY KEY (user_id, cadence),
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);
CREATE TABLE dump_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
dump_id TEXT NOT NULL REFERENCES dumps(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX idx_dump_slug_aliases_dump ON dump_slug_aliases(dump_id, created_at);
CREATE TABLE playlist_slug_aliases (
user_id TEXT NOT NULL,
slug TEXT NOT NULL,
playlist_id TEXT NOT NULL REFERENCES playlists(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (user_id, slug)
);
CREATE INDEX idx_playlist_slug_aliases_playlist ON playlist_slug_aliases(playlist_id, created_at);
CREATE TABLE user_aliases (
username TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
created_at TEXT NOT NULL,
PRIMARY KEY (username)
);
CREATE UNIQUE INDEX idx_user_aliases_lower ON user_aliases(lower(username));
CREATE INDEX idx_user_aliases_user ON user_aliases(user_id, created_at);
CREATE TABLE chat_messages ( CREATE TABLE chat_messages (
id TEXT NOT NULL PRIMARY KEY, id TEXT NOT NULL PRIMARY KEY,
user_id TEXT NOT NULL, user_id TEXT NOT NULL,
@@ -218,7 +280,70 @@ CREATE TABLE chat_messages (
created_at TEXT NOT NULL, created_at TEXT NOT NULL,
updated_at TEXT, updated_at TEXT,
reply_to_id TEXT, reply_to_id TEXT,
mentions_everyone INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
); );
CREATE INDEX idx_chat_messages_created_id ON chat_messages(created_at, id); CREATE INDEX idx_chat_messages_created_id ON chat_messages(created_at, id);
CREATE INDEX idx_chat_messages_user ON chat_messages(user_id); CREATE INDEX idx_chat_messages_user ON chat_messages(user_id);
CREATE TABLE federation_keys (
kid TEXT PRIMARY KEY,
private_jwk TEXT NOT NULL,
public_jwk TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE user_identities (
user_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
issuer TEXT NOT NULL,
subject TEXT NOT NULL,
picture TEXT,
synced_at TEXT NOT NULL
);
CREATE UNIQUE INDEX idx_user_identities_subject
ON user_identities(issuer, subject);
CREATE TABLE federation_peers (
issuer TEXT PRIMARY KEY,
status TEXT NOT NULL,
name TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE federation_logins (
state TEXT PRIMARY KEY,
issuer TEXT NOT NULL,
code_verifier TEXT NOT NULL,
nonce TEXT NOT NULL,
invite_token TEXT,
created_at TEXT NOT NULL
);
CREATE TABLE oauth_codes (
code_hash TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
redirect_uri TEXT NOT NULL,
code_challenge TEXT NOT NULL,
nonce TEXT NOT NULL,
expires_at TEXT NOT NULL
);
CREATE TABLE federation_grants (
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
client_id TEXT NOT NULL,
client_name TEXT,
granted_at TEXT NOT NULL,
last_used_at TEXT NOT NULL,
PRIMARY KEY (user_id, client_id)
);
CREATE TABLE federation_outbox (
id TEXT PRIMARY KEY,
client_id TEXT NOT NULL,
event TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE INDEX idx_federation_outbox_next ON federation_outbox(next_attempt_at);

69
api/db/startup.ts
View File

@@ -1,57 +1,80 @@
/** /**
* Housekeeping run when the database is opened. * Deletes what has expired and seeds the first admin, each time connection.ts
* opens the database.
* *
* @module * @module
*/ */
import { randomBytes, scryptSync } from "node:crypto";
import type { DatabaseSync } from "node:sqlite"; import type { DatabaseSync } from "node:sqlite";
import { import {
ATTACHMENTS_DIR, ATTACHMENTS_DIR,
ORPHANED_ATTACHMENTS_RETENTION_HOURS, ORPHANED_ATTACHMENTS_RETENTION_HOURS,
UNUSED_INVITES_RETENTION_DAYS, UNUSED_INVITES_RETENTION_DAYS,
} from "../config.ts"; } from "../config.ts";
import { removeScaledImages } from "../lib/ffmpeg.ts";
import { hashPassword } from "../lib/password.ts";
/** /**
* Runs once when the database is opened, after the migrations: drops what has * Returns the time `ms` ago as an ISO string, the format dates are stored in.
* expired (reset tokens, unused invites, orphaned attachments), and seeds the */
* default admin (admin/admin) on an empty instance. function isoAgo(ms: number): string {
return new Date(Date.now() - ms).toISOString();
}
/**
* Runs after the migrations, each time the server starts. It deletes used or
* expired reset tokens, invites past their validity, and attachments nothing
* links to or whose resource is gone. On an instance with no users, it creates
* the user `admin` with the password `admin`, which whoever installs it must
* change.
*/ */
export async function runStartupTasks(db: DatabaseSync): Promise<void> { export async function runStartupTasks(db: DatabaseSync): Promise<void> {
// Purge expired/used password reset tokens // Dates are stored as ISO strings and compared as text, so the cutoffs are
// ISO strings too. SQLite's datetime() has a space where ISO has a `T`.
db.prepare( db.prepare(
`DELETE FROM password_reset_tokens WHERE expires_at < datetime('now') OR used_at IS NOT NULL;`, `DELETE FROM password_reset_tokens WHERE expires_at < ? OR used_at IS NOT NULL;`,
).run(); ).run(isoAgo(0));
// Purge expired unused invites
db.prepare( db.prepare(
`DELETE FROM invites WHERE used_at IS NULL AND created_at < datetime('now', '-${UNUSED_INVITES_RETENTION_DAYS} days');`, `DELETE FROM invites WHERE used_at IS NULL AND created_at < ?;`,
).run(); ).run(isoAgo(UNUSED_INVITES_RETENTION_DAYS * 86_400_000));
// Prune orphaned attachments (uploaded but never linked to a resource) older than 1 hour // An image is uploaded as soon as it's pasted, and linked only when the text
// it's pasted into is saved, so a fresh unlinked one is left alone. One whose
// dump, comment, playlist, message or user is gone goes whatever its age.
const attachmentCutoff = isoAgo(
ORPHANED_ATTACHMENTS_RETENTION_HOURS * 3_600_000,
);
const ORPHANED = `
(resource_id IS NULL AND created_at < ?)
OR (resource_id IS NOT NULL AND NOT CASE resource_type
WHEN 'dump' THEN EXISTS (SELECT 1 FROM dumps WHERE id = resource_id)
WHEN 'comment' THEN EXISTS (SELECT 1 FROM comments WHERE id = resource_id AND deleted = 0)
WHEN 'playlist' THEN EXISTS (SELECT 1 FROM playlists WHERE id = resource_id)
WHEN 'chat' THEN EXISTS (SELECT 1 FROM chat_messages WHERE id = resource_id)
WHEN 'user' THEN EXISTS (SELECT 1 FROM users WHERE id = resource_id)
ELSE 0
END)`;
const orphanedAttachments = db.prepare( const orphanedAttachments = db.prepare(
`SELECT id FROM attachments WHERE resource_id IS NULL AND created_at < datetime('now', '-${ORPHANED_ATTACHMENTS_RETENTION_HOURS} hour');`, `SELECT id FROM attachments WHERE ${ORPHANED};`,
).all() as { id: string }[]; ).all(attachmentCutoff) as { id: string }[];
if (orphanedAttachments.length > 0) { if (orphanedAttachments.length > 0) {
for (const { id } of orphanedAttachments) { for (const { id } of orphanedAttachments) {
await Deno.remove(`${ATTACHMENTS_DIR}/${id}`).catch(() => {}); await Deno.remove(`${ATTACHMENTS_DIR}/${id}`).catch(() => {});
await removeScaledImages(`${ATTACHMENTS_DIR}/${id}`);
} }
db.prepare( db.prepare(`DELETE FROM attachments WHERE ${ORPHANED};`)
`DELETE FROM attachments WHERE resource_id IS NULL AND created_at < datetime('now', '-${ORPHANED_ATTACHMENTS_RETENTION_HOURS} hour');`, .run(attachmentCutoff);
).run();
} }
// Create default admin user if no users exist // Registration needs an invite, so the first user can only come from here.
const userCount = db.prepare(`SELECT COUNT(*) as count FROM users`).get() as { const userCount = db.prepare(`SELECT COUNT(*) as count FROM users`).get() as {
count: number; count: number;
}; };
if (userCount.count === 0) { if (userCount.count === 0) {
const salt = randomBytes(16).toString("hex");
const hash = scryptSync("admin", salt, 64).toString("hex");
const passwordHash = `${hash}.${salt}`;
db.prepare( db.prepare(
`INSERT INTO users (id, username, password_hash, role, created_at, email) VALUES (?, 'admin', ?, 'admin', datetime('now'), 'admin@localhost')`, `INSERT INTO users (id, username, password_hash, role, created_at, email) VALUES (?, 'admin', ?, 'admin', ?, 'admin@localhost')`,
).run(crypto.randomUUID(), passwordHash); ).run(crypto.randomUUID(), await hashPassword("admin"), isoAgo(0));
console.log( console.log(
"Created default admin user (username: admin, password: admin)", "Created default admin user (username: admin, password: admin)",
); );

35
api/lib/api-exception.ts
View File

@@ -1,25 +1,40 @@
/** /**
* The exception that carries an HTTP error out of any layer below a route. * Lets any layer fail a request with an HTTP status, which the error middleware
* turns into the response.
* *
* @module * @module
*/ */
import type { APIErrorCode } from "../../shared/api.ts"; import { APIErrorCode } from "../../shared/api.ts";
/** Passed to an {@link APIException}; any other status is a type error. */
export type APIStatus = 400 | 401 | 403 | 404 | 409 | 429 | 500 | 502;
const CODES: Record<APIStatus, APIErrorCode> = {
400: APIErrorCode.BAD_REQUEST,
401: APIErrorCode.UNAUTHORIZED,
403: APIErrorCode.FORBIDDEN,
404: APIErrorCode.NOT_FOUND,
409: APIErrorCode.CONFLICT,
429: APIErrorCode.RATE_LIMITED,
500: APIErrorCode.SERVER_ERROR,
502: APIErrorCode.BAD_GATEWAY,
};
/** /**
* An error meant for the client, thrown anywhere below a route; the error * Thrown by a route, a service or a remote fetch to answer the request with an
* middleware turns it into an `APIFailure` response. * error. The error middleware sends it as an `APIFailure`, message included,
* so the message must be fit for the user. Use 401 for a caller who isn't
* signed in, 403 for one who is but may not, and 404 rather than 403 when the
* caller mustn't learn that the thing exists.
*/ */
export class APIException extends Error { export class APIException extends Error {
/** The error code the response carries. */
readonly code: APIErrorCode; readonly code: APIErrorCode;
/** The HTTP status the response is sent with. */ readonly status: APIStatus;
readonly status: number;
/** Creates an exception answering `status` with `code` and `message`. */ constructor(status: APIStatus, message: string) {
constructor(code: APIErrorCode, status: number, message: string) {
super(message); super(message);
this.code = code;
this.status = status; this.status = status;
this.code = CODES[status];
} }
} }

48
api/lib/auth.ts
View File

@@ -1,26 +1,23 @@
/** /**
* Authenticates the tokens a request carries: session tokens in the * Checks session and media tokens against the live user row, not just their
* `Authorization` header and media tokens in a media URL's query string. * signature. The auth middleware, the WebSocket route, the media routes and the
* routes open to anonymous callers all authenticate through it.
* *
* @module * @module
*/ */
import type { Context } from "@oak/oak"; import type { Context } from "@oak/oak";
import { verifyJWT, verifyMediaToken } from "./jwt.ts"; import { verifyJWT, verifyMediaToken } from "./jwt.ts";
import { APIErrorCode } from "../../shared/api.ts";
import { APIException } from "./api-exception.ts"; import { APIException } from "./api-exception.ts";
import type { AuthPayload } from "../model/auth.ts"; import type { AuthPayload } from "../model/auth.ts";
import type { Role } from "../../shared/user.ts"; import type { Role } from "../../shared/user.ts";
import { getSessionState } from "../services/user-service.ts"; import { getSessionState } from "../services/user-service.ts";
/** /**
* Checks a signature-verified token against the live user row: the user must * Returns the live role of a verified token's user, or null when the user is
* still exist, and the token must not predate their last password change * gone or the token predates their last password change
* (`users.tokens_valid_after`). Returns the live role, or null to refuse. * (`users.tokens_valid_after`). `iat` counts whole seconds, so a token issued
* * in the same second as the change survives it.
* `iat` has one-second resolution, so a token issued in the same second as the
* change survives it; one issued after it (the fresh one the change hands back)
* always does.
*/ */
function checkTokenOwner( function checkTokenOwner(
userId: string, userId: string,
@@ -38,9 +35,10 @@ function checkTokenOwner(
} }
/** /**
* Verifies a session token end to end: signature, expiry, a user that still * Returns the payload of a valid session token, or null. Besides the signature
* exists and no password change since it was issued. The role comes from the * and expiry, it refuses a token whose user is gone or which predates a
* DB, not the token, since tokens last 7 days and roles change under them. * password change. The returned role comes from the database, not the token,
* since a token lasts 7 days and a role can change under it.
*/ */
export async function authenticate( export async function authenticate(
token: string, token: string,
@@ -53,8 +51,9 @@ export async function authenticate(
} }
/** /**
* Resolves the `?token=` on a media URL to the viewer's user id. Takes a media * Returns the user id behind the `?token=` of a file, thumbnail or playlist
* token, or a session token from a client that predates them. * cover URL, or null when it isn't valid. A session token is accepted too, for
* clients from before media tokens.
*/ */
export async function authenticateMedia(token: string): Promise<string | null> { export async function authenticateMedia(token: string): Promise<string | null> {
const media = await verifyMediaToken(token); const media = await verifyMediaToken(token);
@@ -65,18 +64,13 @@ export async function authenticateMedia(token: string): Promise<string | null> {
} }
/** /**
* Returns the user id of an optional Bearer token, or null for an anonymous * Returns the caller's user id on a route open to anonymous callers, or null
* request. * when the request has no token. A placeholder such as `Bearer undefined`
* counts as no token. An invalid token throws instead of reading as anonymous,
* so that a stale session gets a 401 the client signs out on, rather than a
* misleading 404 on a private resource.
* *
* Distinguishes *missing* credentials from *invalid* ones: * @throws {APIException} 401 when a token is present but invalid.
* - No `Authorization: Bearer …` header (or a placeholder value such as the
* `Bearer undefined` that the client emits when logged out) → anonymous (null).
* - A real token that fails verification (expired / bad signature / revoked) →
* throws 401, so a stale client session surfaces as an auth error it can react
* to, instead of being silently downgraded to anonymous (which yields a
* misleading 404 on private resources).
*
* @throws {APIException} 401 when a token is present but fails verification.
*/ */
export async function parseOptionalAuth(ctx: Context): Promise<string | null> { export async function parseOptionalAuth(ctx: Context): Promise<string | null> {
const authHeader = ctx.request.headers.get("Authorization"); const authHeader = ctx.request.headers.get("Authorization");
@@ -87,7 +81,7 @@ export async function parseOptionalAuth(ctx: Context): Promise<string | null> {
const payload = await authenticate(token); const payload = await authenticate(token);
if (!payload) { if (!payload) {
throw new APIException(APIErrorCode.UNAUTHORIZED, 401, "Invalid token"); throw new APIException(401, "Invalid token");
} }
return payload.userId; return payload.userId;
} }

59
api/lib/canonical-url.ts
View File

@@ -1,22 +1,20 @@
/** /**
* The canonical form of a URL, used *only* to recognise that two links point at the * Spots reposts and links to known dumps, by reducing a URL to a key shared by
* same thing — the duplicate check the create form runs while it fetches a * every link to the same page or video. `dump-service` stores it in
* preview. It is a lookup key, never something we display or fetch: `dumps.url` * `dumps.url_canonical` when a dump is posted or edited, `GET /api/dumps?url=`
* keeps the exact string the poster submitted. * matches on it to warn the create form of a repost, and backlink-service
* compares links by it. The key is never fetched or shown, so
* it may be lossy: it forces `https`, drops `www.`, tracking parameters and
* timestamps.
* *
* Because it is only ever compared against other canonical forms, it may be * Migration 0013 backfilled existing rows with a frozen copy of these rules.
* lossy in ways a real URL never could be — it forces `https`, drops `www.`, * Changing them needs a new backfill migration, or old rows keep old keys.
* and throws away share/tracking parameters and timestamps, so
* `http://www.example.com/a/?utm_source=x` and `https://example.com/a` collapse
* to one key.
*
* Stored in `dumps.url_canonical`. Migration 0013 carries a frozen copy of this
* logic to backfill existing rows; changing the rules here therefore needs a
* new backfill migration, or old rows keep keys computed under the old rules.
* *
* @module * @module
*/ */
import { siteHostOf } from "../../shared/url.ts";
const TRACKING_PARAMS = new Set([ const TRACKING_PARAMS = new Set([
"fbclid", "fbclid",
"gclid", "gclid",
@@ -42,7 +40,7 @@ function isTrackingParam(key: string): boolean {
return k.startsWith("utm_") || TRACKING_PARAMS.has(k); return k.startsWith("utm_") || TRACKING_PARAMS.has(k);
} }
/** YouTube's hosts, already stripped of a leading "www.". */ /** YouTube's hosts, without their "www.". */
const YOUTUBE_HOSTS = new Set([ const YOUTUBE_HOSTS = new Set([
"youtube.com", "youtube.com",
"m.youtube.com", "m.youtube.com",
@@ -51,12 +49,23 @@ const YOUTUBE_HOSTS = new Set([
]); ]);
/** /**
* Returns the video a YouTube URL points at, in any of the shapes people paste * Returns `u`'s host in lower case and without "www.", when it is one of
* (`youtu.be/ID`, `/watch?v=ID`, `/embed/ID`, `/shorts/ID`, `/live/ID`). * YouTube's (youtu.be aside), and null otherwise. The YouTube preview provider
* Timestamps are deliberately ignored: the same video linked at 2:30 is still * uses it too, so both agree on what a YouTube link is.
* the same video for duplicate purposes.
*/ */
function youtubeVideoId( export function youtubeHost(u: URL): string | null {
const host = siteHostOf(u);
return YOUTUBE_HOSTS.has(host) ? host : null;
}
/**
* Returns the id of the video a YouTube URL points at, in any of the shapes
* people paste (`youtu.be/ID`, `/watch?v=ID`, `/embed/ID`, `/shorts/ID`,
* `/live/ID`), and null for any other URL. `host` is lower case, without
* "www.". The canonical key drops the timestamp, since the same video linked
* at 2:30 is still a repost; the YouTube provider reads it separately.
*/
export function youtubeVideoId(
host: string, host: string,
pathname: string, pathname: string,
params: URLSearchParams, params: URLSearchParams,
@@ -71,8 +80,8 @@ function youtubeVideoId(
} }
/** /**
* Returns the canonical form of `raw` (see the module doc), or null when it * Returns the canonical key of `raw`, or null when it isn't an http(s) URL. A
* isn't an http(s) URL. * YouTube video or playlist becomes one fixed URL whatever shape it came in.
*/ */
export function canonicalizeUrl(raw: string): string | null { export function canonicalizeUrl(raw: string): string | null {
let u: URL; let u: URL;
@@ -83,7 +92,7 @@ export function canonicalizeUrl(raw: string): string | null {
} }
if (u.protocol !== "http:" && u.protocol !== "https:") return null; if (u.protocol !== "http:" && u.protocol !== "https:") return null;
const host = u.hostname.toLowerCase().replace(/^www\./, ""); const host = siteHostOf(u);
if (!host) return null; if (!host) return null;
const videoId = youtubeVideoId(host, u.pathname, u.searchParams); const videoId = youtubeVideoId(host, u.pathname, u.searchParams);
@@ -94,7 +103,7 @@ export function canonicalizeUrl(raw: string): string | null {
: null; : null;
if (listId) return `https://youtube.com/playlist?list=${listId}`; if (listId) return `https://youtube.com/playlist?list=${listId}`;
// A single trailing slash is never meaningful; "/" itself becomes "". // Trailing slashes are dropped; "/" itself becomes "".
const path = u.pathname.replace(/\/+$/, ""); const path = u.pathname.replace(/\/+$/, "");
const params = [...u.searchParams.entries()] const params = [...u.searchParams.entries()]
@@ -102,8 +111,8 @@ export function canonicalizeUrl(raw: string): string | null {
.sort(([a, av], [b, bv]) => a.localeCompare(b) || av.localeCompare(bv)); .sort(([a, av], [b, bv]) => a.localeCompare(b) || av.localeCompare(bv));
const query = new URLSearchParams(params).toString(); const query = new URLSearchParams(params).toString();
// A bare "#section" anchor points into the same page, so it is dropped — // A "#section" anchor points into the same page and is dropped, but "#/route"
// but "#/route" and "#!/route" address distinct pages of a hash-routed app. // and "#!/route" are distinct pages of a hash-routed app.
const hash = /^#!?\//.test(u.hash) ? u.hash : ""; const hash = /^#!?\//.test(u.hash) ? u.hash : "";
return `https://${host}${path}${query ? `?${query}` : ""}${hash}`; return `https://${host}${path}${query ? `?${query}` : ""}${hash}`;

91
api/lib/css-color.ts Normal file
View File

@@ -0,0 +1,91 @@
/**
* Reads the CSS colors the server meets: a dumped page's `theme-color`, which
* the generic link preview keeps as its tint (`normalizeCssColor`), and the
* theme stylesheets' tokens, which the emails resolve (`theme-tokens.ts`).
*
* @module
*/
/** An sRGB color with its alpha, each channel 0–255 and alpha 0–1. */
export interface Rgba {
r: number;
g: number;
b: number;
a: number;
}
/**
* The named colors sites actually put in `theme-color`, and `transparent`,
* which the themes write; the rest of CSS's names aren't worth carrying.
*/
const NAMED: Record<string, string> = {
transparent: "#00000000",
black: "#000000",
white: "#ffffff",
red: "#ff0000",
green: "#008000",
blue: "#0000ff",
yellow: "#ffff00",
orange: "#ffa500",
purple: "#800080",
gray: "#808080",
grey: "#808080",
silver: "#c0c0c0",
maroon: "#800000",
navy: "#000080",
teal: "#008080",
olive: "#808000",
lime: "#00ff00",
aqua: "#00ffff",
cyan: "#00ffff",
fuchsia: "#ff00ff",
magenta: "#ff00ff",
};
/** Reads a channel of `rgb()`, a number or a percentage of 255. */
function channel(text: string): number {
const n = parseFloat(text);
const value = text.endsWith("%") ? (n / 100) * 255 : n;
return Math.max(0, Math.min(255, value));
}
/** Reads the alpha of `rgba()`, a number or a percentage of 1. */
function alpha(text: string): number {
const n = parseFloat(text);
return Math.max(0, Math.min(1, text.endsWith("%") ? n / 100 : n));
}
/**
* Returns the color `text` writes as a hex (3, 4, 6 or 8 digits), an `rgb()`
* or `rgba()` of numbers or percentages, or one of the common names; undefined
* for any other form. Channels out of range are clamped.
*/
export function parseCssColor(text: string): Rgba | undefined {
const value = text.trim().toLowerCase();
const hex = /^#([0-9a-f]{3,8})$/.exec(NAMED[value] ?? value)?.[1];
if (hex) {
if (hex.length === 5 || hex.length === 7) return undefined;
const full = hex.length <= 4 ? [...hex].map((c) => c + c).join("") : hex;
const byte = (i: number) => parseInt(full.slice(i * 2, i * 2 + 2), 16);
return {
r: byte(0),
g: byte(1),
b: byte(2),
a: full.length === 8 ? byte(3) / 255 : 1,
};
}
const args = /^rgba?\(([^)]+)\)$/.exec(value)?.[1];
if (!args) return undefined;
const parts = args.split(/[\s,/]+/).filter(Boolean);
if (parts.length !== 3 && parts.length !== 4) return undefined;
const [r, g, b] = parts.slice(0, 3).map(channel);
const a = parts[3] === undefined ? 1 : alpha(parts[3]);
return [r, g, b, a].some(Number.isNaN) ? undefined : { r, g, b, a };
}
/** Writes `color` as a lowercase `#rrggbb`, its alpha dropped. */
export function hexOf({ r, g, b }: Rgba): string {
return `#${
[r, g, b].map((c) => Math.round(c).toString(16).padStart(2, "0")).join("")
}`;
}

46
api/lib/dump-media.ts Normal file
View File

@@ -0,0 +1,46 @@
/**
* Resolves the dump whose media a route under `/api/dumps/:dumpId/` serves:
* its file (`routes/files.ts`), its thumbnail (`routes/thumbnails.ts`) or its
* waveform (`routes/waveforms.ts`). Media elements and `<img>`s can't send an
* Authorization header, so these routes take the viewer's media token from
* the query string, and are cached by the file's version.
*
* @module
*/
import type { Context } from "@oak/oak";
import { APIException } from "./api-exception.ts";
import { authenticateMedia } from "./auth.ts";
import { type Dump, dumpFileVersion } from "../../shared/dump.ts";
import { UUID_RE } from "../../shared/validation.ts";
import { getDump } from "../services/dump-service.ts";
/**
* Returns dump `dumpId` as the viewer of `ctx`'s `?token=` may see it, and the
* Cache-Control its media get: kept for good under a URL naming the file's
* current version (`?v=`, as `dumpFileVersion` gives), revalidated under any
* other, and never in a shared cache for a private dump. It also keeps the
* token out of the Referer of whatever the response links to.
*
* @throws {APIException} 400 when `dumpId` isn't an id, which names files on
* disk, and 404 when the viewer can't see the dump.
*/
export async function resolveMediaDump(
ctx: Context,
dumpId: string,
): Promise<{ dump: Dump; cacheControl: string }> {
if (!UUID_RE.test(dumpId)) throw new APIException(400, "Invalid dump ID");
const token = ctx.request.url.searchParams.get("token");
const viewerId = token ? await authenticateMedia(token) : null;
const dump = getDump(dumpId, viewerId ?? undefined);
ctx.response.headers.set("Referrer-Policy", "no-referrer");
const scope = dump.isPrivate ? "private" : "public";
const current = ctx.request.url.searchParams.get("v") ===
dumpFileVersion(dump);
return {
dump,
cacheControl: current
? `${scope}, max-age=31536000, immutable`
: `${scope}, no-cache`,
};
}

512
api/lib/ffmpeg.ts Normal file
View File

@@ -0,0 +1,512 @@
/**
* Makes what the server derives from an upload with ffmpeg: a video's still,
* which the thumbnails route serves, an audio file's waveform, which the
* waveform route serves, and the scaled-down copies of any image, which
* `serveImage` (lib/upload.ts) serves for every route that answers one. Each
* is made on its first request and kept under a name this module gives, a
* dump's in `THUMBNAILS_DIR` and the scaled copies in `SCALED_DIR`, until
* dump-service replaces or deletes the upload and
* {@link removeGeneratedMedia} drops them. {@link generatedMediaSweepJob}
* takes what that leaves behind.
*
* @module
*/
import { dirname, relative } from "node:path";
import {
DUMPS_DIR,
SCALED_DIR,
THUMBNAILS_DIR,
UPLOADS_DIR,
} from "../config.ts";
import { IMAGE_WIDTHS, type ImageWidth } from "../../shared/image.ts";
import type { Job } from "./scheduler.ts";
import { type DumpWaveform, peaksOf } from "../../shared/waveform.ts";
import type { APIResponse } from "../../shared/api.ts";
/**
* Spares a process per uncached request: whether ffmpeg runs is probed once
* per process, and kept here.
*/
let ffmpegProbe: Promise<boolean> | undefined;
function ffmpegAvailable(): Promise<boolean> {
ffmpegProbe ??= (async () => {
try {
const { success } = await new Deno.Command("ffmpeg", {
args: ["-version"],
stderr: "null",
stdout: "null",
}).output();
return success;
} catch {
return false;
}
})();
return ffmpegProbe;
}
/**
* Lists the container formats ffmpeg may read a video as. The upload's type is
* only what the uploader claimed, and ffmpeg guesses the format from the bytes,
* so a "video" could be an HLS or concat playlist pointing it at local files or
* internal URLs. With this list, and `-protocol_whitelist file`, ffmpeg reads
* nothing but the file, and only as a video.
*/
const VIDEO_FORMATS =
"mov,mp4,m4a,3gp,3g2,mj2,matroska,webm,avi,flv,mpegts,mpeg,ogg,asf,h264,hevc";
/**
* Lists the formats ffmpeg may read an image as, for the reason given at
* {@link VIDEO_FORMATS}.
*/
const IMAGE_FORMATS = "jpeg_pipe,png_pipe,webp_pipe,bmp_pipe,gif";
/**
* Lists the formats ffmpeg may read an audio file as, for the reason given at
* {@link VIDEO_FORMATS}; an `.m4a` or a `.webm` is read by its container's.
*/
const AUDIO_FORMATS =
"mp3,ogg,flac,wav,aiff,aac,asf,caf,w64,wv,ape,mov,mp4,m4a,matroska,webm";
/**
* Returns where the still and the waveform generated from dump `dumpId`'s
* upload are kept.
*/
export function generatedPaths(dumpId: string): {
still: string;
waveform: string;
} {
const base = `${THUMBNAILS_DIR}/${dumpId}`;
return { still: `${base}.jpg`, waveform: `${base}.waveform.json` };
}
/**
* The formats a scaled copy comes in: WebP for a browser, and a JPEG or a PNG
* for what doesn't read it, as some mail clients don't.
*/
export type ScaledFormat = "webp" | "jpg" | "png";
const SCALED_FORMATS: readonly ScaledFormat[] = ["webp", "jpg", "png"];
/**
* Returns where the copy of the upload at `srcPath` scaled to `width` is
* kept, in `format`: under `SCALED_DIR`, at the upload's own path below
* `UPLOADS_DIR`.
*/
export function scaledImagePath(
srcPath: string,
width: ImageWidth,
format: ScaledFormat = "webp",
): string {
return `${SCALED_DIR}/${relative(UPLOADS_DIR, srcPath)}.w${width}.${format}`;
}
/**
* Deletes the scaled copies of the upload at `srcPath`, once it is deleted.
* One that is replaced needs no call: its copies are made again, being older
* than it ({@link ensureGenerated}).
*/
export async function removeScaledImages(srcPath: string): Promise<void> {
await Promise.all(
IMAGE_WIDTHS.flatMap((width) =>
SCALED_FORMATS.map((format) =>
Deno.remove(scaledImagePath(srcPath, width, format)).catch(() => {})
)
),
);
}
/**
* Deletes what was generated from dump `dumpId`'s upload, once dump-service
* has replaced or deleted it. The routes make them again from a new one.
*/
export async function removeGeneratedMedia(dumpId: string): Promise<void> {
const { still, waveform } = generatedPaths(dumpId);
await Promise.all([
...[still, waveform].map((path) => Deno.remove(path).catch(() => {})),
removeScaledImages(`${DUMPS_DIR}/${dumpId}`),
]);
}
/**
* Writes `outPath` with `write`, which gets a path beside it to fill, renamed
* over `outPath` once done, so a concurrent reader never gets half a file.
*/
async function writeAtomically(
outPath: string,
write: (tmpPath: string) => Promise<void>,
): Promise<void> {
await Deno.mkdir(dirname(outPath), { recursive: true });
const tmpPath = `${outPath}.${crypto.randomUUID()}.tmp`;
try {
await write(tmpPath);
await Deno.rename(tmpPath, outPath);
} catch (err) {
await Deno.remove(tmpPath).catch(() => {});
throw err;
}
}
/**
* Runs ffmpeg with `args`, writing to `outPath`.
*
* @throws {Error} When ffmpeg fails.
*/
function runFfmpeg(args: string[], outPath: string): Promise<void> {
return writeAtomically(outPath, async (tmpPath) => {
const { success } = await new Deno.Command("ffmpeg", {
args: ["-protocol_whitelist", "file", ...args, "-y", tmpPath],
stdout: "null",
stderr: "null",
}).output();
if (!success) throw new Error("ffmpeg failed");
});
}
/** Writes a 320-pixel-wide JPEG still of a video, one second in. */
export function videoStill(srcPath: string, outPath: string): Promise<void> {
return runFfmpeg([
"-format_whitelist",
VIDEO_FORMATS,
"-ss",
"00:00:01",
"-i",
srcPath,
"-frames:v",
"1",
"-vf",
"scale=320:-1",
"-f",
"image2",
"-c:v",
"mjpeg",
], outPath);
}
/**
* Maps a JPEG's EXIF orientation to the filters that turn its pixels upright.
* A browser applies the orientation to the upload itself; the copy carries no
* EXIF, so it is turned here, and not by ffmpeg's own autorotation, which
* only its recent versions apply to images.
*/
const ORIENTATION_FILTERS: Record<number, string> = {
2: "hflip",
3: "hflip,vflip",
4: "vflip",
5: "transpose=0",
6: "transpose=1",
7: "transpose=3",
8: "transpose=2",
};
/** What ffmpeg is told to write each {@link ScaledFormat} with. */
const ENCODERS: Record<ScaledFormat, string[]> = {
webp: ["-c:v", "libwebp", "-q:v", "80", "-loop", "0", "-f", "webp"],
jpg: ["-c:v", "mjpeg", "-q:v", "4", "-f", "image2"],
png: ["-c:v", "png", "-f", "image2"],
};
/**
* What {@link scaleImage} is told of the image it scales, which
* `scalableImage` (lib/upload.ts) reads from its first bytes.
*/
export interface ImageTraits {
/** The EXIF orientation to turn the image by, 1 when it is upright. */
orientation: number;
/**
* Whether its frames are an animation, as a GIF's are. A still may hold
* more than one image too, a phone's JPEG carrying a gain map or a depth
* map after the photo, and ffmpeg reads each as a frame.
*/
animated: boolean;
}
/**
* Writes a copy of an image at most `width` wide, never wider than the image
* itself, as a WebP unless `format` names another. WebP keeps a PNG's
* transparency and a GIF's animation, at a fraction of either's bytes.
*/
export function scaleImage(
srcPath: string,
outPath: string,
width: ImageWidth,
{ orientation, animated }: ImageTraits,
format: ScaledFormat = "webp",
): Promise<void> {
const upright = ORIENTATION_FILTERS[orientation];
// A JPEG's colour planes halve its height, which they need even.
const height = format === "jpg" ? -2 : -1;
// Only an animation keeps its frames, and only as a WebP: of anything else
// the first is the image, and a copy of them all would flash through what
// follows it.
const frames = animated && format === "webp" ? [] : ["-frames:v", "1"];
return runFfmpeg([
"-noautorotate",
"-format_whitelist",
IMAGE_FORMATS,
"-i",
srcPath,
"-vf",
`${upright ? `${upright},` : ""}scale='min(${width},iw)':${height}`,
"-an",
...frames,
...ENCODERS[format],
], outPath);
}
/**
* Returns `bytes`, an image held in memory, scaled as {@link scaleImage}
* does, or null when it can't be, because ffmpeg is missing or failed. The
* image proxy scales what it fetched with it, and keeps the result itself.
*/
export async function scaleImageBytes(
bytes: Uint8Array,
width: ImageWidth,
traits: ImageTraits,
): Promise<Uint8Array | null> {
if (!await ffmpegAvailable()) return null;
const srcPath = `${SCALED_DIR}/tmp/${crypto.randomUUID()}`;
const outPath = `${srcPath}.webp`;
try {
await Deno.mkdir(dirname(srcPath), { recursive: true });
await Deno.writeFile(srcPath, bytes);
await scaleImage(srcPath, outPath, width, traits);
return await Deno.readFile(outPath);
} catch {
return null;
} finally {
await Promise.all(
[srcPath, outPath].map((path) => Deno.remove(path).catch(() => {})),
);
}
}
/**
* The sample rate the audio is decoded at for its waveform. A bar stands for
* a second or more of a song, so 8 kHz keeps every peak worth drawing at a
* sixth of the samples.
*/
const PEAKS_SAMPLE_RATE = 8000;
/** How many samples each running peak covers before `peaksOf` buckets them. */
const PEAKS_WINDOW = 256;
/**
* Decodes the audio of `srcPath` to its waveform's peaks. ffmpeg streams the
* samples, mono at {@link PEAKS_SAMPLE_RATE}, and only the peak of each
* {@link PEAKS_WINDOW} is kept, so an hour of audio holds a few hundred
* kilobytes at most.
*
* @throws {Error} When ffmpeg fails or the file holds no audio.
*/
async function audioPeaks(srcPath: string): Promise<number[]> {
const child = new Deno.Command("ffmpeg", {
args: [
"-protocol_whitelist",
"file",
"-format_whitelist",
AUDIO_FORMATS,
"-i",
srcPath,
"-vn",
"-ac",
"1",
"-ar",
String(PEAKS_SAMPLE_RATE),
"-f",
"s16le",
"-",
],
stdin: "null",
stdout: "piped",
stderr: "null",
}).spawn();
const windows: number[] = [];
let peak = 0;
let count = 0;
// A chunk may end halfway through a sample, whose first byte waits here.
let pending: number | null = null;
const take = (sample: number) => {
peak = Math.max(peak, Math.abs(sample) / 32768);
if (++count === PEAKS_WINDOW) {
windows.push(peak);
peak = 0;
count = 0;
}
};
for await (const chunk of child.stdout) {
let i = 0;
if (pending !== null) {
take(((chunk[0] << 24) >> 16) | pending);
pending = null;
i = 1;
}
for (; i + 1 < chunk.length; i += 2) {
// Little-endian, signed: the high byte is shifted into the sign bit
// and back.
take(((chunk[i + 1] << 24) >> 16) | chunk[i]);
}
if (i < chunk.length) pending = chunk[i];
}
if (count > 0) windows.push(peak);
const { success } = await child.status;
if (!success || windows.length === 0) throw new Error("ffmpeg failed");
return peaksOf(windows);
}
/**
* Writes the waveform of the audio file `srcPath` as the waveform route's
* whole response, so the route serves it as a file, with its ETag.
*/
export function audioWaveform(srcPath: string, outPath: string): Promise<void> {
return writeAtomically(outPath, async (tmpPath) => {
const body: APIResponse<DumpWaveform> = {
success: true,
data: { peaks: await audioPeaks(srcPath) },
};
await Deno.writeTextFile(tmpPath, JSON.stringify(body));
});
}
/**
* Lets a feed full of cards asking for the same new still at once wait on a
* single ffmpeg run. It holds the generations under way, by output path.
*/
const inFlight = new Map<string, Promise<void>>();
/**
* Spares an ffmpeg run per request for an image it can't read: the
* generations that failed, by output path, with the date of the source they
* failed on. A new upload gets a new try.
*/
const failed = new Map<string, number>();
/** Returns the date `path` was last written, or null when it doesn't exist. */
async function writtenAt(path: string): Promise<number | null> {
try {
return (await Deno.stat(path)).mtime?.getTime() ?? 0;
} catch {
return null;
}
}
/**
* Makes `outPath` with `generate` unless it exists. Resolves false when it
* can't be made, because ffmpeg is missing or failed.
*
* With `srcPath`, the file it is made from, `outPath` takes that file's date,
* and is made again when the two differ: an upload replaced in place, as an
* avatar or a cover is, gets new copies without anyone deleting the old.
* Without it, the caller deletes `outPath` when its source changes.
*/
export async function ensureGenerated(
outPath: string,
generate: () => Promise<void>,
srcPath?: string,
): Promise<boolean> {
const made = await writtenAt(outPath);
let version: number | null = null;
if (srcPath) {
version = await writtenAt(srcPath);
if (version === null) return false;
if (made === version) return true;
if (failed.get(outPath) === version) return false;
} else if (made !== null) {
return true;
}
if (!await ffmpegAvailable()) return false;
let pending = inFlight.get(outPath);
if (!pending) {
pending = generate()
.then(() =>
version === null
? undefined
: Deno.utime(outPath, new Date(), new Date(version))
)
.catch((err) => {
if (version !== null) failed.set(outPath, version);
throw err;
})
.finally(() => inFlight.delete(outPath));
inFlight.set(outPath, pending);
}
return await pending.then(() => true, () => false);
}
/**
* How old a temporary file must be for the sweep to take it: one being
* written, by a generation under way, is seconds old, and one a crash left
* is never finished.
*/
const ABANDONED_AFTER_MS = 60 * 60_000;
/** Yields the path of every file under `dir`, and nothing when it is missing. */
async function* filesUnder(dir: string): AsyncGenerator<string> {
let entries: Deno.DirEntry[];
try {
entries = await Array.fromAsync(Deno.readDir(dir));
} catch {
return;
}
for (const entry of entries) {
const path = `${dir}/${entry.name}`;
if (entry.isDirectory) yield* filesUnder(path);
else if (entry.isFile) yield path;
}
}
/**
* Deletes the generated files nothing serves any more, and returns how many:
* the scaled copies of an upload that is gone, which the services' own
* removals missed or never made, as for a user's avatar; the previews of
* image dumps under their former name, `<id>.preview.jpg` or `.png`, which
* the scaled copies replaced; and the temporary files a generation left
* unfinished. Nothing else of `THUMBNAILS_DIR` is touched, the custom
* thumbnails kept there being uploads.
*/
export async function sweepGeneratedMedia(): Promise<number> {
let removed = 0;
const remove = (path: string) =>
Deno.remove(path).then(() => {
removed++;
}, () => {});
const abandoned = async (path: string) => {
const at = await writtenAt(path);
return at !== null && Date.now() - at > ABANDONED_AFTER_MS;
};
for await (const path of filesUnder(THUMBNAILS_DIR)) {
if (
/\.preview\.(jpg|png)$/.test(path) ||
(path.endsWith(".tmp") && await abandoned(path))
) await remove(path);
}
for await (const path of filesUnder(SCALED_DIR)) {
// A copy is named after its upload's path: see scaledImagePath.
const copy = relative(SCALED_DIR, path).match(
/^(.+)\.w\d+\.(?:webp|jpg|png)$/,
);
const unused = copy
? await writtenAt(`${UPLOADS_DIR}/${copy[1]}`) === null
: await abandoned(path);
if (unused) await remove(path);
}
return removed;
}
/**
* Runs {@link sweepGeneratedMedia} a minute after the server starts, which
* clears what a new version no longer uses as soon as it is deployed, then
* once a day. `main.ts` starts it.
*/
export const generatedMediaSweepJob: Job = {
name: "media-sweep",
everyMs: 24 * 60 * 60_000,
firstAfterMs: 60_000,
async run() {
const removed = await sweepGeneratedMedia();
if (removed > 0) {
console.log(`[media-sweep] removed ${removed} generated files unused`);
}
},
};

33
api/lib/index-html.ts
View File

@@ -1,25 +1,36 @@
/** /**
* The built frontend's `index.html`, with its runtime placeholders filled: the * Fills the placeholders of the frontend's `index.html` with settings read at
* site name, emoji, icon version and Bandcamp player mode. * runtime, so they change with a restart and no rebuild. The static and OG
* middlewares serve what it returns.
* *
* @module * @module
*/ */
import { BANDCAMP_PLAYER, OG_SITE_NAME, SITE_EMOJI } from "../config.ts"; import {
BANDCAMP_PLAYER,
DEFAULT_STYLE,
EMAIL_ENABLED,
FEDERATION,
OG_SITE_NAME,
SITE_EMOJI,
} from "../config.ts";
import { emojiToCodepoint } from "./site-icons.ts"; import { emojiToCodepoint } from "./site-icons.ts";
/** The cache-busting version of the icon URLs. */ /**
* Versions the icon URLs, so browsers fetch new icons when the emoji changes.
*/
const ICON_VERSION = emojiToCodepoint(SITE_EMOJI); const ICON_VERSION = emojiToCodepoint(SITE_EMOJI);
/** /**
* `index.html` with its placeholders filled, by root. Kept until the file * Caches each root's filled file until its mtime changes, as a rebuild under a
* changes (a rebuild under a running server): a stat per navigation instead of * running server does, so a navigation costs a stat instead of a read.
* a read and four replaceAll.
*/ */
const indexHtml = new Map<string, { mtime: number; html: string }>(); const indexHtml = new Map<string, { mtime: number; html: string }>();
/** /**
* Returns `${root}/index.html` with its placeholders filled. * Returns `${root}/index.html` with the site name, emoji, icon version,
* Bandcamp player mode, default style, and whether email and each way of
* federated sign-in are on filled in.
* *
* @throws {Error} When the file can't be read. * @throws {Error} When the file can't be read.
*/ */
@@ -32,7 +43,11 @@ export async function loadIndexHtml(root: string): Promise<string> {
.replaceAll("__SITE_NAME__", OG_SITE_NAME) .replaceAll("__SITE_NAME__", OG_SITE_NAME)
.replaceAll("__SITE_EMOJI__", SITE_EMOJI) .replaceAll("__SITE_EMOJI__", SITE_EMOJI)
.replaceAll("__ICON_VERSION__", ICON_VERSION) .replaceAll("__ICON_VERSION__", ICON_VERSION)
.replaceAll("__BANDCAMP_PLAYER__", BANDCAMP_PLAYER); .replaceAll("__BANDCAMP_PLAYER__", BANDCAMP_PLAYER)
.replaceAll("__DEFAULT_STYLE__", DEFAULT_STYLE)
.replaceAll("__EMAIL_ENABLED__", EMAIL_ENABLED ? "on" : "off")
.replaceAll("__FEDERATION_INCOMING__", FEDERATION.incoming ? "on" : "off")
.replaceAll("__FEDERATION_OUTGOING__", FEDERATION.outgoing ? "on" : "off");
indexHtml.set(root, { mtime, html }); indexHtml.set(root, { mtime, html });
return html; return html;
} }

248
api/lib/jwt.ts
View File

@@ -1,15 +1,30 @@
/** /**
* Signed tokens and password hashes: session tokens, media tokens, password * Signs and verifies the server's four kinds of token (session, media,
* reset tokens, and scrypt hashing. * password reset and unsubscribe), all HS256 with the one key. Only a token's signature and
* expiry are checked here: `lib/auth.ts` checks a session or media token
* against its user.
*
* It also signs and verifies the tokens instances exchange for federated
* sign-in (identity tokens and events), EdDSA with the instance's Ed25519 key,
* whose public half other instances fetch: `instance-key-service.ts` keeps
* the key, and the federation services pass it in.
* *
* Importing this module throws when `GERBEUR_JWT_SECRET` is unset. * Importing this module throws when `GERBEUR_JWT_SECRET` is unset.
* *
* @module * @module
*/ */
import { randomBytes, scrypt, timingSafeEqual } from "node:crypto"; import {
import { Buffer } from "node:buffer"; createLocalJWKSet,
import { jwtVerify, SignJWT } from "@panva/jose"; exportJWK,
generateKeyPair,
importJWK,
type JSONWebKeySet,
type JWK,
type JWTPayload,
jwtVerify,
SignJWT,
} from "@panva/jose";
import { import {
type AuthPayload, type AuthPayload,
@@ -18,6 +33,8 @@ import {
type PasswordResetPayload, type PasswordResetPayload,
} from "../model/auth.ts"; } from "../model/auth.ts";
import { JWT_SECRET } from "../config.ts"; import { JWT_SECRET } from "../config.ts";
import { type DigestCadence, isDigestCadence } from "../../shared/settings.ts";
import type { AuthTokens, Role } from "../../shared/user.ts";
if (!JWT_SECRET) { if (!JWT_SECRET) {
throw new Error( throw new Error(
@@ -26,22 +43,26 @@ if (!JWT_SECRET) {
} }
const JWT_KEY = new TextEncoder().encode(JWT_SECRET); const JWT_KEY = new TextEncoder().encode(JWT_SECRET);
// ── Password reset tokens ───────────────────────────────────────────────────── /**
* Signs a password reset token for `userId`, valid for an hour. The token
/** Signs a single-use password reset token for `userId`, valid for an hour. */ * isn't single-use by itself: `requestPasswordReset` stores it, and
* `resetPassword` marks it used.
*/
export async function createPasswordResetToken( export async function createPasswordResetToken(
userId: string, userId: string,
): Promise<string> { ): Promise<string> {
return await new SignJWT({ purpose: "password-reset", userId }) return await new SignJWT({ purpose: "password-reset", userId })
.setProtectedHeader({ alg: "HS256" }) .setProtectedHeader({ alg: "HS256" })
// Two requests in the same second would otherwise sign the same token,
// which is the primary key of `password_reset_tokens`.
.setJti(crypto.randomUUID()) .setJti(crypto.randomUUID())
.setExpirationTime("1h") .setExpirationTime("1h")
.sign(JWT_KEY); .sign(JWT_KEY);
} }
/** /**
* Verifies a password reset token and returns its payload, or null when it is * Returns the payload of a password reset token, or null when it is invalid,
* invalid, expired or not a reset token. * expired or another kind of token. Whether it was used is the caller's check.
*/ */
export async function verifyPasswordResetToken( export async function verifyPasswordResetToken(
token: string, token: string,
@@ -55,29 +76,47 @@ export async function verifyPasswordResetToken(
} }
} }
// ── Auth tokens ─────────────────────────────────────────────────────────────── /**
* Signs a session token carrying `payload`, valid for 7 days.
/** Signs a session token carrying `payload`, valid for 7 days. */ * {@link issueTokens} issues it on sign-up, sign-in and password change, and
* the client sends it as a bearer token.
*/
export async function createJWT( export async function createJWT(
payload: Omit<AuthPayload, "exp">, payload: Omit<AuthPayload, "exp">,
): Promise<string> { ): Promise<string> {
return await new SignJWT(payload) return await new SignJWT(payload)
.setProtectedHeader({ alg: "HS256" }) .setProtectedHeader({ alg: "HS256" })
// `iat` is what a password change revokes against (users.tokens_valid_after). // A password change revokes tokens by comparing `iat` with
// `users.tokens_valid_after`.
.setIssuedAt() .setIssuedAt()
.setExpirationTime("7d") .setExpirationTime("7d")
.sign(JWT_KEY); .sign(JWT_KEY);
} }
// ── Media tokens ────────────────────────────────────────────────────────────── /**
* Returns a new session token and media token for `user`, as sign-up
* (`POST /api/users`), sign-in (`POST /api/sessions`) and a password change
* (`PUT /api/me/password`) answer them.
*/
export async function issueTokens(
user: { id: string; username: string; role: Role },
): Promise<AuthTokens> {
return {
token: await createJWT({
userId: user.id,
username: user.username,
role: user.role,
}),
mediaToken: await createMediaToken(user.id),
};
}
/** /**
* Signs a media token for `userId`, valid for 7 days. * Signs a media token for `userId`, valid for 7 days. Media elements can't send
* * an `Authorization` header, so the URLs of private files, thumbnails and
* Media elements can't send an Authorization header, so a private dump's file * playlist covers carry this token as `?token=`, where it lands in logs and
* and thumbnail URLs carry a token in the query string, where it lands in * browser history. It only reads media, so a leaked URL doesn't expose the
* access logs and browser history. That token is this one: it only reads media, * account.
* so a leaked URL exposes the owner's private files, not their account.
*/ */
export async function createMediaToken(userId: string): Promise<string> { export async function createMediaToken(userId: string): Promise<string> {
return await new SignJWT({ purpose: "media", userId }) return await new SignJWT({ purpose: "media", userId })
@@ -88,8 +127,8 @@ export async function createMediaToken(userId: string): Promise<string> {
} }
/** /**
* Verifies a media token and returns its user id and issue time, or null when * Returns the user id and issue time of a media token, or null when it is
* it is invalid, expired or not a media token. * invalid, expired or another kind of token.
*/ */
export async function verifyMediaToken( export async function verifyMediaToken(
token: string, token: string,
@@ -106,8 +145,9 @@ export async function verifyMediaToken(
} }
/** /**
* Verifies a session token's signature and expiry and returns its payload, or * Returns the payload of a session token whose signature and expiry are valid,
* null. Checking it against the live user is `authenticate`'s job (lib/auth.ts). * or null. Call `authenticate` (`lib/auth.ts`) instead, which also checks the
* token against its user.
*/ */
export async function verifyJWT(token: string): Promise<AuthPayload | null> { export async function verifyJWT(token: string): Promise<AuthPayload | null> {
try { try {
@@ -124,38 +164,134 @@ export async function verifyJWT(token: string): Promise<AuthPayload | null> {
} }
} }
/** Hashes `password` with scrypt and a random salt, as `hash.salt` in hex. */ /**
export function hashPassword(password: string): Promise<string> { * Signs the token of a digest's unsubscribe link, which turns off `cadence`
const salt = randomBytes(16).toString("hex"); * for `userId` without signing in. It never expires, so the link in an old
* email still works; turning the digest back on doesn't revoke it, which is
return new Promise((resolve, reject) => { * harmless since all it can do is turn it off again.
scrypt(password, salt, 64, (err, derivedKey) => { */
if (err) reject(err); export async function createUnsubscribeToken(
else resolve(`${derivedKey.toString("hex")}.${salt}`); userId: string,
}); cadence: DigestCadence,
}); ): Promise<string> {
return await new SignJWT({ purpose: "unsubscribe", userId, cadence })
.setProtectedHeader({ alg: "HS256" })
.sign(JWT_KEY);
} }
/** Checks `password` against a hash made by {@link hashPassword}. */ /**
export function verifyPassword( * Returns the user and cadence of an unsubscribe token, or null when it is
password: string, * invalid or another kind of token.
storedHash: string, */
): Promise<boolean> { export async function verifyUnsubscribeToken(
const [hash, salt] = storedHash.split("."); token: string,
// A malformed stored hash can't match anything; scrypt would throw on the ): Promise<{ userId: string; cadence: DigestCadence } | null> {
// missing salt instead of answering. try {
if (!hash || !salt) return Promise.resolve(false); const { payload } = await jwtVerify(token, JWT_KEY);
if (
return new Promise((resolve, reject) => { payload.purpose !== "unsubscribe" || typeof payload.userId !== "string" ||
scrypt(password, salt, 64, (err, derivedKey) => { !isDigestCadence(payload.cadence)
if (err) reject(err); ) {
else { return null;
const expected = Buffer.from(hash ?? "", "hex"); }
resolve( return { userId: payload.userId, cadence: payload.cadence };
expected.length === derivedKey.length && } catch {
timingSafeEqual(expected, derivedKey), return null;
);
} }
}); }
});
/**
* The algorithm of the tokens instances exchange: Ed25519 signatures, short
* and fast, with no parameters to get wrong.
*/
const INSTANCE_ALG = "EdDSA";
/**
* An instance's key, ready to sign, from {@link importInstanceKey}. Its
* `kid` names the public half in the instance's JWKS.
*/
export interface InstanceSigningKey {
kid: string;
privateKey: CryptoKey;
}
/**
* Returns a new Ed25519 key pair as JWKs, with a random `kid`, for
* `instance-key-service.ts` to store when the instance has none. The public
* JWK carries the `kid`, `alg` and `use` a JWKS entry needs.
*/
export async function generateInstanceKey(): Promise<
{ kid: string; privateJwk: JWK; publicJwk: JWK }
> {
const { privateKey, publicKey } = await generateKeyPair(INSTANCE_ALG, {
crv: "Ed25519",
extractable: true,
});
const kid = crypto.randomUUID();
return {
kid,
privateJwk: await exportJWK(privateKey),
publicJwk: {
...await exportJWK(publicKey),
kid,
alg: INSTANCE_ALG,
use: "sig",
},
};
}
/** Returns the stored private JWK `privateJwk` as a key to sign with. */
export async function importInstanceKey(
kid: string,
privateJwk: JWK,
): Promise<InstanceSigningKey> {
const privateKey = await importJWK(privateJwk, INSTANCE_ALG);
if (!(privateKey instanceof CryptoKey)) {
throw new Error("The instance key is not an asymmetric key");
}
return { kid, privateKey };
}
/**
* Signs `payload` with the instance's key, from `issuer` (this instance) to
* `audience` (the instance it is for), valid for `expiresIn`, with a random
* `jti`. The identity tokens of `POST /api/oauth/tokens` and the events of
* the outbox are signed with it.
*/
export async function signInstanceToken(
payload: JWTPayload,
key: InstanceSigningKey,
claims: { issuer: string; audience: string; subject: string },
expiresIn: string,
): Promise<string> {
return await new SignJWT(payload)
.setProtectedHeader({ alg: INSTANCE_ALG, kid: key.kid })
.setIssuer(claims.issuer)
.setAudience(claims.audience)
.setSubject(claims.subject)
.setIssuedAt()
.setJti(crypto.randomUUID())
.setExpirationTime(expiresIn)
.sign(key.privateKey);
}
/**
* Returns the payload of `token` once its signature checks against `jwks`,
* the keys of `issuer`, and it is addressed to `audience` and unexpired.
*
* @throws {Error} jose's error when it doesn't verify; its `code` is
* `ERR_JWKS_NO_MATCHING_KEY` when no key of `jwks` has its `kid`, which
* tells the caller to refetch the keys, the issuer having rotated them.
*/
export async function verifyInstanceToken(
token: string,
jwks: JSONWebKeySet,
claims: { issuer: string; audience: string },
): Promise<JWTPayload> {
const { payload } = await jwtVerify(token, createLocalJWKSet(jwks), {
issuer: claims.issuer,
audience: claims.audience,
algorithms: [INSTANCE_ALG],
});
return payload;
} }

24
api/lib/like.ts Normal file
View File

@@ -0,0 +1,24 @@
/**
* Builds the patterns of the search queries' `LIKE`, for the dump, playlist
* and user searches. A query's `%` and `_` are matched as themselves rather
* than as wildcards: usernames hold underscores, and "a_b" must not find
* "axb".
*
* @module
*/
/**
* Returns the pattern matching text that contains `query`, or that starts
* with it when `where` is `"prefix"`. The SQL must read `LIKE ? ESCAPE '\'`,
* which the {@link LIKE} fragment spells.
*/
export function likePattern(
query: string,
where: "anywhere" | "prefix" = "anywhere",
): string {
const escaped = query.replace(/[\\%_]/g, "\\$&");
return where === "prefix" ? `${escaped}%` : `%${escaped}%`;
}
/** Spells `LIKE ? ESCAPE '\'`, for a {@link likePattern} parameter. */
export const LIKE = "LIKE ? ESCAPE '\\'";

24
api/lib/pagination.ts
View File

@@ -1,6 +1,7 @@
/** /**
* Page-based pagination: reading `page` and `limit` from a query string, and * Pages every list route the same way. It reads the page a list route's query
* cutting fetched rows into a page. * string asks for, and cuts the rows a service fetched into that page, which
* the routes send as `PaginatedData`.
* *
* @module * @module
*/ */
@@ -8,9 +9,9 @@
import { PAGINATION_DEFAULT_LIMIT, PAGINATION_MAX_LIMIT } from "../config.ts"; import { PAGINATION_DEFAULT_LIMIT, PAGINATION_MAX_LIMIT } from "../config.ts";
/** /**
* Reads the `page` and `limit` query parameters. `page` is at least 1 and * Reads the `page` and `limit` query parameters of a list route. It never
* defaults to 1; `limit` is clamped to [1, {@link PAGINATION_MAX_LIMIT}] and * fails: a missing or invalid `page` reads as 1, and `limit` is kept between 1
* defaults to `defaultLimit`. * and {@link PAGINATION_MAX_LIMIT}.
*/ */
export function parsePagination( export function parsePagination(
params: URLSearchParams, params: URLSearchParams,
@@ -30,18 +31,19 @@ export function parsePagination(
return { page, limit }; return { page, limit };
} }
/** One page of a list, and whether another follows. */ /**
* Returned to the routes by the services' list functions, and sent as
* `PaginatedData`.
*/
export interface Page<T> { export interface Page<T> {
/** The page's items, in order. */
items: T[]; items: T[];
/** Whether another page follows this one. */
hasMore: boolean; hasMore: boolean;
} }
/** /**
* Turns rows fetched with `LIMIT limit + 1` into a page: the extra row, if it * Turns rows fetched with `LIMIT limit + 1` into a page. The extra row, when it
* came back, is dropped and says there's a next page. No COUNT(*) over the * comes back, is dropped and tells that another page follows, which spares a
* whole list just to know that. * COUNT(*) over the whole list.
*/ */
export function pageOf<T>(rows: T[], limit: number): Page<T> { export function pageOf<T>(rows: T[], limit: number): Page<T> {
return rows.length > limit return rows.length > limit

48
api/lib/password.ts Normal file
View File

@@ -0,0 +1,48 @@
/**
* Hashes and checks passwords, for sign-up, sign-in, password changes and the
* admin that `db/startup.ts` seeds. It reads no config, so the startup tasks
* can use it before the server's settings are checked.
*
* @module
*/
import { randomBytes, scrypt, timingSafeEqual } from "node:crypto";
import { Buffer } from "node:buffer";
/**
* Hashes `password` with scrypt and a random salt, as `hash.salt` in hex, for
* `users.password_hash`.
*/
export function hashPassword(password: string): Promise<string> {
const salt = randomBytes(16).toString("hex");
return new Promise((resolve, reject) => {
scrypt(password, salt, 64, (err, derivedKey) => {
if (err) reject(err);
else resolve(`${derivedKey.toString("hex")}.${salt}`);
});
});
}
/** Checks `password` against a hash made by {@link hashPassword}. */
export function verifyPassword(
password: string,
storedHash: string,
): Promise<boolean> {
const [hash, salt] = storedHash.split(".");
// scrypt would throw on a missing salt, and a malformed hash matches nothing.
if (!hash || !salt) return Promise.resolve(false);
return new Promise((resolve, reject) => {
scrypt(password, salt, 64, (err, derivedKey) => {
if (err) reject(err);
else {
const expected = Buffer.from(hash ?? "", "hex");
resolve(
expected.length === derivedKey.length &&
timingSafeEqual(expected, derivedKey),
);
}
});
});
}

9
api/lib/permissions.ts
View File

@@ -1,5 +1,7 @@
/** /**
* Permission checks against an authenticated caller. * Checks an authenticated caller's permissions. Routes call it where the
* answer depends on more than the permission, such as "the owner, or a
* moderator"; `requirePermission` gates a whole route instead.
* *
* @module * @module
*/ */
@@ -7,7 +9,10 @@
import { hasPermission, type Permission } from "../../shared/permissions.ts"; import { hasPermission, type Permission } from "../../shared/permissions.ts";
import type { AuthPayload } from "../model/auth.ts"; import type { AuthPayload } from "../model/auth.ts";
/** Checks whether the caller's role grants `permission`. */ /**
* Checks whether the caller's role grants `permission`. The role is the live
* one `authenticate` put in the payload, not the token's.
*/
export function can(payload: AuthPayload, permission: Permission): boolean { export function can(payload: AuthPayload, permission: Permission): boolean {
return hasPermission(payload.role, permission); return hasPermission(payload.role, permission);
} }

73
api/lib/playlist-members.ts
View File

@@ -1,26 +1,31 @@
/** /**
* Which of a playlist's dumps a given viewer may see. * Decides which of a playlist's dumps a viewer may see. The playlist service
* filters its listings with it, the WebSocket service filters each client's
* `playlist_dumps_updated` message the same way, and the queries that list
* playlists count their dumps by the same rule.
* *
* @module * @module
*/ */
import type { Playlist } from "../../shared/playlist.ts"; import type { Playlist } from "../../shared/playlist.ts";
/** A playlist's dump, with what deciding who may see it takes. */ /**
* The fields of a playlist's dump that {@link isMemberVisibleTo} needs. A
* `Dump` has them. For its broadcasts, the playlist service reads only these.
*/
export interface PlaylistMember { export interface PlaylistMember {
/** The dump's id. */
id: string; id: string;
/** Whether the dump is private. */
isPrivate: boolean; isPrivate: boolean;
/** The id of the dump's poster. */ /** The dump's poster, not the playlist's owner. */
userId: string; userId: string;
} }
/** /**
* Checks whether `viewerId` sees this member when the playlist is shown to them: a * Checks whether `viewerId` may see `member` in `playlist`. A public dump is
* public dump, or a private one of the owner's own, to the owner. Everyone * visible to all. A private dump is visible only to the playlist's owner, and
* else — moderators too — doesn't see private dumps, and neither does the * only if they posted it: moderators don't see it, and neither does the owner
* owner see someone else's dump once it has gone private. * once someone else's dump in their playlist goes private. A null or undefined
* `viewerId` is a signed-out viewer.
*/ */
export function isMemberVisibleTo( export function isMemberVisibleTo(
member: PlaylistMember, member: PlaylistMember,
@@ -30,3 +35,53 @@ export function isMemberVisibleTo(
return !member.isPrivate || return !member.isPrivate ||
(member.userId === viewerId && viewerId === playlist.userId); (member.userId === viewerId && viewerId === playlist.userId);
} }
/**
* The fields of a playlist's dump that {@link dumpFiguresFor} needs: those
* that tell who may see it, and when it was added. The playlist service reads
* them for its broadcasts.
*/
export interface DatedPlaylistMember extends PlaylistMember {
addedAt: Date;
}
/** A figure of the dumps of `playlists p` that the viewer `?1` may see. */
const visibleDumps = (figure: string) =>
`(SELECT ${figure} FROM playlist_dumps pd
INNER JOIN dumps d ON d.id = pd.dump_id
WHERE pd.playlist_id = p.id
AND (d.is_private = 0 OR (d.user_id = p.user_id AND p.user_id = ?1))
)`;
/**
* The `dump_count` and `last_dump_at` columns of a query on `playlists p`:
* how many of the playlist's dumps the viewer may see, by
* {@link isMemberVisibleTo}'s rule, and when the latest of them was added,
* null without any. Both read the viewer's id, or `""` for a signed-out one,
* as `?1`: it is the query's first parameter, and each `?` after it takes the
* next.
*/
export const VISIBLE_DUMP_FIGURES = `${visibleDumps("COUNT(*)")} as dump_count,
${visibleDumps("MAX(pd.added_at)")} as last_dump_at`;
/**
* Returns what `playlist` says of its dumps to the viewer, `members` being
* all of them: the `dumpCount` and `lastDumpAt` that
* {@link VISIBLE_DUMP_FIGURES} reads in a query. The WebSocket service puts
* them on the playlist each socket is sent.
*/
export function dumpFiguresFor(
members: DatedPlaylistMember[],
playlist: Pick<Playlist, "userId">,
viewerId: string | null | undefined,
): Required<Pick<Playlist, "dumpCount">> & Pick<Playlist, "lastDumpAt"> {
const visible = members.filter((m) =>
isMemberVisibleTo(m, playlist, viewerId)
);
return {
dumpCount: visible.length,
lastDumpAt: visible.length
? new Date(Math.max(...visible.map((m) => m.addedAt.getTime())))
: undefined,
};
}

36
api/lib/rate-limit.ts
View File

@@ -1,26 +1,35 @@
/** /**
* In-memory rate limiting with token buckets, keyed by the caller's choice * Limits how often something may happen, with token buckets kept in memory,
* (e.g. `${kind}:${userId}`) — per user rather than per socket or request, so * which a restart empties. Logins, password reset requests and the chat's
* more tabs or parallel requests don't buy more throughput. * typing messages use it.
* *
* @module * @module
*/ */
import type { Job } from "./scheduler.ts";
interface Bucket { interface Bucket {
tokens: number; tokens: number;
at: number; at: number;
} }
const buckets = new Map<string, Bucket>(); const buckets = new Map<string, Bucket>();
/** A token bucket's shape. */ /**
* Passed to {@link takeToken} by each limited action, which declares its own
* as a constant.
*/
export interface RateLimit { export interface RateLimit {
/** Burst size. */ /** How many tokens a full bucket holds, and so the largest burst. */
capacity: number; capacity: number;
/** Tokens regained per second. */ /** Tokens regained per second. */
refill: number; refill: number;
} }
/** Takes one token from `key`'s bucket, or returns false when it is empty. */ /**
* Takes one token from `key`'s bucket, or returns false when it is empty. The
* caller picks the key, such as `typing:<userId>`, and keys by user or account
* so that more tabs or parallel requests don't buy more tokens.
*/
export function takeToken(key: string, limit: RateLimit): boolean { export function takeToken(key: string, limit: RateLimit): boolean {
const now = Date.now(); const now = Date.now();
const bucket = buckets.get(key) ?? { tokens: limit.capacity, at: now }; const bucket = buckets.get(key) ?? { tokens: limit.capacity, at: now };
@@ -35,11 +44,18 @@ export function takeToken(key: string, limit: RateLimit): boolean {
return true; return true;
} }
// Idle buckets have refilled by now for every limit in use; drop them so the /**
// map tracks only keys that have been active lately. * Drops the buckets untouched for an hour, every minute. Every limit in use
setInterval(() => { * refills within the hour, so such a bucket is full and dropping it changes
* nothing. `main.ts` starts it.
*/
export const rateLimitSweepJob: Job = {
name: "rate-limit",
everyMs: 60_000,
run() {
const cutoff = Date.now() - 60 * 60_000; const cutoff = Date.now() - 60 * 60_000;
for (const [key, bucket] of buckets) { for (const [key, bucket] of buckets) {
if (bucket.at < cutoff) buckets.delete(key); if (bucket.at < cutoff) buckets.delete(key);
} }
}, 60_000); },
};

76
api/lib/reserved-slugs.ts
View File

@@ -1,46 +1,78 @@
/** /**
* Which slugs a category may take. Category slugs live at the URL root (e.g. * Checks which slugs a dump category may take. A dump category's feed lives at
* /music), so they must never collide with the frontend's own routes. * the URL root (`/music`), so its slug must not shadow another top-level page.
* The category service runs this check when a dump category is created or
* renamed, and the migrations that reserve more words rename the categories
* that had one with {@link renameReservedCategories}. A playlist category's
* page is `/playlists/<slug>`, under a reserved word, so it reserves nothing.
*
* Dump and playlist slugs reserve nothing: they live under their owner's
* `dumps/` and `playlists/`, where nothing else is named by a user. A user
* can't clash with a category either, since `@` is never in a slug.
* *
* @module * @module
*/ */
import type { DatabaseSync } from "node:sqlite";
import { FEED_TABS } from "../../shared/paths.ts";
/** /**
* Top-level path segments that already have meaning in the SPA router (see * Lists the top-level path segments a category slug may not take: the feed
* src/App.tsx), plus a few reserved for infrastructure. * tabs (`/hot`), the SPA's own pages (`src/App.tsx`), and `api`, `ws`,
* `assets` and `emojibase`, which the server answers before the SPA. Keep it
* in step with the route table and the build's top-level folders.
*/ */
export const RESERVED_SLUGS: readonly string[] = [ export const RESERVED_SLUGS: readonly string[] = [
"~", ...FEED_TABS,
"dumps",
"users",
"playlists",
"search", "search",
"login", "login",
"register", "register",
"notifications", "notifications",
"reset-password", "reset-password",
"unsubscribe",
"settings",
"manage", "manage",
"community",
"playlists",
"authorize",
"federation",
"api", "api",
"ws",
"assets", "assets",
"emojibase",
]; ];
/**
* A well-formed category slug: lowercase alphanumerics,
* hyphen-separated, no leading, trailing or double hyphen.
*/
export const CATEGORY_SLUG_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
/** Checks whether `slug` is well-formed (see {@link CATEGORY_SLUG_RE}). */
export function isValidSlugFormat(slug: string): boolean {
return CATEGORY_SLUG_RE.test(slug);
}
/** Checks whether `slug` is one of the {@link RESERVED_SLUGS}. */ /** Checks whether `slug` is one of the {@link RESERVED_SLUGS}. */
export function isReservedSlug(slug: string): boolean { export function isReservedSlug(slug: string): boolean {
return RESERVED_SLUGS.includes(slug); return RESERVED_SLUGS.includes(slug);
} }
/** Checks whether `slug` is well-formed and not reserved. */ /**
export function isSlugAvailable(slug: string): boolean { * Renames each category whose slug is now reserved, such as `hot`, to the
return isValidSlugFormat(slug) && !isReservedSlug(slug); * first of `<slug>-2`, `<slug>-3`… that is free, and logs it. Migration 0021
* runs it for v4's words, 0022 for `community`, 0024 for `authorize` and
* `federation`, and 0025 for `playlists`.
*
* It reads every category: until 0025 gave categories a `kind`, each was a
* dump category, and 0025 makes no other. A later migration that reserves
* more words must leave the playlist categories alone, and limit it to
* `kind = 'dump'`, which the earlier ones can't, having no such column.
*/
export function renameReservedCategories(db: DatabaseSync): void {
const rows = db.prepare(`SELECT id, slug FROM categories;`).all() as {
id: string;
slug: string;
}[];
const taken = new Set(rows.map((r) => r.slug));
for (const { id, slug } of rows) {
if (!isReservedSlug(slug)) continue;
let n = 2;
while (taken.has(`${slug}-${n}`) || isReservedSlug(`${slug}-${n}`)) n++;
const renamed = `${slug}-${n}`;
taken.add(renamed);
db.prepare(`UPDATE categories SET slug = ? WHERE id = ?;`).run(renamed, id);
console.log(
`[migrate] category "${slug}" is now "${renamed}": its slug is reserved`,
);
}
} }

46
api/lib/scheduler.ts Normal file
View File

@@ -0,0 +1,46 @@
/**
* Runs the server's recurring tasks. Each module that needs one exports a
* {@link Job}, and `main.ts` passes them all to {@link startJobs} when it runs
* directly, so nothing recurs in the tests, which call a job's `run` when
* they need it.
*
* @module
*/
/** A recurring task, exported by its module for `main.ts` to start. */
export interface Job {
/** Names it in the log, as `[digest]`. */
name: string;
/** How long from one run's start to the next. */
everyMs: number;
/** How long after the start the first run waits, `everyMs` by default. */
firstAfterMs?: number;
/** Logs what it did itself, if anything; its errors are logged here. */
run(): void | Promise<void>;
}
/**
* Starts running each of `jobs` on its own timer. A run still going when the
* next is due makes that one skip, and a run that throws or rejects is logged
* and the next one runs anyway.
*/
export function startJobs(jobs: readonly Job[]): void {
for (const job of jobs) {
let running = false;
const tick = async () => {
if (running) return;
running = true;
try {
await job.run();
} catch (err) {
console.error(`[${job.name}] run failed:`, err);
} finally {
running = false;
}
};
setTimeout(() => {
tick();
setInterval(tick, job.everyMs);
}, job.firstAfterMs ?? job.everyMs);
}
}

25
api/lib/self-url.ts
View File

@@ -1,11 +1,14 @@
/** /**
* Recognises links to this site's own pages, which previews resolve from the * Recognises links to this site's own pages. The self provider previews them
* database instead of fetching. * from the database, remote/fetch.ts accepts them as dump links even on a
* private host such as localhost, and migration 0021 rewrote the v3 ones in
* stored text.
* *
* @module * @module
*/ */
import { FRONTEND_URL, PUBLIC_URL } from "../config.ts"; import { FRONTEND_URL, PUBLIC_URL } from "../config.ts";
import { parsePagePath } from "../../shared/paths.ts";
const SELF_HOSTS = new Set( const SELF_HOSTS = new Set(
[PUBLIC_URL, FRONTEND_URL].flatMap((u) => { [PUBLIC_URL, FRONTEND_URL].flatMap((u) => {
@@ -17,14 +20,22 @@ const SELF_HOSTS = new Set(
}), }),
); );
const SELF_PATH_RE = /^\/(dumps|users|playlists)\/[^/]+$/; /**
* Checks whether `host` (with its port, as `URL.host` gives it) is where this
* site is served, at its public URL or its frontend's.
*/
export function isSelfHost(host: string): boolean {
return SELF_HOSTS.has(host.toLowerCase());
}
/** /**
* Checks whether `url` links to one of this site's own dump, user or playlist * Checks whether `url` links to one of this site's own dump, user or playlist
* pages. Matched on the host as well as the path: `/users/bob` on some other * pages, by `parsePagePath`; an editor doesn't count. Matched on the
* site is someone else's page, not ours. * host as well as the path: `/@bob` on some other site is someone else's
* page, not ours.
*/ */
export function isSelfUrl(url: URL): boolean { export function isSelfUrl(url: URL): boolean {
return SELF_HOSTS.has(url.host.toLowerCase()) && if (!isSelfHost(url.host)) return false;
SELF_PATH_RE.test(url.pathname); const page = parsePagePath(url.pathname);
return !!page && !(page.kind !== "user" && page.edit);
} }

24
api/lib/serve-file.ts
View File

@@ -1,6 +1,7 @@
/** /**
* Serving files from disk with ETags and byte ranges, streamed a chunk at a * Sends files from disk with ETags and byte ranges, streamed a chunk at a
* time. * time. The files and thumbnails routes call it, and lib/upload.ts for the
* other uploads.
* *
* @module * @module
*/ */
@@ -67,21 +68,24 @@ function matchesEtag(header: string | null, etag: string): boolean {
return header.split(",").some((tag) => strip(tag) === strip(etag)); return header.split(",").some((tag) => strip(tag) === strip(etag));
} }
/** How {@link serveFile} answers. */ /**
* Passed to {@link serveFile} by its caller, which knows the file's type and
* how long it may be cached.
*/
export interface ServeFileOptions { export interface ServeFileOptions {
/** The `Content-Type` sent. */
mime: string; mime: string;
/** The `Cache-Control` sent. */
cacheControl: string; cacheControl: string;
/** Answer a single `Range: bytes=…` with a 206 (media files). */ /**
* Answers a single `Range: bytes=…` with a 206. The files route sets it, so
* that audio and video can seek.
*/
ranges?: boolean; ranges?: boolean;
} }
/** /**
* Sends a file from disk with an ETag (a 304 when the client already has it) * Sends a file from disk with an ETag, answering 304 when the client already
* and, when `ranges` is set, single byte ranges. The body is streamed. * has it, and single byte ranges when `ranges` is set. It returns false, having
* * sent nothing, when the file doesn't exist, and leaves the 404 to the caller.
* Returns false, having sent nothing, when the file doesn't exist.
*/ */
export async function serveFile( export async function serveFile(
ctx: Context, ctx: Context,

48
api/lib/site-icons.ts
View File

@@ -1,12 +1,10 @@
/** /**
* Generates the full favicon set and web manifest from a single emoji. * Renders the site's favicons and web manifest from its emoji. The `icons`
* task (`api/scripts/generate-icons.ts`) runs it before the server starts, so
* the emoji can change without a rebuild.
* *
* Twemoji ships each emoji as a plain vector-path SVG, so it rasterizes without * The glyph comes from Twemoji as plain vector paths, so rendering it needs no
* any system emoji font (which matters in the minimal Alpine runtime image). * emoji font, which the Alpine runtime image lacks.
* `@resvg/resvg-wasm` does the rasterization; the .ico is assembled by hand.
*
* Invoked by api/scripts/generate-icons.ts at server startup (into dist/ in
* prod, public/ in dev) — see that script for how the output dir is chosen.
* *
* @module * @module
*/ */
@@ -19,9 +17,10 @@ const APPLE_SIZE = 180;
const ICO_SIZES = [16, 32, 48]; const ICO_SIZES = [16, 32, 48];
/** /**
* Returns the name Twemoji files `emoji` under: its codepoints joined with * Returns the name Twemoji files `emoji` under: its codepoints in hex, joined
* "-", variation selectors (FE0F) stripped unless the emoji is a ZWJ sequence. * with "-", without variation selectors (FE0F) unless the emoji is a ZWJ
* Also doubles as a stable cache-busting version string for the icon URLs. * sequence. `lib/index-html.ts` also uses it as the icon URLs' cache-busting
* version.
*/ */
export function emojiToCodepoint(emoji: string): string { export function emojiToCodepoint(emoji: string): string {
const cleaned = emoji.includes("‍") ? emoji : emoji.replace(/️/g, ""); const cleaned = emoji.includes("‍") ? emoji : emoji.replace(/️/g, "");
@@ -47,7 +46,7 @@ function renderPng(svg: string, size: number, background?: string): Uint8Array {
return resvg.render().asPng(); return resvg.render().asPng();
} }
/** Packs PNG payloads into a (PNG-compressed) .ico container. */ /** Packs PNG images into an .ico file, which may hold PNGs as they are. */
function buildIco(images: { size: number; png: Uint8Array }[]): Uint8Array { function buildIco(images: { size: number; png: Uint8Array }[]): Uint8Array {
const HEADER = 6; const HEADER = 6;
const ENTRY = 16; const ENTRY = 16;
@@ -99,24 +98,23 @@ function buildManifest(siteName: string, bgColor: string): string {
return JSON.stringify(manifest, null, 2) + "\n"; return JSON.stringify(manifest, null, 2) + "\n";
} }
/** What {@link generateSiteIcons} renders, and where. */ /**
export interface GenerateSiteIconsOptions { * Passed to {@link generateSiteIcons} by `api/scripts/generate-icons.ts`, from
/** Emoji to render the icons from. */ * the site's config.
emoji: string;
/** Directory to write the assets into (created if missing). */
outDir: string;
/** Site name written into the manifest. */
siteName: string;
/**
* Background color for the Apple touch icon and manifest (iOS ignores
* transparency).
*/ */
export interface GenerateSiteIconsOptions {
emoji: string;
/** Created if missing. */
outDir: string;
siteName: string;
/** Fills the Apple touch icon, and the manifest's theme and background. */
bgColor: string; bgColor: string;
} }
/** /**
* Generates `favicon.svg`, `favicon.ico`, `favicon-<size>x<size>.png` (16–512), * Writes `favicon.svg`, `favicon.ico`, `favicon-<size>x<size>.png` (16 to 512),
* `apple-touch-icon.png` and `manifest.webmanifest` into `outDir`. * `apple-touch-icon.png` and `manifest.webmanifest` into `opts.outDir`, the
* directory the site's static files are served from.
* *
* @throws {Error} When the emoji's glyph can't be fetched. * @throws {Error} When the emoji's glyph can't be fetched.
*/ */
@@ -137,7 +135,7 @@ export async function generateSiteIcons(
for (const size of PNG_SIZES) { for (const size of PNG_SIZES) {
await Deno.writeFile(out(`favicon-${size}x${size}.png`), pngs.get(size)!); await Deno.writeFile(out(`favicon-${size}x${size}.png`), pngs.get(size)!);
} }
// Apple touch icon gets a solid background — iOS doesn't honor transparency. // iOS draws a transparent touch icon on black.
await Deno.writeFile( await Deno.writeFile(
out("apple-touch-icon.png"), out("apple-touch-icon.png"),
renderPng(svg, APPLE_SIZE, bgColor), renderPng(svg, APPLE_SIZE, bgColor),

92
api/lib/slugify.ts
View File

@@ -1,32 +1,86 @@
/** /**
* URL slugs for records that have a title. * Picks and checks the slugs of dumps and playlists, which name them in their
* page URLs under their owner: `/@bob/dumps/<slug>`, `/@bob/playlists/<slug>`.
* Each kind has its own slug space per owner, shared by the items' current
* slugs and their aliases (the slugs they had before). `services/slug-service.ts`
* changes and resolves slugs with these, and migration 0021 gave every item
* one; both pass the database, since this module doesn't open it.
* *
* @module * @module
*/ */
/** A UUID, which a route parameter may carry instead of a slug. */ import type { DatabaseSync } from "node:sqlite";
export const UUID_RE = import { APIException } from "./api-exception.ts";
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; import { isValidSlug, SLUG_MAX } from "../../shared/slug.ts";
/** A kind of item that has a slug under its owner. */
export type SlugKind = "dump" | "playlist";
/** Where each {@link SlugKind}'s slugs live. */
export const SLUG_TABLES: Record<
SlugKind,
{ items: string; aliases: string; itemId: string }
> = {
dump: { items: "dumps", aliases: "dump_slug_aliases", itemId: "dump_id" },
playlist: {
items: "playlists",
aliases: "playlist_slug_aliases",
itemId: "playlist_id",
},
};
/** /**
* Turns `title` into a URL-safe slug: lowercase ASCII, dash-separated, at most * Returns the id of the item whose current slug or alias is `slug` in
* 60 characters, `untitled` when nothing is left. * `userId`'s space for `kind`, or undefined when it is free.
*/ */
export function slugify(title: string): string { export function slugHolder(
const slug = title db: DatabaseSync,
.toLowerCase() kind: SlugKind,
.normalize("NFD") userId: string,
.replace(/[\u0300-\u036f]/g, "") // strip diacritics slug: string,
.replace(/[^a-z0-9]+/g, "-") // non-alphanumeric → dash ): string | undefined {
.replace(/^-+|-+$/g, "") // trim leading/trailing dashes const { items, aliases, itemId } = SLUG_TABLES[kind];
.substring(0, 60); const row = db.prepare(
return slug || "untitled"; `SELECT id FROM ${items} WHERE user_id = ? AND slug = ?
UNION ALL
SELECT ${itemId} FROM ${aliases} WHERE user_id = ? AND slug = ?
LIMIT 1;`,
).get(userId, slug, userId, slug) as { id: string } | undefined;
return row?.id;
} }
/** /**
* Returns the slug of a record: its title's slug followed by the start of its * Returns `base`, or `base-2`, `base-3`… cut to fit, whichever is free in
* `id`, so it stays unique by construction. * `userId`'s space for `kind`. `base` must be a valid slug, as `slugify`
* makes. The services take it on creating an item, the migration on
* re-slugging one, inside the transaction that writes it.
*/ */
export function makeSlug(title: string, id: string): string { export function uniqueSlug(
return `${slugify(title)}-${id.substring(0, 8)}`; db: DatabaseSync,
kind: SlugKind,
userId: string,
base: string,
): string {
for (let n = 1;; n++) {
const suffix = n === 1 ? "" : `-${n}`;
const candidate = `${
base.substring(0, SLUG_MAX - suffix.length).replace(/-+$/, "")
}${suffix}`;
if (!slugHolder(db, kind, userId, candidate)) return candidate;
}
}
/**
* Throws a 400 unless `slug` is a valid slug, lowercase words of letters and
* digits joined by hyphens, at most `max` long. slug-service checks a dump's
* or a playlist's with it, and category-service a category's, with that
* kind's own limit.
*/
export function validateSlug(slug: string, max: number = SLUG_MAX): void {
if (!isValidSlug(slug, max)) {
throw new APIException(
400,
`Slug must be lowercase letters, numbers and hyphens, at most ${max} characters`,
);
}
} }

32
api/lib/static.ts
View File

@@ -1,6 +1,7 @@
/** /**
* Serving the built frontend: static files, and `index.html` with its * Serves the built frontend, as its static files and, for every client-side
* placeholders filled for every client-side route. * route, `index.html` with its placeholders filled. main.ts installs it after
* the routers.
* *
* @module * @module
*/ */
@@ -8,38 +9,39 @@
import { Context, Next, send } from "@oak/oak"; import { Context, Next, send } from "@oak/oak";
import { loadIndexHtml } from "./index-html.ts"; import { loadIndexHtml } from "./index-html.ts";
/** Answers with the filled `index.html` of `root`, always revalidated. */
async function serveIndexHtml( async function serveIndexHtml(
context: Context<Record<string, object>>, context: Context<Record<string, object>>,
root: string, root: string,
) { ) {
const html = await loadIndexHtml(root); const html = await loadIndexHtml(root);
context.response.type = "text/html"; context.response.type = "text/html";
// Always revalidated: it names the hashed chunks of the current build. // It names the hashed chunks of the current build, so it is always
// revalidated.
context.response.headers.set("Cache-Control", "no-cache"); context.response.headers.set("Cache-Control", "no-cache");
context.response.body = html; context.response.body = html;
} }
/** /**
* Build output whose URL changes whenever its content does: Vite's hashed * Prefixes the build output whose URL changes whenever its content does, which
* chunks, and the emoji data under its package version (see vite.config.ts). * are Vite's hashed chunks and the emoji data under its package version (see
* vite.config.ts).
*/ */
const IMMUTABLE_PREFIXES = ["/assets/", "/emojibase/"]; const IMMUTABLE_PREFIXES = ["/assets/", "/emojibase/"];
const ONE_YEAR_MS = 365 * 24 * 60 * 60 * 1000; const ONE_YEAR_MS = 365 * 24 * 60 * 60 * 1000;
/** /**
* Returns a middleware serving static files from the first of `staticPaths` * Returns the middleware that serves a static file from the first of
* that has them, and `index.html` from the first for any other path so * `staticPaths` that has it, and `index.html` of the first for any other path.
* client-side routes load on direct navigation. Hashed build output is cached * Hashed build output is cached for a year, and a missing chunk is a 404.
* for a year, and a missing chunk is a 404. * main.ts passes dist/ then public/.
*/ */
export function routeStaticFilesFrom( export function routeStaticFilesFrom(
staticPaths: string[], staticPaths: string[],
): (context: Context<Record<string, object>>, next: Next) => Promise<void> { ): (context: Context<Record<string, object>>, next: Next) => Promise<void> {
return async (context: Context<Record<string, object>>, next: Next) => { // It is the last middleware and always answers, so it never calls `next`.
return async (context: Context<Record<string, object>>, _next: Next) => {
const pathname = context.request.url.pathname; const pathname = context.request.url.pathname;
// Serve index.html with runtime placeholder replacement
if (pathname === "/" || pathname === "/index.html") { if (pathname === "/" || pathname === "/index.html") {
await serveIndexHtml(context, staticPaths[0]); await serveIndexHtml(context, staticPaths[0]);
return; return;
@@ -53,7 +55,8 @@ export function routeStaticFilesFrom(
const compress = context.request.headers.has("accept-encoding"); const compress = context.request.headers.has("accept-encoding");
for (const path of staticPaths) { for (const path of staticPaths) {
try { try {
// send() picks a prebuilt .br or .gz sibling when the browser accepts it. // send() picks a prebuilt .br or .gz sibling when the browser accepts
// it.
await send(context, pathname, { await send(context, pathname, {
root: path, root: path,
brotli: compress, brotli: compress,
@@ -74,8 +77,7 @@ export function routeStaticFilesFrom(
return; return;
} }
// SPA fallback: serve index.html so client-side routes work on direct navigation // Any other path is a client-side route, opened directly or reloaded.
await serveIndexHtml(context, staticPaths[0]); await serveIndexHtml(context, staticPaths[0]);
await next();
}; };
} }

60
api/lib/ttl-cache.ts Normal file
View File

@@ -0,0 +1,60 @@
/**
* Keeps what the server fetched from another site for a while, by key, so a
* page or a feed asking again doesn't refetch it: Bandcamp tracklists, YouTube
* playlists, and link previews. Each holder sets how long an entry lives and
* how many are kept; past that count, the oldest written goes first.
*
* @module
*/
/**
* A map whose entries expire `ttlMs` after they are written, holding
* `maxEntries` at most.
*
* @example
* const playlists = new TtlCache<YouTubePlaylist>(6 * 3_600_000, 200);
* const hit = playlists.get(listId) ?? playlists.set(listId, await fetch());
*/
export class TtlCache<V> {
#entries = new Map<string, { value: V; at: number }>();
constructor(readonly ttlMs: number, readonly maxEntries: number) {}
/** Returns the value under `key` unless it is missing or expired. */
get(key: string): V | undefined {
const hit = this.#entries.get(key);
if (!hit) return undefined;
if (Date.now() - hit.at >= this.ttlMs) {
this.#entries.delete(key);
return undefined;
}
return hit.value;
}
/** Stores `value` under `key` from now on, and returns it. */
set(key: string, value: V): V {
this.#entries.delete(key);
// A Map keeps insertion order, so the first key is the oldest write.
if (this.#entries.size >= this.maxEntries) {
const oldest = this.#entries.keys().next().value;
if (oldest !== undefined) this.#entries.delete(oldest);
}
this.#entries.set(key, { value, at: Date.now() });
return value;
}
/**
* Removes `key`, or only when `value` is still what it holds, so a late
* failure can't remove the entry that replaced its own.
*/
delete(key: string, value?: V): void {
if (value === undefined || this.#entries.get(key)?.value === value) {
this.#entries.delete(key);
}
}
/** Empties the cache, for the tests' reset between cases. */
clear(): void {
this.#entries.clear();
}
}

248
api/lib/upload.ts
View File

@@ -1,35 +1,30 @@
/** /**
* Checking uploaded images and serving uploaded files back. * Checks the images users upload and serves them back, scaled down to the
* width a request names. The avatar, playlist cover, dump thumbnail and
* attachment routes call it.
* *
* @module * @module
*/ */
import type { Context } from "@oak/oak"; import type { Context } from "@oak/oak";
import { APIErrorCode } from "../../shared/api.ts";
import { APIException } from "./api-exception.ts"; import { APIException } from "./api-exception.ts";
import { serveFile } from "./serve-file.ts"; import { serveFile } from "./serve-file.ts";
import { dirname } from "node:path";
import { MAX_IMAGE_SIZE_BYTES } from "../config.ts";
import { type ImageWidth, parseImageWidth } from "../../shared/image.ts";
import { import {
ALLOWED_IMAGE_MIMES, ensureGenerated,
ATTACHMENTS_DIR, type ImageTraits,
AVATARS_DIR, type ScaledFormat,
DUMPS_DIR, scaledImagePath,
MAX_IMAGE_SIZE_BYTES, scaleImage,
PLAYLIST_IMAGES_DIR, } from "./ffmpeg.ts";
UPLOADS_DIR,
} from "../config.ts";
export {
ALLOWED_IMAGE_MIMES,
ATTACHMENTS_DIR,
AVATARS_DIR,
DUMPS_DIR,
PLAYLIST_IMAGES_DIR,
UPLOADS_DIR,
};
/** /**
* Returns the MIME type of a JPEG, PNG, GIF or WebP image from its magic * Returns the MIME type of a JPEG, PNG, GIF or WebP image from its magic
* bytes, ignoring the type the browser declared; null for anything else. * bytes, or null for anything else. These four are the only image types
* accepted anywhere. The type the browser declared is never
* trusted, since the file is later served with the type found here.
*/ */
export function detectImageMime(data: Uint8Array): string | null { export function detectImageMime(data: Uint8Array): string | null {
if (data[0] === 0xFF && data[1] === 0xD8 && data[2] === 0xFF) { if (data[0] === 0xFF && data[1] === 0xD8 && data[2] === 0xFF) {
@@ -52,7 +47,8 @@ export function detectImageMime(data: Uint8Array): string | null {
} }
/** /**
* Checks an uploaded image's size and content, and returns its MIME type. * Checks an uploaded image's size and content, and returns its MIME type,
* which the route stores to serve the image with.
* *
* @throws {APIException} 400 when it exceeds {@link MAX_IMAGE_SIZE_BYTES} or * @throws {APIException} 400 when it exceeds {@link MAX_IMAGE_SIZE_BYTES} or
* isn't a JPEG, PNG, GIF or WebP image. * isn't a JPEG, PNG, GIF or WebP image.
@@ -60,15 +56,13 @@ export function detectImageMime(data: Uint8Array): string | null {
export function validateImageUpload(data: Uint8Array): string { export function validateImageUpload(data: Uint8Array): string {
if (data.length > MAX_IMAGE_SIZE_BYTES) { if (data.length > MAX_IMAGE_SIZE_BYTES) {
throw new APIException( throw new APIException(
APIErrorCode.BAD_REQUEST,
400, 400,
"File too large (max 5 MB)", `File too large (max ${MAX_IMAGE_SIZE_BYTES / (1024 * 1024)} MB)`,
); );
} }
const mime = detectImageMime(data); const mime = detectImageMime(data);
if (!mime) { if (!mime) {
throw new APIException( throw new APIException(
APIErrorCode.BAD_REQUEST,
400, 400,
"File content is not a recognised image (JPEG, PNG, GIF, WebP)", "File content is not a recognised image (JPEG, PNG, GIF, WebP)",
); );
@@ -77,9 +71,176 @@ export function validateImageUpload(data: Uint8Array): string {
} }
/** /**
* Serves an uploaded file inline as `mime`, cached for `cacheMaxAge` seconds * Returns the EXIF orientation of a JPEG, 1 to 8, from the start of its
* (privately when `isPrivate`). Answers 404 and resolves false when the file * bytes, and 1, upright, when it states none. A phone stores a photo as its
* is missing. * sensor saw it and says here how to turn it.
*/
function jpegOrientation(data: Uint8Array): number {
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
// The segments follow the two-byte SOI, each a marker and its length.
let at = 2;
while (at + 4 <= data.length && data[at] === 0xFF) {
const marker = data[at + 1];
const length = view.getUint16(at + 2);
// APP1 holding "Exif\0\0", then a TIFF header and its first directory.
if (marker === 0xE1 && view.getUint32(at + 4) === 0x45786966) {
const tiff = at + 10;
if (tiff + 8 > data.length) return 1;
const little = view.getUint16(tiff) === 0x4949;
const ifd = tiff + view.getUint32(tiff + 4, little);
if (ifd + 2 > data.length) return 1;
const entries = view.getUint16(ifd, little);
for (let i = 0; i < entries; i++) {
const entry = ifd + 2 + i * 12;
if (entry + 12 > data.length) return 1;
if (view.getUint16(entry, little) === 0x0112) {
const value = view.getUint16(entry + 8, little);
return value >= 1 && value <= 8 ? value : 1;
}
}
return 1;
}
// The image data starts: no EXIF came before it.
if (marker === 0xDA) return 1;
at += 2 + length;
}
return 1;
}
/**
* How much of a file {@link scalableImage} needs: the EXIF segment, which
* comes first in a JPEG, is 64 KB at most.
*/
const IMAGE_HEAD_BYTES = 64 * 1024 + 32;
/** Returned by {@link scalableImage}, for `scaleImage`. */
export interface ScalableImage extends ImageTraits {
/**
* The format of its copy for a reader without WebP: a JPEG, or a PNG for
* an image that may be transparent, which a JPEG would fill with black.
*/
compat: "jpg" | "png";
}
/**
* Returns how to scale the image whose first {@link IMAGE_HEAD_BYTES} are
* `head`, or null when it is to be served as it is: an animated WebP, which
* ffmpeg can't read, or anything but a JPEG, PNG, GIF, WebP or BMP.
*/
export function scalableImage(head: Uint8Array): ScalableImage | null {
const mime = detectImageMime(head);
// A file dump may be a BMP ("BM"), which no upload route here accepts.
if (!mime) {
return head[0] === 0x42 && head[1] === 0x4D
? { orientation: 1, animated: false, compat: "jpg" }
: null;
}
// "RIFF....WEBPVP8X", whose flags then say whether frames follow.
const animatedWebp = mime === "image/webp" && head[15] === 0x58 &&
(head[20] & 0x02) !== 0;
if (animatedWebp) return null;
return mime === "image/jpeg"
? { orientation: jpegOrientation(head), animated: false, compat: "jpg" }
: { orientation: 1, animated: mime === "image/gif", compat: "png" };
}
/** Reads the first {@link IMAGE_HEAD_BYTES} of the file at `path`. */
async function readHead(path: string): Promise<Uint8Array | null> {
let file: Deno.FsFile;
try {
file = await Deno.open(path, { read: true });
} catch {
return null;
}
try {
const head = new Uint8Array(IMAGE_HEAD_BYTES);
let filled = 0;
while (filled < head.length) {
const n = await file.read(head.subarray(filled));
if (n === null) break;
filled += n;
}
return head.subarray(0, filled);
} finally {
file.close();
}
}
/** The type each format of a scaled copy is served as. */
const SCALED_MIMES: Record<ScaledFormat, string> = {
webp: "image/webp",
jpg: "image/jpeg",
png: "image/png",
};
/**
* Returns the copy of the image at `path` scaled to `width`, made now unless
* it was already, as its path and type, or null when there is none to serve:
* the image can't be scaled ({@link scalableImage}), or ffmpeg is missing or
* failed on it. The copy is a WebP, or with `compat` the JPEG or PNG a reader
* without WebP takes.
*/
async function scaledCopy(
path: string,
width: ImageWidth,
compat: boolean,
): Promise<{ path: string; mime: string } | null> {
// The other format depends on the image, so it is read before the copy is
// looked for; a WebP's path is known without.
let how = compat ? await describe(path) : undefined;
if (how === null) return null;
const format: ScaledFormat = how ? how.compat : "webp";
const scaledPath = scaledImagePath(path, width, format);
const made = await ensureGenerated(scaledPath, async () => {
how ??= await describe(path);
if (!how) throw new Error("Not an image that can be scaled");
await scaleImage(path, scaledPath, width, how, format);
}, path);
return made ? { path: scaledPath, mime: SCALED_MIMES[format] } : null;
}
/** Returns how to scale the image at `path`: see {@link scalableImage}. */
async function describe(path: string): Promise<ScalableImage | null> {
const head = await readHead(path);
return head && scalableImage(head);
}
/**
* Sends the image at `path`, of type `mime`, with `cacheControl`: a WebP copy
* at most as wide as the request's `?w=` names (`shared/image.ts`), or as
* `defaultWidth` without one, and the image as it is when the request names
* no width or the copy can't be made. With `?f=compat`, the copy is a JPEG,
* or a PNG for an image that may be transparent, which the digest's emails
* ask for, since not every mail client reads WebP. Every route that answers
* one of the site's own images ends here, so each takes both. It resolves
* false, having sent nothing, when the file is missing.
*/
export async function serveImage(
ctx: Context,
path: string,
mime: string,
cacheControl: string,
defaultWidth?: ImageWidth,
): Promise<boolean> {
// The browser must take the checked type, never guess another from the
// bytes.
ctx.response.headers.set("X-Content-Type-Options", "nosniff");
const width = parseImageWidth(ctx.request.url.searchParams.get("w")) ??
defaultWidth;
const compat = ctx.request.url.searchParams.get("f") === "compat";
const scaled = width ? await scaledCopy(path, width, compat) : null;
if (
scaled &&
await serveFile(ctx, scaled.path, { mime: scaled.mime, cacheControl })
) return true;
return await serveFile(ctx, path, { mime, cacheControl });
}
/**
* Serves an uploaded image inline as `mime`, or scaled down as
* {@link serveImage} does, cached for `cacheMaxAge` seconds, and only by the
* browser when `isPrivate`. It answers 404 and resolves false when the file
* is missing. The caller checks first that the viewer may see it.
*/ */
export async function serveUploadedFile( export async function serveUploadedFile(
ctx: Context, ctx: Context,
@@ -87,14 +248,39 @@ export async function serveUploadedFile(
mime: string, mime: string,
cacheMaxAge = 3600, cacheMaxAge = 3600,
isPrivate = false, isPrivate = false,
defaultWidth?: ImageWidth,
): Promise<boolean> { ): Promise<boolean> {
ctx.response.headers.set("Content-Disposition", "inline"); ctx.response.headers.set("Content-Disposition", "inline");
ctx.response.headers.set("X-Content-Type-Options", "nosniff"); // A frontend on another origin may still show the image.
ctx.response.headers.set("Cross-Origin-Resource-Policy", "cross-origin"); ctx.response.headers.set("Cross-Origin-Resource-Policy", "cross-origin");
const served = await serveFile(ctx, filePath, { const served = await serveImage(
ctx,
filePath,
mime, mime,
cacheControl: `${isPrivate ? "private" : "public"}, max-age=${cacheMaxAge}`, `${isPrivate ? "private" : "public"}, max-age=${cacheMaxAge}`,
}); defaultWidth,
);
if (!served) ctx.response.status = 404; if (!served) ctx.response.status = 404;
return served; return served;
} }
/**
* Writes `data` to `filePath`, replacing whatever is there only once the whole
* file is written, so a failed write leaves the old file whole. The upload
* routes call it before recording the new file, so that clients told of it
* never fetch it early.
*/
export async function replaceUploadedFile(
filePath: string,
data: Uint8Array,
): Promise<void> {
const tmpPath = `${filePath}.${crypto.randomUUID()}.tmp`;
await Deno.mkdir(dirname(filePath), { recursive: true });
try {
await Deno.writeFile(tmpPath, data);
await Deno.rename(tmpPath, filePath);
} catch (err) {
await Deno.remove(tmpPath).catch(() => {});
throw err;
}
}

626
api/mail/digest.ts Normal file
View File

@@ -0,0 +1,626 @@
/**
* Renders the weekly digest and the monthly recap, as the drafts approved
* them: the weekly one has four sections (what happened to the reader, what
* the people and playlists they follow posted, the period's top dumps, and
* the community), the monthly one only the last two. `digest-service.ts`
* gathers a {@link DigestContent} and sends what this returns, drawn in the
* reader's own style and scheme.
*
* @module
*/
import { EMAIL_LOCALE, OG_SITE_NAME } from "../config.ts";
import type { DigestCadence } from "../../shared/settings.ts";
import { i18n } from "./i18n.ts";
import { DIGEST_MESSAGES } from "../../shared/digest-messages.ts";
import {
EmailKit,
type EmailLook,
escMd,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/** The kinds of activity a weekly digest counts. */
export type ActivityKind =
| "upvotes"
| "followers"
| "comments"
| "mentions"
| "likes"
| "playlistFollows";
/** What a highlight of the weekly digest says happened. */
export type HighlightKind =
| "mentionComment"
| "mentionDump"
| "mentionPlaylist"
| "mentionChat"
| "mentionEveryone"
| "comment"
| "playlistFollow"
| "userFollow"
| "commentLike";
/** A dump in a digest's lists, with its links made absolute. */
export interface DigestDump {
title: string;
url: string;
poster: string;
/** The link's host, or null for an upload. */
domain: string | null;
/** An absolute image URL, or null to show no image. */
thumbnail: string | null;
votes: number;
comments: number;
/** Posted by the reader, which the lists mark. */
mine: boolean;
/** The followed playlist it was added to, when that's why it's listed. */
addedTo?: string;
}
/** One line of the weekly digest's "For you", made from a notification. */
export interface DigestHighlight {
kind: HighlightKind;
/** Who did it. */
who: string;
/**
* What they did it on, absent for a follow of the reader or the chat. Its
* `url` is null once it is deleted, and the title then shows unlinked.
*/
target?: { title: string; url: string | null };
/** The comment or message, flattened and shortened. */
quote?: string;
at: Date;
}
/**
* Passed to {@link renderDigestEmail} by `digest-service.ts`. Each section is
* null when it has nothing to show, and is left out; the service sends
* nothing when all of them are.
*/
export interface DigestContent {
cadence: DigestCadence;
/** The period covered, from `from` to `to`. */
from: Date;
to: Date;
reader: string;
/** Weekly only. */
activity: {
counts: Partial<Record<ActivityKind, number>>;
highlights: DigestHighlight[];
/** Every notification of the period, which the link to them counts. */
total: number;
} | null;
/** Weekly only. */
network: { dumps: DigestDump[]; more: number } | null;
top: DigestDump[];
community: {
members: { username: string; invitedBy: string | null }[];
membersMore: number;
playlists: { title: string; url: string; owner: string; dumps: number }[];
playlistsMore: number;
chatMessages: number;
totals: { dumps: number; upvotes: number; comments: number };
} | null;
links: {
notifications: string;
followed: string;
chat: string;
/** The community page's view of the cadence's period. */
community: string;
settings: string;
/** The one-click page that turns this cadence off. */
unsubscribe: string;
};
}
const ACTIVITY_ORDER: ActivityKind[] = [
"upvotes",
"followers",
"comments",
"mentions",
"likes",
"playlistFollows",
];
/** Returns the label of `n` of a kind of activity, as in "12 upvotes". */
function countLabel(kind: ActivityKind, n: number): string {
switch (kind) {
case "upvotes":
return i18n._({ ...DIGEST_MESSAGES.countUpvotes, values: { n } });
case "followers":
return i18n._({
id: "email.digest.count.followers",
message: "{n, plural, one {new follower} other {new followers}}",
values: { n },
});
case "comments":
return i18n._({ ...DIGEST_MESSAGES.countComments, values: { n } });
case "mentions":
return i18n._({
id: "email.digest.count.mentions",
message: "{n, plural, one {mention} other {mentions}}",
values: { n },
});
case "likes":
return i18n._({
id: "email.digest.count.likes",
message: "{n, plural, one {comment like} other {comment likes}}",
values: { n },
});
case "playlistFollows":
return i18n._({
id: "email.digest.count.playlistFollows",
message: "{n, plural, one {playlist follow} other {playlist follows}}",
values: { n },
});
}
}
/**
* Returns the sentence of a highlight, `who` and `target` already formatted
* for the part it goes in (HTML or Markdown).
*/
function highlightSentence(
kind: HighlightKind,
who: string,
target: string,
): string {
const values = { who, target };
switch (kind) {
case "mentionComment":
return i18n._({
id: "email.digest.hl.mentionComment",
message: "{who} mentioned you in a comment on {target}",
values,
});
case "mentionDump":
case "mentionChat":
return i18n._({
id: "email.digest.hl.mentionIn",
message: "{who} mentioned you in {target}",
values,
});
case "mentionPlaylist":
return i18n._({
id: "email.digest.hl.mentionPlaylist",
message: "{who} mentioned you in the playlist {target}",
values,
});
case "mentionEveryone":
return i18n._({
id: "email.digest.hl.mentionEveryone",
message: "{who} mentioned everyone in {target}",
values,
});
case "comment":
return i18n._({
id: "email.digest.hl.comment",
message: "{who} commented on your dump {target}",
values,
});
case "playlistFollow":
return i18n._({
id: "email.digest.hl.playlistFollow",
message: "{who} followed your playlist {target}",
values,
});
case "userFollow":
return i18n._({
id: "email.digest.hl.userFollow",
message: "{who} started following you",
values,
});
case "commentLike":
return i18n._({
id: "email.digest.hl.commentLike",
message: "{who} liked your comment on {target}",
values,
});
}
}
const byLine = (username: string) =>
i18n._({
id: "email.digest.by",
message: "by @{username}",
values: { username },
});
const addedToLine = (playlist: string) =>
i18n._({
id: "email.digest.addedTo",
message: "added to {playlist}",
values: { playlist },
});
const uploadLabel = () =>
i18n._({ id: "email.digest.upload", message: "upload" });
const invitedLine = (inviter: string, reader: string) =>
inviter === reader
? i18n._(DIGEST_MESSAGES.invitedByYou)
: i18n._({ ...DIGEST_MESSAGES.invitedBy, values: { username: inviter } });
const andMoreLine = (n: number) =>
i18n._({
id: "email.digest.andMore",
message: "and {n} more",
values: { n },
});
const playlistMeta = (owner: string, n: number) =>
i18n._({
id: "email.digest.playlistMeta",
message: "by @{username} · {n, plural, one {# dump} other {# dumps}}",
values: { username: owner, n },
});
/**
* Returns the digest `content` describes, written as Markdown and drawn in
* `look`, with its subject and hidden preheader.
*/
export function renderDigestEmail(
content: DigestContent,
look: EmailLook,
): RenderedEmail {
const monthly = content.cadence === "monthly";
const nf = new Intl.NumberFormat(EMAIL_LOCALE);
const range = (year: boolean) =>
new Intl.DateTimeFormat(
EMAIL_LOCALE,
year
? { month: "short", day: "numeric", year: "numeric" }
: { month: "short", day: "numeric" },
).formatRange(content.from, content.to);
const day = (d: Date) =>
new Intl.DateTimeFormat(EMAIL_LOCALE, { weekday: "short" }).format(d);
const figure = (n: number, label: string) =>
`**${nf.format(n)}** ${escMd(label)}`;
const yours = i18n._({ id: "email.digest.yours", message: "yours" });
/** A dump as a list item: its thumbnail, title, pill and detail. */
const dumpItem = (d: DigestDump, bullet: string, image = true) =>
`${bullet} ${image && d.thumbnail ? `![](${d.thumbnail}) ` : ""}${
mdLink(d.title, d.url)
}${d.mine ? ` *${escMd(yours)}*` : ""} · ${
escMd(
`${d.addedTo ? addedToLine(d.addedTo) : byLine(d.poster)}${
d.addedTo ? "" : ` · ${d.domain ?? uploadLabel()}`
} · ${nf.format(d.votes)} ${countLabel("upvotes", d.votes)} · ${
nf.format(d.comments)
} ${countLabel("comments", d.comments)}`,
)
}`;
const md: string[] = [
`*${
escMd(
`${
monthly
? i18n._(DIGEST_MESSAGES.monthly)
: i18n._(DIGEST_MESSAGES.weekly)
} · ${range(true)}`,
)
}*`,
`# ${
escMd(
monthly
? i18n._({
id: "email.digest.monthly.title",
message: "The month on {site}",
values: { site: OG_SITE_NAME },
})
: i18n._({
id: "email.digest.weekly.title",
message: "Your week, {username}",
values: { username: content.reader },
}),
)
}`,
];
const { activity, network, community } = content;
if (activity) {
md.push(
`## ${escMd(i18n._({ id: "email.digest.forYou", message: "For you" }))}`,
ACTIVITY_ORDER.filter((k) => (activity.counts[k] ?? 0) > 0)
.map((k) =>
figure(activity.counts[k]!, countLabel(k, activity.counts[k]!))
)
.join(" · "),
activity.highlights.map((h) => {
const target = h.kind === "mentionChat" || h.kind === "mentionEveryone"
? mdLink(
i18n._({ id: "email.digest.theChat", message: "the chat" }),
content.links.chat,
)
: h.target?.url
? mdLink(h.target.title, h.target.url)
: h.target
? escMd(h.target.title)
: "";
const line = `- ${
highlightSentence(h.kind, `**@${escMd(h.who)}**`, target)
} *${escMd(day(h.at))}*`;
return h.quote ? `${line}\n > ${escMd(h.quote)}` : line;
}).join("\n"),
`${
mdLink(
i18n._({
id: "email.digest.seeAll",
message:
"{n, plural, one {See the notification} other {See all # notifications}}",
values: { n: activity.total },
}),
content.links.notifications,
)
} →`,
);
}
if (network) {
md.push(
`## ${
escMd(
i18n._({
id: "email.digest.network",
message: "From people you follow",
}),
)
}`,
network.dumps.map((d) => dumpItem(d, "-")).join("\n"),
);
if (network.more > 0) {
md.push(`${
mdLink(
i18n._({
id: "email.digest.more",
message: "{n} more on your Followed feed",
values: { n: network.more },
}),
content.links.followed,
)
} →`);
}
}
if (content.top.length > 0) {
const [first] = content.top;
md.push(
`## ${
escMd(
monthly
? i18n._(DIGEST_MESSAGES.topMonth)
: i18n._(DIGEST_MESSAGES.topWeek),
)
}`,
);
// The recap gives its first dump a wide image and the next two a thumbnail.
if (monthly && first.thumbnail) md.push(`![](${first.thumbnail})`);
md.push(
content.top.map((d, i) =>
dumpItem(d, `${i + 1}.`, monthly ? i === 1 || i === 2 : true)
).join("\n"),
);
}
if (community) {
const { dumps, upvotes, comments } = community.totals;
md.push(
`## ${escMd(i18n._(DIGEST_MESSAGES.community))}`,
);
if (monthly) {
md.push(
[
figure(
dumps,
i18n._({ ...DIGEST_MESSAGES.statDumps, values: { n: dumps } }),
),
figure(upvotes, countLabel("upvotes", upvotes)),
figure(comments, countLabel("comments", comments)),
figure(
community.chatMessages,
i18n._({
...DIGEST_MESSAGES.statChat,
values: { n: community.chatMessages },
}),
),
].join(" · "),
);
}
if (community.members.length > 0) {
md.push(
`### ${
escMd(
i18n._(DIGEST_MESSAGES.newMembers),
)
}`,
[
...community.members.map((m) =>
`- **@${escMd(m.username)}**${
m.invitedBy
? ` ${escMd(invitedLine(m.invitedBy, content.reader))}`
: ""
}`
),
...(community.membersMore > 0
? [`- ${escMd(andMoreLine(community.membersMore))}`]
: []),
].join("\n"),
);
}
if (community.playlists.length > 0) {
md.push(
`### ${escMd(i18n._(DIGEST_MESSAGES.newPlaylists))}`,
[
...community.playlists.map((p) =>
`- ${mdLink(p.title, p.url)} · ${
escMd(playlistMeta(p.owner, p.dumps))
}`
),
...(community.playlistsMore > 0
? [`- ${escMd(andMoreLine(community.playlistsMore))}`]
: []),
].join("\n"),
);
}
if (!monthly && community.chatMessages > 0) {
md.push(
`### ${
escMd(i18n._({ id: "email.digest.inChat", message: "In the chat" }))
}`,
`${
escMd(
i18n._({
id: "email.digest.chatWeek",
message:
"{n, plural, one {# message this week} other {# messages this week}}",
values: { n: community.chatMessages },
}),
)
} · ${
mdLink(
i18n._({ id: "email.digest.openChat", message: "Open the chat" }),
content.links.chat,
)
}`,
);
}
if (!monthly && dumps + upvotes + comments > 0) {
md.push(
`*${
escMd(i18n._({
id: "email.digest.totals",
message:
"{dumps, plural, one {# dump} other {# dumps}}, {upvotes, plural, one {# upvote} other {# upvotes}} and {comments, plural, one {# comment} other {# comments}}.",
values: { dumps, upvotes, comments },
}))
}*`,
);
}
md.push(`${
mdLink(
i18n._({
id: "email.digest.communityPage",
message: "See it all on the community page",
}),
content.links.community,
)
} →`);
}
md.push(
"---",
escMd(
monthly
? i18n._({
id: "email.digest.monthly.reason",
message:
"You're getting the monthly recap because it's on in your settings.",
})
: i18n._({
id: "email.digest.weekly.reason",
message:
"You're getting the weekly digest because it's on in your settings.",
}),
),
`${
mdLink(
monthly
? i18n._(DIGEST_MESSAGES.monthlyOff)
: i18n._(DIGEST_MESSAGES.weeklyOff),
content.links.unsubscribe,
)
} · ${
mdLink(
i18n._({ id: "email.settings", message: "Settings" }),
content.links.settings,
)
}`,
);
return new EmailKit(look).render({
subject: monthly
? i18n._({
id: "email.digest.monthly.subject",
message: "The month on {site}, {range}",
values: { site: OG_SITE_NAME, range: range(false) },
})
: i18n._({
id: "email.digest.weekly.subject",
message: "Your week on {site}, {range}",
values: { site: OG_SITE_NAME, range: range(false) },
}),
preheader: preheader(content),
markdown: md.join("\n\n"),
});
}
/**
* Returns the line an inbox shows beside the subject: the counts that make
* the digest worth opening, at most three of them.
*/
function preheader(content: DigestContent): string {
const parts: string[] = [];
if (content.cadence === "weekly") {
for (const k of ["upvotes", "followers", "comments"] as const) {
const n = content.activity?.counts[k] ?? 0;
if (n > 0) parts.push(`${n} ${countLabel(k, n)}`);
}
const n = (content.network?.dumps.length ?? 0) +
(content.network?.more ?? 0);
if (n > 0) {
parts.push(
i18n._({
id: "email.digest.pre.network",
message:
"{n, plural, one {# new dump from people you follow} other {# new dumps from people you follow}}",
values: { n },
}),
);
}
} else {
const members = (content.community?.members.length ?? 0) +
(content.community?.membersMore ?? 0);
const playlists = (content.community?.playlists.length ?? 0) +
(content.community?.playlistsMore ?? 0);
if (content.top.length > 0) {
parts.push(
i18n._({
id: "email.digest.pre.topMonth",
message:
"{n, plural, one {the most upvoted dump of the month} other {the # most upvoted dumps of the month}}",
values: { n: content.top.length },
}),
);
}
if (members > 0) {
parts.push(
i18n._({
id: "email.digest.pre.members",
message: "{n, plural, one {# new member} other {# new members}}",
values: { n: members },
}),
);
}
if (playlists > 0) {
parts.push(
i18n._({
id: "email.digest.pre.playlists",
message: "{n, plural, one {# new playlist} other {# new playlists}}",
values: { n: playlists },
}),
);
}
}
if (parts.length === 0 && content.top.length > 0) {
parts.push(
i18n._({
id: "email.digest.pre.topWeek",
message:
"{n, plural, one {the most upvoted dump of the week} other {the # most upvoted dumps of the week}}",
values: { n: content.top.length },
}),
);
}
if (parts.length === 0) return "";
const line = new Intl.ListFormat(EMAIL_LOCALE, { type: "conjunction" })
.format(parts.slice(0, 3));
return `${line.charAt(0).toUpperCase()}${line.slice(1)}.`;
}

36
api/mail/i18n.ts Normal file
View File

@@ -0,0 +1,36 @@
/**
* Lets the emails speak the instance's language from the same catalogs as
* the interface. A template passes each message as a descriptor holding its
* id and its English, as `i18n._({ id: "email.reset.title", message: "Reset
* your password" })`, or one of `shared/digest-messages.ts`, which a page
* says too. `lingui extract` reads them into `locales/<lang>.po` beside the
* interface's messages, and `lingui compile` turns the catalogs into the
* `locales/<lang>.mjs` imported here, failing when a language lacks one. The
* ids are named rather than hashed from the English, since the server runs
* no Lingui macro.
*
* @module
*/
import { type I18n, type Messages, setupI18n } from "@lingui/core";
import { EMAIL_LOCALE } from "../config.ts";
import { LOCALES } from "../../shared/locale.ts";
const catalogs: [string, Messages][] = await Promise.all(
LOCALES.map(async (locale) =>
[
locale,
(await import(`../../locales/${locale}.mjs`)).messages as Messages,
] as [string, Messages]
),
);
/**
* Translates the emails' messages, in {@link EMAIL_LOCALE}. Every catalog is
* loaded, so a test can activate another language.
*/
export const i18n: I18n = setupI18n({
locale: EMAIL_LOCALE,
messages: Object.fromEntries(catalogs),
});

822
api/mail/layout.ts Normal file
View File

@@ -0,0 +1,822 @@
/**
* Every email the site sends shares one layout: the site's emoji and name,
* then a card holding the content, then a footer. The templates beside this
* module (welcome, password reset, digest) build their content with an
* {@link EmailKit}, which draws each block in a style from `styles.ts`, and
* `email-service.ts` sends what they return.
*
* Mail clients drop stylesheets, so every block carries inline styles. A
* reader on the "auto" scheme gets the light palette inline and the dark one
* in a `prefers-color-scheme` block that Apple's mail apps honour; elsewhere
* the light one shows.
*
* @module
*/
import { Marked, Parser, type Token, type Tokens } from "marked";
import {
DEFAULT_STYLE,
EMAIL_LOCALE,
OG_SITE_NAME,
SITE_EMOJI,
} from "../config.ts";
import {
type ColorScheme,
DEFAULT_COLOR_SCHEME,
type StyleName,
} from "../../shared/theme.ts";
import type { UserSettings } from "../../shared/settings.ts";
import { EMAIL_STYLES, type EmailPalette, type EmailStyle } from "./styles.ts";
/**
* Returned by each template to its caller, which hands it to `sendEmail`
* with a recipient.
*/
export interface RenderedEmail {
subject: string;
html: string;
/**
* The plain-text part, for clients that show no HTML. It is Markdown, so it
* reads as text and renders as the email's outline.
*/
text: string;
}
/**
* Passed to {@link EmailKit}, made by {@link lookOf}: the instance's style for
* the emails that aren't a user's own, their settings for a digest.
*/
export interface EmailLook {
style: StyleName;
scheme: ColorScheme;
}
/**
* Returns the look of a reader's settings. An unset style is the instance's,
* as on the site, and an unset scheme the system's; with no settings at all,
* as for the welcome and reset emails, both are.
*/
export function lookOf(
settings: Pick<UserSettings, "style" | "colorScheme"> | null = null,
): EmailLook {
return {
style: settings?.style ?? DEFAULT_STYLE,
scheme: settings?.colorScheme ?? DEFAULT_COLOR_SCHEME,
};
}
/**
* Names a color of an {@link EmailPalette}, as {@link EmailKit.paint} takes
* it, so that the dark block can recolor the element by the same name.
*/
export type PaletteRole = Exclude<keyof EmailPalette, "gradient">;
const ROLES: PaletteRole[] = [
"outer",
"card",
"text",
"sec",
"muted",
"accent",
"link",
"onAccent",
"rule",
"soft",
"quote",
"date",
"brand",
"border",
];
/**
* Escapes `value` for HTML text and attributes. Everything users wrote (names,
* titles, quotes) goes through it, since none of it may become markup.
*/
export function esc(value: string): string {
return value.replace(
/[&<>"']/g,
(c) =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[
c
]!,
);
}
/**
* Escapes `value` for Markdown, so that user text in a plain-text part (a
* title with `*` or `[`, say) stays text.
*/
export function escMd(value: string): string {
return value.replace(/[\\`*_[\]<>#!|~]/g, "\\$&");
}
/** Returns a Markdown link to `url` labelled with `text`, escaped. */
export function mdLink(text: string, url: string): string {
const safeUrl = url.replace(
/[()\s]/g,
(c) => `%${c.charCodeAt(0).toString(16).toUpperCase().padStart(2, "0")}`,
);
return `[${escMd(text)}](${safeUrl})`;
}
const gradient = (colors: string[]) =>
`linear-gradient(90deg,${colors.join(",")})`;
/**
* Draws an email in one {@link EmailLook}. A template writes the email as
* Markdown and hands it to {@link EmailKit.render}, which draws each block in
* the style. The block methods return HTML: text passed to them must already
* be escaped with {@link esc}.
*/
export class EmailKit {
/** The style being drawn. */
readonly style: EmailStyle;
/** The palette inlined into the HTML. */
readonly palette: EmailPalette;
/** The palette that replaces it in dark mode, on the "auto" scheme only. */
readonly darkPalette: EmailPalette | null;
/** The font stack, for templates that set their own blocks. */
readonly font: string;
constructor(look: EmailLook) {
this.style = EMAIL_STYLES[look.style];
this.palette = look.scheme === "dark" ? this.style.dark : this.style.light;
this.darkPalette = look.scheme === DEFAULT_COLOR_SCHEME
? this.style.dark
: null;
this.font = this.style.font;
}
/** Returns `px`, scaled for the style, as a CSS length. */
fs(px: number): string {
return `${Math.round(px * this.style.scale * 2) / 2}px`;
}
/**
* Returns the `class` and `style` attributes of an element colored by
* roles: `fg` for its text, `bg` for its fill, `bd` for its border. The
* classes let the dark block recolor it; `css` is the rest of its style.
*/
paint(
roles: {
fg?: PaletteRole;
bg?: PaletteRole;
bd?: PaletteRole;
extra?: string[];
},
css = "",
): string {
const p = this.palette;
const classes: string[] = [...(roles.extra ?? [])];
let style = "";
if (roles.fg) {
classes.push(`f-${roles.fg}`);
style += `color:${p[roles.fg] ?? p.text};`;
}
if (roles.bg) {
classes.push(`b-${roles.bg}`);
style += `background-color:${p[roles.bg] ?? p.card};`;
}
if (roles.bd) {
classes.push(`d-${roles.bd}`);
style += `border-color:${p[roles.bd] ?? p.rule};`;
}
const cls = classes.length ? ` class="${classes.join(" ")}"` : "";
// The colors come last: a shorthand such as `border-top` in `css` would
// otherwise reset them.
return `${cls} style="${css}${style}"`;
}
/** Returns a link to `href` around `html`, in the link color. */
link(html: string, href: string, strong = false): string {
const s = this.style;
return `<a href="${esc(href)}"${
this.paint(
{ fg: "link" },
`text-decoration:${s.linkUnderline ? "underline" : "none"};${
strong ? `font-weight:${s.heavy};` : ""
}`,
)
}>${html}</a>`;
}
/** Returns the email's heading, in the style's title type. */
heading(html: string, marginBottom = 24): string {
const t = this.style.title;
return `<h1${
this.paint(
{ fg: t.accent ? "accent" : "text" },
`font-family:${this.font};font-size:${
this.fs(t.size)
};line-height:1.15;font-weight:${t.weight};${
t.upper ? "text-transform:uppercase;" : ""
}${
t.spacing ? `letter-spacing:${t.spacing};` : ""
}margin:0 0 ${marginBottom}px;`,
)
}>${html}</h1>`;
}
/** Returns a paragraph. `css` adds to or overrides its style. */
para(html: string, css = ""): string {
return `<p${
this.paint(
{ fg: "text" },
`font-family:${this.font};font-size:${
this.fs(15)
};line-height:1.6;margin:0 0 16px;${css}`,
)
}>${html}</p>`;
}
/** Returns a smaller paragraph in the muted color. */
note(html: string, css = ""): string {
return `<p${
this.paint(
{ fg: "muted" },
`font-family:${this.font};font-size:${
this.fs(13.5)
};line-height:1.6;margin:0 0 16px;${css}`,
)
}>${html}</p>`;
}
/** Returns a button linking to `href`. */
button(label: string, href: string): string {
const s = this.style;
const frame = s.frame === "hard"
? `border:2px solid ${this.palette.text};box-shadow:2px 2px 0 ${this.palette.text};`
: "";
return `<table role="presentation" cellpadding="0" cellspacing="0" style="margin:6px 0 22px;"><tr><td${
this.paint(
{
bg: "accent",
bd: s.frame === "hard" ? "text" : undefined,
extra: s.frame === "hard" ? ["sh-btn"] : [],
},
`border-radius:${s.radius.btn}px;${frame}`,
)
}><a href="${esc(href)}"${
this.paint(
{ fg: "onAccent" },
`display:inline-block;padding:13px 22px;font-family:${this.font};font-size:${
this.fs(13.5)
};font-weight:${
Math.max(600, s.heavy)
};letter-spacing:0.08em;text-transform:uppercase;line-height:1;text-decoration:none;`,
)
}>${label}</a></td></tr></table>`;
}
/** Returns a small uppercase pill, as on one's own dump in a list. */
pill(label: string): string {
return `<span${
this.paint(
{ fg: "sec", bg: "soft" },
`display:inline-block;margin-left:6px;padding:1px 7px;border-radius:${this.style.radius.pill}px;font-size:${
this.fs(11)
};font-weight:600;letter-spacing:0.08em;text-transform:uppercase;vertical-align:2px;`,
)
}>${label}</span>`;
}
/** Returns an image block, rounded as the style rounds images. */
image(
src: string,
alt: string,
width: number,
height: number,
href?: string,
): string {
const img = `<img src="${
esc(src)
}" width="${width}" height="${height}" alt="${
esc(alt)
}" style="display:block;max-width:100%;height:auto;border:0;border-radius:${
Math.min(this.style.radius.img, width / 8)
}px;">`;
return href ? `<a href="${esc(href)}">${img}</a>` : img;
}
/** Returns a small bold heading inside a section. */
subhead(html: string): string {
return `<div${
this.paint(
{ fg: "text" },
`font-family:${this.font};font-size:${this.fs(14)};font-weight:${
Math.max(600, this.style.heavy)
};margin:0 0 8px;`,
)
}>${html}</div>`;
}
/**
* Returns a section of the card: a divider in the style's rule (none before
* the first), an uppercase title in the accent color, and `html`.
*/
section(title: string, html: string, first = false): string {
const s = this.style;
let divider = "";
let border = "";
if (!first) {
const stops = this.palette.gradient;
if (s.rule === "gradient" && stops) {
divider =
`<div class="grad" style="height:3px;line-height:3px;font-size:0;background-color:${
stops[0]
};background-image:${gradient(stops)};margin:0 0 26px;">&nbsp;</div>`;
} else {
border = `border-top:${
s.rule === "thick" ? 2 : 1
}px solid;padding-top:26px;`;
}
}
return `<div${
this.paint(
{ bd: border ? (s.rule === "thick" ? "text" : "rule") : undefined },
`${border}padding-bottom:26px;`,
)
}>${divider}<div${
this.paint(
{ fg: "accent" },
`font-family:${this.font};font-size:${this.fs(12)};font-weight:${
Math.max(600, s.heavy)
};letter-spacing:0.12em;text-transform:uppercase;margin:0 0 16px;`,
)
}>${title}</div>${html}</div>`;
}
/**
* Returns the email written as `markdown`: the Markdown itself is the
* plain-text part, and its blocks, drawn in this kit's style, are the HTML.
* Each template writes its email once this way. The blocks map as follows.
*
* - `# Title` is the heading; an `*emphasised*` paragraph above it, the
* eyebrow. Elsewhere an emphasised paragraph is a note.
* - `## Section` starts a section, with its divider; `### Name` a subhead.
* - A link alone in a paragraph is a button, an image alone a wide image,
* and a fenced block a box for a raw link.
* - A list item whose text starts with an image, then a link, is a row
* with that thumbnail and the link as its title; what follows is its
* detail, and an emphasised word right after the link a pill. Other items
* are sentences, where emphasis marks the date. A numbered list ranks its
* rows, and a quote under an item is set as one.
* - A bold number is a figure.
* - What follows the last `---` is the footer.
*/
render(
{ subject, preheader, markdown }: {
subject: string;
preheader: string;
markdown: string;
},
): RenderedEmail {
const cut = markdown.lastIndexOf("\n---\n");
const body = cut === -1 ? markdown : markdown.slice(0, cut);
const foot = cut === -1 ? "" : markdown.slice(cut + 5);
return {
subject,
html: this.document({
subject,
preheader,
body: this.blocks(this.marked.lexer(body)),
footer: this.marked.lexer(foot)
.filter((t): t is Tokens.Paragraph => t.type === "paragraph")
.map((t) => this.inline(t.tokens, "footer")).join("<br>"),
}),
text: markdown,
};
}
/** How {@link inline} draws emphasis and links, by where they sit. */
private mode: "text" | "footer" | "row" | "sentence" = "text";
/** Draws Markdown's inline tokens with the kit's colors. */
// The kit comes in as an argument: the renderer's methods need marked's
// `this` (its parser) as well as the kit's colors.
private readonly marked: Marked = ((kit: EmailKit) => {
return new Marked({
renderer: {
link({ href, tokens }) {
const html = this.parser.parseInline(tokens);
return kit.mode === "footer"
? `<a href="${esc(href)}"${
kit.paint({ fg: "muted" }, "text-decoration:underline;")
}>${html}</a>`
: kit.link(html, href);
},
strong({ tokens }) {
const html = this.parser.parseInline(tokens);
// A figure, its digits grouped by any space: `\s` takes the no-break
// ones locales group thousands with.
const figure = /^[\d\s.,]+$/.test(html);
return `<b${
kit.paint(
{ fg: "text" },
figure
? `font-size:${kit.fs(22)};font-weight:${kit.style.heavy};`
: `font-weight:${Math.max(600, kit.style.heavy)};`,
)
}>${html}</b>`;
},
em({ tokens }) {
const html = this.parser.parseInline(tokens);
if (kit.mode === "row") return kit.pill(html);
if (kit.mode === "sentence") {
return `<span${
kit.paint(
{ fg: kit.palette.date ? "date" : "muted" },
`font-size:${kit.fs(12.5)};white-space:nowrap;${
kit.palette.date ? "font-style:italic;" : ""
}`,
)
}>${html}</span>`;
}
return `<i>${html}</i>`;
},
codespan({ text }) {
return `<code style="font-family:Menlo,Consolas,'Courier New',monospace;">${text}</code>`;
},
image({ href, text }) {
return kit.image(href, text, 104, 78);
},
},
});
})(this);
private inline(tokens: Token[], mode: EmailKit["mode"]): string {
this.mode = mode;
try {
return Parser.parseInline(tokens, this.marked.defaults);
} finally {
this.mode = "text";
}
}
/** Draws the body's blocks, gathering each `##` section with its content. */
private blocks(tokens: Token[]): string {
const out: string[] = [];
let section: { title: string; parts: string[] } | null = null;
let first = true;
let headed = false;
const push = (html: string) =>
section ? section.parts.push(html) : out.push(html);
const close = () => {
if (!section) return;
out.push(this.section(section.title, section.parts.join(""), first));
first = false;
section = null;
};
for (const token of tokens) {
switch (token.type) {
case "heading": {
const html = this.inline(token.tokens ?? [], "text");
if (token.depth === 1) {
headed = true;
push(this.heading(html));
} else if (token.depth === 2) {
close();
section = { title: html, parts: [] };
} else {
push(this.subhead(html));
}
break;
}
case "paragraph": {
const kids = (token.tokens ?? []).filter((t) =>
!(t.type === "text" && !t.raw.trim())
);
const only = kids.length === 1 ? kids[0] : null;
if (only?.type === "link") {
push(
this.button(this.inline(only.tokens ?? [], "text"), only.href),
);
} else if (only?.type === "image") {
push(
`${
this.image(only.href, only.text, 528, 264)
}<div style="height:14px;line-height:14px;font-size:0;">&nbsp;</div>`,
);
} else if (only?.type === "em") {
const html = this.inline(only.tokens ?? [], "text");
push(headed ? this.note(html) : this.eyebrow(html));
} else {
push(this.para(this.inline(token.tokens ?? [], "text")));
}
break;
}
case "list":
push(this.list(token as Tokens.List));
break;
case "blockquote":
push(this.quote(this.inline(
(token.tokens ?? []).flatMap((t) =>
"tokens" in t ? t.tokens ?? [] : []
),
"text",
)));
break;
case "code":
push(this.linkBox(esc(token.text)));
break;
case "html":
push(token.text);
break;
}
}
close();
return out.join("");
}
/** Draws a list as rows, ranked when it is numbered. */
private list(list: Tokens.List): string {
const start = typeof list.start === "number" ? list.start : 1;
return list.items.map((item, i) => {
const text = item.tokens.find((t) =>
t.type === "text" || t.type === "paragraph"
) as Tokens.Text | Tokens.Paragraph | undefined;
const quote = item.tokens.find((t) => t.type === "blockquote") as
| Tokens.Blockquote
| undefined;
let rest: Token[] = [...(text?.tokens ?? [])];
const skipSpace = () => {
while (rest[0]?.type === "text" && !rest[0].raw.trim()) {
rest = rest.slice(1);
}
};
skipSpace();
const image = rest[0]?.type === "image"
? rest.shift() as Tokens.Image
: null;
skipSpace();
const link = rest[0]?.type === "link"
? rest.shift() as Tokens.Link
: null;
let content: string;
if (link) {
const pills: Token[] = [];
skipSpace();
while (rest[0]?.type === "em") {
pills.push(rest.shift()!);
skipSpace();
}
// The detail follows the title after a " · ".
if (rest[0]?.type === "text") {
const t = rest[0] as Tokens.Text;
const trimmed = t.text.replace(/^\s*·\s*/, "");
rest[0] = { ...t, text: trimmed, raw: trimmed, tokens: undefined };
}
content = `<a href="${esc(link.href)}"${
this.paint(
{ fg: "text" },
`font-size:${
this.fs(15)
};line-height:1.35;font-weight:${this.style.heavy};text-decoration:none;`,
)
}>${this.inline(link.tokens, "text")}</a>${this.inline(pills, "row")}${
rest.length
? `<div${
this.paint(
{ fg: "muted" },
`font-size:${this.fs(13)};line-height:1.5;margin-top:3px;`,
)
}>${this.inline(rest, "text")}</div>`
: ""
}`;
} else {
content = `<div${
this.paint(
{ fg: "sec" },
`font-size:${this.fs(14.5)};line-height:1.5;`,
)
}>${this.inline(rest, "sentence")}</div>`;
}
if (quote) {
content += this.quote(this.inline(
quote.tokens.flatMap((t) => "tokens" in t ? t.tokens ?? [] : []),
"text",
));
}
const size = list.ordered ? [64, 48] : [104, 78];
return `<table role="presentation" width="100%" cellpadding="0" cellspacing="0" style="margin:0 0 ${
link || image ? 16 : 10
}px;"><tr>${
list.ordered
? `<td width="30" valign="top"${
this.paint(
{ fg: "accent" },
`font-family:${this.font};font-size:${this.fs(20)};font-weight:${
Math.max(600, this.style.heavy)
};line-height:1.2;`,
)
}>${start + i}</td>`
: ""
}${
image
? `<td width="${size[0]}" valign="top" style="padding-right:${
list.ordered ? 12 : 14
}px;">${
this.image(image.href, image.text, size[0], size[1], link?.href)
}</td>`
: ""
}<td valign="top" style="font-family:${this.font};">${content}</td></tr></table>`;
}).join("");
}
/** Returns the small uppercase line above the heading. */
private eyebrow(html: string): string {
return `<div${
this.paint(
{ fg: "muted" },
`font-family:${this.font};font-size:${
this.fs(12)
};font-weight:600;letter-spacing:0.12em;text-transform:uppercase;margin:0 0 8px;`,
)
}>${html}</div>`;
}
/** Returns a quoted comment or message. */
private quote(html: string): string {
return `<div${
this.paint(
{ fg: "quote" },
`font-family:${this.font};font-size:${
this.fs(14.5)
};line-height:1.5;font-style:italic;margin:4px 0 0;`,
)
}>“${html}”</div>`;
}
/** Returns a raw link set in a box, for clients that break buttons. */
private linkBox(html: string): string {
return `<div${
this.paint(
{ fg: "sec", bg: "soft" },
`font-family:Menlo,Consolas,'Courier New',monospace;font-size:12px;line-height:1.5;border-radius:${
Math.min(this.style.radius.img, 6)
}px;padding:10px 12px;margin:0 0 22px;word-break:break-all;`,
)
}>${html}</div>`;
}
/**
* Returns the whole HTML document: the brand, the card holding `body`, and
* the `footer`. `preheader` is the hidden first line inboxes show beside the
* subject.
*/
private document(
{ subject, preheader, body, footer }: {
subject: string;
preheader: string;
body: string;
footer: string;
},
): string {
const s = this.style;
const p = this.palette;
const R = s.radius.card;
const bar = s.bar ? p.gradient ?? null : null;
const frame = s.frame === "hard"
? `border:2px solid ${p.text};box-shadow:5px 5px 0 ${p.text};`
: s.frame === "ridge"
? `border:3px ridge ${p.border};`
: s.frame === "hairline"
? `border:1px solid ${p.rule};`
: s.frame === "soft"
? `box-shadow:0 14px 34px -10px ${p.shadow};`
: "";
const frameRoles = s.frame === "hard"
? { bd: "text" as const, extra: ["sh-card"] }
: s.frame === "ridge"
? { bd: "border" as const }
: s.frame === "hairline"
? { bd: "rule" as const }
: s.frame === "soft"
? { extra: ["sh-soft"] }
: {};
const scheme = this.darkPalette
? "light dark"
: p === s.dark
? "dark"
: "light";
return `<!doctype html>
<html lang="${EMAIL_LOCALE}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="color-scheme" content="${scheme}">
<meta name="supported-color-schemes" content="${scheme}">
<title>${esc(subject)}</title>
${
s.webFont
? `<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=${s.webFont}&display=swap">`
: ""
}
<style>
body{margin:0;padding:0;}
@media (max-width:620px){.wrap{padding:20px 10px 28px!important}.card{padding:24px 20px 8px!important}}
${this.darkCss()}
</style>
</head>
<body${this.paint({ bg: "outer" }, "margin:0;padding:0;")}>
<div style="display:none;max-height:0;overflow:hidden;opacity:0;">${
esc(preheader)
}</div>
<table role="presentation" width="100%" cellpadding="0" cellspacing="0"${
this.paint({ bg: "outer" })
}>
<tr><td align="center" class="wrap" style="padding:32px 16px 40px;">
<table role="presentation" width="100%" cellpadding="0" cellspacing="0" style="max-width:600px;">
<tr><td style="padding:0 6px 16px;">
<table role="presentation" cellpadding="0" cellspacing="0"><tr>
<td style="padding-right:10px;vertical-align:middle;font-size:24px;line-height:1;">${
esc(SITE_EMOJI)
}</td>
<td${
this.paint(
{ fg: p.brand ? "brand" : "text" },
`vertical-align:middle;font-family:${this.font};font-weight:700;font-size:${
this.fs(21)
};line-height:1;letter-spacing:-0.01em;`,
)
}>${esc(OG_SITE_NAME)}</td>
</tr></table>
</td></tr>
${
bar
? `<tr><td class="grad" style="height:5px;line-height:5px;font-size:0;background-color:${
bar[0]
};background-image:${
gradient(bar)
};border-radius:${R}px ${R}px 0 0;">&nbsp;</td></tr>`
: ""
}
<tr><td${
this.paint(
{
bg: "card",
...frameRoles,
extra: ["card", ...(frameRoles.extra ?? [])],
},
`border-radius:${
bar ? `0 0 ${R}px ${R}px` : `${R}px`
};${frame}padding:34px 36px 12px;`,
)
}>
${body}
</td></tr>
<tr><td${
this.paint(
{ fg: "muted" },
`padding:22px 6px 0;font-family:${this.font};font-size:${
this.fs(12.5)
};line-height:1.6;`,
)
}>${footer}</td></tr>
</table>
</td></tr>
</table>
</body>
</html>`;
}
/**
* Returns the CSS that recolors every painted element in dark mode, or
* nothing off the "auto" scheme.
*/
private darkCss(): string {
const d = this.darkPalette;
if (!d) return "";
const rules: string[] = [];
for (const role of ROLES) {
const color = d[role];
if (!color) continue;
rules.push(
`.f-${role}{color:${color}!important}`,
`.b-${role}{background-color:${color}!important}`,
`.d-${role}{border-color:${color}!important}`,
);
}
rules.push(
`.sh-card{box-shadow:5px 5px 0 ${d.text}!important}`,
`.sh-btn{box-shadow:2px 2px 0 ${d.text}!important}`,
);
if (d.shadow) {
rules.push(
`.sh-soft{box-shadow:0 14px 34px -10px ${d.shadow}!important}`,
);
}
if (d.gradient) {
rules.push(
`.grad{background-color:${d.gradient[0]}!important;background-image:${
gradient(d.gradient)
}!important}`,
);
}
return `@media (prefers-color-scheme:dark){${rules.join("")}}`;
}
}

92
api/mail/password-reset.ts Normal file
View File

@@ -0,0 +1,92 @@
/**
* Renders the email carrying a password reset link. `requestPasswordReset`
* (`password-reset-service.ts`) sends it, in the instance's style and
* language: whoever asks isn't signed in, so their own settings don't apply.
*
* @module
*/
import { OG_SITE_NAME } from "../config.ts";
import { i18n } from "./i18n.ts";
import {
EmailKit,
escMd,
lookOf,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/**
* Returns the reset email of `username`, whose button opens `resetUrl`. The
* link stays usable for `hours`, which the email states.
*/
export function renderPasswordResetEmail(
username: string,
resetUrl: string,
hours: number,
): RenderedEmail {
const markdown = [
`# ${
escMd(i18n._({ id: "email.reset.title", message: "Reset your password" }))
}`,
escMd(
i18n._({
id: "email.reset.hi",
message: "Hi {username},",
values: { username },
}),
),
escMd(i18n._({
id: "email.reset.body",
message:
"Someone asked to reset the password of your {site} account. The button below lets you choose a new one. It works once, for {hours, plural, one {# hour} other {# hours}}.",
values: { site: OG_SITE_NAME, hours },
})),
mdLink(
i18n._({ id: "email.reset.button", message: "Reset password" }),
resetUrl,
),
`*${
escMd(
i18n._({
id: "email.reset.fallback",
message:
"If the button doesn't work, paste this link into your browser:",
}),
)
}*`,
"```\n" + resetUrl + "\n```",
`*${
escMd(
i18n._({
id: "email.reset.ignore",
message:
"If you didn't ask for this, ignore this email. Your password stays the same.",
}),
)
}*`,
"---",
escMd(
i18n._({
id: "email.reset.reason",
message:
"You're getting this email because a password reset was requested for this address.",
}),
),
].join("\n\n");
return new EmailKit(lookOf()).render({
subject: i18n._({
id: "email.reset.subject",
message: "Reset your {site} password",
values: { site: OG_SITE_NAME },
}),
preheader: i18n._({
id: "email.reset.preheader",
message:
"The link works once, for {hours, plural, one {# hour} other {# hours}}.",
values: { hours },
}),
markdown,
});
}

253
api/mail/styles.ts Normal file
View File

@@ -0,0 +1,253 @@
/**
* Carries each site style into email, so a digest looks like the site its
* reader chose and the other emails like the instance's style. A mail client
* takes no stylesheet of ours, so each style is resolved here into plain
* values, which `layout.ts` inlines.
*
* What a theme defines is read from it through `theme-tokens.ts`, never
* copied: the colors, the gradient, the font and the corner radii. What is
* the email's own is stated here: how large it sets its type, how its card
* stands out, which divider it draws, and which of the theme's own tokens
* carry the touches that make it recognisable (Kanagawa's syntax hues,
* GeoCities' rainbow, Melange's spice and warm shadow).
*
* @module
*/
import { STYLE_NAMES, type StyleName } from "../../shared/theme.ts";
import { readTokens, type Scheme } from "./theme-tokens.ts";
/**
* The colors of one style in one scheme, by role, each read from a token of
* its theme. The optional ones carry a style's particular touches.
*/
export interface EmailPalette {
/** The page around the card (`--color-bg`). */
outer: string;
/** The card holding the content (`--color-surface`). */
card: string;
/** `--color-text`. */
text: string;
/** `--color-text-secondary`. */
sec: string;
/** `--color-text-muted`. */
muted: string;
/** Section titles, ranks and button fills (`--color-accent`). */
accent: string;
/** `--color-link`. */
link: string;
/** Text on an accent fill (`--color-on-accent`). */
onAccent: string;
/** Dividers and hairlines (`--color-border`). */
rule: string;
/** Pills and the reset link's box, as the site's badges (`--color-badge-bg`). */
soft: string;
/** Quoted comments and messages (`--color-text-secondary` unless set). */
quote: string;
/** Kanagawa's fujiGray dates; others use `muted`. */
date?: string;
/** Kanagawa's oniViolet wordmark; others use `text`. */
brand?: string;
/** GeoCities' ridged border. */
border?: string;
/** Melange's warm card shadow, translucent. */
shadow?: string;
/** The colors of GeoCities' rainbow or Melange's spice, left to right. */
gradient?: string[];
}
/**
* How a style draws an email. `layout.ts` builds every block from it, so
* adding a style to `shared/theme.ts` means adding its {@link Recipe} here.
*/
export interface EmailStyle {
/**
* The theme's font stack (`--font-body`). Its first family is a web font
* only Apple's mail apps load, through {@link webFont}; other clients fall
* back along the stack.
*/
font: string;
/** The Google Fonts `family` parameter of the web font, if it is one. */
webFont?: string;
/** Multiplies every font size: Times sets its serif larger, Kanagawa its monospace smaller. */
scale: number;
/**
* Corner radii, in pixels: the card's as the site's cards
* (`--radius-xl`), the images' as its thumbnails (`--radius-md`), the
* buttons' as its buttons (`--radius-lg`), the pills' as its badges
* (`--radius-sm`). The avatars' is the email's own, since the site rounds
* them with a literal 50% that some styles square.
*/
radius: {
card: number;
img: number;
btn: number;
pill: number;
avatar: number;
};
/** The heading: its size in pixels, weight, and Brutalist's uppercase and spacing. */
title: {
size: number;
weight: number;
upper?: boolean;
spacing?: string;
accent?: boolean;
};
/** The weight of titles of items, figures and names. */
heavy: number;
/** How the card stands out from the page. */
frame: "none" | "hard" | "ridge" | "hairline" | "soft";
/** The divider between sections; "gradient" draws the palette's. */
rule: "hairline" | "thick" | "gradient";
/** Draws the palette's gradient as a bar along the card's top. */
bar?: boolean;
/** Underlines links, for styles whose links are the color of the text. */
linkUnderline?: boolean;
light: EmailPalette;
dark: EmailPalette;
}
/** The roles every style reads, and the token each is read from. */
const PALETTE_TOKENS = {
outer: "--color-bg",
card: "--color-surface",
text: "--color-text",
sec: "--color-text-secondary",
muted: "--color-text-muted",
accent: "--color-accent",
link: "--color-link",
onAccent: "--color-on-accent",
rule: "--color-border",
soft: "--color-badge-bg",
quote: "--color-text-secondary",
} satisfies Partial<Record<keyof EmailPalette, string>>;
/**
* What a style states about its email, beside what its theme defines: the
* {@link EmailStyle} less the values read from the theme, the axes of its
* web font, its avatars' radius, and the tokens of its own touches.
*/
interface Recipe
extends Omit<EmailStyle, "font" | "webFont" | "radius" | "light" | "dark"> {
/** The Google Fonts axes of the theme's first family; none for a system font. */
webFontAxes?: string;
avatarRadius: number;
/** The tokens of the palette's optional roles, or of a base role it reads elsewhere. */
tokens?: Partial<Record<keyof EmailPalette, string>>;
}
const RECIPES: Record<StyleName, Recipe> = {
smooth: {
webFontAxes: "wght@400;600;700",
scale: 1,
avatarRadius: 16,
title: { size: 30, weight: 600 },
heavy: 600,
frame: "none",
rule: "hairline",
},
brutalist: {
webFontAxes: "wght@400;700",
scale: 1,
avatarRadius: 0,
title: { size: 30, weight: 900, upper: true, spacing: "0.03em" },
heavy: 700,
frame: "hard",
rule: "thick",
},
geocities: {
scale: 1,
avatarRadius: 0,
title: { size: 30, weight: 700, accent: true },
heavy: 700,
frame: "ridge",
rule: "gradient",
bar: true,
linkUnderline: true,
tokens: { border: "--color-border", gradient: "--geo-rainbow" },
},
nyt: {
webFontAxes: "ital@0;1",
scale: 1.12,
avatarRadius: 16,
title: { size: 38, weight: 400 },
heavy: 400,
frame: "hairline",
rule: "hairline",
linkUnderline: true,
},
kanagawa: {
webFontAxes: "ital,wght@0,400;0,600;1,400",
scale: 0.9,
avatarRadius: 4,
title: { size: 26, weight: 600 },
heavy: 600,
frame: "hairline",
rule: "hairline",
tokens: {
quote: "--kngw-string",
date: "--kngw-comment",
brand: "--kngw-keyword",
},
},
melange: {
webFontAxes: "wght@400;600",
scale: 1,
avatarRadius: 16,
title: { size: 32, weight: 600 },
heavy: 600,
frame: "soft",
rule: "hairline",
bar: true,
tokens: { shadow: "--shadow-color", gradient: "--mel-spice" },
},
};
/**
* Returns the palette of `style` in `scheme` from its theme's tokens. Every
* color is flattened onto the card, as a translucent one shows on the site,
* but the shadow, which falls on the page.
*/
function readPalette(
style: StyleName,
scheme: Scheme,
own: Recipe["tokens"] = {},
): EmailPalette {
const tokens = readTokens(style, scheme);
const roles = { ...PALETTE_TOKENS, ...own };
const palette = Object.fromEntries(
Object.entries(roles)
.filter(([role]) => role !== "shadow" && role !== "gradient")
.map(([role, token]) => [role, tokens.color(token, "--color-surface")]),
) as unknown as EmailPalette;
if (own.shadow) palette.shadow = tokens.color(own.shadow);
if (own.gradient) palette.gradient = tokens.stops(own.gradient);
return palette;
}
/** Returns the email look of `style`, its theme read in both schemes. */
function readStyle(style: StyleName): EmailStyle {
const { webFontAxes, avatarRadius, tokens: own, ...recipe } = RECIPES[style];
const tokens = readTokens(style, "light");
const font = tokens.font("--font-body");
const family = font.split(",")[0].replace(/'/g, "").replaceAll(" ", "+");
return {
...recipe,
font,
webFont: webFontAxes ? `${family}:${webFontAxes}` : undefined,
radius: {
card: tokens.px("--radius-xl"),
img: tokens.px("--radius-md"),
btn: tokens.px("--radius-lg"),
pill: tokens.px("--radius-sm"),
avatar: avatarRadius,
},
light: readPalette(style, "light", own),
dark: readPalette(style, "dark", own),
};
}
/** Every style's email look, keyed by its name, read once at load. */
export const EMAIL_STYLES = Object.fromEntries(
STYLE_NAMES.map((style) => [style, readStyle(style)]),
) as Record<StyleName, EmailStyle>;

276
api/mail/theme-tokens.ts Normal file
View File

@@ -0,0 +1,276 @@
/**
* Lets the emails draw with the site's own values. It reads a style's design
* tokens out of the stylesheets the client loads, `src/tokens.css` and then
* `src/themes/<style>.css`, so `styles.ts` names a token wherever it needs a
* color, a gradient, a font or a radius, and a change to a theme reaches its
* emails with nothing copied to keep in step.
*
* Mail clients know no custom properties, so a token comes back resolved for
* one scheme: its `var()`s followed, its `light-dark()` picked, its
* `color-mix()` computed. Only the rules that set tokens on the root element
* are read (`:root`, the style's `[data-style]`, and either of those with the
* scheme's `[data-color-scheme]`); a theme states its forced schemes that way
* beside each `prefers-color-scheme` block, so the media blocks are skipped.
*
* @module
*/
import { BASE_STYLE, type StyleName } from "../../shared/theme.ts";
import { hexOf, parseCssColor, type Rgba } from "../lib/css-color.ts";
/** A scheme a token is resolved for. "auto" is drawn from both. */
export type Scheme = "light" | "dark";
/**
* Returned by {@link readTokens}: the tokens of one style in one scheme, read
* as the kind of value each is.
*/
export interface ThemeTokens {
/** Returns the resolved value of `token`, or throws when it isn't set. */
value(token: string): string;
/**
* Returns the color of `token` as `#rrggbb`. A translucent one is laid over
* `over` (another token) first, as it shows on the site; without `over` it
* stays `rgba()`, as a shadow's must.
*/
color(token: string, over?: string): string;
/** Returns the colors of the gradient `token`, in order, without positions. */
stops(token: string): string[];
/**
* Returns the font stack of `token`, single-quoted for a `style` attribute.
* A family the site bundles as "X Variable" is named "X", as a mail client
* or Google Fonts knows it.
*/
font(token: string): string;
/** Returns the length of `token` in pixels. */
px(token: string): number;
}
const SOURCES = ["../../src/tokens.css", "../../src/themes/{style}.css"];
/** Returns `css` without its comments. */
function stripComments(css: string): string {
return css.replace(/\/\*[\s\S]*?\*\//g, "");
}
/** Returns the top-level rules of `css` as their selector and body. */
function topRules(css: string): { selector: string; body: string }[] {
const rules = [];
let i = 0;
while (i < css.length) {
const open = css.indexOf("{", i);
if (open < 0) break;
let depth = 1;
let j = open + 1;
while (depth && j < css.length) {
if (css[j] === "{") depth++;
else if (css[j] === "}") depth--;
j++;
}
rules.push({
selector: css.slice(i, open).trim(),
body: css.slice(open + 1, j - 1),
});
i = j;
}
return rules;
}
/**
* Splits `text` at each `sep` outside parentheses and quotes, trimming the
* parts: the arguments of a CSS function, or the declarations of a rule.
*/
function splitTop(text: string, sep: string): string[] {
const parts: string[] = [];
let depth = 0;
let quote = "";
let start = 0;
for (let i = 0; i < text.length; i++) {
const c = text[i];
if (quote) {
if (c === quote) quote = "";
} else if (c === '"' || c === "'") quote = c;
else if (c === "(") depth++;
else if (c === ")") depth--;
else if (c === sep && depth === 0) {
parts.push(text.slice(start, i).trim());
start = i + 1;
}
}
parts.push(text.slice(start).trim());
return parts.filter((part) => part !== "");
}
/**
* Replaces each call of the CSS function `name` in `value`, innermost last
* first, with what `replace` returns for its arguments.
*/
function replaceCalls(
value: string,
name: string,
replace: (args: string[]) => string,
): string {
for (;;) {
const at = value.lastIndexOf(`${name}(`);
if (at < 0) return value;
let depth = 0;
let end = at + name.length;
do {
if (value[end] === "(") depth++;
else if (value[end] === ")") depth--;
end++;
} while (depth && end < value.length);
const args = splitTop(value.slice(at + name.length + 1, end - 1), ",");
value = value.slice(0, at) + replace(args) + value.slice(end);
}
}
/** Parses a CSS color in the forms the themes write, or throws. */
function parseColor(text: string): Rgba {
const color = parseCssColor(text);
if (!color) throw new Error(`Unreadable color: ${text}`);
return color;
}
/** Writes `color` as `#rrggbb`, or as `rgba()` when it is translucent. */
function formatColor(color: Rgba): string {
if (color.a >= 1) return hexOf(color);
const [r, g, b] = [color.r, color.g, color.b].map(Math.round);
return `rgba(${r},${g},${b},${+color.a.toFixed(3)})`;
}
/** Returns `top` laid over `under`, both straight (not premultiplied) alpha. */
function over(top: Rgba, under: Rgba): Rgba {
const a = top.a + under.a * (1 - top.a);
if (a === 0) return { r: 0, g: 0, b: 0, a: 0 };
const mix = (t: number, u: number) =>
(t * top.a + u * under.a * (1 - top.a)) / a;
return {
r: mix(top.r, under.r),
g: mix(top.g, under.g),
b: mix(top.b, under.b),
a,
};
}
/**
* Computes `color-mix(in srgb, …)` from its arguments, the two colors with
* their optional percentages, as CSS does: premultiplied, and scaled down
* when the percentages sum under 100%.
*/
function colorMix(args: string[]): string {
const [space, ...colors] = args;
if (space !== "in srgb" || colors.length !== 2) {
throw new Error(`Unsupported color-mix: ${args.join(", ")}`);
}
const parsed = colors.map((arg) => {
const pct = arg.match(/\s([\d.]+)%$/);
return {
color: parseColor(pct ? arg.slice(0, pct.index) : arg),
pct: pct ? parseFloat(pct[1]) / 100 : null,
};
});
const [p1, p2] = [
parsed[0].pct ?? 1 - (parsed[1].pct ?? 0.5),
parsed[1].pct ?? 1 - (parsed[0].pct ?? 0.5),
];
const sum = p1 + p2;
const [w1, w2] = [p1 / sum, p2 / sum];
const [c1, c2] = [parsed[0].color, parsed[1].color];
const a = c1.a * w1 + c2.a * w2;
const mix = (x: number, y: number) =>
a === 0 ? 0 : (x * c1.a * w1 + y * c2.a * w2) / a;
return formatColor({
r: mix(c1.r, c2.r),
g: mix(c1.g, c2.g),
b: mix(c1.b, c2.b),
a: a * Math.min(sum, 1),
});
}
/**
* Returns the tokens of `style` in `scheme`, read from the site's stylesheets
* at the time of the call. `styles.ts` reads each style once, at load.
*/
export function readTokens(style: StyleName, scheme: Scheme): ThemeTokens {
const styleSel = style === BASE_STYLE
? `(?:\\[data-style="${style}"\\]|:not\\(\\[data-style\\]\\))`
: `\\[data-style="${style}"\\]`;
const rootRule = new RegExp(
`^(?::root)?(?:${styleSel})?(?:\\[data-color-scheme="${scheme}"\\])?$`,
);
// Later declarations win, as in the cascade: the base tokens, then the
// theme's, then its forced scheme's.
const declared = new Map<string, string>();
for (const source of SOURCES) {
const css = stripComments(Deno.readTextFileSync(
new URL(source.replace("{style}", style), import.meta.url),
));
for (const { selector, body } of topRules(css)) {
if (selector.startsWith("@")) continue;
if (!selector.split(",").some((part) => rootRule.test(part.trim()))) {
continue;
}
for (const declaration of splitTop(body, ";")) {
const colon = declaration.indexOf(":");
const name = declaration.slice(0, colon).trim();
if (!name.startsWith("--")) continue;
declared.set(
name,
declaration.slice(colon + 1).replace(/\s+/g, " ").trim(),
);
}
}
}
const resolved = new Map<string, string>();
const value = (token: string): string => {
const known = resolved.get(token);
if (known !== undefined) return known;
const raw = declared.get(token);
if (raw === undefined) throw new Error(`${style} sets no ${token}`);
let result = replaceCalls(
raw,
"var",
([name, fallback]) =>
declared.has(name) ? value(name) : fallback ?? value(name),
);
result = replaceCalls(
result,
"light-dark",
([light, dark]) => scheme === "light" ? light : dark,
);
result = replaceCalls(result, "color-mix", colorMix);
resolved.set(token, result);
return result;
};
return {
value,
color(token, under) {
const color = parseColor(value(token));
return formatColor(under ? over(color, parseColor(value(under))) : color);
},
stops(token) {
const gradient = value(token).match(/^linear-gradient\((.*)\)$/)?.[1];
if (!gradient) throw new Error(`${style} ${token} is no linear gradient`);
return splitTop(gradient, ",")
.filter((stop) => !/^(?:to\s|[\d.-]+(?:deg|turn|rad)$)/.test(stop))
.map((stop) => formatColor(parseColor(splitTop(stop, " ")[0])));
},
font(token) {
return splitTop(value(token), ",")
.map((family) => family.replace(/["']/g, "").replace(/ Variable$/, ""))
.map((family) => /^[\w-]+$/.test(family) ? family : `'${family}'`)
.join(", ");
},
px(token) {
const length = value(token).match(/^([\d.]+)(?:px)?$/)?.[1];
if (length === undefined) {
throw new Error(`${style} ${token} is no pixel length`);
}
return parseFloat(length);
},
};
}

80
api/mail/welcome.ts Normal file
View File

@@ -0,0 +1,80 @@
/**
* Renders the email that greets a new account. `sendWelcomeEmail`
* (`user-service.ts`) sends it after sign-up, in the instance's style and
* language. It opens with the instance's `GERBEUR_WELCOME_EMAIL_BODY`, or the
* built-in body, then says the digests are on, since they are by default, and
* where to turn them off.
*
* @module
*/
import { FRONTEND_URL, OG_SITE_NAME, WELCOME_EMAIL_BODY } from "../config.ts";
import { i18n } from "./i18n.ts";
import {
EmailKit,
escMd,
lookOf,
mdLink,
type RenderedEmail,
} from "./layout.ts";
/**
* Returns the welcome email of `username`. The instance's body has
* `{{username}}` and `{{site_name}}` filled in; raw HTML in it passes
* through, since it comes from whoever runs the instance, never from a user.
*/
export function renderWelcomeEmail(username: string): RenderedEmail {
// A username may hold a `_`, which Markdown would read as emphasis.
const body = WELCOME_EMAIL_BODY
? WELCOME_EMAIL_BODY
.replaceAll("{{username}}", escMd(username))
.replaceAll("{{site_name}}", escMd(OG_SITE_NAME))
: i18n._({
id: "email.welcome.body",
message:
"# Welcome to {site}!\n\nHi **{username}**,\n\nYour account has been created successfully. Welcome aboard!",
values: { site: escMd(OG_SITE_NAME), username: escMd(username) },
});
const markdown = [
body,
mdLink(
i18n._({
id: "email.welcome.button",
message: "Open {site}",
values: { site: OG_SITE_NAME },
}),
FRONTEND_URL,
),
`*${
escMd(
i18n._({
id: "email.welcome.digests",
message:
"Once a week you'll get a digest of what happened here, and once a month a recap of the most upvoted dumps. You can turn either off in your settings.",
}),
)
}*`,
"---",
escMd(
i18n._({
id: "email.welcome.reason",
message:
"You're getting this email because an account was created on {site} with this address.",
values: { site: OG_SITE_NAME },
}),
),
].join("\n\n");
return new EmailKit(lookOf()).render({
subject: i18n._({
id: "email.welcome.subject",
message: "Welcome to {site}",
values: { site: OG_SITE_NAME },
}),
preheader: i18n._({
id: "email.welcome.preheader",
message: "Your account is ready.",
}),
markdown,
});
}

191
api/main.ts
View File

@@ -1,7 +1,12 @@
/** /**
* The server's entry point: builds the Oak application — error and OG * Builds the Oak application and, when run directly, starts the recurring
* middleware, CORS, every router, then the static frontend — and listens on * jobs and listens on {@link LISTEN_HOST}:{@link PORT}, having first signed
* {@link LISTEN_HOST}:{@link PORT} when run directly. * the users out of other instances if they may no longer sign in there
* (`FEDERATION.outgoing`). A request goes
* through the error middleware, the compression of what is answered, the OG
* tags and CORS, then the routers, one
* per resource of the API (`API.md` lists them), and last the built frontend,
* which answers every path no router did.
* *
* @module * @module
*/ */
@@ -10,39 +15,67 @@ import { Application } from "@oak/oak";
import { oakCors } from "@tajpouria/cors"; import { oakCors } from "@tajpouria/cors";
import attachmentsRouter from "./routes/attachments.ts"; import attachmentsRouter from "./routes/attachments.ts";
import avatarsRouter from "./routes/avatars.ts";
import categoriesRouter from "./routes/categories.ts";
import chatRouter from "./routes/chat.ts";
import communityRouter from "./routes/community.ts";
import commentsRouter from "./routes/comments.ts";
import dumpsRouter from "./routes/dumps.ts"; import dumpsRouter from "./routes/dumps.ts";
import filesRouter from "./routes/files.ts"; import filesRouter from "./routes/files.ts";
import thumbnailsRouter from "./routes/thumbnails.ts"; import federationRouter from "./routes/federation.ts";
import usersRouter from "./routes/users.ts";
import avatarsRouter from "./routes/avatars.ts";
import wsRouter from "./routes/ws.ts";
import previewRouter from "./routes/preview.ts";
import bandcampRouter from "./routes/bandcamp.ts";
import youtubeRouter from "./routes/youtube.ts";
import playlistsRouter from "./routes/playlists.ts";
import commentsRouter from "./routes/comments.ts";
import chatRouter from "./routes/chat.ts";
import followsRouter from "./routes/follows.ts"; import followsRouter from "./routes/follows.ts";
import notificationsRouter from "./routes/notifications.ts";
import invitesRouter from "./routes/invites.ts"; import invitesRouter from "./routes/invites.ts";
import meRouter from "./routes/me.ts";
import notificationsRouter from "./routes/notifications.ts";
import oauthRouter from "./routes/oauth.ts";
import passwordResetsRouter from "./routes/password-resets.ts";
import playlistsRouter from "./routes/playlists.ts";
import remoteRouter from "./routes/remote.ts";
import searchRouter from "./routes/search.ts"; import searchRouter from "./routes/search.ts";
import categoriesRouter from "./routes/categories.ts"; import sessionsRouter from "./routes/sessions.ts";
import thumbnailsRouter from "./routes/thumbnails.ts";
import waveformsRouter from "./routes/waveforms.ts";
import unsubscribeRouter from "./routes/unsubscribe.ts";
import usersRouter from "./routes/users.ts";
import wellKnownRouter from "./routes/well-known.ts";
import wsRouter from "./routes/ws.ts";
import { ALLOWED_ORIGINS, LISTEN_HOST, PORT, PUBLIC_URL } from "./config.ts"; import {
ALLOWED_ORIGINS,
FEDERATION,
LISTEN_HOST,
PORT,
PUBLIC_URL,
} from "./config.ts";
import { compressMiddleware } from "./middleware/compress.ts";
import { errorMiddleware } from "./middleware/error.ts"; import { errorMiddleware } from "./middleware/error.ts";
import { ogMiddleware } from "./middleware/og.ts"; import { ogMiddleware } from "./middleware/og.ts";
import { routeStaticFilesFrom } from "./lib/static.ts"; import { routeStaticFilesFrom } from "./lib/static.ts";
import { startJobs } from "./lib/scheduler.ts";
import { rateLimitSweepJob } from "./lib/rate-limit.ts";
import { generatedMediaSweepJob } from "./lib/ffmpeg.ts";
import { digestJob } from "./services/digest-service.ts";
import { communityJob } from "./services/community-service.ts";
import { pingJob, typingExpiryJob } from "./services/ws-service.ts";
import {
outboxJob,
withdrawAllGrants,
} from "./services/identity-provider-service.ts";
/** The application, exported so the tests can drive it without listening. */ /**
* The whole server, exported so the tests' helpers can send it requests
* without listening.
*/
const app: Application = new Application(); const app: Application = new Application();
const cors = oakCors({ origin: ALLOWED_ORIGINS }); const cors = oakCors({ origin: ALLOWED_ORIGINS });
app.use(errorMiddleware); app.use(errorMiddleware);
app.use(compressMiddleware);
app.use(ogMiddleware); app.use(ogMiddleware);
// Only invoke oakCors when the origin is present and allowed. // oakCors calls next() without awaiting it when there is no origin, which
// The library calls next() without await when origin is falsy, which // corrupts the response in Oak's async chain. So it only sees requests from an
// corrupts the response in Oak's async middleware chain. // allowed origin, and the others get no CORS headers.
app.use(async (ctx, next) => { app.use(async (ctx, next) => {
const origin = ctx.request.headers.get("origin"); const origin = ctx.request.headers.get("origin");
if (origin && ALLOWED_ORIGINS.includes(origin)) { if (origin && ALLOWED_ORIGINS.includes(origin)) {
@@ -50,78 +83,37 @@ app.use(async (ctx, next) => {
} }
await next(); await next();
}); });
app.use( for (
dumpsRouter.routes(), const router of [
dumpsRouter.allowedMethods(), meRouter,
); followsRouter,
app.use( notificationsRouter,
filesRouter.routes(), avatarsRouter,
filesRouter.allowedMethods(), usersRouter,
); sessionsRouter,
app.use( passwordResetsRouter,
thumbnailsRouter.routes(), unsubscribeRouter,
thumbnailsRouter.allowedMethods(), dumpsRouter,
); filesRouter,
app.use( thumbnailsRouter,
attachmentsRouter.routes(), waveformsRouter,
attachmentsRouter.allowedMethods(), commentsRouter,
); playlistsRouter,
app.use( categoriesRouter,
usersRouter.routes(), searchRouter,
usersRouter.allowedMethods(), chatRouter,
); communityRouter,
app.use( invitesRouter,
avatarsRouter.routes(), attachmentsRouter,
avatarsRouter.allowedMethods(), remoteRouter,
); federationRouter,
app.use( oauthRouter,
wsRouter.routes(), wellKnownRouter,
wsRouter.allowedMethods(), wsRouter,
); ]
app.use( ) {
previewRouter.routes(), app.use(router.routes(), router.allowedMethods());
previewRouter.allowedMethods(), }
);
app.use(
bandcampRouter.routes(),
bandcampRouter.allowedMethods(),
);
app.use(
youtubeRouter.routes(),
youtubeRouter.allowedMethods(),
);
app.use(
playlistsRouter.routes(),
playlistsRouter.allowedMethods(),
);
app.use(
commentsRouter.routes(),
commentsRouter.allowedMethods(),
);
app.use(
chatRouter.routes(),
chatRouter.allowedMethods(),
);
app.use(
followsRouter.routes(),
followsRouter.allowedMethods(),
);
app.use(
notificationsRouter.routes(),
notificationsRouter.allowedMethods(),
);
app.use(
invitesRouter.routes(),
invitesRouter.allowedMethods(),
);
app.use(
searchRouter.routes(),
searchRouter.allowedMethods(),
);
app.use(
categoriesRouter.routes(),
categoriesRouter.allowedMethods(),
);
app.use(routeStaticFilesFrom([ app.use(routeStaticFilesFrom([
`${Deno.cwd()}/dist`, `${Deno.cwd()}/dist`,
`${Deno.cwd()}/public`, `${Deno.cwd()}/public`,
@@ -138,6 +130,21 @@ app.addEventListener(
); );
if (import.meta.main) { if (import.meta.main) {
if (!FEDERATION.outgoing) {
const withdrawn = await withdrawAllGrants();
if (withdrawn > 0) {
console.log(`Signed users out of other instances (${withdrawn} grants)`);
}
}
startJobs([
digestJob,
communityJob,
pingJob,
typingExpiryJob,
rateLimitSweepJob,
generatedMediaSweepJob,
outboxJob,
]);
await app.listen({ hostname: LISTEN_HOST, port: PORT }); await app.listen({ hostname: LISTEN_HOST, port: PORT });
} }

66
api/middleware/auth.ts
View File

@@ -1,6 +1,7 @@
/** /**
* Middleware gating routes on authentication, the admin role, or a * Gates routes on a session token, and on a permission after it. The routers
* permission. * put these middleware before a handler; a route open to anonymous callers
* calls `parseOptionalAuth` (`lib/auth.ts`) instead.
* *
* @module * @module
*/ */
@@ -12,22 +13,22 @@ import {
type State, type State,
} from "@oak/oak"; } from "@oak/oak";
import { authenticate } from "../lib/auth.ts"; import { authenticate } from "../lib/auth.ts";
import { APIErrorCode } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts"; import { APIException } from "../lib/api-exception.ts";
import type { AuthPayload } from "../model/auth.ts"; import type { AuthPayload } from "../model/auth.ts";
import { can } from "../lib/permissions.ts"; import { can } from "../lib/permissions.ts";
import type { Permission } from "../../shared/permissions.ts"; import type { Permission } from "../../shared/permissions.ts";
/** The request state once {@link authMiddleware} has run. */ /**
* Set on `ctx.state` by {@link authMiddleware} for the handlers after it.
* `user` carries the caller's live role, not the one in their token.
*/
export interface AuthState extends State { export interface AuthState extends State {
/** The authenticated caller, with their live role. */
user: AuthPayload; user: AuthPayload;
} }
/** /**
* A router context whose state is guaranteed to carry the authenticated user * The context type of a handler behind {@link authMiddleware}. Pass the route
* (populated by {@link authMiddleware}). Generic over the route path so * path, as in `AuthContext<"/:key/role">`, to keep `ctx.params` typed.
* `ctx.params` stays typed per-route — e.g. `AuthContext<"/:userId/role">`.
*/ */
export type AuthContext< export type AuthContext<
R extends string = string, R extends string = string,
@@ -35,8 +36,8 @@ export type AuthContext<
> = RouterContext<R, P, AuthState>; > = RouterContext<R, P, AuthState>;
/** /**
* Requires a valid session token in the `Authorization` header and puts its * Requires a valid session token in the `Authorization` header, and puts its
* caller in `ctx.state.user`. * payload in `ctx.state.user` for the rest of the route.
* *
* @throws {APIException} 401 when the token is missing or invalid. * @throws {APIException} 401 when the token is missing or invalid.
*/ */
@@ -47,20 +48,14 @@ export async function authMiddleware<R extends string>(
const authHeader = ctx.request.headers.get("Authorization"); const authHeader = ctx.request.headers.get("Authorization");
if (!authHeader || !authHeader.startsWith("Bearer ")) { if (!authHeader || !authHeader.startsWith("Bearer ")) {
throw new APIException( throw new APIException(401, "Missing or invalid token");
APIErrorCode.UNAUTHORIZED,
401,
"Missing or invalid token",
);
} }
const token = authHeader.substring(7); const token = authHeader.substring(7);
// Also checks the user still exists and hasn't changed their password since
// the token was issued, and takes the live role from the DB.
const payload = await authenticate(token); const payload = await authenticate(token);
if (!payload) { if (!payload) {
throw new APIException(APIErrorCode.UNAUTHORIZED, 401, "Invalid token"); throw new APIException(401, "Invalid token");
} }
ctx.state.user = payload; ctx.state.user = payload;
@@ -69,30 +64,11 @@ export async function authMiddleware<R extends string>(
} }
/** /**
* Requires the caller to be an admin. Must run after {@link authMiddleware}. * Returns a middleware that requires the caller to hold `permission`. It must
* * run after {@link authMiddleware}, as in
* @throws {APIException} 403 when the caller isn't an admin. * `router.patch(path, authMiddleware, requirePermission("user:manage"), …)`.
*/ * The returned middleware throws an {@link APIException} (403) when the caller
export async function adminOnlyMiddleware<R extends string>( * lacks the permission.
ctx: AuthContext<R>,
next: Next,
): Promise<void> {
if (ctx.state.user.role !== "admin") {
throw new APIException(
APIErrorCode.UNAUTHORIZED,
403,
"Admin access required",
);
}
await next();
}
/**
* Returns a middleware requiring the caller to hold `permission`, which throws
* an {@link APIException} (403) when they don't. Must run after
* {@link authMiddleware}. Generic over the route path so it composes with
* path-typed route handlers.
*/ */
export function requirePermission( export function requirePermission(
permission: Permission, permission: Permission,
@@ -102,11 +78,7 @@ export function requirePermission(
next: Next, next: Next,
): Promise<void> => { ): Promise<void> => {
if (!ctx.state.user || !can(ctx.state.user, permission)) { if (!ctx.state.user || !can(ctx.state.user, permission)) {
throw new APIException( throw new APIException(403, "Insufficient permissions");
APIErrorCode.UNAUTHORIZED,
403,
"Insufficient permissions",
);
} }
await next(); await next();
}; };

108
api/middleware/compress.ts Normal file
View File

@@ -0,0 +1,108 @@
/**
* Compresses what the routes answer in JSON, and the HTML page, for the
* browsers that accept it: a page of the feed is some twenty kilobytes as it
* is, and a fifth of that compressed. main.ts installs it after the error
* middleware, around everything that sets a body. The built frontend's files
* aren't its concern: lib/static.ts sends the copies compressed at build time.
*
* @module
*/
import type { Context, Next } from "@oak/oak";
import { promisify } from "node:util";
import { brotliCompress, constants as zlib, gzip } from "node:zlib";
const brotli = promisify(brotliCompress);
const gz = promisify(gzip);
/**
* Below this, the bytes saved aren't worth the work; it is the threshold the
* build compresses the frontend's files from (vite.config.ts).
*/
const MIN_BYTES = 1024;
/**
* Brotli's quality for a response made on the spot. Past 5 it takes several
* times longer for a few percent; the frontend's files, compressed once, are
* at 11.
*/
const BROTLI_QUALITY = 5;
/**
* Returns the coding to answer with for an `Accept-Encoding` header, brotli
* before gzip, or null when the client takes neither. A coding refused with
* `q=0` isn't taken.
*/
function pickCoding(header: string | null): "br" | "gzip" | null {
if (!header) return null;
const accepted = new Set<string>();
for (const part of header.split(",")) {
const [name, ...params] = part.trim().toLowerCase().split(";");
const q = params.map((p) => p.trim().match(/^q=([\d.]+)$/)?.[1])
.find((v) => v !== undefined);
if (q === undefined || Number(q) > 0) accepted.add(name.trim());
}
return accepted.has("br") ? "br" : accepted.has("gzip") ? "gzip" : null;
}
/**
* Returns the text of a body this middleware compresses, or null for any
* other: an object or array, which Oak would send as JSON, or a string whose
* type its route has set. Files, streams and bytes are left alone, as they
* are images, media and files compressed already.
*/
function textOf(ctx: Context): string | null {
const body = ctx.response.body;
if (typeof body === "string") {
// Oak guesses an untyped string's type from the string itself, which it
// couldn't from its compressed bytes.
return ctx.response.type || ctx.response.headers.has("Content-Type")
? body
: null;
}
if (body === null || typeof body !== "object") return null;
const proto = Object.getPrototypeOf(body);
if (proto !== Object.prototype && !Array.isArray(body)) return null;
ctx.response.type = "json";
return JSON.stringify(body);
}
/**
* Sends the response's JSON or text compressed, in brotli or gzip as the
* request accepts, when it is at least {@link MIN_BYTES} long. A response
* already encoded, a `HEAD`, and one marked `no-transform` go out as they are.
*/
export async function compressMiddleware(
ctx: Context,
next: Next,
): Promise<void> {
await next();
const headers = ctx.response.headers;
if (
ctx.request.method === "HEAD" || headers.has("Content-Encoding") ||
headers.get("Cache-Control")?.includes("no-transform")
) return;
const text = textOf(ctx);
if (text === null) return;
const raw = new TextEncoder().encode(text);
if (raw.length < MIN_BYTES) return;
// A cache must not hand the compressed copy to a client that can't read it.
headers.append("Vary", "Accept-Encoding");
const coding = pickCoding(ctx.request.headers.get("Accept-Encoding"));
if (!coding) return;
const compressed = coding === "br"
? await brotli(raw, {
params: {
[zlib.BROTLI_PARAM_MODE]: zlib.BROTLI_MODE_TEXT,
[zlib.BROTLI_PARAM_QUALITY]: BROTLI_QUALITY,
[zlib.BROTLI_PARAM_SIZE_HINT]: raw.length,
},
})
: await gz(raw);
headers.set("Content-Encoding", coding);
ctx.response.body = new Uint8Array(compressed);
}

9
api/middleware/error.ts
View File

@@ -1,5 +1,7 @@
/** /**
* Turns errors thrown while handling a request into `APIFailure` responses. * Turns the errors a request's handling throws into `APIFailure` responses.
* main.ts installs it first, so it catches what every later middleware and
* route throws.
* *
* @module * @module
*/ */
@@ -10,8 +12,9 @@ import { APIErrorCode, APIFailure } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts"; import { APIException } from "../lib/api-exception.ts";
/** /**
* Answers an {@link APIException} with its status and code, and anything else * Answers an {@link APIException} with its status, code and message. Anything
* with a 500 `SERVER_ERROR`, logging both. * else is logged with its stack and answered with a 500 `SERVER_ERROR` whose
* message gives nothing away.
*/ */
export async function errorMiddleware( export async function errorMiddleware(
ctx: Context, ctx: Context,

113
api/middleware/og.ts
View File

@@ -1,16 +1,24 @@
/** /**
* Open Graph and Twitter card tags for shared links: a dump, user or public * Gives links to the site rich previews in chat apps and social sites, which
* playlist page is answered with `index.html` carrying that page's title, * don't run the app. It answers a link to a dump, a profile or a public
* description and image, so link unfurlers that don't run the app see them. * playlist with `index.html` carrying that page's Open Graph and Twitter card
* tags, and a link by a former name with a 301 to the current one. main.ts
* installs it before the routers.
* *
* @module * @module
*/ */
import { Context, Next } from "@oak/oak"; import { Context, Next } from "@oak/oak";
import { getDump } from "../services/dump-service.ts"; import { getDumpByPath } from "../services/dump-service.ts";
import { getUserByUsername } from "../services/user-service.ts"; import { resolveUsername } from "../services/user-service.ts";
import { getPlaylistById } from "../services/playlist-service.ts"; import { getPlaylistByPath } from "../services/playlist-service.ts";
import { OG_SITE_NAME } from "../config.ts"; import {
dumpPath,
parsePagePath,
playlistPath,
userPath,
} from "../../shared/paths.ts";
import { FRONTEND_URL, OG_SITE_NAME, PUBLIC_URL } from "../config.ts";
import { loadIndexHtml } from "../lib/index-html.ts"; import { loadIndexHtml } from "../lib/index-html.ts";
interface OGMeta { interface OGMeta {
@@ -18,8 +26,8 @@ interface OGMeta {
description?: string; description?: string;
imageUrl?: string; imageUrl?: string;
/** /**
* True only for real artwork — a square site icon must not claim a wide * Set for a dump's artwork only. It asks for the large card, which would
* card. * stretch a square icon or avatar.
*/ */
imageIsWide?: boolean; imageIsWide?: boolean;
url: string; url: string;
@@ -39,7 +47,7 @@ function buildTags(meta: OGMeta): string {
: "summary"; : "summary";
const tags = [ const tags = [
`<title>${escapeAttr(meta.title)}</title>`, `<title>${escapeAttr(meta.title)}</title>`,
`<meta property="og:site_name" content="${OG_SITE_NAME}" />`, `<meta property="og:site_name" content="${escapeAttr(OG_SITE_NAME)}" />`,
`<meta property="og:type" content="website" />`, `<meta property="og:type" content="website" />`,
`<meta property="og:url" content="${escapeAttr(meta.url)}" />`, `<meta property="og:url" content="${escapeAttr(meta.url)}" />`,
`<meta property="og:title" content="${escapeAttr(meta.title)}" />`, `<meta property="og:title" content="${escapeAttr(meta.title)}" />`,
@@ -68,8 +76,8 @@ function inject(html: string, meta: OGMeta): string {
} }
/** /**
* Returns the filled `index.html` of the build, or of the working directory * Returns the filled `index.html` of dist/, or else the source one, which is
* when there is no build; null when there is neither. * all there is in development. Returns null when neither can be read.
*/ */
async function findIndexHtml(): Promise<string | null> { async function findIndexHtml(): Promise<string | null> {
for (const root of [`${Deno.cwd()}/dist`, Deno.cwd()]) { for (const root of [`${Deno.cwd()}/dist`, Deno.cwd()]) {
@@ -82,14 +90,23 @@ async function findIndexHtml(): Promise<string | null> {
return null; return null;
} }
const DUMP_RE = /^\/dumps\/([^/]+)$/; /** Answers `canonical` with a 301, keeping the request's query string. */
/** A profile, with an optional `/~/<tab>` suffix so its tabs keep OG meta. */ function redirect(ctx: Context, canonical: string): void {
const USER_RE = /^\/users\/([^/]+)(?:\/~\/[^/]+)?$/; ctx.response.status = 301;
const PLAYLIST_RE = /^\/playlists\/([^/]+)$/; // Revalidated each time: a slug taken back would otherwise loop through a
// redirect the browser kept.
ctx.response.headers.set("Cache-Control", "no-cache");
ctx.response.redirect(`${FRONTEND_URL}${canonical}${ctx.request.url.search}`);
}
/** /**
* Serves `index.html` with OG tags for a dump, user or public playlist page * Serves `index.html` with the tags of the dump, user or public playlist a
* that exists, and passes every other request on. * path names (`parsePagePath`), and passes the request on when it names none
* that exists or the visitor may see. A path naming one by a former name, a
* user's or a slug, or in another case, gets a 301 to its current path. The
* page's URL is on `FRONTEND_URL` and the images' on `PUBLIC_URL`, not on the
* request's origin, which a reverse proxy may have rewritten to an internal
* host or plain http.
*/ */
export async function ogMiddleware( export async function ogMiddleware(
ctx: Context, ctx: Context,
@@ -97,70 +114,70 @@ export async function ogMiddleware(
): Promise<unknown> { ): Promise<unknown> {
const { pathname } = ctx.request.url; const { pathname } = ctx.request.url;
if (pathname.startsWith("/api/") || pathname.includes(".")) { // A file's path never starts with `@`, so parsePagePath passes it on, while
return await next(); // a remote user's has dots in it: `/@bob@music.example`.
} if (pathname.startsWith("/api/")) return await next();
const origin = ctx.request.url.origin; const page = parsePagePath(pathname);
const pageUrl = `${origin}${pathname}`; if (!page || (page.kind !== "user" && page.edit)) return await next();
const origin = PUBLIC_URL;
let meta: OGMeta | null = null; let meta: OGMeta | null = null;
let canonical: string | null = null;
const dumpMatch = pathname.match(DUMP_RE);
const userMatch = pathname.match(USER_RE);
const playlistMatch = pathname.match(PLAYLIST_RE);
if (dumpMatch) {
try { try {
const dump = getDump(dumpMatch[1]); const user = resolveUsername(page.username);
if (page.kind === "dump") {
const dump = getDumpByPath(user.id, page.slug);
canonical = dumpPath(dump);
let imageUrl: string | undefined; let imageUrl: string | undefined;
if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) { if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) {
imageUrl = `${origin}/api/files/${dump.id}`; imageUrl = `${origin}/api/dumps/${dump.id}/file`;
} else if (dump.richContent?.thumbnailUrl) { } else if (dump.richContent?.thumbnailUrl) {
imageUrl = dump.richContent.thumbnailUrl; imageUrl = dump.richContent.thumbnailUrl;
} }
// No artwork: fall back to our own site icon rather than the target's // Without artwork, the site's icon: the linked site's favicon is too
// favicon, which is typically too small to survive a social card (and may // small for a card, and may refuse hotlinking.
// be hotlink-protected).
meta = { meta = {
title: dump.title, title: dump.title,
description: dump.comment, description: dump.comment,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`, imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
imageIsWide: !!imageUrl, imageIsWide: !!imageUrl,
url: pageUrl, url: `${FRONTEND_URL}${canonical}`,
}; };
} catch { /* not found or private — serve default */ } } else if (page.kind === "user") {
} else if (userMatch) { canonical = [userPath(user.username), ...page.rest].join("/");
try {
const user = getUserByUsername(userMatch[1]);
const imageUrl = user.avatarMime const imageUrl = user.avatarMime
? `${origin}/api/avatars/${user.id}` ? `${origin}/api/users/${user.id}/avatar`
: undefined; : undefined;
meta = { meta = {
title: user.username, title: user.username,
description: user.description, description: user.description,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`, imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
url: pageUrl, url: `${FRONTEND_URL}${canonical}`,
}; };
} catch { /* not found */ } } else {
} else if (playlistMatch) { const playlist = getPlaylistByPath(user.id, page.slug);
try { // getPlaylistByPath doesn't check visibility, unlike getDumpByPath.
const playlist = getPlaylistById(playlistMatch[1]);
if (playlist.isPublic) { if (playlist.isPublic) {
canonical = playlistPath(playlist);
const imageUrl = playlist.imageMime const imageUrl = playlist.imageMime
? `${origin}/api/playlists/${playlist.id}/image` ? `${origin}/api/playlists/${playlist.id}/cover`
: undefined; : undefined;
meta = { meta = {
title: playlist.title, title: playlist.title,
description: playlist.description, description: playlist.description,
imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`, imageUrl: imageUrl ?? `${origin}/apple-touch-icon.png`,
url: pageUrl, url: `${FRONTEND_URL}${canonical}`,
}; };
} }
} catch { /* not found or private */ }
} }
} catch { /* Unknown, or private. */ }
if (!meta) return await next(); if (!meta || !canonical) return await next();
if (canonical !== pathname.replace(/\/+$/, "")) {
return redirect(ctx, canonical);
}
const html = await findIndexHtml(); const html = await findIndexHtml();
if (!html) return await next(); if (!html) return await next();

43
api/model/auth.ts
View File

@@ -1,35 +1,36 @@
/** /**
* The payloads of the tokens the server signs, and their guards. * Checks what a verified token says before it is trusted. It types the payloads
* of session and password reset tokens, and `lib/jwt.ts` runs their guards when
* it verifies one.
* *
* @module * @module
*/ */
import { isRole, type Role } from "../../shared/user.ts"; import { isRole, type Role } from "../../shared/user.ts";
/** What a session token carries. */ /**
export interface AuthPayload { * Signed into a session token by `createJWT`, and returned by `authenticate`
/** The user's id. */ * to the auth middleware as `ctx.state.user`. By then `role` is the user's live
userId: string; * role, while `username` is still the one at issue time.
/** The user's name when the token was issued. */
username: string;
/**
* The user's role; the live one replaces it once the token is authenticated.
*/ */
export interface AuthPayload {
userId: string;
username: string;
role: Role; role: Role;
/** When the token expires, in seconds since the epoch. */ /** Seconds since the epoch. */
exp: number; exp: number;
/** When the token was issued, in seconds since the epoch. */ /** Seconds since the epoch, compared with `users.tokens_valid_after`. */
iat?: number; iat?: number;
} }
/** /**
* Checks whether `obj` is an {@link AuthPayload}, and not a purpose-scoped * Checks whether `obj` is an {@link AuthPayload}. A media or reset token's
* token's payload. * payload, which has a `purpose`, never is.
*/ */
export function isAuthPayload(obj: unknown): obj is AuthPayload { export function isAuthPayload(obj: unknown): obj is AuthPayload {
return !!obj && return !!obj &&
typeof obj === "object" && typeof obj === "object" &&
// Purpose-scoped tokens (password reset, media) are never session tokens. // A media or reset token must not pass for a session token.
!("purpose" in obj) && !("purpose" in obj) &&
"userId" in obj && typeof obj.userId === "string" && "userId" in obj && typeof obj.userId === "string" &&
"username" in obj && typeof obj.username === "string" && "username" in obj && typeof obj.username === "string" &&
@@ -37,17 +38,21 @@ export function isAuthPayload(obj: unknown): obj is AuthPayload {
"exp" in obj && typeof obj.exp === "number"; "exp" in obj && typeof obj.exp === "number";
} }
/** What a password reset token carries. */ /**
* Signed into a reset link's token by `createPasswordResetToken`, and returned
* by `verifyPasswordResetToken` to `resetPassword`.
*/
export interface PasswordResetPayload { export interface PasswordResetPayload {
/** The token's purpose, which makes it a reset token. */
purpose: "password-reset"; purpose: "password-reset";
/** The id of the user whose password it resets. */
userId: string; userId: string;
/** When the token expires, in seconds since the epoch. */ /** Seconds since the epoch. */
exp: number; exp: number;
} }
/** Checks whether `obj` is a {@link PasswordResetPayload}. */ /**
* Checks whether `obj` is a {@link PasswordResetPayload}. It doesn't check
* `exp`, which `jwtVerify` has already enforced.
*/
export function isPasswordResetPayload( export function isPasswordResetPayload(
obj: unknown, obj: unknown,
): obj is PasswordResetPayload { ): obj is PasswordResetPayload {

27
api/model/category.ts
View File

@@ -1,35 +1,35 @@
/** /**
* The `categories` table: its row type, its guard, and its mapper to the API's * Turns the rows the category service reads from `categories` into the {@link
* {@link Category}. * Category} the category routes send.
* *
* @module * @module
*/ */
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
import type { Category } from "../../shared/category.ts"; import {
type Category,
type CategoryKind,
isCategoryKind,
} from "../../shared/category.ts";
/** A row of the `categories` table. */ /** Returned by the category service's `SELECT`s on `categories`. */
export interface CategoryRow { export interface CategoryRow {
/** The category's id. */
id: string; id: string;
/** The category's URL slug, at the site root. */ kind: CategoryKind;
slug: string; slug: string;
/** The category's display name. */
name: string; name: string;
/** The category's place in the list, ascending. */ /** Sorts the categories, lowest first. */
position: number; position: number;
/** When the category was created. */
created_at: string; created_at: string;
/** When the category was last changed, if ever. */
updated_at: string | null; updated_at: string | null;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link CategoryRow}. */ /** Checks whether a row `node:sqlite` returned is a {@link CategoryRow}. */
export function isCategoryRow(obj: unknown): obj is CategoryRow { export function isCategoryRow(obj: unknown): obj is CategoryRow {
return !!obj && typeof obj === "object" && return !!obj && typeof obj === "object" &&
"id" in obj && typeof obj.id === "string" && "id" in obj && typeof obj.id === "string" &&
"kind" in obj && isCategoryKind(obj.kind) &&
"slug" in obj && typeof obj.slug === "string" && "slug" in obj && typeof obj.slug === "string" &&
"name" in obj && typeof obj.name === "string" && "name" in obj && typeof obj.name === "string" &&
"position" in obj && typeof obj.position === "number" && "position" in obj && typeof obj.position === "number" &&
@@ -38,10 +38,11 @@ export function isCategoryRow(obj: unknown): obj is CategoryRow {
(typeof obj.updated_at === "string" || obj.updated_at === null); (typeof obj.updated_at === "string" || obj.updated_at === null);
} }
/** Converts a row to the API's {@link Category}. */ /** Converts `row` to the {@link Category} the category routes send. */
export function categoryRowToApi(row: CategoryRow): Category { export function categoryRowToApi(row: CategoryRow): Category {
return { return {
id: row.id, id: row.id,
kind: row.kind,
slug: row.slug, slug: row.slug,
name: row.name, name: row.name,
position: row.position, position: row.position,

47
api/model/chat.ts
View File

@@ -1,6 +1,6 @@
/** /**
* The `chat_messages` table: its row type, its guard, and its mapper to the * Turns the rows of `chat_messages` into the {@link ChatMessage} the API sends.
* API's {@link ChatMessage}. * Only `services/chat-service.ts` uses it, on the rows of its one query.
* *
* @module * @module
*/ */
@@ -9,44 +9,32 @@ import type { SQLOutputValue } from "node:sqlite";
import type { ChatMessage } from "../../shared/chat.ts"; import type { ChatMessage } from "../../shared/chat.ts";
/** /**
* A row of the `chat_messages` table, joined with its author and, when it is * Returned by chat-service's query, which joins each message to its author and
* a reply, the replied-to message and its author. * to the message it replies to. `reply_to_id` is kept when the message it
* points to is deleted, while `reply_to_author` and `reply_to_body` then read
* null.
*/ */
export interface ChatMessageRow { export interface ChatMessageRow {
/** The message's id. */
id: string; id: string;
/** The author's id. */
user_id: string; user_id: string;
/** The message, in Markdown. */
body: string; body: string;
/** When the message was sent. */
created_at: string; created_at: string;
/** When the message was last edited, if ever. */
updated_at: string | null; updated_at: string | null;
/** The author's name (joined). */
author_username: string; author_username: string;
/** The author's avatar type, null without one (joined). */
author_avatar_mime: string | null; author_avatar_mime: string | null;
/**
* The id of the message this replies to, kept even once that message is
* deleted.
*/
reply_to_id: string | null; reply_to_id: string | null;
/**
* The replied-to message's author (joined); null when there's no reply or it
* was since deleted.
*/
reply_to_author: string | null; reply_to_author: string | null;
/** /** Cut to chat-service's `REPLY_SNIPPET_LEN` characters. */
* The start of the replied-to message (joined); null when there's no reply or
* it was since deleted.
*/
reply_to_body: string | null; reply_to_body: string | null;
/** Any other column the query selects. */ /** 1 or 0. */
mentions_everyone: number;
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link ChatMessageRow}. */ /**
* Checks whether `obj`, a row read by chat-service, is a
* {@link ChatMessageRow}.
*/
export function isChatMessageRow(obj: unknown): obj is ChatMessageRow { export function isChatMessageRow(obj: unknown): obj is ChatMessageRow {
return !!obj && typeof obj === "object" && return !!obj && typeof obj === "object" &&
"id" in obj && typeof obj.id === "string" && "id" in obj && typeof obj.id === "string" &&
@@ -64,10 +52,14 @@ export function isChatMessageRow(obj: unknown): obj is ChatMessageRow {
"reply_to_author" in obj && "reply_to_author" in obj &&
(typeof obj.reply_to_author === "string" || obj.reply_to_author === null) && (typeof obj.reply_to_author === "string" || obj.reply_to_author === null) &&
"reply_to_body" in obj && "reply_to_body" in obj &&
(typeof obj.reply_to_body === "string" || obj.reply_to_body === null); (typeof obj.reply_to_body === "string" || obj.reply_to_body === null) &&
"mentions_everyone" in obj && typeof obj.mentions_everyone === "number";
} }
/** Converts a row to the API's {@link ChatMessage}. */ /**
* Converts a row to the {@link ChatMessage} that chat-service returns, turning
* its timestamps into Dates and its nulls into absent fields.
*/
export function chatMessageRowToApi(row: ChatMessageRow): ChatMessage { export function chatMessageRowToApi(row: ChatMessageRow): ChatMessage {
return { return {
id: row.id, id: row.id,
@@ -80,5 +72,6 @@ export function chatMessageRowToApi(row: ChatMessageRow): ChatMessage {
replyToId: row.reply_to_id ?? undefined, replyToId: row.reply_to_id ?? undefined,
replyToAuthor: row.reply_to_author ?? undefined, replyToAuthor: row.reply_to_author ?? undefined,
replyToBody: row.reply_to_body ?? undefined, replyToBody: row.reply_to_body ?? undefined,
mentionsEveryone: row.mentions_everyone === 1 || undefined,
}; };
} }

32
api/model/comment.ts
View File

@@ -1,6 +1,6 @@
/** /**
* The `comments` table: its row type, its guard, and its mapper to the API's * Turns the rows the comment service reads from `comments` into the {@link
* {@link Comment}. * Comment} the comment routes send.
* *
* @module * @module
*/ */
@@ -8,38 +8,31 @@
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
import type { Comment } from "../../shared/comment.ts"; import type { Comment } from "../../shared/comment.ts";
/** A row of the `comments` table, joined with its author. */ /**
* Returned by the comment service's `SELECT`s on `comments`, joined with
* `users` for the `author_` columns.
*/
export interface CommentRow { export interface CommentRow {
/** The comment's id. */
id: string; id: string;
/** The id of the dump it is posted under. */
dump_id: string; dump_id: string;
/** The author's id. */
user_id: string; user_id: string;
/** The id of the comment it replies to; null at the top level. */ /** It is null on a top-level comment. */
parent_id: string | null; parent_id: string | null;
/** The comment, in Markdown. */
body: string; body: string;
/** When the comment was posted. */
created_at: string; created_at: string;
/** When the comment was last edited, if ever. */
updated_at: string | null; updated_at: string | null;
/** /**
* 1 once deleted: the body is emptied but the row stays, so its replies * 1 once deleted. The body is emptied but the row stays, so its replies keep
* keep their place. * their place in the thread.
*/ */
deleted: number; deleted: number;
/** How many users like the comment. */
like_count: number; like_count: number;
/** The author's name (joined). */
author_username: string; author_username: string;
/** The author's avatar type, null without one (joined). */
author_avatar_mime: string | null; author_avatar_mime: string | null;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link CommentRow}. */ /** Checks whether a row `node:sqlite` returned is a {@link CommentRow}. */
export function isCommentRow( export function isCommentRow(
obj: unknown, obj: unknown,
): obj is CommentRow { ): obj is CommentRow {
@@ -61,7 +54,10 @@ export function isCommentRow(
obj.author_avatar_mime === null); obj.author_avatar_mime === null);
} }
/** Converts a row to the API's {@link Comment}. */ /**
* Converts `row` to the {@link Comment} the comment routes send and the
* WebSocket broadcasts.
*/
export function commentRowToApi(row: CommentRow): Comment { export function commentRowToApi(row: CommentRow): Comment {
return { return {
id: row.id, id: row.id,

90
api/model/dump.ts
View File

@@ -1,6 +1,8 @@
/** /**
* The `dumps` table: its row type, its guard, the column lists services select * Turns the rows of `dumps` into the API's {@link Dump} the same way for every
* it with, and its mappers to and from the API's {@link Dump}. * service that reads dumps: each selects them with the column lists here,
* checks each row with {@link isDumpRow} and converts it with {@link
* dumpRowToApi}.
* *
* @module * @module
*/ */
@@ -8,67 +10,70 @@
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
import type { Dump, RichContent } from "../../shared/dump.ts"; import type { Dump, RichContent } from "../../shared/dump.ts";
/** A row of the `dumps` table, with its live comment count. */ /**
* Read from the `dumps` table with {@link DUMP_SELECT_COLUMNS}, or written back
* from a {@link Dump} by {@link dumpApiToRow}. It lacks `url_canonical`, which
* only the duplicate lookup reads.
*/
export interface DumpRow { export interface DumpRow {
/** The dump's id. */
id: string; id: string;
/** `url` for a link, `file` for an upload. */
kind: string; kind: string;
/** The dump's title. */
title: string; title: string;
/** The poster's description, in Markdown. */
comment: string | null; comment: string | null;
/** The poster's id. */
user_id: string; user_id: string;
/** When the dump was posted. */
created_at: string; created_at: string;
/** When the dump was last edited, if ever. */
updated_at: string | null; updated_at: string | null;
/** The dump's URL slug. */ /** Unique among its poster's dumps, and set on every row since 0021. */
slug: string | null; slug: string;
/** The link exactly as submitted, for a URL dump. */ /** The link as the poster submitted it. */
url: string | null; url: string | null;
/** The link's preview, as {@link RichContent} JSON. */ /** {@link RichContent} as JSON. */
rich_content: string | null; rich_content: string | null;
/** The uploaded file's name, for a file dump. */
file_name: string | null; file_name: string | null;
/** The uploaded file's type, for a file dump. */
file_mime: string | null; file_mime: string | null;
/** The uploaded file's size in bytes, for a file dump. */
file_size: number | null; file_size: number | null;
/** How many users upvoted the dump. */
vote_count: number; vote_count: number;
/** How many comments it has, not counting deleted ones (computed). */ /** Computed by a subquery, which counts the comments that aren't deleted. */
comment_count: number; comment_count: number;
/** 1 when only the poster sees the dump. */ /** The poster's current name, computed by a subquery. */
owner_username: string;
is_private: number; is_private: number;
/** The uploaded thumbnail's type, null without one. */
custom_thumbnail_mime: string | null; custom_thumbnail_mime: string | null;
/** Any other column the query selects. */ /** Lets a query select more columns, such as a join's. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** /**
* The stored columns of {@link DumpRow} — keep in sync with it and * Lists the stored columns of {@link DumpRow}. A column added to the table must
* {@link isDumpRow}. Services select with these instead of repeating the * be added here, to {@link DumpRow} and to {@link isDumpRow}.
* column list themselves.
*/ */
export const DUMP_COLUMNS = export const DUMP_COLUMNS =
"id, kind, title, slug, comment, user_id, created_at, updated_at, url, rich_content, file_name, file_mime, file_size, vote_count, is_private, custom_thumbnail_mime"; "id, kind, title, slug, comment, user_id, created_at, updated_at, url, rich_content, file_name, file_mime, file_size, vote_count, is_private, custom_thumbnail_mime";
/** {@link DUMP_COLUMNS}, for a query aliasing `dumps` as `d`. */ // The same columns, for a query that aliases `dumps` as `d`.
export const DUMP_COLUMNS_ALIASED = const DUMP_COLUMNS_ALIASED = DUMP_COLUMNS.split(", ").map((c) => `d.${c}`)
"d.id, d.kind, d.title, d.slug, d.comment, d.user_id, d.created_at, d.updated_at, d.url, d.rich_content, d.file_name, d.file_mime, d.file_size, d.vote_count, d.is_private, d.custom_thumbnail_mime"; .join(", ");
/** {@link DUMP_COLUMNS} and the computed `comment_count`: a full {@link DumpRow}. */ /**
* Selects a whole {@link DumpRow}, the stored columns and the computed
* `comment_count` and `owner_username`.
*/
export const DUMP_SELECT_COLUMNS: string = `${DUMP_COLUMNS}, export const DUMP_SELECT_COLUMNS: string = `${DUMP_COLUMNS},
(SELECT COUNT(*) FROM comments WHERE dump_id = dumps.id AND deleted = 0) as comment_count`; (SELECT COUNT(*) FROM comments WHERE dump_id = dumps.id AND deleted = 0) as comment_count,
(SELECT username FROM users WHERE id = dumps.user_id) as owner_username`;
/** {@link DUMP_SELECT_COLUMNS}, for a query aliasing `dumps` as `d`. */ /**
* Selects a whole {@link DumpRow} in a query that aliases `dumps` as `d` to
* join it with other tables.
*/
export const DUMP_SELECT_COLUMNS_ALIASED: string = `${DUMP_COLUMNS_ALIASED}, export const DUMP_SELECT_COLUMNS_ALIASED: string = `${DUMP_COLUMNS_ALIASED},
(SELECT COUNT(*) FROM comments WHERE dump_id = d.id AND deleted = 0) as comment_count`; (SELECT COUNT(*) FROM comments WHERE dump_id = d.id AND deleted = 0) as comment_count,
(SELECT username FROM users WHERE id = d.user_id) as owner_username`;
/** Checks whether `obj` is a {@link DumpRow}. */ /**
* Checks whether `obj`, a row from a query, is a {@link DumpRow}. It checks
* each column's type, not that `kind` is `url` or `file`.
*/
export function isDumpRow(obj: unknown): obj is DumpRow { export function isDumpRow(obj: unknown): obj is DumpRow {
return !!obj && return !!obj &&
typeof obj === "object" && typeof obj === "object" &&
@@ -81,7 +86,7 @@ export function isDumpRow(obj: unknown): obj is DumpRow {
"created_at" in obj && typeof obj.created_at === "string" && "created_at" in obj && typeof obj.created_at === "string" &&
"updated_at" in obj && "updated_at" in obj &&
(typeof obj.updated_at === "string" || obj.updated_at === null) && (typeof obj.updated_at === "string" || obj.updated_at === null) &&
"slug" in obj && (typeof obj.slug === "string" || obj.slug === null) && "slug" in obj && typeof obj.slug === "string" &&
"url" in obj && (typeof obj.url === "string" || obj.url === null) && "url" in obj && (typeof obj.url === "string" || obj.url === null) &&
"rich_content" in obj && "rich_content" in obj &&
(typeof obj.rich_content === "string" || obj.rich_content === null) && (typeof obj.rich_content === "string" || obj.rich_content === null) &&
@@ -93,22 +98,27 @@ export function isDumpRow(obj: unknown): obj is DumpRow {
(typeof obj.file_size === "number" || obj.file_size === null) && (typeof obj.file_size === "number" || obj.file_size === null) &&
"vote_count" in obj && typeof obj.vote_count === "number" && "vote_count" in obj && typeof obj.vote_count === "number" &&
"comment_count" in obj && typeof obj.comment_count === "number" && "comment_count" in obj && typeof obj.comment_count === "number" &&
"owner_username" in obj && typeof obj.owner_username === "string" &&
"is_private" in obj && typeof obj.is_private === "number" && "is_private" in obj && typeof obj.is_private === "number" &&
"custom_thumbnail_mime" in obj && "custom_thumbnail_mime" in obj &&
(typeof obj.custom_thumbnail_mime === "string" || (typeof obj.custom_thumbnail_mime === "string" ||
obj.custom_thumbnail_mime === null); obj.custom_thumbnail_mime === null);
} }
/** Converts a row to the API's {@link Dump}. */ /**
* Converts a row to the API's {@link Dump}, parsing its preview's JSON. It
* leaves `categoryIds` out: the services add them with `attachCategoryIds`.
*/
export function dumpRowToApi(row: DumpRow): Dump { export function dumpRowToApi(row: DumpRow): Dump {
return { return {
id: row.id, id: row.id,
kind: row.kind as "url" | "file", kind: row.kind as "url" | "file",
title: row.title, title: row.title,
slug: row.slug ?? undefined, slug: row.slug,
url: row.url ?? undefined, url: row.url ?? undefined,
comment: row.comment ?? undefined, comment: row.comment ?? undefined,
userId: row.user_id, userId: row.user_id,
ownerUsername: row.owner_username,
createdAt: new Date(row.created_at), createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined, updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
richContent: row.rich_content richContent: row.rich_content
@@ -124,13 +134,16 @@ export function dumpRowToApi(row: DumpRow): Dump {
}; };
} }
/** Converts a {@link Dump} back to a row. */ /**
* Converts a {@link Dump} back to a row, for a service that writes an edited
* dump's columns.
*/
export function dumpApiToRow(dump: Dump): DumpRow { export function dumpApiToRow(dump: Dump): DumpRow {
return { return {
id: dump.id, id: dump.id,
kind: dump.kind, kind: dump.kind,
title: dump.title, title: dump.title,
slug: dump.slug ?? null, slug: dump.slug,
comment: dump.comment ?? null, comment: dump.comment ?? null,
user_id: dump.userId, user_id: dump.userId,
created_at: dump.createdAt.toISOString(), created_at: dump.createdAt.toISOString(),
@@ -142,6 +155,7 @@ export function dumpApiToRow(dump: Dump): DumpRow {
file_size: dump.fileSize ?? null, file_size: dump.fileSize ?? null,
vote_count: dump.voteCount, vote_count: dump.voteCount,
comment_count: dump.commentCount, comment_count: dump.commentCount,
owner_username: dump.ownerUsername,
is_private: dump.isPrivate ? 1 : 0, is_private: dump.isPrivate ? 1 : 0,
custom_thumbnail_mime: dump.thumbnailMime ?? null, custom_thumbnail_mime: dump.thumbnailMime ?? null,
}; };

18
api/model/follow.ts
View File

@@ -1,5 +1,6 @@
/** /**
* The `follows` table: its row type and its guard. * Checks the rows `follow-service.ts` reads from `follows` to answer what a
* user follows.
* *
* @module * @module
*/ */
@@ -7,25 +8,22 @@
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
/** /**
* A row of the `follows` table: one user following either another user or a * Read by `getFollowStatus`. Exactly one of `followed_user_id` and
* playlist. * `followed_playlist_id` is set, as the table's CHECK constraint requires.
*/ */
export interface FollowRow { export interface FollowRow {
/** The follow's id. */
id: string; id: string;
/** The id of the user following. */
follower_id: string; follower_id: string;
/** The id of the user followed, when it is a user. */
followed_user_id: string | null; followed_user_id: string | null;
/** The id of the playlist followed, when it is a playlist. */
followed_playlist_id: string | null; followed_playlist_id: string | null;
/** When the follow began. */
created_at: string; created_at: string;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link FollowRow}. */ /**
* Checks whether `obj`, a row `getFollowStatus` selected, is a
* {@link FollowRow}.
*/
export function isFollowRow( export function isFollowRow(
obj: unknown, obj: unknown,
): obj is FollowRow { ): obj is FollowRow {

22
api/model/invite.ts
View File

@@ -1,26 +1,32 @@
/** /**
* The `invites` table: its row type and its guard. * Checks the rows `invite-service.ts` reads from `invites` to vet an invite
* before sign-up.
* *
* @module * @module
*/ */
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
/** A row of the `invites` table. */ /**
* Stored by `createInvite`, and read back by `validateInvite` when the invite
* link is opened or used to sign up.
*/
export interface InviteRow { export interface InviteRow {
/** The invite's token, which registering redeems. */ /** The random token the invite link carries, which is the primary key. */
token: string; token: string;
/** The id of the user who made the invite. */ /** Becomes the new user's `invited_by`. */
inviter_id: string; inviter_id: string;
/** When the invite was redeemed; null while unused. */ /** Set when an account is created with the invite. */
used_at: string | null; used_at: string | null;
/** When the invite was made. */ /** Starts the 7-day window in which the invite works. */
created_at: string; created_at: string;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link InviteRow}. */ /**
* Checks whether `obj`, a row `validateInvite` selected, is an
* {@link InviteRow}.
*/
export function isInviteRow( export function isInviteRow(
obj: unknown, obj: unknown,
): obj is InviteRow { ): obj is InviteRow {

35
api/model/notification.ts
View File

@@ -1,6 +1,6 @@
/** /**
* The `notifications` table: its row type, its guard, and its mapper to the * Turns the rows of `notifications`, once checked, into the {@link
* API's {@link Notification}. * Notification} that `GET /api/me/notifications` answers.
* *
* @module * @module
*/ */
@@ -12,30 +12,33 @@ import type {
NotificationType, NotificationType,
} from "../../shared/notification.ts"; } from "../../shared/notification.ts";
/** A row of the `notifications` table. */ /**
* Stored by the triggers of `notification-service.ts`, and read back by
* `getNotificationsForUser`.
*/
export interface NotificationRow { export interface NotificationRow {
/** The notification's id. */
id: string; id: string;
/** The id of the user notified. */
user_id: string; user_id: string;
/** The notification's {@link NotificationType}. */ /** A {@link NotificationType}, not checked by the guard. */
type: string; type: string;
/** The notification's {@link NotificationData}, as JSON. */ /** The {@link NotificationData} as JSON, not checked by the guard. */
data: string; data: string;
/** 1 once the user has read it. */ /** 0 or 1. */
read: number; read: number;
/** When the notification was made. */
created_at: string; created_at: string;
/** /**
* What caused the notification: a second one with the same * Names the event that caused the notification. A unique index keeps a
* `(user_id, source_key)` isn't stored. Null when duplicates are allowed. * second row with the same `(user_id, source_key)` out; null allows
* duplicates.
*/ */
source_key: string | null; source_key: string | null;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link NotificationRow}. */ /**
* Checks whether `obj`, a row `getNotificationsForUser` selected, is a
* {@link NotificationRow}.
*/
export function isNotificationRow( export function isNotificationRow(
obj: unknown, obj: unknown,
): obj is NotificationRow { ): obj is NotificationRow {
@@ -50,7 +53,11 @@ export function isNotificationRow(
(typeof obj.source_key === "string" || obj.source_key === null); (typeof obj.source_key === "string" || obj.source_key === null);
} }
/** Converts a row to the API's {@link Notification}. */ /**
* Converts a row to the {@link Notification} the client gets.
*
* @throws {SyntaxError} When `data` isn't valid JSON.
*/
export function notificationRowToApi(row: NotificationRow): Notification { export function notificationRowToApi(row: NotificationRow): Notification {
return { return {
id: row.id, id: row.id,

24
api/model/password-reset.ts
View File

@@ -1,26 +1,34 @@
/** /**
* The `password_reset_tokens` table: its row type and its guard. * Checks the rows `password-reset-service.ts` reads from
* `password_reset_tokens` to make each reset link work once.
* *
* @module * @module
*/ */
import type { SQLOutputValue } from "node:sqlite"; import type { SQLOutputValue } from "node:sqlite";
/** A row of the `password_reset_tokens` table. */ /**
* Stored by `requestPasswordReset` when it emails a reset link, and read back
* by `resetPassword` when the link is used.
*/
export interface PasswordResetTokenRow { export interface PasswordResetTokenRow {
/** The reset token. */ /** The signed token itself, which the reset link carries. */
token: string; token: string;
/** The id of the user whose password it resets. */
user_id: string; user_id: string;
/** When the token stops working. */ /**
* Lets `db/startup.ts` purge old rows. The token's own `exp` is what ends
* the link.
*/
expires_at: string; expires_at: string;
/** When the token was used; null while unused. */ /** Set when the link is used, and reset to null if the reset then fails. */
used_at: string | null; used_at: string | null;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link PasswordResetTokenRow}. */ /**
* Checks whether `obj`, a row `resetPassword` selected, is a
* {@link PasswordResetTokenRow}.
*/
export function isPasswordResetTokenRow( export function isPasswordResetTokenRow(
obj: unknown, obj: unknown,
): obj is PasswordResetTokenRow { ): obj is PasswordResetTokenRow {

38
api/model/playlist.ts
View File

@@ -1,6 +1,6 @@
/** /**
* The `playlists` table: its row type, its guard, and its mapper to the API's * Turns the rows the playlist and follow services read from `playlists` into
* {@link Playlist}. * the {@link Playlist} the API sends.
* *
* @module * @module
*/ */
@@ -9,33 +9,27 @@ import type { SQLOutputValue } from "node:sqlite";
import type { Playlist } from "../../shared/playlist.ts"; import type { Playlist } from "../../shared/playlist.ts";
/** /**
* A row of the `playlists` table. Queries may add `dump_count` and * Returned by the playlist and follow services' `SELECT`s on `playlists`,
* `owner_username`, which {@link playlistRowToApi} carries over. * which join the owner's name as `owner_username`. A query may add the
* `dump_count` and `last_dump_at` columns (`VISIBLE_DUMP_FIGURES`), which
* {@link playlistRowToApi} carries over when present.
*/ */
export interface PlaylistRow { export interface PlaylistRow {
/** The playlist's id. */
id: string; id: string;
/** The owner's id. */
user_id: string; user_id: string;
/** The playlist's title. */
title: string; title: string;
/** The playlist's URL slug. */ /** Unique among its owner's playlists, and set on every row since 0021. */
slug: string | null; slug: string;
/** The owner's description, in Markdown. */ owner_username: string;
description: string | null; description: string | null;
/** 1 when anyone may see the playlist, 0 when only its owner does. */
is_public: number; is_public: number;
/** When the playlist was created. */
created_at: string; created_at: string;
/** When the playlist was last changed, if ever. */
updated_at: string | null; updated_at: string | null;
/** The cover image's type, null without one. */
image_mime: string | null; image_mime: string | null;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link PlaylistRow}. */ /** Checks whether a row `node:sqlite` returned is a {@link PlaylistRow}. */
export function isPlaylistRow( export function isPlaylistRow(
obj: unknown, obj: unknown,
): obj is PlaylistRow { ): obj is PlaylistRow {
@@ -43,7 +37,8 @@ export function isPlaylistRow(
"id" in obj && typeof obj.id === "string" && "id" in obj && typeof obj.id === "string" &&
"user_id" in obj && typeof obj.user_id === "string" && "user_id" in obj && typeof obj.user_id === "string" &&
"title" in obj && typeof obj.title === "string" && "title" in obj && typeof obj.title === "string" &&
"slug" in obj && (typeof obj.slug === "string" || obj.slug === null) && "slug" in obj && typeof obj.slug === "string" &&
"owner_username" in obj && typeof obj.owner_username === "string" &&
"description" in obj && "description" in obj &&
(typeof obj.description === "string" || obj.description === null) && (typeof obj.description === "string" || obj.description === null) &&
"is_public" in obj && typeof obj.is_public === "number" && "is_public" in obj && typeof obj.is_public === "number" &&
@@ -54,21 +49,22 @@ export function isPlaylistRow(
(typeof obj.image_mime === "string" || obj.image_mime === null); (typeof obj.image_mime === "string" || obj.image_mime === null);
} }
/** Converts a row to the API's {@link Playlist}. */ /** Converts `row` to the {@link Playlist} the playlist routes send. */
export function playlistRowToApi(row: PlaylistRow): Playlist { export function playlistRowToApi(row: PlaylistRow): Playlist {
return { return {
id: row.id, id: row.id,
userId: row.user_id, userId: row.user_id,
title: row.title, title: row.title,
slug: row.slug ?? undefined, slug: row.slug,
description: row.description ?? undefined, description: row.description ?? undefined,
isPublic: Boolean(row.is_public), isPublic: Boolean(row.is_public),
createdAt: new Date(row.created_at), createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined, updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
imageMime: row.image_mime ?? undefined, imageMime: row.image_mime ?? undefined,
dumpCount: typeof row.dump_count === "number" ? row.dump_count : undefined, dumpCount: typeof row.dump_count === "number" ? row.dump_count : undefined,
ownerUsername: typeof row.owner_username === "string" lastDumpAt: typeof row.last_dump_at === "string"
? row.owner_username ? new Date(row.last_dump_at)
: undefined, : undefined,
ownerUsername: row.owner_username,
}; };
} }

24
api/model/rows.ts Normal file
View File

@@ -0,0 +1,24 @@
/**
* Makes a damaged row fail its query with a 500 rather than silently vanish
* from a list. Every service checks the rows of its list queries with it.
*
* @module
*/
import { APIException } from "../lib/api-exception.ts";
/**
* Returns `rows` turned by `toApi`, once `isRow` has accepted each.
*
* @throws {APIException} 500 when a row doesn't pass `isRow`.
*/
export function parseRows<R, T>(
rows: unknown[],
isRow: (row: unknown) => row is R,
toApi: (row: R) => T,
): T[] {
if (!rows.every(isRow)) {
throw new APIException(500, "Malformed data in the database");
}
return rows.map(toApi);
}

66
api/model/settings.ts Normal file
View File

@@ -0,0 +1,66 @@
/**
* Checks the rows `settings-service.ts` reads from `user_settings`, and turns
* them into the {@link UserSettings} the settings endpoints answer.
*
* @module
*/
import type { SQLOutputValue } from "node:sqlite";
import type { UserSettings } from "../../shared/settings.ts";
import { isColorScheme, isStyleName } from "../../shared/theme.ts";
/**
* Stored by `updateSettings` and read back by `getSettings`. A user without a
* row has the defaults.
*/
export interface UserSettingsRow {
user_id: string;
/** A style name, or null for the instance's. */
style: string | null;
color_scheme: string | null;
/** 0 or 1. */
digest_weekly: number;
/** 0 or 1. */
digest_monthly: number;
updated_at: string;
[key: string]: SQLOutputValue;
}
/**
* Names the column of each setting, for the queries that pick users by one,
* as `digest-service.ts` does by its digest's switch.
*/
export const SETTINGS_COLUMNS: Record<keyof UserSettings, string> = {
style: "style",
colorScheme: "color_scheme",
digestWeekly: "digest_weekly",
digestMonthly: "digest_monthly",
};
/**
* Checks whether `obj`, a row `getSettings` selected, is a
* {@link UserSettingsRow}.
*/
export function isUserSettingsRow(obj: unknown): obj is UserSettingsRow {
return !!obj && typeof obj === "object" &&
"user_id" in obj && typeof obj.user_id === "string" &&
"style" in obj && (obj.style === null || typeof obj.style === "string") &&
"color_scheme" in obj &&
(obj.color_scheme === null || typeof obj.color_scheme === "string") &&
"digest_weekly" in obj && typeof obj.digest_weekly === "number" &&
"digest_monthly" in obj && typeof obj.digest_monthly === "number" &&
"updated_at" in obj && typeof obj.updated_at === "string";
}
/**
* Converts a row to the {@link UserSettings} the client gets. A style or
* scheme the code no longer knows, as after one is removed, reads as unset.
*/
export function settingsRowToApi(row: UserSettingsRow): UserSettings {
return {
style: isStyleName(row.style) ? row.style : null,
colorScheme: isColorScheme(row.color_scheme) ? row.color_scheme : null,
digestWeekly: row.digest_weekly === 1,
digestMonthly: row.digest_monthly === 1,
};
}

80
api/model/user.ts
View File

@@ -1,6 +1,7 @@
/** /**
* The `users` table: its row type, its guard, its mappers to and from the * Turns the rows of `users`, once checked, into the {@link User} that
* stored {@link User}, and the views of a user the API sends. * `user-service.ts` returns. A route turns a `User` into what it may send with
* {@link toPublicUser} or {@link toSelfUser}.
* *
* @module * @module
*/ */
@@ -13,56 +14,80 @@ import {
type SelfUser, type SelfUser,
} from "../../shared/user.ts"; } from "../../shared/user.ts";
/** A user as stored: what they see of themselves, plus the password hash. */ /**
* Returned by the lookups of `user-service.ts`. It must never be sent as is:
* it carries the password hash.
*/
export interface User extends SelfUser { export interface User extends SelfUser {
/** The scrypt hash of the password, as `hash.salt`. */ /**
passwordHash: string; * From `hashPassword`, as `hash.salt`, or null for an account from another
* instance ({@link isRemoteUser}), which signs in there and never with a
* password here. The row stores that as `''`.
*/
passwordHash: string | null;
} }
/** Returns what anyone may see of `user`: no hash, no email. */ /**
* Checks whether `user`'s account lives on another instance, whose user signs
* in here through federation (`federation-service.ts`). Their password,
* email, avatar and name are that instance's to change, so the routes that
* change them here refuse.
*/
export function isRemoteUser(user: User): boolean {
return user.passwordHash === null;
}
/**
* Returns what anyone may see of `user`, without the hash and the email. Every
* route that sends another user's profile goes through it.
*/
export function toPublicUser( export function toPublicUser(
{ passwordHash: _, email: _email, ...user }: User, { passwordHash: _, email: _email, ...user }: User,
): PublicUser { ): PublicUser {
return user; return user;
} }
/** Returns what `user` sees of themselves: everything but the hash. */ /**
* Returns what `user` sees of themselves, without the hash, for the routes
* that answer the caller's own account.
*/
export function toSelfUser({ passwordHash: _, ...user }: User): SelfUser { export function toSelfUser({ passwordHash: _, ...user }: User): SelfUser {
return user; return user;
} }
/** A row of the `users` table. */ /**
* Selected by `user-service.ts` and `follow-service.ts`, and converted by
* {@link userRowToApi}. It leaves out `chat_read_at` and `tokens_valid_after`,
* which only the chat's read state and `getSessionState` query.
*/
export interface UserRow { export interface UserRow {
/** The user's id. */
id: string; id: string;
/** The user's name, unique. */
username: string; username: string;
/** The scrypt hash of the password, as `hash.salt`. */ /** `''` for an account from another instance, which has no password. */
password_hash: string; password_hash: string;
/** The user's role. */
role: Role; role: Role;
/** When the user registered. */
created_at: string; created_at: string;
/** When the profile was last changed, if ever. */ /** Bumped by any profile or avatar change. */
updated_at: string | null; updated_at: string | null;
/** The avatar's type, null without one. */ /** Null without an avatar. */
avatar_mime: string | null; avatar_mime: string | null;
/** The user's profile text, in Markdown. */ /** Markdown. */
description: string | null; description: string | null;
/** The id of the user whose invite they registered with. */ /** Null for the seeded admin, who had no inviter. */
invited_by: string | null; invited_by: string | null;
/** /**
* The inviter's name; present only when the query joins it (`LEFT JOIN users * Joined from the inviter's row rather than stored, and undefined when the
* i ON i.id = u.invited_by`). * query doesn't join it. The guard doesn't check it.
*/ */
invited_by_username: string | null; invited_by_username: string | null;
/** The user's email address. */
email: string; email: string;
/** Any other column the query selects. */
[key: string]: SQLOutputValue; [key: string]: SQLOutputValue;
} }
/** Checks whether `obj` is a {@link UserRow}. */ /**
* Checks whether `obj`, a row selected with the columns of {@link UserRow}, is
* one.
*/
export function isUserRow(obj: unknown): obj is UserRow { export function isUserRow(obj: unknown): obj is UserRow {
return !!obj && return !!obj &&
typeof obj === "object" && typeof obj === "object" &&
@@ -82,12 +107,12 @@ export function isUserRow(obj: unknown): obj is UserRow {
"email" in obj && typeof obj.email === "string"; "email" in obj && typeof obj.email === "string";
} }
/** Converts a row to the stored {@link User}. */ /** Converts a row to a {@link User}, turning nulls into undefined. */
export function userRowToApi(row: UserRow): User { export function userRowToApi(row: UserRow): User {
return { return {
id: row.id, id: row.id,
username: row.username, username: row.username,
passwordHash: row.password_hash, passwordHash: row.password_hash || null,
role: row.role, role: row.role,
createdAt: new Date(row.created_at), createdAt: new Date(row.created_at),
updatedAt: row.updated_at ? new Date(row.updated_at) : undefined, updatedAt: row.updated_at ? new Date(row.updated_at) : undefined,
@@ -100,12 +125,15 @@ export function userRowToApi(row: UserRow): User {
}; };
} }
/** Converts a {@link User} back to a row, without the join-only columns. */ /**
* Converts a {@link User} back to a row, for `updateUser`. A `User` has no
* inviter id, so `invited_by` comes out null: don't write it back.
*/
export function userApiToRow(user: User): UserRow { export function userApiToRow(user: User): UserRow {
return { return {
id: user.id, id: user.id,
username: user.username, username: user.username,
password_hash: user.passwordHash, password_hash: user.passwordHash ?? "",
role: user.role, role: user.role,
created_at: user.createdAt.toISOString(), created_at: user.createdAt.toISOString(),
updated_at: user.updatedAt?.toISOString() ?? null, updated_at: user.updatedAt?.toISOString() ?? null,

108
api/remote/bandcamp-streams.ts
View File

@@ -1,34 +1,28 @@
/** /**
* Resolves a Bandcamp page to its streamable tracks, at play time. * Resolves a Bandcamp page to its streamable tracks when the native player
* * starts it, for `GET /api/remote/bandcamp`. The stream URLs are signed and
* Bandcamp's signed stream URLs expire 24h after the page fetch, so they can * expire 24 hours after the page is fetched, so unlike the embed URL they
* never be baked into `rich_content` at dump-creation time the way `embedUrl` * can't be stored in the dump's preview. The browser plays them from bcbits
* is. This service is the play-time counterpart: the frontend asks for a page, * directly.
* gets a whole tracklist back, and plays the mp3s directly from bcbits.
* *
* @module * @module
*/ */
import { APIErrorCode } from "../../shared/api.ts";
import { APIException } from "../lib/api-exception.ts"; import { APIException } from "../lib/api-exception.ts";
import { TtlCache } from "../lib/ttl-cache.ts";
import { fetchWithTimeout, isValidHttpUrl, readTextCapped } from "./fetch.ts"; import { fetchWithTimeout, isValidHttpUrl, readTextCapped } from "./fetch.ts";
import { BANDCAMP_HOST_RE, parseTralbum } from "./bandcamp-tralbum.ts"; import { BANDCAMP_HOST_RE, parseTralbum } from "./bandcamp-tralbum.ts";
import type { Tralbum } from "../../shared/player.ts"; import type { Tralbum } from "../../shared/player.ts";
/** /**
* How long a resolved page is cached: comfortably inside the 24h signature * Lives half the signatures' life, so a cached URL still has at least 12
* life, so a cache hit is never a URL that is about to expire mid-listen. * hours left when it is handed out.
*/ */
const TTL_MS = 12 * 60 * 60 * 1000; const cache = new TtlCache<Tralbum>(12 * 3_600_000, 200);
/** The most pages cached; the oldest write goes first. */
const MAX_ENTRIES = 200;
/** Resolved pages, by {@link cacheKey}. */
const cache = new Map<string, Tralbum>();
/** /**
* Returns the cache key of `url`: a page is the same page regardless of query, * Returns `url` without its query, fragment and trailing slashes, and with its
* fragment or case. * host in lowercase.
*/ */
function cacheKey(url: string): string { function cacheKey(url: string): string {
try { try {
@@ -41,30 +35,11 @@ function cacheKey(url: string): string {
} }
} }
function readCache(key: string): Tralbum | null {
const hit = cache.get(key);
if (!hit) return null;
if (Date.now() - hit.resolvedAt > TTL_MS) {
cache.delete(key);
return null;
}
return hit;
}
function writeCache(key: string, value: Tralbum): void {
// Map preserves insertion order, so the first key is the oldest write.
if (cache.size >= MAX_ENTRIES) {
const oldest = cache.keys().next().value;
if (oldest !== undefined) cache.delete(oldest);
}
cache.set(key, value);
}
/** /**
* Fetches a Bandcamp page and returns its tracklist, cached. * Returns the tracklist of the Bandcamp page `pageUrl`, from the cache when it
* * was resolved in the last 12 hours. `force` skips the cache: the player sets
* `force` bypasses the cache — used by the client's re-resolve path, so a * it when it re-resolves a queue whose URLs expired or are about to, which a
* cached entry whose URLs have started 403ing can't keep being handed back. * cached entry could hand back.
* *
* @throws {APIException} 400 when `pageUrl` isn't a public bandcamp.com URL, * @throws {APIException} 400 when `pageUrl` isn't a public bandcamp.com URL,
* 502 when Bandcamp can't be reached or doesn't answer with a page, 404 when * 502 when Bandcamp can't be reached or doesn't answer with a page, 404 when
@@ -75,30 +50,22 @@ export async function resolveBandcamp(
{ force = false }: { force?: boolean } = {}, { force = false }: { force?: boolean } = {},
): Promise<Tralbum> { ): Promise<Tralbum> {
if (!isValidHttpUrl(pageUrl)) { if (!isValidHttpUrl(pageUrl)) {
throw new APIException( throw new APIException(400, "Invalid URL");
APIErrorCode.VALIDATION_ERROR,
400,
"Invalid URL",
);
} }
let hostname: string; let hostname: string;
try { try {
hostname = new URL(pageUrl).hostname; hostname = new URL(pageUrl).hostname;
} catch { } catch {
throw new APIException(APIErrorCode.VALIDATION_ERROR, 400, "Invalid URL"); throw new APIException(400, "Invalid URL");
} }
if (!BANDCAMP_HOST_RE.test(hostname)) { if (!BANDCAMP_HOST_RE.test(hostname)) {
throw new APIException( throw new APIException(400, "Not a Bandcamp URL");
APIErrorCode.VALIDATION_ERROR,
400,
"Not a Bandcamp URL",
);
} }
const key = cacheKey(pageUrl); const key = cacheKey(pageUrl);
if (!force) { if (!force) {
const hit = readCache(key); const hit = cache.get(key);
if (hit) return hit; if (hit) return hit;
} }
@@ -106,44 +73,25 @@ export async function resolveBandcamp(
try { try {
res = await fetchWithTimeout(pageUrl, 8000); res = await fetchWithTimeout(pageUrl, 8000);
} catch { } catch {
throw new APIException( throw new APIException(502, "Could not reach Bandcamp");
APIErrorCode.SERVER_ERROR,
502,
"Could not reach Bandcamp",
);
} }
// Every path out of here that doesn't read the body has to release it, or the // A body left unread keeps its connection open until GC, and this endpoint
// connection stays open until GC. This endpoint is unauthenticated: a caller // is public: every early return cancels it.
// hammering random /track/<slug> URLs would otherwise leak one body a request.
if (!res.ok) { if (!res.ok) {
await res.body?.cancel(); await res.body?.cancel();
throw new APIException( throw new APIException(502, `Bandcamp returned ${res.status}`);
APIErrorCode.SERVER_ERROR,
502,
`Bandcamp returned ${res.status}`,
);
} }
if (!(res.headers.get("content-type") ?? "").startsWith("text/html")) { if (!(res.headers.get("content-type") ?? "").startsWith("text/html")) {
await res.body?.cancel(); await res.body?.cancel();
throw new APIException( throw new APIException(502, "Bandcamp did not return a page");
APIErrorCode.SERVER_ERROR,
502,
"Bandcamp did not return a page",
);
} }
const parsed = parseTralbum(await readTextCapped(res), pageUrl); const parsed = parseTralbum(await readTextCapped(res), pageUrl);
if (!parsed) { if (!parsed) {
// Covers "no data-tralbum", "hasAudio: false" and preorder-only releases // No data, no audio or nothing streamable yet (a preorder). The client
// with nothing streamable — all cases where the client falls back to the // plays the embed instead.
// iframe embed rather than showing an error. throw new APIException(404, "No streamable tracks on this page");
throw new APIException(
APIErrorCode.NOT_FOUND,
404,
"No streamable tracks on this page",
);
} }
writeCache(key, parsed); return cache.set(key, parsed);
return parsed;
} }

66
api/remote/bandcamp-tralbum.ts
View File

@@ -1,59 +1,23 @@
/** /**
* Parser for Bandcamp's `data-tralbum` attribute. * Reads a release out of a Bandcamp page, for `bandcamp-streams.ts`. Every
* * track and album page carries the whole release, stream URLs included, as
* Every Bandcamp track and album page carries the whole release — titles, * HTML-escaped JSON in a `data-tralbum` attribute, so one fetch covers an
* durations and signed mp3 stream URLs — as one HTML-escaped JSON blob on a * album. The format is undocumented, so every field is treated as optional.
* `data-tralbum` attribute. One fetch of an album page therefore yields every
* track's stream URL; there is no per-track request.
*
* The stream URLs are signed and expire 24h after the page was fetched, which
* is why nothing here is ever persisted to `rich_content` — see
* `api/routes/bandcamp.ts` for the resolve-at-play-time endpoint.
*
* The shape is undocumented, so every field is treated as optional and the
* whole parse is best-effort: a `null` return means "not a Bandcamp page",
* which callers turn into a fallback to the iframe embed.
* *
* @module * @module
*/ */
import type { Tralbum, TralbumTrack } from "../../shared/player.ts"; import type { Tralbum, TralbumTrack } from "../../shared/player.ts";
import { decodeHtmlEntities } from "./html.ts";
/** /**
* The hostnames Bandcamp serves its own sites from, anchored at the end so that `bandcamp.com.example.org` cannot pass. Artists on custom * Matches the hostnames `bandcamp-streams.ts` accepts: bandcamp.com and its
* domains do not match this and keep using the embed. * subdomains, and not `bandcamp.com.example.org`. An artist on a custom domain
* doesn't match, and the player falls back to the embed.
*/ */
export const BANDCAMP_HOST_RE = /(?:^|\.)bandcamp\.com$/; export const BANDCAMP_HOST_RE = /(?:^|\.)bandcamp\.com$/;
/** /** Returns the URL of the artwork `artId`; undefined when it isn't numeric. */
* Decodes HTML entities in a single pass.
*
* The single pass is the point: `decodeHtmlEntities` in remote/html.ts
* replaces `&amp;` before `&quot;`, so an escaped-ampersand-then-quot sequence
* (`&amp;quot;`, i.e. the literal text `&quot;` inside a title) decodes twice
* and yields a bare `"` — which lands inside the JSON string and breaks the
* parse. Matching every entity form in one regex means each is replaced once.
*/
function decodeEntitiesOnce(input: string): string {
const named: Record<string, string> = {
quot: '"',
apos: "'",
amp: "&",
lt: "<",
gt: ">",
nbsp: " ",
};
return input.replace(
/&(?:#(\d+)|#[xX]([0-9a-fA-F]+)|([a-zA-Z]+));/g,
(match, dec: string, hex: string, name: string) => {
if (dec) return String.fromCodePoint(Number(dec));
if (hex) return String.fromCodePoint(parseInt(hex, 16));
return named[name.toLowerCase()] ?? match;
},
);
}
/** Returns the URL of the large square artwork numbered `artId`. */
function artworkUrl(artId: unknown): string | undefined { function artworkUrl(artId: unknown): string | undefined {
if (typeof artId !== "number" && typeof artId !== "string") return undefined; if (typeof artId !== "number" && typeof artId !== "string") return undefined;
const id = String(artId).trim(); const id = String(artId).trim();
@@ -63,8 +27,9 @@ function artworkUrl(artId: unknown): string | undefined {
} }
/** /**
* Pulls the `data-tralbum` blob out of a Bandcamp page and normalises it. * Returns the release on the Bandcamp page `html`, fetched from `sourceUrl`;
* Returns null when the page has no blob, no audio, or nothing parseable. * null when the page has no release data, no audio or no streamable track.
* Stream URLs outside bcbits.com are dropped.
*/ */
export function parseTralbum(html: string, sourceUrl: string): Tralbum | null { export function parseTralbum(html: string, sourceUrl: string): Tralbum | null {
const match = html.match(/\bdata-tralbum=(["'])([\s\S]*?)\1/); const match = html.match(/\bdata-tralbum=(["'])([\s\S]*?)\1/);
@@ -72,7 +37,7 @@ export function parseTralbum(html: string, sourceUrl: string): Tralbum | null {
let raw: Record<string, unknown>; let raw: Record<string, unknown>;
try { try {
raw = JSON.parse(decodeEntitiesOnce(match[2])); raw = JSON.parse(decodeHtmlEntities(match[2]));
} catch { } catch {
return null; return null;
} }
@@ -84,8 +49,8 @@ export function parseTralbum(html: string, sourceUrl: string): Tralbum | null {
const t = (entry ?? {}) as Record<string, unknown>; const t = (entry ?? {}) as Record<string, unknown>;
const file = (t.file ?? {}) as Record<string, unknown>; const file = (t.file ?? {}) as Record<string, unknown>;
const mp3 = typeof file["mp3-128"] === "string" ? file["mp3-128"] : null; const mp3 = typeof file["mp3-128"] === "string" ? file["mp3-128"] : null;
// Only ever hand out bcbits URLs. Without this the endpoint would relay // A hostile page could put any host here, and the endpoint would hand it
// whatever host a hostile page put in the blob straight to the browser. // to the browser.
const streamUrl = mp3 && isBcbitsUrl(mp3) ? mp3 : null; const streamUrl = mp3 && isBcbitsUrl(mp3) ? mp3 : null;
return { return {
index, index,
@@ -117,7 +82,6 @@ export function parseTralbum(html: string, sourceUrl: string): Tralbum | null {
}; };
} }
/** Checks whether `url` is an https URL on Bandcamp's bcbits.com CDN. */
function isBcbitsUrl(url: string): boolean { function isBcbitsUrl(url: string): boolean {
try { try {
const { protocol, hostname } = new URL(url); const { protocol, hostname } = new URL(url);

207
api/remote/fetch.ts
View File

@@ -1,12 +1,17 @@
/** /**
* The guarded fetch every request to another site goes through: each hop must * Fetches pages and images from other sites for link previews, the image proxy
* resolve to a public address, the whole exchange has a deadline, and pages are * and the Bandcamp resolver, without letting a user-supplied URL reach the
* read up to a cap. * server's own network. Each hop must resolve to a public address, the whole
* exchange has a deadline, and pages are read up to a cap.
* *
* @module * @module
*/ */
import { isPrivateHostname, resolvePublicAddress } from "./public-address.ts"; import {
bareHostname,
isPrivateHostname,
resolvePublicAddress,
} from "./public-address.ts";
import { isSelfUrl } from "../lib/self-url.ts"; import { isSelfUrl } from "../lib/self-url.ts";
const FETCH_HEADERS = { const FETCH_HEADERS = {
@@ -20,15 +25,11 @@ const FETCH_HEADERS = {
const MAX_REDIRECTS = 5; const MAX_REDIRECTS = 5;
/** /**
* Fetches `url` through a `curl --insecure` subprocess and returns a minimal * Fetches `url` with `curl --insecure`, for a site whose TLS certificate Deno
* Response. * rejects. Returns a Response carrying only the `Location` and `Content-Type`
* Used as a last resort when Deno's rustls rejects the server's TLS certificate * headers, or null when curl is missing or fails. The connection is pinned to
* due to an unsupported algorithm (e.g. UnsupportedSignatureAlgorithm). * `address`, the one {@link resolvePublicAddress} checked, and redirects come
* Returns null if curl is unavailable or exits non-zero. * back unfollowed so that {@link fetchWithTimeout} checks each hop.
*
* `address` pins the connection to the IP already checked by
* resolvePublicAddress, and curl doesn't follow redirects itself: a 3xx comes
* back with its Location so fetchWithTimeout can check the next hop too.
*/ */
async function fetchViaCurl( async function fetchViaCurl(
url: URL, url: URL,
@@ -46,9 +47,11 @@ async function fetchViaCurl(
"--proto", "--proto",
"=http,https", "=http,https",
"--resolve", "--resolve",
`${url.hostname.replace(/^\[(.*)\]$/, "$1")}:${port}:${pinned}`, `${bareHostname(url)}:${port}:${pinned}`,
"--max-time", "--max-time",
String(Math.ceil(timeoutMs / 1000)), String(Math.ceil(timeoutMs / 1000)),
// curl fails on a larger file rather than stopping at a cap, so this is
// looser than MAX_PAGE_BYTES, where readTextCapped still cuts.
"--max-filesize", "--max-filesize",
String(2 * MAX_PAGE_BYTES), String(2 * MAX_PAGE_BYTES),
"--user-agent", "--user-agent",
@@ -78,20 +81,7 @@ async function fetchViaCurl(
if (contentType) headers.set("content-type", contentType); if (contentType) headers.set("content-type", contentType);
const bytes = await Deno.readFile(tmpPath); const bytes = await Deno.readFile(tmpPath);
if (!contentType.startsWith("text/")) {
return new Response(bytes, { status: statusCode, headers }); return new Response(bytes, { status: statusCode, headers });
}
// Decode using the charset declared in the Content-Type header so that
// pages served in ISO-8859-1, windows-1252, etc. are read correctly.
const charset = /charset=([\w-]+)/i.exec(contentType)?.[1] ?? "utf-8";
let bodyText: string;
try {
bodyText = new TextDecoder(charset, { fatal: false }).decode(bytes);
} catch {
bodyText = new TextDecoder("utf-8", { fatal: false }).decode(bytes);
}
headers.set("content-type", contentType.replace(/;.*$/, "; charset=utf-8"));
return new Response(bodyText, { status: statusCode, headers });
} catch { } catch {
return null; return null;
} finally { } finally {
@@ -100,15 +90,15 @@ async function fetchViaCurl(
} }
/** /**
* The most bytes of a fetched page that are read. Metadata lives in the <head>, * The default cap of {@link readTextCapped}. The metadata previews read is in
* and a "page" past this is a download, not something to scrape. * the `<head>`, and a page past this is a download.
*/ */
export const MAX_PAGE_BYTES = 4 * 1024 * 1024; export const MAX_PAGE_BYTES = 4 * 1024 * 1024;
/** /**
* Hands `body` on, and clears the request's deadline once it has been read to * Wraps `body` so that the request's deadline `timer` is cleared only once the
* the end or cancelled. Until then the deadline still applies: a server that * body is read to the end or cancelled. A server that answers at once and then
* answers at once and then drips its body can't hold the request open. * drips its body is still cut off.
*/ */
function withDeadline( function withDeadline(
body: ReadableStream<Uint8Array>, body: ReadableStream<Uint8Array>,
@@ -138,14 +128,14 @@ function withDeadline(
} }
/** /**
* Reads a response's body as text, stopping at `maxBytes` (the rest is * Reads the body of `res` up to `maxBytes`; the rest is cancelled, never
* cancelled, never downloaded). * downloaded. The image proxy reads through it, and {@link readTextCapped}.
*/ */
export async function readTextCapped( export async function readBytesCapped(
res: Response, res: Response,
maxBytes = MAX_PAGE_BYTES, maxBytes: number,
): Promise<string> { ): Promise<Uint8Array> {
if (!res.body) return ""; if (!res.body) return new Uint8Array();
const reader = res.body.getReader(); const reader = res.body.getReader();
const chunks: Uint8Array[] = []; const chunks: Uint8Array[] = [];
let size = 0; let size = 0;
@@ -167,10 +157,45 @@ export async function readTextCapped(
offset += part.byteLength; offset += part.byteLength;
if (offset >= bytes.byteLength) break; if (offset >= bytes.byteLength) break;
} }
return new TextDecoder().decode(bytes); return bytes;
} }
/** Makes one request, following no redirects. The timeout covers the body too. */ /**
* Returns the charset a page declares, in its `Content-Type` header or else in
* a `<meta>` near its top, as HTML allows; UTF-8 when it declares none.
*/
function declaredCharset(contentType: string, bytes: Uint8Array): string {
const fromHeader = /charset=["']?([\w-]+)/i.exec(contentType)?.[1];
if (fromHeader) return fromHeader;
// Latin-1 maps every byte to a character, so the ASCII of the tag reads
// right whatever the page's encoding.
const head = new TextDecoder("latin1").decode(bytes.subarray(0, 1024));
return /<meta[^>]+charset=["']?([\w-]+)/i.exec(head)?.[1] ?? "utf-8";
}
/**
* Reads the body of `res` as text, up to `maxBytes`, in the charset the page
* declares; the rest is cancelled, never downloaded. {@link fetchPage} and the
* Bandcamp and YouTube resolvers read pages through it.
*/
export async function readTextCapped(
res: Response,
maxBytes = MAX_PAGE_BYTES,
): Promise<string> {
const bytes = await readBytesCapped(res, maxBytes);
const charset = declaredCharset(res.headers.get("content-type") ?? "", bytes);
try {
return new TextDecoder(charset).decode(bytes);
} catch {
// TextDecoder throws on a label it doesn't know rather than guess.
return new TextDecoder().decode(bytes);
}
}
/**
* Makes one request to `url`, following no redirects, with a deadline that
* covers the body too.
*/
async function fetchOnce( async function fetchOnce(
url: URL, url: URL,
address: string, address: string,
@@ -191,10 +216,9 @@ async function fetchOnce(
throw err; throw err;
} }
// Deno/rustls rejected the server's TLS certificate (expired, self-signed, // Deno can't be told to skip certificate checks for one request, so a
// or an unsupported signature algorithm) and offers no way to override // site with an expired, self-signed or unusually signed certificate is
// verification programmatically. Fall back to a `curl --insecure` // fetched with curl instead.
// subprocess, which uses its own TLS stack.
const curlRes = await fetchViaCurl(url, address, timeoutMs); const curlRes = await fetchViaCurl(url, address, timeoutMs);
if (curlRes) return curlRes; if (curlRes) return curlRes;
@@ -212,10 +236,37 @@ async function fetchOnce(
} }
/** /**
* Fetches a user-supplied URL, following up to five redirects. Every hop — the * Fetches `url` as {@link fetchWithTimeout} does, and also returns the URL the
* URL itself and each redirect — must resolve to a public address (see * last hop answered from, which {@link fetchPage} resolves the page's links
* {@link resolvePublicAddress}), so a link or a redirect can't point the * against.
* server at its own network. */
async function fetchFollowing(
url: string,
timeoutMs: number,
): Promise<{ res: Response; url: URL }> {
let current = new URL(url);
for (let hop = 0;; hop++) {
if (current.protocol !== "http:" && current.protocol !== "https:") {
throw new TypeError(`Unsupported protocol ${current.protocol}`);
}
const address = await resolvePublicAddress(current);
const res = await fetchOnce(current, address, timeoutMs);
const location = res.headers.get("location");
if (res.status < 300 || res.status >= 400 || !location) {
return { res, url: current };
}
await res.body?.cancel();
if (hop >= MAX_REDIRECTS) throw new TypeError("Too many redirects");
current = new URL(location, current);
}
}
/**
* Fetches a user-supplied URL, following up to five redirects, each of which
* must resolve to a public address too. `timeoutMs` bounds each hop, body
* included. The caller must read or cancel the body, or its connection stays
* open.
* *
* @throws {TypeError} When a hop isn't http(s), there are too many redirects, * @throws {TypeError} When a hop isn't http(s), there are too many redirects,
* or the request fails. * or the request fails.
@@ -225,33 +276,55 @@ export async function fetchWithTimeout(
url: string, url: string,
timeoutMs = 5000, timeoutMs = 5000,
): Promise<Response> { ): Promise<Response> {
let current = new URL(url); return (await fetchFollowing(url, timeoutMs)).res;
for (let hop = 0;; hop++) {
if (current.protocol !== "http:" && current.protocol !== "https:") {
throw new TypeError(`Unsupported protocol ${current.protocol}`);
}
const address = await resolvePublicAddress(current);
const res = await fetchOnce(current, address, timeoutMs);
const location = res.headers.get("location");
if (res.status < 300 || res.status >= 400 || !location) return res;
await res.body?.cancel();
if (hop >= MAX_REDIRECTS) throw new TypeError("Too many redirects");
current = new URL(location, current);
}
} }
/** /**
* Checks whether `raw` is an http(s) URL a dump may link to: not on a private * Returned by {@link fetchPage} to the link preview providers.
* host, unless it is one of this site's own pages. */
export interface FetchedPage {
/**
* The URL the page was served from, after redirects: the base its relative
* links resolve against.
*/
url: string;
/** The page's text, up to {@link MAX_PAGE_BYTES}; missing when it isn't HTML. */
html?: string;
}
/**
* Fetches a page for its link preview, through {@link fetchWithTimeout}. A
* response that isn't HTML is cancelled unread.
*
* @throws {Error} When the page answers with an error status, so that an error
* page never becomes a preview; and as {@link fetchWithTimeout} does.
*/
export async function fetchPage(url: string): Promise<FetchedPage> {
const { res, url: finalUrl } = await fetchFollowing(url, 5000);
if (!res.ok) {
await res.body?.cancel();
throw new Error(`${finalUrl} answered ${res.status}`);
}
const contentType = res.headers.get("content-type") ?? "";
if (!/^(text\/html|application\/xhtml\+xml)\b/i.test(contentType)) {
await res.body?.cancel();
return { url: finalUrl.href };
}
return { url: finalUrl.href, html: await readTextCapped(res) };
}
/**
* Checks whether `raw` is an http(s) URL on a host that isn't private, or a
* link to one of this site's own pages. Routes and services check a URL with it
* before storing, previewing or proxying it; it does no DNS lookup, which
* {@link fetchWithTimeout} does.
*/ */
export function isValidHttpUrl(raw: string): boolean { export function isValidHttpUrl(raw: string): boolean {
try { try {
const u = new URL(raw); const u = new URL(raw);
if (u.protocol !== "http:" && u.protocol !== "https:") return false; if (u.protocol !== "http:" && u.protocol !== "https:") return false;
// This site's own pages are fine even on a private host (e.g. localhost in // This site may run on a private host (localhost in development), and the
// development): selfProvider reads them from the DB, no HTTP involved. // self provider reads its pages from the database without fetching them.
// Every actual fetch is checked again, after DNS, in fetchWithTimeout.
return !isPrivateHostname(u) || isSelfUrl(u); return !isPrivateHostname(u) || isSelfUrl(u);
} catch { } catch {
return false; return false;

271
api/remote/gerbeur-instance.ts Normal file
View File

@@ -0,0 +1,271 @@
/**
* Talks to other gerbeur instances for federated sign-in: it reads their
* metadata and public keys, trades a code for an identity token, delivers
* events, and fetches a user's avatar. `federation-service.ts`,
* `identity-provider-service.ts` and `federation-peer-service.ts` reach other
* instances only through it.
*
* An identity is only as safe as the connection it came over, so unlike
* `fetch.ts` it never falls back to skipping certificate checks, and follows
* no redirect. It reaches only https instances on public addresses, unless
* `GERBEUR_FEDERATION_ALLOW_PRIVATE` lets two instances on one machine talk.
*
* @module
*/
import type { JSONWebKeySet } from "@panva/jose";
import {
type InstanceMetadata,
isInstanceMetadata,
} from "../../shared/federation.ts";
import {
FEDERATION_ALLOW_PRIVATE,
MAX_IMAGE_SIZE_BYTES,
OG_SITE_NAME,
} from "../config.ts";
import { resolvePublicAddress } from "./public-address.ts";
import { readBytesCapped, readTextCapped } from "./fetch.ts";
import { TtlCache } from "../lib/ttl-cache.ts";
/** How long a request to another instance may take, body included. */
const TIMEOUT_MS = 8000;
/** Caps a metadata document, a key set or a token response. */
const MAX_JSON_BYTES = 64 * 1024;
const USER_AGENT = `gerbeur (${OG_SITE_NAME}; federation)`;
/**
* Keeps each instance's metadata and keys for 10 minutes, so that a sign-in
* or an event doesn't fetch them each time. A key set is refetched sooner
* when a token names a key it lacks ({@link fetchKeys} with `fresh`), at most
* once a minute: anyone can send a token naming a key that doesn't exist.
*/
const metadataCache = new TtlCache<InstanceMetadata>(10 * 60_000, 500);
const keysCache = new TtlCache<JSONWebKeySet>(10 * 60_000, 500);
const keysRefetched = new TtlCache<true>(60_000, 500);
/** Keeps which issuer an instance's pages belong to ({@link resolveIssuer}). */
const issuerCache = new TtlCache<string>(10 * 60_000, 500);
/**
* Checks that `url` may be reached: https on a public address, or anything
* under `GERBEUR_FEDERATION_ALLOW_PRIVATE`.
*
* @throws {Error} When it may not.
*/
export async function assertReachable(url: URL): Promise<void> {
if (FEDERATION_ALLOW_PRIVATE) {
if (url.protocol !== "https:" && url.protocol !== "http:") {
throw new Error(`Unsupported protocol ${url.protocol}`);
}
return;
}
if (url.protocol !== "https:") {
throw new Error(`Refusing to reach ${url.origin} without https`);
}
await resolvePublicAddress(url);
}
/**
* Makes one request to another instance, after {@link assertReachable}, with
* the deadline covering the body. The caller reads or cancels the body.
*/
async function request(url: string, init: RequestInit = {}): Promise<Response> {
const target = new URL(url);
await assertReachable(target);
return await fetch(target, {
...init,
redirect: "error",
signal: AbortSignal.timeout(TIMEOUT_MS),
headers: {
"User-Agent": USER_AGENT,
"Accept": "application/json",
...init.headers,
},
});
}
/**
* Returns the JSON body of `res`, up to {@link MAX_JSON_BYTES}.
*
* @throws {Error} When the status isn't a success or the body isn't JSON.
*/
async function readJson(res: Response, what: string): Promise<unknown> {
if (!res.ok) {
await res.body?.cancel();
throw new Error(`${what} answered ${res.status}`);
}
return JSON.parse(await readTextCapped(res, MAX_JSON_BYTES));
}
/**
* Returns the metadata `origin` serves at `/.well-known/openid-configuration`,
* whichever issuer it names.
*
* @throws {Error} When it can't be reached or isn't a gerbeur instance's.
*/
async function metadataAt(origin: string): Promise<InstanceMetadata> {
const body = await readJson(
await request(`${origin}/.well-known/openid-configuration`),
origin,
);
if (!isInstanceMetadata(body)) {
throw new Error(`${origin} is not a gerbeur instance this one can trust`);
}
return body;
}
/**
* Returns the metadata of the instance at `origin`, its issuer, cached. It
* must name `origin` as its issuer and speak this protocol's version.
*
* @throws {Error} When it can't be reached or isn't a gerbeur instance.
*/
export async function fetchMetadata(origin: string): Promise<InstanceMetadata> {
const hit = metadataCache.get(origin);
if (hit) return hit;
const body = await metadataAt(origin);
if (body.issuer !== origin) {
throw new Error(`${origin} names itself ${body.issuer}`);
}
return metadataCache.set(origin, body);
}
/**
* Returns the issuer of the instance at `origin`, cached: `origin` itself, or,
* when `origin` is where the instance's pages are and its API lives at
* another, the issuer the pages' metadata names, whose own metadata must name
* those pages back as where its `/authorize` is. People know an instance by
* the address in their browser, which a sign-in form or the trust list may be
* given; a handle and the stored identities name the issuer.
*
* @throws {Error} When `origin` can't be reached, isn't a gerbeur instance,
* or names an issuer that doesn't name it back.
*/
export async function resolveIssuer(origin: string): Promise<string> {
const hit = issuerCache.get(origin);
if (hit) return hit;
if (metadataCache.get(origin)) return origin;
const body = await metadataAt(origin);
if (body.issuer === origin) {
metadataCache.set(origin, body);
return issuerCache.set(origin, origin);
}
const pages =
new URL((await fetchMetadata(body.issuer)).authorization_endpoint)
.origin;
if (pages !== origin) {
throw new Error(`${origin} names ${body.issuer}, whose pages are ${pages}`);
}
return issuerCache.set(origin, body.issuer);
}
/**
* Returns the public keys of the instance at `origin`, cached unless `fresh`
* asks to refetch them and they weren't in the last minute.
*
* @throws {Error} When they can't be fetched.
*/
export async function fetchKeys(
origin: string,
fresh = false,
): Promise<JSONWebKeySet> {
const refetch = fresh && !keysRefetched.get(origin);
const hit = refetch ? undefined : keysCache.get(origin);
if (hit) return hit;
if (refetch) keysRefetched.set(origin, true);
const { jwks_uri } = await fetchMetadata(origin);
const body = await readJson(await request(jwks_uri), `${origin}'s keys`);
if (
!body || typeof body !== "object" ||
!Array.isArray((body as { keys?: unknown }).keys)
) {
throw new Error(`${origin}'s keys are not a key set`);
}
return keysCache.set(origin, body as JSONWebKeySet);
}
/**
* Trades `code` for an identity token at the token endpoint of the instance
* at `origin`, and returns the token, unverified.
*
* @throws {Error} When the instance refuses or answers no token.
*/
export async function exchangeCode(
origin: string,
params: {
code: string;
clientId: string;
redirectUri: string;
codeVerifier: string;
},
): Promise<string> {
const { token_endpoint } = await fetchMetadata(origin);
const body = await readJson(
await request(token_endpoint, {
method: "POST",
headers: { "Content-Type": "application/x-www-form-urlencoded" },
body: new URLSearchParams({
grant_type: "authorization_code",
code: params.code,
client_id: params.clientId,
redirect_uri: params.redirectUri,
code_verifier: params.codeVerifier,
}),
}),
`${origin}'s token endpoint`,
);
const idToken = (body as { id_token?: unknown } | null)?.id_token;
if (typeof idToken !== "string") {
throw new Error(`${origin} answered no identity token`);
}
return idToken;
}
/**
* Sends the signed event `event` to the events endpoint of the instance at
* `origin`, and returns whether it took it. A 4xx other than 429 means it never
* will, which the caller treats as delivered.
*
* @throws {Error} When it can't be reached, or answers 429 or a 5xx.
*/
export async function postEvent(
origin: string,
event: string,
): Promise<boolean> {
const { gerbeur_events_endpoint } = await fetchMetadata(origin);
const res = await request(gerbeur_events_endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ event }),
});
await res.body?.cancel();
if (res.ok) return true;
if (res.status === 429 || res.status >= 500) {
throw new Error(`${origin}'s events endpoint answered ${res.status}`);
}
return false;
}
/**
* Returns the image at `url`, a user's avatar on their home instance, up to
* the size an avatar may be.
*
* @throws {Error} When it can't be fetched.
*/
export async function fetchAvatar(url: string): Promise<Uint8Array> {
const res = await request(url, { headers: { "Accept": "image/*" } });
if (!res.ok) {
await res.body?.cancel();
throw new Error(`The avatar at ${url} answered ${res.status}`);
}
return await readBytesCapped(res, MAX_IMAGE_SIZE_BYTES + 1);
}
/** Empties the caches, for the tests' reset between cases. */
export function clearInstanceCaches(): void {
metadataCache.clear();
issuerCache.clear();
keysCache.clear();
keysRefetched.clear();
}

732
api/remote/html.ts
View File

@@ -1,98 +1,246 @@
/** /**
* Scraping helpers: the metadata a link preview reads from a page's HTML — * Reads a link preview's metadata out of a page's HTML. {@link parsePage}
* meta and OG tags, the title, JSON-LD, images, icons and brand colors. * scans the page once for the tags that carry it (`<meta>`, `<link>`, `<img>`,
* `<title>` and JSON-LD scripts) without building a document, and returns a
* {@link Page} that the providers' `genericPreview` picks from. Every field is
* undefined when the page doesn't say.
* *
* @module * @module
*/ */
/** Decodes the named and numeric HTML entities in `str`. */ import { hexOf, parseCssColor } from "../lib/css-color.ts";
function decodeHtmlEntities(str: string): string {
return str /**
.replace(/&amp;/gi, "&") * The named entities decoded besides numeric ones: the XML five, and those
.replace(/&lt;/gi, "<") * page titles and descriptions actually use, Latin-1 letters and typography.
.replace(/&gt;/gi, ">") */
.replace(/&quot;/gi, '"') const NAMED_ENTITIES: Record<string, string> = {
.replace(/&apos;/gi, "'") amp: "&",
.replace(/&#(\d+);/g, (_, dec) => String.fromCodePoint(Number(dec))) lt: "<",
.replace( gt: ">",
/&#x([0-9a-f]+);/gi, quot: '"',
(_, hex) => String.fromCodePoint(parseInt(hex, 16)), apos: "'",
nbsp: " ",
aelig: "æ",
AElig: "Æ",
oelig: "œ",
OElig: "Œ",
szlig: "ß",
oslash: "ø",
Oslash: "Ø",
eth: "ð",
ETH: "Ð",
thorn: "þ",
THORN: "Þ",
hellip: "…",
lsquo: "‘",
rsquo: "’",
sbquo: "‚",
ldquo: "“",
rdquo: "”",
bdquo: "„",
laquo: "«",
raquo: "»",
lsaquo: "‹",
rsaquo: "›",
ndash: "–",
mdash: "—",
bull: "•",
middot: "·",
dagger: "†",
Dagger: "‡",
prime: "′",
Prime: "″",
euro: "€",
pound: "£",
yen: "¥",
cent: "¢",
copy: "©",
reg: "®",
trade: "™",
deg: "°",
plusmn: "±",
times: "×",
divide: "÷",
micro: "µ",
para: "¶",
sect: "§",
iexcl: "¡",
iquest: "¿",
ordf: "ª",
ordm: "º",
sup2: "²",
sup3: "³",
frac14: "¼",
frac12: "½",
frac34: "¾",
shy: "­",
ensp: " ",
emsp: " ",
thinsp: " ",
zwnj: "‌",
zwj: "‍",
};
// The accented letters, `&eacute;` and the like, composed from their letter
// and accent rather than listed one by one.
for (
const [accent, mark, letters] of [
["grave", "̀", "aeiouAEIOU"],
["acute", "́", "aeiouyAEIOUY"],
["circ", "̂", "aeiouAEIOU"],
["tilde", "̃", "anoANO"],
["uml", "̈", "aeiouyAEIOUY"],
["ring", "̊", "aA"],
["cedil", "̧", "cC"],
]
) {
for (const letter of letters) {
NAMED_ENTITIES[letter + accent] = (letter + mark).normalize("NFC");
}
}
/** Returns the character of code point `n`, or undefined when there is none. */
function fromCodePoint(n: number): string | undefined {
if (n <= 0 || n > 0x10ffff || (n >= 0xd800 && n <= 0xdfff)) {
return undefined;
}
return String.fromCodePoint(n);
}
/**
* Decodes the entities in `str` in a single pass, so `&amp;quot;` reads
* `&quot;`. An entity it doesn't know, or a code point that doesn't exist, is
* left as written. Also used by `bandcamp-tralbum.ts` on the JSON a page
* carries in an attribute.
*/
export function decodeHtmlEntities(str: string): string {
return str.replace(
/&(?:#(\d{1,8})|#[xX]([0-9a-fA-F]{1,6})|([a-zA-Z][a-zA-Z0-9]{1,31}));/g,
(match, dec?: string, hex?: string, name?: string) => {
if (dec) return fromCodePoint(Number(dec)) ?? match;
if (hex) return fromCodePoint(parseInt(hex, 16)) ?? match;
return NAMED_ENTITIES[name!] ?? NAMED_ENTITIES[name!.toLowerCase()] ??
match;
},
); );
} }
/** /**
* Returns the `content` of the first `<meta>` tag whose `attr` equals `value` * Returns `raw` resolved against `base` as an absolute http(s) URL; undefined
* (either attribute ordering). Each tag is matched in isolation so * for a `data:` URI, another scheme or a URL that doesn't parse. Every URL a
* the `content` capture can never span across tag boundaries. * page names goes through it, so `/og.png` and `//cdn/og.png` work.
*/ */
function extractMetaContent( export function absoluteHttpUrl(
html: string, raw: string | undefined,
attr: string, base: string,
value: string,
): string | undefined { ): string | undefined {
const escaped = value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); if (!raw?.trim()) return undefined;
const keyRe = new RegExp(`\\b${attr}=["']${escaped}["']`, "i"); try {
const contentRe = /\bcontent=(["'])([\s\S]*?)\1/i; const url = new URL(raw.trim(), base);
const tagRe = /<meta\b[^>]*>/gi; return url.protocol === "http:" || url.protocol === "https:"
let m: RegExpExecArray | null; ? url.href
while ((m = tagRe.exec(html)) !== null) { : undefined;
const tag = m[0]; } catch {
if (!keyRe.test(tag)) continue;
const content = contentRe.exec(tag);
if (content) return decodeHtmlEntities(content[2]);
}
return undefined; return undefined;
}
} }
/** Returns the content of `<meta property="og:<tag>">`. */ /** A tag's attributes by lowercase name, values decoded; the first one wins. */
export function extractOgTag( type Attrs = Map<string, string>;
html: string,
tag: string, /**
): string | undefined { * Matches the start of a tag the scan reads, or of a comment. Quoted values
return extractMetaContent(html, "property", `og:${tag}`); * may hold `>`, but no tag spans a `<`, so each attempt stops at the next one
* and the scan stays linear in the page's size.
*/
const TAG_RE =
/<(meta|link|img|title|script)\b((?:[^<>"']|"[^"<]*"|'[^'<]*')*)>|<!--/gi;
/** Matches one attribute: unquoted, single- or double-quoted, or bare. */
const ATTR_RE =
/([^\s"'<>/=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+)))?/g;
function parseAttrs(source: string): Attrs {
const attrs: Attrs = new Map();
for (const m of source.matchAll(ATTR_RE)) {
const name = m[1].toLowerCase();
if (attrs.has(name)) continue;
attrs.set(name, decodeHtmlEntities(m[2] ?? m[3] ?? m[4] ?? ""));
}
return attrs;
}
/** What {@link scanTags} found, in page order. */
interface Scan {
metas: Attrs[];
links: Attrs[];
imgs: Attrs[];
/** The raw text of the first `<title>`. */
title?: string;
/** The bodies of the `application/ld+json` scripts. */
jsonLd: string[];
} }
/** /**
* Returns the content of `<meta name="…" content="…">` (both attribute * Collects the tags {@link parsePage} reads in one pass. Comments and the
* orderings). * bodies of scripts and the title are skipped over, so a tag written inside
* them isn't read; one left open runs to the end of the page, as in a browser.
*/ */
export function extractMetaName( function scanTags(html: string): Scan {
html: string, const scan: Scan = { metas: [], links: [], imgs: [], jsonLd: [] };
name: string, const re = new RegExp(TAG_RE);
): string | undefined { let m: RegExpExecArray | null;
return extractMetaContent(html, "name", name); while ((m = re.exec(html)) !== null) {
const tag = m[1]?.toLowerCase();
if (!tag) {
const end = html.indexOf("-->", re.lastIndex);
if (end < 0) break;
re.lastIndex = end + 3;
continue;
}
const attrs = parseAttrs(m[2]);
if (tag === "meta") scan.metas.push(attrs);
else if (tag === "link") scan.links.push(attrs);
else if (tag === "img") scan.imgs.push(attrs);
else {
const close = new RegExp(`</${tag}\\s*>`, "gi");
close.lastIndex = re.lastIndex;
const end = close.exec(html);
if (!end) break;
const body = html.slice(re.lastIndex, end.index);
if (tag === "title") scan.title ??= body;
else if (
attrs.get("type")?.trim().toLowerCase() === "application/ld+json"
) {
scan.jsonLd.push(body);
}
re.lastIndex = close.lastIndex;
}
}
return scan;
} }
/** Returns the text content of the `<title>` element. */ /** Returns `value` trimmed, or undefined when nothing is left. */
export function extractPageTitle(html: string): string | undefined { function nonBlank(value: string | undefined): string | undefined {
const match = html.match(/<title[^>]*>([^<]+)<\/title>/i); const trimmed = value?.trim();
return match ? decodeHtmlEntities(match[1].trim()) : undefined; return trimmed ? trimmed : undefined;
} }
// ── Brand color helpers ───────────────────────────────────────────────────────
/** /**
* Returns the page's `theme-color`. * Returns the page's `theme-color`, as written: pass it to
* * {@link normalizeCssColor}. Sites often ship one per color scheme with the
* Sites routinely ship the tag twice, scoped by `media`, and often list the * dark one first, so the unscoped tag wins, then the light one, then any.
* dark-scheme one first — so unlike `extractMetaName` this can't just take the
* first match. Prefer the unscoped tag, then the light-scheme one, then any.
*/ */
export function extractThemeColor(html: string): string | undefined { function themeColorOf(metas: Attrs[]): string | undefined {
const tagRe = /<meta\b[^>]*>/gi;
const contentRe = /\bcontent=(["'])([\s\S]*?)\1/i;
const mediaRe = /\bmedia=(["'])([\s\S]*?)\1/i;
let unscoped: string | undefined; let unscoped: string | undefined;
let light: string | undefined; let light: string | undefined;
let any: string | undefined; let any: string | undefined;
for (const meta of metas) {
let m: RegExpExecArray | null; if (meta.get("name")?.toLowerCase() !== "theme-color") continue;
while ((m = tagRe.exec(html)) !== null) { const content = nonBlank(meta.get("content"));
const tag = m[0];
if (!/\bname=["']theme-color["']/i.test(tag)) continue;
const content = contentRe.exec(tag)?.[2];
if (!content) continue; if (!content) continue;
const media = mediaRe.exec(tag)?.[2]; const media = meta.get("media");
if (!media) unscoped ??= content; if (!media) unscoped ??= content;
else if (/light/i.test(media)) light ??= content; else if (/light/i.test(media)) light ??= content;
any ??= content; any ??= content;
@@ -100,237 +248,182 @@ export function extractThemeColor(html: string): string | undefined {
return unscoped ?? light ?? any; return unscoped ?? light ?? any;
} }
/** Returns the `color` of `<link rel="mask-icon">` (Safari pinned tabs). */ /** Returns the lowercase tokens of a `<link>`'s `rel`. */
export function extractMaskIconColor(html: string): string | undefined { function relTokens(link: Attrs): string[] {
const linkRe = /<link[^>]+>/gi; return (link.get("rel") ?? "").toLowerCase().split(/\s+/);
let m: RegExpExecArray | null; }
while ((m = linkRe.exec(html)) !== null) {
const tag = m[0]; /**
if (!/\brel=["'][^"']*mask-icon[^"']*["']/i.test(tag)) continue; * Returns the `color` of `<link rel="mask-icon">`, Safari's pinned-tab icon,
const color = /\bcolor=(["'])([\s\S]*?)\1/i.exec(tag)?.[2]; * as written. The generic preview's last resort for a brand color.
*/
function maskIconColorOf(links: Attrs[]): string | undefined {
for (const link of links) {
if (!relTokens(link).includes("mask-icon")) continue;
const color = nonBlank(link.get("color"));
if (color) return color; if (color) return color;
} }
return undefined; return undefined;
} }
/** /**
* The handful of named colors that actually show up in `theme-color`. The full * Returns the absolute URL of the page's largest icon, among its `icon` and
* 148-name table isn't worth carrying for the long tail. * `apple-touch-icon` links, by declared `sizes`; the first one when none
* declares a size. A `mask-icon` is no candidate: it is a one-color shape for
* Safari to tint, black when drawn as is.
*
* An SVG icon wins over any size: it declares none, yet scales to any.
*/ */
const NAMED_COLORS: Record<string, string> = { function bestIconOf(links: Attrs[], baseUrl: string): string | undefined {
black: "#000000", let best: { href: string; area: number } | undefined;
white: "#ffffff", for (const link of links) {
red: "#ff0000", const rel = relTokens(link);
green: "#008000", if (
blue: "#0000ff", !rel.includes("icon") && !rel.includes("apple-touch-icon") &&
yellow: "#ffff00", !rel.includes("apple-touch-icon-precomposed")
orange: "#ffa500", ) continue;
purple: "#800080", const href = absoluteHttpUrl(link.get("href"), baseUrl);
gray: "#808080", if (!href) continue;
grey: "#808080", const sizes = (link.get("sizes") ?? "").toLowerCase();
silver: "#c0c0c0", const isSvg = /\.svg(\?|$)/i.test(href) ||
maroon: "#800000", link.get("type")?.toLowerCase() === "image/svg+xml" ||
navy: "#000080", sizes === "any";
teal: "#008080", const area = isSvg ? Number.MAX_SAFE_INTEGER : Math.max(
olive: "#808000", 0,
lime: "#00ff00", ...[...sizes.matchAll(/(\d+)x(\d+)/g)].map(([, w, h]) => +w * +h),
aqua: "#00ffff", );
cyan: "#00ffff", if (!best || area > best.area) best = { href, area };
fuchsia: "#ff00ff", }
magenta: "#ff00ff", return best?.href;
};
function clampByte(n: number): number {
return Math.max(0, Math.min(255, Math.round(n)));
} }
/** /**
* Normalizes a CSS color to canonical lowercase `#rrggbb`, or `undefined` when * Returned by the JSON-LD reading of {@link parsePage}, which the generic
* it isn't one of the forms sites actually use. Alpha is dropped — the color is * preview ranks below the page's OG and Twitter tags.
* only ever used as a tint over the app's own surface.
*/ */
export function normalizeCssColor(raw: string | undefined): string | undefined {
if (!raw) return undefined;
const value = raw.trim().toLowerCase();
const named = NAMED_COLORS[value];
if (named) return named;
const hex = /^#([0-9a-f]{3,8})$/.exec(value)?.[1];
if (hex) {
if (hex.length === 3 || hex.length === 4) {
const [r, g, b] = [...hex.slice(0, 3)];
return `#${r}${r}${g}${g}${b}${b}`;
}
if (hex.length === 6 || hex.length === 8) return `#${hex.slice(0, 6)}`;
return undefined;
}
const rgb = /^rgba?\(([^)]+)\)$/.exec(value)?.[1];
if (rgb) {
const parts = rgb.split(/[\s,/]+/).filter(Boolean).slice(0, 3);
if (parts.length !== 3) return undefined;
const bytes = parts.map((p) => {
const n = parseFloat(p);
if (!Number.isFinite(n)) return NaN;
return clampByte(p.endsWith("%") ? (n / 100) * 255 : n);
});
if (bytes.some(Number.isNaN)) return undefined;
return `#${bytes.map((b) => b.toString(16).padStart(2, "0")).join("")}`;
}
return undefined;
}
// ── JSON-LD helpers ───────────────────────────────────────────────────────────
/** What a page's JSON-LD says about it. */
export interface JsonLdResult { export interface JsonLdResult {
/** The node's `name` or `headline`. */ /** From `name`, or else `headline`. */
title?: string; title?: string;
/** The node's `description`. */
description?: string; description?: string;
/** /** From `image`, or else `thumbnailUrl` or `thumbnail`. */
* The node's `image`, `thumbnailUrl` or `thumbnail`, when it is an absolute
* URL.
*/
thumbnailUrl?: string; thumbnailUrl?: string;
} }
/**
* The JSON-LD types that describe the site around a page rather than the
* page itself. Their name is the site's or the company's, so a node of one of
* them is read only when no other node has a title or an image.
*/
const SITE_LD_TYPES = new Set([
"WebSite",
"Organization",
"Corporation",
"NewsMediaOrganization",
"Person",
"BreadcrumbList",
"SearchAction",
"ImageObject",
"SiteNavigationElement",
"WPHeader",
"WPFooter",
]);
function ldString(v: unknown): string | undefined { function ldString(v: unknown): string | undefined {
if (typeof v === "string" && v.trim()) return v.trim(); if (typeof v === "string") return nonBlank(v);
if (Array.isArray(v) && typeof v[0] === "string" && v[0].trim()) { if (Array.isArray(v) && typeof v[0] === "string") return nonBlank(v[0]);
return v[0].trim();
}
return undefined; return undefined;
} }
function ldImage(v: unknown): string | undefined { function ldImage(v: unknown, baseUrl: string): string | undefined {
if ( if (typeof v === "string") return absoluteHttpUrl(v, baseUrl);
typeof v === "string" && if (Array.isArray(v)) return ldImage(v[0], baseUrl);
(v.startsWith("http://") || v.startsWith("https://"))
) return v;
if (Array.isArray(v)) return ldImage(v[0]);
if (v && typeof v === "object") { if (v && typeof v === "object") {
const o = v as Record<string, unknown>; const o = v as Record<string, unknown>;
return ldImage(o.url ?? o.contentUrl); return ldImage(o.url ?? o.contentUrl, baseUrl);
} }
return undefined; return undefined;
} }
function ldExtractNode(data: unknown): JsonLdResult { /** Lists every node of `data`, through arrays and `@graph`, in order. */
if (Array.isArray(data)) { function ldNodes(data: unknown): Record<string, unknown>[] {
for (const item of data) { if (Array.isArray(data)) return data.flatMap(ldNodes);
const r = ldExtractNode(item); if (!data || typeof data !== "object") return [];
if (r.title || r.thumbnailUrl) return r;
}
return {};
}
if (!data || typeof data !== "object") return {};
const o = data as Record<string, unknown>; const o = data as Record<string, unknown>;
if (o["@graph"]) return ldExtractNode(o["@graph"]); return o["@graph"] ? ldNodes(o["@graph"]) : [o];
return {
title: ldString(o.name ?? o.headline),
description: ldString(o.description),
thumbnailUrl: ldImage(o.image ?? o.thumbnailUrl ?? o.thumbnail),
};
} }
/** /**
* Parses every `<script type="application/ld+json">` block and returns the first * Returns the first JSON-LD node about the page that has a title or an image,
* node that yields a title or image. Handles `@graph`, arrays, and the common * else the first about its site ({@link SITE_LD_TYPES}); empty when there is
* `image` shapes (string, string[], ImageObject). * none. An image may be a URL, a list of them or an `ImageObject`. Blocks that
* aren't valid JSON are skipped.
*/ */
export function extractJsonLd(html: string): JsonLdResult { function jsonLdOf(blocks: string[], baseUrl: string): JsonLdResult {
const pattern = let fallback: JsonLdResult | undefined;
/<script[^>]+type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi; for (const block of blocks) {
let match: RegExpExecArray | null; let data: unknown;
while ((match = pattern.exec(html)) !== null) {
try { try {
const result = ldExtractNode(JSON.parse(match[1])); data = JSON.parse(block);
if (result.title || result.thumbnailUrl) return result; } catch {
} catch { /* invalid JSON — skip */ } continue;
} }
return {}; for (const node of ldNodes(data)) {
const result: JsonLdResult = {
title: ldString(node.name ?? node.headline),
description: ldString(node.description),
thumbnailUrl: ldImage(
node.image ?? node.thumbnailUrl ?? node.thumbnail,
baseUrl,
),
};
if (!result.title && !result.thumbnailUrl) continue;
const types = [node["@type"]].flat();
if (!types.some((t) => typeof t === "string" && SITE_LD_TYPES.has(t))) {
return result;
}
fallback ??= result;
}
}
return fallback ?? {};
} }
/** /**
* Returns the `src` of the first `<img>` whose declared width or height is at * Returns the source of an `<img>`: its `src`, or for a lazy-loaded image
* least `minSize` pixels (default 200). Skips data URIs. Resolves relative URLs. * whose `src` is a `data:` placeholder, its `data-src` or `data-lazy-src`.
*/ */
export function extractLargeImage( function imageSource(img: Attrs, baseUrl: string): string | undefined {
html: string, const src = img.get("src")?.trim();
return src && !src.startsWith("data:")
? absoluteHttpUrl(src, baseUrl)
: absoluteHttpUrl(img.get("data-src") ?? img.get("data-lazy-src"), baseUrl);
}
/** Returns an `<img>`'s declared `name` dimension in pixels; 0 when none. */
function declaredSize(img: Attrs, name: "width" | "height"): number {
return parseInt(img.get(name) ?? "") || 0;
}
/**
* Returns the absolute URL of the first `<img>` whose declared width and height
* are both at least `minSize` pixels; an image without them doesn't count.
*/
function largeImageOf(
imgs: Attrs[],
baseUrl: string, baseUrl: string,
minSize = 200, minSize = 200,
): string | undefined { ): string | undefined {
const imgPattern = /<img[^>]+>/gi; for (const img of imgs) {
let match: RegExpExecArray | null; if (
while ((match = imgPattern.exec(html)) !== null) { declaredSize(img, "width") < minSize ||
const tag = match[0]; declaredSize(img, "height") < minSize
const src = /\bsrc=["']([^"']+)["']/i.exec(tag)?.[1]; ) continue;
if (!src || src.startsWith("data:")) continue; const src = imageSource(img, baseUrl);
const w = parseInt(/\bwidth=["']?(\d+)/i.exec(tag)?.[1] ?? "0"); if (src) return src;
const h = parseInt(/\bheight=["']?(\d+)/i.exec(tag)?.[1] ?? "0");
if (w >= minSize && h >= minSize) {
try {
return new URL(src, baseUrl).toString();
} catch {
continue;
}
}
} }
return undefined; return undefined;
} }
/** /** Filename words that mark an image as page furniture rather than content. */
* Collects all `<link rel="icon">` / `<link rel="apple-touch-icon">` tags, ranks
* them by declared size (largest wins), and returns the best resolved URL.
* Falls back to the first match when no `sizes` attribute is present.
*
* SVG icons are ranked above everything: they carry no `sizes` attribute, so
* they'd otherwise score 0 and lose to a 16×16 PNG, yet they scale cleanly to
* whatever size the placeholder renders them at.
*/
export function extractBestIcon(
html: string,
baseUrl: string,
): string | undefined {
const linkRe = /<link[^>]+>/gi;
const relRe = /\brel=["']([^"']+)["']/i;
const hrefRe = /\bhref=["']([^"']+)["']/i;
const sizesRe = /\bsizes=["']([^"']+)["']/i;
const candidates: { href: string; area: number }[] = [];
let m: RegExpExecArray | null;
while ((m = linkRe.exec(html)) !== null) {
const tag = m[0];
const rel = relRe.exec(tag)?.[1] ?? "";
if (!/\bicon\b/i.test(rel) && !/apple-touch-icon/i.test(rel)) continue;
const href = hrefRe.exec(tag)?.[1];
if (!href) continue;
const sizesStr = sizesRe.exec(tag)?.[1] ?? "";
const sm = sizesStr.match(/(\d+)x(\d+)/i);
const isSvg = /\.svg(\?|$)/i.test(href) ||
/\btype=["']image\/svg\+xml["']/i.test(tag);
const area = isSvg
? Number.MAX_SAFE_INTEGER
: sm
? parseInt(sm[1]) * parseInt(sm[2])
: 0;
try {
candidates.push({ href: new URL(href, baseUrl).toString(), area });
} catch {
continue;
}
}
if (candidates.length === 0) return undefined;
candidates.sort((a, b) => b.area - a.area);
return candidates[0].href;
}
/** Filename tokens that mark an image as UI chrome rather than content. */
const UI_IMAGE_KEYWORDS = new Set([ const UI_IMAGE_KEYWORDS = new Set([
"logo", "logo",
"icon", "icon",
@@ -347,77 +440,108 @@ const UI_IMAGE_KEYWORDS = new Set([
"pixel", "pixel",
"bg", "bg",
"background", "background",
"placeholder",
"spacer",
"blank",
"lazy",
"loading",
]); ]);
/** /**
* Checks whether `src` looks like UI chrome: an SVG, or a filename with one of * Checks whether `src` looks like page furniture: an SVG, or a filename with
* the {@link UI_IMAGE_KEYWORDS}. * one of the {@link UI_IMAGE_KEYWORDS} between its `-`, `_` or `.`.
*/ */
function isUiImage(src: string): boolean { function isUiImage(src: string): boolean {
if (/\.svg(\?|$)/i.test(src)) return true; if (/\.svg(\?|$)/i.test(src)) return true;
const filename = src.split("?")[0].split("/").pop() ?? ""; const filename = src.split("?")[0].split("/").pop() ?? "";
const baseName = filename.replace(/\.[^.]+$/, ""); // strip extension const baseName = filename.replace(/\.[^.]+$/, "");
// Split on common filename separators (-, _, .) and check each token
return baseName.toLowerCase().split(/[-_.]/).some((t) => return baseName.toLowerCase().split(/[-_.]/).some((t) =>
UI_IMAGE_KEYWORDS.has(t) UI_IMAGE_KEYWORDS.has(t)
); );
} }
/** /**
* Returns the `src` of the first `<img>` that looks like content rather than UI * Returns the absolute URL of the first `<img>` that looks like content rather
* chrome. Skips SVGs, data URIs, and images whose filename tokens match common * than a logo, icon, tracking pixel or other page furniture: one declared
* icon/logo/nav patterns (logo, icon, sprite, etc.). * smaller than 50 pixels either way doesn't count. The generic preview's last
* Resolves relative and protocol-relative URLs. * resort for a thumbnail.
*/ */
export function extractFirstContentImage( function firstContentImageOf(
html: string, imgs: Attrs[],
baseUrl: string, baseUrl: string,
): string | undefined { ): string | undefined {
const imgPattern = /<img[^>]+>/gi; for (const img of imgs) {
let match: RegExpExecArray | null; const w = declaredSize(img, "width");
while ((match = imgPattern.exec(html)) !== null) { const h = declaredSize(img, "height");
const tag = match[0]; if ((w && w < 50) || (h && h < 50)) continue;
const src = /\bsrc=["']([^"']+)["']/i.exec(tag)?.[1]; const src = imageSource(img, baseUrl);
if (!src || src.startsWith("data:")) continue; if (src && !isUiImage(src)) return src;
if (isUiImage(src)) continue;
try {
return new URL(src, baseUrl).toString();
} catch {
continue;
}
} }
return undefined; return undefined;
} }
/** /**
* Returns the `href` of the first `<link rel="…">` whose rel contains * Returned by {@link parsePage}: what a page says about itself, entities
* `relFragment`, resolved to an absolute URL using `baseUrl`. * decoded, values trimmed and URLs absolute.
*/ */
export function extractLinkHref( export interface Page {
html: string, /**
relFragment: string, * Returns the `content` of the first `<meta>` whose `property` or `name` is
baseUrl: string, * `key`, in any case: sites write `og:` tags under either attribute, and
): string | undefined { * Twitter's too.
const escaped = relFragment.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); */
const patterns = [ meta(key: string): string | undefined;
new RegExp( /** The text of the `<title>` element. */
`<link[^>]+rel=["'][^"']*${escaped}[^"']*["'][^>]+href=["']([^"']+)["']`, title?: string;
"i", jsonLd: JsonLdResult;
), /** The `theme-color`, as written: pass it to {@link normalizeCssColor}. */
new RegExp( themeColor?: string;
`<link[^>]+href=["']([^"']+)["'][^>]+rel=["'][^"']*${escaped}[^"']*["']`, /** The `color` of the Safari pinned-tab icon, as written. */
"i", maskIconColor?: string;
), /** The largest icon the page links, which the thumbnail placeholder draws. */
]; bestIcon?: string;
for (const pattern of patterns) { /** The first image declared at least 200 pixels both ways. */
const match = html.match(pattern); largeImage?: string;
if (match) { /** The first image that doesn't look like page furniture. */
try { firstContentImage?: string;
return new URL(match[1], baseUrl).toString(); }
} catch {
return undefined; /**
} * Reads `html`, served from `pageUrl`, in one pass. The generic preview and
} * the providers built on it read a page through it.
} */
return undefined; export function parsePage(html: string, pageUrl: string): Page {
const scan = scanTags(html);
const metas = new Map<string, string>();
for (const meta of scan.metas) {
const content = nonBlank(meta.get("content"));
if (!content) continue;
for (const attr of ["property", "name"]) {
const key = meta.get(attr)?.trim().toLowerCase();
if (key && !metas.has(key)) metas.set(key, content);
}
}
return {
meta: (key) => metas.get(key.toLowerCase()),
title: nonBlank(scan.title && decodeHtmlEntities(scan.title)),
jsonLd: jsonLdOf(scan.jsonLd, pageUrl),
themeColor: themeColorOf(scan.metas),
maskIconColor: maskIconColorOf(scan.links),
bestIcon: bestIconOf(scan.links, pageUrl),
largeImage: largeImageOf(scan.imgs, pageUrl),
firstContentImage: firstContentImageOf(scan.imgs, pageUrl),
};
}
/**
* Returns `raw` as a lowercase `#rrggbb`, from a hex, `rgb()`, `rgba()` or
* common named color; undefined for any other form. The generic preview
* stores it as the preview's `accentColor`, a tint over the app's own surface,
* so alpha is dropped, and a fully transparent one is no color at all.
*/
export function normalizeCssColor(raw: string | undefined): string | undefined {
const color = raw ? parseCssColor(raw) : undefined;
return color && color.a > 0 ? hexOf(color) : undefined;
} }

71
api/remote/image-proxy.ts
View File

@@ -1,17 +1,25 @@
/** /**
* The image proxy's fetch and cache: remote thumbnails are served through * Shows the thumbnails a browser can't load from their own site: those on plain
* this site, so viewers' browsers never contact the hosts. * http, which an https page would block, and those behind hotlink protection.
* It fetches and caches the images `GET /api/remote/image` serves, which the
* frontend routes a thumbnail through when it is on http or when loading it
* directly failed, and scales them down as the site's own are.
* *
* @module * @module
*/ */
import { fetchWithTimeout } from "./fetch.ts"; import { fetchWithTimeout, readBytesCapped } from "./fetch.ts";
import type { ImageWidth } from "../../shared/image.ts";
import { scaleImageBytes } from "../lib/ffmpeg.ts";
import { scalableImage } from "../lib/upload.ts";
/** An image fetched for the proxy. */ /**
* Returned to `GET /api/remote/image` by {@link fetchProxiedImage}, which sends
* it back under `contentType`.
*/
export interface ProxiedImage { export interface ProxiedImage {
/** The image's bytes. */
bytes: Uint8Array; bytes: Uint8Array;
/** The image's MIME type, as its host sent it. */ /** One of the raster types, without parameters. */
contentType: string; contentType: string;
} }
@@ -26,14 +34,12 @@ const PROXYABLE_IMAGE_TYPES = new Set([
"image/vnd.microsoft.icon", "image/vnd.microsoft.icon",
]); ]);
/** How long a proxied image is served from the cache. */
const IMAGE_CACHE_TTL_MS = 24 * 60 * 60 * 1000; const IMAGE_CACHE_TTL_MS = 24 * 60 * 60 * 1000;
/** The cache's size bound, in image bytes. */
const IMAGE_CACHE_MAX_BYTES = 32 * 1024 * 1024; const IMAGE_CACHE_MAX_BYTES = 32 * 1024 * 1024;
/** /**
* Proxied images, by URL: every viewer of a feed asks for the same thumbnails, * Proxied images, by URL and by the width they were scaled to, since every
* which were fetched from their hosts again for each one. Bounded by total * viewer of a feed asks for the same thumbnails. The least recently used go
* size; the least recently used go first. Map order is recency order. * first, and the Map's order is that of use.
*/ */
const imageCache = new Map< const imageCache = new Map<
string, string,
@@ -51,7 +57,7 @@ function readImageCache(
imageCacheBytes -= hit.bytes.byteLength; imageCacheBytes -= hit.bytes.byteLength;
return undefined; return undefined;
} }
imageCache.set(url, hit); // most recently used imageCache.set(url, hit); // Moves it to the end: most recently used.
return hit; return hit;
} }
@@ -77,7 +83,7 @@ function writeImageCache(
imageCacheBytes += bytes.byteLength; imageCacheBytes += bytes.byteLength;
} }
/** Forgets every proxied image (for the tests). */ /** Empties the cache, for the tests' reset between cases. */
export function clearImageCache(): void { export function clearImageCache(): void {
imageCache.clear(); imageCache.clear();
imageCacheBytes = 0; imageCacheBytes = 0;
@@ -85,20 +91,44 @@ export function clearImageCache(): void {
/** /**
* Returns the image at `url`, from the cache or else from its host; null when * Returns the image at `url`, from the cache or else from its host; null when
* the host answers with something that isn't a raster image of 5 MB at most. * the host answers with an error, or with anything but a raster image of 5 MB
* at most. With `width`, it comes as a WebP copy scaled down to it, or as it
* is when no copy can be made (`scalableImage`, and ffmpeg). The route checks
* `url` with `isValidHttpUrl` first.
* *
* @throws {Error} When the host can't be reached (see {@link fetchWithTimeout}). * @throws {Error} When the host can't be reached, as
* {@link fetchWithTimeout} reports it.
*/ */
export async function fetchProxiedImage( export async function fetchProxiedImage(
url: string, url: string,
width?: ImageWidth | null,
): Promise<ProxiedImage | null> { ): Promise<ProxiedImage | null> {
if (!width) return await fetchOriginal(url);
const key = `${url}#w${width}`;
const cached = readImageCache(key);
if (cached) return cached;
const original = await fetchOriginal(url);
if (!original) return null;
const how = scalableImage(original.bytes);
const bytes = how && await scaleImageBytes(original.bytes, width, how);
if (!bytes) return original;
writeImageCache(key, bytes, "image/webp");
return { bytes, contentType: "image/webp" };
}
/** Returns the image at `url` as its host sends it: see above. */
async function fetchOriginal(url: string): Promise<ProxiedImage | null> {
const cached = readImageCache(url); const cached = readImageCache(url);
if (cached) return cached; if (cached) return cached;
const res = await fetchWithTimeout(url, 8000); const res = await fetchWithTimeout(url, 8000);
// Raster formats only. The bytes go out under this site's origin, so an if (!res.ok) {
// SVG (which can carry script) or anything else a browser might execute await res.body?.cancel();
// would run with the app's origin if opened directly. return null;
}
// Raster formats only: the bytes go out under this site's origin, where an
// SVG, which can carry script, would run if opened directly.
const contentType = (res.headers.get("content-type") ?? "") const contentType = (res.headers.get("content-type") ?? "")
.split(";")[0].trim().toLowerCase(); .split(";")[0].trim().toLowerCase();
if (!PROXYABLE_IMAGE_TYPES.has(contentType)) { if (!PROXYABLE_IMAGE_TYPES.has(contentType)) {
@@ -106,14 +136,15 @@ export async function fetchProxiedImage(
return null; return null;
} }
const MAX_SIZE = 5 * 1024 * 1024; // 5 MB const MAX_SIZE = 5 * 1024 * 1024;
const contentLength = Number(res.headers.get("content-length") ?? "0"); const contentLength = Number(res.headers.get("content-length") ?? "0");
if (contentLength > MAX_SIZE) { if (contentLength > MAX_SIZE) {
await res.body?.cancel(); await res.body?.cancel();
return null; return null;
} }
const bytes = new Uint8Array(await res.arrayBuffer()); // Without a Content-Length, the cap is found by reading one byte past it.
const bytes = await readBytesCapped(res, MAX_SIZE + 1);
if (bytes.length > MAX_SIZE) return null; if (bytes.length > MAX_SIZE) return null;
writeImageCache(url, bytes, contentType); writeImageCache(url, bytes, contentType);

55
api/remote/providers/bandcamp.ts
View File

@@ -1,47 +1,64 @@
/** /**
* The Bandcamp link provider: previews from a release page's OG tags. * Previews Bandcamp links from their page, for `rich-content-service.ts`.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts"; import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts"; import { parsePage } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts"; import { fetchPage } from "../fetch.ts";
import { BANDCAMP_HOST_RE } from "../bandcamp-tralbum.ts";
import { genericPreview } from "./generic.ts";
const BANDCAMP_REGEX = /(?:^|\.)bandcamp\.com/; /**
* Returns the embed player's URL a page's `og:video` names, over https; only
* a player on bandcamp.com counts, since the frontend frames it.
*/
function embeddedPlayerUrl(raw: string | undefined): string | undefined {
if (!raw) return undefined;
try {
const url = new URL(raw);
if (
(url.protocol !== "https:" && url.protocol !== "http:") ||
url.hostname !== "bandcamp.com" ||
!url.pathname.startsWith("/EmbeddedPlayer/")
) return undefined;
url.protocol = "https:";
return url.href;
} catch {
return undefined;
}
}
/** The provider for bandcamp.com pages, with the embed player from `og:video`. */ /**
* Claims bandcamp.com and its subdomains, where artists have their pages, and
* takes the embed player's URL from the page's `og:video`. The native player
* fetches the page's streams itself at play time, through
* `GET /api/remote/bandcamp`.
*/
export const bandcampProvider: RichContentProvider = { export const bandcampProvider: RichContentProvider = {
name: "bandcamp", name: "bandcamp",
matches(url: string): boolean { matches(url: string): boolean {
try { try {
return BANDCAMP_REGEX.test(new URL(url).hostname); return BANDCAMP_HOST_RE.test(new URL(url).hostname);
} catch { } catch {
return false; return false;
} }
}, },
async fetch(url: string): Promise<RichContent> { async fetch(url: string): Promise<RichContent> {
const res = await fetchWithTimeout(url); const fetched = await fetchPage(url);
const contentType = res.headers.get("content-type") ?? ""; if (fetched.html === undefined) {
if (!contentType.startsWith("text/html")) {
await res.body?.cancel();
return { type: "bandcamp", siteName: "Bandcamp", url }; return { type: "bandcamp", siteName: "Bandcamp", url };
} }
const page = parsePage(fetched.html, fetched.url);
const html = await readTextCapped(res);
return { return {
...genericPreview(page, url, fetched.url),
type: "bandcamp", type: "bandcamp",
siteName: "Bandcamp", siteName: "Bandcamp",
url, embedUrl: embeddedPlayerUrl(page.meta("og:video")),
title: extractOgTag(html, "title"),
description: extractOgTag(html, "description"),
thumbnailUrl: extractOgTag(html, "image"),
embedUrl: extractOgTag(html, "video") ?? undefined,
}; };
}, },
}; };

164
api/remote/providers/generic.ts
View File

@@ -1,100 +1,92 @@
/** /**
* The fallback link provider, for any page no other provider claims. * Previews any page no other provider claims, for `rich-content-service.ts`,
* from whatever metadata its HTML carries. Its reading of a page,
* {@link genericPreview}, is also the base of the Bandcamp, SoundCloud and
* YouTube channel previews.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts"; import type { RichContentProvider } from "./provider.ts";
import { siteHost } from "../../../shared/url.ts";
import { import {
extractBestIcon, absoluteHttpUrl,
extractFirstContentImage,
extractJsonLd,
extractLargeImage,
extractMaskIconColor,
extractMetaName,
extractOgTag,
extractPageTitle,
extractThemeColor,
normalizeCssColor, normalizeCssColor,
type Page,
parsePage,
} from "../html.ts"; } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts"; import { fetchPage } from "../fetch.ts";
/** Returns the path of `url` without its trailing slashes, `/` for the root. */
function pathOf(url: string): string {
return new URL(url).pathname.replace(/\/+$/, "") || "/";
}
/** /**
* The provider for any HTML page, previewed from its OG and Twitter tags, JSON-LD, and markup: * Strips a trailing ` | Site`, ` – Site` or the like from a `<title>`, which
* title, description, artwork, icon and brand color. Matches every URL, so it * names the page and then its site; the site is in the preview's badge.
* must come last.
*/ */
export const genericProvider: RichContentProvider = { function withoutSiteSuffix(title: string, siteName: string): string {
name: "generic", const escaped = siteName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const suffix = new RegExp(`\\s+[|–—\\-·•]\\s+${escaped}\\s*$`, "i");
return title.replace(suffix, "") || title;
}
matches(_url: string): boolean { /**
return true; // fallback — always matches * Returns the preview of the page `url` that `page` read, served from
}, * `finalUrl` after redirects. Each field falls back from OG tags to Twitter
* tags, JSON-LD and the markup itself. The icon and brand color feed the
async fetch(url: string): Promise<RichContent> { * placeholder the frontend draws when there is no thumbnail.
const res = await fetchWithTimeout(url); */
const contentType = res.headers.get("content-type") ?? ""; export function genericPreview(
page: Page,
if (!contentType.startsWith("text/html")) { url: string,
await res.body?.cancel(); finalUrl: string,
return { type: "generic", url }; ): RichContent {
} // An og:url naming the home page from another page means the OG tags are
// the site's defaults, not this page's: their title and image are skipped.
const html = await readTextCapped(res); // Any other og:url is the page's canonical address, often with a slug the
const ld = extractJsonLd(html); // link left out, and its tags are the page's.
const ogUrl = page.meta("og:url");
// If og:url is present but points to a different page (e.g. the homepage), let useOg = true;
// the og: block is a site-level fallback, not page-specific metadata.
// In that case skip og:title and og:image so page-level signals win.
const ogUrl = extractOgTag(html, "url");
const useOg = !ogUrl || (() => {
try { try {
const ogPath = new URL(ogUrl).pathname.replace(/\/+$/, "") || "/"; useOg = !ogUrl || pathOf(new URL(ogUrl, finalUrl).href) !== "/" ||
const pagePath = new URL(url).pathname.replace(/\/+$/, "") || "/"; pathOf(finalUrl) === "/";
return ogPath === pagePath; } catch { /* an og:url that doesn't parse names no other page */ }
} catch {
return true;
}
})();
// Title: og:title (page-matched) → twitter:title → JSON-LD → <title> const ogSiteName = page.meta("og:site_name");
const title = (useOg ? extractOgTag(html, "title") : undefined) ?? const pageTitle = page.title && ogSiteName
extractMetaName(html, "twitter:title") ?? ? withoutSiteSuffix(page.title, ogSiteName)
ld.title ?? : page.title;
extractPageTitle(html); const title = (useOg ? page.meta("og:title") : undefined) ??
page.meta("twitter:title") ??
page.jsonLd.title ??
pageTitle;
// Site name: og:site_name → hostname const description = page.meta("og:description") ??
const siteName = extractOgTag(html, "site_name") ?? page.meta("twitter:description") ??
new URL(url).hostname.replace(/^www\./, ""); page.jsonLd.description ??
page.meta("description");
// Description: og:description → twitter:description → JSON-LD → <meta name="description"> // Never the favicon: a 16×16 icon cropped into a thumbnail looks broken,
const description = extractOgTag(html, "description") ?? // and the placeholder drawn instead shows the icon properly. A tag naming
extractMetaName(html, "twitter:description") ?? // no usable URL falls through to the next.
ld.description ?? const thumbnailUrl =
extractMetaName(html, "description"); (useOg ? absoluteHttpUrl(page.meta("og:image"), finalUrl) : undefined) ??
absoluteHttpUrl(page.meta("twitter:image"), finalUrl) ??
page.jsonLd.thumbnailUrl ??
page.largeImage ??
page.firstContentImage;
// Image: og:image (page-matched) → twitter:image → JSON-LD → large <img> → // `/favicon.ico` is a guess. When it doesn't load, the placeholder shows
// first content <img>. The chain deliberately stops there: a favicon is not // the site's initial.
// artwork, and pretending otherwise means every art-less page gets a 16×16 const faviconUrl = page.bestIcon ?? `${new URL(finalUrl).origin}/favicon.ico`;
// icon cover-cropped into a 128×72 box. No match here means "no artwork",
// and the frontend draws a generated placeholder instead.
const thumbnailUrl = (useOg ? extractOgTag(html, "image") : undefined) ??
extractMetaName(html, "twitter:image") ??
ld.thumbnailUrl ??
extractLargeImage(html, url) ??
extractFirstContentImage(html, url);
// Icon and brand color are independent facts about the page, so they're
// collected whether or not there's artwork. `/favicon.ico` is a guess; when
// it 404s the placeholder falls back to the site's initial.
const faviconUrl = extractBestIcon(html, url) ??
`${new URL(url).origin}/favicon.ico`;
const accentColor = normalizeCssColor( const accentColor = normalizeCssColor(
extractThemeColor(html) ?? page.themeColor ??
extractMetaName(html, "msapplication-TileColor") ?? page.meta("msapplication-TileColor") ??
extractMaskIconColor(html), page.maskIconColor,
); );
return { return {
@@ -105,7 +97,25 @@ export const genericProvider: RichContentProvider = {
thumbnailUrl, thumbnailUrl,
faviconUrl, faviconUrl,
accentColor, accentColor,
siteName, siteName: ogSiteName ?? siteHost(url),
}; };
}
/** Matches every URL, so it comes last in the service's list. */
export const genericProvider: RichContentProvider = {
name: "generic",
matches(_url: string): boolean {
return true;
},
async fetch(url: string): Promise<RichContent> {
const fetched = await fetchPage(url);
if (fetched.html === undefined) return { type: "generic", url };
return genericPreview(
parsePage(fetched.html, fetched.url),
url,
fetched.url,
);
}, },
}; };

19
api/remote/providers/provider.ts
View File

@@ -1,20 +1,27 @@
/** /**
* The interface every link provider implements. * Lets each kind of site have its own link previews. A provider implements this
* interface, and `rich-content-service.ts` asks its providers in turn and
* fetches a link's preview from the first that matches.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
/** Metadata for the links one kind of site serves. */ /**
* Implemented by each file of `remote/providers/`, and listed in priority order
* by `rich-content-service.ts`.
*/
export interface RichContentProvider { export interface RichContentProvider {
/** The provider's name, for logs. */
name: string; name: string;
/** Checks whether this provider handles `url`. */ /** Called on a URL that `isValidHttpUrl` accepted; never throws. */
matches(url: string): boolean; matches(url: string): boolean;
/** /**
* Fetches `url`'s preview. Rejects when the page can't be fetched; a page * Returns the preview of `url`, which ends up in the dump's `rich_content` or
* that isn't HTML yields a preview with no metadata. * the create form's preview. It rejects when the page can't be fetched or
* answers with an error status, and the service then falls back to a stub.
* A page that isn't HTML yields a preview without metadata. The service
* tidies what it returns, so a provider leaves text as the page wrote it.
*/ */
fetch(url: string): Promise<RichContent>; fetch(url: string): Promise<RichContent>;
} }

77
api/remote/providers/self.ts
View File

@@ -1,25 +1,25 @@
/** /**
* The link provider for this site's own pages, read from the database * Previews links to this site's own dump, user and playlist pages, as
* instead of fetched. * `parsePagePath` reads them, from the database, for
* `rich-content-service.ts`. No request goes out, so this works even when the
* site runs on a private host that `fetchWithTimeout` refuses.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts"; import type { RichContentProvider } from "./provider.ts";
import { getDump } from "../../services/dump-service.ts"; import { getDumpByPath } from "../../services/dump-service.ts";
import { getUserByUsername } from "../../services/user-service.ts"; import { resolveUsername } from "../../services/user-service.ts";
import { getPlaylistById } from "../../services/playlist-service.ts"; import { getPlaylistByPath } from "../../services/playlist-service.ts";
import { APIException } from "../../lib/api-exception.ts";
import { PUBLIC_URL, THEME_COLOR } from "../../config.ts"; import { PUBLIC_URL, THEME_COLOR } from "../../config.ts";
import { isSelfUrl } from "../../lib/self-url.ts"; import { isSelfUrl } from "../../lib/self-url.ts";
import { parsePagePath } from "../../../shared/paths.ts";
const DUMP_RE = /^\/dumps\/([^/]+)$/;
const USER_RE = /^\/users\/([^/]+)$/;
const PLAYLIST_RE = /^\/playlists\/([^/]+)$/;
/** /**
* Internal links carry gerbeur's own branding, so an art-less one still gets a * Gives a link to this site without a thumbnail a placeholder in the site's
* recognizable placeholder instead of a generic tint. * icon and color.
*/ */
const SELF_BRANDING = { const SELF_BRANDING = {
siteName: "gerbeur", siteName: "gerbeur",
@@ -28,9 +28,10 @@ const SELF_BRANDING = {
} as const; } as const;
/** /**
* The provider for links to this site's dump, user and playlist pages, read * Comes before the generic provider, and claims what `isSelfUrl` accepts. A
* from the database. Its `fetch` rejects with the services' `APIException` (404) for a * profile's tab previews as the profile. Its `fetch` rejects with a 404
* page that doesn't exist or is private. * `APIException` for a missing page, a private dump or a private playlist:
* the preview is stored in the linking dump, which others may see.
*/ */
export const selfProvider: RichContentProvider = { export const selfProvider: RichContentProvider = {
name: "self", name: "self",
@@ -44,14 +45,15 @@ export const selfProvider: RichContentProvider = {
}, },
fetch(url: string): Promise<RichContent> { fetch(url: string): Promise<RichContent> {
const { pathname } = new URL(url); // matches() let only these paths through.
const page = parsePagePath(new URL(url).pathname)!;
const user = resolveUsername(page.username);
const dumpMatch = pathname.match(DUMP_RE); if (page.kind === "dump") {
if (dumpMatch) { const dump = getDumpByPath(user.id, page.slug);
const dump = getDump(dumpMatch[1]);
let thumbnailUrl: string | undefined; let thumbnailUrl: string | undefined;
if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) { if (dump.kind === "file" && dump.fileMime?.startsWith("image/")) {
thumbnailUrl = `${PUBLIC_URL}/api/files/${dump.id}`; thumbnailUrl = `${PUBLIC_URL}/api/dumps/${dump.id}/file`;
} else if (dump.richContent?.thumbnailUrl) { } else if (dump.richContent?.thumbnailUrl) {
thumbnailUrl = dump.richContent.thumbnailUrl; thumbnailUrl = dump.richContent.thumbnailUrl;
} }
@@ -65,27 +67,13 @@ export const selfProvider: RichContentProvider = {
}); });
} }
const userMatch = pathname.match(USER_RE); if (page.kind === "playlist") {
if (userMatch) { const playlist = getPlaylistByPath(user.id, page.slug);
const user = getUserByUsername(userMatch[1]); if (!playlist.isPublic) {
const thumbnailUrl = user.avatarMime throw new APIException(404, "Playlist not found");
? `${PUBLIC_URL}/api/avatars/${user.id}`
: undefined;
return Promise.resolve({
type: "generic",
url,
...SELF_BRANDING,
title: user.username,
description: user.description,
thumbnailUrl,
});
} }
const playlistMatch = pathname.match(PLAYLIST_RE);
if (playlistMatch) {
const playlist = getPlaylistById(playlistMatch[1]);
const thumbnailUrl = playlist.imageMime const thumbnailUrl = playlist.imageMime
? `${PUBLIC_URL}/api/playlists/${playlist.id}/image` ? `${PUBLIC_URL}/api/playlists/${playlist.id}/cover`
: undefined; : undefined;
return Promise.resolve({ return Promise.resolve({
type: "generic", type: "generic",
@@ -97,7 +85,16 @@ export const selfProvider: RichContentProvider = {
}); });
} }
// Should not reach here if matches() is correct const thumbnailUrl = user.avatarMime
return Promise.resolve({ type: "generic", url }); ? `${PUBLIC_URL}/api/users/${user.id}/avatar`
: undefined;
return Promise.resolve({
type: "generic",
url,
...SELF_BRANDING,
title: user.username,
description: user.description,
thumbnailUrl,
});
}, },
}; };

66
api/remote/providers/soundcloud.ts
View File

@@ -1,46 +1,72 @@
/** /**
* The SoundCloud link provider: previews from a track page's OG tags. * Previews SoundCloud links from their page, for `rich-content-service.ts`.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts"; import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts"; import { parsePage } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts"; import { fetchPage } from "../fetch.ts";
import { genericPreview } from "./generic.ts";
/** The provider for soundcloud.com pages, with SoundCloud's embed player. */ /** SoundCloud's own hosts, the short links' `on.` included. */
const SOUNDCLOUD_HOSTS = new Set([
"soundcloud.com",
"www.soundcloud.com",
"m.soundcloud.com",
"on.soundcloud.com",
]);
/**
* Returns the soundcloud.com address of the page, which the embed player
* takes: its `og:url`, or else the URL it was served from. A short link only
* redirects, so it is never the answer; tracking parameters are dropped.
*/
function pageAddress(ogUrl: string | undefined, finalUrl: string): string {
for (const raw of [ogUrl, finalUrl]) {
if (!raw) continue;
try {
const url = new URL(raw);
if (
!SOUNDCLOUD_HOSTS.has(url.hostname) ||
url.hostname === "on.soundcloud.com"
) continue;
return `https://soundcloud.com${url.pathname}`;
} catch {
continue;
}
}
return finalUrl;
}
/**
* Claims SoundCloud's hosts, short links included, and builds the embed
* player's URL from the page's address.
*/
export const soundcloudProvider: RichContentProvider = { export const soundcloudProvider: RichContentProvider = {
name: "soundcloud", name: "soundcloud",
matches(url: string): boolean { matches(url: string): boolean {
try { try {
return new URL(url).hostname === "soundcloud.com"; return SOUNDCLOUD_HOSTS.has(new URL(url).hostname);
} catch { } catch {
return false; return false;
} }
}, },
async fetch(url: string): Promise<RichContent> { async fetch(url: string): Promise<RichContent> {
const res = await fetchWithTimeout(url); const fetched = await fetchPage(url);
const contentType = res.headers.get("content-type") ?? ""; const page = fetched.html === undefined
? undefined
if (!contentType.startsWith("text/html")) { : parsePage(fetched.html, fetched.url);
await res.body?.cancel(); const address = pageAddress(page?.meta("og:url"), fetched.url);
return { type: "soundcloud", siteName: "SoundCloud", url };
}
const html = await readTextCapped(res);
return { return {
...(page ? genericPreview(page, url, fetched.url) : { url }),
type: "soundcloud", type: "soundcloud",
siteName: "SoundCloud", siteName: "SoundCloud",
url,
title: extractOgTag(html, "title"),
description: extractOgTag(html, "description"),
thumbnailUrl: extractOgTag(html, "image"),
embedUrl: `https://w.soundcloud.com/player/?url=${ embedUrl: `https://w.soundcloud.com/player/?url=${
encodeURIComponent(url) encodeURIComponent(address)
}&visual=true&auto_play=false`, }&visual=true&auto_play=false`,
}; };
}, },

96
api/remote/providers/youtube.ts
View File

@@ -1,50 +1,43 @@
/** /**
* The YouTube link provider: videos, playlists and channels. * Previews YouTube videos, playlists and channels, for
* `rich-content-service.ts`. Videos and playlists are read from YouTube's
* oEmbed endpoint, channels from their page.
* *
* @module * @module
*/ */
import type { RichContent } from "../../../shared/dump.ts"; import type { RichContent } from "../../../shared/dump.ts";
import type { RichContentProvider } from "./provider.ts"; import type { RichContentProvider } from "./provider.ts";
import { extractOgTag } from "../html.ts"; import { parsePage } from "../html.ts";
import { fetchWithTimeout, readTextCapped } from "../fetch.ts"; import { fetchPage, fetchWithTimeout } from "../fetch.ts";
import { youtubeHost, youtubeVideoId } from "../../lib/canonical-url.ts";
import { siteHostOf } from "../../../shared/url.ts";
import { genericPreview } from "./generic.ts";
/** Returns the id of the video a youtu.be or youtube.com URL points at. */ /** Returns the id of the video `url` points at, in any shape YouTube uses. */
function extractVideoId(url: string): string | null { function extractVideoId(url: string): string | null {
try { try {
const u = new URL(url); const u = new URL(url);
if (u.hostname === "youtu.be") { const host = siteHostOf(u);
return u.pathname.slice(1).split("/")[0] || null; return youtubeVideoId(host, u.pathname, u.searchParams);
}
if (u.hostname === "youtube.com" || u.hostname === "www.youtube.com") {
if (u.pathname === "/watch" || u.pathname.startsWith("/watch?")) {
return u.searchParams.get("v");
}
if (
u.pathname.startsWith("/embed/") || u.pathname.startsWith("/shorts/")
) {
return u.pathname.split("/")[2] || null;
}
}
} catch { } catch {
// invalid URL
}
return null; return null;
}
} }
/** Returns the `list` parameter of a URL: the playlist it plays. */ /** Returns the `list` parameter of a YouTube URL, and null for any other. */
function extractPlaylistId(url: string): string | null { function extractPlaylistId(url: string): string | null {
try { try {
return new URL(url).searchParams.get("list"); const u = new URL(url);
return youtubeHost(u) ? u.searchParams.get("list") : null;
} catch { } catch {
return null; return null;
} }
} }
/** /**
* Returns the playback offset in seconds of a YouTube URL. * Returns the start time of a YouTube URL in seconds, from `t` (`90`, `90s`,
* Accepts `t` (plain seconds, `90s`, or `1h2m3s` / `20m5s` forms) and `start` * `1h2m3s`) or `start`; null when there is none that parses.
* (plain seconds). Returns null when absent or unparseable.
*/ */
function extractStartSeconds(url: string): number | null { function extractStartSeconds(url: string): number | null {
let raw: string | null; let raw: string | null;
@@ -66,22 +59,15 @@ function extractStartSeconds(url: string): number | null {
} }
/** /**
* Returns the path of a channel URL: /channel/UC…, /@handle, /c/name or * Returns the path of a YouTube channel URL (`/channel/UC…`, `/@handle`,
* /user/name. * `/c/name` or `/user/name`); null for any other URL.
*/ */
function extractChannelPath(url: string): string | null { function extractChannelPath(url: string): string | null {
try { try {
const u = new URL(url); const u = new URL(url);
if (u.hostname === "youtube.com" || u.hostname === "www.youtube.com") { if (youtubeHost(u) && /^\/(channel\/|@|c\/|user\/)/.test(u.pathname)) {
if (
u.pathname.startsWith("/channel/") ||
u.pathname.startsWith("/@") ||
u.pathname.startsWith("/c/") ||
u.pathname.startsWith("/user/")
) {
return u.pathname; return u.pathname;
} }
}
} catch { } catch {
// invalid URL // invalid URL
} }
@@ -90,7 +76,7 @@ function extractChannelPath(url: string): string | null {
/** /**
* Fetches the title and thumbnail of `url` from YouTube's oEmbed endpoint; * Fetches the title and thumbnail of `url` from YouTube's oEmbed endpoint;
* empty when that fails. * empty when that fails, since the preview can do without them.
*/ */
async function fetchOEmbed( async function fetchOEmbed(
url: string, url: string,
@@ -109,14 +95,16 @@ async function fetchOEmbed(
} }
await res.body?.cancel(); await res.body?.cancel();
} catch { } catch {
// oembed failed — carry on // YouTube unreachable, or its answer not JSON.
} }
return {}; return {};
} }
/** /**
* The provider for YouTube videos (with their playlist and start time in the embed), * Comes first in the service's list. A video's embed keeps the link's playlist
* playlists (as a `videoseries` embed) and channels (not embeddable). * and start time, a playlist embeds as a `videoseries`, and a channel gets no
* embed. Its `fetch` never rejects: whatever YouTube doesn't answer is left
* out.
*/ */
export const youtubeProvider: RichContentProvider = { export const youtubeProvider: RichContentProvider = {
name: "youtube", name: "youtube",
@@ -134,34 +122,21 @@ export const youtubeProvider: RichContentProvider = {
const listId = extractPlaylistId(url); const listId = extractPlaylistId(url);
const channelPath = extractChannelPath(url); const channelPath = extractChannelPath(url);
// ── Channel ───────────────────────────────────────────────────────────────
if (channelPath && !videoId) { if (channelPath && !videoId) {
// oEmbed doesn't support channel URLs — scrape og:image from the page instead // oEmbed doesn't take channel URLs.
let title: string | undefined; let channel: RichContent | undefined;
let thumbnailUrl: string | undefined;
try { try {
const res = await fetchWithTimeout(url); const fetched = await fetchPage(url);
if (res.ok) { if (fetched.html !== undefined) {
const html = await readTextCapped(res); const page = parsePage(fetched.html, fetched.url);
title = extractOgTag(html, "title"); channel = genericPreview(page, url, fetched.url);
thumbnailUrl = extractOgTag(html, "image");
} else {
await res.body?.cancel();
} }
} catch { } catch {
// scrape failed — carry on with undefined values // YouTube unreachable: the preview goes without.
} }
return { return { ...channel, type: "youtube", siteName: "YouTube", url };
type: "youtube",
siteName: "YouTube",
url,
title,
thumbnailUrl,
// channels are not embeddable as a player
};
} }
// ── Playlist (no specific video) ──────────────────────────────────────────
if (listId && !videoId) { if (listId && !videoId) {
const { title, thumbnailUrl } = await fetchOEmbed(url); const { title, thumbnailUrl } = await fetchOEmbed(url);
return { return {
@@ -175,7 +150,7 @@ export const youtubeProvider: RichContentProvider = {
}; };
} }
// ── Video (with optional playlist context) ──────────────────────────────── // Built, not taken from oEmbed, so it is there even when oEmbed fails.
const thumbnailUrl = `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`; const thumbnailUrl = `https://img.youtube.com/vi/${videoId}/hqdefault.jpg`;
const { title } = await fetchOEmbed(url); const { title } = await fetchOEmbed(url);
@@ -188,7 +163,6 @@ export const youtubeProvider: RichContentProvider = {
type: "youtube", type: "youtube",
siteName: "YouTube", siteName: "YouTube",
url, url,
videoId: videoId!,
title, title,
thumbnailUrl, thumbnailUrl,
embedUrl: `https://www.youtube.com/embed/${videoId}?${embedParams}`, embedUrl: `https://www.youtube.com/embed/${videoId}?${embedParams}`,

36
api/remote/public-address.ts
View File

@@ -1,8 +1,8 @@
/** /**
* Guards for server-side fetches of user-supplied URLs (link previews, the * Keeps a user-supplied URL from making the server fetch what only it can
* image proxy, Bandcamp resolution): the server must not become a way to reach * reach: loopback, the LAN, container networks or the cloud metadata endpoint
* addresses only it can see — loopback, the LAN, container networks, or the * at 169.254.169.254. It tells private addresses from public ones, and
* cloud metadata endpoint at 169.254.169.254. * `fetch.ts` checks every hop with it.
* *
* @module * @module
*/ */
@@ -83,8 +83,9 @@ function isPrivateIPv6(b: number[]): boolean {
} }
/** /**
* Checks whether `ip` is an IP literal outside the public internet. Non-IPs * Checks whether `ip` is an IPv4 or IPv6 address outside the public internet.
* are false. * An IPv4 address carried in IPv6 is judged as IPv4. Anything that isn't an IP
* address is false, so a hostname must be resolved first.
*/ */
export function isPrivateAddress(ip: string): boolean { export function isPrivateAddress(ip: string): boolean {
const v4 = parseIPv4(ip); const v4 = parseIPv4(ip);
@@ -99,16 +100,17 @@ function isIpLiteral(host: string): boolean {
/** /**
* Returns `url`'s hostname without the brackets `URL.hostname` keeps around * Returns `url`'s hostname without the brackets `URL.hostname` keeps around
* an IPv6 literal. * an IPv6 literal: the address checks here read it, and `fetch.ts` hands it
* to curl's `--resolve`.
*/ */
function bareHostname(url: URL): string { export function bareHostname(url: URL): string {
return url.hostname.replace(/^\[(.*)\]$/, "$1"); return url.hostname.replace(/^\[(.*)\]$/, "$1");
} }
/** /**
* Checks whether `url`'s host is private without DNS: a private IP literal, * Checks whether the host of `url` is private without a DNS lookup: a private
* or a name that is local by definition. Names still need resolving (see * IP address, or `localhost` and its subdomains. `isValidHttpUrl` uses it to
* resolvePublicAddress) — this only rejects what is obvious without DNS. * reject a link early; only {@link resolvePublicAddress} is a real guard.
*/ */
export function isPrivateHostname(url: URL): boolean { export function isPrivateHostname(url: URL): boolean {
const host = bareHostname(url).toLowerCase(); const host = bareHostname(url).toLowerCase();
@@ -117,13 +119,13 @@ export function isPrivateHostname(url: URL): boolean {
} }
/** /**
* Resolves `url`'s host and returns one of its addresses, or throws if the * Resolves the host of `url` and returns one of its addresses, all of which
* host is private or any address it resolves to is (a name with both a public * must be public: the fetch may pick any of them. `fetchWithTimeout` calls it
* and a private record is refused outright, since the fetch may pick either). * before each hop.
* *
* The fetch itself resolves again, so a DNS server answering differently the * Only the curl fallback connects to the address returned. Deno's fetch
* second time (rebinding) isn't fully excluded; the curl fallback pins the * resolves the name again, so a DNS server that answers differently the second
* address this returns. * time (rebinding) can still get through.
* *
* @throws {Error} When the host is private, can't be resolved, or resolves to * @throws {Error} When the host is private, can't be resolved, or resolves to
* a private address. * a private address.

Some files were not shown because too many files have changed in this diff Show More