multi-topic, publish from gh-pages branch
All checks were successful
Build and deploy static pages / build-and-push (push) Successful in 19s
All checks were successful
Build and deploy static pages / build-and-push (push) Successful in 19s
This commit is contained in:
460
README.md
460
README.md
@@ -11,6 +11,8 @@ Hugo-generated website that helps researchers:
|
||||
|
||||
The site is domain-driven: a researcher picks a research area (e.g. "edge and cloud systems") and the assistant builds a curated, ranked, up-to-date snapshot of where to publish, when to submit, and what to read. The output is a static Hugo site that can be rebuilt on demand.
|
||||
|
||||
Multiple research topics live under the same Hugo instance as subsites (`/cloud-edge/`, `/embedded-ai/`, …). Each topic has its own venue list, deadlines, digests, and calendar. Layouts, theme, and JavaScript are shared.
|
||||
|
||||
Work is split between **automated scripts** (data fetching, Hugo content generation) and **agent tasks** (domain curation, paper selection, deadline gap-filling). The README below documents both halves so the site can be kept fresh over time.
|
||||
|
||||
---
|
||||
@@ -23,25 +25,30 @@ publish-assistant/
|
||||
│ ├── hugo.toml
|
||||
│ ├── themes/PaperMod/ # PaperMod theme (git submodule)
|
||||
│ ├── layouts/
|
||||
│ │ ├── _default/calendar.html # FullCalendar layout override
|
||||
│ │ └── partials/extend_head.html
|
||||
│ │ ├── _default/calendar.html # FullCalendar layout
|
||||
│ │ ├── _default/digests.html # Digest list layout
|
||||
│ │ └── partials/
|
||||
│ │ ├── header.html # Section-aware nav override
|
||||
│ │ └── extend_head.html # Calendar CSS injection
|
||||
│ ├── content/ # Generated — do not edit by hand
|
||||
│ │ ├── venues/
|
||||
│ │ │ ├── _index.md # Venues overview (generated)
|
||||
│ │ │ ├── conferences/ # One subdir per tracked conference
|
||||
│ │ │ └── journals/ # One subdir per tracked journal
|
||||
│ │ ├── calendar/ # Aggregated deadline view + FullCalendar
|
||||
│ │ └── digests/ # Per-issue paper digests
|
||||
│ └── data/ # Structured data — edit these
|
||||
│ ├── venues.yaml # Master venue list ← primary edit target
|
||||
│ ├── deadlines.yaml # Deadline cache; manual entries preserved
|
||||
│ ├── best_papers.yaml # Best-paper awards (written by pa-fetch-best-papers)
|
||||
│ ├── rankings/
|
||||
│ │ ├── icore.csv # ICORE rankings cache
|
||||
│ │ └── scimago.csv # SCImago rankings cache (optional)
|
||||
│ └── papers/
|
||||
│ ├── <V>-<Y>-candidates.yaml # Full paper list (pa-fetch-papers)
|
||||
│ └── <V>-<Y>-digest.yaml # Curated selection (agent, Task 3)
|
||||
│ │ ├── _index.md # Global landing page
|
||||
│ │ └── <topic>/ # One subdir per topic, e.g. cloud-edge/
|
||||
│ │ ├── _index.md # Topic home page
|
||||
│ │ ├── venues/ # Venue pages (generated)
|
||||
│ │ ├── calendar/ # Deadline calendar (generated)
|
||||
│ │ └── digests/ # Paper digests (generated)
|
||||
│ └── data/
|
||||
│ ├── topics.yaml # Topic registry (nav + build loop)
|
||||
│ ├── rankings/ # Shared across all topics
|
||||
│ │ ├── icore.csv # ICORE conference rankings
|
||||
│ │ └── scimago.csv # SCImago journal rankings (optional)
|
||||
│ └── <topic>/ # One subdir per topic, e.g. cloud-edge/
|
||||
│ ├── venues.yaml # Master venue list ← primary edit target
|
||||
│ ├── deadlines.yaml # Deadline cache; manual entries preserved
|
||||
│ ├── best_papers.yaml # Best-paper awards
|
||||
│ └── papers/
|
||||
│ ├── <V>-<Y>-candidates.yaml # Full paper list (pa-fetch-papers)
|
||||
│ └── <V>-<Y>-digest.yaml # Curated selection (agent, Task 3)
|
||||
├── src/publish_assistant/ # Python package
|
||||
│ ├── fetch_icore.py
|
||||
│ ├── fetch_scimago.py
|
||||
@@ -55,6 +62,8 @@ publish-assistant/
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Topic isolation**: each topic owns its `site/data/<topic>/` directory (venues, deadlines, papers) and generates into `site/content/<topic>/`. The shared `site/data/rankings/` CSVs are reused by every topic. The nav bar automatically shows topic-relative Venues / Calendar / Digests links when inside a topic, and lists all topics from `topics.yaml` on the root page.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
@@ -71,22 +80,24 @@ uv run pa-fetch-papers --venue OSDI --year 2025
|
||||
uv run pa-generate
|
||||
|
||||
# Local dev server:
|
||||
hugo server --source site
|
||||
./build.sh --dev
|
||||
# or, to process one topic without re-fetching:
|
||||
./build.sh --skip-rankings --topic cloud-edge --dev
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Sources
|
||||
|
||||
| Source | What it provides | Automatable? | Known issues |
|
||||
| ---------------------------------------------------------------------------- | ---------------------------------------- | ------------ | --------------------------------------------------------------------------------------- |
|
||||
| [ICORE](https://portal.core.edu.au/conf-ranks/) | Conference rankings (A*, A, B, C) | Yes | Pagination uses `javascript:jumpPage('N')` — handled in `fetch_icore.py` |
|
||||
| [SCImago](https://www.scimagojr.com/) | Journal quartiles, SJR, H-index | Blocked | Anti-bot returns HTML; add data manually to `venues.yaml` under `scimago_quartile` etc. |
|
||||
| [DBLP](https://dblp.org/) | Paper metadata by venue | Yes | Use `dblp_key` field in `venues.yaml` |
|
||||
| [OpenAlex](https://openalex.org/) | Papers, open-access links | Yes | Fallback when DBLP is thin |
|
||||
| [WikiCFP](http://wikicfp.com/) | Submission deadlines | Partially | See detailed notes below |
|
||||
| [Conference websites](.) | Authoritative deadlines | Partially | See Task 2 |
|
||||
| [jeffhuang.com/best_paper_awards/](https://jeffhuang.com/best_paper_awards/) | Best paper awards since 1996, ~32 venues | Yes | Manually maintained; run `pa-fetch-best-papers` annually |
|
||||
| Source | What it provides | Automatable? | Known issues |
|
||||
| --- | --- | --- | --- |
|
||||
| [ICORE](https://portal.core.edu.au/conf-ranks/) | Conference rankings (A*, A, B, C) | Yes | Pagination uses `javascript:jumpPage('N')` — handled in `fetch_icore.py` |
|
||||
| [SCImago](https://www.scimagojr.com/) | Journal quartiles, SJR, H-index | Blocked | Anti-bot returns HTML; add data manually to `venues.yaml` under `scimago_quartile` etc. |
|
||||
| [DBLP](https://dblp.org/) | Paper metadata by venue | Yes | Use `dblp_key` field in `venues.yaml` |
|
||||
| [OpenAlex](https://openalex.org/) | Papers, open-access links | Yes | Fallback when DBLP is thin |
|
||||
| [WikiCFP](http://wikicfp.com/) | Submission deadlines | Partially | See detailed notes below |
|
||||
| [Conference websites](.) | Authoritative deadlines | Partially | See Task 2 |
|
||||
| [jeffhuang.com/best_paper_awards/](https://jeffhuang.com/best_paper_awards/) | Best paper awards since 1996, ~32 venues | Yes | Manually maintained; run `pa-fetch-best-papers` annually |
|
||||
|
||||
---
|
||||
|
||||
@@ -96,22 +107,22 @@ WikiCFP is the primary deadline source but has several quirks that required work
|
||||
|
||||
**HTML structure**: Detail pages use `<th>` for row labels (not `<td>`). The parser in `fetch_cfp_details()` specifically looks for `<th>` + `<td>` pairs. Do not revert to `find_all("td")` — it will find zero deadline rows.
|
||||
|
||||
**Direct ID lookup**: Add `wikicfp_id: "<event_id>"` to a venue entry in `venues.yaml` to skip the search and fetch that page directly. This avoids wrong matches on common acronyms. Verified IDs for this domain:
|
||||
**Direct ID lookup**: Add `wikicfp_id: "<event_id>"` to a venue entry in `venues.yaml` to skip the search and fetch that page directly. This avoids wrong matches on common acronyms. Verified IDs for the cloud-edge topic:
|
||||
|
||||
| Venue | WikiCFP event ID |
|
||||
| ---------- | ---------------- |
|
||||
| SOSP | 191399 |
|
||||
| EuroSys | 186524 |
|
||||
| SoCC | 191071 |
|
||||
| Middleware | 190153 |
|
||||
| IPDPS | 189093 |
|
||||
| HPDC | 191029 |
|
||||
| Venue | WikiCFP event ID |
|
||||
| --- | --- |
|
||||
| SOSP | 191399 |
|
||||
| EuroSys | 186524 |
|
||||
| SoCC | 191071 |
|
||||
| Middleware | 190153 |
|
||||
| IPDPS | 189093 |
|
||||
| HPDC | 191029 |
|
||||
|
||||
**Skipping search**: Set `wikicfp_id: false` to skip WikiCFP entirely for a venue (e.g., ATC, SC, SEC — where the search returns wrong events). Deadlines for these must be filled manually.
|
||||
|
||||
**Conferences not on WikiCFP** (for edge/cloud systems domain): OSDI, NSDI, USENIX ATC, SC, MobiSys, SEC. Use `wikicfp_id: false` for all of them.
|
||||
**Conferences not on WikiCFP** (for systems/networking): OSDI, NSDI, USENIX ATC, SC, MobiSys, SEC. Use `wikicfp_id: false` for all of them.
|
||||
|
||||
**Manual deadline entries**: Add entries with `source: manual` to `site/data/deadlines.yaml`. The fetcher preserves all `source: manual` entries across runs. Format:
|
||||
**Manual deadline entries**: Add entries with `source: manual` to `site/data/<topic>/deadlines.yaml`. The fetcher preserves all `source: manual` entries across runs. Format:
|
||||
```yaml
|
||||
ATC:
|
||||
source: manual
|
||||
@@ -131,7 +142,7 @@ ATC:
|
||||
conferences:
|
||||
- acronym: SOSP
|
||||
full_name: "ACM Symposium on Operating Systems Principles"
|
||||
domain: [edge-and-cloud, operating-systems, distributed-systems]
|
||||
domain: [<topic-slug>, operating-systems, distributed-systems]
|
||||
url: "https://sigops.org/s/conferences/sosp/"
|
||||
dblp_key: "conf/sosp"
|
||||
wikicfp_id: "191399" # direct lookup; omit to use search; false to skip entirely
|
||||
@@ -139,7 +150,7 @@ conferences:
|
||||
journals:
|
||||
- acronym: TPDS
|
||||
full_name: "IEEE Transactions on Parallel and Distributed Systems"
|
||||
domain: [edge-and-cloud, parallel-computing, distributed-systems]
|
||||
domain: [<topic-slug>, parallel-computing, distributed-systems]
|
||||
issn: "1045-9219"
|
||||
url: "https://www.computer.org/csdl/journal/td"
|
||||
dblp_key: "journals/tpds"
|
||||
@@ -153,14 +164,30 @@ journals:
|
||||
|
||||
---
|
||||
|
||||
## `topics.yaml` Schema
|
||||
|
||||
```yaml
|
||||
topics:
|
||||
- slug: cloud-edge
|
||||
title: "Edge and Cloud Systems"
|
||||
description: "Conferences and journals for edge computing, cloud systems, and distributed systems."
|
||||
- slug: embedded-ai
|
||||
title: "Embedded AI"
|
||||
description: "Conferences and journals for on-device ML, edge inference, and TinyML."
|
||||
```
|
||||
|
||||
The `slug` must match the directory names under `site/data/` and `site/content/`. It also becomes the URL prefix (`/cloud-edge/`, `/embedded-ai/`). The `title` appears in the nav bar on the root page and on the topic's section home.
|
||||
|
||||
---
|
||||
|
||||
## Scripts
|
||||
|
||||
All scripts are installed as CLI entry points by `uv sync`.
|
||||
|
||||
### `pa-fetch-icore`
|
||||
Downloads ICORE rankings. Handles the portal's JavaScript-based pagination.
|
||||
Downloads ICORE rankings. Shared across all topics — run once per build.
|
||||
```bash
|
||||
uv run pa-fetch-icore # fetch all
|
||||
uv run pa-fetch-icore # fetch all
|
||||
uv run pa-fetch-icore --query "distributed systems"
|
||||
```
|
||||
|
||||
@@ -172,14 +199,16 @@ uv run pa-fetch-scimago --area 1705 # networks
|
||||
```
|
||||
|
||||
### `pa-fetch-deadlines`
|
||||
Fetches submission deadlines from WikiCFP. Preserves `source: manual` entries.
|
||||
Fetches submission deadlines from WikiCFP for a specific topic. Preserves `source: manual` entries.
|
||||
```bash
|
||||
uv run pa-fetch-deadlines
|
||||
uv run pa-fetch-deadlines \
|
||||
--venues site/data/cloud-edge/venues.yaml \
|
||||
--output site/data/cloud-edge/deadlines.yaml
|
||||
```
|
||||
After running: check `site/data/deadlines.yaml` for the `missing:` list, then do Task 2.
|
||||
After running: check `deadlines.yaml` for the `missing:` list, then do Task 2.
|
||||
|
||||
### `pa-fetch-best-papers`
|
||||
Scrapes [jeffhuang.com/best_paper_awards/](https://jeffhuang.com/best_paper_awards/) and writes `site/data/best_papers.yaml`. Run once per year (the source is updated annually).
|
||||
Scrapes [jeffhuang.com/best_paper_awards/](https://jeffhuang.com/best_paper_awards/) and writes `best_papers.yaml`. Run once per year.
|
||||
```bash
|
||||
uv run pa-fetch-best-papers
|
||||
```
|
||||
@@ -192,119 +221,246 @@ uv run pa-fetch-papers --venue TPDS --year 2024 --source openalex
|
||||
```
|
||||
|
||||
### `pa-generate`
|
||||
Regenerates all Hugo content from data files. Safe to re-run at any time.
|
||||
Regenerates all Hugo content for one topic from its data files. Safe to re-run at any time.
|
||||
```bash
|
||||
# Explicit (for a specific topic):
|
||||
uv run pa-generate \
|
||||
--venues site/data/cloud-edge/venues.yaml \
|
||||
--deadlines site/data/cloud-edge/deadlines.yaml \
|
||||
--papers-dir site/data/cloud-edge/papers \
|
||||
--content site/content/cloud-edge \
|
||||
--base-path /cloud-edge
|
||||
|
||||
# Defaults (cloud-edge):
|
||||
uv run pa-generate
|
||||
```
|
||||
`--base-path` prefixes all internal links in generated markdown (e.g. `/cloud-edge/venues/…`). It must match the topic slug in the URL.
|
||||
|
||||
Preserved fields (never overwritten): `notes`, `deadline_source`.
|
||||
Stripped fields (removed on regen to avoid stale data): `url` (Hugo reserved), `papers`.
|
||||
|
||||
### `build.sh`
|
||||
Full pipeline orchestrator.
|
||||
Full pipeline orchestrator. Loops over all topics in `topics.yaml` by default.
|
||||
```bash
|
||||
./build.sh
|
||||
./build.sh --skip-rankings # skip icore/scimago fetches
|
||||
./build.sh --skip-deadlines # use cached deadlines.yaml
|
||||
./build.sh --dev # hugo server instead of build
|
||||
./build.sh # full build, all topics
|
||||
./build.sh --topic cloud-edge # one topic only
|
||||
./build.sh --skip-rankings # skip icore/scimago fetches (use cached CSVs)
|
||||
./build.sh --skip-deadlines # use cached deadlines.yaml
|
||||
./build.sh --dev # hugo server instead of build
|
||||
./build.sh --topic cloud-edge --dev # dev server, one topic
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hugo Content Structure
|
||||
|
||||
### Multi-topic routing
|
||||
|
||||
Hugo treats `site/content/<topic>/` as a section. All pages inside it are served under `/<topic>/`. The nav bar partial (`site/layouts/partials/header.html`) detects `.Section` at render time:
|
||||
- **Inside a topic** (`cloud-edge`, `embedded-ai`, …): renders Venues / Calendar / Digests links relative to that section.
|
||||
- **At the root** (`/`): renders one link per topic from `site/data/topics.yaml`.
|
||||
|
||||
To add a topic: populate `site/data/topics.yaml` + `site/data/<topic>/venues.yaml`, then run `./build.sh --topic <slug>`. The new section appears in the global nav automatically.
|
||||
|
||||
### Venues
|
||||
|
||||
**`site/content/venues/_index.md`** — overview, links to conferences and journals. Generated.
|
||||
**`site/content/<topic>/venues/_index.md`** — overview, links to conferences and journals. Generated.
|
||||
|
||||
**`site/content/venues/conferences/_index.md`** — table of all conferences sorted by ICORE rank with deadlines. Generated.
|
||||
**`site/content/<topic>/venues/conferences/_index.md`** — table of all conferences sorted by ICORE rank with deadlines, domains, and digest count. Generated.
|
||||
|
||||
**`site/content/venues/journals/_index.md`** — table of all journals sorted by SCImago quartile. Generated.
|
||||
**`site/content/<topic>/venues/journals/_index.md`** — table of all journals sorted by SCImago quartile. Generated.
|
||||
|
||||
Each venue page body is **fully generated Markdown** — PaperMod renders body content, not front matter fields. The body includes a metadata table and a deadline section.
|
||||
Each venue page body is **fully generated Markdown**. The body includes a metadata table (rank, domains, latest digest link), an upcoming deadline block, and previous-edition info blocks (paper count, topics, digest link).
|
||||
|
||||
### Calendar
|
||||
|
||||
**`site/content/calendar/_index.md`** — uses `layout: calendar`, which activates `site/layouts/_default/calendar.html`. The layout renders a FullCalendar (CDN) month grid above the deadline table. Events are embedded as a JSON-ready YAML list in the `events` front matter field. Colors: orange = abstract deadline, red = paper deadline, blue = conference dates.
|
||||
**`site/content/<topic>/calendar/_index.md`** — uses `layout: calendar`. Events are embedded as JSON-ready YAML in the `events:` front matter field by `pa-generate`. Colors: orange = abstract deadline, red = paper deadline, blue = conference dates.
|
||||
|
||||
### Digests
|
||||
|
||||
**`site/content/digests/<VENUE>-<YEAR>/index.md`** — uses `index.md` (not `_index.md`) to be a leaf page, not a section. Body contains the full paper list with TL;DR and why-notable for each paper. Papers data lives in `site/data/papers/<V>-<Y>-digest.yaml`; do not put it in front matter (it was stripped for causing empty pages with PaperMod).
|
||||
**`site/content/<topic>/digests/<VENUE>-<YEAR>/index.md`** — leaf page (`index.md`, not `_index.md`). Body contains the full paper list with TL;DR and why-notable for each paper. Papers data lives in `site/data/<topic>/papers/<V>-<Y>-digest.yaml`.
|
||||
|
||||
---
|
||||
|
||||
## Agent Instructions
|
||||
## Agent Prompts
|
||||
|
||||
Tasks requiring agent involvement (domain knowledge, judgment, web research).
|
||||
Ready-to-use prompts. Paste directly into Claude Code (or any agent) as a starting point.
|
||||
|
||||
---
|
||||
|
||||
### Task 1 — Bootstrap a domain
|
||||
### Bootstrap a new topic
|
||||
|
||||
**Trigger:** User asks to set up tracking for a new research domain.
|
||||
```
|
||||
Bootstrap a new publish-assistant topic for "<TOPIC NAME>" (slug: <slug>).
|
||||
|
||||
**Steps:**
|
||||
Steps:
|
||||
1. Research the top 10–15 conferences and 5–10 journals for this domain. Use
|
||||
csrankings.org, the ICORE portal (portal.core.edu.au/conf-ranks/), and
|
||||
SCImago (scimagojr.com) for rankings.
|
||||
2. For each conference: record acronym, full name, ICORE rank, official
|
||||
website URL for the upcoming edition, DBLP stream key (conf/<key>), and
|
||||
WikiCFP event ID if you can find it (set wikicfp_id: false for short or
|
||||
ambiguous acronyms).
|
||||
3. For each journal: record acronym, full name, ISSN, SCImago quartile + SJR
|
||||
+ H-index (add inline to venues.yaml — the CSV download is blocked), DBLP
|
||||
stream key (journals/<key>), submission model (rolling / special issues).
|
||||
4. Create site/data/<slug>/venues.yaml following the schema in the README.
|
||||
5. Create site/data/<slug>/papers/ (empty directory).
|
||||
6. Add the topic to site/data/topics.yaml:
|
||||
- slug: <slug>
|
||||
title: "<TOPIC NAME>"
|
||||
description: "<one-line description>"
|
||||
7. Create site/content/<slug>/_index.md:
|
||||
---
|
||||
title: "<TOPIC NAME>"
|
||||
description: "<one-line description>"
|
||||
draft: false
|
||||
---
|
||||
8. Run: ./build.sh --skip-rankings --topic <slug>
|
||||
(Use --skip-rankings to reuse cached ICORE data if already fresh.)
|
||||
9. For any conferences under deadlines.yaml missing:, do the manual deadline
|
||||
task (Task 2 in the README).
|
||||
10. Run: ./build.sh --skip-rankings --topic <slug>
|
||||
Then: hugo --source site --minify
|
||||
Confirm the site builds cleanly and /<slug>/venues/ loads correctly.
|
||||
|
||||
1. Ask for the domain name and any seed venues.
|
||||
2. Research: identify top 10–15 conferences and 5–10 journals. Use [csrankings.org](https://csrankings.org), [ICORE portal](https://portal.core.edu.au/conf-ranks/), and [SCImago](https://www.scimagojr.com/) for rankings.
|
||||
3. For each conference: record acronym, full name, ICORE rank, official 2025/2026 website URL, DBLP stream key (`conf/<key>`).
|
||||
4. For each journal: record acronym, full name, ISSN, SCImago quartile + SJR (add inline to `venues.yaml` — CSV download is blocked), DBLP stream key (`journals/<key>`), submission model (rolling / special issues).
|
||||
5. Append all venues to `site/data/venues.yaml`.
|
||||
6. For WikiCFP: add `wikicfp_id: "<id>"` if you can find the event page (search at wikicfp.com). Set `wikicfp_id: false` for conferences where search returns wrong matches (short/common acronyms are risky).
|
||||
7. Run `uv run pa-fetch-icore` and `uv run pa-fetch-deadlines`.
|
||||
8. For conferences not found by the deadline fetcher, do Task 2 immediately.
|
||||
9. Run `uv run pa-generate` and `hugo --source site` to validate.
|
||||
|
||||
**One-shot checklist for agents:**
|
||||
- [ ] `site/data/venues.yaml` populated with all venues
|
||||
- [ ] `wikicfp_id` set or `wikicfp_id: false` on every conference
|
||||
- [ ] SCImago data added inline to every journal entry
|
||||
- [ ] `uv run pa-fetch-icore` succeeded (check `site/data/rankings/icore.csv`)
|
||||
- [ ] `uv run pa-fetch-deadlines` ran (check `site/data/deadlines.yaml`)
|
||||
- [ ] All conferences in `deadlines.yaml:missing` handled via Task 2
|
||||
- [ ] `uv run pa-generate` ran cleanly
|
||||
- [ ] `hugo --source site --minify` built without errors
|
||||
Checklist before finishing:
|
||||
- [ ] site/data/<slug>/venues.yaml has all venues with correct dblp_key
|
||||
- [ ] wikicfp_id set or false on every conference
|
||||
- [ ] SCImago data added inline for every journal
|
||||
- [ ] site/data/topics.yaml updated
|
||||
- [ ] site/content/<slug>/_index.md created
|
||||
- [ ] ./build.sh --skip-rankings --topic <slug> runs without errors
|
||||
- [ ] hugo --source site --minify succeeds
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Add a venue to an existing topic
|
||||
|
||||
```
|
||||
Add <ACRONYM> to the publish-assistant topic "<slug>".
|
||||
|
||||
1. Look up:
|
||||
- Full name and ICORE rank (conferences) or SCImago quartile + SJR + H-index (journals)
|
||||
- Official website URL for the upcoming edition
|
||||
- DBLP stream key at dblp.org
|
||||
- WikiCFP event ID, or note if this acronym is ambiguous (set wikicfp_id: false)
|
||||
2. Append the entry to site/data/<slug>/venues.yaml following the existing schema.
|
||||
3. Run: uv run pa-fetch-deadlines \
|
||||
--venues site/data/<slug>/venues.yaml \
|
||||
--output site/data/<slug>/deadlines.yaml
|
||||
4. If the conference appears under missing: in deadlines.yaml, find the deadline
|
||||
on the official CFP page and add a source: manual entry.
|
||||
5. Run: uv run pa-generate (defaults to cloud-edge) or with explicit --venues /
|
||||
--content / --base-path flags for the target topic.
|
||||
6. Confirm the new venue page appears at /<slug>/venues/conferences/<acronym>/
|
||||
(or journals/) with correct metadata.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Build a digest for a venue + year
|
||||
|
||||
```
|
||||
Build a paper digest for <ACRONYM> <YEAR> in the publish-assistant topic "<slug>".
|
||||
|
||||
1. Run: uv run pa-fetch-papers --venue <ACRONYM> --year <YEAR>
|
||||
This writes site/data/<slug>/papers/<ACRONYM>-<YEAR>-candidates.yaml.
|
||||
|
||||
2. Run: uv run pa-fetch-best-papers
|
||||
(Skip if site/data/best_papers.yaml already exists and is recent.)
|
||||
|
||||
3. Read the candidates file. Select 8–15 papers that are:
|
||||
- Methodologically novel (new algorithms, system designs, formal proofs)
|
||||
- Attracting community attention (highly cited if issue is ≥1 year old;
|
||||
well-known authors or top-venue co-publications if recent)
|
||||
- Representative of the breadth of the issue (avoid over-indexing on one subtheme)
|
||||
- Preferably open-access (arXiv, USENIX, ACM OpenTOC)
|
||||
Flag any paper in best_papers.yaml (include it; it's a strong signal).
|
||||
|
||||
4. For each selected paper write:
|
||||
- tldr: one sentence, the core technical contribution
|
||||
- why_notable: 1–2 sentences — novelty, impact, surprising result, or
|
||||
influential technique; what would make a program committee member
|
||||
recommend this paper to colleagues
|
||||
|
||||
5. Write site/data/<slug>/papers/<ACRONYM>-<YEAR>-digest.yaml:
|
||||
venue: <ACRONYM>
|
||||
year: <YEAR>
|
||||
date: "<YYYY-MM-DD of first conference day>"
|
||||
tags: [<3–5 topic tags>]
|
||||
selected:
|
||||
- dblp_key: "..."
|
||||
title: "..."
|
||||
tldr: "..."
|
||||
why_notable: "..."
|
||||
|
||||
6. Run: uv run pa-generate (or with explicit flags for the topic)
|
||||
7. Confirm the digest page at /<slug>/digests/<acronym>-<year>/ renders correctly.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Annual cycle refresh for a topic
|
||||
|
||||
```
|
||||
Refresh the publish-assistant topic "<slug>" for the new conference cycle.
|
||||
|
||||
1. For each conference in site/data/<slug>/venues.yaml:
|
||||
a. Check whether the url field points to the upcoming edition (many venues
|
||||
use year-specific URLs like osdi26, 2027.eurosys.org, mobisys/2026/).
|
||||
Update any that point to past editions.
|
||||
b. Verify wikicfp_id still points to the upcoming edition by visiting
|
||||
http://wikicfp.com/cfp/servlet/event.showcfp?eventid=<ID>.
|
||||
If it points to a past event, search WikiCFP for the new edition.
|
||||
If the new event page doesn't exist yet, set wikicfp_id: false and add
|
||||
a source: manual entry to deadlines.yaml; restore the ID once it appears.
|
||||
|
||||
2. Run: uv run pa-fetch-deadlines \
|
||||
--venues site/data/<slug>/venues.yaml \
|
||||
--output site/data/<slug>/deadlines.yaml
|
||||
|
||||
3. For each entry under missing: in deadlines.yaml, visit the conference CFP
|
||||
page and add a source: manual entry to deadlines.yaml.
|
||||
|
||||
4. Run: ./build.sh --skip-rankings --topic <slug>
|
||||
Then confirm hugo --source site --minify succeeds and all venue pages show
|
||||
correct upcoming deadlines.
|
||||
|
||||
Checklist:
|
||||
- [ ] All conference url fields updated to upcoming edition
|
||||
- [ ] All wikicfp_id values verified (or set to false + manual entry)
|
||||
- [ ] missing: list in deadlines.yaml is empty
|
||||
- [ ] Build succeeds, no broken links
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agent Tasks (Reference)
|
||||
|
||||
### Task 1 — Bootstrap a topic
|
||||
See the "Bootstrap a new topic" prompt above. The one-shot checklist is embedded in the prompt.
|
||||
|
||||
### Task 2 — Fill in missing deadlines
|
||||
|
||||
**Trigger:** `pa-fetch-deadlines` lists conferences under `missing:`, or a deadline looks wrong.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. For each missing conference, visit the official website. Conferences typically have a "Call for Papers" page with an "Important Dates" section.
|
||||
2. Also check [WikiCFP](http://wikicfp.com/cfp/servlet/tool.search?q=<ACRONYM>&year=f) manually — if you find the right event ID, add `wikicfp_id` to `venues.yaml` so future runs fetch it automatically.
|
||||
1. For each missing conference, visit the official website's "Call for Papers" / "Important Dates" page.
|
||||
2. Also check WikiCFP manually — if you find the right event ID, add `wikicfp_id` to `venues.yaml` so future runs fetch it automatically.
|
||||
3. Extract: abstract deadline, paper deadline, notification, camera-ready, event dates, location.
|
||||
4. Add a `source: manual` entry to `site/data/deadlines.yaml`. This entry survives future `pa-fetch-deadlines` runs.
|
||||
5. Run `uv run pa-generate` to propagate.
|
||||
4. Add a `source: manual` entry to `site/data/<topic>/deadlines.yaml`.
|
||||
5. Run `uv run pa-generate` (with explicit flags for the topic).
|
||||
|
||||
**Conferences reliably NOT on WikiCFP** (for systems/networking):
|
||||
- USENIX family: OSDI, NSDI, USENIX ATC, USENIX Security — use usenix.org directly
|
||||
- SC (Supercomputing) — use sc<YY>.supercomputing.org/program/papers/
|
||||
- MobiSys — use sigmobile.org/mobisys/<YEAR>/
|
||||
- SEC (Edge Computing) — use acm-ieee-sec.org/<YEAR>/
|
||||
- Short or common acronyms (ATC, SEC) collide with unrelated events — always use `wikicfp_id: false` and fetch manually
|
||||
**Conferences reliably NOT on WikiCFP** (systems/networking):
|
||||
- USENIX family: OSDI, NSDI, USENIX ATC, USENIX Security — use usenix.org
|
||||
- SC — use sc`<YY>`.supercomputing.org
|
||||
- MobiSys — use sigmobile.org/mobisys/`<YEAR>`/
|
||||
- SEC — use acm-ieee-sec.org/`<YEAR>`/
|
||||
- Short/common acronyms (ATC, SEC) collide with unrelated events — always use `wikicfp_id: false`
|
||||
|
||||
---
|
||||
|
||||
### Task 3 — Build a digest for a conference issue
|
||||
|
||||
**Trigger:** User asks to build a digest for a specific venue + year.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Run `uv run pa-fetch-papers --venue <ACRONYM> --year <YEAR>` to get the candidate pool (`site/data/papers/<V>-<Y>-candidates.yaml`).
|
||||
2. Check `site/data/best_papers.yaml` (run `pa-fetch-best-papers` first if it doesn't exist). Papers with matching titles in the best-papers list should be included and flagged.
|
||||
3. Select 8–15 papers that are:
|
||||
- Methodologically novel (new algorithms, systems designs, formal proofs);
|
||||
- Attracting community attention (highly cited if the issue is ≥ 1 year old; in top venues / co-authored by known researchers if recent);
|
||||
- Representative of the breadth of the issue (avoid over-indexing on one subtheme);
|
||||
- Preferably open-access (arXiv, USENIX, ACM OpenTOC).
|
||||
4. For each selected paper, write:
|
||||
- `tldr`: one sentence, the core technical contribution.
|
||||
- `why_notable`: 1–2 sentences — novelty, impact, surprising result, or influential technique.
|
||||
5. Write `site/data/papers/<V>-<Y>-digest.yaml` with a `selected:` list.
|
||||
6. Run `uv run pa-generate` — the digest page is created at `site/content/digests/<V>-<Y>/index.md`.
|
||||
### Task 3 — Build a digest
|
||||
See the "Build a digest for a venue + year" prompt above.
|
||||
|
||||
**Digest YAML format:**
|
||||
```yaml
|
||||
@@ -320,80 +476,34 @@ selected:
|
||||
why_notable: "Became one of the most-cited LLM systems papers of 2024; disaggregation is now standard in production inference stacks."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4 — Refresh rankings
|
||||
|
||||
**Trigger:** ICORE releases a new round (every 2–3 years); SCImago releases new data (annually, each spring).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Run `uv run pa-fetch-icore` for fresh ICORE data.
|
||||
2. For SCImago: download the CSV manually from [scimagojr.com](https://www.scimagojr.com/journalrank.php) (CSV download button on the rankings page) and place it at `site/data/rankings/scimago.csv`. The automated fetch is blocked.
|
||||
3. Check `site/data/venues.yaml` — for journals, compare `scimago_quartile` / `scimago_sjr` against the new CSV. Update inline values if changed.
|
||||
4. Run `uv run pa-generate`.
|
||||
|
||||
---
|
||||
2. For SCImago: download the CSV manually from [scimagojr.com](https://www.scimagojr.com/journalrank.php) and place at `site/data/rankings/scimago.csv`. The automated fetch is blocked.
|
||||
3. For each journal in `venues.yaml`, compare `scimago_quartile` / `scimago_sjr` against new CSV. Update inline values if changed.
|
||||
4. Run `./build.sh --skip-deadlines` (or per-topic with `--topic <slug>`).
|
||||
|
||||
### Task 5 — Add a new venue mid-cycle
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Look up ICORE rank (conferences) or SCImago quartile (journals).
|
||||
2. Find the DBLP stream key at [dblp.org](https://dblp.org).
|
||||
3. Add to `site/data/venues.yaml`.
|
||||
4. For conferences: look up WikiCFP event ID or set `wikicfp_id: false`.
|
||||
5. Run `uv run pa-fetch-deadlines` + `uv run pa-generate`.
|
||||
|
||||
---
|
||||
See the "Add a venue to an existing topic" prompt above.
|
||||
|
||||
### Task 6 — Annual cycle refresh
|
||||
|
||||
**Trigger:** A new conference cycle begins (roughly each autumn/spring depending on the venue). Signs: event URLs return 404s, WikiCFP fetches pull wrong editions, or deadlines are over a year old.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Update edition URLs** — for each conference in `venues.yaml`, check whether `url` points to the upcoming edition. Many venues use year-specific URLs (`osdi26`, `2027.eurosys.org`, `mobisys/2026/`). Update these to the new edition. Generic/stable URLs (e.g., `sigops.org/s/conferences/sosp/`) do not need changing.
|
||||
|
||||
2. **Refresh WikiCFP IDs** — for each conference with a `wikicfp_id`, verify the ID still matches the upcoming edition by visiting `http://wikicfp.com/cfp/servlet/event.showcfp?eventid=<ID>`. If it points to a past event, search WikiCFP for the new edition and update the ID. If the new event page does not exist yet, set `wikicfp_id: false` temporarily and add a `source: manual` deadline entry; restore the ID once the page appears.
|
||||
|
||||
3. Run `uv run pa-fetch-deadlines` and check the `missing:` list. Fill gaps via Task 2.
|
||||
|
||||
4. Run `uv run pa-generate` and `hugo --source site --minify` to validate.
|
||||
|
||||
**One-shot checklist for agents:**
|
||||
- [ ] All conference `url` fields in `venues.yaml` point to the upcoming edition
|
||||
- [ ] All `wikicfp_id` values verified against the upcoming edition (or set to `false` with a manual entry)
|
||||
- [ ] `uv run pa-fetch-deadlines` ran cleanly; `missing:` list is empty
|
||||
- [ ] `uv run pa-generate` + `hugo --source site --minify` succeed
|
||||
|
||||
---
|
||||
|
||||
## What the Build Script Cannot Do
|
||||
|
||||
| Task | Why automation fails | Agent task |
|
||||
| ----------------------------------- | ---------------------------- | ---------- |
|
||||
| Initial venue curation | Requires domain knowledge | Task 1 |
|
||||
| Fetching missing deadlines | No standard CFP structure | Task 2 |
|
||||
| Selecting notable papers | Requires reading + judgment | Task 3 |
|
||||
| Writing `why_notable` | Requires synthesis | Task 3 |
|
||||
| Detecting meaningful rank changes | Requires domain context | Task 4 |
|
||||
| Evaluating new venues for inclusion | Requires community awareness | Task 5 |
|
||||
See the "Annual cycle refresh for a topic" prompt above.
|
||||
|
||||
---
|
||||
|
||||
## Known Gotchas (for agents picking this up)
|
||||
|
||||
- **`--base-path` must match the topic slug**: `pa-generate --base-path /cloud-edge` prefixes all internal links in generated markdown. If you run `pa-generate` without this flag (old default was `/cloud-edge`), all links on the new topic would resolve to `/cloud-edge/...` instead. Always pass explicit `--base-path /<slug>` when generating for a non-default topic, or use `build.sh --topic <slug>` which sets it automatically.
|
||||
- **Hugo `url` field**: reserved by Hugo to override the page URL. `venues.yaml` uses `url:` but `generate_content.py` maps it to `homepage:` in front matter. Never write `url:` in Hugo front matter via the generator.
|
||||
- **PaperMod renders body, not front matter**: all visible content must be in the Markdown body (after the second `---`). Front matter is used only for metadata and Hugo taxonomy. If a page looks empty, check that `_conf_body()` / `_journal_body()` etc. are being called.
|
||||
- **PaperMod renders body, not front matter**: all visible content must be in the Markdown body. Front matter is used only for metadata and Hugo taxonomy. If a page looks empty, check that `_conf_body()` / `_journal_body()` etc. are being called.
|
||||
- **`_index.md` vs `index.md`**: section pages use `_index.md` (list template), leaf pages use `index.md` (single template). Digest pages are `index.md` — using `_index.md` makes them section pages and breaks pagination.
|
||||
- **Rankings CSVs are in `site/data/rankings/`**: Hugo's data loader is configured to ignore `data/rankings/*.csv` via `ignoreFiles` in `hugo.toml`. If you move these files or add new CSVs, update `ignoreFiles` accordingly — Hugo cannot parse arbitrary CSV as a data map and will error on build.
|
||||
- **ICORE pagination**: the ICORE portal uses `javascript:jumpPage('N')` links, not standard `?page=N` URLs. `fetch_icore.py` handles this. If you get only 50 results instead of ~900, pagination is broken.
|
||||
- **SCImago blocked**: `pa-fetch-scimago` will raise a clear error if anti-bot HTML is returned. Add data inline to `venues.yaml` instead.
|
||||
- **WikiCFP `<th>` labels**: deadline detail pages use `<th>` for label cells, not `<td>`. The parser looks for `<th>+<td>` pairs. A `find_all("td")` approach finds nothing.
|
||||
- **Calendar events**: `pa-generate` embeds events as a YAML list in the `events:` front matter field of `content/calendar/_index.md`. The custom layout at `site/layouts/_default/calendar.html` reads `.Params.events` and initializes FullCalendar. Do not remove the `layout: calendar` front matter field.
|
||||
- **FullCalendar CDN**: loaded from `cdn.jsdelivr.net`. The `extend_head.html` partial injects the CSS; the layout injects the JS. Both are conditional on `layout == "calendar"`.
|
||||
- **Deadline preservation**: never strip deadline fields from `deadlines.yaml` just because the submission window has closed. Keep all fields (abstract deadline, submission deadline, notification, camera-ready) for every conference whose event date is still in the future. Remove an entry only once the conference has taken place. An agent doing a "refresh" or "cleanup" must not treat a past submission deadline as stale data worth deleting.
|
||||
- **WikiCFP IDs are edition-specific**: each year's event gets a new WikiCFP event ID. The IDs in the table above are for specific editions and will be wrong once a new cycle begins. When `pa-fetch-deadlines` returns stale or mismatched data, check whether the `wikicfp_id` in `venues.yaml` still points to the upcoming edition. If the new event page doesn't exist on WikiCFP yet, set `wikicfp_id: false` and add a `source: manual` entry; update the ID once the new page appears. See Task 6 for the full annual refresh checklist.
|
||||
- **WikiCFP `<th>` labels**: deadline detail pages use `<th>` for label cells, not `<td>`. The parser looks for `<th>+<td>` pairs.
|
||||
- **Calendar events**: `pa-generate` embeds events as YAML in the `events:` front matter field. The layout at `site/layouts/_default/calendar.html` reads `.Params.events` and initializes FullCalendar. Do not remove the `layout: calendar` front matter field.
|
||||
- **Deadline preservation**: never strip deadline fields from `deadlines.yaml` just because the submission window has closed. Remove an entry only once the conference has taken place.
|
||||
- **WikiCFP IDs are edition-specific**: each year's event gets a new ID. IDs verified at one point in time will be wrong once a new cycle begins. See Task 6 for the annual refresh checklist.
|
||||
|
||||
---
|
||||
|
||||
@@ -402,6 +512,6 @@ selected:
|
||||
- **iCal feed** — generate a `.ics` file from deadline data so researchers can subscribe from their calendar app.
|
||||
- **Email/RSS notifications** — alert when a deadline is within N weeks.
|
||||
- **Citation tracking** — periodically re-query OpenAlex for citation counts on digest papers.
|
||||
- **Multi-domain support** — a single repo tracking multiple domains; run the pipeline per domain with a `--domain` flag.
|
||||
- **Best paper badge** — `pa-generate` could cross-reference `best_papers.yaml` with digest candidates and add a `best_paper_award: true` flag, then render a badge in `_digest_body()`.
|
||||
- **Automated WikiCFP ID discovery** — when a venue has no `wikicfp_id`, attempt a search and record the result in `venues.yaml` for future runs (reduces manual work when bootstrapping new domains).
|
||||
- **Automated WikiCFP ID discovery** — when a venue has no `wikicfp_id`, attempt a search and record the result in `venues.yaml` for future runs (reduces manual work when bootstrapping new topics).
|
||||
- **Cross-topic venue pages** — some venues (e.g., ASPLOS) span multiple topics. A future `cross_listed: [embedded-ai, cloud-edge]` field in `venues.yaml` could render the venue under multiple topic sections without duplicating data.
|
||||
|
||||
Reference in New Issue
Block a user