Publish Assistant

Hugo-generated website that helps researchers:

• identify important venues to publish in;
• keep track of submission deadlines;
• retrieve important papers for each issue.

---

Overview

The site is domain-driven: a researcher picks a research area (e.g. "edge and cloud systems") and the assistant builds a curated, ranked, up-to-date snapshot of where to publish, when to submit, and what to read. The output is a static Hugo site that can be rebuilt on demand.

Work is split between automated scripts (data fetching, Hugo content generation) and agent tasks (domain curation, paper selection, deadline gap-filling). The README below documents both halves so the site can be kept fresh over time.

---

Repository Layout

publish-assistant/
├── site/                                 # Hugo project
│   ├── hugo.toml
│   ├── themes/PaperMod/                  # PaperMod theme (git submodule)
│   ├── layouts/
│   │   ├── _default/calendar.html        # FullCalendar layout override
│   │   └── partials/extend_head.html
│   ├── content/                          # Generated — do not edit by hand
│   │   ├── venues/
│   │   │   ├── _index.md                 # Venues overview (generated)
│   │   │   ├── conferences/              # One subdir per tracked conference
│   │   │   └── journals/                 # One subdir per tracked journal
│   │   ├── calendar/                     # Aggregated deadline view + FullCalendar
│   │   └── digests/                      # Per-issue paper digests
│   └── data/                             # Structured data — edit these
│       ├── venues.yaml                   # Master venue list ← primary edit target
│       ├── deadlines.yaml                # Deadline cache; manual entries preserved
│       ├── best_papers.yaml              # Best-paper awards (written by pa-fetch-best-papers)
│       ├── rankings/
│       │   ├── icore.csv                 # ICORE rankings cache
│       │   └── scimago.csv               # SCImago rankings cache (optional)
│       └── papers/
│           ├── <V>-<Y>-candidates.yaml   # Full paper list (pa-fetch-papers)
│           └── <V>-<Y>-digest.yaml       # Curated selection (agent, Task 3)
├── src/publish_assistant/                # Python package
│   ├── fetch_icore.py
│   ├── fetch_scimago.py
│   ├── fetch_deadlines.py
│   ├── fetch_papers.py
│   ├── fetch_best_papers.py
│   └── generate_content.py
├── pyproject.toml                        # uv project; CLI entry points
├── uv.lock
├── build.sh
└── README.md

---

Setup

uv sync          # installs all dependencies + registers CLI tools

# Available commands after sync:
uv run pa-fetch-icore
uv run pa-fetch-scimago
uv run pa-fetch-deadlines
uv run pa-fetch-best-papers
uv run pa-fetch-papers --venue OSDI --year 2025
uv run pa-generate

# Local dev server:
hugo server --source site

---

Data Sources

Source	What it provides	Automatable?	Known issues
ICORE	Conference rankings (A*, A, B, C)	Yes	Pagination uses javascript:jumpPage('N') — handled in fetch_icore.py
SCImago	Journal quartiles, SJR, H-index	Blocked	Anti-bot returns HTML; add data manually to venues.yaml under scimago_quartile etc.
DBLP	Paper metadata by venue	Yes	Use dblp_key field in venues.yaml
OpenAlex	Papers, open-access links	Yes	Fallback when DBLP is thin
WikiCFP	Submission deadlines	Partially	See detailed notes below
Conference websites	Authoritative deadlines	Partially	See Task 2
jeffhuang.com/best_paper_awards/	Best paper awards since 1996, ~32 venues	Yes	Manually maintained; run pa-fetch-best-papers annually

---

WikiCFP Integration — Detailed Notes

WikiCFP is the primary deadline source but has several quirks that required workarounds:

HTML structure: Detail pages use <th> for row labels (not <td>). The parser in fetch_cfp_details() specifically looks for <th> + <td> pairs. Do not revert to find_all("td") — it will find zero deadline rows.

Direct ID lookup: Add wikicfp_id: "<event_id>" to a venue entry in venues.yaml to skip the search and fetch that page directly. This avoids wrong matches on common acronyms. Verified IDs for this domain:

Venue	WikiCFP event ID
SOSP	191399
EuroSys	186524
SoCC	191071
Middleware	190153
IPDPS	189093
HPDC	191029

Skipping search: Set wikicfp_id: false to skip WikiCFP entirely for a venue (e.g., ATC, SC, SEC — where the search returns wrong events). Deadlines for these must be filled manually.

Conferences not on WikiCFP (for edge/cloud systems domain): OSDI, NSDI, USENIX ATC, SC, MobiSys, SEC. Use wikicfp_id: false for all of them.

Manual deadline entries: Add entries with source: manual to site/data/deadlines.yaml. The fetcher preserves all source: manual entries across runs. Format:

ATC:
  source: manual
  event_dates: Nov 16-18, 2026
  location: Hong Kong
  submission_deadline: Jun 10, 2026
  notification: Sep 18, 2026
  camera_ready: Oct 16, 2026
  cfp_url: https://sigops.org/s/conferences/atc/2026/cfp.html

---

venues.yaml Schema

conferences:
  - acronym: SOSP
    full_name: "ACM Symposium on Operating Systems Principles"
    domain: [edge-and-cloud, operating-systems, distributed-systems]
    url: "https://sigops.org/s/conferences/sosp/"
    dblp_key: "conf/sosp"
    wikicfp_id: "191399"      # direct lookup; omit to use search; false to skip entirely

journals:
  - acronym: TPDS
    full_name: "IEEE Transactions on Parallel and Distributed Systems"
    domain: [edge-and-cloud, parallel-computing, distributed-systems]
    issn: "1045-9219"
    url: "https://www.computer.org/csdl/journal/td"
    dblp_key: "journals/tpds"
    submission_model: rolling
    scimago_quartile: Q1      # add manually — SCImago CSV download is blocked
    scimago_sjr: "1.560"
    scimago_h_index: "131"

Important: Hugo reserves the front matter field url as a page URL override. generate_content.py maps venues.yaml:url → front matter field homepage to avoid this conflict.

---

Scripts

All scripts are installed as CLI entry points by uv sync.

pa-fetch-icore

Downloads ICORE rankings. Handles the portal's JavaScript-based pagination.

uv run pa-fetch-icore                                   # fetch all
uv run pa-fetch-icore --query "distributed systems"

pa-fetch-scimago

Downloads SCImago CSV. Currently blocked by anti-bot. Will raise a descriptive error if it receives HTML instead of CSV. Add journal data manually to venues.yaml instead.

uv run pa-fetch-scimago --list-areas    # show area codes
uv run pa-fetch-scimago --area 1705     # networks

pa-fetch-deadlines

Fetches submission deadlines from WikiCFP. Preserves source: manual entries.

uv run pa-fetch-deadlines

After running: check site/data/deadlines.yaml for the missing: list, then do Task 2.

pa-fetch-best-papers

Scrapes jeffhuang.com/best_paper_awards/ and writes site/data/best_papers.yaml. Run once per year (the source is updated annually).

uv run pa-fetch-best-papers

pa-fetch-papers

Fetches paper lists from DBLP (with OpenAlex fallback).

uv run pa-fetch-papers --venue OSDI --year 2024
uv run pa-fetch-papers --venue TPDS --year 2024 --source openalex

pa-generate

Regenerates all Hugo content from data files. Safe to re-run at any time.

uv run pa-generate

Preserved fields (never overwritten): notes, deadline_source. Stripped fields (removed on regen to avoid stale data): url (Hugo reserved), papers.

build.sh

Full pipeline orchestrator.

./build.sh
./build.sh --skip-rankings    # skip icore/scimago fetches
./build.sh --skip-deadlines   # use cached deadlines.yaml
./build.sh --dev              # hugo server instead of build

---

Hugo Content Structure

Venues

site/content/venues/_index.md — overview, links to conferences and journals. Generated.

site/content/venues/conferences/_index.md — table of all conferences sorted by ICORE rank with deadlines. Generated.

site/content/venues/journals/_index.md — table of all journals sorted by SCImago quartile. Generated.

Each venue page body is fully generated Markdown — PaperMod renders body content, not front matter fields. The body includes a metadata table and a deadline section.

Calendar

site/content/calendar/_index.md — uses layout: calendar, which activates site/layouts/_default/calendar.html. The layout renders a FullCalendar (CDN) month grid above the deadline table. Events are embedded as a JSON-ready YAML list in the events front matter field. Colors: orange = abstract deadline, red = paper deadline, blue = conference dates.

Digests

site/content/digests/<VENUE>-<YEAR>/index.md — uses index.md (not _index.md) to be a leaf page, not a section. Body contains the full paper list with TL;DR and why-notable for each paper. Papers data lives in site/data/papers/<V>-<Y>-digest.yaml; do not put it in front matter (it was stripped for causing empty pages with PaperMod).

---

Agent Instructions

Tasks requiring agent involvement (domain knowledge, judgment, web research).

---

Task 1 — Bootstrap a domain

Trigger: User asks to set up tracking for a new research domain.

Steps:

1. Ask for the domain name and any seed venues.
2. Research: identify top 10–15 conferences and 5–10 journals. Use csrankings.org, ICORE portal, and SCImago for rankings.
3. For each conference: record acronym, full name, ICORE rank, official 2025/2026 website URL, DBLP stream key (conf/<key>).
4. For each journal: record acronym, full name, ISSN, SCImago quartile + SJR (add inline to venues.yaml — CSV download is blocked), DBLP stream key (journals/<key>), submission model (rolling / special issues).
5. Append all venues to site/data/venues.yaml.
6. For WikiCFP: add wikicfp_id: "<id>" if you can find the event page (search at wikicfp.com). Set wikicfp_id: false for conferences where search returns wrong matches (short/common acronyms are risky).
7. Run uv run pa-fetch-icore and uv run pa-fetch-deadlines.
8. For conferences not found by the deadline fetcher, do Task 2 immediately.
9. Run uv run pa-generate and hugo --source site to validate.

One-shot checklist for agents:

• site/data/venues.yaml populated with all venues
• wikicfp_id set or wikicfp_id: false on every conference
• SCImago data added inline to every journal entry
• uv run pa-fetch-icore succeeded (check site/data/rankings/icore.csv)
• uv run pa-fetch-deadlines ran (check site/data/deadlines.yaml)
• All conferences in deadlines.yaml:missing handled via Task 2
• uv run pa-generate ran cleanly
• hugo --source site --minify built without errors

---

Task 2 — Fill in missing deadlines

Trigger: pa-fetch-deadlines lists conferences under missing:, or a deadline looks wrong.

Steps:

1. For each missing conference, visit the official website. Conferences typically have a "Call for Papers" page with an "Important Dates" section.
2. Also check WikiCFP manually — if you find the right event ID, add wikicfp_id to venues.yaml so future runs fetch it automatically.
3. Extract: abstract deadline, paper deadline, notification, camera-ready, event dates, location.
4. Add a source: manual entry to site/data/deadlines.yaml. This entry survives future pa-fetch-deadlines runs.
5. Run uv run pa-generate to propagate.

Conferences reliably NOT on WikiCFP (for systems/networking):

• USENIX family: OSDI, NSDI, USENIX ATC, USENIX Security — use usenix.org directly
• SC (Supercomputing) — use sc.supercomputing.org/program/papers/
• MobiSys — use sigmobile.org/mobisys//
• SEC (Edge Computing) — use acm-ieee-sec.org//
• Short or common acronyms (ATC, SEC) collide with unrelated events — always use wikicfp_id: false and fetch manually

---

Task 3 — Build a digest for a conference issue

Trigger: User asks to build a digest for a specific venue + year.

Steps:

1. Run uv run pa-fetch-papers --venue <ACRONYM> --year <YEAR> to get the candidate pool (site/data/papers/<V>-<Y>-candidates.yaml).
2. Check site/data/best_papers.yaml (run pa-fetch-best-papers first if it doesn't exist). Papers with matching titles in the best-papers list should be included and flagged.
3. Select 8–15 papers that are:
   • Methodologically novel (new algorithms, systems designs, formal proofs);
   • Attracting community attention (highly cited if the issue is ≥ 1 year old; in top venues / co-authored by known researchers if recent);
   • Representative of the breadth of the issue (avoid over-indexing on one subtheme);
   • Preferably open-access (arXiv, USENIX, ACM OpenTOC).
4. For each selected paper, write:
   • tldr: one sentence, the core technical contribution.
   • why_notable: 1–2 sentences — novelty, impact, surprising result, or influential technique.
5. Write site/data/papers/<V>-<Y>-digest.yaml with a selected: list.
6. Run uv run pa-generate — the digest page is created at site/content/digests/<V>-<Y>/index.md.

Digest YAML format:

venue: OSDI
year: 2024
date: "2024-07-10"
tags: [llm-serving, distributed-systems, storage]

selected:
  - dblp_key: "conf/osdi/ZhongLCHZL0024"
    title: "DistServe: Disaggregating Prefill and Decoding ..."
    tldr: "Separates prefill and decode onto different GPU pools, eliminating head-of-line blocking."
    why_notable: "Became one of the most-cited LLM systems papers of 2024; disaggregation is now standard in production inference stacks."

---

Task 4 — Refresh rankings

Trigger: ICORE releases a new round (every 2–3 years); SCImago releases new data (annually, each spring).

Steps:

1. Run uv run pa-fetch-icore for fresh ICORE data.
2. For SCImago: download the CSV manually from scimagojr.com (CSV download button on the rankings page) and place it at site/data/rankings/scimago.csv. The automated fetch is blocked.
3. Check site/data/venues.yaml — for journals, compare scimago_quartile / scimago_sjr against the new CSV. Update inline values if changed.
4. Run uv run pa-generate.

---

Task 5 — Add a new venue mid-cycle

Steps:

1. Look up ICORE rank (conferences) or SCImago quartile (journals).
2. Find the DBLP stream key at dblp.org.
3. Add to site/data/venues.yaml.
4. For conferences: look up WikiCFP event ID or set wikicfp_id: false.
5. Run uv run pa-fetch-deadlines + uv run pa-generate.

---

Task 6 — Annual cycle refresh

Trigger: A new conference cycle begins (roughly each autumn/spring depending on the venue). Signs: event URLs return 404s, WikiCFP fetches pull wrong editions, or deadlines are over a year old.

Steps:

1. Update edition URLs — for each conference in venues.yaml, check whether url points to the upcoming edition. Many venues use year-specific URLs (osdi26, 2027.eurosys.org, mobisys/2026/). Update these to the new edition. Generic/stable URLs (e.g., sigops.org/s/conferences/sosp/) do not need changing.

2. Refresh WikiCFP IDs — for each conference with a wikicfp_id, verify the ID still matches the upcoming edition by visiting http://wikicfp.com/cfp/servlet/event.showcfp?eventid=<ID>. If it points to a past event, search WikiCFP for the new edition and update the ID. If the new event page does not exist yet, set wikicfp_id: false temporarily and add a source: manual deadline entry; restore the ID once the page appears.

3. Run uv run pa-fetch-deadlines and check the missing: list. Fill gaps via Task 2.

4. Run uv run pa-generate and hugo --source site --minify to validate.

One-shot checklist for agents:

• All conference url fields in venues.yaml point to the upcoming edition
• All wikicfp_id values verified against the upcoming edition (or set to false with a manual entry)
• uv run pa-fetch-deadlines ran cleanly; missing: list is empty
• uv run pa-generate + hugo --source site --minify succeed

---

What the Build Script Cannot Do

Task	Why automation fails	Agent task
Initial venue curation	Requires domain knowledge	Task 1
Fetching missing deadlines	No standard CFP structure	Task 2
Selecting notable papers	Requires reading + judgment	Task 3
Writing why_notable	Requires synthesis	Task 3
Detecting meaningful rank changes	Requires domain context	Task 4
Evaluating new venues for inclusion	Requires community awareness	Task 5

---

Known Gotchas (for agents picking this up)

• Hugo url field: reserved by Hugo to override the page URL. venues.yaml uses url: but generate_content.py maps it to homepage: in front matter. Never write url: in Hugo front matter via the generator.
• PaperMod renders body, not front matter: all visible content must be in the Markdown body (after the second ---). Front matter is used only for metadata and Hugo taxonomy. If a page looks empty, check that _conf_body() / _journal_body() etc. are being called.
• _index.md vs index.md: section pages use _index.md (list template), leaf pages use index.md (single template). Digest pages are index.md — using _index.md makes them section pages and breaks pagination.
• ICORE pagination: the ICORE portal uses javascript:jumpPage('N') links, not standard ?page=N URLs. fetch_icore.py handles this. If you get only 50 results instead of ~900, pagination is broken.
• SCImago blocked: pa-fetch-scimago will raise a clear error if anti-bot HTML is returned. Add data inline to venues.yaml instead.
• WikiCFP <th> labels: deadline detail pages use <th> for label cells, not <td>. The parser looks for <th>+<td> pairs. A find_all("td") approach finds nothing.
• Calendar events: pa-generate embeds events as a YAML list in the events: front matter field of content/calendar/_index.md. The custom layout at site/layouts/_default/calendar.html reads .Params.events and initializes FullCalendar. Do not remove the layout: calendar front matter field.
• FullCalendar CDN: loaded from cdn.jsdelivr.net. The extend_head.html partial injects the CSS; the layout injects the JS. Both are conditional on layout == "calendar".
• Deadline preservation: never strip deadline fields from deadlines.yaml just because the submission window has closed. Keep all fields (abstract deadline, submission deadline, notification, camera-ready) for every conference whose event date is still in the future. Remove an entry only once the conference has taken place. An agent doing a "refresh" or "cleanup" must not treat a past submission deadline as stale data worth deleting.
• WikiCFP IDs are edition-specific: each year's event gets a new WikiCFP event ID. The IDs in the table above are for specific editions and will be wrong once a new cycle begins. When pa-fetch-deadlines returns stale or mismatched data, check whether the wikicfp_id in venues.yaml still points to the upcoming edition. If the new event page doesn't exist on WikiCFP yet, set wikicfp_id: false and add a source: manual entry; update the ID once the new page appears. See Task 6 for the full annual refresh checklist.

---

Future Considerations

• iCal feed — generate a .ics file from deadline data so researchers can subscribe from their calendar app.
• Email/RSS notifications — alert when a deadline is within N weeks.
• Citation tracking — periodically re-query OpenAlex for citation counts on digest papers.
• Multi-domain support — a single repo tracking multiple domains; run the pipeline per domain with a --domain flag.
• Best paper badge — pa-generate could cross-reference best_papers.yaml with digest candidates and add a best_paper_award: true flag, then render a badge in _digest_body().
• Automated WikiCFP ID discovery — when a venue has no wikicfp_id, attempt a search and record the result in venues.yaml for future runs (reduces manual work when bootstrapping new domains).