Scraping 52 government websites that all publish differently

It's Half-Staff Because tracks whether the flag is at half-staff, and why, across all fifty states, DC, and the federal government. The answers come from proclamations published by 52 different offices. No two offices publish them quite the same way.

Watching all of them means working with whatever each office happens to provide.

Four kinds of source

Every source gets one of four approaches, roughly in order of how much trouble it is to maintain:

RSS or Atom feeds. The easiest, and the largest share of sources. A feed already contains structured entries, dates, and links, so the scraper never has to understand the page layout. These rarely break.
JSON APIs. Some state websites load their press releases from an API. It is usually undocumented, but it tends to survive redesigns because the site itself still depends on it.
HTML scraping. These scrapers read the press-release listing page and use CSS selectors to find each title, date, and link. They work until a redesign changes the markup. Most of my maintenance time goes here.
A full browser (Playwright). A stubborn few block plain HTTP requests or render their content with JavaScript. For those, the scraper opens a browser and lets the page load normally. It is slower and heavier than the other approaches, but sometimes it is the only option that works.

The mix is not fixed. A state drops its RSS feed, or rebuilds on a JavaScript-heavy platform, and that source moves from the easy column to the expensive one. The proportions drift; the four approaches stay the same.

The federal source is the easy exception. Presidential proclamations come through the Federal Register's documented API.

Keeping the data trustworthy

Deduplication happens twice. I first check each document by source URL. Some offices republish the same order at a new URL, and others do not provide a stable URL at all, so I also compare titles within the same jurisdiction. That second check keeps a republished proclamation from appearing twice on the map.

The system closes orders on its own. Offices rarely publish an announcement when flags return to full staff. The active proclamation simply disappears. When a scraper finds no current half-staff documents for a jurisdiction, the system closes any order that remains open. Otherwise old orders would linger on the map and look current.

Routine scrapes have a cutoff date. Many sources expose years of press releases on the same page or feed. A cutoff prevents a normal run from pulling all of that history into the database. Historical backfills are separate, deliberate runs.

Each source has a small scraper of its own. They all sit on shared plumbing, but the source-specific selectors and rules live in separate files. When Ohio redesigns its website, I can fix Ohio without touching the other 51 sources.

Every run is logged to the database with the source, time, and result. When Georgia goes quiet, I need enough history to tell whether its scraper broke or the governor's office simply has nothing new to publish.

What maintenance actually looks like

These websites change on their own schedules and without warning. A scraper stays useful only as long as I maintain it.

The four approaches make that workload manageable. Feed and API scrapers rarely need attention, leaving most of the maintenance time for HTML and browser-based sources. I assume that any given week may bring an unannounced change somewhere in the list.

When a site changes today, one jurisdiction may go quiet until I repair its scraper. The failure stays contained to that source while the rest of the map keeps working.