It's Half-Staff Because tracks whether the flag is at half-staff, and why, across all fifty states, DC, and the federal government. The answers come from proclamations published by 52 different offices. No two offices publish them quite the same way.

Watching all of them means working with whatever each office happens to provide.

Four kinds of source

Every source gets one of four approaches, roughly in order of how much trouble it is to maintain:

  • RSS or Atom feeds. The easiest, and the largest share of sources. A feed already contains structured entries, dates, and links, so the scraper never has to understand the page layout. These rarely break.
  • JSON APIs. Some state websites load their press releases from an API. It is usually undocumented, but it tends to survive redesigns because the site itself still depends on it.
  • HTML scraping. These scrapers read the press-release listing page and use CSS selectors to find each title, date, and link. They work until a redesign changes the markup. Most of my maintenance time goes here.
  • A full browser (Playwright). A stubborn few block plain HTTP requests or render their content with JavaScript. For those, the scraper opens a browser and lets the page load normally. It is slower and heavier than the other approaches, but sometimes it is the only option that works.

The mix is not fixed. A state drops its RSS feed, or rebuilds on a JavaScript-heavy platform, and that source moves from the easy column to the expensive one. The proportions drift; the four approaches stay the same.

The federal source is the easy exception. Presidential proclamations come through the Federal Register's documented API.

What one of them looks like

Every scraper, whatever its type, satisfies the same interface. It returns documents, or it returns nothing:

TypeScript
export interface RawDocument {
  externalId: string;
  title: string;
  rawText: string;
  sourceUrl: string;
  signingDate: string;
  publicationDate: string;
}

export interface Scraper {
  readonly sourceId: string;
  scrape(): Promise<ScrapeResult>;
}

Texas is a good example of the HTML case, and of how little code a source needs once the shared plumbing handles fetching and parsing. The page shows only the current status, with no history list, so this scraper returns one document or zero:

TypeScript
export class TexasScraper extends HtmlScraper {
  constructor(sourceId: string) {
    super(sourceId, 'https://gov.texas.gov/flag-status', 'Texas');
  }

  protected async extractDocuments(
    $: cheerio.CheerioAPI
  ): Promise<RawDocument[]> {
    // The page has an h1 like "Flag Status: Full-Staff" or "... Half-Staff"
    const headings = $('h1, h2').toArray().map((el) => $(el).text().trim());

    const statusHeading = headings.find((h) =>
      h.toLowerCase().includes('flag status')
    );

    if (!statusHeading || !statusHeading.toLowerCase().includes('half-staff')) {
      return []; // Full-staff or unrecognized -- nothing to report
    }

    const rawText =
      $('main, .content-area, article').first().text().trim() ||
      $('body').text().trim();

    const today = new Date().toISOString().split('T')[0];

    return [{
      externalId: `tx-flag-status-${today}`,
      title: statusHeading,
      rawText,
      sourceUrl: `${this.baseUrl}#${today}`,
      signingDate: today,
      publicationDate: today,
    }];
  }
}

That return [] does more than skip a record. The next section explains what it sets off.

Keeping the data trustworthy

Deduplication happens twice. I first check each document by source URL. Some offices republish the same order at a new URL, and others do not provide a stable URL at all, so I also compare titles within the same jurisdiction. That second check keeps a republished proclamation from appearing twice on the map.

TypeScript
// First pass: exact source URL.
if (doc.sourceUrl) {
  const { data: existing } = await this.supabase
    .from('proclamations')
    .select('id')
    .eq('source_url', doc.sourceUrl)
    .limit(1)
    .maybeSingle();

  if (existing) {
    logger.info(`Skipping duplicate: ${doc.title}`);
    return;
  }
}

// Fallback: same title, same jurisdiction. Catches republished orders
// and sources whose URLs are unstable or missing.
const { data: titleMatch } = await this.supabase
  .from('proclamations')
  .select('id')
  .eq('jurisdiction_id', source.jurisdiction_id)
  .eq('reason', doc.title)
  .limit(1)
  .maybeSingle();

The system closes orders on its own. Offices rarely publish an announcement when flags return to full staff. The active proclamation simply disappears. When a scraper finds no current half-staff documents for a jurisdiction, the system closes any order that remains open. Otherwise old orders would linger on the map and look current.

This is what the Texas scraper's empty array triggers. No documents means the flag is back up:

TypeScript
const result = await scraper.scrape();

if (result.documents.length === 0) {
  // No half-staff documents -- auto-close any open proclamations for
  // this source's jurisdiction so the map does not show stale data.
  await this.closeOpenProclamations(scraper.sourceId);
  await this.finishRun(runId, 'no_change');
  return;
}

That design divides scraper failures into two kinds, and only one of them is dangerous.

A hard failure is safe. If the page 404s or the request dies, the fetch throws, the pipeline catches it, the run is recorded as an error, and nothing gets auto-closed:

TypeScript
if (!res.ok) {
  throw new Error(
    `${this.stateName} page returned ${res.status}: ${res.statusText}`
  );
}

A silent failure is the one to worry about. If a state redesigns its site and my selectors stop matching, the page still returns 200, extractDocuments still returns an empty array, and that is indistinguishable from a state whose flags went back up. The map would show full staff for a jurisdiction I can no longer actually read. That is what the run log below is for.

Routine scrapes have a cutoff date. Many sources expose years of press releases on the same page or feed. A cutoff prevents a normal run from pulling all of that history into the database. Historical backfills are separate, deliberate runs.

Each source has a small scraper of its own. They all sit on shared plumbing, but the source-specific selectors and rules live in separate files. When Ohio redesigns its website, I can fix Ohio without touching the other 51 sources.

Every run is logged to the database with the source, time, and result — new_proclamation, no_change, or error. When Georgia goes quiet, I need enough history to tell whether its scraper broke or the governor's office simply has nothing new to publish. A source that has reported no_change every run for two months, having previously reported activity, is the signature of a scraper that is quietly reading the wrong page.

What maintenance actually looks like

These websites change on their own schedules and without warning. A scraper stays useful only as long as I maintain it.

The four approaches make that workload manageable. Feed and API scrapers rarely need attention, leaving most of the maintenance time for HTML and browser-based sources. I assume that any given week may bring an unannounced change somewhere in the list.

When a site changes today, one jurisdiction may go quiet until I repair its scraper. The failure stays contained to that source while the rest of the map keeps working.