codeforboston/maple

Legislator Profile - Scrape Biographies from Legislature Site

Open

#2,161 opened on Jun 9, 2026

View on GitHub
 (1 comment) (0 reactions) (1 assignee)TypeScript (168 forks)auto 404
Ready for Developmentbackendgood first issuescraper

Repository metrics

Stars
 (56 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

The Legislator pages on the Legislature's site actually have biographies:https://malegislature.gov/Legislators/Profile/RLR0/Biography - we should use those as a default for our Legislator Profile page's biography section. Given that this is relevant data for all members, not just members with MAPLE accounts, it makes the most sense to store this on the Member document itself.

I would guess they only update these once a session at most, so I think this is fine as a one-off script that will run and scrape this for all legislators. (We may eventually want this longer-term, but given that we also want to avoid overwriting a biography the legislator sets in MAPLE itself, that may be self-defeating).

Success Criteria

  • Add biography field to Members (in generalCourts/194/members)
    • (There may also eventually be an override in profiles for legislator who make accounts on MAPLE and want to customize them, but that is out of scope for this ticket)
  • Create a scraper script that:
    • Iterates over all legislators in the current court (better yet, make court configurable by a param and just default to the current general court)
    • Scrapes the biography from the html of the legislature's legislator page (found at https://malegislature.gov/Legislators/Profile/<<MEMBER_CODE>>/Biography)
    • Writes the biographies to the corresponding member in Firestore

Contributor guide