codeforboston/maple

Handle Bill Text Limit

Open

#2169 aperta il 16 giu 2026

Vedi su GitHub
 (1 commento) (0 reazioni) (0 assegnatari)TypeScript (168 fork)auto 404
good first issuescraper

Metriche repository

Star
 (56 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Summary

Now that we're scraping the text of longer bills from the PDFs, we're seeing occasional errors where a Bill would exceed the maximum acceptable size of a Firebase document (~1 MiB). This prevents scraping of the bill altogether - which is

Instead of failing altogether, we should handle this case more gracefully:

  • Catch failed Firestore writes around maximum allowed size and re-try saving the Bill document without the DocumentText (basically, the current behavior where we only have the PDF)

Success Criteria

  • Bills with excessively long DocumentText can be mostly successfully scraped (by excluding the long DocumentText field)

Additional Info

  • Error: In fetchBillBatch: Error: 3 INVALID_ARGUMENT: Document 'projects/digital-testimony-dev/databases/(default)/documents/generalCourts/194/bills/H5500' cannot be written because its size (1,373,153 bytes) exceeds the maximum allowed size of 1,048,576 bytes.

  • Example Bill: Bill H5500 in Court 194

Guida contributor