Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Add dataset: chronicling_america

Open
  • Is something wrong?
#85 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Domain
data-engineering

Research direction

Start by locating the dataset loading scripts, then inspect the Chronicling America newspapers API at https://chroniclingamerica.loc.gov/newspapers.json and its linked title, issue, OCR, and image endpoints. Review the date and other filters needed for loading; done means the dataset can be added through the open API and its access and licence details are recorded.

Written by the indexing model from the issue text.

Description

dataset
A URL for this dataset

https://chroniclingamerica.loc.gov/about/api/#bulk-data

Dataset description

Chronicling America is a Library of Congress project to digitise historic newspapers. The collection contains mostly English but also contains other languages. Breakdown by language: https://public.tableau.com/app/profile/chronicling.america#!/vizhome/ChroniclingAmericaLanguageCoverageBubble/All_Lang

Various ways of accessing this data include bulk downloads and an API. The API may be the most helpful way of accessing this dataset (via dataset loading script) because this dataset is not static (more titles are digitised and added on a rolling basis).

The 'newspapers' API (https://chroniclingamerica.loc.gov/newspapers.json) is probably the best starting point. This starts instead from a list of Newspaper titles for which digital content is held. A title, i.e. https://chroniclingamerica.loc.gov/lccn/sn86072192.json, contains a bunch of metadata.

Screenshot 2022-09-27 at 16 32 26.

This API also contains all the issues for that title. For each issue, you get a set of pages. Each page contains the plain text generated from the OCR for that page, e.g. https://chroniclingamerica.loc.gov/lccn/sn82014726/1888-04-07/ed-1/seq-1/ocr.txt and a link to the image of that page, e.g. https://chroniclingamerica.loc.gov/lccn/sn82014726/1888-04-07/ed-1/seq-1.jp2.

My suggested approach to loading this dataset would be to call https://chroniclingamerica.loc.gov/newspapers.json at the start of the script and, depending on some filters defined in the loading script, i.e. start/end date of interest, build up a list of relevant URLs for the text/images for each page.

If you want to work on this dataset, please cc @davanstrien and @albertvillanova!

Dataset modality

Mixed

Dataset licence

Other license

Other licence

https://chroniclingamerica.loc.gov/about/#rights

How can you access this data

Via an open API

size of dataset

10GB

Confirm the dataset has an open licence
  • To the best of my knowledge, this dataset is accessible via an open licence
Contact details for data custodian

No response

Dominant language
No language data
Stars
91
Forks
8
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bigscience-workshop/lam

All issues in bigscience-workshop/lam

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.