Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

MediaWiki API Project: Category Under/Over-Coverage

Aperta
  • Qualcosa non va?
#239 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
jupyter-notebook, pandas, python, streamlit
Ambito
analytics, data, data-engineering, data-visualization

Direzione di ricerca

Inizia dalla documentazione collegata di MediaWiki Action API, Categorymembers, Revisions, Langlinks e PageAssessments, quindi esamina gli artefatti proposti e i deliverable della dashboard. Definisci una profondità di attraversamento limitata e un insieme di metriche prima dell’implementazione; il lavoro è completo quando sono presenti gli output parquet/CSV, la dashboard, il methods README e i test per l’attraversamento, la deduplicazione, i contatori e i controlli di coerenza delle metriche.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

complexity: medium CoP: Data Science feature: missing good first issue project: EDA role: data analysis role: data science role: missing size: 3pt size: missing
Overview

Quantify category-level under/over-coverage on Wikipedia by crawling sensitive topical categories (and their subcategories) and measuring article counts, content depth, references, and quality signals—then ranking categories against their peers to surface gaps.

Action Items

If this is the beginning (research & design)

  • Define seed categories (e.g., human rights, policing, immigration, public health, Indigenous issues). Set a max traversal depth (start with 1–2).
  • Decide metrics: pages per category, median article length, references/article, % with infobox, interlanguage links count, PageAssessments quality/importance (if enabled), and additions per month (from category-membership timestamps).
  • Methods plan: breadth-first search of subcategories via list=categorymembers; fetch page features via prop=revisions (size), prop=revisions&rvprop=content (for <ref> and infobox detection), prop=langlinks (coverage across languages), prop=pageassessments (quality, if available).
  • Tooling (pick pairs): requests or httpx; pandas or polars; storage in duckdb or sqlite; visualization in Altair or Plotly.

If researched and ready (implementation steps)

  1. Seed & crawl

    • For each top-level category: list=categorymembers&cmtype=subcat|page&cmlimit=max&cmnamespace=0|14.
    • BFS to a fixed depth; persist (category, pageid, title, member_timestamp). De-duplicate pages across subcats.

  2. Feature extraction

    • Size & timestamps: prop=revisions&rvprop=size|timestamp&rvlimit=1&rvdir=older/newer as needed.
    • References & infobox: fetch wikitext for sampled/latest revids and count <ref occurrences and {{Infobox.
    • Interlanguage breadth: prop=langlinks&lllimit=max → count.
    • Quality (optional): prop=pageassessments → map to quality/importance bins.

  3. Aggregation

    • Per category: page_count, median size, refs/article (median), % with infobox, median langlinks, quality mix, and monthly additions using member_timestamp.
    • Normalize within the cohort (z-scores/percentiles) and compute an Under-Coverage Index (e.g., average rank of page_count, size, refs/article, langlinks).

  4. Deliver

    • Artifacts: category_members.parquet, page_features.parquet, category_monthly.parquet, category_metrics.csv.
    • Dashboard: ranked table of categories (Under-Coverage Index), sparkline of monthly additions, metric drill-downs.
    • Methods README with traversal depth, regex rules, and limitations.

  5. Quality & Ops

    • Caching + retries with exponential backoff; honor maxlag; persist continue tokens and query manifests.
    • Tests: traversal correctness, duplicate handling, regex counters, and metric sanity checks.
    • Error handling: wrap I/O in try/except with helpful print() for file-not-found; raise with trace on dtype mismatches; warn on recoverables.
    • Optional: scheduled monthly refresh (GitHub Actions).
Resources/Instructions

API docs to pin in repo

MediaWiki Action API overview: https://www.mediawiki.org/wiki/API:Action_API
Category members (pages & subcats): https://www.mediawiki.org/wiki/API:Categorymembers
Revisions (size, timestamps, content): https://www.mediawiki.org/wiki/API:Revisions
Langlinks (interlanguage links): https://www.mediawiki.org/wiki/API:Langlinks
PageAssessments (if enabled): https://www.mediawiki.org/wiki/Extension:PageAssessments#API
Query & continuation etiquette: https://www.mediawiki.org/wiki/API:Query

Suggested libraries (choose pairs)

  • HTTP: requests | httpx
  • DataFrames: pandas | polars
  • Storage: duckdb | sqlite
  • Viz: Altair | Plotly
  • Parsing: mwparserfromhell | wikitextparser

Sample queries (copy to notes)

# Members of a category (pages + subcats), with timestamps
action=query&list=categorymembers&cmtitle=Category:Human_rights&cmtype=page|subcat&cmprop=ids|title|type|timestamp&cmlimit=max

# Latest revision (size) for a page
action=query&prop=revisions&rvprop=size|timestamp&rvlimit=1&titles=<TITLE>

# Fetch content for counting <ref> and {{Infobox
action=query&prop=revisions&revids=<REVID>&rvslots=main&rvprop=content

# Interlanguage links count
action=query&prop=langlinks&lllimit=max&titles=<TITLE>

# Page assessments (if available)
action=query&prop=pageassessments&titles=<TITLE>

Ethics & caveats

  • Category systems are noisy; membership can be incomplete or inconsistent. Document traversal depth and exclusions (hidden cats if you choose).

  • Ref/infobox counts are proxies for depth, not quality. Treat PageAssessments as optional (not all wikis have it).

  • Aggregate reporting only; no editor-level profiling.

  • If this issue requires access to 311 data, please answer the following questions:

    • Not applicable.
    • N/A
    • N/A
    • N/A

Project Outline (detailed plan for this idea) in details:

Research question
Which sensitive topical categories are thin (few pages, short articles, sparse references, low interlanguage coverage) relative to peer categories, and how are they changing month to month?

Data sources & modules

  • list=categorymembers for pages/subcats (+ timestamps).
  • prop=revisions (size/content).
  • prop=langlinks (language breadth).
  • prop=pageassessments (quality/importance, when available).

Method

  1. BFS the category graph to a fixed depth, collecting member pages and timestamps.
  2. For each page, compute size, ref count, infobox presence, langlinks count, and optional quality labels.
  3. Aggregate per category; build Under-Coverage Index and rank.
  4. Produce monthly addition trends and highlight categories in the bottom decile on ≥2 core metrics.

Key metrics

  • page_count, median_size, median_refs_per_article, %_infobox, median_langlinks, quality mix.
  • Monthly additions; 12-mo Δ in page_count and size.
  • Under-Coverage Index and bottom-decile flags.

Deliverables

  • Clean tables (category_members.parquet, page_features.parquet, category_monthly.parquet, category_metrics.csv).
  • Notebook + reports/category_under_over_coverage.md.
  • Streamlit/Altair dashboard (ranked table, sparklines, metric drill-downs).

Limitations

  • Category membership timestamps can reflect when categorization occurred, not page creation; call that out.
  • Regex heuristics for refs/infobox may miss template edge cases—validate on a labeled subset and report precision.
  • Interlanguage links vary by page age; consider age-adjusted comparisons if needed.
Lingua principale
Jupyter Notebook
Stelle
33
Fork
23
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di hackforla/data-science

Tutte le issue di hackforla/data-science

Issue simili

Altre issue su Data Visualization

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.