research: back-test a second-gift conversion model on DonorsChoose

Open
  • Is something wrong?
#346 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data, machine-learning

Research direction

Start by getting the local DonorsChoose file as described in docs/results/datasets.md, then read philanthropy.datasets.load_donorschoose to see which first-gift fields are available. Use philanthropy.model_selection.compare_to_rule for the rule baselines and paired bootstrap, and scripts/make_results_pages.py for the Results row format. Done means a test-year report of top-10% hit rate and AUC against the best validation rule, recorded as either a Results row or a rejected note with the numbers.

Written by the indexing model from the issue text.

Description

enhancement help wanted roadmap
Goal

Find out whether a model beats a simple rule at picking which first-time donors will give again within 12 months.

Acceptance criteria
  • Population: donors whose first gift falls in fiscal year T. Label: another gift within 12 months of the first. load_donorschoose truncates gift_date to the first of the month, so define "within 12 months" in whole months and say so; a second gift in the same month as the first is ambiguous and must be handled explicitly (drop or count, stated).
  • Baseline rules: "first gift above the median", plus one-column rules on what the loader actually returns (donor_type, was_matched, the two gift-card flags). There is no recurring-gift field in this file. The reported baseline is the best rule on the validation fold.
  • Model: DonorPropensityModel on features known at the first gift only (amount, month, donor_type, was_matched, gift-card flags), so nothing after the first gift leaks in. load_donorschoose returns no project fields; joining them would need a second file and a note on its licence.
  • Protocol: choose settings on a validation fiscal year, report once on a later test year: top-10% hit rate and AUC with a paired bootstrap interval against the best rule.
  • Outcome recorded either way: a Results row if it wins, or a "rejected" note with the numbers if it doesn't.
Scope

Research and a Results-page row. No new public estimator unless the model clearly wins on the test year.

Where to start
  • philanthropy.datasets.load_donorschoose (local file; docs/results/datasets.md says where to get it)
  • philanthropy.model_selection.compare_to_rule for the bootstrap and verdict logic to copy. It only accepts task in {"upgrade", "lapse", "ask"}, so it cannot run this comparison as is; adding a task is part of the public API and only follows an adopt result
  • scripts/make_results_pages.py for how existing rows are generated
Dependencies
  • None

Phase order is a release plan, not a rule against picking up independent work early. Comment on this issue before you start so two people don't build the same thing.

Dominant language
Python
Stars
11
Forks
23
Avg merge
15h 13m
Merged PRs (30d)
30

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PhilanthroPy-Project/PhilanthroPy

All issues in PhilanthroPy-Project/PhilanthroPy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.