feast-dev/feast

Restructure test suite into 4-tier directory layout: unit, universal, component and integration

Open

#6048 opened on Mar 3, 2026

View on GitHub
 (1 comment) (2 reactions) (0 assignees)Python (5,029 stars) (896 forks)batch import
good first issuekind/featuresize/XLstarter-ticket

Description

Is your feature request related to a problem? Please describe.

Our current test organization relies on pytest markers to separate unit and integration tests, but this system is failing:

  • test-python-unit runs pytest sdk/python/tests (the entire tests directory) and filters by markers — any integration test missing @pytest.mark.integration silently runs as a "unit" test
  • At least 3 integration test files (test_mcp_feature_server.py, test_hybrid_online_store.py, test_dbt_integration.py) are missing the marker and currently run during unit CI
  • Unit tests import heavyweight dependencies (ray, pyspark) that slow down the entire suite and require special environment setup
  • The conftest.py marker-filtering logic in pytest_collection_modifyitems is a house of cards — markers are opt-in and people forget them

Describe the solution you'd like

Replace marker-based filtering with a folder-based 4-tier layout:

tests/
  unit/ → Fast, no heavy deps, pure logic
  universal/ → Cross-backend parameterized tests (same test, many backends)
  component/ → Backend-specific tests (needs ray/spark/etc installed)
  integration/ → Needs external services (K8s, Docker, Keycloak, cloud)

Key benefits:*

  • pytest tests/unit — path-based, no marker gymnastics, no leakage
  • pytest tests/component/ray — runs only ray-specific tests
  • pytest tests/universal --config=duckdb — runs universal tests against DuckDB
  • The folder is the categorization — no markers to forget
  • Ray CI = component/ray + universal(ray config). Clean and composable.

Sub-tasks

  • Create directory skeleton and move feature_repos/
  • Move ray tests to component/ray/
  • Move spark tests to component/spark/
  • Move universal offline store tests to universal/offline_store/
  • Move universal online store tests to universal/online_store/
  • Move universal registration and materialization tests to universal/
  • Reclassify misplaced tests (unit ↔ integration boundary fixes)
  • Refactor root conftest.py — move fixtures, remove marker filtering
  • Update Makefile targets for new directory layout
  • Update CI workflows for new directory layout

Migration Principles

  1. Each sub-task is a self-contained PR that can be merged and tested independently
  2. No functional changes to tests — only file moves and import path updates
  3. Old Makefile targets should remain as aliases during transition (deprecated, then removed)
  4. Run the relevant CI after each PR to verify nothing broke

Thanks to @tokoko for idea and kicking off discussion.

Contributor guide