Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[SPIKE] Feed Import Automation - Implementation Proposal

Aperta
  • Qualcosa non va?
#1,678 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
openapi, postgresql, python
Ambito
api, databases

Direzione di ricerca

Start by reading docs/OperationsAPI.yaml and docs/DatabaseCatalogAPI.yaml to understand the API choice still pending discussion, then review feeds_operations_impl.py and the referenced db_utils.py query. The proposal also suggests a MatchingFeed DTO in db_model; completion would require settling the endpoint and API, then implementing the matching behavior and result representation described in the issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

As drafted in: https://www.notion.so/mobilitydata/SPIKE-Extending-Operations-API-for-Feed-Import-Automation-3566e644546a805dbd4ed40640b68dc6?source=copy_link

Problem

On average we have 5-10 weekly incoming issues to the mobility-database-catalogs repo specifically created by the feed submission form. These are issues relating to either the creation, or update of schedule or realtime feeds in the database.

There is good potential for automating these types of issues because the incoming format is predictable, and the manual workflow for importing them is already well defined. Specifically, a main source of work in completing these issues is checking for the pre-existence of the feed before creating or updating it in the database.

Feed Import Automation 2.pdf

Apart from checking for the pre-existence of feeds in MobilityDatabase, there are several other benefits to automating this type of import:

  • check for URL validity (HTTP200? HEAD request to receive .zip download?) There are a few safeguards we can add to catch feed errors before introducing them to the database
  • sometimes user error results in the feed form not being completed (for instance, user may not realize that an old URL exists in MobilityDatabase, and issue is incorrectly treated as feed creation rather than feed update)
  • feed active status can be correctly set before importing into MobilityDatabase

This pre-existing database match check can be automated as an operation performed by an API endpoint call. To achieve this, we can look to the existing Operations API which provides a good location to implement this functionality. We can also consider extending the Public API itself (to discuss in meeting).

OpenAPI Schema

There are two candidate APIs to extend with a feed matching endpoint:

Schema file Gen script Output Deployed as
docs/DatabaseCatalogAPI.yaml scripts/api-gen.sh api/src/feeds_gen/ Public API (api/)
docs/OperationsAPI.yaml scripts/api-operations-gen.sh functions-python/operations_api/src/feeds_gen/ Operations Cloud Function

Pending a discussion with team, we can decide which API to add the matching endpoint to. The Public API is a good choice if we intend to make calls during Feed Form Submission. If we intend to add lots of specific heuristics that affect matching, we may opt to include it in the Operations API because Public API should be more open ended?

Implementation & Code Changes

From higher level to lower level:

docs/OperationsAPI.yaml or docs/DatabaseCatalogAPI.yaml

add API endpoint for example /v1/operations/feeds:match: or /v1/search/feeds:match:

functions-python/operations_api/src/feeds_operations/impl

After regenerating, implement the stub in feeds_operations_impl.py or its own impl file.

Matching Logic

We can't simply return ORM from the query results. Because we are returning other match-specific metadata like criteria, match_type, confidence and potentially others, we should create another DTO like this:

class MatchingFeed:
    """Result of feed(s) matching on Producer URL, Provider Name and Location

    Attributes:
        stable_id: Unique stable identifier (e.g. 'mdb-123').
        data_type: Feed data type ('gtfs', 'gtfs_rt').
        status: Feed lifecycle status (e.g. 'active', 'inactive').
        operational_status: Publication status (e.g. 'published', 'wip').
        provider: Human-readable transit provider name.
        producer_url: The producer URL stored in the database for this feed.
        normalized_url: The input URL after normalization.
        criteria: [producer_url, provider_name, location]        
        match_type: 'exact' or 'fuzzy'.
        confidence: Score in [0.0, 1.0] — 1.0 for exact, 0.8 for fuzzy.

api/src/shared/common/db_utils can be used to return exact matches on producer_url. It can be extended to also include fuzzy matches (pg_trgm can be used to do trigram matching, we can use LIKE condition etc.)

api/src/shared/common/db_utils or optionally a new util file matching_utils.py can include functions for finding provider name-based matches and location-based matches:

get_feed_query_by_provider_name and get_feed_query_by_location/_get_feed_query_by_bounding_box

db_model I recommend creating a new db_model/matching_feed_impl.py with the MatchingFeed DTO drafted above.
DB queries will not simply return ORM entities, but rather domain-specific results so we have to also include an impl in api/src/shared/db_models to represent this.

Lingua principale
Python
Stelle
34
Fork
7
Merge medio
12h 7m
PR unite (30g)
7

Preparare l'ambiente

  • Include un Dockerfile o un file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MobilityData/mobility-feed-api

Tutte le issue di MobilityData/mobility-feed-api

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.