Live data source call log can't show what each HTTP call returned

Open
#32,797 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

  • #32798 by @sdolidze — closed without merging

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
typescript
Domain
api, backend

Research direction

The issue describes adding per HTTP call records for live data source queries, but names no files or tests. Start by tracing where api_source_calls is recorded and where integrations process paginated responses; check existing coverage for RudderStack and OTel. Done means each call exposes its method, path, status, response rows and bytes, duration, and error while preserving the existing query level meaning and row_count.

Written by the indexing model from the issue text.

Description

Log every HTTP call a live data source query makes to its integration, with the rows and bytes each response held.

api_source_calls has one row per query, despite its name. It sums the calls (api_calls, response_bytes) but keeps nothing per call. When a query pages through an integration, there is no way to see how many rows each page returned, how big it was, or which call failed. A query that reads 20 pages of 500 records shows only api_calls = 20 and 104 MB.

  • One row per HTTP call, linked to its query: method, path, status, rows in the response, bytes, duration, error.
  • Rows are read generically from the JSON response, so integrations need no code.
  • RudderStack and OTel get the same per-call view.
  • api_source_calls keeps its meaning; row_count stays the result's row count.
Dominant language
TypeScript
Stars
6.2k
Forks
809
Avg merge
12h 13m
Merged PRs (30d)
854

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lightdash/lightdash

All issues in lightdash/lightdash

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.