Duplicated Fitbit data on output storage for certain topics

Abierto
  • ¿Algo no está bien?
#154 0 comentarios 0 reacciones 1 asignado Ver en GitHub

@mpgxvii ya está trabajando en esto.

Desde el 15/1/2025.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Problem

For certain Fitbit topics on a client deployment, selected topics started showing duplicated data rows in output storage analogous to:

key.subject key.source key.project value.time value.timeRecieved value.value
my-subject-id my-source-id my-project-id 000000001 000000005 20
my-subject-id my-source-id my-project-id 000000001 0000000010 20

Note that Fitbit data recorded at the same time appears as a separate line; the only difference is the value.timeReceived.

This was observed for the following Fitbit topics:

connect_fitbit_breathing_rate
connect_fitbit_intraday_heart_rate_variability
connect_fitbit_sleep_stages
connect_fitbit_activity_log
connect_fitbit_skin_temperature
connect_fitbit_sleep_classic

it was noted to NOT occur for the following Fitbit topics:

connect_fitbit_intraday_calories
connect_fitbit_intraday_steps
connect_fitbit_intraday_heart_rate

Some observations

  • Data deduplication by output-restructure normally takes all columns into account. For this reason the value.timeReceived is causing the same piece of data to be added as a separate line.
  • I suspect that previous from older versions of Fitbit connector, the current connector repeatedly downloads the same date range for the problematic topics. The inclusion of the value.timeReceived column during de-deduplication causes the data to appear multiple times in the output files.
  • Many of the problematic topics were updated in PR #118
  • The topics that work well all have a 1d time period hard-coded in the request URL whereas others do not:
FitbitIntradayCaloriesRoute.java:    return baseUrl + "/1/user/%s/activities/calories/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitRestingHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d.json?timezone=UTC";
FitbitIntradayStepsRoute.java:    return baseUrl + "/1/user/%s/activities/steps/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitIntradayHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d/1sec/time/%s/%s.json?timezone=UTC";

FitbitSkinTemperatureRoute.java:    return baseUrl + "/1/user/%s/temp/skin/date/%s/%s.json";
FitbitBreathingRateRoute.java:    return baseUrl + "/1/user/%s/br/date/%s/%s/all.json";
FitbitIntradayHeartRateVariabilityRoute.java:    return baseUrl + "/1/user/%s/hrv/date/%s/%s/all.json";
FitbitIntradaySpo2Route.java:    return baseUrl + "/1/user/%s/spo2/date/%s/%s/all.json";
FitbitTimeZoneRoute.java:    return baseUrl + "/1/user/%s/profile.json";
FitbitSleepRoute.java:    return baseUrl + "/1.2/user/%s/sleep/list.json?sort=asc&afterDate=%s&limit=100&offset=0";
FitbitActivityLogRoute.java:    return baseUrl + "/1/user/%s/activities/list.json?sort=asc&afterDate=%s&limit=20&offset=0";
Lenguaje dominante
Java
Estrellas
5
Forks
6
Merge medio
6 d 1 h
PR fusionados (30 d)
4

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de RADAR-base/RADAR-REST-Connector

Todos los issues de RADAR-base/RADAR-REST-Connector

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.