[Crashlytics] ANR events of the outgoing app version drop by about 50% for 3-5 days during each new version rollout

Abierto
  • ¿Algo no está bien?
#8,659 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
android, kotlin
Área
mobile-dev, observability-sre

Línea de trabajo

Start by reproducing the rollout comparison using the BigQuery realtime and batch tables and the app's ApplicationExitInfo records via getHistoricalProcessExitReasons. Then inspect the SessionReportingCoordinator change referenced in #8250; done means identifying whether collection, sampling, rate limiting, processing, or previous-session attribution explains the measured drop and documenting a verified conclusion.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

api: crashlytics needs-attention
[REQUIRED] Step 2: Describe your environment
  • Android Studio version: not relevant, production data
  • Firebase Component: Crashlytics (firebase-crashlytics, firebase-crashlytics-ndk)
  • Component version: Firebase BoM 34.17.0, Crashlytics Gradle plugin 3.0.7
[REQUIRED] Step 3: Describe the problem

When a new version of our app starts its staged rollout, the number of ANR events that Crashlytics records for the outgoing version drops by about half for 3 to 5 days. Then it returns to its usual level. The incoming version does not show the drop. We saw it on three consecutive releases. Our own count of ANR exits from ApplicationExitInfo stays flat in the same days, so the app does not have fewer ANRs.

Steps to reproduce:
  1. Ship version N to all users, then start a staged rollout of version N+1.
  2. Each day, count Crashlytics ANR events of version N in the BigQuery export (realtime and batch tables).
  3. Each day, count ApplicationExitInfo records with REASON_ANR that the app itself reads on start with getHistoricalProcessExitReasons and sends to our own telemetry.
  4. Compare the two counts.

Ratio of Crashlytics ANR events to our ApplicationExitInfo ANR records, per version and day (UTC). The rollout of the next version started on the day of the first low value.

Version Before During the rollout of the next version After
A 2.68 1.25, 1.22, 1.26 2.07
B 3.17 2.41, 2.05, 1.57 2.52
C 2.18 1.25, 1.03, 1.13 1.93

Our own ApplicationExitInfo ANR count stays at about 0.4% of app starts during the same days.

What we ruled out:

  • The drop is the same in the realtime and the batch BigQuery export.
  • It is the same for all rows and for distinct event ids, with and without a user id.
  • It is not a weekday effect: on the same day, version C fell by 43% and the version before it by 1%.
  • It is not only an attribution effect at the update. We see it also between two consecutive app starts on the same version. For restarts on version C that carry an ApplicationExitInfo ANR, the share with a matching Crashlytics ANR event between the two starts was 81%, 46% and 35% on three consecutive days.
  • Event timestamps and the time to the next start stay within seconds. Upload lag grows during the rollout, but days that are a week old still show the drop.

Questions:

  1. Is there a known behavior in collection, sampling, rate limiting or processing that lowers ANR reporting for an app version while a newer version rolls out?
  2. Can a change in how the previous session's ANR is attached on the next launch cause this? For example the SessionReportingCoordinator change in #8250.
  3. Is there a setting that keeps ANR reporting complete for older versions?

We can share the queries and the full daily table.

Relevant Code:
// No custom Crashlytics code: default ANR collection.
// Our own reference count, read on app start (API 30+):
val exits = activityManager.getHistoricalProcessExitReasons(null, 0, 5)
val last = exits.firstOrNull { it.processName == packageName }
// last.reason == ApplicationExitInfo.REASON_ANR is sent to our telemetry once per exit timestamp
Lenguaje dominante
Java
Estrellas
2.6k
Forks
733
Merge medio
2 d 15 h
PR fusionados (30 d)
72

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de firebase/firebase-android-sdk

Todos los issues de firebase/firebase-android-sdk

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.