[Crashlytics] ANR events of the outgoing app version drop by about 50% for 3-5 days during each new version rollout

Aperta
  • Qualcosa non va?
#8,659 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
android, kotlin
Ambito
mobile-dev, observability-sre

Direzione di ricerca

Start by reproducing the rollout comparison using the BigQuery realtime and batch tables and the app's ApplicationExitInfo records via getHistoricalProcessExitReasons. Then inspect the SessionReportingCoordinator change referenced in #8250; done means identifying whether collection, sampling, rate limiting, processing, or previous-session attribution explains the measured drop and documenting a verified conclusion.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

api: crashlytics needs-attention
[REQUIRED] Step 2: Describe your environment
  • Android Studio version: not relevant, production data
  • Firebase Component: Crashlytics (firebase-crashlytics, firebase-crashlytics-ndk)
  • Component version: Firebase BoM 34.17.0, Crashlytics Gradle plugin 3.0.7
[REQUIRED] Step 3: Describe the problem

When a new version of our app starts its staged rollout, the number of ANR events that Crashlytics records for the outgoing version drops by about half for 3 to 5 days. Then it returns to its usual level. The incoming version does not show the drop. We saw it on three consecutive releases. Our own count of ANR exits from ApplicationExitInfo stays flat in the same days, so the app does not have fewer ANRs.

Steps to reproduce:
  1. Ship version N to all users, then start a staged rollout of version N+1.
  2. Each day, count Crashlytics ANR events of version N in the BigQuery export (realtime and batch tables).
  3. Each day, count ApplicationExitInfo records with REASON_ANR that the app itself reads on start with getHistoricalProcessExitReasons and sends to our own telemetry.
  4. Compare the two counts.

Ratio of Crashlytics ANR events to our ApplicationExitInfo ANR records, per version and day (UTC). The rollout of the next version started on the day of the first low value.

Version Before During the rollout of the next version After
A 2.68 1.25, 1.22, 1.26 2.07
B 3.17 2.41, 2.05, 1.57 2.52
C 2.18 1.25, 1.03, 1.13 1.93

Our own ApplicationExitInfo ANR count stays at about 0.4% of app starts during the same days.

What we ruled out:

  • The drop is the same in the realtime and the batch BigQuery export.
  • It is the same for all rows and for distinct event ids, with and without a user id.
  • It is not a weekday effect: on the same day, version C fell by 43% and the version before it by 1%.
  • It is not only an attribution effect at the update. We see it also between two consecutive app starts on the same version. For restarts on version C that carry an ApplicationExitInfo ANR, the share with a matching Crashlytics ANR event between the two starts was 81%, 46% and 35% on three consecutive days.
  • Event timestamps and the time to the next start stay within seconds. Upload lag grows during the rollout, but days that are a week old still show the drop.

Questions:

  1. Is there a known behavior in collection, sampling, rate limiting or processing that lowers ANR reporting for an app version while a newer version rolls out?
  2. Can a change in how the previous session's ANR is attached on the next launch cause this? For example the SessionReportingCoordinator change in #8250.
  3. Is there a setting that keeps ANR reporting complete for older versions?

We can share the queries and the full daily table.

Relevant Code:
// No custom Crashlytics code: default ANR collection.
// Our own reference count, read on app start (API 30+):
val exits = activityManager.getHistoricalProcessExitReasons(null, 0, 5)
val last = exits.firstOrNull { it.processName == packageName }
// last.reason == ApplicationExitInfo.REASON_ANR is sent to our telemetry once per exit timestamp
Lingua principale
Java
Stelle
2.6k
Fork
733
Merge medio
2g 15h
PR unite (30g)
72

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di firebase/firebase-android-sdk

Tutte le issue di firebase/firebase-android-sdk

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.