[Bug][gh-copilot] DORA dashboard uses seat-assigning organization as organization adoption denominator
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 35/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Tranquilo
Línea de trabajo
Comienza rastreando las 27 ubicaciones de consultas MySQL y las consultas PostgreSQL correspondientes en los dashboards incluidos de github-copilot-dora-correlation. Luego inspecciona CollectCopilotSeatAssignments, copilotSeatResponse y ExtractSeats para comprender los datos de asientos de enterprise y la atribución a organizaciones. La implementación terminada debe definir una población licenciada válida con alcance de organización para la adopción actual e histórica, sin clasificar la adopción NULL como >75%.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Search before asking
- I searched the issues and found no issue covering this organization attribution and denominator problem.
Related, but not duplicates:
- #8964 updates Copilot API endpoints and schemas.
- #8960 adds AI adoption phases/cohorts.
- #9002 implements parts of those changes.
What happened
The bundled github-copilot-dora-correlation dashboards calculate organization adoption as:
daily_active_users /
COUNT(_tool_copilot_seats WHERE organization = selected_scope.organization)
This assumes that Copilot seats assigned to members of an organization are also assigned by that organization.
GitHub explicitly supports assigning licenses through a central or "shell" organization. Its current attribution rules state that once a user has a Copilot seat anywhere in the enterprise, their usage is attributed to every organization in that enterprise where they are a member.
Therefore, an organization can have valid organization-scoped usage metrics while having zero seats assigned directly by that organization.
Observed result:
_tool_copilot_enterprise_daily_metricscontained daily active-user data for the selected organization scope._tool_copilot_seatscontained no seats attributed to that organization.- The dashboard denominator was zero, so adoption was
NULL. - Adoption trend, current adoption, correlation, and tier-based panels showed no or misleading data.
NULLadoption was classified as>75%by tier queries because theirCASEexpressions useELSE '>75%'.
The MySQL DORA dashboard currently contains this seat-denominator pattern in 27 query locations. The PostgreSQL version appears to use the same assumption.
There is also a plugin-level enterprise seat attribution concern:
CollectCopilotSeatAssignmentscorrectly calls the enterprise seats endpoint when an enterprise is configured.copilotSeatResponsedoes not parse the response'sorganizationfield.ExtractSeatssets every row'sOrganizationtoconnection.Organization.- The enterprise endpoint can return multiple seat rows for the same user when access is granted through multiple organizations or enterprise teams.
This means enterprise seat records cannot reliably represent organization membership or organization-level license eligibility.
GitHub documentation:
Relevant behavior from the documentation:
If licenses are assigned in a dedicated "shell" organization for administrative purposes within the enterprise, users still appear in the metrics for all other organizations in the enterprise they belong to.
What do you expect to happen
Organization-scoped dashboards should use an organization-scoped licensed-user population rather than filtering enterprise seats by their assigning organization.
How to reproduce
- Create an enterprise containing:
- a central organization used to assign Copilot seats
- a separate product organization
- Assign Copilot seats to users through the central organization.
- Make those users active members of the product organization.
- Enable GitHub Copilot usage metrics for the enterprise.
- Configure the DevLake
gh-copilotplugin with the enterprise and product organization. - Collect Copilot reports and seat assignments.
- Select the product organization in the bundled GitHub Copilot + DORA Correlation dashboard.
- Confirm that organization-scoped active-user metrics exist.
- Observe that the dashboard denominator is zero or otherwise incorrect because it filters seats by assigning organization.
- Observe that
NULLadoption weeks can be labeled>75%.
Diagnostic query:
SELECT
day,
daily_active_users
FROM _tool_copilot_enterprise_daily_metrics
WHERE connection_id = ?
AND scope_id = ?
ORDER BY day DESC;
Current denominator pattern:
SELECT COUNT(*)
FROM _tool_copilot_seats s
WHERE s.connection_id = ?
AND s.organization = ?;
The first query returns organization usage while the second returns no organization-assigned seats.
Anything else
Counting rows from _tool_copilot_user_daily_metrics is not a valid replacement denominator. Those reports represent daily user activity and produce weekday/weekend fluctuations rather than the complete licensed population.
The current-state seat table also cannot reconstruct historical weekly eligibility. A historical adoption trend needs either:
- a daily licensed-population snapshot, or
- an effective-dated license/membership source.
The dashboard should avoid silently applying today's seat state to historical dates.
Version
Observed with the gh-copilot plugin and bundled dashboard from the DevLake v1.0.3-beta10 generation. The same denominator pattern is present on the current main branch.
Are you willing to submit PR?
- Yes, I am willing to submit a PR.
Code of Conduct
- I agree to follow this project's Code of Conduct.
- Lenguaje dominante
- Go
- Estrellas
- 3.1k
- Forks
- 812
- Merge medio
- 1 d 23 h
- PR fusionados (30 d)
- 51
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de apache/devlake
-
type/bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
-
type/bug
Dificultad 4/5 3-5 días Aptitud para principiantes 52/100
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
Todos los issues de apache/devlake
Issues similares
-
bug github_actions
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
registrystack/registry-stack#1393 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
JakeChampion/lang#10213 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
oasisprotocol/oasis-sdk#2523 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100