Opentelemetry Tracing

Description

Tracing is a form of telemetry data which tracks fine-grain information on requests as they flow through an application.

Tracing could help us in a number of ways:

• Debugging: If someone says "my request hung," can pull a trace and immediately see: was it stuck waiting for model spin-up, or in queue, or the model crashed?
• Development Help: Can be used to study system performance before/after a new feature was added (or help debug if something new isn't working)
• Very fine-grained benchmarking: For example, determining if there any features of a pickled nnsight experiment which correlate directly with experiment length.

Overall, it provides a very clean way to visualize and study how user requests flow through our system. This could provide useful insights into how particular components of NDIF perform under varying conditions/contexts.

Desired outcome

Implement some level of cross-service tracing using opentelemetry, with span data collected in jaeger and visualized in grafana. Even just getting a basic prototype which works across our 3 services (api, queue, ray) would be an incredible milestone.