gRPC plugin framework should be able to recover from panics · jaegertracing/jaeger#1742

(7 comments) (0 reactions) (0 assignees)Go (2.326 forks)batch import

area/storagehelp wanted

Métricas do repositório

Stars: (18.974 stars)
Métricas de merge de PR: (Mesclagem média 4d 11h) (85 fundiu PRs em 30d)

Description

Requirement - what kind of business use case are you trying to solve?

We are implementing a custom gRPC-based storage plugin as per this doc.

Problem - what in Jaeger blocks you from solving the requirement?

There are two related problems:

When gRPC plugin panics the reason for panic is not shown in Jaeger logs, making the cause very hard to identify (related to: #1529)
When plugin panics it is not restarted, so Jaeger enters an unusable state

Impact:

Makes it very hard to make any plugin prod-ready
Makes developer experience of writing a plugin very frustrating
Plugin code becomes littered with defer/recover to prevent it from fully crashing and display helpful debug info

Proposal - what do you suggest to solve the problem or improve the existing situation?

Reason and a stacktrace should be already dumped into stderr of a plugin process at the time of panic, so Jaeger should be able to capture and log it.

Crashed plugins should be restarted.

Any open questions to address

Guia do colaborador

Direção de pesquisa: Investigue os caminhos do código do plugin gRPC onde podem ocorrer panics, depois implemente a recuperação usando defer/recover e lógica de reinicialização.
Pilha de tecnologia: gogrpc
Domain: backend
Tipo Issue: Funcionalidade
Difficulty: 3
Tempo estimado: Meio dia
Status da atividade: Ativo
Clarity: Claro
Prerequisites: GogRPC
Simpatia para novatos: 60