Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug] BE aborts via std::terminate when a VARIANT read exceeds the query memory limit (allocation in noexcept release path)

Abierto
  • ¿Algo no está bien?
#68,898 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
22/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
database

Línea de trabajo

Start at release_batch_container in be/src/core/value/variant/variant_batch_builder.cpp (around line 134) and at _take_encoded_metadata() (around line 342), then check the noexcept functions there that call it, including release_scratch(). Run or write a test that drives the VARIANT V2 assembly path under a tight memory tracker limit. Done means the query returns MEM_LIMIT_EXCEEDED without the BE terminating, and the noexcept release paths no longer allocate.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Search before asking
  • I had searched in the issues and found no similar issues.
Version

4.1.4 (cloud mode, arm64); master is the same. On master release_batch_container is be/src/core/value/variant/variant_batch_builder.cpp:134 and _take_encoded_metadata() is :342.

What's Wrong?

A query that reads a VARIANT V2 column and exceeds its memory limit aborts the BE process with std::terminate, instead of failing the query with MEM_LIMIT_EXCEEDED. When the query runs on every BE, every BE aborts at once. In our case all 4 BEs aborted within about 10 s and restarted.

FE error for the query: [MEM_ALLOC_FAILED]Allocator mem tracker check failed, [MEM_LIMIT_EXCEEDED]failed alloc size 13.68 MB.

Symbolized BE stack:

#0  std::terminate handler
#3  __clang_call_terminate
#4  doris::VariantMetadataBuilder::_take_encoded_metadata()   (allocation, core/custom_allocator.h:133)
#5  doris::VariantBatchBuilder::finish_batch()                (core/value/variant/variant_batch_builder.cpp:1728)
#6  doris::segment_v2::variant_v2::VariantAssembler::assemble(...)
    ... segment iterator / scan operator frames

Cause:

template <typename Container>
void release_batch_container(Container& container) noexcept {
    Container().swap(container);
}

PaddedPODArray<char> VariantMetadataBuilder::_take_encoded_metadata() noexcept {
    ...
    release_batch_container(_impl->keys);   // Keys = std::deque<PaddedPODArray<char>, CustomStdAllocator<...>>
    ...
}

A default-constructed libstdc++ std::deque allocates its map and first node. With CustomStdAllocator, that allocation goes through MemoryAllocator::alloc, which throws doris::Exception (MEM_ALLOC_FAILED, "Allocator mem tracker check failed") once the thread's memory tracker is over its limit. The throw happens inside a noexcept function, so the runtime calls std::terminate. release_scratch() is also noexcept and uses the same helper. Other noexcept functions in this file may allocate as well, and are worth auditing for the same pattern.

What You Expected?

The query fails with MEM_LIMIT_EXCEEDED and the BE keeps running.

How to Reproduce?

A table with a VARIANT column (Variant V2) holding about 1B rows of nested JSON. A query that materializes the VARIANT for many rows under the default query memory limit, e.g.:

SELECT count(*), sum(length(j))
FROM (SELECT CAST(v AS STRING) AS j FROM t
      WHERE ts >= '...' AND ts < '...'
      ORDER BY ts DESC, id LIMIT 1000000) x;

Any query whose VARIANT read pushes the memory tracker over its limit while finish_batch runs should reproduce it. (Separately, the CAST is evaluated in the scan projection for every row before the top-N, rather than being deferred by lazy materialization. That makes this query far more memory-hungry than its LIMIT suggests.)

Anything Else?

Possible fix: make the release path unable to allocate or throw. For example, fall back to clear() when constructing an empty container throws, or drop noexcept where no caller relies on it, so the memory-limit exception reaches the query as a normal error.

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Lenguaje dominante
Java
Estrellas
16.1k
Forks
4k
Merge medio
2 d 23 h
PR fusionados (30 d)
686

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/doris

Todos los issues de apache/doris

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.