BatchSpanProcessor#force_flush reports success before an in-flight export finishes and ignores its timeout while waiting

Open
#2,497 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
57/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
ruby

Research direction

Read sdk/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb, focusing on force_flush, the worker’s batch handling, and export_batch; the issue says this file is unchanged on main. Reproduce with the provided blocking exporter and check existing processor tests for flush and timeout coverage. Done when force_flush waits for in-flight exports, includes their result, and returns TIMEOUT within its deadline.

Written by the indexing model from the issue text.

Description

Description of the bug

BatchSpanProcessor#force_flush doesn't account for a batch the background worker has already taken off the queue and is still exporting. It also waits on @export_mutex with no deadline. Together these have two effects:

  1. It reports success while spans are still unexported. force_flush snapshots only spans. If the worker has just shifted a batch and is inside export_batch, that batch is in neither the queue nor the snapshot, so force_flush returns SUCCESS straight away while the export is still running. That export can still fail, and its failure is never reflected in the result. A caller that flushes before handing span IDs to another system, or before a FaaS-style suspend, can't rely on SUCCESS meaning that everything that ended before the call has been exported.
  2. It ignores timeout: while the worker is exporting. When the snapshot isn't empty, export_batch runs @export_mutex.synchronize, which waits for the worker's in-flight export with no bound. With a slow or stuck exporter, force_flush(timeout: 1) blocks for as long as that export takes and still returns SUCCESS. TracerProvider#force_flush holds its own @mutex for the whole call, so concurrent flushes on the same provider, and shutdown, queue up behind it.

The spec says ForceFlush should export all spans for which the processor has not yet completed the export, and that it should complete or abort within a timeout.

Version

opentelemetry-sdk 1.13.1; sdk/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb on main is unchanged in this respect. Ruby 3.3.

Reproduction
gem "opentelemetry-sdk", "1.13.1"
require "opentelemetry/sdk"

# An exporter whose first export blocks until released.
class BlockingExporter
  def initialize
    @started = Queue.new
    @release = Queue.new
    @calls = 0
  end
  attr_reader :started, :release

  def export(spans, timeout: nil)
    @calls += 1
    if @calls == 1
      @started << true
      @release.pop
      return OpenTelemetry::SDK::Trace::Export::FAILURE
    end
    OpenTelemetry::SDK::Trace::Export::SUCCESS
  end

  def force_flush(timeout: nil) = OpenTelemetry::SDK::Trace::Export::SUCCESS
  def shutdown(timeout: nil) = OpenTelemetry::SDK::Trace::Export::SUCCESS
end

OpenTelemetry.logger = Logger.new(File::NULL)
exporter = BlockingExporter.new
bsp = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(exporter, schedule_delay: 50, max_export_batch_size: 1, max_queue_size: 10)
provider = OpenTelemetry::SDK::Trace::TracerProvider.new
provider.add_span_processor(bsp)
tracer = provider.tracer("repro")

tracer.in_span("in-flight") {}
exporter.started.pop # the worker now holds the span and is exporting it

# 1. force_flush with an empty queue returns SUCCESS while the batch is still in flight (and will fail).
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
r1 = bsp.force_flush(timeout: 1)
puts "empty-queue force_flush: #{r1 == 0 ? 'SUCCESS' : r1} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - t).round(2)}s (in-flight export still running)"

# 2. With a queued span, force_flush blocks on the export mutex past its timeout.
tracer.in_span("queued") {}
Thread.new { sleep 3; exporter.release << true }
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
r2 = bsp.force_flush(timeout: 1)
puts "queued force_flush(timeout: 1): result #{r2} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - t).round(2)}s"
provider.shutdown(timeout: 5)

Output:

empty-queue force_flush: SUCCESS after 0.0s (in-flight export still running)
queued force_flush(timeout: 1): result 0 after 3.0s

The first call returns SUCCESS although the in-flight export then fails. The second blocks for three times its timeout and also returns SUCCESS (0).

Expected behaviour
  • force_flush also waits for an export that is in flight when it is called, and includes that export's result in its return value.
  • The wait honours timeout: and returns TIMEOUT when the deadline passes, leaving the in-flight export to finish in the background.
Possible approach

Track whether the worker is exporting (set when it takes a batch under @mutex, cleared when export_batch returns) together with that export's result. Have force_flush wait on a condition variable with the remaining timeout, instead of on @export_mutex.synchronize, until the in-flight export finishes. Then export its snapshot as it does today. That also removes the unbounded wait on @export_mutex. We're happy to send a PR if this direction sounds right.

Dominant language
Ruby
Stars
613
Forks
306
Avg merge
3d 11h
Merged PRs (30d)
21

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from open-telemetry/opentelemetry-ruby

All issues in open-telemetry/opentelemetry-ruby

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.