BatchSpanProcessor#force_flush reports success before an in-flight export finishes and ignores its timeout while waiting
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 57/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- ruby
- Domain
- observability-sre
Research direction
Read sdk/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb, focusing on force_flush, the worker’s batch handling, and export_batch; the issue says this file is unchanged on main. Reproduce with the provided blocking exporter and check existing processor tests for flush and timeout coverage. Done when force_flush waits for in-flight exports, includes their result, and returns TIMEOUT within its deadline.
Written by the indexing model from the issue text.
Description
Description of the bug
BatchSpanProcessor#force_flush doesn't account for a batch the background worker has already taken off the queue and is still exporting. It also waits on @export_mutex with no deadline. Together these have two effects:
- It reports success while spans are still unexported.
force_flushsnapshots onlyspans. If the worker has just shifted a batch and is insideexport_batch, that batch is in neither the queue nor the snapshot, soforce_flushreturnsSUCCESSstraight away while the export is still running. That export can still fail, and its failure is never reflected in the result. A caller that flushes before handing span IDs to another system, or before a FaaS-style suspend, can't rely onSUCCESSmeaning that everything that ended before the call has been exported. - It ignores
timeout:while the worker is exporting. When the snapshot isn't empty,export_batchruns@export_mutex.synchronize, which waits for the worker's in-flight export with no bound. With a slow or stuck exporter,force_flush(timeout: 1)blocks for as long as that export takes and still returnsSUCCESS.TracerProvider#force_flushholds its own@mutexfor the whole call, so concurrent flushes on the same provider, andshutdown, queue up behind it.
The spec says ForceFlush should export all spans for which the processor has not yet completed the export, and that it should complete or abort within a timeout.
Version
opentelemetry-sdk 1.13.1; sdk/lib/opentelemetry/sdk/trace/export/batch_span_processor.rb on main is unchanged in this respect. Ruby 3.3.
Reproduction
gem "opentelemetry-sdk", "1.13.1"
require "opentelemetry/sdk"
# An exporter whose first export blocks until released.
class BlockingExporter
def initialize
@started = Queue.new
@release = Queue.new
@calls = 0
end
attr_reader :started, :release
def export(spans, timeout: nil)
@calls += 1
if @calls == 1
@started << true
@release.pop
return OpenTelemetry::SDK::Trace::Export::FAILURE
end
OpenTelemetry::SDK::Trace::Export::SUCCESS
end
def force_flush(timeout: nil) = OpenTelemetry::SDK::Trace::Export::SUCCESS
def shutdown(timeout: nil) = OpenTelemetry::SDK::Trace::Export::SUCCESS
end
OpenTelemetry.logger = Logger.new(File::NULL)
exporter = BlockingExporter.new
bsp = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(exporter, schedule_delay: 50, max_export_batch_size: 1, max_queue_size: 10)
provider = OpenTelemetry::SDK::Trace::TracerProvider.new
provider.add_span_processor(bsp)
tracer = provider.tracer("repro")
tracer.in_span("in-flight") {}
exporter.started.pop # the worker now holds the span and is exporting it
# 1. force_flush with an empty queue returns SUCCESS while the batch is still in flight (and will fail).
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
r1 = bsp.force_flush(timeout: 1)
puts "empty-queue force_flush: #{r1 == 0 ? 'SUCCESS' : r1} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - t).round(2)}s (in-flight export still running)"
# 2. With a queued span, force_flush blocks on the export mutex past its timeout.
tracer.in_span("queued") {}
Thread.new { sleep 3; exporter.release << true }
t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
r2 = bsp.force_flush(timeout: 1)
puts "queued force_flush(timeout: 1): result #{r2} after #{(Process.clock_gettime(Process::CLOCK_MONOTONIC) - t).round(2)}s"
provider.shutdown(timeout: 5)
Output:
empty-queue force_flush: SUCCESS after 0.0s (in-flight export still running)
queued force_flush(timeout: 1): result 0 after 3.0s
The first call returns SUCCESS although the in-flight export then fails. The second blocks for three times its timeout and also returns SUCCESS (0).
Expected behaviour
force_flushalso waits for an export that is in flight when it is called, and includes that export's result in its return value.- The wait honours
timeout:and returnsTIMEOUTwhen the deadline passes, leaving the in-flight export to finish in the background.
Possible approach
Track whether the worker is exporting (set when it takes a batch under @mutex, cleared when export_batch returns) together with that export's result. Have force_flush wait on a condition variable with the remaining timeout, instead of on @export_mutex.synchronize, until the in-flight export finishes. Then export its snapshot as it does today. That also removes the unbounded wait on @export_mutex. We're happy to send a PR if this direction sounds right.
- Dominant language
- Ruby
- Stars
- 613
- Forks
- 306
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 21
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from open-telemetry/opentelemetry-ruby
-
enhancement metrics spec-compliance spec:development stale
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
open-telemetry/opentelemetry-ruby#2369 · 2 comments ·
Maintainers usually reply within 2 days
-
Make instrument name conflict detection case-insensitivePossibly taken @naveedsajjdawan claimed this 41 days ago. Openbug good first issue metrics spec-compliance spec:stable stale
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
open-telemetry/opentelemetry-ruby#2357 · 2 comments · 1 assignee ·
Maintainers usually reply within 2 days
-
keep metrics
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
open-telemetry/opentelemetry-ruby#2129 · 2 comments ·
Maintainers usually reply within 2 days
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
open-telemetry/opentelemetry-ruby#2500 · 2 comments ·
Maintainers usually reply within 2 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
open-telemetry/opentelemetry-ruby#2437 · 2 comments ·
Maintainers usually reply within 2 days
All issues in open-telemetry/opentelemetry-ruby
Similar issues
-
🐛 bug 👩💻 équipe CE 🧑🏭 entreprises
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
betagouv/conseillers-entreprises#4808 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
yosiat/panko_serializer#250 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
cline/mcp-marketplace#3886 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100