Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it

Open Beginner friendly
#3,748 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python, rust
Domain
documentation

Research direction

Start in the Python SDK under foreign/python at the consumer_group API and review its poll_interval type and docstring, then inspect the high-level SDK documentation example that uses a 1ms interval. Document the CPU impact of leaving the interval unset and show a non-1ms example; done means Python users can discover the setting and its idle-polling consequence without reading Rust source.

Written by the indexing model from the issue text.

Description

bug python sdk

TL;DR — Leaving poll_interval unset on consumer_group() makes the consumer re-poll as fast as the broker can answer. On a 1-vCPU host with an essentially idle topic that measured ~30% of the core (iggy-server 17.3% + consumer 12.6%) and ~34,700 context switches/sec; passing poll_interval=250ms took both processes below 0.3% with no functional change. This is my omission rather than an SDK bug — I'm filing it because two of my codebases made the identical omission independently, and because the consequence isn't discoverable from Python. Please re-triage as docs/enhancement if that fits better.

Bug description

My worker created its consumer without passing poll_interval. consume_messages(callback, shutdown_event) takes no interval either, so consumer_group() is the only place it can be set. With it unset the SDK leaves it None, and the consumer polls continuously.

The cost, on a topic receiving a handful of messages per day:

unset poll_interval=250ms
iggy-server 17.3 % < 0.3 %
consumer process 12.6 % < 0.3 %
context switches/sec ~34,700 ~2,300–3,900
PSI cpu some avg10 70.2 1.6

Stopping just the broker and the consumer took the whole machine to 100% idle, so this loop was effectively everything the host was doing. The server's own CPU tracks the client's poll rate — it drops below 0.3% while a consumer is still attached and polling at 250ms.

Three things made it hard to notice:

  1. In core/sdk/src/clients/consumer_builder.rs, polling_retry_interval is given a default of IggyDuration::ONE_SECOND, while polling_interval is a bare Option<IggyDuration> with no default — seeing the first handled for you makes the second look handled too.
  2. The Python stub types it poll_interval: Optional[datetime.timedelta] = None, and the complete docstring for consumer_group is "Creates a new consumer group consumer. Returns the consumer or a PyRuntimeError on failure." — a Python user cannot learn what omitting it does without reading the Rust source.
  3. The only poll_interval example in the docs is .poll_interval(IggyDuration::from_str("1ms")?) — 1000 polls/sec — presented with no note that it's a throughput setting for dedicated hardware.

That two separate projects of mine, written months apart, both omitted it is what makes me think it's the shape of the API rather than one careless call site.

Suggested direction

Either a small non-zero default for polling_interval, or (cheaper) one sentence in the Python consumer_group docstring plus a non-1ms example in the docs.

One question, not a complaint

Does Iggy support or plan server-side long-polling — the broker holding a poll open until a message arrives or a timeout expires (Kafka's fetch.max.wait.ms, SQS long polling)? That makes an idle consumer nearly free regardless of what the client sets, and would remove this category at the source rather than documenting around it.

Affected area / component

Python SDK (foreign/python), consumer configuration; core/sdk/src/clients/consumer_builder.rs defaults; docs (high-level SDK page).

Deployment

Single iggy-server binary under systemd, TCP transport on loopback, one stream / one topic / one partition, one consumer group.

Versions

Server iggy-server 0.8.0, Python SDK apache-iggy 0.8.0 (PyPI).

Hardware / environment

Proxmox LXC container, 1 vCPU, 4 GiB RAM, no swap, ~20 other services co-resident. Local disk.

Sample code

Before (the omission):

consumer = await client.consumer_group(
    group, stream, topic,
    polling_strategy=PollingStrategy.Next(),
    auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
)
await consumer.consume_messages(on_message, shutdown)

After:

consumer = await client.consumer_group(
    group, stream, topic,
    polling_strategy=PollingStrategy.Next(),
    auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
    poll_interval=timedelta(milliseconds=250),
)
Reproduction
  1. Run iggy-server and one Python consumer built as above, on a box where you can see per-process CPU clearly (a single core makes it obvious).
  2. Publish nothing — leave the topic idle.
  3. Observe iggy-server and the consumer process in top, and the context-switch rate in vmstat 2.
  4. Add poll_interval=timedelta(milliseconds=250) and repeat.

On Linux, cat /proc/pressure/cpu is the clearest single signal — it went from some avg10=70.2 to some avg10=1.6 across that change.

Note on framing

I want to be straightforward that this may be working as intended: Iggy is a throughput broker, and "set the interval, that's what it's for" is a fair answer. If that's the call, I'd suggest keeping only the documentation part — the Python docstring never mentions the consequence, and the sole published example is 1ms.

Dominant language
Rust
Stars
4.9k
Forks
442
Avg merge
2d 2h
Merged PRs (30d)
160

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iggy

All issues in apache/iggy

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.