Discussion: Enable 5 minutes of out-of-order by default in Prometheus

Proposal

This is an issue to kick start a discussion to switch the default configuration for the out-of-order window.

Today the default value of the window is 0 meaning no out-of-order samples are allowed into Prometheus by default. it's up to the client to allow this use case and thus its up to the client to toggle the window and set it to a certain amount of minutes.

If we use Prometheus in the traditional way were it is pull based and it does scrapes based on a the configured interval this is completely fine and probably the out-of-order feature won't be needed.

However if Prometheus is used with remote-write due to a specific use case, certain network topologies, OTEL ingestion, scale or other reasons then we might run into issues. It's a paradigm change where order is hard to maintain.

It is this second use case where enabling out-of-order by default to a small window like 5m or 10m starts making a bit of sense.

• An example is using Prometheus with OTEL. Due to how OTEL clients work, encouraging users to batch up writes and then remote writing them all at once to improve compression rate, out-of-order is a natural outcome.

For any of these cases, the user needs to know about the existence of this feature and how to tweak it to avoid data loss. Maybe it does make sense to enable it to a small window to solve 99% of the race conditions between the writes?