Prometheus startup should discard data beyond the retention time

What did you do?

Came across a Prometheus that has been crashlooping for a month because its disk is full.

What did you expect to see?

At startup, delete WAL and storage blocks that are beyond retention time, before starting to write any new data.

What did you see instead?

The data is from well before the retention time (1635876000 = 02 Nov 2021) so can all be deleted, but it tries to start writing before doing any deleting and is crashlooping.

• Logs:

level=info ts=2021-12-14T14:44:04.005Z caller=main.go:400 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2021-12-14T14:44:04.005Z caller=main.go:438 msg="Starting Prometheus" version="(version=2.30.3, branch=HEAD, revision=f29caccc42557f6a8ec30ea9b3c8c089391bd5df)"
level=info ts=2021-12-14T14:44:04.005Z caller=main.go:443 build_context="(go=go1.17.1, user=root@5cff4265f0e3, date=20211005-16:10:52)"
level=info ts=2021-12-14T14:44:04.005Z caller=main.go:444 host_details="(Linux 5.4.120+ #1 SMP Wed Aug 18 10:20:32 PDT 2021 x86_64 billing-prometheus-1 (none))"
[...]
level=info ts=2021-12-14T14:44:04.007Z caller=main.go:822 msg="Starting TSDB ..."
level=info ts=2021-12-14T14:44:04.007Z caller=web.go:575 component=web msg="Router prefix" prefix=/billing-prometheus
level=info ts=2021-12-14T14:44:04.008Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1635811200079 maxt=1635876000000 ulid=01FKHE3PR8GXJ2F155YHP1QEV9
level=info ts=2021-12-14T14:44:04.008Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1635876000001 maxt=1635940800000 ulid=01FKKBXK4ZCAD0TFQWZVKRAKHQ
[...]
level=info ts=2021-12-14T14:44:04.009Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1637150400000 maxt=1637157600000 ulid=01FMQ6V1SNNTAGTSJXNVTBEG56
level=error ts=2021-12-14T14:44:04.059Z caller=main.go:915 err="opening storage failed: open /prometheus/data/wal/00002813: no space left on device"