Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Increased IO Wait in postgres

Ouverte
  • Quelque chose ne va pas ?
#995 6 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
25/100
Type d'issue
Bug
Clarté
À clarifier
Activité
À l'abandon
Stack technique
go, postgresql
Domaine
backend, databases

Piste de recherche

The report identifies River v0.20.2, PostgreSQL, and the BatchCompleter, JobScheduler, and Elector log paths; start by comparing those timeout failures with PostgreSQL I/O and job-state activity under the described workload. Done would require a reproducible cause and a narrowly defined corrective change, but the issue supplies no repository file or test to target.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Introduction

We have been using River for about a year, and we’ve had almost no issues that we couldn’t solve by tuning the configs or modifying the workers. Overall, we are very satisfied. However, in the last week, we encountered an issue twice that we haven’t been able to resolve yet.

We have the following versions in production:

github.com/riverqueue/river v0.20.2
github.com/riverqueue/river/riverdriver/riverpgxv5 v0.20.2
github.com/riverqueue/river/rivertype v0.20.2
Use-case

We run a go app in k8s, the rest API pods emit events, we insert these events to postgres as river jobs, and another deployment with river workers in it processes the events (make HTTP calls for the webhook subscriptions).

The problem

We noticed that fewer webhook calls were being made that the usual.
We checked what's happening in our DB with select state, count(*) from webhooks.river_job;:

   state   | count  
-----------+--------
 available |   1.404
 completed | 272.082
 discarded |  44.583
 retryable |  98.424
 running   |   2.739

Then we checked the error fields of the retryable jobs, and all of them errored with "Job rescued by rescuer".
We initially thought our workers couldn't process the jobs in time, but we only do an HTTP call in each job which has a 2 second timeout, so we were almost certain it's not the case.
Then we found many BatchCompleter: Completer error (will retry after sleep) and BatchCompleter: Too many errors; giving up log entries, all showing timeout: context deadline exceeded error. There were also JobScheduler: Error scheduling jobs and Elector: Error attempting reelection errors with the same context deadline exceeded cause.

For postgres we have a 4vCPU 16GB memory Cloud SQL instance. (Which is shared among some apps, but the current one is the most significant.)
Image
Here we see a significant increase in IO wait. We eventually switch to maintenance mode around 7:30PM and tried to fix the error, that's why the load decreased, it wasn't natural. (The horizontal line in the chart is a different metric, it does not mean we hit the CPU limit or something. The CPU utilization was ~70-80% so we never hit 100%, when we doulbed the resources the utilization was ~60%.)

What we tried
  • We deleted all the completed jobs, but the issue persisted
  • We updated all the retryable and running jobs to available status, but the issue persisted
  • We increased the DB resources to 8vCPU 32GB ram, the issue persisted
  • We stopped the app completely, removed everything but the 120k available jobs from the river_jobs table, did a VACUUM, but when we started the app, the BatchCompleter: Completer error (will retry after sleep) errors appeared almost instantly, and only ~30% of the jobs were completed, the rest stayed in running, eventually in retryable state.

Now we will update river to the newest version, and see if the issue appears again.
We don't expect anyone to solve our problems, but we are curious if you seen something similar before, or have some tips what might be causing this behavior.

Another minor detail is that we use river in multiple databases in this cluster, even in multiple schemas in the same database, in case this has anything to do with our issue.

Langage dominant
Go
Étoiles
5.7k
Forks
187
Merge moyen
22 h 47 min
PR mergées (30 j)
90

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de riverqueue/river

Toutes les issues de riverqueue/river

Issues similaires

Plus d'issues Go

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.