Queries can fail in the AWS SQS/lambda integration poller, and are never retried

Open
  • Is something wrong?
#8,236 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
aws, java
Domain
backend, cloud, distributed-systems

Research direction

Look at QueryQueueStack and QueryStack to understand the SQS queue configuration with maxReceiveCount. Examine the query processing lambda integration and the failure tracking mechanism. The goal is to modify the system to detect and record queries that fail before lambda invocation, possibly by adding dead-letter queues and a failure tracking lambda.

Written by the indexing model from the issue text.

Description

bug parent-issue under-discussion
Description / Background

In the nightly test suite last night MultipleTablesST failed because one of 200 queries was lost in the AWS poller before it was ever delivered to the query processing lambda. We've got maxReceiveCount set to 1 for the query queues, which also includes failures in the AWS infrastructure for the SQS/lambda integration. The SQS metrics show 200 messages sent, 200 received but only 199 deleted, and there were only 199 invocations of the lambda.

Steps to reproduce
  1. Run MultipleTablesST, or make a lot of queries over a long period.
  2. Occasionally, a query is dropped and not processed.
  3. The SQS metrics show a message was received but not deleted.
  4. The query processing lambda was not invoked.
Expected behaviour

When a query fails it should be recorded in the query tracker.

Ideally queries should be retried if a recoverable failure occurs. This may be a difficult distinction to make, because there are other cases where a retry would be a waste of time. For example, for a web socket query the client may go away, or not be able to process the results. In that case the query will fail, but cannot be effectively retried.

It might be better to severely limit our use of retries, to avoid wasted compute and confusion about why a retried query was slow.

In the case where a query failed before ever invoking the lambda, if we can detect that specifically we might want to retry for just that case.

Technical Notes / Implementation Details

QueryQueueStack and QueryStack set maxReceiveCount(1) for both query queues.

It would help to add intermediate queues for dead letters to ensure they're recorded, both for parent queries and subqueries. An extra failure tracking lambda could record the failure in the query tracker if it hasn't already been recorded.

We could consider how the failure tracking lambda could detect the case where a query failed before ever invoking the lambda.

Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.