Cleanup: Automated data retention system

Open
#1,183 0 comments 0 reactions 1 assignee View on GitHub

@Prajna1999 is already working on this.

Since Sep 4, 2026.

Assessment

This issue has not been assessed yet.

Description

Is your feature request related to a problem?
We need a system to automatically clean up query and response data in the LLM calls table after a specified retention period. Without this, the database may become cluttered and inefficient.

Describe the solution you'd like

  • Implement a cron job to purge query/response data after a configurable retention window (default 1 week).
  • Handle one-time backlog separately from regular cron operations.
  • Write a cleanup strategy document detailing cron design, batching, and backlog handling.
  • Conduct a staging load test by duplicating ~20k LLM call rows ~5x in the copy_dev database to measure cleanup time. Do not use production data.
  • Ensure S3 cleanup is managed separately and not included in the DB cron/migration code.
Original issue

Context

query + response data in the LLM calls table will be cleaned up via a cron job after a configurable retention window (starting with 1 week).

A step-by-step cleanup strategy doc is to be written (cron design, batching, one-time backlog handling).

Scope / Acceptance criteria

  • Cron job to purge query/response data in the LLM calls table after a configurable retention window (default 1 week).
  • Handle one-time backlog separately from steady-state cron.
  • Write cleanup strategy doc (cron design, batching, backlog).
  • Staging load test: reuse the empty copy_dev database, duplicate staging's ~20k LLM call rows ~5x to reach ~1 lakh, and measure cleanup time. Do NOT pull prod data.
  • Keep S3 cleanup separate — do not bundle S3 deletion into the DB cron/migration code.

Notes

  • Guardrails input/output data is retained separately (anonymized, tagged as product enhancement) and is NOT part of this cleanup.
Dominant language
Python
Stars
18
Forks
10
Avg merge
2d 20h
Merged PRs (30d)
14

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ProjectTech4DevAI/kaapi-backend

All issues in ProjectTech4DevAI/kaapi-backend

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.