cube-js/cube

How one can disable cache or preaggregation for large datasets?

Open

#5341 opened on Sep 22, 2022

View on GitHub
 (2 comments) (2 reactions) (0 assignees)Rust (19,563 stars) (1,965 forks)batch import
help wanted

Description

Problem

Sometimes users can run smth like 'Select many dimensions and metrics for long period ". When it happens, huge portion of data is added to memory cache (we are still using redis, but I think problem will remain after adding cube store).

How can we disable adding result of query to cache by its size (e.g. limit 30 mb)? For us it's cheaper to make roundrip to db)

Contributor guide

Tech stack
javascripttypescriptnodejsredis
Domain
databaseperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic understanding of Cube.jsFamiliarity with caching concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Review Cube.js caching implementation, likely in `packages/cubejs server` or `src/cache`. The user requests a per query cache size limit. Investigate configuration options and propose adding a new setting like `maxCacheSize` to skip caching for queries exceeding a threshold. Check if there are any open PRs or comments discussing similar approaches.