Move transfer session FSICs to their own table

Open

#166 opened on May 9, 2022 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 35/100

Issue type: Refactor
Clarity: Mostly clear
Activity status: Stale
Tech stack: django, python
Domain: api, backend, database, performance

Research direction

Start by locating the transfer session model, its JSON FSIC fields, the API serialization, and the queuing operation queries. Review commit 9f919c997b79171143e953e9f8d65f5a5c8740da alongside the recent queuing changes. Done means the new TransferSessionCounter table and constraint exist, API fields remain compatible, and queuing queries join against the table.

Written by the indexing model from the issue text.

Description

DEV: backend P1 - important TAG: performance

Background

The transfer session model tracks both client and server FSICs within JSON database fields for a directional sync. As time has gone on, these fields have begun to represent a significant amount of data. The data within these fields are then used to generate complex query filters against other tables within the database when queuing data for a sync.

Summary

As someone concerned with performance and scaling of Morango, I'd like to move these FSIC fields to a new, dedicated table, thereby reducing the size of the transfer session table (currently 200+GB). I'd like to take advantage of some performance improvements that can be made by leveraging this data directly within the database when queuing records for syncing.

Deliverables

A new table TransferSessionCounter should be created with fields:
- transfer_session FK
- instance_id UUID
- partition nullable text field
- server_counter nullable integer
- client_counter nullable integer
The table should be have a unique constraint on transfer_session and instance_id
API endpoints should serialize the table's data into two separate client_fsic and server_fsic fields to maintain compatibility
Queuing operation queries should join against this new table

Notes

Some progress has been made but was done prior to several recent changes to queuing queries https://github.com/learningequality/morango/commit/9f919c997b79171143e953e9f8d65f5a5c8740da

Dominant language: Python
Stars: 15
Forks: 23
Avg merge: 5d 1h
Merged PRs (30d): 1

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

PATCH /interviews/{id} 的 goal 字段没有任何长度上限，500KB 字符串直接落库且每次 GET detail 都全量带回 Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

xinnan-tech/xiaozhi-fde-talk#263 · Opened Sep 24, 2026
rules: REQ-0480 'left fold' jargon is redundant with its own plain restatement Open

rules

Difficulty 1/5 Under an hour Newbie friendliness 90/100

elong0527/yamaa#978 · Opened Sep 24, 2026
Validation discards output from runners that report on stderr (jest, unittest) Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

huggingface/Repo2RLEnv#163 · 1 comment · Opened Sep 24, 2026
Source link in quantize_embeddings have shifted code highlights Open

Difficulty 1/5 Under an hour Newbie friendliness 95/100

huggingface/sentence-transformers#4074 · Opened Sep 24, 2026
Queue insertion button ("插入排队消息") missing in api_server sessions — canInsert allowlist only covers cli/global_agent Open

comp/dashboard invalid P3

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

NousResearch/hermes-agent#121143 · Opened Sep 24, 2026

More Python issues