SFTtech/openage

Reduce converter memory usage with on-demand read

Open

#1,256 opened on May 4, 2020

View on GitHub
 (4 comments) (1 reaction) (0 assignees)Python (12,130 stars) (1,135 forks)batch import
area: assetsgood first issuelang: python

Description

Required skills: Python

Difficulty: Medium

The new converter (#1151) uses member objects to operate on data from the .dat files. While these provide the necessary functions for conversion, they are also significantly larger than the original dat file entries (20x - 400x the size). This results in a comparatively high memory usage, especially for DE2 which uses ~1.1 GB of memory for all dat file entries. Note that the converter does not create objects for all entries by default, but the size difference compared to the source is still significant.

A solution to this problem could be that we read certain structures into member objects on-demand during conversion, instead of all converting them during the initial read.

This would require:

  • A special ValueMember class that references an offset in the .dat file and a GenieStructure reference for the dataformat.
  • Manual loading and unloading of the structure from file
  • Dynamic loading and unloading of the structure from file
  • Identifying places in the converter where manual loading makes sense

Contributor guide

Tech stack
python
Domain
toolingperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonOOPconverter architecture
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Start by studying the existing converter implementation in the repository, particularly the member objects and how they are loaded from `.dat` files. Understand the memory overhead by profiling the current conversion process, especially for DE2 which uses ~1.1 GB. Then design a `ValueMember` class that references file offsets and a `GenieStructure` for lazy loading. Look at the linked PR #1151 for context on the new converter. Identify candidate structures in the converter where manual on demand loading makes sense.