Squirrel/Squirrel.Windows

System.OutOfMemoryException when creating delta for (large) files

Open

#651 opened on Apr 5, 2016

View on GitHub
 (8 comments) (5 reactions) (0 assignees)C++ (7,025 stars) (1,061 forks)batch import
bug:old-versionhelp wanted

Description

The following happens with version 1.2.5 (via https://github.com/electron/windows-installer version 2.0.5) during diffing:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.File.InternalReadAllBytes(String path, Boolean checkHost)
   at Squirrel.DeltaPackageBuilder.createDeltaForSingleFile(FileInfo targetFile, DirectoryInfo workingDirectory, Dictionary`2 baseFileListing)
   at Squirrel.DeltaPackageBuilder.CreateDeltaPackage(ReleasePackage basePackage, ReleasePackage newPackage, String outputFile)
   at Squirrel.Update.Program.Releasify(String package, String targetDir, String packagesDir, String bootstrapperExe, String backgroundGif, String signingOpts, String baseUrl, String setupIcon, Boolean generateMsi)
   at Squirrel.Update.Program.executeCommandLine(String[] args)
   at Squirrel.Update.Program.main(String[] args)
   at Squirrel.Update.Program.Main(String[] args)

Note that there is still enough physical memory when this happens (about 4gb). The files are about 160mb. Works without problems for files that are around 50mb.

Contributor guide

Tech stack
csharp
Domain
desktop
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
C#memory profilingsquirrel.windows codebase
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Investigate the file DeltaPackageBuilder.cs, specifically the createDeltaForSingleFile method. The error occurs when using File.ReadAllBytes which loads the entire file into memory. Consider using streaming APIs (e.g., FileStream) to process the diff without loading the whole file. Also examine the baseFileListing dictionary for potential memory issues. Review the 8 comments in the issue for any proposed solutions or workarounds.