IL2CPP: the first Deserialize through the generated resolver allocates 11 MB, growing with the number of registered types
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
Research direction
Start by examining the source generator's emitted GeneratedMessagePackResolverGetFormatterHelper and the resolver's FormatterCache<T> initialization path; the issue describes both as the source of eager IL2CPP initialization. Use the attached Unity 2022.3.57f1 repro and compare the first GetFormatter<Ping>() or Deserialize<Ping>() with later calls. Done means first-use cost depends on the requested type rather than all 2,139 registrations.
Written by the indexing model from the issue text.
Description
Bug description
With the source-generated resolver, the first MessagePackSerializer.Deserialize<T>() of a generated type pays for
every type registered in the assembly, not just T. On IL2CPP, deserializing a 2-byte payload into a one-field class
allocates 11 MB of native memory and initializes 5,755 classes when the assembly has 2,139 registrations. Nearly all of
it is the generated resolver's first GetFormatter<T>(), and the cost grows with the number of registrations.
In our game (Unity 2022.3.57, MessagePack 3.x), the first deserialization raises IL2CPP VM memory by more than 18 MB.
Repro steps
Attached: MessagePackIl2CppRepro.zip, a Unity 2022.3.57f1 project. It installs MessagePack 3.1.12 the way
MessagePack's README describes: MessagePack from NuGet through NuGetForUnity, and the MessagePack.Unity git package
at #v3.1.12. Unity needs git to fetch the two git packages, and the IL2CPP module for the editor's platform (on
Windows, also Visual Studio's C++ build tools).
- Open the project in Unity 2022.3.57f1.
- Choose Repro > Build IL2CPP Player (or File > Build And Run; the project already uses IL2CPP).
- Run the player. The results appear on screen and in
Player.log.
The project contains a generated message catalog (600 classes, 60 structs and 80 enums whose members use List<T>,
T[], Dictionary<K, V> and so on), which gives the generated resolver 2,139 registrations. The probe never uses those
types. It only runs:
MessagePackSerializer.Deserialize<int>(new byte[] { 0x01 }); // 1. warms up the standard resolvers
GeneratedMessagePackResolver.Instance.GetFormatter<Ping>(); // 2. what Deserialize<Ping> reaches through StandardResolver
MessagePackSerializer.Deserialize<Ping>(new byte[] { 0x91, 0x01 }); // 3. Ping: [MessagePackObject] class with one int field
MessagePackSerializer.Deserialize<Ping>(new byte[] { 0x91, 0x01 }); // 4.
Expected behavior
The first deserialization of a type loads only what that type needs, so its cost does not depend on how many other
types are registered.
Actual behavior
macOS standalone IL2CPP player on Apple silicon. Native memory is Profiler.GetTotalAllocatedMemoryLong(), which
includes IL2CPP's metadata; the other columns come from il2cpp_stats_get_value. The numbers were identical in every run.
| Step | Native memory | Classes initialized | Methods | Generic classes | Inflated methods |
|---|---|---|---|---|---|
1. Deserialize<int>, warm-up |
1,364 KB | 398 | 1,501 | 799 | 2,765 |
2. GeneratedMessagePackResolver.Instance.GetFormatter<Ping>() |
11,013 KB | 5,731 | 20,867 | 5,025 | 25,936 |
3. Deserialize<Ping> #1 |
64 KB | 24 | 44 | 39 | 93 |
4. Deserialize<Ping> #2 |
0 | 0 | 0 | 0 | 0 |
First use of Ping (2 + 3) |
11,077 KB | 5,755 | 20,911 | 5,064 | 26,029 |
Steps 2 and 3 together are what the first Deserialize<Ping> costs. Step 2, the generated resolver's first lookup,
accounts for 99% of it; step 3 is the rest of the resolver chain and the formatter itself.
- Version used: MessagePack 3.1.12 (NuGet), MessagePack.Unity
#v3.1.12 - Runtime: Unity 2022.3.57f1, IL2CPP, .NET Standard 2.1
Additional context
For this project, the generator emits this helper (excerpt; each list has 2,139 entries):
private static class GeneratedMessagePackResolverGetFormatterHelper
{
private static readonly global::System.Collections.Generic.Dictionary<global::System.Type, int> closedTypeLookup = new global::System.Collections.Generic.Dictionary<global::System.Type, int>(2139)
{
{ typeof(global::Repro.Messages.Msg0101[]), 0 },
// ...
{ typeof(global::System.Collections.Generic.List<global::Repro.Messages.Msg0100>), 897 },
// ...
};
internal static object GetFormatter(global::System.Type t)
{
if (closedTypeLookup.TryGetValue(t, out int closedKey))
{
switch (closedKey)
{
case 0: return new MsgPack::Formatters.ArrayFormatter<global::Repro.Messages.Msg0101>();
// ...
case 897: return new MsgPack::Formatters.ListFormatter<global::Repro.Messages.Msg0100>();
// ...
The first GeneratedMessagePackResolver.GetFormatter<T>() call for any T runs the static constructor of
FormatterCache<T>, which calls GeneratedMessagePackResolverGetFormatterHelper.GetFormatter(typeof(T)). That first
call:
- runs the helper's static constructor, which evaluates every
typeof(X), so IL2CPP creates the runtime class of every
registered type, including generic instantiations such asList<Msg0100>; - runs
GetFormatter(Type)for the first time. IL2CPP initializes all the metadata a method references on its first
call, so this initializes every formatter class and its generic instantiations, although only one formatter is created.
- Dominant language
- C#
- Stars
- 6.8k
- Forks
- 778
- Avg merge
- 22h 8m
- Merged PRs (30d)
- 2
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from MessagePack-CSharp/MessagePack-CSharp
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 1/5 Under an hour Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
MessagePack-CSharp/MessagePack-CSharp#2276 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
MessagePack-CSharp/MessagePack-CSharp#2252 · 3 comments ·
-
dynamic type fails sourcegenerator CS8785 'MessagePackGenerator' failed to generate source.Possibly taken @1AbdullahButt claimed this 7 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
MessagePack-CSharp/MessagePack-CSharp#2318 · 1 comment ·
All issues in MessagePack-CSharp/MessagePack-CSharp
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
Maintainers usually reply within 1 day
-
quality 提案 要トリアージ
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Maintainers usually reply within 1 day
-
The Language and Region lists in Options read to a screen reader as "LanguageInfo { Id = ar, ... }"Openbug localization small ui
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
hexide-io/HexIDE#4176 · 1 comment ·
Maintainers usually reply within 1 day
-
kind/chore status/draft
Difficulty 1/5 Under an hour Newbie friendliness 72/100
Altinn/altinn-profile#1102 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
MarcelRoozekrans/Rag.NET#1030 ·
Maintainers usually reply within 1 day