Caches.Cache: its finalizer keeps a dropped check alive for one more full GC, and MailboxProcessor-mode caches are never collected
Los mantenedores suelen responder en 3 días
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 18/100
Línea de trabajo
Start in src/Compiler/Utilities/Caches.fs: the finalizer near L397-L398, Dispose around L308-L309 and the eviction loop around L289-L301. Also look at getTypeSubsumptionCache in src/Compiler/Checking/TypeRelations.fs and getOverloadResolutionCache in OverloadResolutionCache.fs. Done means the finalizer is removed and the reflection repro shows the value collected after one full GC in MailboxProcessor mode. A linked pull request (#20758) already exists, so this is not open for a new contributor.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Cache<'Key, 'Value> in Caches.fs has a finalizer that calls Dispose (L397-L398). I measured two effects of it with FCS 43.12.201. Main has the same code in the places linked below.
1. A dropped check survives one extra full GC
MemoizationTable creates its cache with NoEviction (illib.fs L1103), and InfoReader has 11 of these tables. With NoEviction and Immediate, Dispose has no eviction processor to stop (L308-L309). It only counts the disposal for the metrics (L387-L390). MemoizationTable is not disposable, so the finalizer runs for each table.
An object with a finalizer survives the collection that finds it dead, and so does everything it references, until the finalizer has run and a later collection of its generation happens. A table references the infos it computed, and through them the type checker state. So when a check is dropped, that state survives the first full GC and is promoted to gen 2. A process that goes idle can wait a long time for the next gen 2 GC.
Measured:
-
A
Cachecreated through reflection, holding a 50 MB value. Long weak references, which stay set while an object waits for its finalizer:Mode Finalizer Value alive after one full GC After WaitForPendingFinalizersand a second full GCNoEvictionkept yes no NoEvictionGC.SuppressFinalizeno no Immediatekept yes no ImmediateGC.SuppressFinalizeno no -
FsAutoComplete on the Fantomas solution, transparent compiler. I called
ClearCachefor every project, then ran oneGC.Collect(2, GCCollectionMode.Aggressive, true, true)and took a heap dump. No root reached 62 MB of the heap, and 44.2 MB of that was reachable only from deadCacheinstances. An earlier run left 135 MB unreachable. WithGC.WaitForPendingFinalizers()and a second collection, 0.8 MB was left. FsAutoComplete now does that as a workaround.
2. A cache in MailboxProcessor mode is never collected
MailboxProcessor is the default eviction mode (L157). The eviction loop waits in mb.Receive() (L289-L301). FSharp.Core waits there with Async.AwaitWaitHandle, which registers the wait with the thread pool, and the loop references the cache. The root path from a dump:
root Stack System.Threading.PortableThreadPool+WaitThread
._registeredWaits -> System.Threading.RegisteredWaitHandle[]
.[] -> System.Threading.RegisteredWaitHandle
._callbackHelper -> System.Threading._ThreadPoolWaitOrTimerCallback
._waitOrTimerCallback -> System.Threading.WaitOrTimerCallback
._target -> <StartupCode$FSharp-Core>.$Async+AwaitWaitHandle@1918-3
.ctxt -> Microsoft.FSharp.Control.AsyncActivationContents<System.Boolean>
.cont@ -> Microsoft.FSharp.Control.AsyncPrimitives+Bind@589<EvictionQueueMessage<...>, ...>
.ctxt -> Microsoft.FSharp.Control.AsyncActivationContents<EvictionQueueMessage<...>>
.cont@ -> Microsoft.FSharp.Control.AsyncPrimitives+Bind@589<Unit, EvictionQueueMessage<...>>
.part2 -> <StartupCode$FSharp-Compiler-Service>.$Caches+processNext@295-1<...>
.this -> FSharp.Compiler.Caches.Cache<...>
So the cache stays reachable until something disposes it. The finalizer is meant to stop the loop when Dispose was not called, but it can never run. In the reflection test above, a MailboxProcessor cache and its 50 MB value were still alive after three full GCs, with or without GC.SuppressFinalize.
Outside CompilationMode.OneOff, getTypeSubsumptionCache (TypeRelations.fs L42-L49) and getOverloadResolutionCache (OverloadResolutionCache.fs L48-L63) create a cache of this kind for each TcGlobals, kept in a WeakMap. Nothing disposes them, so every TcGlobals leaves its cache and a thread pool wait behind.
Measured: an fsi script creates five FSharpCheckers one after the other, checks a script with a type test with each, and drops them. Then three full GCs, each followed by WaitForPendingFinalizers, and a heap dump:
| Type | Instances a root reaches |
|---|---|
TcGlobals |
1 (fsi's own) |
FSharpChecker |
1 (fsi's own) |
Cache<TTypeCacheKey, bool> (typeSubsumptionCache) |
6 |
RegisteredWaitHandle |
9 |
The background compiler and the transparent compiler give the same counts. The entries do not reference the typed tree, and in FsAutoComplete the caches held 0.3 to 2 MB each. So each TcGlobals leaks little, but nothing ever frees it. My script did not use the overload resolution cache, which is created the same way.
Proposal
- Remove the finalizer. With
NoEvictionandImmediateit has nothing to clean up, and withMailboxProcessorit cannot run. - Give the caches that nobody disposes,
typeSubsumptionCacheandoverloadResolutionCache, a lifetime they can end. Two options: useEvictionMode.Immediatefor them, or keep the eviction loop from making the cache reachable while it waits. Disposing them would need an owner, and theWeakMapis not one.
Repro: Cache lifetime per eviction mode (reflection)
#r "nuget: FSharp.Compiler.Service, 43.12.201"
open System
open System.Reflection
open System.Runtime.CompilerServices
open System.Collections.Generic
open Microsoft.FSharp.Reflection
let asm = typeof<FSharp.Compiler.CodeAnalysis.FSharpChecker>.Assembly
let flags = BindingFlags.Public ||| BindingFlags.NonPublic ||| BindingFlags.Static ||| BindingFlags.Instance
let cacheType = asm.GetType("FSharp.Compiler.Caches.Cache`2", true).MakeGenericType(typeof<string>, typeof<byte[]>)
let getDefault = asm.GetType("FSharp.Compiler.Caches.CacheOptions", true).GetMethod("getDefault", flags).MakeGenericMethod(typeof<string>)
[<MethodImpl(MethodImplOptions.NoInlining)>]
let create (mode: string) (suppressFinalizer: bool) =
let defaults = getDefault.Invoke(null, [| box (EqualityComparer<string>.Default :> IEqualityComparer<string>) |])
let optionsType = defaults.GetType()
let fields = FSharpValue.GetRecordFields(defaults, flags)
let modeCase = FSharpType.GetUnionCases(asm.GetType("FSharp.Compiler.Caches.EvictionMode", true), flags) |> Array.find (fun c -> c.Name = mode)
fields[FSharpType.GetRecordFields(optionsType, flags) |> Array.findIndex (fun p -> p.Name = "EvictionMode")] <- FSharpValue.MakeUnion(modeCase, [||], flags)
let cache = (cacheType.GetConstructors(flags) |> Array.exactlyOne).Invoke([| FSharpValue.MakeRecord(optionsType, fields, flags); box (Some "repro") |])
let value = Array.zeroCreate<byte> (50 * 1024 * 1024)
cacheType.GetMethod("TryAdd", flags).Invoke(cache, [| box "key"; box value |]) |> ignore
Threading.Thread.Sleep 200
if suppressFinalizer then GC.SuppressFinalize cache
// Long weak reference: it stays set while the object waits for its finalizer
WeakReference(value, true)
for mode in [ "NoEviction"; "Immediate"; "MailboxProcessor" ] do
for suppress in [ false; true ] do
let value = create mode suppress
GC.Collect(2, GCCollectionMode.Forced, true, true)
let afterOne = value.IsAlive
GC.WaitForPendingFinalizers()
GC.Collect(2, GCCollectionMode.Forced, true, true)
GC.WaitForPendingFinalizers()
GC.Collect(2, GCCollectionMode.Forced, true, true)
printfn "%-16s suppressed %-5b value alive after one GC %-5b after finalizers and two more GCs %b" mode suppress afterOne value.IsAlive
Repro: one typeSubsumptionCache left behind per dropped checker
Run with dotnet fsi leak.fsx background (or transparent), take a heap dump with dotnet-dump collect --type Heap -p <pid> while it sleeps, and count the instances a GC root reaches (I used ClrMD).
#r "nuget: FSharp.Compiler.Service, 43.12.201"
open System
open System.Runtime.CompilerServices
open FSharp.Compiler.CodeAnalysis
open FSharp.Compiler.Text
let useTransparentCompiler = fsi.CommandLineArgs[1] = "transparent"
let file = IO.Path.Combine(__SOURCE_DIRECTORY__, "Script.fsx")
// The type test makes the checker use the type subsumption cache
let source = SourceText.ofString "let f (o: obj) = match o with :? string as s -> s.Length | :? System.IComparable -> 1 | _ -> 0\nprintfn \"%d\" (f (box \"a\"))"
[<MethodImpl(MethodImplOptions.NoInlining)>]
let check () =
let checker = FSharpChecker.Create(useTransparentCompiler = useTransparentCompiler)
let options, _ = checker.GetProjectOptionsFromScript(file, source, assumeDotNetFramework = false) |> Async.RunSynchronously
let _, answer = checker.ParseAndCheckFileInProject(file, 0, source, options) |> Async.RunSynchronously
match answer with
| FSharpCheckFileAnswer.Succeeded _ -> ()
| FSharpCheckFileAnswer.Aborted -> failwith "aborted"
for _ in 1..5 do check ()
for _ in 1..3 do
GC.Collect(2, GCCollectionMode.Forced, true, true)
GC.WaitForPendingFinalizers()
printfn "pid %d" Environment.ProcessId
Threading.Thread.Sleep 120000
Related: #20755 (another thing the transparent compiler keeps in memory, found the same way).
- Lenguaje dominante
- F#
- Estrellas
- 4.3k
- Forks
- 880
- Merge medio
- 4 d 14 min
- PR fusionados (30 d)
- 99
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de dotnet/fsharp
-
Semantic classification cache for opened documents is never populated (written to the unopened-documents cache)Posiblemente ocupada @xperiandri la tomó hace 37 días. AbiertoNeeds-Triage
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
Los mantenedores suelen responder en 3 días
-
Needs-Triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
dotnet/fsharp#20265 · 1 comentario ·
Los mantenedores suelen responder en 3 días
-
Bug Needs-Triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 3 días
-
Feature Request Needs-Triage
Dificultad 3/5 1-2 días Aptitud para principiantes 57/100
Los mantenedores suelen responder en 3 días
-
Feature Request Needs-Triage
Dificultad 5/5 Más de una semana Aptitud para principiantes 35/100
Los mantenedores suelen responder en 3 días
Todos los issues de dotnet/fsharp
Issues similares
-
type: bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
blamechris/chroxy#8579 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
needs triage
Dificultad 2/5 1-3 horas Aptitud para principiantes 83/100
darnitdevorg/darnit#587 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 72/100
cline/mcp-marketplace#2932 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
nextlevelbuilder/ui-ux-pro-max-skill#525 · 2 comentarios ·
Los mantenedores suelen responder en 1 día