Compiling in a long-lived process emits types in a different order (FileIndex is process-wide)

Abierto
  • ¿Algo no está bien?
#20,834 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
fsharp
Área
build-system, compilers

Línea de trabajo

Start with TypeDefsBuilder.AddTypeDef in src/Compiler/CodeGen/IlxGen.fs and FileIndex.fileIndexTable in src/Compiler/Utilities/range.fs; trace how fsc.fs assigns source file indices and how CreateILModule.fs creates the ReflectedDefinitions resource name. Run the supplied repro for fresh and reused FSharpChecker processes. Done means repeated compilation produces identical bytes and MVIDs, including projects with reflected definitions; the issue describes multiple process-wide ordering leaks and a candidate fix.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Feature Request

When one process compiles more than once (FSharpChecker.Compile, a compile server, any long-lived host), the emitted assembly can depend on what the process compiled before. The output is still valid, but it is not byte-identical to what a fresh fsc produces. So the MVID of the assembly and of its reference assembly change, and every dependent project rebuilds.

This sequence reproduces it: in one process, compile B (which references a prebuilt A.dll), then compile A. The TypeDef table of A comes out in reverse file order.

Repro steps

  1. Console app on FSharp.Compiler.Service (reproduced on 43.13.101-rc2.26509.113 and on main at d0b662a911).
  2. Process 1: compile A (One.fs, Two.fs, Three.fs, each with one type) and keep the output as A-ref.dll.
  3. Process 2: compile B with -r:A-ref.dll, then compile A again with identical arguments.
  4. Compare the TypeDef order and the bytes of the two A.dlls.
Program.fs 
open System
open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable
open System.Security.Cryptography
open FSharp.Compiler.CodeAnalysis

// args: fresh|after <workDir> <Microsoft.NETCore.App.Ref/.../ref/net10.0>
let mode, root, refPack = (fun (a: string[]) -> a[0], a[1], a[2]) (Environment.GetCommandLineArgs() |> Array.skip 1)
let checker = FSharpChecker.Create()

let refs =
    [ yield! Directory.GetFiles(refPack, "*.dll"); typeof<list<int>>.Assembly.Location ]
    |> List.map (fun r -> "-r:" + r)

let write dir name (text: string) =
    Directory.CreateDirectory dir |> ignore
    let p = Path.Combine(dir, name)
    File.WriteAllText(p, text)
    p

let compile out extraRefs files =
    let args =
        [| yield! [ "fsc.exe"; "-o:" + out; "--target:library"; "--noframework"; "--deterministic+"; "--debug-"; "--optimize+" ]
           yield! refs
           yield! extraRefs
           yield! files |]
    let diags, exn = checker.Compile args |> Async.RunSynchronously
    for d in diags do printfn "  %O" d
    match exn with Some e -> raise e | None -> ()

let sourcesA () =
    let dir = Path.Combine(root, "A")
    [ write dir "One.fs" "namespace A\ntype One() = member _.X = 1"
      write dir "Two.fs" "namespace A\ntype Two() = member _.X = 2"
      write dir "Three.fs" "namespace A\ntype Three() = member _.X = 3" ]

let report label (dll: string) =
    use pe = new PEReader(File.OpenRead dll)
    let md = pe.GetMetadataReader()
    let names =
        [ for h in md.TypeDefinitions do
            let t = md.GetTypeDefinition h
            if md.GetString t.Namespace = "A" then md.GetString t.Name ]
    let hash = SHA256.HashData(File.ReadAllBytes dll) |> Convert.ToHexString
    printfn "%-26s types: %-15s sha256: %s  mvid: %O" label (String.Join(" ", names)) hash[..15] (md.GetGuid(md.GetModuleDefinition().Mvid))

let outA = Path.Combine(root, "out", "A.dll")
Directory.CreateDirectory(Path.GetDirectoryName outA) |> ignore

match mode with
| "fresh" ->
    compile outA [] (sourcesA ())
    report "A, fresh process" outA
    File.Copy(outA, Path.Combine(root, "A-ref.dll"), true)
| "after" ->
    let b = write (Path.Combine(root, "B")) "B.fs" "module B\nlet x = A.One().X + A.Two().X + A.Three().X"
    compile (Path.Combine(root, "out", "B.dll")) [ "-r:" + Path.Combine(root, "A-ref.dll") ] [ b ]
    compile outA [] (sourcesA ())
    report "A, after B, same process" outA
| m -> failwithf "unknown mode %s" m

Run dotnet app.dll fresh <dir> <refpack>, then dotnet app.dll after <dir> <refpack>.

Expected behavior

The second compile of A is byte-identical to the first: One Two Three, same hash, same MVID.

Actual behavior

A, fresh process           types: One Two Three   sha256: 1867DAF63BFC6329  mvid: 685208f9-72bc-79f3-01ab-cada7eaf998b
A, after B, same process   types: Three Two One   sha256: 5BD304310F3CA5C3  mvid: 216756e6-903a-c1bf-6f7e-bff3e80b3d7c

The same happens with signature files, nested modules and closures (for example Alpha Beta Gamma becomes Beta Alpha Gamma).

Cause

TypeDefsBuilder.AddTypeDef (src/Compiler/CodeGen/IlxGen.fs) sorts type definitions by struct (false, 0, m.FileIndex, m.StartLine, m.StartColumn, idx, name). FileIndex comes from the process-wide FileIndex.fileIndexTable (src/Compiler/Utilities/range.fs), which gives a file its number the first time anything in the process names it.

  • A fresh fsc names the source files in compilation order: fsc.fs calls fileIndexOfFile on every source file before parsing (the ilSourceDocs loop).
  • Importing B's references unpickles A's ranges (u_range → mkRange), which names A's files earlier and in another order.
  • When A is compiled later, its files keep those earlier numbers, so its types are emitted in that order.

TypeDefsBuilder is the only place in IlxGen where FileIndex sets output order. Its other uses (raw data field keys, memoize_file) are lookups.

Candidate fix

Map each FileIndex to the file's position in this compilation before sorting, in GenerateCode. A colleague has a patch that does this. It finds each file's position from the ranges of the entities each CheckedImplFile defines (Range, SigRange, DefinitionRange) plus the QualifiedNameOfFile range. A signature file sorts just before its implementation, and any file that is not a source file (FSharp.Core, references, range0) keeps its current place before or after the source files, ordered by raw index. I built main with and without that patch and ran the repro above plus a richer project (a signature file, nested modules, closures, an object expression, a generic type, async/seq):

  • With the patch, the second compile is byte-identical to the fresh one.
  • In a fresh process, main and the patched build produce byte-identical output for both projects.

Passing the ordered source file list from fsc.fs to IlxGen may be simpler than deriving positions from entity ranges. Known limit of either approach: types whose range is in a file that is not a source file (for example code inlined from a reference) are still ordered by raw index among themselves.

Second leak of the same kind: ReflectedDefinitions resource name

With the patch applied, a project that contains [<ReflectedDefinition>] still differs after an earlier compile in the same process. The only difference is the resource name:

< resource ReflectedDefinitions-R-4921-1376909745
> resource ReflectedDefinitions-R-9265-1376909745

CreateILModule.fs builds that name with string (newUnique ()), a process-wide counter, so it depends on how much the process compiled before. Without the quotation, the patched build gives byte-identical output.

Known workarounds

Compile each project in a fresh process.

Related information

  • Linux, .NET 10 runtime, .NET SDK 11.0.100-rc.2
  • FCS 43.13.101-rc2.26509.113, and main at d0b662a911
Lenguaje dominante
F#
Estrellas
4.3k
Forks
884
Merge medio
2 d 11 h
PR fusionados (30 d)
73

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/fsharp

Todos los issues de dotnet/fsharp

Issues similares

Más issues de Build System

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.