Compiling in a long-lived process emits types in a different order (FileIndex is process-wide)

Aberta
  • Algo errado?
#20,834 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 2 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
fsharp
Domínio
build-system, compilers

Direção de pesquisa

Start with TypeDefsBuilder.AddTypeDef in src/Compiler/CodeGen/IlxGen.fs and FileIndex.fileIndexTable in src/Compiler/Utilities/range.fs; trace how fsc.fs assigns source file indices and how CreateILModule.fs creates the ReflectedDefinitions resource name. Run the supplied repro for fresh and reused FSharpChecker processes. Done means repeated compilation produces identical bytes and MVIDs, including projects with reflected definitions; the issue describes multiple process-wide ordering leaks and a candidate fix.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Feature Request

When one process compiles more than once (FSharpChecker.Compile, a compile server, any long-lived host), the emitted assembly can depend on what the process compiled before. The output is still valid, but it is not byte-identical to what a fresh fsc produces. So the MVID of the assembly and of its reference assembly change, and every dependent project rebuilds.

This sequence reproduces it: in one process, compile B (which references a prebuilt A.dll), then compile A. The TypeDef table of A comes out in reverse file order.

Repro steps

  1. Console app on FSharp.Compiler.Service (reproduced on 43.13.101-rc2.26509.113 and on main at d0b662a911).
  2. Process 1: compile A (One.fs, Two.fs, Three.fs, each with one type) and keep the output as A-ref.dll.
  3. Process 2: compile B with -r:A-ref.dll, then compile A again with identical arguments.
  4. Compare the TypeDef order and the bytes of the two A.dlls.
Program.fs 
open System
open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable
open System.Security.Cryptography
open FSharp.Compiler.CodeAnalysis

// args: fresh|after <workDir> <Microsoft.NETCore.App.Ref/.../ref/net10.0>
let mode, root, refPack = (fun (a: string[]) -> a[0], a[1], a[2]) (Environment.GetCommandLineArgs() |> Array.skip 1)
let checker = FSharpChecker.Create()

let refs =
    [ yield! Directory.GetFiles(refPack, "*.dll"); typeof<list<int>>.Assembly.Location ]
    |> List.map (fun r -> "-r:" + r)

let write dir name (text: string) =
    Directory.CreateDirectory dir |> ignore
    let p = Path.Combine(dir, name)
    File.WriteAllText(p, text)
    p

let compile out extraRefs files =
    let args =
        [| yield! [ "fsc.exe"; "-o:" + out; "--target:library"; "--noframework"; "--deterministic+"; "--debug-"; "--optimize+" ]
           yield! refs
           yield! extraRefs
           yield! files |]
    let diags, exn = checker.Compile args |> Async.RunSynchronously
    for d in diags do printfn "  %O" d
    match exn with Some e -> raise e | None -> ()

let sourcesA () =
    let dir = Path.Combine(root, "A")
    [ write dir "One.fs" "namespace A\ntype One() = member _.X = 1"
      write dir "Two.fs" "namespace A\ntype Two() = member _.X = 2"
      write dir "Three.fs" "namespace A\ntype Three() = member _.X = 3" ]

let report label (dll: string) =
    use pe = new PEReader(File.OpenRead dll)
    let md = pe.GetMetadataReader()
    let names =
        [ for h in md.TypeDefinitions do
            let t = md.GetTypeDefinition h
            if md.GetString t.Namespace = "A" then md.GetString t.Name ]
    let hash = SHA256.HashData(File.ReadAllBytes dll) |> Convert.ToHexString
    printfn "%-26s types: %-15s sha256: %s  mvid: %O" label (String.Join(" ", names)) hash[..15] (md.GetGuid(md.GetModuleDefinition().Mvid))

let outA = Path.Combine(root, "out", "A.dll")
Directory.CreateDirectory(Path.GetDirectoryName outA) |> ignore

match mode with
| "fresh" ->
    compile outA [] (sourcesA ())
    report "A, fresh process" outA
    File.Copy(outA, Path.Combine(root, "A-ref.dll"), true)
| "after" ->
    let b = write (Path.Combine(root, "B")) "B.fs" "module B\nlet x = A.One().X + A.Two().X + A.Three().X"
    compile (Path.Combine(root, "out", "B.dll")) [ "-r:" + Path.Combine(root, "A-ref.dll") ] [ b ]
    compile outA [] (sourcesA ())
    report "A, after B, same process" outA
| m -> failwithf "unknown mode %s" m

Run dotnet app.dll fresh <dir> <refpack>, then dotnet app.dll after <dir> <refpack>.

Expected behavior

The second compile of A is byte-identical to the first: One Two Three, same hash, same MVID.

Actual behavior

A, fresh process           types: One Two Three   sha256: 1867DAF63BFC6329  mvid: 685208f9-72bc-79f3-01ab-cada7eaf998b
A, after B, same process   types: Three Two One   sha256: 5BD304310F3CA5C3  mvid: 216756e6-903a-c1bf-6f7e-bff3e80b3d7c

The same happens with signature files, nested modules and closures (for example Alpha Beta Gamma becomes Beta Alpha Gamma).

Cause

TypeDefsBuilder.AddTypeDef (src/Compiler/CodeGen/IlxGen.fs) sorts type definitions by struct (false, 0, m.FileIndex, m.StartLine, m.StartColumn, idx, name). FileIndex comes from the process-wide FileIndex.fileIndexTable (src/Compiler/Utilities/range.fs), which gives a file its number the first time anything in the process names it.

  • A fresh fsc names the source files in compilation order: fsc.fs calls fileIndexOfFile on every source file before parsing (the ilSourceDocs loop).
  • Importing B's references unpickles A's ranges (u_range → mkRange), which names A's files earlier and in another order.
  • When A is compiled later, its files keep those earlier numbers, so its types are emitted in that order.

TypeDefsBuilder is the only place in IlxGen where FileIndex sets output order. Its other uses (raw data field keys, memoize_file) are lookups.

Candidate fix

Map each FileIndex to the file's position in this compilation before sorting, in GenerateCode. A colleague has a patch that does this. It finds each file's position from the ranges of the entities each CheckedImplFile defines (Range, SigRange, DefinitionRange) plus the QualifiedNameOfFile range. A signature file sorts just before its implementation, and any file that is not a source file (FSharp.Core, references, range0) keeps its current place before or after the source files, ordered by raw index. I built main with and without that patch and ran the repro above plus a richer project (a signature file, nested modules, closures, an object expression, a generic type, async/seq):

  • With the patch, the second compile is byte-identical to the fresh one.
  • In a fresh process, main and the patched build produce byte-identical output for both projects.

Passing the ordered source file list from fsc.fs to IlxGen may be simpler than deriving positions from entity ranges. Known limit of either approach: types whose range is in a file that is not a source file (for example code inlined from a reference) are still ordered by raw index among themselves.

Second leak of the same kind: ReflectedDefinitions resource name

With the patch applied, a project that contains [<ReflectedDefinition>] still differs after an earlier compile in the same process. The only difference is the resource name:

< resource ReflectedDefinitions-R-4921-1376909745
> resource ReflectedDefinitions-R-9265-1376909745

CreateILModule.fs builds that name with string (newUnique ()), a process-wide counter, so it depends on how much the process compiled before. Without the quotation, the patched build gives byte-identical output.

Known workarounds

Compile each project in a fresh process.

Related information

  • Linux, .NET 10 runtime, .NET SDK 11.0.100-rc.2
  • FCS 43.13.101-rc2.26509.113, and main at d0b662a911
Linguagem predominante
F#
Estrelas
4.3k
Forks
884
Merge médio
2d 11h
PRs com merge (30d)
73

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dotnet/fsharp

Todas as issues de dotnet/fsharp

Issues semelhantes

Mais issues de Build System

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.