Compiling in a long-lived process emits types in a different order (FileIndex is process-wide)

Đang mở
  • Có gì không ổn?
#20,834 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 2 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
fsharp
Lĩnh vực
build-system, compilers

Hướng nghiên cứu

Start with TypeDefsBuilder.AddTypeDef in src/Compiler/CodeGen/IlxGen.fs and FileIndex.fileIndexTable in src/Compiler/Utilities/range.fs; trace how fsc.fs assigns source file indices and how CreateILModule.fs creates the ReflectedDefinitions resource name. Run the supplied repro for fresh and reused FSharpChecker processes. Done means repeated compilation produces identical bytes and MVIDs, including projects with reflected definitions; the issue describes multiple process-wide ordering leaks and a candidate fix.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Feature Request

When one process compiles more than once (FSharpChecker.Compile, a compile server, any long-lived host), the emitted assembly can depend on what the process compiled before. The output is still valid, but it is not byte-identical to what a fresh fsc produces. So the MVID of the assembly and of its reference assembly change, and every dependent project rebuilds.

This sequence reproduces it: in one process, compile B (which references a prebuilt A.dll), then compile A. The TypeDef table of A comes out in reverse file order.

Repro steps

  1. Console app on FSharp.Compiler.Service (reproduced on 43.13.101-rc2.26509.113 and on main at d0b662a911).
  2. Process 1: compile A (One.fs, Two.fs, Three.fs, each with one type) and keep the output as A-ref.dll.
  3. Process 2: compile B with -r:A-ref.dll, then compile A again with identical arguments.
  4. Compare the TypeDef order and the bytes of the two A.dlls.
Program.fs 
open System
open System.IO
open System.Reflection.Metadata
open System.Reflection.PortableExecutable
open System.Security.Cryptography
open FSharp.Compiler.CodeAnalysis

// args: fresh|after <workDir> <Microsoft.NETCore.App.Ref/.../ref/net10.0>
let mode, root, refPack = (fun (a: string[]) -> a[0], a[1], a[2]) (Environment.GetCommandLineArgs() |> Array.skip 1)
let checker = FSharpChecker.Create()

let refs =
    [ yield! Directory.GetFiles(refPack, "*.dll"); typeof<list<int>>.Assembly.Location ]
    |> List.map (fun r -> "-r:" + r)

let write dir name (text: string) =
    Directory.CreateDirectory dir |> ignore
    let p = Path.Combine(dir, name)
    File.WriteAllText(p, text)
    p

let compile out extraRefs files =
    let args =
        [| yield! [ "fsc.exe"; "-o:" + out; "--target:library"; "--noframework"; "--deterministic+"; "--debug-"; "--optimize+" ]
           yield! refs
           yield! extraRefs
           yield! files |]
    let diags, exn = checker.Compile args |> Async.RunSynchronously
    for d in diags do printfn "  %O" d
    match exn with Some e -> raise e | None -> ()

let sourcesA () =
    let dir = Path.Combine(root, "A")
    [ write dir "One.fs" "namespace A\ntype One() = member _.X = 1"
      write dir "Two.fs" "namespace A\ntype Two() = member _.X = 2"
      write dir "Three.fs" "namespace A\ntype Three() = member _.X = 3" ]

let report label (dll: string) =
    use pe = new PEReader(File.OpenRead dll)
    let md = pe.GetMetadataReader()
    let names =
        [ for h in md.TypeDefinitions do
            let t = md.GetTypeDefinition h
            if md.GetString t.Namespace = "A" then md.GetString t.Name ]
    let hash = SHA256.HashData(File.ReadAllBytes dll) |> Convert.ToHexString
    printfn "%-26s types: %-15s sha256: %s  mvid: %O" label (String.Join(" ", names)) hash[..15] (md.GetGuid(md.GetModuleDefinition().Mvid))

let outA = Path.Combine(root, "out", "A.dll")
Directory.CreateDirectory(Path.GetDirectoryName outA) |> ignore

match mode with
| "fresh" ->
    compile outA [] (sourcesA ())
    report "A, fresh process" outA
    File.Copy(outA, Path.Combine(root, "A-ref.dll"), true)
| "after" ->
    let b = write (Path.Combine(root, "B")) "B.fs" "module B\nlet x = A.One().X + A.Two().X + A.Three().X"
    compile (Path.Combine(root, "out", "B.dll")) [ "-r:" + Path.Combine(root, "A-ref.dll") ] [ b ]
    compile outA [] (sourcesA ())
    report "A, after B, same process" outA
| m -> failwithf "unknown mode %s" m

Run dotnet app.dll fresh <dir> <refpack>, then dotnet app.dll after <dir> <refpack>.

Expected behavior

The second compile of A is byte-identical to the first: One Two Three, same hash, same MVID.

Actual behavior

A, fresh process           types: One Two Three   sha256: 1867DAF63BFC6329  mvid: 685208f9-72bc-79f3-01ab-cada7eaf998b
A, after B, same process   types: Three Two One   sha256: 5BD304310F3CA5C3  mvid: 216756e6-903a-c1bf-6f7e-bff3e80b3d7c

The same happens with signature files, nested modules and closures (for example Alpha Beta Gamma becomes Beta Alpha Gamma).

Cause

TypeDefsBuilder.AddTypeDef (src/Compiler/CodeGen/IlxGen.fs) sorts type definitions by struct (false, 0, m.FileIndex, m.StartLine, m.StartColumn, idx, name). FileIndex comes from the process-wide FileIndex.fileIndexTable (src/Compiler/Utilities/range.fs), which gives a file its number the first time anything in the process names it.

  • A fresh fsc names the source files in compilation order: fsc.fs calls fileIndexOfFile on every source file before parsing (the ilSourceDocs loop).
  • Importing B's references unpickles A's ranges (u_range → mkRange), which names A's files earlier and in another order.
  • When A is compiled later, its files keep those earlier numbers, so its types are emitted in that order.

TypeDefsBuilder is the only place in IlxGen where FileIndex sets output order. Its other uses (raw data field keys, memoize_file) are lookups.

Candidate fix

Map each FileIndex to the file's position in this compilation before sorting, in GenerateCode. A colleague has a patch that does this. It finds each file's position from the ranges of the entities each CheckedImplFile defines (Range, SigRange, DefinitionRange) plus the QualifiedNameOfFile range. A signature file sorts just before its implementation, and any file that is not a source file (FSharp.Core, references, range0) keeps its current place before or after the source files, ordered by raw index. I built main with and without that patch and ran the repro above plus a richer project (a signature file, nested modules, closures, an object expression, a generic type, async/seq):

  • With the patch, the second compile is byte-identical to the fresh one.
  • In a fresh process, main and the patched build produce byte-identical output for both projects.

Passing the ordered source file list from fsc.fs to IlxGen may be simpler than deriving positions from entity ranges. Known limit of either approach: types whose range is in a file that is not a source file (for example code inlined from a reference) are still ordered by raw index among themselves.

Second leak of the same kind: ReflectedDefinitions resource name

With the patch applied, a project that contains [<ReflectedDefinition>] still differs after an earlier compile in the same process. The only difference is the resource name:

< resource ReflectedDefinitions-R-4921-1376909745
> resource ReflectedDefinitions-R-9265-1376909745

CreateILModule.fs builds that name with string (newUnique ()), a process-wide counter, so it depends on how much the process compiled before. Without the quotation, the patched build gives byte-identical output.

Known workarounds

Compile each project in a fresh process.

Related information

  • Linux, .NET 10 runtime, .NET SDK 11.0.100-rc.2
  • FCS 43.13.101-rc2.26509.113, and main at d0b662a911
Ngôn ngữ chính
F#
Star
4.3k
Fork
884
Merge trung bình
2 ngày 11 giờ
Pull request đã merge (30 ngày)
73

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của dotnet/fsharp

Tất cả issue của dotnet/fsharp

Issue tương tự

Thêm issue về Build System

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.