swiftlang/swift-package-manager

[SR-8671] ARG_MAX limit exceeded when compiles huge project

Open

#5341 opened on Aug 31, 2018

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Swift (10,145 stars) (1,462 forks)batch import
buggood first issue

Description
Previous ID SR-8671
Radar rdar://problem/43940293
Original Reporter Pogonets (JIRA User)
Type Bug

Attachment: Download
Votes 3
Component/s Package Manager
Labels Bug, StarterBug
Assignee None
Priority Medium

md5: 7fd3bd1478f560273a7837164b9d76a9

Issue Description:

macOS have a relative small ARG_MAX limit (256kb)

When you compile huge project with SwiftPM they pass all sources one by one as arguments. Each file is passed as absolute path. And with count of files ~1000 (800 in our case) command line args become too long to even start command.

Error message is meaningless by the way

unable to spawn process (File exists)

This issue even easier to reach when linking multiple swift modules to single dynamic library (all object files from all modules passed to compiler as single line).

This is pretty common issue for build tools and usually solved with response files. Response files supported by clang, ld.gold and many other tools.

Contributor guide

Tech stack
swift
Domain
build systemperformance
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
SwiftSwiftPM internalsARG MAX limitclang response files
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Examine the SwiftPM source code, particularly the `SwiftCompilerTool` or `BuildOperation` files, to understand how command lines are constructed for clang. Implement a check for command line length against ARG MAX, and if exceeded, write a temporary response file containing the input files and pass `@file` to clang. Ensure the solution also applies to the linking step. Test with a large project to verify the fix.