Intermittent DirectoryNotEmptyException when running analyze@v2

Start with the reported CodeQL database finalize command, raw logs, and the Windows self-hosted runner details. Investigate the intermittent cleanup failure around the csharp database's cache\working directory and concurrent processes. Done means the finalize step no longer intermittently raises DirectoryNotEmptyException in the described workflow.

Getting the below error intermittently on Perform CodeQL Analysis step, specifically when running database finalize command. Ultimately the error is java.nio.file.DirectoryNotEmptyException, the directory in question being C:\actions-runner_work_temp\codeql_databases\csharp\db-csharp\default\cache\working

C:\actions-runner\_work\_tool\CodeQL\2.16.3\x64\codeql\codeql.exe database finalize --finalize-dataset --threads=4 --ram=12799 C:\actions-runner\_work\_temp\codeql_databases\csharp Running pre-finalize script C:\actions-runner\_work\_tool\CodeQL\2.16.3\x64\codeql\csharp\tools\pre-finalize.cmd in C:\actions-runner\_work\REPO\REPO. Running TRAP import for CodeQL database at C:\actions-runner\_work\_temp\codeql_databases\csharp... Falling back to 2.15.x RAM sizing, for CodeQL Action compatibility. Importing TRAP files Merging relations A fatal error occurred: Could not clear working directory: C:\actions-runner\_work\_temp\codeql_databases\csharp\db-csharp\default\cache\working after saving relation files. (eventual cause: MultiIOException "java.nio.file.DirectoryNotEmptyException: C:\actions-runner\_work\_temp\codeql_d...") A fatal error occurred: Dataset import for C:\actions-runner\_work\_temp\codeql_databases\csharp\db-csharp failed with code 2. Error: A fatal error occurred.

Looking at raw logs:

2024-02-28T19:28:53.9246062Z [2024-02-28 19:25:22] [ERROR] An exception occurred during TRAP import. The database may be partial.
2024-02-28T19:28:53.9246522Z java.util.concurrent.CompletionException: Could not clear working directory: C:\actions-runner_work_temp\codeql_databases\csharp\db-csharp\default\cache\working after saving relation files.
2024-02-28T19:28:53.9246862Z (eventual cause: MultiIOException "java.nio.file.DirectoryNotEmptyException: C:\actions-runner_work_temp\codeql_d...")
2024-02-28T19:28:53.9247154Z at java.base/java.util.concurrent.CompletableFuture.encodeRelay(Unknown Source)
2024-02-28T19:28:53.9247446Z at java.base/java.util.concurrent.CompletableFuture.completeRelay(Unknown Source)
2024-02-28T19:28:53.9247714Z at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(Unknown Source)
2024-02-28T19:28:53.9248000Z at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
2024-02-28T19:28:53.9248324Z at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
2024-02-28T19:28:53.9248615Z at com.semmle.util.concurrent.FutureUtils.lambda$futureAutoclose$14(FutureUtils.java:227)
2024-02-28T19:28:53.9248890Z at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
2024-02-28T19:28:53.9249156Z at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
2024-02-28T19:28:53.9249441Z at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
2024-02-28T19:28:53.9249709Z at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
2024-02-28T19:28:53.9249991Z at com.semmle.util.concurrent.Paralleliser.startMoreJobs(Paralleliser.java:134)
2024-02-28T19:28:53.9250284Z at com.semmle.util.concurrent.Paralleliser.lambda$startMoreJobs$2(Paralleliser.java:119)
2024-02-28T19:28:53.9250581Z at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
2024-02-28T19:28:53.9251475Z at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
2024-02-28T19:28:53.9251767Z at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
2024-02-28T19:28:53.9252018Z at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source)
2024-02-28T19:28:53.9252301Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2024-02-28T19:28:53.9252549Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2024-02-28T19:28:53.9252724Z at java.base/java.lang.Thread.run(Unknown Source)

Details:

• GHES v3.9.5
• Self-hosted Windows runner
• Running CodeQL workflow using advanced setup on 100+ repos in enterprise setting. This error is not restricted to any certain repo or repos.
• Began getting this error on v2.16.0, although we jumped there from v2.12.x so unsure about versions in between. Recently updated to v2.16.3 and still seeing it.

When I see this error, I re-trigger the workflow manually and it usually runs fine the second time, although it could get the same error again. Behavior is very intermittent.

My best guess at this point is that there are perhaps processes running concurrently that may lock some working files as read-only, or something similar.

Any assistance is greatly appreciated!