[Bug] Unclosed files and pipes in tool scripts and Python modules

Open
  • Is something wrong?
#8,122 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, python
Domain
backend, testing

Research direction

Start with one independent leak from the issue, such as the unclosed pipe in scripts/v.rast.stats/v.rast.stats.py or the /dev/null handle in a named tool; the issue also lists python/grass/pygrass/modules/interface/env.py and scripts/g.extension/g.extension.py. Read the relevant code, reproduce its warning with PYTHONDEVMODE=1, then run that tool's testsuite under the same setting. Done means the selected leak is closed and its reproducer emits no ResourceWarning.

Written by the indexing model from the issue text.

Description

Python

Describe the bug

I made an agent (Claude Code) and asked it to find the places where we open a file or a socket and never close it. It built GRASS, then ran the pytest tests and the full gunittest testsuite (with the NC sample dataset, as in the Ubuntu CI job) with Python's development mode (python -X dev, or PYTHONDEVMODE=1 so that tool scripts started as subprocesses are covered too). Dev mode enables ResourceWarning, which reports files, pipes and database connections that are never closed. The agent then traced each warning back to the code, did a static pass over the rest of the code, and wrote the reproducers below, which it ran on main at commit 79cd2b35f714274b60df4ef91e2f385c8e3d1197 to produce the output shown.

This issue covers the small, independent leaks in tool scripts and Python modules that should be easy to fix. The larger grass.temporal SQLite leak is tracked separately.

To reproduce

Enable dev mode for the shell you run the commands in:

export PYTHONDEVMODE=1        # also reaches Python scripts launched by GRASS
export PYTHONTRACEMALLOC=20   # optional: adds "Object allocated at" tracebacks

The warnings are printed on stderr when the object is garbage collected, so the line number in a warning is where the GC happened, not where the file was opened. Use PYTHONTRACEMALLOC to see where it was opened.

v.rast.stats (the stdout of gs.pipe_command() is never closed, in get_nr_of_categories() and perform_stats()):

grass --tmp-project XY --exec bash -c '
g.region n=10 s=0 e=10 w=0 res=1
r.mapcalc expression="r = int(row())"
v.in.region output=v
v.db.addtable map=v
v.rast.stats map=v raster=r column_prefix=r method=average
'

v.rast.stats:186: ResourceWarning: unclosed file <_io.TextIOWrapper name=4 encoding='UTF-8'>
v.rast.stats:212: ResourceWarning: unclosed file <_io.TextIOWrapper name=5 encoding='UTF-8'>
<sys>:0: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='w' ...>

v.db.droptable (nuldev = open(os.devnull, "w") is never closed; the same pattern is in v.db.reconnect.all, v.db.univar, v.in.e00, v.rast.stats and v.report):

grass --tmp-project XY --exec bash -c '
g.region n=10 s=0 e=10 w=0 res=1
v.in.region output=v
v.db.addtable map=v
v.db.droptable -f map=v
'

m.proj (the input file and the output file are opened and never closed):

printf '1 1\n' > pts.txt
grass --tmp-project XY --exec m.proj input=pts.txt \
    proj_in="+proj=longlat +datum=WGS84" proj_out="+proj=utm +zone=17 +datum=WGS84"

grass.pygrass.modules.interface.env.G_debug() (opens GRASS_DEBUG_FILE on every debug message and never closes it, so it leaks one file per call):

cat > gd.py <<'EOF'
import gc
from grass.pygrass.modules.interface.env import G_debug
for i in range(3):
    G_debug(1, "hello")
gc.collect()
EOF
GRASS_DEBUG_FILE=$PWD/dbg.txt grass --tmp-project XY --exec bash -c 'g.gisenv set="DEBUG=3"; python3 gd.py'

g.extension (is_binary_string(open(filename, "rb").read(1024)) in a loop over every file of an add-on, and the /dev/null handle outdev). I saw 34 warnings from the first and 9 from the second when running its testsuite:

cd scripts/g.extension/testsuite
grass --tmp-mapset ~/grassdata/nc_spm_full_v2beta1 --exec python3 test_addons_modules.py

A test has the same problem: raster/r.in.gdal/testsuite/test_r_in_gdal.py:310 opens a file without closing it.

Expected behavior

Files and pipes are closed when the code is done with them, so a run with PYTHONDEVMODE=1 prints no ResourceWarning from these tools.

Screenshots

Not applicable.

System description

  • Operating System: Ubuntu 24.04.5 LTS, x86_64
  • GRASS version: 8.6.0dev (main at commit 79cd2b35f714274b60df4ef91e2f385c8e3d1197, "locale: Update translation files (135) (#8111)"), built with CMake without GUI and PDAL
  • g.version -rge: proj=9.4.0, gdal=3.8.4, geos=3.12.1, sqlite=3.45.1
  • Python 3.13.17
  • Test data: nc_spm_full_v2beta1

Additional context

Found by reading the code (no warning reproduced)
  • python/grass/utils/download.py (extract_tar) and scripts/g.extension/g.extension.py:1834: tarfile.open(name) is never closed. I did not get a warning from CPython when I tried extract_tar() on a .tar.gz, but the missing close() is clear from the code. A with block would fix both.
  • Unclosed urlopen() responses (sockets): scripts/g.manual/g.manual.py:72 and several in scripts/g.extension/g.extension.py (around lines 551, 571, 949, 983, 1891, 1903). No test downloads from the network, so these were not exercised.
  • Callers of gs.pipe_command() that never close the pipe, like v.rast.stats does: scripts/v.report/v.report.py, scripts/d.rast.edit/d.rast.edit.py, scripts/i.image.mosaic/i.image.mosaic.py. Using the returned Popen as a context manager closes the pipe.
  • Files that are only closed on the normal path, so a gs.fatal() leaks them: python/grass/temporal/univar_statistics.py (two places) and python/grass/temporal/register.py:167.
  • scripts/db.test/db.test.py: inf = open(infile) is never closed.

Ruff's SIM115 (open-file-with-context-handler) flags most of the open() cases. Many of these files have it disabled in pyproject.toml; enabling it again for each file after it is fixed would keep it from regressing.

These can be fixed in small PRs grouped by tool, each checked with its testsuite under PYTHONDEVMODE=1.

I did not look at C code beyond a rough check for fopen()/popen() without a matching close; the ones I read were either closed elsewhere or in tools that exit right after. A valgrind --track-fds=yes pass over the C tools would check that properly.

Dominant language
C
Stars
1.2k
Forks
447
Avg merge
1d 19h
Merged PRs (30d)
62

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OSGeo/grass

All issues in OSGeo/grass

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.