Too Many Open Files - Causes Container (and Docker host) to Stop Functioning
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- docker, go
- Domain
- backend, infrastructure
Research direction
No source file or test is named. Start by reproducing a repository check with the reported 2M files and examining open-file limits inside the container and on its host; done means isolating whether rest-server or the environment causes the lockout and identifying a concrete fix or configuration change.
Written by the indexing model from the issue text.
Description
Output of rest-server --version
rest-server version rest-server 0.14.0 compiled with go1.24.3 on linux/amd64
Problem description / Steps to reproduce
I am getting filesystem lockout errors when running a 'check' function from my restic frontend. The errors look something like this from the docker logs:
ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp3847862705250625670: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp8186166381684637943: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp1594954907149831599: too many open files in system ERROR: open /data/locks/f582ae9a31b06ac2721f860effb2e6da27eb9611424b806708d4f141a5d6799b.rest-server-temp491155930888957836: too many open files in system ERROR: lstat /data/locks/d32b409c3ea7bf22fd9c3c96749380d0d0a4fdbdaadfbbffcca5d74ff3aa7d89.rest-server-temp2286757414886870265: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system ERROR: open /data/locks: too many open files in system
I am currently running a prune on the repo now to see if I can clean out some of the unnecessary files to make the check easier, but we'll see. I will start experimenting with ulimit in a bit from inside the container, though this issue seems to affect the host as well.
Looks like the total amount of files in the repo currently exceeds 2M:
/PrimaryPool/RESTServer/data# find . -type f | wc -l
2090515
Expected behavior
Server should not reach a filesystem max limit with 40TB of data.
Actual behavior
Container ceases to function until restarted and throws the above errors.
Do you have any idea what may have caused this?
I finally finished the initial backup for a large amount of data (around 40TB), and went to run a check on the repo.
I think this may be along the lines of https://github.com/restic/rest-server/issues/215, where large datasets for these backups are going to run into some default filesystem issues. Though I'm not sure if deeper folder structure would help this particular problem.
Did rest-server help you today? Did it make you happy in any way?
Rest-server has been a godsend for acting as an endpoint on the large backup end of things. I look forward to figuring this out, since the backups already work and I can't imagine that this is going to be too complicated, though it will likely require adjustments on the host (and maybe the...super-host(?) docker is running on a VM, on a proxmox instance...)
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 179
- Avg merge
- 15d 9h
- Merged PRs (30d)
- 2
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from restic/rest-server
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
restic/rest-server#389 ·
-
state: need feedback
Difficulty 3/5 1-2 days Newbie friendliness 55/100
restic/rest-server#391 · 5 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
restic/rest-server#390 · 1 comment · 1 reaction ·
-
Exit after configurable idle period for servers with limited RAM or are off/suspended when unusedOpen
Difficulty 4/5 3-5 days Newbie friendliness 50/100
restic/rest-server#383 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 48/100
restic/rest-server#378 · 1 comment ·
All issues in restic/rest-server
Similar issues
-
automation models
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
txn2/mcp-data-platform#1984 ·
Maintainers usually reply within 1 day
-
agentic-workflows
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
kind/docs prio/P2
Difficulty 1/5 Under an hour Newbie friendliness 95/100
agent-substrate/substrate#1986 ·
Maintainers usually reply within 1 day