System hangs when a server is killed
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- distributed-systems
Research direction
Reproduce from parameter_server/example/linear using the provided ../../script/ps.sh command and hostfile, then compare behavior after killing a server versus a worker. Check the current project location noted in the repository description, and consider the issue resolved only when server failure behavior is documented or the system's expected continuation is verified.
Written by the indexing model from the issue text.
Description
The system hangs (node where scheduler is running stops displaying iteration outcomes on the terminal. Logs are also not generated) whenever a server process is killed (both on local machine or on remote machine). Is it the expected behaviour ? Shouldn't it still continue to run with gradients being updated on the backup/replicated server node as described in the paper ?
Here are the steps that I ran (from "parameter_server/example/linear" dir):
../../script/ps.sh start -nw 4 -ns 3 -hostfile hostfile ../../build/linear -app_file ctr/online_l1lr.conf -num_replicas 2 -report_interval 1
Then I killed a server process on one of the nodes. This stops the system. Killing a worker node, still continues the SGD and converges eventually.
Any help in this regard will be highly appreciated.
Thanks,
Danish
- Dominant language
- C++
- Stars
- 649
- Forks
- 233
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dmlc/parameter_server
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
dmlc/parameter_server#31 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 15/100
dmlc/parameter_server#29 · 2 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
dmlc/parameter_server#28 · 1 comment ·
-
running error.Open
Difficulty 5/5 Over a week Newbie friendliness 5/100
dmlc/parameter_server#27 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
dmlc/parameter_server#25 · 1 comment ·
All issues in dmlc/parameter_server
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
ChicoState/autovalidate#195 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
petercorke/robotics-toolbox-python#709 ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 2 days