FedML-AI/FedML

Communication stuck randomly

Open

#161 创建于 2021年12月9日

在 GitHub 查看
 (2 评论) (0 反应) (3 负责人)Python (743 fork)batch import
bughelp wanted

仓库指标

Star
 (3,930 star)
PR 合并指标
 (30 天内没有已合并 PR)

描述

I modified the way of communication in distribution FL, making the communication parallel. I mainly change the code of the section: In file FedAvgServerManager.py

for receiver_id in range(1, self.size):
       # self.send_message_sync_model_to_client(receiver_id,global_model_params,client_indexes[receiver_id-1])
        multi_processing=mp.Process(target=self.send_message_sync_model_to_client,args=(receiver_id,global_model_params,client_indexes[receiver_id-1]))
        multi_processing.start()

I'm using multiprocessing module to make it work in different process. But there are few strange things I have met. (I'm using GRPC backend, and set communication rounds as 5)

  1. when the server and simulated client on the same machine, it works as expected
  2. when I use another PC running simulated client, it sticks randomly. Sometimes it works, sometimes not.
  3. when I use raspberrypi, it sticks every time. But mostly stick in communication round 3 or 4.
  4. when I use jetson-nano, it sticks at the first round.
  5. when I use them both, it sticks at communication total round 3 or 4.( e.g. first round to simulated client, second round to pi, third round to jetson, and it get stuck!)

I have used wireshark to capture tcp communication package, and I found something interesting. From what I known, communication of GRPC is using persistent connection, which means it should open and keep a connection to client for every clients. But in dump file, every time before it get stuck, server send a tcp [RST] or [ACK, RST] flag to client ,cutting down the connection, but client get no error message. I can not figure out why. Interestingly, when the server communicate with simulated client on the same machine, server sends [RST] flag after every communication round, which I think is not properly. And all these tests are in the same local area network.

Here're some of my tcp dump file: tcpdump.zip

ubuntu-ubuntu is server and simulated client (which works fine but seems wrong) ubuntu-helix1 is server and another pc running simulated client ubuntu-raspberrypi is server and raspberrypi

the server and clients give no error message

贡献者指南