FedML-AI/FedML

Communication stuck randomly

Open

#161 opened on Dec 9, 2021

View on GitHub
 (2 comments) (0 reactions) (3 assignees)Python (743 forks)batch import
bughelp wanted

Repository metrics

Stars
 (3,930 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

I modified the way of communication in distribution FL, making the communication parallel. I mainly change the code of the section: In file FedAvgServerManager.py

for receiver_id in range(1, self.size):
       # self.send_message_sync_model_to_client(receiver_id,global_model_params,client_indexes[receiver_id-1])
        multi_processing=mp.Process(target=self.send_message_sync_model_to_client,args=(receiver_id,global_model_params,client_indexes[receiver_id-1]))
        multi_processing.start()

I'm using multiprocessing module to make it work in different process. But there are few strange things I have met. (I'm using GRPC backend, and set communication rounds as 5)

  1. when the server and simulated client on the same machine, it works as expected
  2. when I use another PC running simulated client, it sticks randomly. Sometimes it works, sometimes not.
  3. when I use raspberrypi, it sticks every time. But mostly stick in communication round 3 or 4.
  4. when I use jetson-nano, it sticks at the first round.
  5. when I use them both, it sticks at communication total round 3 or 4.( e.g. first round to simulated client, second round to pi, third round to jetson, and it get stuck!)

I have used wireshark to capture tcp communication package, and I found something interesting. From what I known, communication of GRPC is using persistent connection, which means it should open and keep a connection to client for every clients. But in dump file, every time before it get stuck, server send a tcp [RST] or [ACK, RST] flag to client ,cutting down the connection, but client get no error message. I can not figure out why. Interestingly, when the server communicate with simulated client on the same machine, server sends [RST] flag after every communication round, which I think is not properly. And all these tests are in the same local area network.

Here're some of my tcp dump file: tcpdump.zip

ubuntu-ubuntu is server and simulated client (which works fine but seems wrong) ubuntu-helix1 is server and another pc running simulated client ubuntu-raspberrypi is server and raspberrypi

the server and clients give no error message

Contributor guide