[yb-ctl] start_node after multiple stopped servers fails on waiting for all processes to be up

Jira Link: DB-1592 @rajukumaryb reported this earlier

Workflow seemed to be a manual attempt of restarting to a new version

• create a default rf3 local cluster with yb-ctl
• work on code and get new binary ready
• stop all tservers
• start them one by one

The first 2 servers would fail with errors such as:

Starting node tserver-1.
Waiting for cluster to be ready.
Viewing file /home/raju/ybtmp2/yugabyte-data/node-3/disk-1/tserver.err:
In YugaByte DB, setting LC_COLLATE to C and all other locale settings to en_US.UTF-8 by default. Locale support will be enhanced as part of addressing https://github.com/YugaByte/yugabyte-db/issues/15572020-01-30 21:30:02.295 UTC [12645] LOG:  YugaByte is ENABLED in PostgreSQL. Transactions are enabled.
2020-01-30 21:30:02.322 UTC [12645] LOG:  listening on IPv4 address "127.0.0.3", port 5433
2020-01-30 21:30:02.398 UTC [12645] LOG:  redirecting log output to logging collector process
2020-01-30 21:30:02.398 UTC [12645] HINT:  Future log output will appear in directory "/home/raju/ybtmp2/yugabyte-data/node-3/disk-1/yb-data/tserver/logs".
*** Aborted at 1580426294 (unix time) try "date -d @1580426294" if you are using GNU date ***
PC: @     0x7f0143f59bb7 epoll_wait
*** SIGTERM (@0x3e800000abe) received by PID 12315 (TID 0x7f01532dbbc0) from PID 2750; stack trace: ***
    @     0x7f014423b890 (unknown)
    @     0x7f0143f59bb7 epoll_wait
    @     0x561c1b78bc99  boost::asio::detail::epoll_reactor::run(long, boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) (thirdparty/installed/uninstrumented/include/boost/asio/detail/impl/epoll_reactor.ipp:471)
    @     0x561c1b78c4c1  boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) (thirdparty/installed/uninstrumented/include/boost/asio/detail/impl/scheduler.ipp:385)
    @     0x561c1b78c4c1  boost::asio::detail::scheduler::run(boost::system::error_code&) (thirdparty/installed/uninstrumented/include/boost/asio/detail/impl/scheduler.ipp:154)
    @     0x561c1b788d8f  boost::asio::io_context::run(boost::system::error_code&) (thirdparty/installed/uninstrumented/include/boost/asio/impl/io_context.ipp:70)
    @     0x561c1b788d8f  TabletServerMain (src/yb/tserver/tablet_server_main.cc:271)
    @     0x561c1b78921d  main (src/yb/tserver/tablet_server_main.cc:292)
    @     0x7f0143e59b97 __libc_start_main
    @     0x561c1b78620a _start
    @                0x0 (unknown)
Traceback (most recent call last):
  File "/media/raju/morespace/yugabyte-db/submodules/yugabyte-installation/bin/yb-ctl", line 1969, in <module>
    control.run()
  File "/media/raju/morespace/yugabyte-db/submodules/yugabyte-installation/bin/yb-ctl", line 1946, in run
    self.args.func()
  File "/media/raju/morespace/yugabyte-db/submodules/yugabyte-installation/bin/yb-ctl", line 1804, in start_node_cmd_impl
    self.wait_for_cluster_or_raise()
  File "/media/raju/morespace/yugabyte-db/submodules/yugabyte-installation/bin/yb-ctl", line 1551, in wait_for_cluster_or_raise
    raise RuntimeError("Timed out waiting for a YugaByte DB cluster!")
RuntimeError: Timed out waiting for a YugaByte DB cluster!
Viewing file /tmp/tmpuyFtM_:
2020-01-30 23:24:27,310 INFO: Server tserver-1 already running
2020-01-30 23:24:27,311 INFO: Waiting for master and tserver processes to come up.
2020-01-30 23:24:38,160 INFO: PIDs found: {'tserver': [3473, 6744, None], 'master': [12300, 12303, 12306]}
2020-01-30 23:24:38,161 ERROR: Failed waiting for master and tserver processes to come up.
^^^ Encountered errors ^^^

I think the wait for cluster to be ready likely waits for all servers that have local data to also be alive.

cc @jj-kim @mbautin