ipfs swarm connect should only use address provided in argument by default

We need a way to tell ipfs swarm connect that it should only use the address we give to it. Too many times I try to connect to a peer with ipfs swarm connect /definitely/a/valid/address and get dial backoffed because ipfs tried 100 other addresses that we're the one i gave it. This is also important in testing other things, I want to test out relay stuff, but often times i try to connect to a peer via relay and get a normal tcp direct connection.