Clickhouse get lost when multiple CH instances are running on the same host
#2,434 opened on May 28, 2018
Repository metrics
- Stars
- (47,419 stars)
- PR merge metrics
- (Avg merge 2d 2h) (1,000 merged PRs in 30d)
Description
On a machine with ip aliases, it is fairly easy to setup multiple clickhouse instances that will be binded to a specified ip.
with
inline bool isLocal(const Cluster::Address & address, const Poco::Net::SocketAddress & resolved_address, UInt16 clickhouse_port)
{
/// If there is replica, for which:
/// - its port is the same that the server is listening;
/// - its host is resolved to set of addresses, one of which is the same as one of addresses of network interfaces of the server machine*;
/// then we must go to this shard without any inter-process communication.
///
/// * - this criteria is somewhat approximate.
///
/// Also, replica is considered non-local, if it has default database set
/// (only reason is to avoid query rewrite).
return address.default_database.empty() && isLocalAddress(resolved_address, clickhouse_port);
}
Clickhouse will treat the two instances as in the same process, which is false, inducing errors in sharding.
I've seen the parameter treat_local_as_remote for the Cluster Class but that is not updateable and default to false
It would be nice to have a settable treat_local_as_remote like setting to enforce tcp instead of local process.
system.clusters table with 2 shards per host
SELECT *
FROM system.clusters
┌─cluster─────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name────┬─host_address─┬──port─┬─is_local─┬─user────┬─default_database─┐
│ clickhouse2 │ 1 │ 1 │ 1 │ 10.0.0.1 │ 10.0.0.1 │ 9000 │ 1 │ default │ │
│ clickhouse2 │ 2 │ 1 │ 1 │ 10.0.0.2 │ 10.0.0.2 │ 9000 │ 1 │ default │ │
│ clickhouse2 │ 3 │ 1 │ 1 │ 10.0.0.3 │ 10.0.0.3 │ 9000 │ 0 │ default │ │
│ clickhouse2 │ 4 │ 1 │ 1 │ 10.0.0.4 │ 10.0.0.4 │ 9000 │ 0 │ default │ │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 0 │ default │ │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │
└─────────────────────────────┴───────────┴──────────────┴─────────────┴──────────────┴──────────────┴───────┴──────────┴─────────┴──────────────────┘
Changing the instance port is a working turn around which I'd like to avoid.
system.clusters table with 2 shards per host with custom tcp ports
<yandex>
<tcp_port>9001</tcp_port>
</yandex>
SELECT *
FROM system.clusters
┌─cluster─────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name────┬─host_address─┬──port─┬─is_local─┬─user────┬─default_database─┐
│ clickhouse2 │ 1 │ 1 │ 1 │ 10.0.0.1 │ 10.0.0.1 │ 9001 │ 1 │ default │ │
│ clickhouse2 │ 2 │ 1 │ 1 │ 10.0.0.2 │ 10.0.0.2 │ 9002 │ 0 │ default │ │
│ clickhouse2 │ 3 │ 1 │ 1 │ 10.0.0.3 │ 10.0.0.3 │ 9001 │ 0 │ default │ │
│ clickhouse2 │ 4 │ 1 │ 1 │ 10.0.0.4 │ 10.0.0.4 │ 9002 │ 0 │ default │ │
│ test_shard_localhost │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9000 │ 0 │ default │ │
│ test_shard_localhost_secure │ 1 │ 1 │ 1 │ localhost │ 127.0.0.1 │ 9440 │ 0 │ default │ │
└─────────────────────────────┴───────────┴──────────────┴─────────────┴──────────────┴──────────────┴───────┴──────────┴─────────┴──────────────────┘
ch: 1.1.54378 Enterprise linux