apache/dubbo

dubbo注册服务和cloud服务如何在nacos上区分开?

Open

#7666 aperta il 30 apr 2021

Vedi su GitHub
 (2 commenti) (2 reazioni) (0 assegnatari)Java (26.453 fork)batch import
help wanted

Metriche repository

Star
 (41.524 star)
Metriche merge PR
 (Merge medio 7g 14h) (20 PR mergiate in 30 g)

Descrizione

Environment

  • Spring Cloud Alibaba 2.2.3.RELEASE
  • Dubbo version: 2.7.8
  • Java version: 1.8

当前有如下问题: 10.20.2.18:20880为阿里云k8s上建立的私网集群负载均衡服务

容器副本两台:172.17.0.190和172.17.0.191
容器(PodIP:172.17.0.190)无法访问 负载均衡:10.20.2.18:20880(轮询到自身172.17.0.190:20880时)
容器(PodIP:172.17.0.190)可以访问 负载均衡:10.20.2.18:20880(轮询到172.17.0.191:20880时)

容器(PodIP:172.17.0.191)无法访问 负载均衡:10.20.2.18:20880(轮询到自身172.17.0.191:20880时)
容器(PodIP:172.17.0.191)可以访问 负载均衡:10.20.2.18:20880(轮询到172.17.0.190:20880时)

具体现象:任何一台容器 消费dubbo时候,一次成功,一次失败。

2

出现这个问题的原因是,nacos配置直接用 负载均衡ip做了注册:

spring.cloud.nacos.discovery.server-addr=xx.xx.xx.xx:8848
spring.cloud.nacos.discovery.ip=10.20.2.18
spring.cloud.nacos.discovery.port=8080

dubbo注册服务时使用PodIP注册(172.17.0.190:20880和172.17.0.191:20880),消费的时候却使用了负载均衡ip,即10.20.2.18:20880

网上搜了一圈,好像K8S容器自身与 负载均衡解析到自身时是不通的。 1

为什么非要指定负载均衡ip?因为公司环境相对复杂,服务器资源在多个阿里云账号下,虽然拉了内网专线

但不属于同一个K8S内网,因此不同账号下的容器内部Pod网络不通(即172网段),需要做负载均衡转发。

因此想到这种解决方案,如果纯spring cloud的话,没有问题,恰恰dubbo和springcloud注册的是同一个服务名(application.name),且指定了同一个ip:10.20.2.18

理想状态,dubbo 服务纯粹仅供系统内部调用,因此注册使用PodIP即可。而对外暴露feign调用的话,使用负载均衡ip做解析

但是目前就是 dubbo虽然注册上去是172网段,但是消费的时候,全部使用10.20.2.18负载均衡网段。

想来想去,这套架构,最合适的方案就是:系统与系统间feign用 负载均衡ip(10.xx),系统内部dubbo用 容器内部网段(172.xx)消费

请问大牛,想要指定dubbo消费网段要怎么做? 或者说,dubbo的服务与 cloud的服务区分开,也能间接解决此问题!

Steps to reproduce this issue

  1. 将nacos注册ip和端口指定到k8s负载均衡ip上 3
  2. 任意一个consumer消费provider,副本数为2,一次可用,一次不可用
  3. 检测的方法还可以进容器,直接 curl http://负载均衡ip:8080/xxx ,一次通,一次不通

Pls. provide [GitHub address] to reproduce this issue.

Expected Result

172.17.0.191 消费 172.17.0.190:20880或172.17.0.191:20880

Actual Result

172.17.0.191 消费 10.20.2.18:20880 一次通,一次拒绝

2021-04-30 14:47:57.689 INFO  [dubbo-client-idleCheck-thread-1] ReconnectTimerTask -  [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [172.17.0.190:0 -> /10.20.2.19:20880]], dubbo version: 2.7.8, current host: 172.17.0.190
2021-04-30 14:47:57.694 ERROR [dubbo-client-idleCheck-thread-1] ReconnectTimerTask -  [DUBBO] Fail to connect to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [172.17.0.190:0 -> /10.20.2.19:20880]], dubbo version: 2.7.8, current host: 172.17.0.190
org.apache.dubbo.remoting.RemotingException: client(url: dubbo://10.20.2.19:20880/com.joyoung.wsc.product.api.service.ProductService?anyhost=true&application=wsc-gateway&check=false&codec=dubbo&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&heartbeat=60000&init=false&interface=com.joyoung.wsc.product.api.service.ProductService&loadbalance=roundrobin&metadata-type=remote&methods=saveSku,updateProduct,getById,addProduct,listProduct,listH5mProduct,deleteSku,deleteProduct,updateProductStatus,getProductById&pid=1&qos.enable=false&register.ip=172.17.0.190&release=2.7.8&remote.application=wsc-product&retries=0&revision=1.0-SNAPSHOT&service.filter=-exception&side=consumer&sticky=false&timeout=60000) failed to connect to server /10.20.2.19:20880, error message is:Connection refused: /10.20.2.19:20880
	at org.apache.dubbo.remoting.transport.netty4.NettyClient.doConnect(NettyClient.java:169)
	at org.apache.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:191)
	at org.apache.dubbo.remoting.transport.AbstractClient.reconnect(AbstractClient.java:247)
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.reconnect(HeaderExchangeClient.java:166)
	at org.apache.dubbo.remoting.exchange.support.header.ReconnectTimerTask.doTask(ReconnectTimerTask.java:49)
	at org.apache.dubbo.remoting.exchange.support.header.AbstractTimerTask.run(AbstractTimerTask.java:87)
	at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:648)
	at org.apache.dubbo.common.timer.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:727)
	at org.apache.dubbo.common.timer.HashedWheelTimer$Worker.run(HashedWheelTimer.java:449)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.20.2.19:20880
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Guida contributor