EIP publicip association not correctly updated on fresh instance
#1321 opened on Sep 6, 2016
Description
I've been directed over here from the eureka folks, as they believe this should just 'work'. Have the following issue running off spring-cloud-netflix:1.1.4.RELEASE. The issue I opened over there is: https://github.com/Netflix/eureka/issues/840
There seems to be a problem with public EIP address association not being correctly updated when a new AWS server starts and has a new Eureka server starting with it. When the server starts up, it correctly registers itself:
2016-09-06 15:55:29.040 WARN 3399 --- [Thread-10] com.netflix.eureka.aws.EIPManager : The selected EIP 54.67.102.122 is associated with another instance i-0666b391 according to AWS, hence skipping this
2016-09-06 15:55:29.628 INFO 3399 --- [Thread-10] com.netflix.eureka.aws.EIPManager :
Associated i-25f11391 running in zone: us-west-1c to elastic IP: X.X.X.X
But, every minute after that we get the following log entry:
2016-09-06 16:24:55.568 INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl : Got 1 instances from neighboring DS node
2016-09-06 16:24:55.568 INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl : Renew threshold is: 1
2016-09-06 16:24:55.568 INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl : Priming AWS connections for all replicas..
2016-09-06 16:24:55.568 INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl : No peers needed to prime.
2016-09-06 16:24:55.568 INFO 3399 --- [Eureka-EIPBinder] c.n.e.r.PeerAwareInstanceRegistryImpl : Changing status to UP
2016-09-06 16:24:55.713 WARN 3399 --- [Eureka-EIPBinder] com.netflix.eureka.aws.EIPManager : The selected EIP X.X.X.X is associated with another instance i-0666b391 according to AWS, hence skipping this
2016-09-06 16:24:55.804 INFO 3399 --- [Eureka-EIPBinder] com.netflix.eureka.aws.EIPManager : My instance i-25f11391 seems to be already associated with the EIP X.X.X.X
Debugging this, the call to isEIPBound() is always failing, and this is because the following is always null:
String myPublicIP = ((AmazonInfo) myInfo.getDataCenterInfo()).get(MetaDataKey.publicIpv4);
It looks like there is stale datacenterinfo and it never gets refreshed (from what I can tell) and there there are no settings available to have it refreshed automatically.
The odd side affect of this, and we noticed, is that the registry continually gets wiped, and reset causing obvious potential issues down stream for our clients.
I have been trying to find where this datacenter info might be refreshed, but am unable to find anything that might actually do that.
The deployed app only has a single main class in it:
@SpringBootApplication
@EnableEurekaServer
@EnableAutoConfiguration
public class EurekaServer {
@Value("${server.port}")
private Integer nonSecurePort;
@Autowired
private InetUtils utils;
public static void main(String[] args) {
new SpringApplicationBuilder(EurekaServer.class).web(true).run(args);
}
@Bean
@Profile("aws")
public EurekaInstanceConfigBean awsEurekaConfig() {
EurekaInstanceConfigBean b = new EurekaInstanceConfigBean(utils);
b.setNonSecurePort(nonSecurePort);
b.setSecurePortEnabled(false);
AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
b.setDataCenterInfo(info);
return b;
}
}