Hazelcast 集群在 Eureka 上不可用答案

【问题标题】：Hazelcast cluster not available on EurekaHazelcast 集群在 Eureka 上不可用
【发布时间】：2018-12-11 09:00:53
【问题描述】：

我在为现有的 Eureka 客户端配置简单的 Hazelcast 集群时遇到问题。我的应用程序是启用了 Eureka 发现的简单 Java SpringBoot 应用程序，它适用于 Eureka，但 Hazelcast 集群不是。

这是我的配置：

hazelcast.xml

<hazelcast
    xsi:schemaLocation="http://www.hazelcast.com/schema/config http://www.hazelcast.com/schema/config/hazelcast-config-3.9.xsd"
    xmlns="http://www.hazelcast.com/schema/config"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<properties>
    <property name="hazelcast.discovery.enabled">true</property>
    <property name="hazelcast.jmx">true</property>
    <property name="hazelcast.http.healthcheck.enabled">true</property>
    <property name="hazelcast.rest.enabled">true</property>
    <property name="hazelcast.logging.type">slf4j</property>
    <property name="hazelcast.diagnostics.enabled">true</property>
    <property name="hazelcast.diagnostics.metric.level">info</property>
    <property name="hazelcast.name">hazelcast-cluster</property>
</properties>

<group>
    <name>xxxx</name>
    <password>xx</password>
</group>

<instance-name>hazelcast-cluster</instance-name>

<network>
    <join>
        <multicast enabled="false"/>
        <tcp-ip enabled="false"/>
        <aws enabled="false"/>
        <discovery-strategies>
            <discovery-strategy class="com.hazelcast.eureka.one.EurekaOneDiscoveryStrategy" enabled="true">
                <properties>
                    <property name="self-registration">true</property>
                    <property name="namespace">hazelcast</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>
    </join>
</network>

maps....

</hazelcast>

Spring 启动应用程序：

@SpringBootApplication
@EnableAutoConfiguration ( exclude = { WebMvcAutoConfiguration.class } )
@EnableDiscoveryClient
@EnableEurekaClient
public class HazelcastClusterApplication {
    public static void main( String[] args ) {
        SpringApplication.run( HazelcastClusterApplication.class, args );
    }  
}

配置类：

@Configuration
public class HazelcastInstanceConfiguration {

    @Bean
    public HazelcastInstance hazelcastInstance(Config config, EurekaClient eurekaClient) { 
        EurekaOneDiscoveryStrategyFactory.setEurekaClient( eurekaClient );
        return Hazelcast.newHazelcastInstance(config);
    }

    @Bean
    public Config config() {    
        Config config = new ClasspathXmlConfig( "hazelcast.xml" );
        return config;
    }
}

bootstrap.yml

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8011/eureka/
    register-with-eureka: true

spring:
  mvc:
    favicon:
      enabled: false
  application:
    name: cache-server
  cloud:
    config:
      discovery:
        enabled: true
        serviceId: config-server
      failFast: false
      retry:
        initialInterval: 10000
        maxInterval: 60000
        maxAttempts: 12
        multiplier: 10.1
  config:
    name: application.*, ${spring.application.name}*.*
server:
  port: 7990

Maven 依赖：

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast</artifactId>
        <version>3.11</version>
    </dependency>

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast-spring</artifactId>
        <version>3.11</version>
    </dependency>

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast-eureka-one</artifactId>
        <version>1.0.2</version>
    </dependency>

运行后，'cache-server' 应用程序在 Eureka 上可用，但我猜 'hazelcast-cluster' 也应该可用，但事实并非如此。还是我的假设是错误的？

启动日志：

INFO 17628 --- [           main] c.h.s.d.integration.DiscoveryService     : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
INFO 17628 --- [           main] c.h.s.d.integration.DiscoveryService     : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
INFO 17628 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_CACHE-SERVER/pc_data:cache-server:7990: registering service...
INFO 17628 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_CACHE-SERVER/pc_data:cache-server:7990 - registration status: 204
INFO 17628 --- [           main] c.h.s.d.integration.DiscoveryService     : [10.230.115.128]:5701 [xxxx] [3.11] Waiting for registration with Eureka...
DEBUG 17628 --- [           main] c.h.i.cluster.impl.DiscoveryJoiner       : [10.230.115.128]:5701 [xxxx] [3.11] This node will assume master role since no possible member where connected to.
DEBUG 17628 --- [           main] c.h.internal.cluster.ClusterService      : [10.230.115.128]:5701 [xxxx] [3.11] Setting master address to [10.230.115.128]:5701
DEBUG 17628 --- [           main] c.h.i.cluster.impl.MembershipManager     : [10.230.115.128]:5701 [xxxx] [3.11] Local member list join version is set to 1
DEBUG 17628 --- [           main] c.h.i.cluster.impl.DiscoveryJoiner       : [10.230.115.128]:5701 [xxxx] [3.11] PostJoin master: [10.230.115.128]:5701, isMaster: true
INFO 17628 --- [           main] c.h.internal.cluster.ClusterService      : [10.230.115.128]:5701 [xxxx] [3.11] 

Members {size:1, ver:1} [
    Member [10.230.115.128]:5701 - f6ed031e-5740-4daa-b583-5b91d98816c2 this
]

然后我的缓存服务器应用程序在 Eureka 上可用，但不是我猜应该的 hazelcast-cluster。

Hazelcast 客户端服务日志（已删减）：

2018-12-11 09:48:01.673  INFO [security-service,,,] 21496 --- [           main] com.hazelcast.client.HazelcastClient     : hz.client_0 [xxxx] [3.11] A non-empty group password is configured for the Hazelcast client. Starting with Hazelcast version 3.11, clients with the same group name, but with different group passwords (that do not use authentication) will be accepted to a cluster. The group password configuration will be removed completely in a future release.
2018-12-11 09:48:01.700  INFO [security-service,,,] 21496 --- [           main] com.hazelcast.core.LifecycleService      : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is STARTING
2018-12-11 09:48:03.233  INFO [security-service,,,] 21496 --- [           main] o.s.c.n.eureka.InstanceInfoFactory       : Setting initial instance status as: STARTING
2018-12-11 09:48:03.249  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Initializing Eureka in region us-east-1
2018-12-11 09:48:03.264  INFO [security-service,,,] 21496 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using JSON encoding codec LegacyJacksonJson
2018-12-11 09:48:03.266  INFO [security-service,,,] 21496 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using JSON decoding codec LegacyJacksonJson
2018-12-11 09:48:03.267  INFO [security-service,,,] 21496 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using XML encoding codec XStreamXml
2018-12-11 09:48:03.267  INFO [security-service,,,] 21496 --- [           main] c.n.d.provider.DiscoveryJerseyProvider   : Using XML decoding codec XStreamXml
2018-12-11 09:48:03.417  INFO [security-service,,,] 21496 --- [           main] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2018-12-11 09:48:03.421  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Disable delta property : false
2018-12-11 09:48:03.421  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Single vip registry refresh property : null
2018-12-11 09:48:03.422  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Force full registry fetch : false
2018-12-11 09:48:03.422  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Application is null : false
2018-12-11 09:48:03.423  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Registered Applications size is zero : true
2018-12-11 09:48:03.424  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Application version is -1: true
2018-12-11 09:48:03.424  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Getting all instance registry info from the eureka server
2018-12-11 09:48:03.426  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : The response status is 200
2018-12-11 09:48:03.426  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Starting heartbeat executor: renew interval is: 5
2018-12-11 09:48:03.426  INFO [security-service,,,] 21496 --- [           main] c.n.discovery.InstanceInfoReplicator     : InstanceInfoReplicator onDemand update allowed rate per min is 4
2018-12-11 09:48:03.442  INFO [security-service,,,] 21496 --- [           main] com.netflix.discovery.DiscoveryClient    : Discovery Client initialized at timestamp 1544518083442 with initial instances count: 3
2018-12-11 09:48:03.488  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:08.468  INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035 - Re-registering apps/SECURITY-SERVICE
2018-12-11 09:48:08.468  INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035: registering service...
2018-12-11 09:48:08.499  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:08.530  INFO [security-service,,,] 21496 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_SECURITY-SERVICE/GPLPLW2002.gft.com:security-service:8035 - registration status: 204
2018-12-11 09:48:13.500  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:18.503  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:23.512  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:28.513  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:33.515  INFO [security-service,,,] 21496 --- [           main] c.h.s.d.integration.DiscoveryService     : hz.client_0 [xxxx] [3.11] Waiting for registration with Eureka...
2018-12-11 09:48:33.557  INFO [security-service,,,] 21496 --- [           main] c.h.client.spi.ClientInvocationService   : hz.client_0 [xxxx] [3.11] Running with 2 response threads
2018-12-11 09:48:33.742  INFO [security-service,,,] 21496 --- [           main] com.hazelcast.core.LifecycleService      : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is STARTED
2018-12-11 09:48:33.772 DEBUG [security-service,,,] 21496 --- [           main] c.h.i.networking.nio.NioNetworking       : hz.client_0 [xxxx] [3.11] TcpIpConnectionManager configured with Non Blocking IO-threading model: 1 input threads and 1 output threads
2018-12-11 09:48:33.774 DEBUG [security-service,,,] 21496 --- [           main] c.h.i.networking.nio.NioNetworking       : hz.client_0 [xxxx] [3.11] IO threads selector mode is SELECT
2018-12-11 09:48:33.818  WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager          : hz.client_0 [xxxx] [3.11] Unable to get alive cluster connection, try in 3000 ms later, attempt 1 of 2.
2018-12-11 09:48:36.822  WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager          : hz.client_0 [xxxx] [3.11] Unable to get alive cluster connection, attempt 2 of 2.
2018-12-11 09:48:36.824  WARN [security-service,,,] 21496 --- [ient_0.cluster-] c.h.c.c.ClientConnectionManager          : hz.client_0 [xxxx] [3.11] Could not connect to cluster, shutting down the client. Unable to connect to any address! The following addresses were tried: []
2018-12-11 09:48:36.828  INFO [security-service,,,] 21496 --- [clientShutdown-] com.hazelcast.core.LifecycleService      : hz.client_0 [xxxx] [3.11] HazelcastClient 3.11 (20181023 - 1500bbb) is SHUTTING_DOWN
2018-12-11 09:48:36.839  INFO [security-service,,,] 21496 --- [clientShutdown-] com.netflix.discovery.DiscoveryClient    : Shutting down DiscoveryClient ...
2018-12-11 09:48:36.843  INFO [security-service,,,] 21496 --- [clientShutdown-] com.netflix.discovery.DiscoveryClient    : Unregistering ...
2018-12-11 09:48:36.844  WARN [security-service,,,] 21496 --- [           main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: o.....org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheManager' defined in class path resource 
.......
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [xxxx/hazelcast/HazelcastClientConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
    ... 150 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 163 common frames omitted
Caused by: java.lang.IllegalStateException: Unable to connect to any address! The following addresses were tried: []
    at com.hazelcast.client.connection.nio.ClusterConnector.connectToClusterInternal(ClusterConnector.java:206)
    at com.hazelcast.client.connection.nio.ClusterConnector.access$400(ClusterConnector.java:56)
    at com.hazelcast.client.connection.nio.ClusterConnector$2.call(ClusterConnector.java:215)
    at com.hazelcast.client.connection.nio.ClusterConnector$2.call(ClusterConnector.java:211)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

我的配置有什么问题，是服务器问题还是客户端问题？

客户端配置：

@Bean
@Autowired
public HazelcastInstance hazelcastInstance( EurekaClient eurekaClient, GroupConfig groupConfig ) {

    EurekaOneDiscoveryStrategyFactory.setEurekaClient( eurekaClient );

    ClientConfig config = new ClientConfig();
    config.setGroupConfig( groupConfig );
    config.setProperty( "hazelcast.discovery.enabled", "true" );

    DiscoveryStrategyConfig discoveryStrategyConfig = new DiscoveryStrategyConfig( new EurekaOneDiscoveryStrategyFactory() );
    discoveryStrategyConfig.addProperty( "use-classpath-eureka-client-props", "false" );
    discoveryStrategyConfig.addProperty( "self-registration", "false" );

    DiscoveryConfig discoveryConfig = new DiscoveryConfig();
    discoveryConfig.addDiscoveryStrategyConfig( discoveryStrategyConfig );
    config.getNetworkConfig().setDiscoveryConfig( discoveryConfig );

    return HazelcastClient.newHazelcastClient( config );
}

【问题讨论】：

标签： spring-boot hazelcast netflix-eureka

【解决方案1】：

问题是您使用相同的 EurekaClient 进行应用发现（端口 7990）和 Hazelcast 发现（端口 5701）。更多信息，请阅读相关GH issue。

你可以通过两种方式解决它：

使用属性use-metadata-for-host-and-port，如here 所述
为您的应用和 Hazelcast 使用单独的 Eureka 客户端和端口

编辑：use-metadata-for-host-and-port 将很快在hazelcast-eureka:1.0.3 中发布。我也准备好了Hazelcast Eureka Code Sample。

【讨论】：

我认为这有助于我理解问题，我在 Eureka 中成功注册了 Hazelcast 集群。现在的问题是我无法从 Eureka Client 应用程序（使用现有、自动装配的 EurekaClient）访问此集群。请你帮助我好吗？我使用客户端项目的 HazelcastInstance 配置更新了帖子描述，groupConfig 已正确注入。 Hazelcast 客户端尝试连接到本地端口而不是 eureka：hz.client_0 [xxxx] [3.11] 尝试以所有者成员身份连接到 [127.0.0.1]:5702
我认为在客户端你还需要指定namespace（设置为hazelcast，就像你在Hazelcast成员中所做的那样）。
我已经编辑了我的答案，请看一下。另外，我添加了代码示例，它将阐明如何使用 Hazelcast Eureka 插件。
@RafałLeszko 谢谢，添加use-metadata-for-host-and-port 帮助我使用 PCF 注册服务在 PCF 上注册 hazelcast 成员

【解决方案2】：

您是否看过Eureka plugin repo 和code sample with Eureka 中的示例配置？我看到你的配置有一些不同。如果仍有问题，请检查并告诉我。

【讨论】：

是的，我试图遵循它，但是缺少重用现有 EurekaClient 对象的示例。公平地说，我不知道在进行 hazelcast 服务器注册时我应该期待什么，它是否应该像普通的 Spring Boot 应用程序一样在 Eureka 上可用？第二个链接的解决方案有点过于复杂，因为它使用分区组，我只需要一个集群实例。