Spring Zuul - 使用 HTTPS 时的随机“转发错误”答案

【问题标题】：Spring Zuul - Random "Forwarding errors" when use HTTPSSpring Zuul - 使用 HTTPS 时的随机“转发错误”
【发布时间】：2017-11-03 11:19:45
【问题描述】：

我们在单个服务器上部署了一组微服务，这些微服务隐藏在 Apache 后面。

在将它们分散到不同的服务器之前，我们希望确保通信安全，因此开启了 HTTPS。

之后 spring zuul 出现问题。随机一些请求失败，因为 spring zuul 无法将它们转发到特定的 micro-service（即使服务在同一台服务器上）。我们收到以下异常：

com.netflix.zuul.exception.ZuulException: Forwarding error
        at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:188)
        at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:163)
        at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:111)
        at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
        at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
        at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
        at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
        at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
        at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
        at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
        at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
        at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:186)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: service-foo timed-out and no fallback available.
        at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:819)
        at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:804)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
        at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1472)
        at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1397)
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
        at rx.observers.Subscribers$5.onError(Subscribers.java:230)
        at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
        at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
        at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
        at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10211)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:142)
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
        at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1$1.run(AbstractCommand.java:1154)
        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:45)
        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
        at org.springframework.cloud.sleuth.instrument.hystrix.SleuthHystrixConcurrencyStrategy$HystrixTraceCallable.call(SleuthHystrixConcurrencyStrategy.java:188)
        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:61)
        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.tick(AbstractCommand.java:1159)
        at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
        at com.netflix.hystrix.AbstractCommand.handleTimeoutViaFallback(AbstractCommand.java:997)
        at com.netflix.hystrix.AbstractCommand.access$500(AbstractCommand.java:60)
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:610)
        at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)
        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
        ... 16 common frames omitted

奇怪的是，几乎所有请求都通过了——只有一些请求失败了。

已经尝试了一些解决方案，但没有帮助：

使用较低版本 (4.4.1) 的 httpclient

使用 Spring Cloud 的低版本 (Dalston.SR2)

增加 Apache 上 MaxRequestWorkers 的数量

延长连接时间（即使失败的 zuul 请求用时不到 3 秒）：

ribbon.ConnectTimeout = 3000

ribbon.ReadTimeout = 20000

所以在我打开“测试脚本”的同时还有更多：

一直在ping我的服务器

通过 HTTPS 向 foo-service 发送请求，避免使用 ZUUL

这两个操作都是与通过 ZUUL 发送请求并行进行的。对 foo-service 的 ping 和直接调用甚至在 Spring Zuul 失败的“时刻”。

当我们切换回 HTTP 时，它又可以工作了。我看到的唯一区别是，在 HTTPS 的情况下，整个流量都通过 Apache。

在异常之前我在日志中看到的最后一件事是：

2017-11-03 12:10:45.718 DEBUG [zuul,9aceeded808781ba,9aceeded808781ba,false] 873 --- [io-8009-exec-49] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-10859: Close connection
2017-11-03 12:10:45.718 DEBUG [zuul,9aceeded808781ba,9aceeded808781ba,false] 873 --- [io-8009-exec-49] o.a.http.impl.execchain.MainClientExec   : Connection discarded
2017-11-03 12:10:45.718 DEBUG [zuul,9aceeded808781ba,9aceeded808781ba,false] 873 --- [io-8009-exec-49] h.i.c.PoolingHttpClientConnectionManager : Connection released: [id: 10859][route: {s}->https://my.server.com:443][total kept alive: 1; route allocated: 4 of 50; total allocated: 5 of 200]
2017-11-03 12:10:45.724  WARN [zuul,9aceeded808781ba,9aceeded808781ba,false] 873 --- [io-8009-exec-49] o.s.c.n.z.filters.post.SendErrorFilter   : Error during filtering

但不知道我还能检查什么来找到问题的根源。

有什么想法吗？我们是否遗漏了一些应该进行的配置更改？

【问题讨论】：

Https 有点慢。您可能需要增加 Hystrix 超时时间

标签： spring-cloud spring-cloud-netflix

【解决方案1】：

我遇到了与您描述的完全相同的问题。根本问题是 HTTPS 协议较慢（尤其是第一次验证客户端和服务器之间的连接时）。默认情况下，Ribbon 对 Hystrix 有 2000 毫秒的超时，而您通常为所有 hystrix 命令覆盖此设置的配置似乎不适用于 Ribbon。

这是由于 AbstractRibbonCommand:93 中方法 getSetter 中的一些代码造成的。如果您检查该行，您将看到断路器的属性正在使用功能区代码中静态类中定义的默认值。

当您的服务连接的 CircuitBreaker 对象启动时，如果没有找到其他对象，它将使用此定义 - 类 HystrixCircuitBreaker:169。我没有深入探讨为什么这里没有选择默认的 Hystrix 配置。

我仍在尝试找出最好的解决方案，但 zuul 服务上的以下属性解决了这个问题：

hystrix.command.RibbonCommand.<my-service-name>.execution.isolation.thread.timeoutInMilliseconds=5000

理想情况下，我希望它使用默认的 hystrix 超时时间，因此我很可能会添加一个配置类，以根据配置的 Zuul 路由为我的所有服务实例化 RibbonCommand。如果我这样做，我会更新这个答案，以便将来参考。

我所指的类是 Hystrix Core 1.5.12 的一部分，它是 Spring Cloud Edgware 版本的一部分。

【讨论】：