无法使用 Nginx-ingress 在 Kubernetes 中升级 websocket 连接

Question

我正在尝试通过 websockets 连接到我的 Mosquitto 代理，但我无法连接，因为连接没有升级。 mosquitto 代理公开端口 9001 以允许 websocket 连接，并且它在具有 nginx-ingress 控制器的 Kubernetes 集群后面运行。

$ kubectl get ingress mosquitto
NAME        HOSTS   ADDRESS   PORTS   AGE
mosquitto   *                 80      14m

.

$kubectl get service

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE

mosquitto                 ClusterIP   10.108.206.11    <none>        9001/TCP,1883/TCP   12m

Mosquitto.yaml：

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mosquitto
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      imagePullSecrets:
        - name: abb-login
      containers:
        - name: mosquitto
          image: ***/mosquitto:k8s2
          imagePullPolicy: Always
          ports:
            - containerPort: 9001
              protocol: TCP
            - containerPort: 1883
              protocol: TCP
          resources: {}

---
apiVersion: v1
kind: Service
metadata:
  name: mosquitto
spec:
  ports:
    - name: "9001"
      port: 9001
      targetPort: 9001
      protocol: TCP
    - name: "1883"
      port: 1883
      targetPort: 1883
      protocol: TCP
  selector:
    app: mosquitto

Mosquitto.conf：

allow_duplicate_messages false

connection_messages true

log_dest stdout stderr
log_timestamp true
log_type all

persistence false

listener 1883

allow_anonymous true

listener 9001
protocol websockets

allow_anonymous false

auth_plugin /usr/lib/mosquitto-auth-plugin/auth-plugin.so
auth_opt_backends http
auth_opt_http_ip 127.0.0.1
auth_opt_http_getuser_uri /api/mosquitto/users
auth_opt_http_superuser_uri /api/mosquitto/admins
auth_opt_http_aclcheck_uri /api/mosquitto/permissions
auth_opt_acl_cacheseconds 1
auth_opt_auth_cacheseconds 0

Ingress.yaml：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mosquitto
  annotations:
    nginx.org/websocket-services: mosquitto
spec:
  rules:
    - http:
        paths:
          - path: /mosquitto-ws
            backend:
              serviceName: mosquitto
              servicePort: 80

来自客户端的错误：

MqttException (0) - java.io.IOException: WebSocket Response header: Incorrect upgrade.
opc-ua-adapter_1  |     at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
opc-ua-adapter_1  |     at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:715)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
opc-ua-adapter_1  |     at java.base/java.lang.Thread.run(Thread.java:834)

Kubernetes ingress-nginx pod 日志：

192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "GET /mosquitto-ws HTTP/1.1" 308 171 "-" "-" 218 0.000 [default-mosquitto-9001] - - - - 5db23bb19698ac94612ff6ebac265bed
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "\x88\x84\xDDi+\x5C\xECY\x1Bl" 400 157 "-" "-" 0 0.000 [] - - - - 2b8f177f0f62389ba7d918f9c36ee72e
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "GET /mosquitto-ws HTTP/1.1" 308 171 "-" "-" 218 0.000 [default-mosquitto-9001] - - - - c99fe7606530ae938297e227e34084c0
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "\x88\x84dB5aUr\x05Q" 400 157 "-" "-" 0 0.000 [] - - - - 375ec1ac17cc3e0f7595cf8c1cc752c3

Answer 1

尝试在 mosquito 入口定义上增加 proxy-read-timeout 和 proxy-send-timeout。

查看 NGinx Ingress 文档： https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets