来自共享 Gitlab 运行器的 SSH 停止工作

Question

这确实以前工作过！

我在管道 SSH 中的部署步骤到 DO 框并从 docker 注册表中提取代码。如前所述，这在以前有效，这是我当时在.gitlab-ci.yml 中的deploy 步骤，从here 下的Using SSH 获得了很好的灵感：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -S)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

一旦我尝试运行deploy 步骤并失败。返回此错误：

...
 $ mkdir -p ~/.ssh
 $ echo "${DEPLOYMENT_SERVER_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
 $ chmod 600 ~/.ssh/id_rsa
 $ eval "$(ssh-agent -s)"
 Agent pid 22
 $ ssh-add ~/.ssh/id_rsa
 Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh gitlab@${DEPLOYMENT_SERVER_IP} "docker stop ${CI_PROJECT_NAME}; docker rm ${CI_PROJECT_NAME}; docker container prune -f; docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH}; docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}; docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest; docker run -d -p ${PORT}:${PORT} --restart always -m 2g --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
Running after_script
00:02
Uploading artifacts for failed job
00:01
 ERROR: Job failed: exit code 255

我最初设置的步骤

• 在 DO 框上运行 ssh-keygen -t rsa -b 2048（无密码）
• 在 DO 框上的 authorized_keys 中添加了 public 密钥
• 将 private 键复制到 CI 变量 DEPLOYMENT_SERVER_PRIVATE_KEY

我知道端口对 SSH 开放，因为我能够从本地计算机通过 SSH 连接到 gitlab 用户。我现在已将我的部署步骤 （基于来自 here、this article 和 this one 的 cmets） 更改为：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - echo ${CI_REGISTRY_USER}
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
      "docker stop ${CI_PROJECT_NAME};
      docker rm ${CI_PROJECT_NAME};
      docker container prune -f;
      docker rmi ${CI_REGISTRY}/${CI_PROJECT_PATH};
      docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY};
      docker pull ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;
      docker run -d -p ${HTTP_PORT}:${HTTP_PORT} --restart=always -m 800m --init --name ${CI_PROJECT_NAME} --net ${NETWORK_NAME} --ip ${NETWORK_IP} ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest;"

还是没用！ ssh 的详细日志吐了出来：

...
 $ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
 /usr/bin/ssh-agent
 $ eval $(ssh-agent -s)
 Agent pid 18
 $ echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
 Identity added: (stdin) ((stdin))
 $ mkdir -p ~/.ssh
 $ chmod 700 ~/.ssh
 $ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
 $ cat ~/.ssh/config
 Host *
    StrictHostKeyChecking no
 $ echo ${CI_REGISTRY_USER}
 gitlab-ci-token
 $ ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh -vvv gitlab@${DEPLOYMENT_SERVER_IP}
 OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
 debug1: Reading configuration data /root/.ssh/config
 debug1: /root/.ssh/config line 1: Applying options for *
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug2: resolve_canonicalize: hostname 134.xxx.xxx.xxx is address
 Pseudo-terminal will not be allocated because stdin is not a terminal.
 debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
 debug2: ssh_connect_direct
 debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
 debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
 ERROR: Job failed: exit code 255

我还添加了 -T 选项 suggested here 以禁用伪 tty 分配，但所做的只是从日志中删除伪行。

编辑

查看 DO 框 (/var/log/auth.log) 上的日志，我得到了错误：

Jun 22 15:53:37 exchange-apis sshd[16159]: Connection closed by 35.190.162.232 port 49750 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16160]: Connection closed by 35.190.162.232 port 49754 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16162]: Connection closed by 35.190.162.232 port 49752 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16163]: Unable to negotiate with 35.190.162.232 port 49756: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
Jun 22 15:53:38 exchange-apis sshd[16161]: Unable to negotiate with 35.190.162.232 port 49758: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]

谷歌搜索此错误，常见原因似乎是由于 OpenSSH 放弃了对 DSA 密钥的支持。但是，当我生成一个 RSA 密钥对时，不确定为什么这会影响我。无论如何，运行dpkg --list | grep openssh 会吐出：

ii  openssh-client                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                    1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines

&sshd -v吐槽：

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

尽管如此，答案仍然有效； here & here 所以我的deploy 舞台现在是：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\tHostkeyAlgorithms +ssh-dss\n\tPubkeyAcceptedKeyTypes +ssh-dss\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - ssh -oHostKeyAlgorithms=+ssh-dss gitlab@${DEPLOYMENT_SERVER_IP} ls

仍然没有看到，我在跑步者的输出和 DO 框上的日志中得到了同样的错误。有什么想法吗？

Answer 1

理想情况下，如果您可以登录 DO 框，您将停止 ssh 服务，并启动 /usr/bin/sshd -de，以便在 SSH 守护进程端建立调试会话，并将日志写入 stderr（而不是系统消息）

但如果你不能，至少尝试生成一个没有密码的 rsa 密钥，用于测试。这意味着您不需要 ssh-agent。
并尝试ssh -Tv gitlab@${DEPLOYMENT_SERVER_IP} ls 看看那里产生了什么日志。

试试classic PEM format

ssh-keygen -t rsa -P "" -m PEM

在对管道进行更多编辑后，我注意到实际上是这一行导致了问题：ssh-keyscan -H ${DEPLOYMENT_SERVER_IP} >> ~/.ssh/known_hosts

如果它导致~/.ssh/known_hosts 格式错误，尤其是${DEPLOYMENT_SERVER_IP} 设置不正确时，可能会出现这种情况。
尝试将echo "DEPLOYMENT_SERVER_IP='${DEPLOYMENT_SERVER_IP}'" 和cat ~/.ssh/known_hosts 命令添加到before_script 部分，以了解更多信息。