【问题标题】:Cloud SQL connection for Kubernetes using proxy使用代理的 Kubernetes 的 Cloud SQL 连接
【发布时间】:2019-07-09 19:57:54
【问题描述】:

我目前正在 Kubernetes 中运行 Spring Boot Pod。云 SQL 代理的 pod 中有一个边车。

下面是我的spring Boot application.properties配置:

server.port=8081
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.continue-on-error=true
spring.datasource.url=jdbc:mysql://localhost:3306/<database_name>
spring.datasource.username=<user_name>
spring.datasource.password=<password>

下面是我的 pom.xml 提取的插件和依赖项:

<properties>
    <java.version>1.8</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.jayway.jsonpath</groupId>
        <artifactId>json-path</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-hateoas</artifactId>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>ca.performance.common</groupId>
        <artifactId>common-http</artifactId>
        <version>1.1.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.cloud.sql</groupId>
        <artifactId>mysql-socket-factory</artifactId>
        <version>1.0.10</version>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-jdbc</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-gcp-starter-sql-mysql</artifactId>
        <version>1.1.0.RELEASE</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

这是我的 deployment.yaml 文件:


apiVersion: v1
kind: Service
metadata:
  name: app-dummy-name
spec:
  selector:
    app: app-dummy-name
  ports:
  - port: 81
    name: http-app-dummy-name
    targetPort: http-api
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-dummy-name
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-dummy-name
  template:
    metadata:
      labels:
        app: app-dummy-name
    spec:
      containers:
      - name: app-dummy-name
        image: <image url>
        ports:
        - containerPort: 8081
          name: http-api
        env:
        - name: DB_HOST
          value: 127.0.0.1:3306
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: password
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy",
                    "-instances=<INSTANCE_CONNECTION_NAME>=:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
        securityContext:
          runAsUser: 2  # non-root user
          allowPrivilegeEscalation: false
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials

我按照this link 的说明进行操作,因此我创建了机密和服务帐户。但是,当我在创建机密后在 Kubernetes 中部署以前的 yaml 文件时,我经常遇到连接拒绝错误:

org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta-data; 
nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; 
nested exception is com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure. 
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

我什至使用代理和相同的 application.properties 配置在本地测试了 Spring boot 应用程序,它工作正常。

【问题讨论】:

  • 只需确保 是上述示例代码中的占位符,而不是实际部署中编写的内容。
  • 它是一个占位符,而不是实际名称
  • 你检查过sidecar容器的日志吗?
  • 是否应该将您的 deployment.yaml 中的 DB_HOST 替换为:NAME:url VALUE:"jdbc:mysql://localhost:3306/&lt;database_name&gt;",就像在您的 application.properties configuration 中一样?与DB_USER 相同 -> 用户名和DB_PASSWORD -> 密码
  • 也尝试在乞求中明确输入用户名和密码——不要从cloudsql-db-credentials中提取它们

标签: mysql spring spring-boot kubernetes cloud-sql-proxy


【解决方案1】:

我正在添加对我有用的部署 yaml,请检查添加以下内容是否有帮助:

在卷下:

  volumes:
  - name: cloudsql
    emptyDir:

在连接中:--dir=/cloudsql

  - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy:1.11
    command: ["/cloud_sql_proxy", "--dir=/cloudsql",
        "-instances=<INSTANCE_CONNECTION_NAME=tcp:5432>",
        "-credential_file=/secrets/cloudsql/credentials.json"]

还要确保您启用了Cloud SQL Administration API

这是我的完整部署 yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-dummy-name
spec:
  replicas: 1
  revisionHistoryLimit: 1
  strategy:
      type: RollingUpdate
  template:
    metadata:
      labels:
        app: app-dummy-name
        tier: backend
    spec:
      securityContext:
        runAsUser: 0
        runAsNonRoot: false
      containers:
      - name: app-dummy-name
        image: <image url>
        ports:
        - containerPort: 80
        env:
        - name: DB_HOST
          value: localhost
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: cloudsql-db-credentials
              key: password
      # proxy_container
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
          "-instances=my-project-id:us-central1:postgres-instance-name=tcp:5432",
          "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: cloudsql
            mountPath: /cloudsql
      # volumes
      volumes:
      - name: cloudsql-instance-credentials
        secret:
          secretName: cloudsql-instance-credentials
      - name: cloudsql
        emptyDir:

这是我的 pre-delpoy 脚本:

#!/bin/bash
# https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine
# 1. Go to the Cloud SQL Service accounts page of the Google Cloud Platform Console.
# GO TO THE SERVICE ACCOUNTS PAGE
# 2.  If needed, select the project that contains your Cloud SQL instance.
# 3. Click Create service account.
# 4. In the Create service account dialog, provide a descriptive name for the service account.
# 5. For Role, select Cloud SQL > Cloud SQL Client.
# Alternatively, you can use the primitive Editor role by selecting Project > Editor, but the Editor role includes permissions across Google Cloud Platform.
#
# 6. If you do not see these roles, your Google Cloud Platform user might not have the resourcemanager.projects.setIamPolicy permission. You can check your permissions by going to the IAM page in the Google Cloud Platform Console and searching for your user id.
# Change the Service account ID to a unique value that you will recognize so you can easily find this service account later if needed.
# 7. Click Furnish a new private key.
# 8. The default key type is JSON, which is the correct value to use.
# 9. Click Create.
# 10. enable Cloud SQL Administration API [here](https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview)
# make sure to choose your project


echo "create cloudsql secret"
kubectl create secret generic cloudsql-instance-credentials \
   --from-file=credentials.json=postgres-sql-credential.json

echo "create cloudsql user and password"
kubectl create secret generic cloudsql-db-credentials \
   --from-literal=username=postgres --from-literal=password=123456789

postgres-sql-credential.json 文件:

{
  "type": "service_account",
  "project_id": "my-project",
  "private_key_id": "1234567890",
  "private_key": "-----BEGIN PRIVATE KEY-----\n123445556\n123445\n-----END PRIVATE KEY-----\n",
  "client_email": "postgres-sql@my-project.iam.gserviceaccount.com",
  "client_id": "1234567890",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/postgres-sq%my-project.iam.gserviceaccount.com"
}

【讨论】:

    【解决方案2】:

    在这次讨论中,我开发了一个 yaml,如下所示。就是通过pgbouncer连接cloud-sql。

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: pgproxy
      namespace: var_namespace_k8s
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: pgproxy
      revisionHistoryLimit: 1
      strategy:
          type: RollingUpdate
      template:
        metadata:
          labels:
            app: pgproxy
            tier: backend
        spec:
          securityContext:
            runAsUser: 0
            runAsNonRoot: false
          containers:
          - name: pgproxy
            image: var_image_pgproxy_k8s
            ports:
            - containerPort: 6432
            volumeMounts:
              - name: pgbouncer-init-volume
                mountPath: /home/pgbouncer/pgbouncer.ini
                subPath: pgbouncer.ini
              - name: pgbouncer-userlist-volume
                mountPath: /home/pgbouncer/userlist.txt
                subPath: userlist.txt
          # proxy_container
          - name: cloudsql-proxy
            image: var_image_cloudsqlproxy_k8s
            command: ["/cloud_sql_proxy", "--dir=/cloudsql",
              "-instances=gcp:us-east:databse=tcp:0.0.0.0:3306",
              "-credential_file=/secrets/cloudsql/cloudsql-service-acnt-key.json"]
            volumeMounts:
              - name: sqlauthproxy-svcaccount-volume
                mountPath: /secrets/cloudsql/cloudsql-service-acnt-key.json
                subPath: cloudsql-service-acnt-key.json
                readOnly: true
              - name: cloudsql
                mountPath: /cloudsql
          # volumes
          volumes:
          - name: pgbouncer-init-volume
            secret:
              secretName: pgbouncer-init
              items:
              - key: pgbouncer.ini
                path: pgbouncer.ini
          - name: pgbouncer-userlist-volume
            secret:
              secretName: pgbouncer-userlist
              items:
              - key: userlist.txt
                path: userlist.txt
          - name: sqlauthproxy-svcaccount-volume
            secret:
              secretName: sqlauthproxy-svcaccount
              items:
              - key: cloudsql-service-acnt-key.json
                path: cloudsql-service-acnt-key.json
          - name: cloudsql
            emptyDir:
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: pgproxy
      namespace: var_namespace_k8s
      annotations:
        cloud.google.com/load-balancer-type: "Internal"
    spec:
      type: LoadBalancer
      selector:
        app: pgproxy
      ports:
        - port: 6432
          targetPort: 6432
    

    输出是:

    NAME                      READY   STATUS             RESTARTS   AGE
    pgproxy-6b84bdb84-7g2l4   1/2     CrashLoopBackOff   7          14m 
    

    kubectl logs pod/pgproxy-6b84bdb84-7g2l4 -c cloudsql-proxy -n db-auth-proxy-gcp的输出

    2021/11/09 13:05:59 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
    2021/11/09 13:05:59 using credential file for authentication; email=svc-keng-edap-poc-user@gcp-keng01.iam.gserviceaccount.com
    2021/11/09 13:05:59 Listening on 0.0.0.0:3306 for gcp:us-east:databse
    2021/11/09 13:05:59 Ready for new connections
    2021/11/09 13:05:59 Generated RSA key in 412.982017ms
    

    尽管没有错误,但状态为“CrashLoopBackOff”。我做错了什么?

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2021-02-12
      • 1970-01-01
      • 2017-04-09
      • 2018-08-04
      • 1970-01-01
      • 1970-01-01
      • 2019-10-07
      • 2022-11-10
      相关资源
      最近更新 更多