无法从 Google Compute Engine 元数据服务检索令牌。状态：404

Question

我正在尝试将 Cloud SQL 代理设置为在我的 GKE 集群中作为 sidecar 运行。配置是通过 Terraform 完成的。我已经设置了工作负载身份、所需的服务帐户等。从 GKE 集群 (kubectl run -it --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test) 中启动 ./cloud_sql_proxy 时，我得到以下输出：

root@workload-identity-test:/# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance=tcp:5432
2020/11/24 17:18:39 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 17:18:40 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/db-proxy@project-id.iam.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7fc5575545f8>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/db-proxy@project-id.iam.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7f06f72f45c0>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 errors parsing config:
        Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/project-id/instances/europe-west4~db-instance?alt=json&prettyPrint=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.admin" not defined

以下是我迄今为止完成的故障排除：

root@workload-identity-test:/# gcloud auth list
                   Credentialed Accounts
ACTIVE  ACCOUNT
*       db-proxy@project-id.iam.gserviceaccount.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

λ gcloud container clusters describe mycluster --format="value(workloadIdentityConfig.workloadPool)"
project-id.svc.id.goog

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster --format="value(config.workloadMetadataConfig.mode)"
GKE_METADATA

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster--format="value(config.oauthScopes)"
https://www.googleapis.com/auth/monitoring;https://www.googleapis.com/auth/devstorage.read_only;https://www.googleapis.com/auth/logging.write;https://www.googleapis.com/auth/cloud-platform;https://www.googleapis.com/auth/userinfo.email;https://www.googleapis.com/auth/compute;https://www.googleapis.com/auth/sqlservice.admin

λ kubectl describe serviceaccount --namespace k8s-namespace ksa-name
Name:                ksa-name
Namespace:           k8s-namespace
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: db-proxy@project-id.iam.gserviceaccount.com
Image pull secrets:  <none>
Mountable secrets:   ksa-name-token-87n4t
Tokens:              ksa-name-token-87n4t
Events:              <none>

λ gcloud iam service-accounts get-iam-policy db-proxy@project-id.iam.gserviceaccount.com
bindings:
- members:
  - serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]
  role: roles/iam.workloadIdentityUser
etag: BwW02zludbY=
version: 1

λ kubectl get networkpolicy --namespace k8s-namespace
No resources found in k8s-namespace namespace.

λ gcloud projects get-iam-policy project-id
bindings:
- members:
  - serviceAccount:db-proxy@project-id.iam.gserviceaccount.com
  role: roles/cloudsql.editor

预期结果（我在另一个集群上运行它，后来更改了配置，找不到我的错误在哪里）：

root@workload-identity-test:~# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance-2=tcp:5432
2020/11/24 18:09:54 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 18:09:56 Listening on 127.0.0.1:5432 for project-id:europe-west4:db-instance-2
2020/11/24 18:09:56 Ready for new connections

我做错了什么？如何进一步排除故障或调试？

Answer 1

这可能是由于创建 Kubernetes 集群时未启用服务帐户，或者未正确配置。尝试检查服务帐户是否被禁用and Enable if it is. 您也可以尝试创建一个新的服务帐户并更改service account in the pods. 或者最后，在运行时尝试provide the credentials to the gcloud 命令。

Answer 2

您能否确认“db-proxy@project-id.iam.gserviceaccount.com”是正确的帐户？我可能看错了，但似乎尝试刷新该帐户的身份验证令牌时出错，错误是该帐户不存在。

Answer 3

我今天遇到了类似的错误，发现是因为 GSA 与 GKE 集群在不同的项目中。似乎 iam.workloadIdentityUser 绑定需要在同一项目中的帐户之间。

这样就成功了：

gcloud iam service-accounts create custom-metrics-adapter \
    --project ${PLATFORM_PROJECT_ID}

gcloud iam service-accounts add-iam-policy-binding \
    "${GSA_NAME}@${PLATFORM_PROJECT_ID}.iam.gserviceaccount.com" \
    --member "serviceAccount:${PLATFORM_PROJECT_ID}.svc.id.goog[${KSA_NAMESPACE}/${KSA_NAME}]" \
    --role "roles/iam.workloadIdentityUser" \
    --project ${PLATFORM_PROJECT_ID}

与

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${KSA_NAME}
  namespace: ${KSA_NAMESPACE}
  annotations:
    iam.gke.io/gcp-service-account: ${GSA_NAME}${PLATFORM_PROJECT_ID}.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: ${KSA_NAMESPACE}
spec:
  template:
    spec:
      serviceAccountName: ${KSA_NAME}
# Deployment spec truncated for clarity

不确定这是否对您有帮助，但也许它会帮助通过搜索错误字符串找到此问题的其他人：

检索失败 http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/${GSA_NAME}@${DIFFERENT_PROJECT_ID}.iam.gserviceaccount.com/token 来自 Google Compute Engine 元数据服务。状态：404 响应：\nb'无法生成访问令牌； IAM 返回 404 不是 找到：未找到请求的实体。

Answer 4

我能够通过创建一个不同名称的服务帐户来解决此问题。只是名字变了，没有别的。如果我删除db-proxy@project-id.iam.gserviceaccount.com 然后再次使用该名称，问题仍然存在。我找不到该帐户的任何其他参考。在我于 20 年 11 月 30 日发表评论后，该问题不再遇到。