If you're running EKS and trying to use the AWS EBS CSI Driver, you've probably hit a cryptic volume attachment error at some point. Here's a breakdown of the exact problem I faced, and how I resolved it using IAM Roles for Service Accounts (IRSA).

📊 The Problem

After installing the EBS CSI Driver via Helm, pods like Prometheus and others failed to bind their PersistentVolumeClaims (PVCs). On checking the controller logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100

You may see an error like this:

Could not attach volume "vol-..." to node "i-...": operation error EC2: AttachVolume, https response error StatusCode: 403, api error UnauthorizedOperation

🔍 How to Check Configuration

Use these commands to validate that everything is wired correctly:

✅ Controller logs (should NOT show AttachVolume 403 errors) 

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100

Bad output:

... AttachVolume ... UnauthorizedOperation ...

Good output:

I0330 00:04:47.133164       1 node.go:944] "CSINode Allocatable value is set" nodeName="i-035ea5232976e526c" count=26

✅ Validate CSINode registration 

kubectl get csinode

Good output:

NAME                         DRIVER              ...
i-035ea5232976e526c          ebs.csi.aws.com     ...

✅ Confirm node tolerations 

kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' | jq

Expected:

[
  {"operator": "Exists"},
  {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"},
  {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}
]

✅ Check IRSA annotations 

kubectl describe sa ebs-csi-controller-sa -n kube-system

Expected:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/cltest-ebs-csi-controller-irsa

✅ Inspect IAM Role and policies 

aws iam get-role --role-name cltest-ebs-csi-controller-irsa
aws iam list-attached-role-policies --role-name cltest-ebs-csi-controller-irsa

Expected:

"PolicyName": "AmazonEBSCSIDriverPolicy"

✅ Evaluate IAM policies

You can also simulate the action to confirm permissions:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam:::role/cltest-ebs-csi-controller-irsa \
  --action-names ec2:AttachVolume ec2:DetachVolume \
  --resource-arns arn:aws:ec2:::instance/* \
  --region

Expected:

"EvalDecision": "allowed"

🔐 IAM Inline Policy

Make sure this inline policy is also present:

resource "aws_iam_role_policy" "ebs_volume_permissions" {
  name = "allow-ebs-volume-ops"
  role = aws_iam_role.ebs_csi_controller_irsa.name

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid    = "EBSVolumeActions",
        Effect = "Allow",
        Action = [
          "ec2:AttachVolume",
          "ec2:DetachVolume"
        ],
        Resource = [
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:volume/*",
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:instance/*"
        ]
      }
    ]
  })
}

💡 Use aws iam get-role-policy --role-name --policy-name allow-ebs-volume-ops to verify.

🎉 Outcome

Once the proper inline policy was added, everything worked. Volumes were attached and Prometheus stopped showing PVC binding issues.

👋 Final Thoughts

The EBS CSI Driver depends on your IAM plumbing being just right. While AmazonEBSCSIDriverPolicy covers most use cases, sometimes additional permissions are needed.

Check your trust policies, policy attachments, and don't forget: inline IAM policies can help patch IAM gaps when needed.

Happy debugging!

🧪 Quick Summary of Checks

Check Command Expected Result
Controller logs kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100 No UnauthorizedOperation errors
CSINode registration kubectl get csinode Node lists ebs.csi.aws.com driver
Tolerations `kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' \ jq`
IRSA annotation kubectl describe sa ebs-csi-controller-sa -n kube-system Has eks.amazonaws.com/role-arn annotation
IAM role policies aws iam list-attached-role-policies Contains AmazonEBSCSIDriverPolicy
IAM simulation aws iam simulate-principal-policy Shows "EvalDecision": "allowed"