Cross account access to S3 using IRSA in EKS with Terraform as IaaC

We have many options to get cross-account access to resources, but when talking about the Kubernetes cluster, things can get a little bit tricky! So, in this blog, I'll share a solution to do it in the safest way using the principle of least privilege.

A typical scenario is to have two accounts, Account A, with an EKS cluster and Account B with an S3 bucket (example_bucket) that needs to be accessed by a pod from account A. We have many options for this:

We can create a bucket policy with the worker role name of the Kubernetes cluster on it.
We can create an IAM role in Account B, grant the role permissions to perform required S3 operations, assume the role with a trust policy, etc.…

Those are some solutions to access the bucket; however, in the way of getting the access, we grant lots of privileges that we don’t need because we just need to give access to a pod, not to the whole cluster. That’s why AWS provides us with IAM Roles for Service Accounts (IRSA)

IRSA allows us to associate an IAM Role with a Kubernetes service account, and this service account can then grant permissions to any pod that uses it.

Using IRSA has the benefit of using the least privileged recommendation and credential isolations, meaning that the container within the pod can only retrieve credentials for the IAM role associated with the service account to which the pod belongs.

For getting IRSA to work, we need these things:

IAM OIDC provider
The IAM role
And finally, associate the IAM Role with the Kubernetes service account.

So, we're going to grant access to a pod from an EKS cluster in Account A to get an object from an S3 bucket in Account B using IRSA with cross-account access.

Pretty interesting, Let’s get our hands dirty!

The diagram below shows what we‘re going to do:

We‘re going to use Terraform as IaaC for this example and Kubernetes YAML for deployment, and Kubernetes service account.

In Account A

We will deploy a pod with a Service Account that will allow that pod to get objects from the bucket located in Account B

In Account B

We will create an IAM Role in Account B with an IAM OIDC provider ARN as the source of trust to allow Account A to assume the role. This resource is the result of using the Kubernetes Cluster’s OIDC URL issuer from Account A; then, we will attach the policies required for accessing the S3 bucket.

Let’s start with Account B first.

We assume we have a file test.txt within a bucket named test-oidc-cross-account-bucket in account B.

Step 1: Create the IAM OIDC Provider

The first thing we need is the cluster OIDC issuer URL from Account A to generate the IAM OIDC Provider in Account B; you can grab it from the EKS terraform module output, cluster_oidc_issuer_url

From the terraform console in account A:

➜ terraform console
Acquiring state lock. This may take a few moments...
> module.eks.cluster_oidc_issuer_url
"https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE1111222EAEEE"

Then in Account B, create the IAM OIDC provider:


resource "aws_iam_openid_connect_provider" "oidc_issuer" {
  url             = "https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE1111222EAEEE"
  thumbprint_list = ["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"]
  client_id_list  = ["sts.amazonaws.com"]

You can also get the thumbprint from the EKS terraform module.

Step 2: Create the IAM role with the trust relationship and attach the policy to get access to S3.

First, we create the assume-role policy that establishes the trust relationship.

Principals will use the ARN of the aws_iam_openid_connect_provider we created in the first step as the identifier.
In condition, will evaluate StringEquals with the OIDC URL (without https://) of the cluster of Account A, and in the values, we will use the Kubernetes service account name in the form system:serviceaccount:<namespace>:<name_of_service_account>

data "aws_iam_policy_document" "irsa" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [
        aws_iam_openid_connect_provider.oidc_issuer.arn #arn OIDC 
      ]
    }
    condition {
      test     = "StringEquals"
      variable = "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE1111222EAEEE:sub" 
      values = [
        "system:serviceaccount:default:s3-sa"
      ]
    }
  }
}

The s3 policy:

data "aws_iam_policy_document" "s3" {
  statement {
    sid    = ""
    effect = "Allow"
    actions = [
      "s3:ListBucket"
    ]
    resources = [
      aws_s3_bucket.test_bucket.arn,
      "${aws_s3_bucket.test_bucket.arn}/*",
    ]
  }
  statement {
    sid    = ""
    effect = "Allow"
    actions = [
      "s3:GetObject",
    ]
    resources = [
      aws_s3_bucket.test_bucket.arn,
      "${aws_s3_bucket.test_bucket.arn}/*",
    ]
  }
}

And then the creation of the resource:

Create the s3 IAM policy
Create the role with the assume_role_policy
Attach the s3 IAM policy to the role

resource "aws_iam_policy" "s3" {
  name   = "s3_read"
  path   = "/"
  policy = data.aws_iam_policy_document.s3.json
}
 
resource "aws_iam_role" "s3" {
  name               = "s3"
  path               = "/"
  assume_role_policy = data.aws_iam_policy_document.irsa.json
}
 
resource "aws_iam_role_policy_attachment" "s3" {
  role       = aws_iam_role.s3.name
  policy_arn = aws_iam_policy.s3.arn
}

Once you create those resources, get the ARN of the role, we will need it in the final step

Great! We finished all the work in Account B

Let’s go to the final part and the best one…

Time to work in Account A

Step 3: Test the permissions!

The next and final task to do is create a service account and a test deployment that uses that service account to grant permissions to get an object of the bucket.

To get that service account authenticated against Account B, you must add an annotation with the ARN of the role created in the previous step in this form:

eks.amazonaws.com/role-arn: "arn:aws:iam::<Account_B_ID>:role/<role_name>"

Let’s create a service account:

Remember: The service account name must be the same you used in the role created in the previous step.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: "s3-sa" #Name we used during the role creation
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::5554444333:role/s3" #role ARN
automountServiceAccountToken: true

And a test deployment that uses that service account: I used the official AWS CLI image just to get the commands from the start.

➜ cat test_deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-deployment
  name: test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-deployment
  template:
    metadata:
      labels:
        app: test-deployment
    spec:
      serviceAccountName: "s3-sa" #Name of the SA we ‘re using
      automountServiceAccountToken: true
      containers:
      - image: amazon/aws-cli
        name: aws
        command: ["sleep","10000"]

Here is where the magic happens.

Apply those YAML.

➜ kubectl apply -f s3_sa.yaml 
serviceaccount/s3-sa created
➜ kubectl apply -f test_deployment.yaml 
deployment.apps/test-deployment created

Check if it’s running, and then let’s get into the pod.

➜ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
test-deployment-7867876f54-xm5pm   1/1     Running   0          100s
 
➜ kubectl exec -it test-deployment-7867876f54-xm5pm -- bash
bash-4.2#

Check the credentials, and you can see that the pod assumed the role we created in account B; success!

bash-4.2# aws sts get-caller-identity
{
    "UserId": "AROA2PKFSO4DYYGMNG4IS:botocore-session-1626387503",
    "Account": "5554444333",
    "Arn": "arn:aws:sts::5554444333:assumed-role/s3/botocore-session-1626387503"
}

Let’s try to get something from the bucket!

bash-4.2# aws s3 ls s3://test-oidc-cross-account-bucket
2021-07-15 22:22:28         63 test.txt
 
bash-4.2# aws s3 cp s3://test-oidc-cross-account-bucket/test.txt .
download: s3://test-oidc-cross-account-bucket/test.txt to ./test.txt
 
bash-4.2# cat test.txt 
this is a test to prove you can get this file from this bucket

M-A-G-I-C!

IRSA is the way to get cross-account access using the least privilege concept. Remember, the critical part is the trust relationship in the role. You can use it with any policy with the permissions you want!

In conclusion, IRSA is a good practice to adopt for managing pods permission because of three main motives: it uses the least privilege principle, it provides credential isolation for each pod and allows audibility through CloudTrail

Want more? Check out our page and social networks! You will find more valuable tips for Kubernetes and AWS cloud!