Setting up AWS IAM for a data team, role by role

Most teams don’t decide who can access what in their AWS account. It just happens: someone needs a bucket, someone shares an admin key, and a year later nobody can say why the analytics intern’s credentials can delete production data. The right time to answer the question is before anything breaks — and the answer is IAM.

Take a typical data team of four. A software engineer maintains the application that exports raw files to S3. A senior data engineer owns the pipeline that transforms those files and loads them into the warehouse. A junior data engineer is learning the stack in a dev environment. An analyst queries the warehouse every morning. Four people, four jobs, one AWS account — and four very different sets of permissions.

Start from the data flow, not from the console

IAM configuration only makes sense once the pipeline is explicit. In our example it looks like this:

1. The application uploads raw CSV files to s3://shop-data/raw/.
2. A Glue crawler registers their schemas in the Data Catalog.
3. A Glue ETL job converts them to Parquet in s3://shop-data/curated/.
4. A Redshift COPY loads the curated files for analysis.
5. The analyst queries the analytics schema in Redshift.

Each role touches a different segment of that flow. Write the mapping down before opening the console: if a permission isn’t justified by the flow, it doesn’t belong in the policy. Everything that follows is just an implementation of that table.

The software engineer: write-only, on purpose

The application’s job ends the moment the file lands in raw/. So the role that the application assumes gets exactly two things: the ability to put objects under the raw/ prefix, and the ability to write its own logs to CloudWatch.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteRawOnly",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::shop-data",
        "arn:aws:s3:::shop-data/raw/*"
      ]
    },
    {
      "Sid": "AppLogs",
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Notice what’s missing: s3:GetObject. The application can deposit files but can never read anything back — a pattern sometimes called a blind write. Three reasons to want it:

• The raw prefix is the sensitive one. Data lands there before any cleaning or masking — names, emails, order history. If the application’s credentials leak, a write-only key can push data in but exfiltrate nothing.
• The application never needs to read. Designing the role around what the code actually does means there is no unused read permission waiting to be misused.
• Blast radius. A bad deployment can only make a mess inside raw/. The curated data and the warehouse are unreachable from this role.

The senior data engineer: broad, but bounded to the project

The pipeline owner needs to read and write across the whole project bucket, manage Glue, manage Redshift, and read logs. Two details matter more than the breadth.

First, the trust policy. This role is also the execution identity of the Glue job, so it trusts two principals: your AWS account (a human assumes it) and glue.amazonaws.com (the service assumes it at runtime). That’s how the job reads S3 and loads Redshift without a single password in the code.

Second, the double S3 ARN. Every S3 statement needs both the bucket ARN and the /* object ARN:

"Resource": [
  "arn:aws:s3:::shop-data",
  "arn:aws:s3:::shop-data/*"
]

The bucket ARN covers bucket-level actions like ListBucket; the /* form covers object-level actions like GetObject. Provide only the first and your Glue job will list the bucket happily, then fail on every actual read — with an error message that never mentions the ARN. It is probably the single most common S3/IAM mistake, and one of the most time-consuming to diagnose.

The junior data engineer: see everything, change nothing

A junior engineer learning the stack needs visibility, not write access: read the dev prefix in S3, inspect Glue job definitions and catalog tables, query a dev schema in Redshift, follow the pipeline in CloudWatch logs.

The policy has the same shape as the senior one — same statement structure, same four service blocks — with the scope tightened at every line: s3:GetObject instead of s3:*, a dev/ prefix instead of the whole bucket, glue:GetJob but deliberately not glue:StartJobRun, and Redshift access constrained by condition to a specific database user:

"Condition": {
  "StringEquals": {
    "redshift:DbName": "dev",
    "redshift:DbUser": "junior_de"
  }
}

That symmetry is the point: you are not learning a different syntax per role, you are adjusting boundaries within one format. And read-only for juniors isn’t distrust — it’s what lets them explore production logs freely without anyone worrying.

The analyst: one door, one room

The analyst needs to query curated data. Their policy is the shortest: GetObject on the curated/ prefix, plus redshift:GetClusterCredentials scoped to the analyst database user.

One caveat that catches many teams: IAM controls whether the analyst can connect; database grants control what they see once inside. A GRANT SELECT ON SCHEMA analytics TO analyst_user inside Redshift is a separate, mandatory step. IAM is the front door; the grants are the rooms. Configure both or the raw schema isn’t actually protected.

Should you memorise the JSON?

No. Policy JSON is repetitive, and a language model will produce a decent first draft from a plain-language description of what a role needs. What you cannot delegate is the review: the model doesn’t know your ARNs, your workgroup IDs, or whether a write action just slipped into a read-only role. Knowing what each field does — trust vs. permission policy, bucket vs. object ARN, condition blocks — is precisely what lets you spot when a generated policy is wrong.

Four roles, four services, one pipeline: each identity touches exactly what its work requires and nothing more. That’s the whole discipline — and it’s much cheaper to set up on day one than to retrofit after an incident.