Access Denied Issues with Cross Account S3 Buckets

Cross account S3 buckets are pretty important for large organizations, however if managed improperly can cause permissions nightmares. AWS now recommends that S3 buckets be created with ACLs (access control lists) disabled, however older S3 buckets may still have this setting enabled. with ACLs, each object in the S3 bucket can have a different owner (AWS account). If another AWS account is writing to your bucket, you may not have permissions to modify or delete the objects, even if you own the bucket. Newer S3 buckets disable this capability by default which prevents this problem from occuring.

If you’re running into this problem, the following code snippet can help you. We will

1. Pull data for every object in the S3 bucket
2. Figure out which AWS accounts own objects in the bucket (represented by the canonical id of the AWS account)
3. Use those accounts to give control of the objects in S3 back to the owner of the S3 bucket.

The first section inspects every object in the S3 bucket to figure out who the different owners are. The IDs printed are the canonical IDs of the AWS accounts that own the objects.

import boto3
from collections import Counter
bucket = "openproblems-bio"
session = boto3.Session()
client = session.client('s3')
owner_ids = Counter()

data = []
for response in client.get_paginator('list_objects').paginate(Bucket=bucket):
    for obj in response['Contents']:
        owner_id = obj['Owner']['ID']
        key = obj['Key']
        data.append((owner_id, key))

owner_ids.update(x[0] for x in data)
print(owner_ids)

This second snippet will give control for those objects back to the bucket owner.

for owner_id, key in data:
    if owner_id == bad_owner:
        print(f'fixing {key}')
        client.put_object_acl(ACL='bucket-owner-full-control', Bucket=bucket, Key=key)

Afterwards - you can disable ACLs on the bucket to prevent this from happening again

This must be run in the AWS account owning the bucket.

$ aws s3api put-bucket-ownership-controls --bucket BUCKET --ownership-controls="Rules=[{ObjectOwnership=BucketOwnerEnforced}]"

SHARE:

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Request a demo today to learn more.