FSx for Lustre and S3 are two complementary methods of storing data in Amazon Web Services (AWS). FSx offers an extremely performant, reliable, true filesystem, but it’s expensive and not accessible via the web or other APIs. S3 offers cheaper object storage that’s accessible from any device connected to the internet, but methods to access objects in S3 as files are limited.

AWS does have a great feature where you can set up a data repository association between a FSx for Lustre filesystem and n S3 bucket. This lets you sync the contents of the filesystem and the bucket, either in one direction or bidirectionally. This is helpful for users who want to access data from a traditional file path on a filesystem, but the data lives in S3. It’s also good if users also want to output results onto the filesystem, but other teams want to access the results from S3.

AWS supports linking FSx and S3 resources in different accounts, and even in different regions (FSx to S3 export only). However, nowhere in the documentation does AWS cover how to configure permissions to enable cross-account data repository associations.

At Deep Origin, we just set this up to let one of our customers access data in a S3 bucket as files in a ComputeBench. I wanted to share a simplified version of bucket polices in case anyone was searching the internet for the same answer!

To access a bucket via the “import” read-only data repository association, add this policy to the source bucket. Change YOUR_BUCKET to the id of the source bucket, and ACCOUNT_WITH_FSx:user/USER to the IAM identifier of the account owning the FSx volume. You can then set up the data repository association from the account owning the FSx volume as you normally would.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:PutBucketNotification"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET/*",
                "arn:aws:s3:::YOUR_BUCKET"
            ],
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_WITH_FSx:user/USER"
            }
        }
    ]
}

For a bucket policy that also allows export of data from FSx to S3, add a few write permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:PutBucketNotification",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:PutObject",
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET/*",
                "arn:aws:s3:::YOUR_BUCKET"
            ],
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_WITH_FSx:user/USER"
            }
        }
    ]
}