amazon web services - Permission denied while elastic beanstalk is retrieving S3 file

Question

I have files stored on S3 and wrote .ebextensions config to automatically copy the them to new instances. I'm receiving this error in the Elastic Beanstalk console:

[Instance: INSTANCEID Module: AWSEBAutoScalingGroup ConfigSet: null] Command failed on instance. Return code: 1 Output: [CMD-AppDeploy/AppDeployStage0/EbExtensionPreBuild] command failed with error code 1: Error occurred during build: Failed to retrieve https://s3-us-west-1.amazonaws.com/MyBucket/MyFolder/_MyFile.txt: HTTP Error 403 : AccessDenied

My .ebextension config file has this section:

files:
    "/target/file/path" :
        mode: "000777"
        owner: ec2-user
        group: ec2-user
        source: https://s3-us-west-1.amazonaws.com/_MyBucket_/_MyFolder_/_MyFile.txt

In attempting to make this file copying work, I've also relaxed permissions by giving the elastic beanstalk IAM role the standard read only access policy to all of S3. It's policy is this:

{
  "Effect": "Allow",
  "Action": [
    "s3:Get*",
    "s3:List*"
  ],
  "Resource": "*"
}

Yet the prebuild copying step still fails. Did I give the source url in the correct format? Is there another security entity/policy involved? Help please :)

Answer 1

The documentation is very sketchy on the subject (probably an ideal candidate for StackExchange Docs!).

To do this correctly with .ebextensions, you need to allow the Beanstalk instance IAMs user in the bucket policy, setup an AWS::CloudFormation::Authentication: auth config and attach config to remote sources. This is kind of a hybrid of all the other answers, but all failed in one way or another for me.

Assuming your IAM instance role is aws-elasticbeanstalk-ec2-role:

1. Set your AWS bucket to allow the Beanstalk IAM User. Edit "bucket policy":

   {
       "Version": "2012-10-17",
       "Id": "BeanstalkS3Copy",
       "Statement": [
           {
               "Sid": "",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "<beanstalk_iam_role_arn>"
               },
               "Action": [
                   "s3:ListBucketVersions",
                   "s3:ListBucket",
                   "s3:GetObjectVersion",
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<bucket_name>",
                   "arn:aws:s3:::<bucket_name>/*"
               ]
           }
       ]
   }
   

   where:

   beanstalk_iam_role_arn = the fully qualified instance IAMs role. See "IAM role" associated with a running instance if available or see environment configuration. Example: arn:aws:iam::12345689:role/aws-elasticbeanstalk-ec2-role

   bucket_name = your bucket name

2. In your .ebextension/myconfig.config, add an S3 authentication block that uses your IAMs instance user:

   Resources:
   AWSEBAutoScalingGroup:
     Metadata:
       AWS::CloudFormation::Authentication:
         S3Auth:
           type: "s3"
           buckets: ["bucket_name"]
           roleName:
             "Fn::GetOptionSetting":
               Namespace: "aws:asg:launchconfiguration"
               OptionName: "IamInstanceProfile" 
               DefaultValue: "aws-elasticbeanstalk-ec2-role"
   

   Set bucket_name appropriately

3. Define a remote file and attach the S3 Authentication block:

   "/etc/myfile.txt" :
      mode: "000400"
      owner: root
      group: root
      authentication: "S3Auth" # Matches to auth block above.
      source: https://s3-eu-west-1.amazonaws.com/mybucket/myfile.txt
   

   Set your source URL appropriately