amazon web services - Jmeter-How to copy the files in S3 from one bucket to another

Question

I need to upload large files from one bucket(source) in AWS S3 to different bucket(destination) in same S3. we have the script which will pick the files from destination and move the files.

I have tried below code and though sampler is passed but file is not copied. Then i have tried to upload from local to S3. As i need to test with large files upto 15GB, its takes lot of time to upload to S3. Is there a way to copy from S3 to S3.

import com.amazonaws.AmazonServiceException;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.CopyObjectRequest;
import java.io.IOException;
public class CopyObjectSingleOperation {
    public static void main(String[] args) throws IOException {
        Regions clientRegion = Regions.DEFAULT_REGION;
        String bucketName = "*** Bucket name ***";
        String sourceKey = "*** Source object key *** ";
        String destinationKey = "*** Destination object key ***";
        try {
            AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                    .withCredentials(new ProfileCredentialsProvider())
                    .withRegion(clientRegion)
                    .build();
            // Copy the object into a new object in the same bucket.
            CopyObjectRequest copyObjRequest = new CopyObjectRequest(bucketName, sourceKey, bucketName, destinationKey);
            s3Client.copyObject(copyObjRequest);
        } catch (AmazonServiceException e) {
            // The call was transmitted successfully, but Amazon S3 couldn't process 
            // it, so it returned an error response.
            e.printStackTrace();
        } catch (SdkClientException e) {
            // Amazon S3 couldn't be contacted for a response, or the client  
            // couldn't parse the response from Amazon S3.
            e.printStackTrace();
        }
    }
}

question from:https://stackoverflow.com/questions/66050588/jmeter-how-to-copy-the-files-in-s3-from-one-bucket-to-another

Answer 1

You can use one of following option;

1. S3 Batch Operations job

2. AWS CLI aws s3 sync e.g. aws s3 sync s3://source-examplebucket s3://destination-examplebucket

3. AWS CLI aws s3 cp e.g. aws s3 cp s3://source-examplebucket/ s3://destination-examplebucket/ --recursive

4. AWS DataSync service