java - parallel stream is not working for multiple file upload to aws s3

Question

The following method is written to parallel upload of files to AWS S3, but the parallel processing is not happening. It is taking 30 sec if uploading 30 files, the 50s if uploading 50 files. I am not sure why the parallel() stream is not working in my use case. Any pointers will be appreciated

public ConcurrentHashMap<String, String> uploadMyFiles(MultipartFile[] files, String prefix) {
    ConcurrentHashMap<String, String> map = new ConcurrentHashMap<String, String>();
    Stream.of(files).parallel().forEach(file -> {
        String fStatus =  uploadMyFile(file, prefix);                   
        map.put(file.getOriginalFilename(), fStatus);
    }); 

    return fileStatusMap;
}


public String uploadMyFile(final MultipartFile multipartFile, String prefix) {
    try {
        final File file = convertMPartFileToFile(multipartFile);
        uploadToMyBucket("mybucket", file, prefix);

    } catch (final AwsServiceException exception) {         
    
        return exception.getMessage()
    }
    return "OK";
}

private File convertMPartFileToFile(final MultipartFile multipartFile) {
    final File file = new File(multipartFile.getOriginalFilename());
    try (final FileOutputStream outputStream = new FileOutputStream(file)) {
        outputStream.write(multipartFile.getBytes());
    } catch (final IOException ex) {
        LOGGER.error("log error here");
    }
    return file;
}

private String uploadToMyBucket(final String bucket, final File file, String fileKey) {

    PutObjectResponse putObjectResult = s3Client
            .putObject(PutObjectRequest.builder().bucket(bucket).key(fileKey).build(), RequestBody.fromFile(file));
    final URL myfileUrl = s3Client.utilities().getUrl(GetUrlRequest.builder().bucket(bucket).key(fileKey).build());

    return myfileUrl.toString();
}

question from:https://stackoverflow.com/questions/65847081/parallel-stream-is-not-working-for-multiple-file-upload-to-aws-s3

Answer 1

I would suggest to use TransferManager along with ExecutorService as shown in examples below;

Parallelizing Large Uploads for Speed and Reliability

Multipart Uploads in Amazon S3