In Brief
In order to keep the uploaded media (S3 objects) private for all the clients on my multi-tenant system I implemented a Cloudfront CDN deployment and configured it (and its Origin S3 Bucket) to force the use of signed URLs in order to GET any of the objects.
The Method
First, the user is authenticated via my system, and then a signed URL is generated and returned to them using the AWS.CloudFront.Signer.getSignedUrl()
method provided by the AWS JS SDK. so they can make the call to CF/S3 to download the object (image, PDF, docx, etc). Pretty standard stuff.
The Problem
The above method works 95% of the time. The user obtains a signed URL from my system and then when they make an XHR to GET the object it's retrieved just fine.
But, 5% of the time a 403 is thrown with a CORS error stating that the client origin is not allowed by Access-Control-Allow-Origin
.
This bug (error) has been confirmed across all environments: localhost, dev.myapp.com, prod.myapp.com. And across all platforms/browsers.
There's such a lack of rhyme or reason to it that I'm actually starting to think this is an AWS bug (they do happen, from time-to-time).
The Debugging Checklist So Far
I've been going out of my mind for days now trying to figure this out. Here's what I've attempted so far:
Have you tried a different browser/platform?
Yes. The issue is present across all client origins, browsers (and
versions), and all platforms.
Is your S3 Bucket configured for CORS correctly?
Yes. It's wide-open in fact. I've even set <MaxAgeSeconds>0</MaxAgeSeconds>
in
order to prevent cacheing of any pre-flight OPTIONS
requests by the
client:
Is the signed URL expired?
Nope. All of the signed URLs are set to expire 24hrs after generation. This problem has shown up even seconds
after any given signed URL is generated.
Is there an issue with the method used to generate the signed URLs?
Unlikely. I'm simply using the AWS.CloudFront.Signer.getSignedUrl()
method of their JS SDK. The signed URLs do work most of the time, so
it would seem very strange that it would be an issue with the signing
process. Also, the error is clearly a CORS error, not a signature
mis-match error.
Is it a timezone/server clock issue?
Nope. The system does serve users across many timezones, but that
theory proved to be false given that the signed URLs are all generated
on the server-side. The timezone of the client doesn't matter, it gets
a signed URL good for 24hrs from the time of generation no matter what
TZ it's in.
Is your CF distro configured properly?
Yes, so far as I can make out by following several AWS guides,
tutorials, docs and such.
Here's a screenshot for brevity. You can see that I've disabled
cacheing entirely in an attempt to rule that out as a cause:
Are you seeing this error for all mime-types?
No. This error hasn't been seen for any images, audio, or video files
(objects). With much testing already done, this error only seems to
show up when attempting to GET a document or PDF file (.doc, .docx,
.pdf). This lead me to believe that this was simply an Accept
header
mis-match error: The client was sending an XHR with the the header
Accept: pdf
, but really the signature was generated for Accept: application/pdf
.
I haven't yet been able to fully rule this out as a
cause. But it's highly unlikely given that the errors are
intermittent. So if it were a Accept
header mis-match problem then it
should be an error every time.
Also, the XHR is sending Accept: */*
so it's highly unlikely this is where the issue is.
The Question
I've really hit a wall on this one. Can anyone see what I'm missing here? The best I can come up with is that this is some sort of "timing" issue. What sort of timing issue, or if it even is a timing issue, I've yet to figure out.
Thanks in advance for any help.
See Question&Answers more detail:
os