lørdag den 3. maj 2014

Amazon S3 curl upload: Policy Condition failed: ["starts-with", "$Filename", ""]

For some time, I have been struggeling with fixing a bash script that uploades files to Amazon S3. It have been working, but I got an error when the configuration of the service was changed:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Invalid according to Policy: Policy Condition failed: ["starts-with", "$Filename", ""]</Message><RequestId>ED7D91115863CB7B</RequestId><HostId>mOQWlKRPdgeVeRJFStqt5N0Jcc/0cD1wqDRSiQUgaw87K7Xg/D5hqWG+zIw8ldDS</HostId></Error>Done.

So what did it mean. Searching Amazons documentation on http://doc.s3.amazonaws.com/proposals/post.html for starts-with provided the answer: A requirement had been put in, that the Filename field I send should start with "".
Why start with an empty string? The doc says, that it is a trick to ensure a field is filled out. So I did not have a Filename field - I simply added it and voila!

Now how does my command look like: I have logged in (the site requires that - the method is different from site to site) already obtained the keys needed for Amazon, which are embedded in the site. The general magic is:

curlargs="-b cookies.txt -c cookies.txt --silent --output $res"
curl $curlargs \
-F "acl=private" \
-F "AWSAccessKeyId=$AWSAccessKeyId" \
-F "Policy=$policy" \
-F "Signature=$sig" \
-F "key=$key" \
-F "success_action_status=201" \
-F "Filename=$fn" \
-F "file=@$fn" \
http://some-service.s3.amazonaws.com/

All other fields, except the $fn which contains the full path to my file, are obtained from the site by screenscraping. The hard part is getting the proper fields and the order correct.
Hope this will help a soul or two: Please comment if it does and please add knowledge.