If you decide to try and interact with AWS Glacier API or certain other AWS services you will need to interact with their signature version 4 authentication. Unfortunately in ColdFusion this is one of the hardest things I've ever had to do. Not really ColdFusion's fault, and not really Amazon's fault. Their documentation is comprehensive (although a little confusing) it is just incredibly fiddly. Hashing is a process where a single wrong character completely changes everything. So one slip up causes failures and it can be difficult to determine what you've done wrong.
I have previously blogged about AWS Signiature Version 2 and using it with AWS SES. This article also touches on HMAC and a few of the other key concepts:
http://webdeveloperpadawan.blogspot.co.uk/2012/02/coldfusion-and-amazon-aws-ses-simple.html
AWS Glacier -
http://aws.amazon.com/glacier/
AWS Glacier is a very low cost storage solution designed for archiving and backing up data. The basic idea is its cheaper than S3 storage but access is limited. So you don't necessarily have immediate access to your backups. Instead access can be requested and files retrieved within a given time period.
In order to keep costs low, Amazon Glacier is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable.
- AWS Website
In order to make an API request to Glacier you are required to authenticate each request using their V4 Signature process. Data in Glacier is stored in "Vaults", similar to an S3 bucket a vault is a storage container. For the purposes of this demo I've created a vault using the AWS web management interface. I will be using the API to list all available vaults. In time I hope to expand tutorials and code to cover more complex operations. However, once you've got the signature sorted that shouldn't be too hard.
Code Glorious Code
Again I just want to make the point that I'm just addressing the signature here. I hope to expand the CFC to better deal with making full requests. That will come in time though.
Setup
This should be fairly self explanatory.
variables.dteNow = DateAdd("s", GetTimeZoneInfo().UTCTotalOffset, now());
variables.strPublicKey = "publickey";
variables.strPrivateKey = "secretkey";
variables.oAwsSig4 = createObject("component","lib.awsSig4").init(strSecretKey = variables.strPrivateKey);
//We need a few custom date formats
variables.strCanonicalDate = variables.oAwsSig4.getCanonicalDateFormat(dteNow = variables.dteNow);
variables.strShortDate = variables.oAwsSig4.getShortDateFormat(dteNow = variables.dteNow);
Step 1 - Create A Canonical Request
The canonical request is basically a standard way to describe the request you are making to AWS. Be that a GET, POST to Glacier or whatever.
This is the aws documentation pseudocode that describes what's happening:
CanonicalRequest =
HTTPRequestMethod + '\n' +
CanonicalURI + '\n' +
CanonicalQueryString + '\n' +
CanonicalHeaders + '\n' +
SignedHeaders + '\n' +
HexEncode(Hash(Payload))
Although I think that's slightly wrong. In my example1 (below), the second empty line is unexplained.
Here's the call I make to the method:
//step 1, create canonical request
strCanonical = variables.oAwsSig4.createCanonical(
strHTTPRequestMethod = "GET",
strCanonicalURI = "/-/vaults",
strCanonicalQueryString = "",
arrCanonicalHeaders = ["date:#variables.strCanonicalDate#","host:glacier.us-east-1.amazonaws.com","x-amz-glacier-version:2012-06-01"],
arrSignedHeaders = ["date","host","x-amz-glacier-version"],
strPayLoad = ""
);
You'll note the payload is empty, so the hash at the bottom is simply a SHA-256 hash of "".
The finished Canonical request should look like this.
Example1:
GET
/-/vaults
date:2013-06-26T13:07:03
host:glacier.us-east-1.amazonaws.com
x-amz-glacier-version:2012-06-01
date;host;x-amz-glacier-version
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Step 2 - Create A String To Sign
The string to sign is a little harder to explain out of context. It's basically the message we will hash which will authorize the request to AWS. It's a short and simple string with a very concise format.
The pseudocode is fine here and describes it quite well:
StringToSign =
Algorithm + '\n' +
RequestDate + '\n' +
CredentialScope + '\n' +
HexEncode(Hash(CanonicalRequest))
Here's my function call:
//Step 2 - Create String To Sign
strStringToSign = variables.oAwsSig4.createStringToSign(
strAlgorithm = "AWS4-HMAC-SHA256",
strRequestDate = variables.oAwsSig4.getStringToSignDateFormat(dteNow = variables.dteNow),
strCredentialScope = strShortDate & "/us-east-1/glacier/aws4_request",
strCanonicalRequest = strCanonical
);
This is the finished string to sign:
AWS4-HMAC-SHA256
20130626T133038Z
20130626/us-east-1/glacier/aws4_request
6d26f46dbf5d48665e06f44a2f9a65368b3b8d9ef45638b1496fbbe6604ed9db
Step 3a - Calculate Signing Key
OK Now it gets complicated. This is where we start running things through the HMAC function and problems quickly occur. If you do it right though, it'll all come together. AWS do this all in one step, but I think its easier as two.
Here's the documentation pseudocode:
kSecret = Your AWS Secret Access Key
kDate = HMAC("AWS4" + kSecret, Date)
kRegion = HMAC(kDate, Region)
kService = HMAC(kRegion, Service)
kSigning = HMAC(kService, "aws4_request")
Here's my function call:
//create singing key
bSigningKey = variables.oAwsSig4.createSigningKey(
dateStamp = strShortDate,
regionName = "us-east-1",
serviceName = "glacier"
);
and here's the function:
<cffunction name="createSigningKey" access="public" returnType="binary" output="false" hint="THIS WORKS DO NOT FUCK WITH IT.">
<cfargument name="dateStamp" type="string" required="true" />
<cfargument name="regionName" type="string" required="true" />
<cfargument name="serviceName" type="string" required="true" />
<cfscript>
var kSecret = JavaCast("string","AWS4" & variables.strSecretKey).getBytes("UTF8");
var kDate = HMAC_SHA256_bin(arguments.dateStamp, kSecret);
var kRegion = HMAC_SHA256_bin(arguments.regionName, kDate);
var kService = HMAC_SHA256_bin(arguments.serviceName, kRegion);
var kSigning = HMAC_SHA256_bin("aws4_request", kService);
return kSigning;
</cfscript>
</cffunction>
I know I've not included the function in the other steps, but I want to highlight the importance of two things:
- kSecret is "AWS4" + Secret Key, then cast into bytes. This is a very important step and where I was going wrong for quite a while.
- variables.strSecretKey is the secret key you get from AWS in your account section. It's obviously secret and shouldn't be disclosed to anyone. In my example it's set in the variables scope of the cfc.
- The function I use HMAC_SHA256_bin accepts a binary argument as param1. This is different from the example in my previous blog post on signature version 2, which used two strings as arguments.
So you can see with four HMAC steps, the tiniest mistake means a totally different response. Obviously AWS will be doing the same thing on their end, so if the two don't match, your request won't get approved.
Step 3b - Sign it!
OK Now we bring it all together
signature = HexEncode(HMAC(derived-signing-key, string-to-sign))
We take the signing key from step 3a and the string to sign from step 2:
bSignature = variables.oAwsSig4.HMAC_SHA256_bin(strStringToSign, bSigningKey);
Step 4 - Put it all together
Now we make our request:
<cfhttp method="GET" url="http://glacier.us-east-1.amazonaws.com/-/vaults">
<cfhttpparam type="header" name="Date" value="#variables.strCanonicalDate#">
<cfhttpparam type="header" name="x-amz-glacier-version" value="2012-06-01" />
<cfhttpparam type="header" name="Authorization" value="AWS4-HMAC-SHA256 Credential=#variables.strPublicKey#/#variables.strShortDate#/us-east-1/glacier/aws4_request,SignedHeaders=date;host;x-amz-glacier-version,Signature=#lcase(binaryEncode(bSignature, 'hex'))#" />
</cfhttp>
- Note we didn't hex encode our bSigniature from before, so I do that in the value. Probably better done elsewhere, but I'll get to it.
- Also note the public key. Again this comes from AWS management console.
- Note the three different dates, one is the strCanonicalDate we created at the top, the other is the short date and finally the hard coded glacier version date.
That's it. That should work! Obviously you need the cfc. Take a look below for that.
Advice
My advice to anyone attempting to do this in ColdFusion or any other language is as such:
- Baby Steps - The documentation is presented in steps. Get each step working perfectly before moving onto the next. They all rely on each other, so a mistake early on will just cascade and waste time later.
- Unit Tests - I'm a huge fan of unit tests anyway, but in this case they really helped. Setting up some great unit tests using AWS examples will help you define your input and your output and tweak your code until you get the response you're looking for.
- Check Responses - If you actually make a request to AWS they will tell you in the response what the problem is. Plus they'll tell you the expected canonical request and the expected string to sign. These can really help iron out any tiny discrepancies
- Try it in Java - CFML Doesn't have a native HMAC function (pre CF10), and converting too and from byte arrays caused endless problems. So I just did a few piecemeal functions in Java and learned from that.
The CFC
Ah of course one final step. The cfc. As I've said I hope to improve it a great deal and perhaps open source it and put it on
cflib. I've also got a bunch of unit tests I wrote which may help people improve it.
Right now I'm just pleased to have got it working and don't want to forget it all so it's going on the blog.
Thanks
I ended up not using the code, but some of Ben Nadel's stuff was really useful to understand. He's written a great cfc that I urge anyone using HMAC in CFML to consider:
http://www.bennadel.com/blog/2412-Crypto-cfc-For-Hmac-SHA1-Hmac-Sha256-and-Hmac-MD5-Code-Generation-In-ColdFusion.htm
<cfcomponent output="false">
<!---
<OWNER> = James Solo
<YEAR> = 2013
In the original BSD license, both occurrences of the phrase "COPYRIGHT HOLDERS AND CONTRIBUTORS" in the disclaimer read "REGENTS AND CONTRIBUTORS".
Here is the license template:
Copyright (c) 2013, James Solo
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--->
<cffunction name="init" access="public" returnType="awsSig4" output="false">
<cfargument name="strSecretKey" type="string" required="true" />
<cfset variables.strSecretKey = arguments.strSecretKey />
<cfset variables.strPublicKey = "mypublickey" />
<cfreturn this>
</cffunction>
<cffunction name="createCanonical" access="public" returnType="string" output="false" hint="Create the canonical request">
<cfargument name="strHTTPRequestMethod" type="string" required="true" />
<cfargument name="strCanonicalURI" type="string" required="true" />
<cfargument name="strCanonicalQueryString" type="string" required="false" default="" />
<cfargument name="arrCanonicalHeaders" type="array" required="true" />
<cfargument name="arrSignedHeaders" type="array" required="true" />
<cfargument name="strPayload" type="string" required="false" default="" />
<cfscript>
var intCount = 0;
var strHeaderString = "";
var strCanonicalRequest =
arguments.strHTTPRequestMethod & chr(010) &
arguments.strCanonicalURI & chr(010) &
arguments.strCanonicalQueryString & chr(010);
//Headers
for(intCount=1; intCount <= arraylen(arrCanonicalHeaders); intCount++){
strCanonicalRequest &= arguments.arrCanonicalHeaders[intCount] & chr(010);
}
strCanonicalRequest &= chr(010);
//Signed headers
for(intCount=1; intCount <= arraylen(arrSignedHeaders); intCount++){
strHeaderString = arguments.arrSignedHeaders[intCount];
strCanonicalRequest &= strHeaderString;
//put a semi-colon between headers, or a new line at end
if(intCount EQ arraylen(arrSignedHeaders)){
strCanonicalRequest &= chr(010);
}else{
strCanonicalRequest &= ";";
}
}
strCanonicalRequest &= lcase(hash(arguments.strPayload, "SHA-256"));
return trim(strCanonicalRequest);
</cfscript>
</cffunction>
<cffunction name="createStringToSign" access="public" returnType="string" output="false" hint="I create the string to sign">
<cfargument name="strAlgorithm" type="string" required="true" />
<cfargument name="strRequestDate" type="string" required="true" />
<cfargument name="strCredentialScope" type="string" required="true" />
<cfargument name="strCanonicalRequest" type="string" required="true" />
<cfscript>
var strStringToSign =
arguments.strAlgorithm & chr(010) &
arguments.strRequestDate & chr(010) &
arguments.strCredentialScope & chr(010) &
lcase(hash(arguments.strCanonicalRequest, "SHA-256"));
return strStringToSign;
</cfscript>
</cffunction>
<cffunction name="createSigningKey" access="public" returnType="binary" output="false" hint="THIS WORKS DO NOT FUCK WITH IT.">
<cfargument name="dateStamp" type="string" required="true" />
<cfargument name="regionName" type="string" required="true" />
<cfargument name="serviceName" type="string" required="true" />
<cfscript>
var kSecret = JavaCast("string","AWS4" & variables.strSecretKey).getBytes("UTF8");
var kDate = HMAC_SHA256_bin(arguments.dateStamp, kSecret);
var kRegion = HMAC_SHA256_bin(arguments.regionName, kDate);
var kService = HMAC_SHA256_bin(arguments.serviceName, kRegion);
var kSigning = HMAC_SHA256_bin("aws4_request", kService);
return kSigning;
</cfscript>
</cffunction>
<cffunction name="HMAC_SHA256_bin" access="public" returntype="binary" output="false" hint="THIS WORKS DO NOT FUCK WITH IT.">
<cfargument name="signMessage" type="string" required="true" />
<cfargument name="signKey" type="binary" required="true" />
<cfset var jMsg = JavaCast("string",arguments.signMessage).getBytes("UTF8") />
<cfset var jKey = arguments.signKey />
<cfset var key = createObject("java","javax.crypto.spec.SecretKeySpec") />
<cfset var mac = createObject("java","javax.crypto.Mac") />
<cfset key = key.init(jKey,"HmacSHA256") />
<cfset mac = mac.getInstance(key.getAlgorithm()) />
<cfset mac.init(key) />
<cfset mac.update(jMsg) />
<cfreturn mac.doFinal() />
</cffunction>
<cffunction name="toHex" access="public" returnType="string" output="false" hint="I convert binary to hex">
<cfargument name="bSignature" type="binary" required="true" />
<cfreturn lcase(binaryEncode(arguments.bSignature, "hex")) />
</cffunction>
<cffunction name="getCanonicalDateFormat" access="public" returnType="string" output="false" hint="I return a formatted date time for the canonical part of the process">
<cfargument name="dteNow" type="date" required="true" />
<cfreturn "#dateformat(arguments.dteNow, 'yyyy-mm-dd')#T#TimeFormat(arguments.dteNow, 'HH:mm:ss')#" />
</cffunction>
<cffunction name="getStringToSignDateFormat" access="public" returnType="string" output="false" hint="I return a formatted date time for the string to sign section">
<cfargument name="dteNow" type="date" required="true" />
<cfreturn "#dateformat(arguments.dteNow, 'yyyymmdd')#T#TimeFormat(arguments.dteNow, 'HHmmss')#Z" />
</cffunction>
<cffunction name="getShortDateFormat" access="public" returnType="string" output="false" hint="I return a short date time">
<cfargument name="dteNow" type="date" required="true" />
<cfreturn "#dateformat(arguments.dteNow, 'yyyymmdd')#" />
</cffunction>
</cfcomponent>
