Less than one year ago,
Amazon launched a new computation service, the
AWS-Lambda. It promises simplify the process of building applications, by hosting and running code for you. All the infrastructure and some scalability and fail over aspects are Amazon's concerns. It also integrates pretty well with other Amazon's services like
SQS,
SNS,
DynamoDB,
S3, etc. The code hosted there can even be called externally by other applications using the aws-sdk.
Here, I'll show how to use this service by doing something very simple. It idea is implement some code that will listen to an event (PUT) in a given S3 bucket, apply some processing on the file content, and send It to a SQS Queue.
This service restricts the language and platform where the code is implemented. A
NodeJS module needs to be exported and called after deployed into the Amazon infrastructure, So, if you are not familiarised with Javascript and Nodejs, I would advice you to step back and look some documentation first.
var AWS = require('aws-sdk');
var parser = require('xml2js').parseString;
var async = require('async');
var s3 = new AWS.S3();
var sqs = new AWS.SQS();
exports.handler = function(event, context) {
var bucketName = event.Records[0].s3.bucket.name;
var fileName = event.Records[0].s3.object.key;
async.waterfall([
function download(next) {
s3.getObject({Bucket: bucketName, Key: fileName}, function (err, data) {
next(err, data);
})
},
function parseXml(response, next) {
parser(response.Body.toString(), function(err, result) {
next(err, result);
})
},
function sendMessage(result, next) {
var message = {
MessageBody: JSON.stringify(result),
QueueUrl: "[YOUR QUEUE URL: i.e: https://....]"
};
sqs.sendMessage(message, function(err, data) {
if(err) {
context.fail("Error: " + err);
} else {
context.succeed("Message sent succefully: " + data.MessageId);
}
context.done();
});
}
], function(err) {
if (err) {
context.fail("Error: " + err);
throw err;
}
});
}
Lets see what is happening here:
- From line 1 to 3: Importing the modules needed on the implementation. All these modules needs to be packed when deploying the application, except by the aws-sdk, which is available by default in runtime.
- Line 9 and 10: Getting information from the event. When listing to an event from a S3 bucket, what you receive is the event metadata. So, if you want to do something with the object that uploaded, you need to extract the event metadata and then get the uploaded object and do something with the content.
- Line 12: The code from this point is a series of callbacks that depends from each other's results. So, to avoid the callback hell scenario, I used an external lib that make these functions dependencies a bit more clear to read.
In order to sure It's everything ok before deploying It, go to the console and perform a "npm install" command. It should check all code dependencies and put them into a specific directory.
Now It's time to set It up on Amazon infrastructure. The AWS-Lambda service let you upload a zip file with Its dependencies inside a Zip file. When using this option, be careful when creating the zip file. The Javascript file that will contains the code shown before needs to be at "/" on the zip file, otherwise It wont work. Worst than that, when running the code, the error message on the console is gonna show an error message that does not point on this direction.
Once is you have your code properly packed, go to the AWS console, access the "Lambda" option and and ask to create a new function. The presented screen should look like this:
There, I'm putting basic information about what I'm uploading. The most relevant information are the Handler (Javascript file + the module name to be called) and Memory and Timeout values (Amazon will use this information billing). There is still the execution role. If you don't have one yet, create It using the options available on the combo box. Once you managed to finish this step, the module is ready to be called. Now, the last step is just go to the bucket and I'm interested to monitor, and trigger this this function every time a new event happens by changing the bucket properties.
An additional and reasonable step would be test this deployed function in order be sure It's everything ok. In order to do that , go the console where all the functions are listed, select the function you want to test, press the "Actions" and select "Edit/Test". A new screen will be presented. On the left side, there is a section called "Sample Events". It simulates some real use cases. To test this function, pick "S3 Put" option and adapt the event setting valid bucket and file names.
If everything went fine, you should able see a message looking like this on the Execution result area:
"Message sent successfully: 9431465....."
Some additional billing information should be displayed also and that is It. From this point you are sure that the function is properly deployed and ready to be used.
The working example can be found
here.
.png)
