How to upload and serve data using Amazon CloudFront and Amazon S3 in Node.js

Most of the applications today serve users across the globe and need a way to deliver their content fast. The content can be images, videos, PDF and so on. Content delivery network (CDN) is a network of servers that are geographically distributed with the intent of serving content to users as fast as possible.

Amazon CloudFront is a content delivery network service which can deliver content fast and secure. In this article, I will describe how to upload files to S3 bucket and serve those files through CloudFront in Node.js. CloudFront will use S3 as an origin in this article.

Prerequisites

Create a bucket in S3 and create a CloudFront distribution in AWS. I will not go into detail about creating S3 bucket and CloudFront distribution in this article. Navigate to IAM and go to ‘Security Credentials’ under user. Create an access key and download the CSV file in case you don’t already have one. We will need this access key later.

Then, click on ‘My Security Credentials’ under the your account.

Under CloudFront keypairs, create a keypair and download the private key and note down your access key ID. We will need them later for the integration.

Creating Node.js application

Let’s create a simple Node.js express server and add two REST API endpoints for file upload and download. Here is the sample project structure.

I am using typescript and ts-node-dev npm modules in this sample project. Therefore, I have tsconfig.json in the project.  

Here is the entire app.ts file. The file contains the logic to initialize express server and REST endpoints. I am also using multer npm module to handle multi-part file upload.

import express from 'express';
import * as fileCtrl from './fileController';
import multer from 'multer';
import crypto from 'crypto';
import path from 'path';

const app = express();
const port = 3000;

app.listen(port, () => {
  console.log('Server listening on port %s.', port);
});

//Multer module for handling multi part file upload.
var storage = multer.diskStorage({
  destination: './files',
  filename: function (req, file, cb) {
    crypto.pseudoRandomBytes(16, function (err, raw) {
      if (err) return cb(err)

      cb(null, raw.toString('hex') + path.extname(file.originalname))
    })
  }
})

app.use(multer({ storage: storage }).single('file'));


app.get('/api/download', asyncHandler(fileCtrl.download));
app.post('/api/upload', asyncHandler(fileCtrl.upload));

export function asyncHandler(handler) {
  return function (req, res, next) {
    if (!handler) {
      next(new Error(`Invalid handler ${handler}, it must be a function.`));
    } else {
      handler(req, res, next).catch(next);
    }
  };
}

Uploading files to Amazon S3 Bucket

Let us look at how to upload files to S3 bucket. We will need to install node module aws-sdk to access S3 buckets from Node.js application.

Once we have installed, the handler for upload endpoint is defined in as follows:

export async function upload(req, res) {
  let response = await uploadFile(req.file.originalname, req.file.path);
  res.send(response);
  res.end();
}

In fileComponent.ts, we need to import aws-sdk module as follows.

import awsSDK from 'aws-sdk';

In the beginning of this article, we downloaded CSV file that contained access key id and secret access key. We will use them to upload files to S3 bucket. Using aws-sdk module, we need to configure access key id and secret access key as follows:

export function uploadFile(filename, fileDirectoryPath) {
  awsSDK.config.update({ accessKeyId: process.env.S3_ACCESS_KEY_ID, secretAccessKey: process.env.S3_SECRET_ACCESS_KEY });
  const s3 = new awsSDK.S3();

  return new Promise(function (resolve, reject) {
    fs.readFile(fileDirectoryPath.toString(), function (err, data) {
      if (err) { reject(err); }
      s3.putObject({
        Bucket: '' + process.env.S3_BUCKET_NAME,
        Key: filename,
        Body: data,
        ACL: 'public-read'
      }, function (err, data) {
        if (err) reject(err);
        resolve("succesfully uploaded");
      });
    });
  });
}

Using putObject() method, we will upload files to S3 bucket. In putObject(), we need to pass the bucket name to which we will upload the files. Note that, depending on your bucket policies, you can send parameters in putObject(). In this example, I have set the canned ACL policy to public-read. Make sure to check your S3 bucket policies.

Note that, I have used environment variables for sensitive information such as access key ID, secret key and so on. If the variables are configured as recommended by AWS, they don’t need to be explicitly passed to the S3 client as above. The S3 client automatically detects the configured environment variables.

Now, we can start the server and test our POST endpoint. Here is an example from Postman.

Once the request is successful, we can see the file in S3 bucket.

Serving Files via Amazon CloudFront

Earlier, we downloaded private key from CloudFront keypairs. We will use that private key and access key ID to access CloudFront in Node.js.

The handler for download API endpoint is as follows.

export async function download(req, res) {
  let response = await getFileLink(req.query.filename);
  res.send(response);
  res.end();
}

In this handler, we are expecting a file name that has to be downloaded via CloudFront.

Let us look at how to access CloudFront in our Node.js app. First, we will install aws-cloudfront-sign npm module. Using this module, we can get signed Amazon CloudFront URLs which enables us to provide users access to our private content. The signed URL also contains additional meta information such as expiration time. This gives more control over the access to our content.

export function getFileLink(filename) {
  return new Promise(function (resolve, reject) {
    var options = { keypairId: process.env.CLOUDFRONT_ACCESS_KEY_ID, privateKeyPath: process.env.CLOUDFRONT_PRIVATE_KEY_PATH };
    var signedUrl = awsCloudFront.getSignedUrl(process.env.CLOUDFRONT_URL + filename, options);
    resolve(signedUrl);
  });
}

Here, we need to pass access key ID, path to private key file and CloudFront URL to getSignedURL(). The CloudFront URL would look something like this: https://XYZ.cloudfront.net. ‘XYZ’ would be your CloudFront distribution name.

Start the server and test the GET endpoint as follows:

Conclusion

In this article, we saw how to upload files to Amazon S3 and serve those files via Amazon CloudFront. I hope you enjoyed this article. Let me know if you have any comments or suggestions in the comments section below.

The example for this article can be found on GitHub repository.

Advertisements

3 thoughts on “How to upload and serve data using Amazon CloudFront and Amazon S3 in Node.js

  1. You don’t have to pass the s3 environment variables into the S3 client, it will use the default/recommended variables by default if they’re already in the environment.

    • Hi, thanks for the pointing it out. I forgot to mention this point. I will update the article.

Comments are closed.