email@encode8.com
JavaScriptTutorials

How to Upload A Remote File to Amazon S3

AWS’ Simple Storage Service or S3 is a very popular object storage service offered by Amazon. A huge number of large-scale websites take advantage of S3. You can easily use it as a Content Delivery Network. To top it all, it has excellent API support that developers can take advantage of. If you are building a web-app and require a storage for user-uploaded data, for instance, you can quickly integrate S3. AWS SDK comes in several flavors — from python to nodejs. In this post I am going to share a simple snippet that you can use to upload data from remote URL to AWS S3, using JavaScript.

The Problem

The Nodejs SDK for S3 does offer an API to upload data to S3, but you can upload data either from browser or from a server which you have access to. However, your project may require you to upload a file that you don’t have a copy of; you only have an URL where the file is available. You can download the file and then upload to S3. However, this method may not be useful when you need to automate this process.

For example, Cloudinary has a very simple API that lets you upload an image or video just by inserting the remote URL of the file. It kind of ‘sucks’ the whole file all by itself and spits out the uploaded URL.

However, Amazon S3 does not allow such remote file upload. Here in this post, I am going to share a snippet that solves the problem.

The tools

I am going to use Axios for making http requests. Axios is a nifty JavaScript library that makes fetching remote content a breeze. For interacting with AWS services, I am going to use the official Nodejs library. So, start with:

npm install aws-sdk axios --save

Configuration

I like to save my configuration in a JSON file. For this simple case, here is how the configuration looks like:

{
    "s3" : {
        "endpoint" : "changeme.with.amazons3.endpoint",
        "api_key" : "changeMeWithAPIKey",
        "api_secret" : "inertAPISecretHERE",
        "bucket" : "bucketname",
        "subfolder" : "subfolderName",
        "acl" : "public-read"
    }
}

Make sure you change the values with the proper ones.

Uploader Class

Here is the uploader class that will upload any remote file to S3:

const config = require('../config.json');
 
const stream = require('stream');
const axios = require('axios');
const AWS = require('aws-sdk');
 
class S3RemoteUploader {
    constructor(remoteAddr){
        this.remoteAddr = remoteAddr;
        this.stream = stream;
        this.axios = axios;
        this.config = config;
        this.AWS = AWS;
        this.AWS.config.update({
            accessKeyId: this.config.api_key,
            secretAccessKey: this.config.api_secret
        }); 
        this.spacesEndpoint = new this.AWS.Endpoint(this.config.endpoint);
        this.s3 = new this.AWS.S3({endpoint: this.spacesEndpoint});

        this.file_name = this.remoteAddr.substring(this.remoteAddr.lastIndexOf('/')+1);
        this.obj_key = this.config.subfolder+'/'+this.file_name;
        this.content_type = 'application/octet-stream';

        this.uploadStream();
    }   

    uploadStream(){
        const pass = new this.stream.PassThrough();
        this.promise = this.s3.upload({
            Bucket: this.config.bucket,
            Key: this.obj_key,
            ACL: this.config.s3.acl,
            Body: pass,
            ContentType: this.content_type
        }).promise();
        return pass;
    }

    initiateAxiosCall() {
        return axios({
            method: 'get',
            url: this.remoteAddr,
            responseType: 'stream'
        });
    }

    async dispatch() {
        await this.initiateAxiosCall().then( (response) => {
            if(response.status===200){
                this.content_type = response.headers['content-type'];
                response.data.pipe(this.uploadStream());
            }
        });
        return this.promise;
    }
}

module.exports = {
    S3RemoteUploader: S3RemoteUploader
}

Here is how you can use this class:

const S3RemoteUploader = require('path/to/S3UploaderClass').S3RemoteUploader;
fileUpload = new S3RemoteUploader('http://remoteurl.tld/file.ext');
fileUpload.dispatch().then( (r) => {
    const uploadedURL = r.Location;
    // do something
})

Test

To test the code (using Jest):

const S3RemoteUploader = require('/path/to/S3UploaderClass').S3RemoteUploader;

test('uploads a file to S3 bucket', ()=> {
    testUpload = new S3RemoteUploader('https://remoteurl.tld/file.ext');
    testUpload.dispatch().then( (r) => {
        expect(r.Location).toBe('https://s3.bucket.url.already.known');
    });
});