3.6.3: Heroku S3

Search…
3.6.3: Heroku S3
Introduction
Heroku is a service built on top of EC2 that is optimised for scalability.
Heroku lets us initialise hundreds of "dynos" (i.e. virtual machines) on a single app deployment command, each running a copy of our app.
Currently, 84 normal sized Heroku dynos costs $2100 USD per month.
With multiple dynos, if a user uploads a file to 1 dyno, that file is trapped on that dyno, because Heroku automatically routes traffic to any of our running dynos, so there is no guarantee a user will always visit the same dyno.
In fact, the Heroku platform explicitly does not support uploads we can send back to the user. If we keep a file inside a dyno long enough (usually about a day), it may disappear. Read about this behaviour here and here.
S3
S3 is AWS' static file storage service, and the file storage service that Heroku recommends. We can store files in S3 on behalf of our users.
Secrets
The following are steps to use S3. We will obtain S3 secret keys we'll need to authenticate our server when uploading a file to S3.
1.Go to the new user creation page to create a user for our current app: https://console.aws.amazon.com/iam/home#/users​
2.Add a user: https://console.aws.amazon.com/iam/home#/users$new?step=details​
3.Name the user after our app.
4.Choose "Attach existing policies directly". Filter for S3 policies. Select the AmazonS3FullAccess policy. In a production app we would select more fine-grained permissions.
1.Copy the generated secret keys to use in our app.
2.Set environment variables to access secret key values in our app.
export ACCESSKEYID=<AWS_KEY>
export SECRETACCESSKEY=<AWS_SECRET>
Important
1.Never commit these keys in our code.
2.If we accidentally commit keys, never push that commit to a GitHub repo, especially a public repo.
3.If we committed and pushed our keys by accident, delete them from AWS immediately.
4.We would be responsible for any charges to our AWS account if these keys were used by anyone else.
Buckets
A bucket is the S3 equivalent of a directory. Depending on the application we can have 1 or more buckets per app. The maximum bucket size is 5TB.
To start, we'll create a single bucket for this app. Multiple-bucket apps are out of the scope of this exercise.
1.Create a bucket here. Make sure to save the name of the bucket.
2.Allow public access on the bucket.
Upload Files to S3 in Express with Multer
We will use additional NPM libraries to use Multer with S3. The following assumes we already have Multer installed and configured from the previous file upload exercise.​
npm install aws-sdk multer-s3
Import the libraries in our app.
import aws from 'aws-sdk';
import multerS3 from 'multer-s3';
Initialise the S3 SDK with our secret keys from environment variables.
const s3 = new aws.S3({
  accessKeyId: process.env.ACCESSKEYID,
  secretAccessKey: process.env.SECRETACCESSKEY,
});
Initialise the Multer SDK with multerS3.
const multerUpload = multer({
  storage: multerS3({
    s3,
    bucket: '<MY_BUCKET_NAME>',
    acl: 'public-read',
    metadata: (request, file, callback) => {
      callback(null, { fieldName: file.fieldname });
    },
    key: (request, file, callback) => {
      callback(null, Date.now().toString());
    },
  }),
});
Assuming a form like the following that includes a file...
<form action="/recipe" method="post" enctype="multipart/form-data">
  <label for="label">recipe label:</label><br />
  <input type="text" id="label" name="label" /><br />
  <label for="photo">recipe photo:</label><br />
  <input type="file" name="photo" />
  <input type="submit" value="Submit" />
</form>
...we can upload that file to S3 using our multerUpload middleware.
app.post('/recipe', multerUpload.single('photo'), (request, response) => {
  console.log(request.file);
  response.send(request.file);
});
The uploaded file location will be available in our request handling middleware at request.file.location. We can record this location in our database to retrieve the file when we need to. We can also inspect the file in the S3 console.
Use S3 with Express and Heroku
To use S3 in our Express apps with Heroku, we'll need to set our S3 secret keys as environment variables or "config vars" in Heroku.
1.Visit the Settings tab of the Heroku app dashboard and paste our S3 secret keys. The config vars form should look like the following.
3.6.2: Heroku Postgres
3.ICE: In-Class Exercises
Last modified 3mo ago
Copy link
Outline
Introduction
Secrets
Buckets
Upload Files to S3 in Express with Multer
Use S3 with Express and Heroku