HTTP: The language of the internet

Protocol for hypermedia communication between distributed computers.

Required: Method, Path, HTTP version
Optional: Headers, Body

GET / HTTP/1.1
Host: austingil.com
Accept: */*
Accept-Encoding: gzip
Connection: keep-alive

Say hi to your dog for me

Browsers are HTTP clients

• Make HTTP requests on our behalf.
• Convert HTTP responses into websites.
• Provide APIs to send files via HTML or JavaScript.

Construct an HTTP request

<form>
  <button>Upload</button>
</form>

Send Data As Request Body

<form method="post">
  <button>Upload</button>
</form>

Access Files

<form method="post">
  File:
  <input type="file" />
  <button>Upload</button>
</form>

Make it accessible

<form method="post">
  <label for="file">File:</label>
  <input type="file" id="file" />
  <button>Upload</button>
</form>

Actually Send File Contents

<form method="post" enctype="multipart/form-data">
  <label for="file">File:</label>
  <input type="file" id="file" name="file" />
  <button>Upload</button>
</form>

Behold The HTML File Upload Form

<form method="post" enctype="multipart/form-data">
  <label for="file">File:</label>
  <input type="file" id="file" name="file" />
  <button>Upload</button>
</form>

Translated To HTTP

POST / HTTP/1.1
Content-Type: multipart/form-data;boundary=
  ---------------------------735323031399963166993862150
Content-Length: 17

-----------------------------735323031399963166993862150
Content-Disposition: form-data; name="file"; filename="nugget.txt"
Content-Type: text/plain

Who's a good boy?
-----------------------------735323031399963166993862150--

We’re Uploading Files

Same same but different

• Still need access to the file system
• Still need an HTTP request (fetch or XMLHttpRequest)
• Still need a POST method
• Still need to put the file in the body

Technically, this works

<input type="file" onchange="handleFileChange">

function handleFileChange(event) {
  fetch('/some-url', {
    method: 'post',
    body: event.target.files[0],
  });
}

Why?

We’ve improved the UX

Node is event-driven

During HTTP requests, each chunk triggers the request.on() method.

function processNodeRequest(request) {
  request.on("data", (data) => {
    console.log(data);
  }
}

We’ve made a terrible mistake…

• Whole file contents in memory?
• Manually parsing HTTP requests?

Use streaming library (formidable)

function processNodeRequest(request) {
  return new Promise((resolve, reject) => {
    const form = formidable({ multiples: true })
    form.parse(request, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
}

We’re Saving Files to Disk

What is Object Storage?

A single, central place to store and access all of your uploads.

Highly available, easily scalable, and often 10x cheaper than server space.

Ideally should be S3-compatible.

What is S3?

S3 ("Simple Storage Service") is AWS’s object storage product.

Has become a standard communication protocol for Object Storage solutions.

Libraries: @aws-sdk/client-s3 @aws-sdk/lib-storage

Getting files to Object Storage

1. Signed URLs
2. Stream through proxy

Signed URLs

• Frontend request asks for a signed URL.
• Backend gets it from Object Storage provider.
• Backend gives signed URL to the frontend.
• Frontend uploads directly to provider.

Streaming through formidable

const form = formidable({
  multiples: true,
  fileWriteStreamHandler: fileWriteStreamHandler,
});

function fileWriteStreamHandler(file) {
  // TODO
}

Getting into the weeds

• formidable wants to write chunks to a stream.
• S3 upload wants to read chunks from a stream.
• We need to store the file’s location somewhere.
• We need to track pending upload requests.

We’re Using Object Storage

Content Delivery Networks (CDNs)

Globally distributed network of servers that caches content close to users.

Initial requests pass through CDN node, grab content from origin, and get cached. Subsequent requests use cached content.

Faster deliver of static assets (HTML, CSS, JavaScript, images, fonts, etc.)

Things to consider…

• Compounding impact for render blocking resources
• Caching strategy is really hard to get right
  • Duration
  • Invalidation
  • Host URL (subpath vs subdomain, cookies, etc)
• CDNs have evolved in the last 25 years
  • Edge-layer security (DDoS, bots, etc)
  • Automatic media optimization
  • Edge compute

We’re Delivering Files Quickly

OWASP File Upload Cheat Sheet

• Extension Validation
• Filename Sanitization
• Upload and Download Limits
• File Storage Location
• Content-Type Validation
• File Content Validation
• Malware Scanning Architecture

Filename Sanitization

Prevent excessively long file names or invalid characters for OS.

const form = formidable({
  // other config options
  filename(file) {
    // return some random string
  },
});

(formidable does this by default)

File Size Limits

File upload & download size impacts storage, bandwidth, and performance.

const form = formidable({
  // other config options
  maxFileSize: 1024 * 1024 * 10, // 10mb
});

(formidable defaults to 200mb max file size)

File Storage Location

Quarantine files away from running application and with limited permissions.

const form = formidable({
  // other config options
  uploadDir: './uploads',
});

(formidable defaults to OS’s temp folder)

Extension Validation **

Only allow files with allowed extensions into your system.

const form = formidable({
  // other config options
  filter(file) {
    const allowed = /\.(jpe?g|png|gif|avif|webp|svg)$/i;
    return allowed.test(file.originalFilename)
  }
});

Content-Type Validation **

Only allow files with allowed MIME-types into your system.

const form = formidable({
  // other config options
  filter(file) {
    const mimetype = file.mimetype ?? '';
    return Boolean(mimetype.includes('image'));
  }
});

(formidable bases file.mimetype from file extension)

File Content Validation **

Scanning files for malware is one of the more important security steps you can take when accepting file uploads.

• The file must already be on our server to scan.
• Scans take too long for the reqquest/response cycle.
• This requires a whole new architecture.

Malware Scanning Architecture

1. Store files in quatantine
2. Save file metadata and scan status in DB
3. Respond to user with upload pending info
4. Allow users to view file metadata
5. Background process scans pending files
6. Clean files get updated in DB
7. Decided on strategy for dirty files

App & API Protector

Akamai customers have access to App & API Protector which provides:

• IP/Geo Firewall
• Denial of Service (DoS) protection
• Web Application Firewall (WAF)
• Malware Protection

App & API Protector Review

We’ve secured our files

Relevant Akamai Products

• VPS - Host your application
• Object Storage - Store uploads cheaply and reliably
• CDN - Deliver files faster
• Malware Protection - Pretty obvious, no?

This is the last slide (stop clicking)

Let’s connect