The process of uploading an image to S3 from a form submitted by a user appears to be finishing successfully

Question

The process of uploading an image to S3 from a form submitted by a user appears to be finishing successfully

Currently, I am working on a Remix app where I have implemented a form to select and upload an image using a simple

<input name="image" type="file" />

The form sends a POST request to a Remix handler. In the handler (refer to the code below), I am attempting to upload this image to Amazon S3 using the v3 SDK. I have experimented with both the PutObjectCommand from client-s3 and the Upload method from lib-storage, but encountered the same outcome...

The upload process seems to be successful as it creates the file in the bucket without any issues. The file size matches exactly what I expect when compared to uploading the file directly into the S3 bucket via the web UI.

However, when trying to open the uploaded file as an "image", it does not display correctly. When viewing the image from the S3 bucket, the browser shows the standard "broken image icon" (similar issue occurs if attempting to open the image in Preview on OSX) - unfortunately, there isn't much information available about what might be wrong. It appears that something may be corrupted or created in a manner that prevents the file from being recognized as an image.

Within the handler, the file data is received as an AsyncIterable<Uint8Array>. My main processing task involves converting this data into a single Uint8Array for the Upload operation (I have already tried setting the body to both buffer and blob with no success).

Could there be an issue with my conversion to a Uint8Array? Is there a more appropriate way to perform this conversion for the upload operation, perhaps using a different data type?

Alternatively, is there a specific configuration that needs to be adjusted regarding how the upload to S3 is being set up?

async ({ name, contentType, data, filename }) => {
  const arrayOfUInt8Array: Uint8Array[] = [];

  let length = 0;
  for await (const x of data)
  {
    arrayOfUInt8Array.push(x);
    length += x.length;
  }

  const uInt8Array = new Uint8Array(length);

  for (const x of arrayOfUInt8Array) {
    uInt8Array.set(x);
  }

  // Tried creating a buffer from the Uint8Array...
  const buff = Buffer.from(uInt8Array);
  // Tried creating a blob from the Uint8Array...
  const blob = new Blob([uInt8Array], { type: contentType });

  const uploadCommand = new Upload({
    client: s3Client,
    params: {
      Bucket: s3Bucket,
      Key: filename,
      Body: uInt8Array,
      ContentType: contentType,
    }
  });

  await uploadCommand.done();

  return '';
},

javascript amazon-s3 image-upload uint8array

Answer 1

Answer №1

According to @Botje's comments, the problem stemmed from how the Uint8Array was constructed. The issue was that the source was being constantly overwritten at the beginning, leaving the rest of the array empty.

Instead of using the original code:

  for (const x of arrayOfUInt8Array) {
    uInt8Array.set(x);
  }

I had to make adjustments and use the following code:

  let i = 0;
  let currentIndex = 0;
  for (const x of arrayOfUInt8Array) {
    uInt8Array.set(x, currentIndex)
    currentIndex += arrayOfUInt8Array[i].length
    i += 1
  }

Answer 2

According to @Botje's comments, the problem stemmed from how the Uint8Array was constructed. The issue was that the source was being constantly overwritten at the beginning, leaving the rest of the array empty.

Instead of using the original code:

  for (const x of arrayOfUInt8Array) {
    uInt8Array.set(x);
  }

I had to make adjustments and use the following code:

  let i = 0;
  let currentIndex = 0;
  for (const x of arrayOfUInt8Array) {
    uInt8Array.set(x, currentIndex)
    currentIndex += arrayOfUInt8Array[i].length
    i += 1
  }

Answer 3

Answer №2

My approach to handling this situation usually involves a two-step strategy:

First, I parse the upload as NodeDiskOnFile using Remix's built-in functionality:

const uploadHandler = unstable_composeUploadHandlers(
    unstable_createFileUploadHandler({
      maxPartSize: 10_000_000,
      file: ({ filename }) => filename,
    }),
    // parse everything else into memory
    unstable_createMemoryUploadHandler()
  );

const formData = await unstable_parseMultipartFormData(
    request,
    uploadHandler
  );

Next, I proceed to upload the files by utilizing presigned URLs:

for (const file of files) {
    const uploadedURL = await uploadFileWithPresignedUrl(file);
    dataWithAWSUrls.uploadedFiles.push(uploadedURL);
  }

Below are the functions that define the structure of presigned URLs:

import {
  GetObjectCommand,
  PutObjectCommand,
  S3Client,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { NodeOnDiskFile } from "@remix-run/node";
import axios from "axios";

import slugify from "slugify";
import { slugifyOptions } from "~/constants/constants";
import { env } from "~/env";

const s3Client = new S3Client({
  region: env.AWS_REGION,
  credentials: {
    accessKeyId: env.AWS_ACCESS_KEY_ID,
    secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
  },
});

async function streamToBuffer(
  readable: ReadableStream<Uint8Array>
): Promise<Buffer> {
  const data: Uint8Array[] = [];
  const reader = readable.getReader();

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      data.push(value);
    }
  } finally {
    reader.releaseLock();
  }

  return Buffer.concat(data.map((chunk) => Buffer.from(chunk)));
}

export async function uploadFileWithPresignedUrl(file: File | NodeOnDiskFile) {
  const buffer = await streamToBuffer(file.stream());

  const slugifiedFileName = slugify(file.name, slugifyOptions);

  const command: any = new PutObjectCommand({
    Bucket: env.AWS_BUCKET_NAME,
    Key: slugifiedFileName,
    Body: buffer,
    ContentType: file.type,
  });

  const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });

  // Upload file with presigned url
  await axios.put(url, buffer, {
    headers: {
      "Content-Type": file.type,
      "Content-Length": buffer.length,
    },
  });

  const publicUrl = `https://${env.AWS_BUCKET_NAME}.s3.amazonaws.com/${slugifiedFileName}`;

  // Return the url of the uploaded file
  return publicUrl;
}

export async function getFileWithPresignedUrl(key: string) {
  const command = new GetObjectCommand({
    Bucket: env.AWS_BUCKET_NAME,
    Key: key,
  });
  const signedUrl = getSignedUrl(s3Client, command, { expiresIn: 3600 });
  return signedUrl;
}

Answer 4

My approach to handling this situation usually involves a two-step strategy:

First, I parse the upload as NodeDiskOnFile using Remix's built-in functionality:

const uploadHandler = unstable_composeUploadHandlers(
    unstable_createFileUploadHandler({
      maxPartSize: 10_000_000,
      file: ({ filename }) => filename,
    }),
    // parse everything else into memory
    unstable_createMemoryUploadHandler()
  );

const formData = await unstable_parseMultipartFormData(
    request,
    uploadHandler
  );

Next, I proceed to upload the files by utilizing presigned URLs:

for (const file of files) {
    const uploadedURL = await uploadFileWithPresignedUrl(file);
    dataWithAWSUrls.uploadedFiles.push(uploadedURL);
  }

Below are the functions that define the structure of presigned URLs:

import {
  GetObjectCommand,
  PutObjectCommand,
  S3Client,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { NodeOnDiskFile } from "@remix-run/node";
import axios from "axios";

import slugify from "slugify";
import { slugifyOptions } from "~/constants/constants";
import { env } from "~/env";

const s3Client = new S3Client({
  region: env.AWS_REGION,
  credentials: {
    accessKeyId: env.AWS_ACCESS_KEY_ID,
    secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
  },
});

async function streamToBuffer(
  readable: ReadableStream<Uint8Array>
): Promise<Buffer> {
  const data: Uint8Array[] = [];
  const reader = readable.getReader();

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      data.push(value);
    }
  } finally {
    reader.releaseLock();
  }

  return Buffer.concat(data.map((chunk) => Buffer.from(chunk)));
}

export async function uploadFileWithPresignedUrl(file: File | NodeOnDiskFile) {
  const buffer = await streamToBuffer(file.stream());

  const slugifiedFileName = slugify(file.name, slugifyOptions);

  const command: any = new PutObjectCommand({
    Bucket: env.AWS_BUCKET_NAME,
    Key: slugifiedFileName,
    Body: buffer,
    ContentType: file.type,
  });

  const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });

  // Upload file with presigned url
  await axios.put(url, buffer, {
    headers: {
      "Content-Type": file.type,
      "Content-Length": buffer.length,
    },
  });

  const publicUrl = `https://${env.AWS_BUCKET_NAME}.s3.amazonaws.com/${slugifiedFileName}`;

  // Return the url of the uploaded file
  return publicUrl;
}

export async function getFileWithPresignedUrl(key: string) {
  const command = new GetObjectCommand({
    Bucket: env.AWS_BUCKET_NAME,
    Key: key,
  });
  const signedUrl = getSignedUrl(s3Client, command, { expiresIn: 3600 });
  return signedUrl;
}

The process of uploading an image to S3 from a form submitted by a user appears to be finishing successfully

Answer №1

Answer №2

Similar questions

Building CRUD functionality using the ExpressJS router

Modify the indicator color of the tabs in Material UI v5

Generate a novel item by organizing the key of an array of objects

Displaying items as objects in search results in Kendo Angular's auto complete feature

How can I simulate or manipulate the element's scrollHeight and clientHeight in testing scenarios?

Exploring Raycasting with Three.js and WhitestormJS

Having trouble with passing post data from jQuery ajax to a php page?

Is there a simple angular directive available for displaying expandable/collapsible text?

Adding 2-dimensional text to a specified location with the help of three.js

Troubleshooting Media Query problem in Angular 6

Interact with a Firebase database table using Node.js

Tracking your daily nutrition using cronometer with the powerful combination of JavaScript

Transferring blank documents to a React-Native server

Issue encountered - Attempting to provide an object as input parameter to puppeteer function was unsuccessful

Customize the appearance of the v-autocomplete component in Vuetify

Emphasize the designated drop zone in Dragula

How do I pass a value from a Vue component's prop to an HTML attribute in Vue.js?

The JWT authentication appears to be functioning properly when tested on "postman," however, it is encountering issues on the website

Dividing JSON File Entries Among Several Files

What steps should I take to make my Vue JS delete function operational?