Utilizing URLs as video sources in the @google/generative AI framework within a Next.js environment

Currently, I am facing an issue while trying to upload videos from a URL using @google/generative-ai in Next.js. While I have successfully learned how to work with videos stored on my local machine, I am struggling to do the same with videos from external sources.

Below is my existing function for uploading videos to @google/generative-ai:

"use server"

const { GoogleGenerativeAI } = require("@google/generative-ai");
import { GoogleAIFileManager, FileState } from "@google/generative-ai/server";
import { redirect } from "next/navigation";
import fetchVideoById from "./fetchVideoById";

// Initialize GoogleAIFileManager with your API_KEY.
const fileManager = new GoogleAIFileManager(process.env.API_KEY);


// Access your API key as an environment variable (see "Set up your API key" above)
const genAI = new GoogleGenerativeAI(process.env.API_KEY);

// Choose a Gemini model.
const model = genAI.getGenerativeModel({
  model: "gemini-1.5-pro",
});

export async function generateSummary(formData) {
  const rows = await fetchVideoById(formData.get("id"))
  const url = rows["url"]

  console.log("Uploading file...")
    const fileManager = new GoogleAIFileManager(process.env.API_KEY);

    // Upload the file and specify a display name.
    const uploadResponse = await fileManager.uploadFile(url, {
      mimeType: "video/mp4",
      displayName: rows["title"],
    });

    // View the response.
    console.log(`Uploaded file ${uploadResponse.file.displayName} as: ${uploadResponse.file.uri}`);
    const name = uploadResponse.file.name;

    // Poll getFile() on a set interval (10 seconds here) to check file state.
    let file = await fileManager.getFile(name);
    while (file.state === FileState.PROCESSING) {
      process.stdout.write(".")
      // Fetch the file from the API again
      file = await fileManager.getFile(name)
    }

    if (file.state === FileState.FAILED) {
      throw new Error("Video processing failed.");
    }

    // When file.state is ACTIVE, the file is ready to be used for inference.
    console.log(`File ${file.displayName} is ready for inference as ${file.uri}`);

    const result = await model.generateContent([
      {
        fileData: {
          mimeType: uploadResponse.file.mimeType,
          fileUri: uploadResponse.file.uri
        }
      },
      { text: "Summarize this video." },
    ]);

  // Handle the response of generated text
  console.log(result.response.text())

  return result.response.text()

  console.log("Deleting file...")

  await fileManager.deleteFile(file.name);

  console.log("Deleted file.")
  
}

The error message I encounter is:

Error: ENOENT: no such file or directory, open 'C:\Users\n_mac\Desktop\Coding\summa\front-end\https:\m9x5emw6q3oaze3r.public.blob.vercel-storage.com\monkeyman64\6999DBC5-2D93-4220-BC43-3C16C9A5D9C6-IZzFC1THZXPSgeAK1NPo3uCVxA091l.mp4'

It seems that the system is searching for the file on my local machine instead of Vercel Blob where the files are actually stored. Any assistance regarding this matter would be highly appreciated.

Answer №1

Upon reviewing the uploadFile script, it appeared that the file was being uploaded as multipart.Source Furthermore, it is evident that the filePath must be the file path of the local PC. Unfortunately, at this stage, direct usage of the URL seems unattainable.

Considering this, why not try downloading data from the URL and uploading it to Gemini without creating a file? How about implementing the following modifications in the script?

Original Code:

const fileManager = new GoogleAIFileManager(process.env.API_KEY);

// Uploading the file with specifying a display name.
const uploadResponse = await fileManager.uploadFile(url, {
  mimeType: "video/mp4",
  displayName: rows["title"],
});

// Displaying the response.
console.log(`Uploaded file ${uploadResponse.file.displayName} as: ${uploadResponse.file.uri}`);
const name = uploadResponse.file.name;

Modified Code:

const fileManager = new GoogleAIFileManager(process.env.API_KEY);


// Data Download.
const res = await fetch(url);
const buffer = await res.arrayBuffer();

// Uploading the downloaded data.
const formData = new FormData();
const metadata = { file: { mimeType: "video/mp4", displayName: rows["title"] } };
formData.append("metadata", new Blob([JSON.stringify(metadata)], { contentType: "application/json" }));
formData.append("file", new Blob([buffer], { type: "video/mp4" }));
const res2 = await fetch(
  `https://generativelanguage.googleapis.com/upload/v1beta/files?uploadType=multipart&key=${fileManager.apiKey}`,
  { method: "post", body: formData }
);
const uploadResponse = await res2.json();


// Displaying the response.
console.log(`Uploaded file ${uploadResponse.file.displayName} as: ${uploadResponse.file.uri}`);
const name = uploadResponse.file.name;
  • This modification utilizes the Node.js fetch API.
  • A successful run of this script verified that MP4 data from the URL can indeed be uploaded to Gemini.

Important Note:

  • This modification assumes the validity of your URL for directly downloading MP4 data. Exercise caution in this regard.

Further Reference:

  • Method: media.upload

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

What are some examples of MySQL node insertion queries with a pair of values?

I need help with my current JavaScript code snippet. var connection = mysql.createConnection({ host: 'localhost', user: 'root', password: 'root', database: 'codify', port: '8889' } ...

The error message "Type 'string' cannot be assigned to type 'Condition<UserObj>' while attempting to create a mongoose query by ID" is indicating a type mismatch issue

One of the API routes in Next has been causing some issues. Here is the code: import {NextApiRequest, NextApiResponse} from "next"; import dbConnect from "../../utils/dbConnect"; import {UserModel} from "../../models/user"; e ...

Refresh your attention on the Vue single page application

I am building a Vue SPA with the Vue router and I am attempting to simulate a page reload by resetting the focus state after changing routes. The following is my code snippet: router.beforeEach((to, from, next) => { document.activeElement.blur(); ne ...

deciphering recursive collection of JSON objects in Angular

I'm currently working with an angular component that holds an array of nested JSON objects. My goal is to use a service to load these nested JSONs into separate objects within an array so I can easily look them up by their ID. I'm wondering if t ...

Move upwards and move downwards

I have a list of groups with names. <ul id="groups" > <li id="group1" ></li> <li id="group2" ></li> <li id="group3"></li> </ul> Additionally, I have sliding containers. <div id="containers" > ...

Take action once all deferred operations have been successfully completed

Below is the code snippet I am working with: loadOpportunities: function () { return $.getJSON("/Opportunity/GetAll").pipe(function (result) { //do stuff }); }, loadTypes: function () { return $.getJSON("/OpportunityTypes/GetAll", nul ...

Automate your Excel tasks with Office Scripts: Calculate the total of values in a column depending on the criteria in another column

As a newcomer to TypeScript, I have set a goal for today - to calculate the total sum of cell values in one column of an Excel file based on values from another column. In my Excel spreadsheet, the calendar weeks are listed in column U and their correspon ...

Is there a way to simulate a KeyboardEvent (DOM_VK_UP) that the browser will process as if it were actually pressed by the user?

Take a look at this code snippet inspired by this solution. <head> <meta charset="UTF-8"> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> </head> <body> <script> $(this). ...

Combine the elements of an array to form a cohesive string

As a newcomer to Javascript, I am feeling a bit puzzled about this conversion. var string = ['a','b','c']; into 'a','b','c' ...

Store user inputs from HTML forms into database using Javascript

I'm looking for assistance on how to store this javascript/html form data in a database. My expertise lies in connecting html/php with SQL, and I need guidance on integrating this form submission process. Below is the code snippet that enables users ...

Exploring the process of performing an AJAX JQuery HTTP request using JavaScript and PHP on the server side - any tips?

Greetings! I have developed a web application using HTML, CSS, and JavaScript. To enhance functionality, I have integrated Bootstrap and jQuery into the project. The application comprises both client-side and server-side components. Let's take a look ...

Callback issues in need of resolving

This problem seems like it should have a simple solution, but I think I've been staring at it for too long. Initially, I had this function inline, but now I want to extract it and use it in other parts of my application. However, I'm struggling ...

How to transform an array into a collection of objects using React

In my React project, I encountered the following data structure (object of objects with keys): const listItems = { 1:{ id: 1, depth: 0, children: [2,5] }, 2:{ id: 2, depth: 1, children: [3,4], parentIndex: 1, disable ...

Show off a sleek slider within a Bootstrap dropdown menu

Is there a way to display a sleek slider within a Bootstrap dropdown element? The issue arises when the slider fails to function if the dropdown is not open from the start, and the prev/next buttons do not respond correctly. For reference, here is my curr ...

What is the best way to utilize JSONP to display an entire HTML code with an alert?

When I attempt to use cross-domain ajax to retrieve the entire HTML page code, I keep receiving a failed message. The console log shows: "Uncaught SyntaxError: Unexpected token <" Below is my AJAX function: function fetchData(){ var url = documen ...

Dynamic data binding in AngularJS with dynamically generated views

The scenario: Currently, I am constructing a wizard in AngularJS that contains tabbed content with each tab representing a step in the wizard. These steps are fetched from the database through a Laravel controller and displayed using ng-repeat. Each tab c ...

Sizing of Info Windows for Google Map Markers

I've been trying to create a Google map displaying multiple locations, but I can't seem to figure out how to adjust the size of the infowindow for each marker. I've spent several hours on this with no luck. Can anyone show me where in this f ...

Steps for fixing Eslint error during the compilation of Next.js framework

In the process of executing the build command npm run build in Next.js, an Eslint error was triggered. info - Checking validity of types warn - The Next.js plugin was not detected in your ESLint configuration. See https://nextjs.org/docs/basic-features/e ...

Determine the number of lines present in a textarea

Is there a way to use JavaScript to determine the number of lines in a textarea without relying on the rows attribute? For example, in this scenario, I would expect to get 4 lines. <textarea rows="4">Long text here foo bar lorem ipsum Long text he ...

There was an error in locating the JavaScript file within the Node.js Express Handlebars application

My website is not displaying the .js file, css file, or image from the public folder. Here are the errors showing up in the console: GET http://localhost:8080/assets/images/burger.jpg 404 (Not Found) GET http://localhost:8080/burgers.js net::ERR_ABORTED ...