The MongoDB GridFS is refusing to accept the buffer being written

Hey everyone, I've been working on this issue for nearly a day now and can't seem to figure it out. I'm utilizing multer's inMemory flag to upload an image from my website. My approach involves writing the buffer received from multer to GridFs (specifically GridStore).

Here is the code snippet for my multer:

product.js (route/controller)

    var DB = rek('database');
    var router = express.Router();
    var multer = require("multer");

    var uploadOptions = {inMemory:true, onFileUploadComplete: uploadDone}

    router.post('/product/:productId/images/', multer(uploadOptions), uploadImageResponse);

    function uploadDone(file) {
        var options = {
            name:file.name,
            mode:"w+",
            content_type:file.mimetype,
            w: "majority",
            chunkSize: 1024,
            metadata:file
        };

        var GridStore = new DB.GridStore(DB.connection, file.name, "w+", options);
        GridStore.open(function(err, GS){
            if(err) throw err;
            return GS.write(file.buffer, function(err, GS){
                console.log("file written");
                if(err) throw err;
                return GS.close(function(err, result){
                    if(err) throw err
                    console.log(result);
                })

            })
        });
    }

Now you may be wondering where DB object comes from. Well, I simply add it to the mongoose object during initialization. Here is what the code looks like:

database.js

var mongoose = require("mongoose"),
    mongooseTimestamps = require("mongoose-concrete-timestamps"),
    autoIncrement = require("mongoose-auto-increment"),
    config = require("../config"),
    Grid = require("gridfs-stream");

mongoose.connect( config.database['development'].url + "" + config.database['development'].name );
var db = mongoose.connection;
db.once("open", function(err){
    if(err) throw err
    mongoose.GridStore = mongoose.mongo.GridStore
})
db.on("error",function(errMsg){
    console.log("Error Connecting to Mongo: " + errMsg);
});
mongoose.set('debug', true);

mongoose.plugin(mongooseTimestamps);
autoIncrement.initialize(db);
module.exports = mongoose;

I've tried various approaches but keep getting no writes and no errors. Even the mongoose output log shows:

POST /product/1000/images 200 4.494 ms - 22
Mongoose: fs.chunks.ensureIndex([ [ 'files_id', 1 ], [ 'n', 1 ] ]) { w: 'majority' }  
Mongoose: fs.files.find({ filename: '2b08f506ed277eda45f9fc400c098aa1.jpg' }) { readPreference: 'primary', w: 'majority' }  
Mongoose: fs.chunks.find({ n: 0, files_id: ObjectId("54bb87aaabf2c0416a50c068") }) { readPreference: 'primary', w: 'majority' }

I'm puzzled by why it's performing a find operation when I'm trying to insert/write to GridFS. So far, I only get as far as the call to GridStore.open before the write never happens and no error is thrown.

Here are the things I've attempted so far:

  1. Using Multer's InMemory - same result
  2. Using Multer's dest attribute and piping the stream from fs to gridfs - same result
  3. Using the gridfs-stream module - Same result
  4. Used the Native GridFS/GridStoire - Same result.

Any assistance would be greatly appreciated.

Answer №1

In this scenario, it's important to understand that the "buffer" in the "inMemory" option doesn't function as an exclusive choice and doesn't imply that the data is solely held in memory. Instead, it serves as a duplicate of the data which is also stored in a temporary file on disk.

Whether you specify "inMemory" or not, files will still be generated (typically in the /tmp directory) but they will be automatically deleted when no longer needed:

var async = require('async'),
    express = require('express'),
    multer = require('multer'),
    fs = require('fs'),
    mongoose = require('mongoose'),
    Grid = require('gridfs-stream'),
    Schema = mongoose.Schema;

It's worth noting that using the "gridfs-stream" package simplifies content uploads, and relying on a buffer copy for performance optimization may not yield significant benefits considering the persistent IO costs and storage requirements involved.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Using jQuery to serialize parameters for AJAX requests

I could use some help figuring out how to set up parameters for a $.ajax submission. Currently, I have multiple pairs of HTML inputs (i pairs): > <input type="hidden" value="31" name="product_id"> <input > type="hidden" value="3" name="qua ...

In Javascript, a variable is being defined by combining a string with another variable through concatenation

Hey, I'm relatively new to working with Javascript and am facing an issue that I believe can be resolved easily with the right syntax. The problem lies in setting a variable (totalRow1) where it needs to be concatenated with a string and another vari ...

Ways to retrieve the identifiers of every child node UL element within a UL container

I am struggling with a basic question related to HTML structure. Here is the code snippet I am working with: <ul> <li> <ul class=t2 id=15> <li class='item'>a<span class='val'>b</ ...

validation of dialog forms

Is there a way to trigger the $("#form").validated() function from the ok button in a jquery-ui-dialog? Please note that I would prefer not to use a submit button within the form. ...

Learn how to retrieve URL parameters using HTML code

I would like to update the URL without refreshing the page as I am using an AJAX function to refresh a specific div. The issue I am encountering is that when the div is reloaded via AJAX, it does not recognize the URL parameter. Below is my code (I have a ...

Query the Firebase database in Angular2 to locate the latitude and longitude values that are nearest to the user's current coordinates

I am working with a database table named locations, which contains a list of places along with their corresponding lat/long coordinates. Additionally, I am utilizing geolocation to retrieve the user's current lat/long. My goal is to identify the loc ...

Analyzing the value of a tab with Protractor测试

Below is my HTML code showcasing a list of tabs: <mat-tab-group> <mat-tab label="A"> <app-A></app-A> </mat-tab> <mat-tab label="B"> <app-B></app-B> </mat ...

The 'fn' argument passed is not a valid function, instead it is a string

I am encountering an issue while trying to retrieve data from my server. The console doesn't provide any specific information about the error. My objective is to fetch JSON data from the server and use it to display the required information. I am util ...

What is the best way to link assets within an Angular custom element (Web Components)?

After successfully creating a web component and referencing an image from my asset folder, everything was running smoothly on my local environment. However, when I published my custom element to Firebase hosting, I encountered some issues. When trying to ...

Searching for a different method in JavaScript that can add items without duplication, as the prependTo function tends to insert multiple items

Every time my code runs successfully, a success message is generated and displayed using prependTo within the HTML. However, the issue arises when the user performs the successful action twice, resulting in two success messages being shown on the screen. ...

Angular input for date is showing wrong value due to timezone problem

My database stores dates in UTC format: this.eventItem.date: "2023-06-21T00:00:00.000Z" The input form field value is showing '2023-06-20' (incorrect day number), which seems to be a timezone issue. I am located in a region with a +3 o ...

Is your Javascript navigation functioning properly on some submenus while encountering issues on others?

If you visit , you'll notice that it's a visually appealing site. The navigation submenus seem to be functioning properly, HOWEVER upon inspecting the element, you'll notice that under the PRICING tab, there are submenus that have the same c ...

What is the reason that the index is consistently the final index when utilizing it to pass to a function while iterating over an array with map()?

Currently, I am utilizing the map function to iterate over an array object fetched from the server. Each object within the array is used to populate a row in a table for displaying data. I need to perform a specific action for each row by invoking a functi ...

What is the best way to send a parameter to the callback function of a jQuery ajax request?

I am facing an issue where I need to pass additional variables to a jQuery ajax callback function. Consider the following scenario: while (K--) { $.get ( "BaseURL" + K, function (zData, K) {ProcessData (zData, K); } ); } func ...

How to duplicate a folder in Yeoman without using templates

Currently, I am utilizing yeoman to generate a scaffold for my application. In order to recursively copy all directories, I have opted to utilize the this.directory method. this.directory('views/static/views', '.views/static/views'); ...

The development chrome extension failed to load due to an invalid port or malformed URL pattern

I'm encountering an issue while trying to load my development chrome extension for debugging. The error message I am receiving is: Issue with 'content_scripts[0].matches[0]' value: Path cannot be empty. Manifest failed to load. This is th ...

Animated CSS sidemenu (utilized as a filtering panel for a table)

Hi there, I'm having some trouble with CSS Animation. I recently started developing websites and am using Bootstrap 4 along with Animate.css for animations. My goal is to have an icon button expand sideways to reveal a div containing select elements f ...

Setting up Travis CI for using locally linked packages: A step-by-step guide

When working on myPackage and myDependency simultaneously, I use the command npm link myDependency to avoid having to constantly publish changes to myDependency and update myPackage. I would like to integrate Travis CI with myPackage, but running npm inst ...

How can we nest a div within the outermost parent element?

Consider a scenario where there are 3 different divs named grandParent, Parent, and Child. Although the parent's tag is placed inside the grandParent, due to the use of position: absolute; property, the parent ends up being displayed outside the grand ...

Trouble arises when attempting to deploy fresh code on Heroku platform

After updating my backend code and deploying it, I've encountered a strange issue. It seems like the changes are only taking effect in certain cases and not being reflected as expected. The updated API functions correctly when tested with Postman and ...