Using MongoDB to find a specific element in an array and then make updates

I have performed aggregation to calculate the total and the objects that contribute to the total.

Now, I need to update the source table with the aggregated object ID for the elements involved in the aggregation, establishing a two-way relationship.

coll.aggregate([
    { "$match": {"elig": 1, "nid" : null, "cncl" : null  } },
    { "$group": {
        "_id": "$nkey",
        "cumqty": {"$sum": "$pr_qty.qty" },
        "netted" : { "$push" : "$_id" } 
    }},
    { "$project": {
        "nkey":"$nkey" ,
        "cumqty": "$cumqty",
        "netted" : "$netted" ,
        "_id" : 0 
    }},
    { "$out": aggcollnm }
])

The aggregated table now contains a list of object IDs created with $push.

For example, let's say doc1, doc2, and doc3 have contributed to the creation of agg1, and agg1 includes doc1, doc2, and doc3 in its list. I want doc1, doc2, and doc3 to have the ID of agg1 as their nettid.

So, I implemented the following solution:

coll.find().forEach( function(elem) {
    coll.update (
        { "_id" : elem._id },
        { "$set" : { nid : aggcoll.aggregate ( [
            { "$unwind" : "$netted" } ,
            { "$match" : { "netted" : elem._id } },
            { "$project" : { "_id" :1 } }
            ] )._firstBatch[0]
        }}
    )
})

This solution worked well with a smaller dataset, but for 1 million documents, it fails with the following error:

2014-06-30T09:48:40.577+0100 Error: getMore: cursor didn't exist on server, possible restart or timeout? at src/mongo/shell/query.js:116 failed to load: ./netting.js

Is there a more efficient way to achieve this task?

Answer №1

Operating on MongoDB 2.6 means that there are more efficient ways to update and your processing methods seem to be backwards. It is recommended to loop through your "aggcoll" and update your target from within:

var batch = coll.initializeOrderedBulkOp();
counter = 0;

aggcoll.find().forEach(function(agg) {
    batch.find({ "_id": { "$in": agg.netted }}).update({ "$set: { "nid": agg._id } });
    counter++;

    if ( counter % 1000 == 0 ) {
        batch.execute();
        counter = 0;
        batch = coll.initializeOrderedBulkOp();
    }
});

if ( counter > 0 )
    batch.execute();

Your previous use of an "inline" aggregate statement was not efficient and would have slowed down the process. The use of bulk operations API reduces server traffic and processing time.

It's unclear why you're updating this way when you should already have the necessary "related" information. Review your original aggregate query:

{ "$group": {
    "_id": "$nkey",

You changed the new collection's primary key from the original "_id" key, even though it was present in all your sourced documents. It would have been more straightforward to leave it as the primary key.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Search for data within both the main document and any sub-documents within a

I am facing a challenge with querying two collections called Orders and Books simultaneously. I need to search for a specific string value in Order (code) and Book (name, number) fields at the same time. The objective is to find any matching values and ret ...

sending functions into angular as opposed to using 'function()'

Lately, I've been immersing myself in Angular development. One thing that caught my interest was the idea of using a declared function instead of a generic "function() {}" placeholder, particularly in scenarios like handling promise callbacks. I encou ...

Tips for extracting the most deeply nested object in a JSON file using JavaScript

Is it possible to access the innermost object without knowing the path names? Consider this JSON example: const data = { first: { second: { third: {innerObject} } } ...

Do we need to use aria-labelledby if we already have label and input associated with "for" and "id"?

Here is the HTML structure for the Text Field component from Adobe Spectrum: The <label> element has a for attribute and the <input> has an id which allows screen readers to read out the label when the input is focused. So, why is aria-label ...

Tips for incorporating MUI into your Redwood JS project

Trying to integrate MUI into Redwood JS has been a challenge for me. I attempted to run the following command in the project directory: yarn add @mui/material Unfortunately, an error message appeared in the console stating: An error Running this command w ...

A comprehensive guide on using Lua Script in Node.js to efficiently insert multiple records into a Redis Hash

How can I efficiently insert multiple records into a Redis Hash using Lua Script in Node.js? I currently have the following code which utilizes multi and exec for inserting data. How can I modify it to use a lua script instead? return new Promise ...

How can I retrieve values of selected checkboxes using the Express Data API?

I have a scenario where I need to retrieve data from input checkboxes only when the checkbox for my express post function is selected. There are 3 checkboxes with values of 1000, 2000, and 3000 as follows: <input type="checkbox" name=" ...

How to troubleshoot Props not functioning in nextjs-typescript?

I'm having trouble with props in my project and I can't seem to figure it out! Here are the three files. I'm still learning typescript but everything seems fine in the code, yet it's not working! Here is index.tsx file: const Home: ...

Exploring Jasmine's Powerful Spying and Mocking Capabilities in JavaScript Prototypes

Hey everyone, I need some help with a JavaScript issue. So, I have a file named FileA.js which contains a prototype called FileAObject.prototype along with a function named funcAlpha(). Here's a snippet of what it looks like: File = FileA function s ...

Having difficulty with loading images lazily in a jQuery Mobile app with LazyLoadXT feature

Struggling to incorporate lazy loading in my jQM app with Lazy Load XT v1.0.6. Oddly, images only appear when switching browser tabs, not while scrolling down. This happens on Firefox and Chrome. <img src="/img/default-img.jpg" data-src="/img/product/ ...

Guide to changing a 10-digit number field to a string field in MongoDB

Looking to change the mobile field to a string in MongoDB. { "_id": "1373b7723", "firstname": "name1", "mobile":1000000099 }, { "_id": "137be30723", "firstname": "name2&qu ...

Emulating a mouse click in jQuery/JavaScript on a webpage link

I am seeking a way to programmatically trigger a click on any link within a webpage using JavaScript. The challenge lies in ensuring that if the link has an 'onclick' event bound to it by another unknown JavaScript function, that event is trigger ...

Generating a fresh instance from a pre-existing object using JavaScript

Currently, I am facing a challenge from devchallenges.io known as the Shoppingify challenge. After carefully reviewing the prompt, I started working on creating a model that should have a specific format when a request is submitted. { "user": 1 ...

What could be causing my if-else statement to malfunction in JavaScript?

I'm working on a form where certain conditions need to be met based on the selected order type. For instance, if a market order is chosen, the stop price and limit price should default to zero and become read-only fields. Similarly, selecting a limit ...

executing a Prisma database migration with various schemas

I am currently immersed in a Prisma project where my goal is to create a node module that can be utilized by other projects. The challenge now is to ensure that the database stays synchronized with the models and the primary project, so all testing platfor ...

Achieving a collapsing navbar on click in Bootstrap 5

Is there a way to collapse this navigation bar after clicking on a link, without using a JavaScript event listener or the data-bs-toggle and data-bs-target methods mentioned in this article? I tried both methods but they are not working with my code. Here ...

Encountered an issue loading a resource due to a lost network connection while using Safari 9 and JBoss WildFly 8.2

After successfully deploying my War file to the JBoss Wildfly 8.2 server, I attempted to access the application link from a remote MAC machine. The application opened correctly, but some functionalities were not working properly. An error message popped u ...

What is the method to create a resizable table column border rather than resizing the bottom corner border?

At the moment, we are able to resize the table border when we drag the corner. However, my goal is to enable resizing on the right side of the full border. https://i.sstatic.net/yFI24.png Below is the CSS code for resizing: th{ resize: horizontal; ove ...

Webpack compatibility issue hindering big.js node module functionality

I'm currently working on compiling (typescript files) and bundling my source code using webpack. Below is the content of my webpack.config.js file: const path = require('path') module.exports = { devtool: 'eval-source-map', en ...

Adjust the Scope in Angular-Charts.js Post-Rendering

I am currently facing a challenge with displaying multiple charts using the angular-charts.js framework. The issue is that I require all the charts to have the same scale, but each chart currently has its own scale based on the displayed data. Unfortunatel ...