What could be causing the sluggish performance of this particular MongoDB query?

I'm currently grappling with how to manage a particular situation that will eventually involve date ranges.

db.article_raw.count({
   "date": {$gt:ISODate("2015-07-08T00:00:00.000Z")},
   "searchTerms.term":"iPhone"
})

Despite knowing that my indexes are somewhat redundant, I have the following three in place as I work through this issue:

{
   "date" : 1,
   "searchTerms.term" : 1
}

{
    "date" : 1
}

{
    "searchTerms.term" : 1
}

This is how the data is structured:

{
    title: "a cool title",
    date: ISODate("2015-07-09T11:58:36.000Z"),
    "searchTerms" : [ 
        {
            "term" : "According to Jim",
            "relevance" : "0.315"
        }, 
        {
            "term" : "iPhone",
            "relevance" : "0.057"
        }
   }
}

Finally, here is the breakdown of the explain() function on the find() version of this query:

{
    "cursor" : "BtreeCursor date & search",
    "isMultiKey" : true,
    "n" : 275,
    "nscannedObjects" : 275,
    "nscanned" : 11022,
    "nscannedObjectsAllPlans" : 16142,
    "nscannedAllPlans" : 26889,
    "scanAndOrder" : false,
    "indexOnly" : false,
    "nYields" : 1074,
    "nChunkSkips" : 0,
    "millis" : 59548,
    "indexBounds" : {
        "date" : [ 
            [ 
                ISODate("2015-07-08T00:00:00.000Z"), 
                Date(9223372036854775807)
            ]
        ],
        "searchTerms.term" : [ 
            [ 
                "iPhone", 
                "iPhone"
            ]
        ]
    },
    "server" : "...",
    "filterSet" : false,
    "stats" : {
        "type" : "FETCH",
        "works" : 11023,
        "yields" : 1074,
        "unyields" : 1074,
        "invalidates" : 90,
        "advanced" : 275,
        "needTime" : 10746,
        "needFetch" : 1,
        "isEOF" : 1,
        "alreadyHasObj" : 0,
        "forcedFetches" : 0,
        "matchTested" : 0,
        "children" : [ 
            {
                "type" : "IXSCAN",
                "works" : 11022,
                "yields" : 1074,
                "unyields" : 1074,
                "invalidates" : 90,
                "advanced" : 275,
                "needTime" : 10746,
                "needFetch" : 0,
                "isEOF" : 1,
                "keyPattern" : "{ date: 1, searchTerms.term: 1 }",
                "isMultiKey" : 1,
                "boundsVerbose" : "field #0['date']: (new Date(1436313600000), new Date(9223372036854775807)], field #1['searchTerms.term']: [\"iPhone\", \"iPhone\"]",
                "yieldMovedCursor" : 0,
                "dupsTested" : 275,
                "dupsDropped" : 0,
                "seenInvalidated" : 0,
                "matchTested" : 0,
                "keysExamined" : 11022,
                "children" : []
            }
        ]
    }
}

Given the number of articles in the system (nearly a million) and the fact that the count result is only around 250, it's puzzling to me why this query is taking 80 seconds to execute if my indexes are set up correctly.

javascriptperformancemongodboptimizationmongodb-query

Answer №1

Sean, when dealing with a large dataset, it is common to encounter this behavior due to the high cardinality when an index is created on a date field. MongoDB utilizes B-Tree indexes, and if the date column has a multitude of unique values (such as when it includes milliseconds), the index may not provide much benefit. To optimize performance, consider adding an additional column with just the date part to apply the index on and use that in your query criteria, especially when searching within specific date ranges.

Best regards,

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Utilize ExpressJS app.use to enable middleware functionality

As I delve into learning ExpressJS, my attention was drawn to a particular code snippet that has left me puzzled. The function app.use is perplexing me and the documentation isn't providing clear insight. Could someone shed some light on what exactly ...

javascriptnode.jsexpress

getStaticProps only runs on IIS after the entire page is refreshed

Using the nextjs getStaticProps feature in my project has been smooth sailing so far. However, after uploading the Static files to IIS, the feature seemed to stop working until I configured a urlRewrite module on it. I noticed that when initially visiting ...

javascriptiisnext.jsurl-rewriting

Performing MongoDB aggregation across multiple collections while populating the matches

In order to create an aggregation that can work across multiple collections, populate data, and execute search queries, I need to work with 3 collections: { "_id" : ObjectId("5d64d2bf48dd17387d77d27a"), "name" : "Rework_1", "kind" : 1, "or ...

mongodbaggregation-framework

Storing an image as an encoded string in MongoDB: Step-by-step guide

Currently, my goal is to transform an image into a string and store it in MongoDB. Additionally, I would like the ability to decode it at a later time. My approach involves solely using Express, MongoDB, and ReactJS. I specifically do not want to upload t ...

javascriptreactjsmongodbexpress

What steps can I take to make my search button work with this JavaScript code?

I am struggling with integrating my custom search bar with a JavaScript code to make it functional. Here is the code snippet for the Search Button: document.getElementById('searchform').onsubmit = function() { window.location = 'http:/ ...

javascripthtmlgoogle-search

When attempting to access the Angular app in Edge, it automatically redirects to open in IE 11 instead

I have encountered an issue with my angular 5 App. It works perfectly fine in Chrome and Firefox, but when I try to open it in Microsoft Edge on Windows 10, the application always opens in the IE 11 browser. There are no errors displayed on the console. W ...

javascriptangularinternet-explorer-11microsoft-edge

Unable to access Vue component method beyond its scope

Having an issue with my Vue component that I'm trying to call a method from outside its wrapper element #app. Is there a way to register the component so I can easily call it like Component.function(); var viewController = new Vue({ el: "#app", ...

javascriptvue.js

Is invalid HTML causing issues with Backbone.js templating in Phonegap?

Summary: When my template file contains valid HTML code, it does not display in Phonegap. The structure of my index.html file is as follows: <body onload="onBodyLoad()"> <div data-role="page"> <a href="#login" id="clickme">C ...

javascriptcordovabackbone.js

Navigate to the middle of a DIV container in Angular 7

Is there a way to programmatically scroll to the center of my element on both the Y and X axes when a specific function is executed? My HTML structure includes the following (I am aiming to scroll to the middle of #viewport): </div> <div # ...

javascripthtmlcssangular

Adding several lines of HTML content from an object using jQuery's append method

Currently, this is what I have: $(document).ready(function() { $.ajax({ type:"GET" , url:'{{url("/api/getcart")}}' , dataType:"json", success: function(data){ $('#price').append(data.cartitems[1].product.pr ...

javascriptjqueryhtmlajax

What steps can be taken to customize the default keyboard shortcuts functionality in Swiper.js?

I am trying to customize the functionality for left/right keys in Swiper.js but I am unable to find a way to do this through the API () It seems that the API only allows you to disable/enable default actions: mySwiper.keyboard.enabled // Whether th ...

javascriptswiper.js

Having trouble with your Node.js application? Specifically with the Ajax and MongoDB CRUD operations running slowly?

It's taking a few seconds for my database updates to reflect on the page when I add or delete entries. Even with AJAX, the process is still slow. As a newbie in NODE JS transitioning from MVC NET, I can't seem to pinpoint the issue. Below is the ...

node.jsajaxmongodbexpressejs

Exploring the DOM: A Guide to Observing the Present State with React Testing Library

Currently, I am delving into React Testing Library, drawing from my extensive TDD experience in various programming languages. The documentation for React Testing Library mentions that if getByText fails, it will "print the state of your DOM under test" h ...

javascriptreactjsreact-testing-libraryreact-testing

Tips for extracting data from arrays with multiple dimensions - (JavaScript)

I am looking to extract string values from a multi-dimensional array and store them in a new array as the answer. For Example : var dimStr = [ "First-one", ["Double-one", "Double-two", "Double-three", ["Triple-one", "Triple-two"] ], "First-two" ] ; The ...

javascriptarraysmultidimensional-arraylogic

performing a query on two tables with sequelize

There must be a more efficient way to accomplish this task with fewer lines of code. I am not very experienced with databases and I am new to sequelize and node. The user id is passed as a parameter and I need to check if there is a corresponding user in ...

javascriptnode.jssequelize.js

Showcase multiple examples of three.js on a single webpage

I need to showcase several 3D objects on my web app within different containers. Currently, I'm creating multiple three.js renderers, each for a separate container. However, I encountered an error message: "WARNING: Too many active WebGL contexts. Old ...

javascriptthree.jswebgl

Confirm that the contents of two Mongodb collections are the same

Is there a simple way to find the equivalent of taking the shasum of the contents of two files? I'm not interested in comparing every item using an eval function as suggested in this post: How to compare 2 mongodb collections? I assume that Mongodb& ...

mongodb

Enhancing web pages using PHP method progress tracking

I am in the process of creating a basic import wizard and I would like to incorporate a notification feature that displays the progress of each method. For example, I have the following methods: public function mainFunction() { $this->firstStep(); ...

javascriptphpajax

The parameters used in the json.parse function in javascript may be difficult to interpret

Currently, I am examining a JavaScript file on this website. It contains the following code: let source = fs.readFileSync("contracts.json"); let contracts = JSON.parse(source)["contracts"]; I'm curious about what exactly the JSON.parse function is d ...

javascriptjson

React default class name ... Before conditional operator takes effect

import './App.css' import {Button} from "antd" <Button shape="round" className={ istrue === "itstrue" ? "trueclass" : "trueclass active" } > TEXT </Button> ...

javascriptcssreactjsantd