Currently, I am in the process of designing a MongoDB database to interact with a script that periodically polls a resource and stores the response in the database. The existing structure includes a collection with four fields: id, name, timestamp, and data.
My goal is to identify which names experienced changes in the data field between script runs and which did not. Here's a pseudocode representation:
if(data[name][timestamp]==data[name][timestamp+1]) //data has not changed
store data in collection 1
else //data has changed between script runs for this name
store data in collection 2
I am seeking a query solution that can achieve this without needing to iterate through and execute JavaScript logic on each item within the collection. With millions of documents present, such an approach would be inefficient.
Would it be beneficial to create a new collection named timestamp each time the script runs? Could this potentially improve speed and organization? Are there alternative schema designs that could prove more effective?
The script operates once daily so namespace limitations are not expected to pose an immediate concern.