Bringing JSON data into MongoDB using a mongo script

I am currently working on a mongo script to efficiently import a jsonArray from a JSON file. The script I am developing is in .js format and I plan to run it using the load() command within the mongo shell. Is there a feasible method for accomplishing this task with a mongo script?

While I am aware that I could use mongoimport as an alternative solution, my preference lies in finding a way to achieve this through a scripting approach.

Below are the contents of my current script, where the import section is yet to be implemented:

var db = connect("localhost:27017/fypgui");
//Import json data into "crimes" collection goes here 
var crimes = db.crimes.find();
while (crimes.hasNext()){
    var item = crimes.next();
    var year =(item.crime_date != null)?(new Date(item.crime_date)).getFullYear():null;
    db.crimes.update( {_id: item._id}, {$set: {crime_year: year}});
}

Answer №1

There's a different perspective on this issue that I'd like to share, despite its age.

If you're working with the mongo shell, there's a way to handle your JSON data effectively.

Simply convert your JSON to valid JavaScript by adding var myData= before it and then use the load() command in your script. This will allow you to access your data through the myData object.

data.js

var myData=
[
        {
                "letter" : "A"
        },
        {
                "letter" : "B"
        },
        {
                "letter" : "C"
        }
]

read.js

#!/usr/bin/mongo --quiet

// read data

load('data.js');

// display letters

for (i in myData) {
  var doc  = myData[i];
  print(doc.letter);
}

To write JSON data, simplify the process by loading the result into a single object. Start by initializing the object with var result={} and then use printjson() to output the data. Redirect the output to a file using standard methods.

write.js

#!/usr/bin/mongo --quiet

var result=[];

// read data from collection etc...
for (var i=65; i<91; i++) {
  result.push({letter: String.fromCharCode(i)});
}

// output
print("var myData=");
printjson(result);

The shebang lines (#!) are designed for Unix systems but can also be used on Windows with tools like Cygwin.

Answer №2

To retrieve the contents of a file as text, you can utilize the undocumented cat() function within the mongo shell:

var text = cat(filename);

If you want to explore other hidden utilities like cat() and writeFile, take a look at this file: shell_utils_extended.cpp

Once you have the file's content, you have the option to modify it or directly input it into JSON.parse for conversion into a JavaScript object:

jsObj = JSON.parse(text);

Remember: while JSON.parse is powerful, it differs from the functionality of the mongoimport tool in terms of JSON parsing capabilities.

mongoimport excels in parsing Mongo's extended JSON in canonical format. (Canonical format files are typically generated by tools like bsondump and mongodump. For further details on JSON formats, refer to MongoDB extended JSON).

Keep in mind that JSON.parse does not support canonical JSON format. While it will successfully read such input and produce a JavaScript object, any additional data type information specific to canonical format JSON will be disregarded.

Answer №3

The mongo shell does not support reading and writing files like a complete programming environment. It is recommended to use mongoimport or create a script in a language that has an official driver for MongoDB. For example, Node.js has syntax similar to the mongo shell, but it operates on an async/event-driven model. Alternatively, Python with PyMongo can be a good choice due to its simplicity and similarity to MongoDB's query language.

Answer №4

Hey there, I wanted to share a useful tip with you! Are you tired of manually importing json files into your mongo db? I used to struggle with that too until I created a simple batch script to automate the process. Interested in checking it out?

https://github.com/aminjellali/batch/blob/master/mongoImporter.bat

@echo off
title = Mongo Data Base importing tool
goto :main

:import_collection
    echo importing %~2
    set file_name=%~2
    set removed_json=%file_name:.json=%
    mongoimport --db %~1 --collection %removed_json% --file %~2
goto :eof

:loop_over_files_in_current_dir
    for /f %%c in ('dir /b *.json') do call :import_collection %~1 %%c
goto :eof

:main
    IF [%1]==[] (
    ECHO FATAL ERROR: Please specify a data base name
    goto :eof
    ) ELSE (
    ECHO @author amin.jellali
    ECHO @email <a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="ed8cc387c38c808483c3878881818c8184ad8a808c84...›protected]</a>
    echo starting import...
    call :loop_over_files_in_current_dir %~1
    echo import done...
    echo hope you enjoyed me
    )
goto :eof

Answer №5

After searching extensively, I finally found the solution I was looking for. Previous answers were either ineffective or merely provided temporary fixes.

Fortunately, I discovered the answer.. You can utilize fs.readFileSync to read (json) files. Below is my complete script for reference:

const dbName = process.env.DB_NAME || 'test_db';
const dbUser = process.env.DB_USER || 'test_user';
const dbPassword = process.env.DB_PASSWORD || 'secret';

db.createUser({
  user: dbUser,
  pwd: dbPassword,
  roles: [
    {
      role: 'readWrite',
      db: dbName,
    },
  ],
});

// Create and seed collections:

const collections = [
  'some_collection',
  'another_collection',
];

for (let i = 0; i < collections.length; i++) {
  const collectionName = collections[i];
  db.createCollection(collectionName);

  let jsonFile;
  try {
    jsonFile = fs.readFileSync(`/docker-entrypoint-initdb.d/mongo.${collectionName}.json`, 'utf8');
  }
  catch (err) {
    console.log(`NOTE: mongo.${collectionName}.json not found! Skip seeding of ${collectionName} collection..`)
    continue;
  }

  const json = JSON.parse(jsonFile);

  // Parse the $oid ids that Mongo uses in exports, but cant use in insertMany:
  for (let i = json.length; i--;) {
    json[i]._id = ObjectId(json[i]._id.$oid);
  }

  db[collectionName].insertMany(json);
}

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Calculate the total number of randomly generated integers using jQuery

Below are examples of the inputs I have generated: <input type="text" id="rfq_ironilbundle_quotes_offer_price_0" name="rfq_ironilbundle_quotes[offer_price][0]" required="required" class="form-control offered-price"> <input type="text" id="rfq_iro ...

Verify that the password is entered correctly in Angular2

My Angular2 form looks like this: this.registerForm = formBuilder.group({ 'name': ['', Validators.required], 'email': ['', Validators.compose([Validators.pattern("[A-Z0-9a-z._%+-]+@[A-Za-z0-9.-]+&bso ...

Navigating with jQuery Scrollbars

Looking to incorporate a bit of animation in jQuery, trying out the following line: window.parent.scroll(coord[0], coord[1]); The next block of code doesn't seem to be achieving what I had in mind. Do you have any suggestions? $(window.parent).anim ...

What is the best way to prevent event propagation in d3 with TypeScript?

When working with JavaScript, I often use the following code to prevent event propagation when dragging something. var drag = d3.behavior.drag() .origin(function(d) { return d; }) .on('dragstart', function(e) { d3.event.sourceEvent ...

How to send the value of a JavaScript loop variable to PHP using AJAX

How can I send variables in a loop to a PHP file using AJAX? var lat; var lng; var array = [22.399602, 114.041176, 22.344043, 114.0168, 22.327529, 114.087181]; console.log(array); for (var i = 0; i < 6; i += 2) { lat = array[i]; console.log("l ...

Deactivating upcoming weeks according to the year in Angular 8

In the user interface, there are dropdowns for selecting a year and a week. Once a year is selected, the list of weeks for that year is displayed in the week dropdown. Now, the requirement is to disable the selection of future weeks. For example, for the ...

What makes Mathematics a unique object in JavaScript programming?

Recently, I've dived into learning Javascript, so pardon me if my doubts seem a bit illogical. I came across the definition for a Math object, and here is the code snippet: interface Math { /** The mathematical constant e. This is Euler's nu ...

Tips on finding the ID of a textbox using the cursor's position

In the container, there are several textboxes. When a button is clicked, I want to insert some text at the cursor position in one of the textboxes. I have managed to insert text into a specific textbox using its ID, but I am facing difficulty in identifyin ...

The Model.findOneAndRemove() method has been updated and can no longer accept a callback function, resulting in

const express = require('express') const mongoose = require('mongoose') var app = express() var Data = require('./noteSchema') mongoose.connect('mongodb://localhost/newDB') mongoose.connection.once("open" ...

The Arabic content in the Json request is failing to be submitted

Encountering a problem when trying to post Arabic text in a Json Request field. The error message states: "The Request was Aborted; The Request was cancelled." I have attempted various encoding methods such as UTF8, ASCII, HTMLEncode, JavaScritpEncode, and ...

Provide Laravel 5.2 with JSON formatted data for processing

I have a text file named data.txt with the following values : {"id":"1","title":"first Testing","completed":"Yes"} {"id":"2","title":"Second Testing","completed":"no"} {"id":"3","title":"Third Testing","completed":"no"} and I also have a table called D ...

Guide to building an extjs view for a json document

I am currently developing a web application using the extjs and Yii framework. My server side is powered by Yii, while the client side utilizes extjs. I am passing Yii framework output to extjs as input in JSON format: CJSON::encode(array("questionId"=> ...

MUI-Datatable rows that can be expanded

I'm attempting to implement nested tables where each row in the main table expands to display a sub-table with specific data when clicked. I've been following the official documentation, but so far without success. Below is a code snippet that I& ...

Creating a line using CSS to connect two elements

I've been attempting to divide a series of circles with a line down the center. However, when I position a line (.Line1) to run between the first and last circle, it appears centered at the top left of the first circle instead of being truly centraliz ...

Organize pairs of strings into a nested array within an array using Angular

Currently, I am working on a project in Angular. In this project, I have a string that contains special characters which I have successfully removed using regular expressions. Now, my goal is to arrange the first two strings within square brackets and the ...

Ag-grid: how to reset a cell when using the "agSelectCellEditor"

According to the documentation on ag-grid: If Backspace or Delete is pressed, the default editor will clear the contents of the cell. However, this behavior does not apply when using the "agSelectCellEditor." Pressing Delete or Backspace puts the cell i ...

How to activate the menu in AngularJS

Within my application, I have a header that contains various menu items. These menu items are fetched from a service and displayed in the header. When hovering over the main list, the submenus appear. My goal is to highlight the parent item as active when ...

Learn how to efficiently generate multiple documents simultaneously in MongoDB utilizing Mongoose in Node.js. In cases where some documents fail to be inserted, the remainder should continue to be processed and

My goal is to insert and create multiple documents in MongoDB using Mongoose in Node.js. Even if some of the inserts fail, perhaps due to duplicate keys, I would like the others to be inserted successfully. Ideally, I want to have an output that displays b ...

Steps to eliminate duplicate embedded documents in a collection

I have a collection of users with various lists of sub documents. The schema looks something like this: { _id: ObjectId(), name: aaa, age: 20, transactions:[ { trans_id: 1, product: mobile, price: 30, ...

What is the best way to implement an event listener in ReactJS?

while following tutorials, I've noticed a common practice: import React from 'react'; function App(){ return( <div> <h1 onClick={functionName}>Hello React</h1> </div> ); } export default App; Howev ...