Ways to boost heap capacity in Corb2 for MarkLogic

I encountered a challenge while running my custom deletion framework on a massive dataset. When attempting to execute Corb2, I started receiving the warning below and occasionally even encountered errors due to insufficient memory.

WARNING: Slow receive! Consider increasing max heap size and using -XX:+UseConcMarkSweepGC

Error:

Sep 23, 2021 1:06:23 PM com.marklogic.developer.corb.QueryUrisLoader closeRequestAndSession
INFO: closing uris session
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
        at java.lang.StringBuilder.append(StringBuilder.java:190)
        at com.marklogic.io.IOHelper.literalStringFromReader(IOHelper.java:50)
        at com.marklogic.io.IOHelper.literalStringFromStream(IOHelper.java:66)
        at com.marklogic.http.MultipartBuffer.getBodyAsString(MultipartBuffer.java:75)
        at com.marklogic.xcc.impl.AbstractResultSequence.instantiateResultItem(AbstractResultSequence.java:138)
        at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:147)
        at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:166)
        at com.marklogic.developer.corb.QueryUrisLoader.populateQueue(QueryUrisLoader.java:212)
        at com.marklogic.developer.corb.QueryUrisLoader.createAndPopulateQueue(QueryUrisLoader.java:201)
        at com.marklogic.developer.corb.QueryUrisLoader.open(QueryUrisLoader.java:122)
        at com.marklogic.developer.corb.Manager.runUrisLoader(Manager.java:809)
        at com.marklogic.developer.corb.Manager.populateQueue(Manager.java:830)
        at com.marklogic.developer.corb.Manager.run(Manager.java:578)
        at com.marklogic.developer.corb.Manager.main(Manager.java:138)

Has anyone successfully used this tool (especially with JavaScript) for handling large datasets before?

javascriptunixmarklogicmarklogic-corb

Answer №1

Instead of struggling to determine the correct amount of memory needed to store all URIs in-memory, you can activate the DISK-QUEUE feature. This will allow the URIs queue to overflow to a temporary disk file that can be accessed as needed, rather than holding everything in memory.

If desired, you have the option to set the location for the temporary files with DISK-QUEUE-TEMP-DIR. If no path is specified, it will default to using the TEMP-DIR or java.io.tmpdir.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

A guide on updating an item in a basic JavaScript file with Node.js

This query pertains to Workbox and Webpack, but no prior knowledge of these libraries is necessary. Background Information (Optional) Currently using the InjectManifest plugin from Workbox 4.3.1 (workbox-webpack-plugin). While this version provides the o ...

javascriptnode.jswebpackfsworkbox

Store in database and forward to a different web address

I have created an interactive quiz using jQuery. I need to add a specific feature at the end of the quiz. if (answered_questions === total_questions) { save the user's score to the database; redirect to specified URL; } The redirect_url is ...

javascriptjquery

Why did my compilation process fail to include the style files despite compiling all other files successfully?

As English is not my first language, I kindly ask for your understanding with any typing mistakes. I have created a workspace with the image depicted here; Afterwards, I executed "tsc -p ." to compile my files; You can view the generated files here Unf ...

javascriptcssreactjstypescriptless

Error message: An error occurred while executing the AJAX PHP code due to a TypeError, specifically stating that the property 'status' cannot be

For some reason, I keep receiving an undefined return for my response. The event handler in index.php triggers the following code: $.post("getData.php", onNewPost()); function onNewPost (response){ if (response.status == "OK") { console.log(resp ...

javascriptphpajax

Having trouble inputting text into input field using ReactJS

I am currently facing a challenge while attempting to modify the value of an input field using ReactJS. The issue I am encountering is the inability to input values into the field. After reviewing several other queries, it appears that simply changing the ...

javascriptreactjs

Sharing and showcasing files directly from a local directory

Recently diving into NodeJS and web development, I've successfully used multer to upload a single file within my web application. The file gets uploaded to my "uploads" folder flawlessly, and now I'm planning on storing the file path in my databa ...

javascriptnode.jsexpressuploadmulter

Is there a way to automatically submit an upload form once a file has been selected?

Need help with automatically submitting a file upload form when a file is selected. Is there a way to bypass the need for the user to click the Submit button? ...

javascripthtml

What is the best way to transmit two distinct sets of data from a child component to the v-model of a parent component?

Currently, I am working on a project using vuejs 2 and typescript. In this project, I need to pass two different sets of data - data and attachments - within the parent component. I am utilizing vue-property-decorator for this purpose. However, I am facing ...

javascripttypescriptvue.js

Enhance your Rails 5 application with a dynamic live search feature powered by Keyup. Say goodbye to

Currently, I have a Rails 5.1.3 application with a simple contact model that stores names and phone numbers. To enable searching within the index view/page, I am utilizing Ransack. A keyup event listener written in Coffeescript captures user input as they ...

javascriptruby-on-railsajaxsearchcoffeescript

Sending a cookie token to the server through the header

This is my first attempt at working with a server Utilizing React and express Storing token in browser cookie from the server //Upon login request res.cookie('token', token, {maxAge: 3600000} ).json({ user: userDoc, message: 'message!&apos ...

javascriptreactjsexpressbackend

Enhance the textarea using Javascript when clicked

I am experimenting with styling my textarea using a combination of JAVASCRIPT and CSS. The goal is to make it expand in size from 20px height to 120px height when clicked, using document.getElementById("tweet_area"). However, I am facing an issue where t ...

javascriptcss

The intended functionality of clicking on an image is exclusively reserved for its immediate parent element

I have a feature on my website that displays an image gallery. When a user clicks on an image, it opens up the image in full screen similar to Facebook's theatre mode. I have written code so that when the user clicks anywhere in the container of the i ...

javascriptjqueryhtmlimage

Upon returning to the website homepage from another page, the JavaScript animation ceases

I implemented a unique typewriter animation on my homepage by utilizing code from this source and making minor HTML edits to customize the content. https://codepen.io/hi-im-si/pen/DHoup. <h5> <body>Hello! I am Jane.</body> < ...

javascripthtmlcssanimation

Transferring class titles between div elements

I would like to implement a feature where class names on div elements shift after a brief delay, perhaps 10 seconds. My goal is for the 3 classes listed below to cycle through the div elements and start over once it reaches the last one. For instance: & ...

javascriptjqueryhtmlcss

Creating a running text (marquee) with CSS is a simple and efficient way to make

I encountered a significant challenge while delving into CSS animation. My goal is to create a "transform: translate" animation that displays text overflowing the content width as depicted in the image below. https://i.stack.imgur.com/sRF6C.png See it i ...

javascripthtmlcsscss-animations

Ways to eliminate empty values from an array in JavaScript

I need help deleting any null elements from my array [ [ null, [ [Array], [Array] ] ] ] I am looking to restructure it as [ [[Array],[Array]], [[Array],[Array]], [[Array],[Array]] ] If there are any undefined/null objects like : [ [[Array],[]], [[A ...

javascriptarraysnull

Incorporate VLC player into a webpage without any visible control options

Is there a way to embed a flash video in a webpage without showing any controls? I managed to embed a flash video using VLC with the following code: <embed src="img/Wildlife.wmv" height="480" width="640"> However, I want the video to play without ...

javascriptjqueryhtmlflashvideo

Sending parameters to a personalized Angular directive

I am currently facing a challenge in creating an Angular directive as I am unable to pass the necessary parameters for displaying it. The directive code looks like this: (function () { "use strict"; angular.module("customDirectives", []) .directive ...

javascriptangularjsangularjs-directive

Develop a cutting-edge front end for Hyperledger Fabric 2.2 with React js and enhance it with a node

Currently, I have developed a unique hyperledger fabric 2.2 blockchain network using javascript and am looking to integrate it with a React.js front end utilizing the node.js API. Despite my efforts to find relevant examples, most resources focus on hyperl ...

javascriptnode.jsreactjshyperledger-fabric

Enhance chat functionality by integrating MySQL database to store and update chat messages

I'm currently working on developing a chat system that allows users to enter the chat and send messages. The messages are being stored in a MySQL database, and here is a snippet of my code... <script> $('input[type=text]').on('ke ...

javascriptphpmysqlajax