Using Puppeteer-cluster along with cheerio in an express router API results in an unexpectedly blank response

Question

Using Puppeteer-cluster along with cheerio in an express router API results in an unexpectedly blank response

I am currently working on developing an API using express, puppeteer-cluster, and cheerio to extract anchor elements containing specific words that can be used as query parameters. My aim is to utilize puppeteer to capture dynamically generated elements as well. However, I am facing an issue where I am receiving an empty array as the output displayed on the browser.

Despite dedicating the past 2 days to mastering this library, I have made no significant progress. Any assistance or guidance would be greatly appreciated.

Update: I have updated all my functions to include async operations, which have enabled them to run successfully. However, the end result still remains empty :(

Update 2: Upon thorough logging and debugging, I have identified that the data.name is being passed as a Promise to the cheerio function. This appears to be the root cause of the issue, although I am uncertain about how to resolve it at this point.

Update 3: One of the key challenges I encountered was the mishandling of the page content (HTML code) being passed to the cheerio function. Despite the content being empty in the browser response, an error is displayed in the console:

Error handling response: TypeError: Cannot read properties of undefined (reading 'innerText').

This leads me to believe that the response is not formatted as JSON. Could it be that res.json() is not the correct method for formatting?

Here is a snippet of my code:

app.js

[code snippet for app.js]

cluster.js

[code snippet for cluster.js]

Despite not encountering any errors, I am presented with an empty output on the screen. Can anyone identify where I may be going wrong with this implementation? :(

javascript express puppeteer cheerio puppeteer-cluster

Answer 1

Answer №1

It is not recommended to combine Puppeteer with another selection library like Cheerio as it can lead to redundancy and potential issues. Using a separate HTML parser with Puppeteer requires snapshotting the HTML and then parsing it with Cheerio, introducing unnecessary complexity.

By opting for Puppeteer's await page.content() instead of referencing document.body.innerHTML, you can avoid these complications.

Additionally, there is no need for Cheerio functions to be asynchronous since the API is fully synchronous.

For a basic setup using Cheerio with Puppeteer, ensure you understand the implications and consider if this extra layer is necessary for your specific use case:

const cheerio = require("cheerio"); // 1.0.0-rc.12
const puppeteer = require("puppeteer"); // ^19.0.0

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const url = "https://www.example.com";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const html = await page.content();
  const $ = cheerio.load(html);

  // perform Cheerio operations synchronously
  console.log($("h1").text()); // => Example Domain
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

The process is similar for puppeteer-cluster: integrate the lines involving await page.content(); into the cluster.task callback that interacts with page.

Answer 2

It is not recommended to combine Puppeteer with another selection library like Cheerio as it can lead to redundancy and potential issues. Using a separate HTML parser with Puppeteer requires snapshotting the HTML and then parsing it with Cheerio, introducing unnecessary complexity.

By opting for Puppeteer's await page.content() instead of referencing document.body.innerHTML, you can avoid these complications.

Additionally, there is no need for Cheerio functions to be asynchronous since the API is fully synchronous.

For a basic setup using Cheerio with Puppeteer, ensure you understand the implications and consider if this extra layer is necessary for your specific use case:

const cheerio = require("cheerio"); // 1.0.0-rc.12
const puppeteer = require("puppeteer"); // ^19.0.0

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const url = "https://www.example.com";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  const html = await page.content();
  const $ = cheerio.load(html);

  // perform Cheerio operations synchronously
  console.log($("h1").text()); // => Example Domain
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

The process is similar for puppeteer-cluster: integrate the lines involving await page.content(); into the cluster.task callback that interacts with page.

Using Puppeteer-cluster along with cheerio in an express router API results in an unexpectedly blank response

Answer №1

Similar questions

Creating a webpage that supports multiple languages using either JavaScript or jQuery

Troubleshooting: Why isn't setMetadata working in NestJS from authGuards

Steps for aligning items in a column horizontally in Bootstrap 5

Show only the record with the greatest price

Understanding the functionality of app.locals within app.get in an Express application and how to effectively parse data

Issue with Wicket: unable to generate sound for resource paths

A step-by-step guide on how to refresh a circular loading indicator

Having trouble with the `click()` function not working on a button while using Selenium in

What is the best way to move the Grid upward when the above content is not visible?

How to Trigger a Javascript Function from an OnItemClick Event in ASP.NET ListView

Having trouble with filtering an array using the some() method of another array?

Creating a stylish background gradient between two handles on a jQuery Slider with CSS

Initiate an "execute.document" command directly from the address bar

The progress indicator for AJAX file uploads is malfunctioning; the addEventListener('progress') method is not functioning as intended

There is an issue with the functionality of Java code that has been adapted from Javascript logic

Learning the integration of vuex with defineCustomElement in Vue 3.2

Focus on an empty <input> tag with just the type attribute

What is the best way to populate a dropdown menu by matching keys from an array within an ng-repeat

What is the best way to position and align an angle with the axis of a moving 3D figure?

Strange interaction observed when working with Record<string, unknown> compared to Record<string, any>