Issue with Puppeteer: element selection resulting in null values or timing out

I am currently working on extracting the innerHTML value of a button from a webpage using puppeteer. My current approach involves awaiting the appearance of the selector before proceeding with the operation.

However, upon executing the code below, the program times out during the wait process.

const puppeteer = require("puppeteer");

const link =
  "https://etherscan.io/tx/0xb06c7d09611cb234bfcd8ccf5bcd7f54c062bee9ca5d262cc5d8f3c4c923bd32";

async function configureBrowser() {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(link);

  return page;
}

async function findFee(page) {
  await page.reload({ waitUntil: ["networkidle0", "domcontentloaded"] });
  await page.waitForSelector("#txfeebutton");
  console.log("boom");
}

const setup = async () => {
  const page = await configureBrowser();
  await findFee(page);
  await browser.close();
};

setup();

Despite confirming the existence of the element within the HTML:

https://i.sstatic.net/98ifd.png

The console output indicates a timeout issue:

https://i.sstatic.net/KJN17.png

javascriptpuppeteerethereumetherscan

Answer №1

When using a user agent string, everything runs smoothly:

const puppeteer = require("puppeteer"); // ^19.0.0

let browser;
(async () => {
  browser = await puppeteer.launch({headless: true});
  const [page] = await browser.pages();
  const ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36";
  await page.setExtraHTTPHeaders({"Accept-Language": "en-US,en;q=0.9"});
  await page.setUserAgent(ua);
  const url = "https://etherscan.io/tx/0xb06c7d09611cb234bfcd8ccf5bcd7f54c062bee9ca5d262cc5d8f3c4c923bd32";
  await page.goto(url);
  const btn = await page.waitForSelector("#txfeebutton");
  console.log(await btn.evaluate(el => el.textContent.trim())); // => ($0.56)
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

To troubleshoot, try running the same script with headless: false first to see if it works. If not, check page.content() when running headlessly. Cloudflare may be detecting your scraper and showing a captcha.

For more information, please refer to:

  • Puppeteer can't find elements when Headless TRUE
  • Why does headless need to be false for Puppeteer to work?

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Dynamic form population with dropdown selection using Ajax - Coldfusion continuation

Following some valuable feedback from my previous inquiry, I have made progress: Refer to the original post - Coldfusion Populate form with dropdown selection using ajax Currently, I have successfully sent a request to my CFC with a remote function, but I ...

javascriptajaxjquerycoldfusion

Image loading event doesn't trigger when used inside the useEffect() hook

I am encountering a problem with my React function where the console log message "Drawing image...." does not appear even though I do not see any errors reported from image.onerror. This issue arises when I fetch image data in base64 format fro ...

javascriptreactjsimage

Can a Vue component in SFC form be utilized as a mixin?

Consider this scenario, where I have a component created in SFC format: // Parent.vue <template> <div> {{ txt }} </div> </template> <script> export default { name: 'ParentComponent', data() { return ...

javascriptvue.jsvuejs2vue-sfc

Fade or animate the opacity in jQuery to change the display type to something other than block

I am currently using display: table and display: table-cell to vertically align multiple divs. However, I have encountered an issue when animating the opacity with jQuery using either fadeTo() or fadeIn. The problem is that it always adds inline style di ...

javascriptjqueryhtmlcss

How to Retrieve JSON Object Using Dynamically Generated String in Angular

Creating a controller in Angular involves retrieving URL parts and parameters using $statePrams. The $http service is then called to retrieve data from a JSON file. Once the data is received, the content of a specific object element - determined by $state ...

javascriptangularjsjson

Encounter a "syntax error Cannot GET /xyz" message using jQuery AJAX

When using $.ajax to request data from a node.js server, I encountered an error while debugging on the client side in Firefox. The console displayed: Syntax error Cannot GET /xyz, where /xyz represents the route for my server. Despite this error, the page ...

javascriptjqueryajaxnode.js

Failed to convert the value "hello" to an ObjectId (string type) for the _id path in the product model

i am currently using node, express, and mongoose with a local mongodb database. all of my routes are functioning correctly except for the last one /hello, which is giving me this error: { "stringValue": "\"hello\"&qu ...

javascriptnode.jsmongodbexpressmongoose

Specify the return type based on specific parameter value

I'm facing a situation where I have two definitions that are identical, but I need them to behave differently based on the value of the limit parameter. Specifically, I want the first definition to return Promise<Cursor<T>> when limit is g ...

javascripttypescripttypescript-typings

Choosing a single radio button value within a nested ng-repeat loop in AngularJS

Help needed with selecting only one radio button value in nested ng-repeat. Please review the source code and suggest any potential mistakes. <form ng-submit="save()"> <div ng-repeat="x in surveyLst"> <div class="row"> < ...

javascriptjqueryhtmlangularjsforms

Check the value of a single bit in JavaScript: Is the bit on (1) or off (0)?

Is there a way in JavaScript to determine if a single Bit is On (1) or Off (0)? function isBitOn(number, index) { // ... ? } // Example: let num = 13; // 1101 isBitOn(num, 0); // true 1 isBitOn(num, 1); // false 0 isBitOn(num, 2); // true 1 isBit ...

javascriptbinarybit-manipulationbitwise-operatorsbit

Having trouble with the "foreach" binding in knockout where the "html" binding buttons are not functioning properly in Javascript

I'm experiencing an issue with the following code: <html lang="en"> <head> <link href="css/bootstrap.min.css" rel="stylesheet"> </head> <body> <script type='text/javascript' src='knockout-3. ...

javascripthtmlknockout.js

Utilizing JQuery's .Toggle function in conjunction with a stylish CSS transform button

I am trying to create a toggle menu that shows and hides itself without requiring the button to be clicked twice. I have added e.preventDefault(); which solves the issue, but now the button does not transform as intended. $(document).ready(function() { ...

javascriptjqueryhtmlcss

Printing a docx file from Node.js to a physical printer

I'm currently working on a project in node.js where I need to enable users to print a docx file directly from the printer. Is there anyone out there who can guide me on how to achieve this using Node.js? ...

javascriptnode.jsprintingprinters

Use node.js and the Awilix modules to add parameters to a class that extends from a parent

Looking to establish a parameter in the child class and utilize it in the base class, I aim to access this variable initialized in the child constructor without invoking super() as I require specific services only in the parent. The child classes are ins ...

javascriptnode.jsoopkoa

Guide to generating a div element with its contents using JSON

On my webpage, there is a button that increases the "counter" value every time it's clicked. I am looking to achieve the following tasks: 1) How can I generate a json file for each div on my page like the example below: <div class="text1" id="1" ...

javascriptjqueryjson

The height of the div container exceeds 100% stretch

I am trying to incorporate a scrollbar for a specific div container. Here is the current code snippet that I have: $(document).ready(() => { for (let i = 0; i < 100; i++) { const newDiv = (`<div>Log Item</div>`); $("#logsCont ...

javascripthtmlcss

Tips for sharing a global variable in Node.js multi-cluster mode while running Socket.IO on an NGINX load balancer but using fork mode with PM2

My socket.io app is currently running with NGINX load balancing on 6 cores, distributing the load among them. When I use pm2 list myapp, it shows that the app is running in fork mode but spanning across 6 processes due to NGINX load balancing. │ myapp-1 ...

javascriptnode.jsnginxsocket.iopm2

What sets Express.js apart from koa2.js in handling asynchronous functions?

I've encountered a situation where I had to set up the router using Express, and it was functioning correctly with the following code: router.get('/',(req,res)=>{ queries.getAll().then(stickers=>{ res.json(stickers) }) ...

javascriptmysqlexpressasync-awaitkoa

Discovering the dimensions of a video in Angular 2 using TypeScript

To determine whether the video is portrait or landscape, I am attempting to retrieve the height and width of the video using the following method: import { Component, OnInit, AfterViewInit } from '@angular/core'; @Component({ selector: ' ...

javascriptangulartypescriptvideo

When incorporating express.static(), the Express .use() callback may be triggered multiple times

I'm in the process of verifying a user's identity, and once that is confirmed I aim to add them as a new user in my personal database using the information provided by the authentication server. The issue at hand is that the function createNewAc ...

javascriptnode.jsexpress