My current challenge involves using npm request and cheerio to extract webpages and analyze their HTML structure. Everything works smoothly in scenarios where the HTML is available upon request. However, I am facing a problem when a website initially displays a loading screen before updating its content with new information / elements after a short delay.
Snippet of my code:
var url = 'website with loading screen prior to content.com';
var request = require('request');
request(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Display the HTML for the Google homepage.
}
})
Desired Solution - I am looking for a way for the request function to either wait for a specific element to appear on the page before reading the body, or alternatively, to wait for a predefined number of seconds before processing the body.
Exploring Alternatives - It's plausible that achieving this with npm request may not be viable. In such a case, I am open to suggestions on alternative approaches. One option I am contemplating is utilizing webdriver.io or phantomjs. Could you provide guidance on which path to take?