Identify dead hyperlinks on a webpage with the help of selenium webdriver while steering clear of links that

I have been trying to identify broken links on a webpage by extracting all anchor tags. However, some of the links are dynamically generated through JavaScript. When I attempt to print out the list of all the links, I encounter a StaleElementReferenceException error due to these dynamically created links. Can anyone explain why this error occurs specifically for the twitter link provided below?

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title></title>
    </head>
<body>
    <div style="padding-top:10px;">
        <a href="https://twitter.com/url" class="twitter-follow-button" data-show-count="false" data-size="large" data-show-screen-name="false">Follow @url</a>
            <script>!function(d, s, id) {
                var js, fjs = d.getElementsByTagName(s)[0], p = /^http:/.test(d.location) ? 'http' : 'https';
                if (!d.getElementById(id)) {
                    js = d.createElement(s);
                    js.id = id;
                    js.src = p + '://platform.twitter.com/widgets.js';
                    fjs.parentNode.insertBefore(js, fjs);
                }
            }(document, 'script', 'twitter-wjs');</script>
    </div>
</body>

Selenium code :

     List<WebElement> links=driver.findElements(By.tagName("a"));    for(WebElement link: links)   {       System.out.println(link.getAttribute("href"));   }

Answer №1

It is highly likely that the issue at hand is unfolding in the following manner:

  1. get(<url>) is initiated, causing the page to start loading
  2. findElements(By.tagName("a")) is executed. This triggers an event in the browser, prompting it to compile a list of matches and send them back to the WebDriver caller.
  3. Meanwhile, the <script> tag is appended, requesting for platform.twitter.com/widgets.js, which commences creating new / altering existing <a> elements.

In essence, due to potential differences between your WebElements coming from two distinct states (pre-load vs. post-load) and the fact that some elements no longer match after the change, the "before" elements are deemed stale.

You have several options you can explore:

  1. Postpone executing findElements(By.tagName("a")) until confirmation that all Twitter widgets have completed loading, such as by waiting for one of the newly generated links to appear.
  2. Handle the StaleElementReferenceException if encountered, then re-run the findElements command within your exception handler.

(2) is likely the easier approach. If the exception occurs only 20% of the time and the lookup process is swift, performance impact should be minimal.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

How can React.Js and Typescript be used to extract information from JSON files?

Hi there! I've been working with MongoDB, Mongoose, and React.Js for my project. One of the features I have is a page where users can view posts. When making a fetch request to the backend, I receive JSON data in response: { "post" ...

"Implementing a click event handler on a button within an iframe

On my website, I have embedded an iframe containing external content. Within the code of this iframe is a button identified by the id "XYZ." I want to create an onclick function for this button. Here's what I've attempted: $( "#XYZ" ).click(fun ...

Placing the date of each blog post at the end of the excerpt

Currently, there is a grid section on the homepage of our site showcasing the two most recent blog posts. I am looking for a Javascript solution to relocate the date to the bottom of each block, outside of the border. For reference, you can visit the follo ...

JavaScript function for converting timestamp to readable date

Can someone help me transform the timestamp 1382086394000 into a readable date format 2013-10-18 08:53:14 by using a JavaScript function? The current function I have is: function cleanDate(d) {return new Date(+d.replace(/\/Date\((\d+)\ ...

Update the content of a div element with the data retrieved through an Ajax response

I am attempting to update the inner HTML of a div after a certain interval. I am receiving the correct response using Ajax, but I am struggling to replace the inner HTML of the selected element with the Ajax response. What could be wrong with my code? HTM ...

Executing Python Selenium Tests in Headless Mode

I'm facing some challenges when trying to run Webdriver on a specific hub in headless mode using Pyvirtualdisplay. The generic code below works without any issues: class TestHub4444TestClass01(unittest.TestCase): def setUp(self): self.di ...

Monitor and retrieve live updates of links using Javascript

I am a beginner in JavaScript and recently developed a successful Chrome extension for Dubtrack. I have been struggling to find a way to make my injected script run in real-time and grab the latest YouTube music video URL. Any assistance would be greatly a ...

What could be causing my AngularJS JSONP request to fail when trying to query Solr?

After attempting to query my Solr server with the provided code snippet: var url ="http://localhost:8080/solr/sdc/selectwt=json&callback=JSON_CALLBACK&q=opioid" $http.jsonp(url ).success(function(res){ console.log(res) }) An error is returned ...

What is the best way to send the selected option from a dropdown to a button click function within the controller?

I need to pass the selected value from a user's country selection dropdown to the ng-click="saveChanges()" function on the submit button. Is there a correct method for achieving this? I want to be able to access the user's selection from the dro ...

What is the best way to invoke a Service from a Directive in AngularJS?

Every time a user clicks, a directive is triggered to change the icon for adding to their favorite list. View: <div class="col col-33"> <i class="icon ion-ios-heart-outline" favorite="place[0].objectId" on-icon="ion-ios-heart-outline" off-ic ...

Utilize an AngularJS controller to verify the presence of a cookie and display a modal pop-up

In the process of developing a Single Page Application (SPA), I have encountered an issue with an HTML element calling an AngularJS controller. Here is what I need: I want the controller to check for a specific cookie: - If the cookie exists, call a ...

Exploring the functionalities of can-deactivate without incorporating routing

I'm facing an issue with a parent component and multiple child components. The parent component contains a form, while each child component also has its own form. Unfortunately, all child components share the same route as the parent component and hav ...

Adjust the Scope in Angular-Charts.js Post-Rendering

I am currently facing a challenge with displaying multiple charts using the angular-charts.js framework. The issue is that I require all the charts to have the same scale, but each chart currently has its own scale based on the displayed data. Unfortunatel ...

Is there a way to transform a .pcm file to a wav file using node.js?

I'm working on a project that generates a pcm file from a stream and saves it after the stream finishes. Now, I am looking for a way to convert this .pcm file to a .wav or another audio format using an npm package. Can anyone suggest a solution or poi ...

Open the CSV document

Why am I receiving an error stating ./vacancy-data.csv cannot be found when attempting to pass the csv file into the csvtojson npm package? The csv file is located in the same directory as the code below: var express = require('express'), route ...

What are some methods for utilizing the data received through this.props.history?

Whenever I need to navigate from one route to another using this.props.history.push("/"), I also want to pass along some extra data. For example: this.props.history.push('/postDetail', {data: item}). However, I am uncertain about where to define ...

The conversion from CSV to JSON using the parse function results in an inaccurate

I am having trouble converting a CSV file to JSON format. Even though I try to convert it, the resulting JSON is not valid. Here is an example of my CSV data: "timestamp","firstName","lastName","range","sName","location" "2019/03/08 12:53:47 pm GMT-4","H ...

Can Selenium be used to retrieve the CSS selector?

My current task involves scraping the following webpage: here. I am focused on extracting all images from this website, but unfortunately, they do not have any size (width, height) attributes included. Consequently, when extracted, the images appear too la ...

Angular2: Utilizing filter and search functionalities for an unordered list

Is there a way to implement filtering for an unordered list in Angular using an input field? This specific component creates an unordered list upon page load by fetching data from a JSON file and using the *ngFor directive along with a service. Below is t ...

The Bootstrap Tooltip seems to be glued in place

Utilizing jQuery, I am dynamically generating a div that includes add and close buttons. Bootstrap tooltips are applied to these buttons for additional functionality. However, a problem arises where the tooltip for the first add button remains visible even ...