Identify dead hyperlinks on a webpage with the help of selenium webdriver while steering clear of links that

I have been trying to identify broken links on a webpage by extracting all anchor tags. However, some of the links are dynamically generated through JavaScript. When I attempt to print out the list of all the links, I encounter a StaleElementReferenceException error due to these dynamically created links. Can anyone explain why this error occurs specifically for the twitter link provided below?

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title></title>
    </head>
<body>
    <div style="padding-top:10px;">
        <a href="https://twitter.com/url" class="twitter-follow-button" data-show-count="false" data-size="large" data-show-screen-name="false">Follow @url</a>
            <script>!function(d, s, id) {
                var js, fjs = d.getElementsByTagName(s)[0], p = /^http:/.test(d.location) ? 'http' : 'https';
                if (!d.getElementById(id)) {
                    js = d.createElement(s);
                    js.id = id;
                    js.src = p + '://platform.twitter.com/widgets.js';
                    fjs.parentNode.insertBefore(js, fjs);
                }
            }(document, 'script', 'twitter-wjs');</script>
    </div>
</body>

Selenium code :

     List<WebElement> links=driver.findElements(By.tagName("a"));    for(WebElement link: links)   {       System.out.println(link.getAttribute("href"));   }

Answer №1

It is highly likely that the issue at hand is unfolding in the following manner:

  1. get(<url>) is initiated, causing the page to start loading
  2. findElements(By.tagName("a")) is executed. This triggers an event in the browser, prompting it to compile a list of matches and send them back to the WebDriver caller.
  3. Meanwhile, the <script> tag is appended, requesting for platform.twitter.com/widgets.js, which commences creating new / altering existing <a> elements.

In essence, due to potential differences between your WebElements coming from two distinct states (pre-load vs. post-load) and the fact that some elements no longer match after the change, the "before" elements are deemed stale.

You have several options you can explore:

  1. Postpone executing findElements(By.tagName("a")) until confirmation that all Twitter widgets have completed loading, such as by waiting for one of the newly generated links to appear.
  2. Handle the StaleElementReferenceException if encountered, then re-run the findElements command within your exception handler.

(2) is likely the easier approach. If the exception occurs only 20% of the time and the lookup process is swift, performance impact should be minimal.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

When Using TypeScript with Serverless, 'this' Becomes Undefined When Private Methods are Called from Public Methods

Currently, I am working on constructing an AWS Serverless function using TypeScript. My focus is on creating an abstract class with a single public method that invokes some private methods. Below is the simplified version of my TypeScript class: export ...

What is the best way to change an int32 to a float32 in Deeplearn.js?

In December 2017, this code did the job perfectly. However, after updating Deeplearn.js to the latest version, it no longer functions as expected: math.scope(function(keep, track) { var i = track(dl.Array3D.fromPixels(rawImageData, 4)); i = math.multi ...

Struggling to get a shader up and running on three.js using shaderfrom

Trying to add this shader to my project: This is the fragment shader code: <script id="fragmentShader" type="x-shader/x-fragment"> #ifdef GL_ES precision highp float; precision highp int; #endif uniform vec2 u_resolutio ...

Mixing box-shadow and blur filters can result in unusual artifacts appearing in Webkit browsers

Recently, I encountered an intriguing and frustrating bug in WebKit browsers. Consider a scenario where there is a parent DIV (let's say .wrapper) and a child DIV (.main). You apply a box-shadow on the .main DIV and a blur filter on the .wrapper DIV. ...

The click event listener only seems to fire on the second attempt

This block of code is designed to function as a search algorithm that also provides keyword suggestions. When a user clicks on a keyword, it is supposed to be placed into an input textbox. The possible keywords are "first" and "second". However, there is ...

Error: Primefaces ajax status failure (dialog has not been defined)

I incorporated the same code found on primefaces.org, specifically this link: http://www.primefaces.org/showcase/ui/ajaxStatusScript.jsf <p:ajaxStatus onstart="statusDialog.show();" onsuccess="statusDialog.hide();"/> <p:dialog modal="true" ...

Ensure exclusive access to variables for sequential updates in jQuery or JavaScript

I am currently developing a series of small functions in jQuery aimed at tracking how long it takes for a user to complete a form. The form consists of various input types and 'continue' buttons located within the form itself. Below are some sa ...

Obtaining a byte array using the file input method

var profileImage = fileInputInByteArray; $.ajax({ url: 'abc.com/', type: 'POST', dataType: 'json', data: { // Other data ProfileImage: profileimage // Other data }, success: { } }) // Code in Web ...

Animation is not applied to Bootstrap Collapse on a row within a table

My Bootstrap 4 table has an issue - when I apply the "collapse" class to a table row, it behaves differently compared to applying it to a div element. Clicking "animate div" smoothly animates the target div, but clicking "animate tr" does not animate the ...

Error: Unable to locate module: Issue: Unable to find '../components/charts/be.js' in '/vercel/workpath0/my-app/pages'

Having some trouble deploying my next.js app through Vercel. Everything works fine locally with the command 'npm run dev'. But when attempting to deploy it on Vercel using a Github remote repository, I encountered the following error: 18:07:58.29 ...

Is there a way to transmit React code using Express?

Currently, I'm in the process of creating a coloring demonstration. Initially, I had an SVG file at hand, but I made the decision to utilize svgr for its conversion into a React component. This strategy will offer me the flexibility to easily modify i ...

I want to retrieve a complete HTML page using an AJAX request

I am trying to retrieve the content of a specific page, but encountering issues when using this function: function getResult() { var url="http://service.semanticproxy.com/processurl/ftfu27m3k66dvc3r43bzfneh/html/http://www.smallbiztechnology.c ...

Trouble mapping an array of objects in ReactJS

I'm encountering an issue where I am unable to map through an array of objects in my component. Although I have used this map method before, it doesn't seem to be working now. Can anyone help me figure out what's going wrong? import React, ...

When utilizing the data property within the useQuery() hook, an error may arise stating "Unable to

This problem has me scratching my head. The code snippet below is triggering this error: TypeError: Cannot read properties of undefined (reading 'map') When I use console.log() to check res.data, everything seems normal and there is data present ...

Connect an Angular Service object to a Controller's Scope and keeping it in sync

I am facing an issue with the interaction between my EmployeeService and EmployeeController. The service contains a specific object which I bind to the controller's scope: app.controller('EmployeeController', function($scope, EmployeeServic ...

Display additional information from a JSON file after choosing an ID with AngularJS Select

After saving a JSON file filled with information, I managed to successfully populate a select menu with the names of each element from the JSON data using this code snippet: <select ng-model="car.marca" ng-options="item.brakeId as item.name for item in ...

a single line within a compartment

I am encountering an issue with the code snippet below, which is responsible for generating the location of different records within a table: return ( <TableRow> <TableCell > // some code_1 < ...

Comparing AngularJS: The difference between ng-click function and directive

I'm facing a dilemma in choosing the best method for triggering an alert when buttons are clicked. There are two methods I've come across, and I need help determining which one is the preferred practice. Can you provide insights into why one meth ...

What steps can be taken to remove the search parameter responsible for the error?

Imagine having a webpage that displays search results based on the parameters in the URL, like this: https://www.someurl.com/categories/somecategory?brands=brand1,brand2,brand3 This URL will show listings for only brand1, brand2, and brand3. Additionally ...

Learn how to effortlessly integrate and utilize a bpmn file in a Vue component by leveraging the raw-loader

As a newcomer to bpmn-js, I am facing the challenge of importing and utilizing a .bpmn file within a vue.js component (BPMNViewer.vue). Despite my efforts in researching, I could only find recommendations to use the raw-loader. Consequently, I am currently ...