The ultimate destination URL after redirection with Selenium in Java using HtmlUnit

I have a question that I believe can be resolved using Selenium. I have a collection of URLs, such as the example below.

http://www.sears.com/search=little tikes&Little Tikes?filter=Brand&keywordSearch=false&vName=Toys+%26+Games&catalogId=12605&catPrediction=false&previousSort=ORIGINAL_SORT_ORDER&viewItems=50&storeId=10153&adCell=W3

If you enter the URL into a browser, it will redirect to another URL, which you can confirm in the address bar of the browser (for example, Firefox). I need to retrieve the redirected URL, regardless of whether the redirection was caused by JavaScript code or not. Is it achievable with the Selenium framework?

I've attempted to use HTMLUnit for this task, but I encountered the following error related to JavaScript execution. Any assistance would be appreciated!

com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "indexOf" of null (script in http://www.sears.com/search=little%20tikes&Little%20Tikes?filter=Brand&keywordSearch=false&catalogId=12605&adCell=W3&catPrediction=false&previousSort=ORIGINAL_SORT_ORDER&viewItems=50&storeId=10153&levels=Toys+%26+Games from (6942, 33) to (6974, 14)#6966)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:669) ~[htmlunit-2.12.jar:2.12]
    ...
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:374) ~[htmlunit-2.12.jar:2.12]
javajavascripthttp-redirectseleniumhtmlunit

Answer №1

If I have understood your question correctly, following these steps should make it easy for you:

1. Retrieve the FirefoxDriver object
2. Execute driver.get("http://www.sears.com/search=little tikes&Little Tikes?filter=Brand&keywordSearch=false&vName=Toys+%26+Games&catalogId=12605&catPrediction=false&previousSort=ORIGINAL_SORT_ORDER&viewItems=50&storeId=10153&adCell=W3");
   This will open the specified URL in Firefox. The URL will then be redirected to its actual destination (based on my understanding of your description).
3. Use driver.getCurrentUrl() to get the current URL.

Feel free to let me know if this solution works for you :)

UPDATE :

WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_9);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setCssEnabled(true);     
HtmlPage page = (HtmlPage) webClient.getPage("http://www.sears.com/search=little tikes&Little Tikes?filter=Brand&keywordSearch=false&vName=Toys+%26+Games&catalogId=12605&catPrediction=false&previousSort=ORIGINAL_SORT_ORDER&viewItems=50&storeId=10153&adCell=W3");
WebResponse response = page.getWebResponse();
String content = response.getContentAsString();
System.out.println(page.getUrl());

Answer №2

If you're using the HTMLUnit Driver, make sure to enable JavaScript (it's turned off by default) like this:

Additionally, HTMLUnit utilizes Rhino as its JavaScript engine, which sets it apart from other mainstream browser JS engines.

HtmlUnitDriver session = new HtmlUnitDriver();
session.setJavascriptEnabled(true);

or

HtmlUnitDriver session = new HtmlUnitDriver(true);

Follow these steps to retrieve the redirected URL:

session.navigate().to("URL");
session.getCurrentUrl(); // This will get the current redirected URL.

I hope this information is useful for you!

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The element 'mesh' is not found within the type 'JSX.IntrinsicElements''

I've been diligently studying the React Three Fiber documentation, but I've hit a roadblock with this particular issue. Take a look at my component https://i.sstatic.net/obiNv.png And here are the dependencies I'm using https://i.sstatic. ...

javascriptreactjsthree.jsreact-three-fiber

What is the best way to access a protected method in an abstract class?

Looking to access a protected method of an abstract class - any suggestions? I've been struggling to access the protected method of the "SingleBrowserLocator" class in Selenium WebDriver. http://selenium.googlecode.com/git/docs/api/java/index.html ...

javaseleniumwebdriver

What is causing the slow performance of this JavaScript array retrieval?

I am working with a large array called frames_to_boxes, consisting of 5000 elements. Each element is an array of Objects belonging to the Box class: class Box { constructor(x, y, width, height, frame, object_class, id) { this.x = x; this.y = y; ...

javascriptperformancevue.js

Using javascript to overlay and position many images over another image

I've searched extensively but haven't been able to find any relevant answers here. Currently, I am developing a hockey scoring chance tracker. My goal is to display an image of the hockey ice where users can click to mark their input. Each click ...

javascriptjqueryhtmlcanvas

Error encountered while building a NeoMAD example for Android - reference issue

Currently, I am in the process of developing a proof of concept application utilizing NeoMAD. To start off, my initial task is to set up the SDK and compile one of their android examples to test on my mobile device. I submitted a request for the evaluatio ...

javaandroideclipseneomad

Having trouble with ExecuteJavaScript? It keeps throwing an error that says, "ReferenceError: OpenQA is not defined."

I am attempting to trigger a right-click event on an anchor tag using the JavaScript executor, but I keep receiving an error message stating "ReferenceError: OpenQA is not defined". I'm uncertain if this is the correct method for calling a JS function ...

c#seleniumwebdriver

using ng-class or ng-style for displaying error messages when validating a text area content

I'm curious about the most effective "angular" method for changing the character count style for validation purposes. I have a text area with a 250-character limit. Instead of restricting users to exactly 250 characters, we want to allow them to excee ...

javascriptcssangularjs

Fixing the Responsive Menu Issues

In order to achieve the desired effect of making the text from menu and logo disappear when minimizing the menu, you can use the following approach: Create a function called `smallNav()` to handle the resizing of the sidebar container: function sm ...

javascripthtmlcssbootstrap-4

Text color animation effect for menu

Here is the layout of my menu: <nav> <a href="#"> <span class="black">Home</span> <span class="red active">Home</span> </a> <a href="#"> <span class="black">Longer m ...

javascriptjqueryhtmlcss

How to Get into a Nested Class in JavaScript

I'm struggling to articulate my question, but I know there's a solution out there. I have profile cards in HTML, each with the class "card" containing two numbers. My goal is to display a progress bar specific to the % relationship of these numbe ...

javascript

Alert: No precise match could be located for CDP version 111

Why is this error occurring: WARNING: Unable to find an exact match for CDP version 111, closest version found: 109? The dependencies in my pom.xml file consist of: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www ...

javaselenium-webdriverpom.xml

Transforming the breakpoint can cause fluctuations in the jQuery UI selectable selected event

I'm running into an issue with a jQuery UI selectable div that contains child divs. The problem is that the child divs are not getting selected until I insert a breakpoint inside the selected handler. For a live example, please see my JS Fiddle: htt ...

javascriptjqueryjquery-ui

What is the best way to show the totals on a calculator screen?

As part of my work, I created a calculator to help potential clients determine their potential savings. Everything seems to be working fine, except for the total fees not appearing for all the boxes. I believe I might be missing the correct process to add ...

javascripthtmlcsscalculator

Struggling to retrieve variables from JSON response in JavaScript

While working on my project, I encountered an issue with accessing an array inside the response object from an API call using AJAX. Despite numerous attempts, I am unable to retrieve the 'quantity' value of the 'expert' object from the ...

javascriptarraysajaxobject

What is the process for inserting text or letters into a checkbox using Material Ui?

I am looking to create circular check boxes with text inside them similar to the image provided. Any help or guidance on achieving this would be greatly appreciated. View the image here ...

javascriptreactjsmaterial-ui

Guide to importing various files with AutoIt

I've been working with the following code snippet: ControlFocus("File Upload","","File1") ControlSetText("File Upload","","File1",'"C:\Users\Desktop\image1.png" "C:\Users\Desktop\image1.png" "C:\Users\Desk ...

javaeclipseselenium-webdriverautoit

Monitoring Unread Message Totals in Twilio Chat

Seeking an efficient method to retrieve and update the count of unread messages in Twilio chat. I have explored similar questions and answers on other platforms, such as this one, which suggested looping through the Channel's array and utilizing Messa ...

javascriptangulartwilio

Utilizing WebDriverWait for locating elements via their XPath

I'm puzzled about how WebDriverWait interacts with find_elements_by_xpath. Does it determine when all related elements have loaded, or does it simply wait until the page is fully loaded? I can comprehend the concept if we are dealing with a specific ...

pythonseleniumwebdriver

Is there a way for me to navigate back to my menu after running a method?

package javaapplication10; import java.util.ArrayList; import java.util.Arrays; import java.util.Scanner; /** * * @author Tom */ public class JavaApplication10 { static int randomWithRange(int min, int max) { int range = (max - min) + 1; retu ...

javaarraysloopsmenu

What is the best way to keep an image centered in the nav-bar as it decreases in size?

As I work on building a website using a template (https://github.com/learning-zone/web-templates/tree/master/victory-school-free-html5-bootstrap-template), I encountered an issue with the navigation bar. The original design appears like this: Before shrin ...

javascripthtmlcss