Running JavaScript on a change function while performing web scraping with Selenium using Java proves to be challenging

Task at hand:

1) Navigate to the following URL:

2) Choose "FO Stocks" from the selection box to display a new table of FO Stocks.

3) Sort the table in descending order by clicking on the "FFM Column" header.

Extract the top 5 rows from the table.

My approach:

Utilizing Selenium Webdriver with Java

1) Successfully accessed the URL in FireFox, attempted loading in Chrome which took longer to load the page entirely.

2) Upon webpage loading, I adjusted the Selection Box Value to "FO" by utilizing the ID of the select box and specifying the value of the option. The code snippet below demonstrates this.

Select dropdown = new Select(driver.findElement(By.id("selId")));
dropdown.selectByValue("fo");

Issue: The table does not update values according to the selected option possibly due to the onchange() function in JavaScript not executing/enabling after changing the value of the selection box on that page.

3) Seeking advice on how to click on this column using Selenium Webdriver (all tips are appreciated).

Code snippet as follows:

public class FirstSeleniumTest {
WebDriver driver;

public void waitForLoad(WebDriver driver) {
System.out.print("waiting for javascript successfully");
new WebDriverWait(driver, 120).until((ExpectedCondition<Boolean>) wd ->
((JavascriptExecutor) wd).executeScript("return document.readyState").equals("complete"));
}

public void launchBrowser() {
// System.setProperty("webdriver.chrome.driver", "C:\\Downloads\\chromedriver_win32\\chromedriver.exe");
// driver = new ChromeDriver();
System.setProperty("webdriver.gecko.driver","C:\\Downloads\\geckodriver-v0.26.0-win64\\geckodriver.exe");
DesiredCapabilities dcap = new DesiredCapabilities();
dcap.setCapability("pageLoadStrategy", "eager");
FirefoxOptions opt=new FirefoxOptions();
opt.merge(dcap);
WebDriver driver = new FirefoxDriver(opt);
System.out.print("Driver loaded successfully");
driver.get("https://www1.nseindia.com/live_market/dynaContent/live_watch/pre_open_market/pre_open_market.htm");
waitForLoad(driver);
Select dropdown = new Select(driver.findElement(By.id("selId")));
waitForLoad(driver);
dropdown.selectByValue("fo");
waitForLoad(driver);
}

public static void main(String[] args)
{
FirstSeleniumTest ft=new FirstSeleniumTest();
ft.launchBrowser();
}
}

Answer №1

To change the selection, consider removing the attribute from the initial element and adding the "selected" attribute to the desired option using jquery/javascript. Then, activate the on change method for the element.

Answer №2

Attempt to activate it by using the following:

$('selId').trigger('change')

Answer №3

Here are the steps that have proven to be effective for me:

  1. To begin, manually launch Chrome from a custom user directory as demonstrated in this image:

    1. Navigate to the Chrome browser exe path, which in Windows is typically located at
      C:\Program Files (x86)\Google\Chrome\Application\
    2. Open CMD in this path and execute the following command:
      chrome.exe --user-data-dir="C:\selenium\AutomationProfile"
      , where the directory C:\selenium\AutomationProfile is your custom user data directory
    3. Manually launch your desired URL in this browser
    4. Wait for the webpage to load completely before closing the browser
  2. Next, use Selenium to launch the browser from the custom user-data-dir with the following code snippet:

public class Test {

    
    public static void main(String[] args) throws InterruptedException {

        System.setProperty("webdriver.chrome.driver", "Your Driver path");
        ChromeOptions option=new ChromeOptions();
        //arguments to launch chrome with Custom user dir path
        option.addArguments("--user-data-dir=C:\\selenum\\AutomationProfile\\");
        WebDriver driver = new ChromeDriver(option);
        
        driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
        driver.get("https://www1.nseindia.com/live_market/dynaContent/live_watch/pre_open_market/pre_open_market.htm"); // WebElement
        Select dropdown = new Select(driver.findElement(By.id("selId")));
        dropdown.selectByValue("fo");
        Thread.sleep(2000);
        dropdown.selectByValue("sme");
        Thread.sleep(2000);
        dropdown.selectByValue("niftybank");
        Thread.sleep(2000);
        System.out.println(driver.getTitle());
        
    }

}

When handling dropdowns, it's important to use necessary webdriver wait methods to scrape table data instead of relying on Thread.sleep.

The reason for launching Chrome in a custom user profile is to ensure faster website loading and avoid any client-side browser blocking that may occur when launched from Selenium, resulting in delayed loading times and no change in data based on dropdown values.

I hope you find these tips helpful!

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Add the URL link according to the specific domain name

I am looking for a way to attach additional URL parameters to any links that contain example.com on a webpage. The current script I have only works if the link does not already have parameters attached to it. <script> document.body.innerHTML = d ...

Verifying text content in an input field using Protractor

I am currently working on a basic protractor test to verify if a specific input tag contains text after a button is clicked. After trying various methods, I am now attempting to utilize protractor.ExpectedConditions to validate the presence of text. Below ...

Continuous Div Carousel with Looping in jQuery

I have developed a basic carousel to cycle through a series of divs, but I am facing some issues. I want the carousel to loop continuously so that after reaching "slide 07," it goes back to "slide 01." Unlike using a carousel plugin, I prefer having an ove ...

Next.js presents a challenge with micro frontend routing

In the process of developing a micro frontend framework, I have three Next.js projects - app1, app2, and base. The role of app1 and app2 is as remote applications while base serves as the host application. Configuration for app1 in next.config.js: const ...

Display recommendations as soon as a specific character is entered in the text box

Issue Description: I am looking to implement a specific feature using Javascript or JQuery. When a user types a designated character, such as the @ symbol, in a text box, I would like the values of an array to be displayed in a drop-down menu. For example, ...

The issue with Selenium automation arises when attempting to interact with a popup window

Currently, I am utilizing Selenium 2.x WebDriver to automate a Java project. However, during the automation process, there is an issue when a "pop up window" appears after clicking a submit button on a particular page, causing the automation to halt. Snip ...

The NextJS API is now pointing to the index.js file rather than the [id].js file

I am currently setting up an API in NextJS. In my /classes folder, I have index.js and [id].js files. The purpose of /classes/ is to retrieve all classes from the database or add a new class. The purpose of /classes/[id] is to fetch a specific class, upda ...

Can Selenium be used with Universal Windows Platform (UWP) applications?

Utilizing Selenium in my UWP application has proven challenging due to the WRC runtime. Unfortunately, I am unable to install the Selenium.WebDriver package designed for .NET Framework 3.5 and 4.0, as well as CoreCompat.Selenium.WebDriver which targets .NE ...

Launching the Node.js application on Heroku resulted in encountering an issue: "Application error - There was a problem within the application

When I access the browser using http://localhost:8080/, I can see the message Hello World with Express. I am currently trying to deploy this application on Heroku. I have followed the tutorial provided by Heroku. 1) Create a new app 2) Choose an App name ...

Using selenium and request to fetch data from the following page

Looking to access the product data on the next page? Here is where you can find it: Interestingly, in the HTML code, the Next button is not represented by a button element, but rather a span tag. Upon inspecting the developer tools, I observed a tag that ...

The form reset function came back as null in the results

I'm attempting to reset the form inputs with the following code: $("#form_employee_job")[0].reset(); Even after executing the code, the inputs remain filled and the console shows undefined What am I overlooking? https://i.sstatic.net/sqzzZ.jpg ...

The functionality of EnableConfigServer is currently facing issues when used with native location in Spring Boot

I'm in the process of setting up a configuration server for all the properties within our application using @EnableConfigServer in Spring Boot. Take a look at the code snippet below: @EnableConfigServer @SpringBootApplication public class ConfigServe ...

What is the method to validate user input in Java?

I need help figuring out how to make the program react when the user inputs "Start," but I'm struggling to capture what they actually wrote. Here's my initial attempt: import java.util.Scanner; public class Suhwag { public static v ...

What are some techniques for locating an element in Selenium when it has no tags?

I'm facing a challenge in my code where I need to ensure that the message "Product successfully added to your shopping cart" appears after adding an item to the cart. Here, there is no specific tag enclosing the text, making it tricky to target for ve ...

How can I set up flat-pickr for date and time input with Vue.js?

I'm currently working with flat-pickr. Let's dive into the configuration part of it. I have a start date field and an end date field. My goal is to make it so that when a time is selected in the start date field, it defaults to 12h AM, and when a ...

Keeping an eye out for modifications within the ng-repeat when updating mongoose from a separate controller

Recently, I encountered a very specific issue that I need help with resolving. Here's the scenario: I have a department controller that loads a collection from MongoDB and displays it in a table using ng-repeat. Each document in the collection has an ...

Order is preserved with Java String's hashcode, allowing for hash-based partitioning

After reviewing the function in the Java API documentation, it appears that it does indeed preserve order. The formula being s[0]*31^(n-1) + s[1]*31^(n-2) + ... + s[n-1]. I have also tested this with some examples and found that the hash codes are in a sim ...

Using Selenium together with Mocha/Chai to determine the absence of an item in the DOM and return false

In my current project, I am using selenium-webdriver with Mocha & Chai in a JavaScript environment. I am trying to determine if a user is logged in by checking for the presence of a specific element in the DOM. If the user is logged in, then the following ...

Dealing with website links in Next.js and Chakra-UI: Tips and Tricks

When incorporating react linkify directly with chakra-ui components such as Text, the links cannot be managed. Issue Example import Linkify from 'react-linkify'; import {Box, Text} from '@chakra-ui/react'; export default function Usag ...

Issue with Selenium C# Web Driver 4.11.0 functionality on deployment server

After updating the selenium C# web driver package from version 4.2 to 4.11, we started utilizing the new features introduced in selenium 4.6 onwards. One of the notable changes is that we no longer need to specify the selenium driver path explicitly as sel ...