Tips for running Scrapy and Selenium on a webpage that utilizes angular JavaScript to serve data

Question

Tips for running Scrapy and Selenium on a webpage that utilizes angular JavaScript to serve data

I have been working on a web scraper that follows this process:

Visit site A -> click on the buy now button -> redirected to Amazon -> scrape data -> return to site A

The issue I am facing is that the site is built using AngularJS, and I am having trouble clicking the button with selenium.click()

To scrape the JavaScript page, I am using Selenium and Scrapy. The page has infinite scrolling, so I need a solution that works for such pages.

Below is the HTML snippet of the element in question:

<a class="external" href="http://www.amazon.com/dp/B01DBR53FU/?tag=097-20&amp;ascsubtag=v7_3_3_3m7_2nhz_0_x01_-srt5-" target="_blank" analytics-on="click" analytics-event="button" analytics-category="outbound" analytics-label="non-lethal-salt-firing-self-defense-gun" ng-click="click(post)" rel="nofollow">Check it out </a>

The syntax I attempted to use was:

 sel.xpath('//*[@class="button"]').click()

This resulted in an error message:

'SelectorList' object has no attribute 'click'

javascript selenium scrapy

Answer 1

Answer №1

'SelectorList' object is unable to perform the 'click' action

This error indicates that the click method cannot be used on the element selected by your xpath.

To resolve this issue in Python, you can try the following:

    element_to_click = self.driver.find_element_by_css_selector('a.external')
 driver.execute_script("arguments[0].scrollIntoView(true);",element_to_click)
    element_to_click.click()

Alternatively, you can use the xpath method like this:

element_to_click = 
    self.driver.find_element_by_xpath("//a[contains(@href,'http://www.amazon.com/dp/B01DBR53FU/?tag=097-20&amp;ascsubtag=v7_3_3_3m7_2nhz_0_x01_-srt5-')]")
#assuming that the href doesn't change.

driver.execute_script("arguments[0].scrollIntoView(true);",element_to_click)   

# scroll element into view   

element_to_click.click()

Another option is to use linkText as suggested in another answer.

Edit 3: An alternative solution is sending a JS click to this locator

driver.execute_script("arguments[0].click();",element_to_click)

Edit 4 : Pay close attention to this edit. Clicking on any gift item on this page is straightforward

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chrome_path  = 'your chromedriver path here'

driver = webdriver.Chrome(chrome_path)

driver.get("http://thisiswhyimbroke.com/gifts/gifts-for-men/")
time.sleep(5)

#clicks on the first gift article
first_article = driver.find_element_by_css_selector('#page-gifts > article:nth-child(1) > div.button > a')
first_article.click()
time.sleep(5)

All the gift articles mentioned are generated by a repeater since it's an Angular web page. They are contained within an article tag.

If you need to click on a different article button, simply adjust the article:nth-child to target the desired child and the code will function accordingly.

Prior to beginning automation, familiarize yourself with creating and utilizing locators to ensure effective automation implementation.

Answer 2

'SelectorList' object is unable to perform the 'click' action

This error indicates that the click method cannot be used on the element selected by your xpath.

To resolve this issue in Python, you can try the following:

    element_to_click = self.driver.find_element_by_css_selector('a.external')
 driver.execute_script("arguments[0].scrollIntoView(true);",element_to_click)
    element_to_click.click()

Alternatively, you can use the xpath method like this:

element_to_click = 
    self.driver.find_element_by_xpath("//a[contains(@href,'http://www.amazon.com/dp/B01DBR53FU/?tag=097-20&amp;ascsubtag=v7_3_3_3m7_2nhz_0_x01_-srt5-')]")
#assuming that the href doesn't change.

driver.execute_script("arguments[0].scrollIntoView(true);",element_to_click)   

# scroll element into view   

element_to_click.click()

Another option is to use linkText as suggested in another answer.

Edit 3: An alternative solution is sending a JS click to this locator

driver.execute_script("arguments[0].click();",element_to_click)

Edit 4 : Pay close attention to this edit. Clicking on any gift item on this page is straightforward

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chrome_path  = 'your chromedriver path here'

driver = webdriver.Chrome(chrome_path)

driver.get("http://thisiswhyimbroke.com/gifts/gifts-for-men/")
time.sleep(5)

#clicks on the first gift article
first_article = driver.find_element_by_css_selector('#page-gifts > article:nth-child(1) > div.button > a')
first_article.click()
time.sleep(5)

All the gift articles mentioned are generated by a repeater since it's an Angular web page. They are contained within an article tag.

If you need to click on a different article button, simply adjust the article:nth-child to target the desired child and the code will function accordingly.

Prior to beginning automation, familiarize yourself with creating and utilizing locators to ensure effective automation implementation.

Answer 3

Answer №2

To target elements, you have the option of utilizing CSS Selector like

".external"

Alternatively, you can also target elements by Link Text such as

"Check it out"

Answer 4

To target elements, you have the option of utilizing CSS Selector like

".external"

Alternatively, you can also target elements by Link Text such as

"Check it out"

Tips for running Scrapy and Selenium on a webpage that utilizes angular JavaScript to serve data

Answer №1

Answer №2

Similar questions

What strategies can be used to effectively structure CSS and JavaScript in a project for optimal organization?

Error: The jarfile node_modules/protractor/selenium/selenium-server-standalone-2.45.0.jar is not accessible by Grunt

Style binding for background image can utilize computed properties or data for dynamic rendering

Can you explain the distinction between Array() and [] in Javascript, and when would it be preferable to use one over the other?

Issues with Ajax Requests

How to prevent npm from being accessed through the command prompt

Explication of syntax not functioning

Interfacing shared memory between a C++ and JavaScript program

Locating the xpath of an element directly beneath the opening <body> tag

Executing Javascript on Selenium RC with PHP is a crucial skill to have in your toolkit

Embedding JSON data in a GSP page

Utilizing d3.js to implement a scatterplot with zoom functionality that focuses solely on zooming the axis without affecting

Error occurs when an arrow function is called before its function definition

The custom filter in AngularJS fails to activate when a click event occurs

Refreshing a particular <div> on a webpage while making an AJAX request

Deleting an HTML column that has a dynamic header name <th> can be achieved by following these steps

Successive vows

Unordered calling of functions in JavaScript - is it possible?

Update the CSS for InputLabel

What is the proper way to insert a line break within a string in HTML code?