Managing JavaScript with Scrapy

Question

Managing JavaScript with Scrapy

Spider for reference:

import scrapy
from scrapy.spiders import Spider
from scrapy.selector import Selector
from script.items import ScriptItem

class RunSpider(scrapy.Spider):
    name = "run"
    allowed_domains = ["stopitrightnow.com"]
    start_urls = (
        'http://www.stopitrightnow.com/',
    )

    def parse(self, response):
        for widget in response.xpath('//div[@class="shopthepost-widget"]'):
            item = ScriptItem()
            item['url'] = widget.xpath('.//a/@href').extract()
            url = item['url']
            yield item

When executing this code in terminal, the output is as follows:

2015-08-21 14:23:51 [scrapy] DEBUG: Scraped from <200 http://www.stopitrightnow.com/>
{'url': []}
<div class="shopthepost-widget" data-widget-id="708473">
<script type="text/javascript">!function(d,s,id){var e, p = /^http:/.test(d.location) ? 'http' : 'https';if(!d.getElementById(id)) {e = d.createElement(s);e.id = id;e.src = p + '://' + 'widgets.rewardstyle.com' + '/js/shopthepost.js';d.body.appendChild(e);}if(typeof window.__stp === 'object') if(d.readyState === 'complete') {window.__stp.init();}}(document, 'script', 'shopthepost-script');</script><br>

This is the HTML structure:

<div class="shopthepost-widget" data-widget-id="708473" data-widget-uid="1"><div id="stp-55d44feabd0eb" class="stp-outer stp-no-controls">
    <a class="stp-control stp-left stp-hidden">&lt;</a>
    <div class="stp-inner" style="width: auto">
        <div class="stp-slide" style="left: -0%">
                        <a href="http://rstyle.me/iA-n/zzhv34c_" target="_blank" rel="nofollow" class="stp-product " data-index="0" style="margin: 0 0px 0 0px">
                <span class="stp-help"></span>
                <img src="//images.rewardstyle.com/img?v=2.13&amp;p=n_24878713">
                            </a>
                        <a href="http://rstyle.me/iA-n/zzhvw4c_" target="_blank" rel="nofollow" class="stp-product " data-index="1" style="margin: 0 0px 0 0px">
                <span class="stp-help"></span>
                <img src="//images.rewardstyle.com/img?v=2.13&amp;p=n_24878708">

It appears that there is a problem when trying to activate the JavaScript in the code provided. While JavaScript cannot run in Scrapy, there might be a way to access those links. I have looked into using Selenium but am struggling to implement it successfully.

Any assistance or guidance is greatly appreciated.

javascript selenium web-scraping scrapy

Answer 1

Answer №1

I managed to solve the issue using ScrapyJS.

Simply follow the setup instructions provided in the official documentation as well as checking out this helpful answer.

This is the test spider code that I implemented:

# -*- coding: utf-8 -*-
import scrapy


class TestSpider(scrapy.Spider):
    name = "run"
    allowed_domains = ["stopitrightnow.com"]
    start_urls = (
        'http://www.stopitrightnow.com/',
    )

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={
                'splash': {
                    'endpoint': 'render.html',
                    'args': {'wait': 0.5}
                }
            })

    def parse(self, response):
        for widget in response.xpath('//div[@class="shopthepost-widget"]'):
            print widget.xpath('.//a/@href').extract()

Furthermore, here are the results displayed on the console:

[u'http://rstyle.me/iA-n/7bk8r4c_', u'http://rstyle.me/iA-n/7bk754c_', u'http://rstyle.me/iA-n/6th5d4c_', u'http://rstyle.me/iA-n/7bm3s4c_', u'http://rstyle.me/iA-n/2xeat4c_', u'http://rstyle.me/iA-n/7bi7f4c_', u'http://rstyle.me/iA-n/66abw4c_', u'http://rstyle.me/iA-n/7bm4j4c_']
[u'http://rstyle.me/iA-n/zzhv34c_', u'http://rstyle.me/iA-n/zzhvw4c_', u'http://rstyle.me/iA-n/zwuvk4c_', u'http://rstyle.me/iA-n/zzhvr4c_', u'http://rstyle.me/iA-n/zzh9g4c_', u'http://rstyle.me/iA-n/zzhz54c_', u'http://rstyle.me/iA-n/zwuuy4c_', u'http://rstyle.me/iA-n/zzhx94c_']

Answer 2

I managed to solve the issue using ScrapyJS.

Simply follow the setup instructions provided in the official documentation as well as checking out this helpful answer.

This is the test spider code that I implemented:

# -*- coding: utf-8 -*-
import scrapy


class TestSpider(scrapy.Spider):
    name = "run"
    allowed_domains = ["stopitrightnow.com"]
    start_urls = (
        'http://www.stopitrightnow.com/',
    )

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={
                'splash': {
                    'endpoint': 'render.html',
                    'args': {'wait': 0.5}
                }
            })

    def parse(self, response):
        for widget in response.xpath('//div[@class="shopthepost-widget"]'):
            print widget.xpath('.//a/@href').extract()

Furthermore, here are the results displayed on the console:

[u'http://rstyle.me/iA-n/7bk8r4c_', u'http://rstyle.me/iA-n/7bk754c_', u'http://rstyle.me/iA-n/6th5d4c_', u'http://rstyle.me/iA-n/7bm3s4c_', u'http://rstyle.me/iA-n/2xeat4c_', u'http://rstyle.me/iA-n/7bi7f4c_', u'http://rstyle.me/iA-n/66abw4c_', u'http://rstyle.me/iA-n/7bm4j4c_']
[u'http://rstyle.me/iA-n/zzhv34c_', u'http://rstyle.me/iA-n/zzhvw4c_', u'http://rstyle.me/iA-n/zwuvk4c_', u'http://rstyle.me/iA-n/zzhvr4c_', u'http://rstyle.me/iA-n/zzh9g4c_', u'http://rstyle.me/iA-n/zzhz54c_', u'http://rstyle.me/iA-n/zwuuy4c_', u'http://rstyle.me/iA-n/zzhx94c_']

Answer 3

Answer №2

To avoid using javascript like Alecxe's method, you can manually inspect where the page is sourcing its content from and integrate that functionality yourself (view this SO thread for more insight).

In this particular scenario, we discover the following: https://i.sstatic.net/4hq90.png

For

<div class="shopthepost-widget" data-widget-id="708473">

, JavaScript is utilized to insert the link "widgets.rewardstyle.com/stps/708473.html".

You have the option of managing this on your own by manually creating requests for these URLs:

def parse(self, response):
    for widget in response.xpath('//div[@class="shopthepost-widget"]'):
        widget_id = widget.xpath('@data-widget-id').extract()[0]
        widget_url = "http://widgets.rewardstyle.com/stps/{id}.html".format(id=widget_id)
        yield Request(widget_url, callback=self.parse_widget)

def parse_widget(self, response):
    for link in response.xpath('//a[contains(@class, "stp-product")]'):
        item = JavasItem()  # Name recommended by the author, see comments below
        item['link'] = links.xpath("@href").extract()
        yield item

    # Proceed with any additional actions desired on the opened page.

If you require maintaining a connection between these widgets and their corresponding post/article, transmit that data within the request using meta.

UPDATE: parse_widget() has been revised. It now employs contains to identify the class due to a trailing space. Alternatively, you could use a CSS selector if you prefer.

Answer 4

To avoid using javascript like Alecxe's method, you can manually inspect where the page is sourcing its content from and integrate that functionality yourself (view this SO thread for more insight).

In this particular scenario, we discover the following: https://i.sstatic.net/4hq90.png

For

<div class="shopthepost-widget" data-widget-id="708473">

, JavaScript is utilized to insert the link "widgets.rewardstyle.com/stps/708473.html".

You have the option of managing this on your own by manually creating requests for these URLs:

def parse(self, response):
    for widget in response.xpath('//div[@class="shopthepost-widget"]'):
        widget_id = widget.xpath('@data-widget-id').extract()[0]
        widget_url = "http://widgets.rewardstyle.com/stps/{id}.html".format(id=widget_id)
        yield Request(widget_url, callback=self.parse_widget)

def parse_widget(self, response):
    for link in response.xpath('//a[contains(@class, "stp-product")]'):
        item = JavasItem()  # Name recommended by the author, see comments below
        item['link'] = links.xpath("@href").extract()
        yield item

    # Proceed with any additional actions desired on the opened page.

If you require maintaining a connection between these widgets and their corresponding post/article, transmit that data within the request using meta.

UPDATE: parse_widget() has been revised. It now employs contains to identify the class due to a trailing space. Alternatively, you could use a CSS selector if you prefer.

Managing JavaScript with Scrapy

Answer №1

Answer №2

Similar questions

Discover the process of incorporating secondary links into your dabeng organizational chart!

Guide to transferring filtered data to the controller

Enhance User Experience with ngDialog Modal's Multi-pane Feature for Angular

Experiencing problems with integrating Slim framework and AngularJS, such as encountering a 404 error

With a GroupAvatar, my Avatar named "max" likes to dance to the beat of its own drum rather than following the rules of my

If the LocalStorage value is empty, relocate to a different location

The footer section of my website seems to have gone missing in the HTML/CSS coding

Selenium's continue session feature is unable to retrieve attributes using the getAttribute

Canvas only draws outside the table, with the exception of the first one

Error in Angular2: "Promise" name not found despite successful installation of global dependencies

Invoke a fresh constructor within a $get method in Angular's provider

Python WebDriver in Selenium Unable to Locate Element

Utilizing NodeJS Puppeteer and the scrapedin framework to extract user email addresses from Linked-in profiles through web scraping techniques

What should be done if an image is not wide enough to stretch it to match the width of the window?

Navigating through different pages and encountering a StaleElementReferenceException with Python Selenium

Why does routing function correctly in a browser with AngularUI Router and Ionic, but not in Ionic View?

Is there a way to Clear All domain cookies using Selenium?

It seems that Firefox is ignoring the word-wrap style when the class of a child element is changed

Transmit data via XMLHttpRequest in smaller portions or through a ReadableStream to minimize memory consumption when handling large datasets

I am working with Vue.js 2.0 and attempting to send an event from a `child component`