I need help accessing and extracting data from a URL that is embedded within a specific tag. The tag in question looks like this:
<script src="http://includes.mpt-static.com/data/7CE5047496" type="text/javascript" charset="utf-8"></script>
So far, I have attempted to use Selenium to open the URL, but it just returns an empty string. It seems that when I manually click on the source URL, a page opens displaying a table of the desired data. However, pasting the URL directly into a browser results in an empty response. Additionally, each time I refresh the page, a new source URL is generated. Can someone explain why this behavior is occurring?
The URL in question is: view-source:
Below is the relevant portion of my code:
import time
from fake_useragent import UserAgent
import urllib2
import csv
from bs4 import BeautifulSoup
import json
from selenium import webdriver
#FAKE-USER_AGENT
ua = UserAgent(cache = False)
headers = {'User-Agent': ua.randome}
#SENDING REQUEST TO PRICETRACKER WEBSITE
product = 'B00N2BW2PK'
page = requests.get('http://www.mypricetrack.com/amazon/'+str(product), headers = headers)
soup = BeautifulSoup(page.text)
#print(soup.prettify())
#GETTING URL FOR DATA
data_link = []
for tag in soup.findAll('script',{'charset':'utf-8'}):
data_link = data_link + [tag['src']]
string2 = data_link[1]
print string2
#OPENING URL FOR DATA
driver = webdriver.Firefox()
driver.get(string2)
time.sleep(5)
htmlSource = driver.page_source
print htmlSource