My current challenge involves loading additional comments by clicking on a JavaScript object and then scraping the page to collect data. To troubleshoot this issue, I am comparing the number of comments displayed before and after clicking the "load more" button within a p tag element. However, I consistently receive the same count for both instances despite the presence of additional comments on the page. Where could my code be incorrect?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.PhantomJS(executable_path='PATH_TO.../phantomjs')
driver.get('http://www.ratemyprofessors.com/ShowRatings.jsp?tid=1500075')
comments = driver.find_elements_by_tag_name('p')
print('Initial Comment Count:', len(comments))
time.sleep(1)
try:
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'loadMore')))
time.sleep(1)
finally:
comments = driver.find_elements_by_tag_name('p')
print('Updated Comment Count:', len(comments))
driver.close()
I have attempted using both 'loadMore' and 'loadmoreBlog' identifiers without success. Any insights or suggestions would be greatly appreciated.