A guide on extracting Disqus comments iframe information using Selenium WebDriver

Recently, I've been experimenting with Selenium WebDriver (Chrome) in an attempt to load a page and extract disqus comment threads. Below is the code snippet I used:

public static void Main(string[] args)
        {
            var driver = new ChromeDriver();
            driver.Navigate().GoToUrl("http://nation.com.pk/blogs/05-Apr-2016/should-qandeel-baloch-s-strip-tease-really-be-a-rallying-cause-for-liberalism");
            var userNameField = driver.FindElementById("disqus_thread").GetAttribute("outerHTML");
            Console.WriteLine(userNameField);
            Console.Read();
        }

The output generated was as follows:

<div id="disqus_thread"><iframe id="dsq-app2" name="dsq-app2" allowtransparency="true" frameborder="0" scrolling="no" tabindex="0" title="Disqus" width="100%" src="http://disqus.com/embed/comments/?base=default&amp;version=af1a2e2611136ef6c314afec2806cef3&amp;f=nawaiwaqt&amp;t_u=http%3A%2F%2Fnation.com.pk%2Fblogs%2F05-Apr-2016%2Fshould-qandeel-baloch-s-strip-tease-really-be-a-rallying-cause-for-liberalism&amp;t_d=Should%20Qandeel%20Baloch%E2%80%99s%20%E2%80%98striptease%E2%80%99%20really%20be%20a%20rallying%20cause%20for%20liberalism%3F&amp;t_t=Should%20Qandeel%20Baloch%E2%80%99s%20%E2%80%98striptease%E2%80%99%20really%20be%20a%20rallying%20cause%20for%20liberalism%3F&amp;s_o=default" style="width: 1px !important; min-width: 100% !important; border: none !important; overflow: hidden !important; height: 652px !important;" horizontalscrolling="no" verticalscrolling="no"></iframe></div>

This result came as a surprise to me since I had expected to retrieve the comments html instead. Any suggestions on how to tweak this code to obtain the comment thread? PS: There is no mention of iframe in the original page source.

javascriptc#seleniumiframedisqus

Answer №1

Here is a method to extract the iframe source code for further manipulation. Hopefully, this will be beneficial to someone else.

var browser = new ChromeDriver();
            int count = 1;
            foreach(string link in File.ReadLines(@"path\to\FileLinks.txt"))
            {
                browser.Navigate().GoToUrl(link);
                browser.SwitchTo().Frame("dsq-app2");
                var code = browser.PageSource;
                File.WriteAllText(@"path\to\Comments\"+count+".html", code);
                Console.WriteLine(link);
                count++;
            }
            browser.Close();

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Every Dynamic Post automatically defaults to the initial object

I am currently developing an app that retrieves feeds from a Wordpress site and displays individual posts in a jQuery mobile list format. Here is the JavaScript code I am using: $(document).ready(function () { var url = 'http://howtodeployit.com/ ...

javascriptjquerywordpress

What is the Firefox Gecko equivalent of the --disable-dev-shm-usage flag found in Chrome?

Welcome to the Docker environment featuring a Docker Container running on ubuntu:18.10 with geckodriver-v0.23.0-linux64 and selenium-3.14.1. While using Chrome, I encountered resource allocation issues in my Docker Container. The problem was resolved by a ...

seleniumdockerselenium-webdriver

Using an array to substitute a string in JavaScript

xyz1 xyz2 xyz3 xyz4 xyz5 xyz6 xyz7 xyz8 xyz9 Given the CSV above, transform the sentence below by removing the double quotes and replacing them with the corresponding values from the CSV: "" is going with "" to "" for something to know. ...

javascriptphparrays

What is the process for calculating collisions when creating a game using the Build an HTML5 Game book?

I'm having trouble grasping the concept of authors using login when determining collisions between two circles (bubbles). You can find more information in the Calculating collisions section. The author explains: The bubble being fired follows a set ...

javascriptmath2d-games

Node JS | Error: Unable to access the property 'first_name' of an undefined object

Just getting started with my MEAN application and I've hit a roadblock while trying to add data to the database. Can someone please help me find a solution for this? This is the main file, the entry point of the application. //Importing modules var ...

javascriptjsonnode.jsmongodb

What reasons could be preventing my Class Library from being referenced?

My class library file doesn't seem to be recognized when I add it to the reference and bin folder. using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Data; namespace SecuritySettings { public clas ...

c#.netasp.netclass

What is the best way to activate inline Javascript that has been added to my webpage? (Using ASP.NET webforms)

One interesting challenge I am facing involves a page section that is dynamically generated by the server. This particular section contains inline JavaScript code written by the server itself (This is ASP.NET webforms). The purpose of this JavaScript is to ...

javascriptjqueryhtmlcssasp.net

Sending properties to a template literal within a loop

Currently, I am working on developing a dynamic select box. The challenge I am facing is in combining the row source data object with the string key or display name. Any thoughts or suggestions would be greatly appreciated. Thank you! const { rowSourceD ...

javascripttemplatesreactjsecmascript-6literals

Organizing communications in NodeJS messaging application

My latest project involves creating a chat room using NodeJS and socket.io which allows multiple users to connect with unique usernames. Everything is running smoothly, except for one minor issue. I want the messages sent by me to appear in a different co ...

javascriptjqueryhtmlcsschat

You need a function property for the child component to be updated

I recently encountered an unusual issue with my React components that has left me puzzled. Here is a condensed version of the app: class Child extends React.Component { componentDidUpdate(prev) { if (!prev.isActive && this.props.isActive) ...

javascriptreactjsreduxreact-redux

Waiting for the execution of the loop to be completed before proceeding - Typescript (Angular)

There's a code snippet triggered on an HTML page when clicked: public salaryConfirmation() { const matDialogConfig: MatDialogConfig = _.cloneDeep(GajiIdSettings.DIALOG_CONFIG); this.warningNameList = []; for(let i=0; i < this.kelolaDat ...

javascriptangulartypescript

JavaScript and CSS Fusion - achieving a mesmerizing 3D image wrapping effect

I am currently attempting to achieve a 3D wrap effect for images on my website, similar to the one showcased in this image. The desired effect needs to closely resemble the image linked above. Below is the CSS code I am using to try to implement this eff ...

javascriptjqueryhtmlcssimage

What is the target of express.static(__dirname) in ExpressJs?

var express = require('express'); var app = express(); port = process.argv[2] || 8000; app.configure(function () { app.use( "/", express.static(__dirname) ); }); app.listen(port); By removing the following line, I enco ...

javascriptnode.jsexpress

Neither selenium nor bs4 are able to locate a div element on the webpage

I am facing a challenge while attempting to scrape data from a Craigslist results page. Despite being able to view the elements I need using Dev Tools, both bs4 and selenium fail to locate them on the page. The search results are contained within list item ...

pythonseleniumselenium-webdriverweb-scrapingbeautifulsoup

Angular2 waits for the JSON file to be fully loaded before proceeding

At the moment, I am loading a JSON file in the following way: translation.service.ts @Injectable() export class TranslationService { private _messages = []; constructor(private _http: Http) { var observable = this._http.get("/app/i18n/l ...

javascriptjsontypescriptangular

"Storing" the chosen option from a dropdown list into a specific property

I'm currently working on a small "Rent a car" application using ASP.NET MVC. My goal is to create a page for administrators where they can add cars to be offered to customers. On this page, they should be able to input details such as the picture of ...

c#asp.netasp.net-mvcdrop-down-menu

Spinner Vue Displays while Loading Image from a URL

I am trying to display a loader spinner while an image is loading, but I am having trouble implementing this. Even after debugging and getting true and false values in the console, the spinner is still not showing up. <template> <div class=&q ...

javascriptvue.js

Organizing outcomes "equitably" through LINQ

I am currently faced with the task of assigning system users to accounts. The allocation process needs to be as fair as possible, where each system user should be assigned an equal number of accounts. For instance, if there are 40 accounts and 20 system us ...

c#.netlinq

What causes the error "Why am I receiving a "Cannot read property 'length' of undefined" when looping through a pug template?

Currently, I am in the process of developing a project using Node and Express. My objective is to have the home page display signup and login links in the nav bar when the user is not logged in. Initially, everything seemed to be working fine, and I was ab ...

javascriptnode.jsexpresspug

Executing a simulated onClick event in jQuery

Attempting to create a simulated onclick event on a drop-down menu has proved challenging for me. An IE object is being used to navigate to a page, where I need to modify a dropdown menu that contains an onchange event: $('select[name="blah"]') ...

javascriptjquery