Utilizing headless Chrome to automatically capture AJAX requests

Question

Utilizing headless Chrome to automatically capture AJAX requests

Chrome officially supports running the browser in headless mode, allowing for programmatic control through the Puppeteer API and/or the CRI library.

I've thoroughly explored the documentation but have not discovered a method to programmatically capture AJAX traffic from instances. I am looking to start a Chrome instance from code, navigate to a page, and access background response/request calls and raw data all within the code without using developer tools or extensions.

If you have any suggestions or examples on how this can be accomplished, I would greatly appreciate it. Thank you!

javascript ajax google-chrome google-chrome-headless puppeteer

Answer 1

Answer №1

Update

In the comment, @Alejandro highlighted that the function resourceType returns a value in lowercase.

page.on('request', request => {
    if (request.resourceType() === 'xhr')
    // do something
});

Original answer

Puppeteer's API simplifies this process:

page.on('request', request => {
  if (request.resourceType === 'XHR')
    // do something
});

You can intercept requests using setRequestInterception, but it's unnecessary here unless you plan to modify the requests.

An example of intercepting image requests is available for reference.

The definitions of resourceTypes can be found here.

Answer 2

Update

In the comment, @Alejandro highlighted that the function resourceType returns a value in lowercase.

page.on('request', request => {
    if (request.resourceType() === 'xhr')
    // do something
});

Original answer

Puppeteer's API simplifies this process:

page.on('request', request => {
  if (request.resourceType === 'XHR')
    // do something
});

You can intercept requests using setRequestInterception, but it's unnecessary here unless you plan to modify the requests.

An example of intercepting image requests is available for reference.

The definitions of resourceTypes can be found here.

Answer 3

Answer №2

After much searching, I have finally discovered a way to achieve my goal. Utilizing chrome-remote-interface (CRI) and node.js, this can be accomplished with the following minimal code snippet.

const CDP = require('chrome-remote-interface');

(async function () {

    // Make sure you have an open Chrome instance with remote debugging enabled
    // For example, run: chrome --remote-debugging-port=9222
    const protocol = await CDP({port: 9222});

    const {Page, Network} = protocol;
    await Page.enable();
    await Network.enable(); // This is necessary in order to use Network.getResponseBody below

    Page.navigate({url: 'http://localhost/'}); // Replace 'http://localhost/' with your desired URL

    const onDataReceived = async (e) => {
        try {
            let response = await Network.getResponseBody({requestId: e.requestId})
            if (typeof response.body === 'string') {
                console.log(response.body);
            }
        } catch (ex) {
            console.log(ex.message)
        }
    }

    protocol.on('Network.dataReceived', onDataReceived)
})();

Answer 4

After much searching, I have finally discovered a way to achieve my goal. Utilizing chrome-remote-interface (CRI) and node.js, this can be accomplished with the following minimal code snippet.

const CDP = require('chrome-remote-interface');

(async function () {

    // Make sure you have an open Chrome instance with remote debugging enabled
    // For example, run: chrome --remote-debugging-port=9222
    const protocol = await CDP({port: 9222});

    const {Page, Network} = protocol;
    await Page.enable();
    await Network.enable(); // This is necessary in order to use Network.getResponseBody below

    Page.navigate({url: 'http://localhost/'}); // Replace 'http://localhost/' with your desired URL

    const onDataReceived = async (e) => {
        try {
            let response = await Network.getResponseBody({requestId: e.requestId})
            if (typeof response.body === 'string') {
                console.log(response.body);
            }
        } catch (ex) {
            console.log(ex.message)
        }
    }

    protocol.on('Network.dataReceived', onDataReceived)
})();

Answer 5

Answer №3

Using Puppeteer's listeners can assist in capturing xhr responses by utilizing the response and request events.

It is advisable to first verify if request.resourceType() is either xhr or fetch.

        listener = page.on('response', response => {
            const isXhr = ['xhr','fetch'].includes(response.request().resourceType())
            if (isXhr){
                console.log(response.url());
                response.text().then(console.log)
            }
        })

Answer 6

Using Puppeteer's listeners can assist in capturing xhr responses by utilizing the response and request events.

It is advisable to first verify if request.resourceType() is either xhr or fetch.

        listener = page.on('response', response => {
            const isXhr = ['xhr','fetch'].includes(response.request().resourceType())
            if (isXhr){
                console.log(response.url());
                response.text().then(console.log)
            }
        })

Answer 7

Answer №4

const browser = await puppeteer.launch();
const page = await browser.newPage();
const clientPage = page["_client"];
clientPage.on("Network.responseReceived", event => {
  if (~event.response.url.indexOf('/api/chart/rank')) {
    console.log(event.response.url);
    clientPage.send('Network.getResponseBody', {
      requestId: event.requestId
    }).then(async response => {
      const data = response.body;
      if (data) {
        try {
          const jsonData = JSON.parse(data);

        }
        catch (e) {
        }
      }
    });
  }
});

await page.setRequestInterception(true);
page.on("request", async request => {
  request.continue();
});
await page.goto('http://www.example.com', { timeout: 0 });

Answer 8

const browser = await puppeteer.launch();
const page = await browser.newPage();
const clientPage = page["_client"];
clientPage.on("Network.responseReceived", event => {
  if (~event.response.url.indexOf('/api/chart/rank')) {
    console.log(event.response.url);
    clientPage.send('Network.getResponseBody', {
      requestId: event.requestId
    }).then(async response => {
      const data = response.body;
      if (data) {
        try {
          const jsonData = JSON.parse(data);

        }
        catch (e) {
        }
      }
    });
  }
});

await page.setRequestInterception(true);
page.on("request", async request => {
  request.continue();
});
await page.goto('http://www.example.com', { timeout: 0 });

Utilizing headless Chrome to automatically capture AJAX requests

Answer №1

Answer №2

Answer №3

Answer №4

Similar questions

show a fresh new page using react router

Is tsconfig.json Utilized by Gatsby When Using Typescript?

Storing information using ajax and JSON through Handsontable to send to a Web API

What is the best way to integrate HybridAuth using an AJAX request?

Trouble with minified scripts on a particular server

"Exploring the power of Ajax: a guide to automatically refreshing the response every

Angular can be used to compare two arrays and display the matching values in a table

React Form with Conditional Input Options and Validation

The V-model is failing to bind properly as anticipated

How to call a function within a component from another component without encountering the "Cannot read property" error

Is there a way to incorporate a CSS file into this without explicitly mentioning the color?

Loading input datalist in Firefox postponed

Getting the value from a .sh (Shell Script) file in React: How to do it?

Retrieve the ID of the image element using Jquery from a collection of images within a div container

Having trouble with opening and closing popup windows in JavaScript while using Android frames?

Currently trapped within the confines of a Next.js 13 application directory, grappling with the implementation of a

Troubleshooting: How to resolve the issue of receiving undefined values when passing API data to a child component with Axios in React

The intricacies of the Jscript ReadLine() function

Send data through Ajax and Jquery to upload files

Error: The function window.intlTelInput is not recognized within the ReactJS framework