Utilizing headless Chrome to automatically capture AJAX requests

Question

Utilizing headless Chrome to automatically capture AJAX requests

Chrome officially supports running the browser in headless mode, allowing for programmatic control through the Puppeteer API and/or the CRI library.

I've thoroughly explored the documentation but have not discovered a method to programmatically capture AJAX traffic from instances. I am looking to start a Chrome instance from code, navigate to a page, and access background response/request calls and raw data all within the code without using developer tools or extensions.

If you have any suggestions or examples on how this can be accomplished, I would greatly appreciate it. Thank you!

javascript ajax google-chrome google-chrome-headless puppeteer

Answer 1

Answer №1

Update

In the comment, @Alejandro highlighted that the function resourceType returns a value in lowercase.

page.on('request', request => {
    if (request.resourceType() === 'xhr')
    // do something
});

Original answer

Puppeteer's API simplifies this process:

page.on('request', request => {
  if (request.resourceType === 'XHR')
    // do something
});

You can intercept requests using setRequestInterception, but it's unnecessary here unless you plan to modify the requests.

An example of intercepting image requests is available for reference.

The definitions of resourceTypes can be found here.

Answer 2

Update

In the comment, @Alejandro highlighted that the function resourceType returns a value in lowercase.

page.on('request', request => {
    if (request.resourceType() === 'xhr')
    // do something
});

Original answer

Puppeteer's API simplifies this process:

page.on('request', request => {
  if (request.resourceType === 'XHR')
    // do something
});

You can intercept requests using setRequestInterception, but it's unnecessary here unless you plan to modify the requests.

An example of intercepting image requests is available for reference.

The definitions of resourceTypes can be found here.

Answer 3

Answer №2

After much searching, I have finally discovered a way to achieve my goal. Utilizing chrome-remote-interface (CRI) and node.js, this can be accomplished with the following minimal code snippet.

const CDP = require('chrome-remote-interface');

(async function () {

    // Make sure you have an open Chrome instance with remote debugging enabled
    // For example, run: chrome --remote-debugging-port=9222
    const protocol = await CDP({port: 9222});

    const {Page, Network} = protocol;
    await Page.enable();
    await Network.enable(); // This is necessary in order to use Network.getResponseBody below

    Page.navigate({url: 'http://localhost/'}); // Replace 'http://localhost/' with your desired URL

    const onDataReceived = async (e) => {
        try {
            let response = await Network.getResponseBody({requestId: e.requestId})
            if (typeof response.body === 'string') {
                console.log(response.body);
            }
        } catch (ex) {
            console.log(ex.message)
        }
    }

    protocol.on('Network.dataReceived', onDataReceived)
})();

Answer 4

After much searching, I have finally discovered a way to achieve my goal. Utilizing chrome-remote-interface (CRI) and node.js, this can be accomplished with the following minimal code snippet.

const CDP = require('chrome-remote-interface');

(async function () {

    // Make sure you have an open Chrome instance with remote debugging enabled
    // For example, run: chrome --remote-debugging-port=9222
    const protocol = await CDP({port: 9222});

    const {Page, Network} = protocol;
    await Page.enable();
    await Network.enable(); // This is necessary in order to use Network.getResponseBody below

    Page.navigate({url: 'http://localhost/'}); // Replace 'http://localhost/' with your desired URL

    const onDataReceived = async (e) => {
        try {
            let response = await Network.getResponseBody({requestId: e.requestId})
            if (typeof response.body === 'string') {
                console.log(response.body);
            }
        } catch (ex) {
            console.log(ex.message)
        }
    }

    protocol.on('Network.dataReceived', onDataReceived)
})();

Answer 5

Answer №3

Using Puppeteer's listeners can assist in capturing xhr responses by utilizing the response and request events.

It is advisable to first verify if request.resourceType() is either xhr or fetch.

        listener = page.on('response', response => {
            const isXhr = ['xhr','fetch'].includes(response.request().resourceType())
            if (isXhr){
                console.log(response.url());
                response.text().then(console.log)
            }
        })

Answer 6

Using Puppeteer's listeners can assist in capturing xhr responses by utilizing the response and request events.

It is advisable to first verify if request.resourceType() is either xhr or fetch.

        listener = page.on('response', response => {
            const isXhr = ['xhr','fetch'].includes(response.request().resourceType())
            if (isXhr){
                console.log(response.url());
                response.text().then(console.log)
            }
        })

Answer 7

Answer №4

const browser = await puppeteer.launch();
const page = await browser.newPage();
const clientPage = page["_client"];
clientPage.on("Network.responseReceived", event => {
  if (~event.response.url.indexOf('/api/chart/rank')) {
    console.log(event.response.url);
    clientPage.send('Network.getResponseBody', {
      requestId: event.requestId
    }).then(async response => {
      const data = response.body;
      if (data) {
        try {
          const jsonData = JSON.parse(data);

        }
        catch (e) {
        }
      }
    });
  }
});

await page.setRequestInterception(true);
page.on("request", async request => {
  request.continue();
});
await page.goto('http://www.example.com', { timeout: 0 });

Answer 8

const browser = await puppeteer.launch();
const page = await browser.newPage();
const clientPage = page["_client"];
clientPage.on("Network.responseReceived", event => {
  if (~event.response.url.indexOf('/api/chart/rank')) {
    console.log(event.response.url);
    clientPage.send('Network.getResponseBody', {
      requestId: event.requestId
    }).then(async response => {
      const data = response.body;
      if (data) {
        try {
          const jsonData = JSON.parse(data);

        }
        catch (e) {
        }
      }
    });
  }
});

await page.setRequestInterception(true);
page.on("request", async request => {
  request.continue();
});
await page.goto('http://www.example.com', { timeout: 0 });

Utilizing headless Chrome to automatically capture AJAX requests

Answer №1

Answer №2

Answer №3

Answer №4

Similar questions

How to resolve a TypeError saying "Object(...) is not a function"?

Adjusting the color of a box in Threejs through dat.gui.min.js controls

Determining the emptiness of an array in Postman using node.js

Prevent the browser back button from being used in a web application that requires CLIENT_CERT authentication

Repeated URL causes Node to redirect

Tips for populating countryList data in Form.Select component within a React.js application

Create an Oval-Shaped Image Cutout

"Encountering an issue with AngularJS where the selected ng-model value is

The scrolltop animation does not appear to be functioning properly on iOS devices

Can a single endpoint be used to provide files to web browsers and data to REST requests simultaneously?

Using a JSON file as a database for a project featuring HTML, CSS, and Vanilla JavaScript

Use ng-repeat to extract information from an array and populate it into a datalist

Scatter plot showcasing repeated values along the x-axis

JavaScript was unable to locate the requested URL on the server

I am experiencing an issue with applying responsiveFontSize() to the new variants in Material UI Typography

BookshelfJS: Establishing a One-to-One Relationship

What is the best way to display a segment of an SVG on a Canvas element?

Tips for smoothly adding new content to a jQuery AJAX autoupdating div

What is the best way to share information among Vue3 single file component instances?

Navigating an indefinite amount of state variables in React.js: A comprehensive guide