Managing URLs in HtmlUnitDriver to either block or allow access.It's important

Question

Managing URLs in HtmlUnitDriver to either block or allow access.It's important

Blocking URLs in PhantomJS and GhostDriver is a simple process. Begin by initializing the driver with a handler:

PhantomJSDriver driver = new PhantomJSDriver();
driver.executePhantomJS(loadFile("/phantomjs/handlers.js"))

Then, set up the handler:

this.onResourceRequested = function (requestData, networkRequest) {
    var allowedUrls = [
        /https?:\/\/localhost.*/,
        /https?:\/\/.*\.example.com\/?.*/
    ];
    var disallowedUrls = [
        /https?:\/\/nonono.com.*/
    ];

    function isUrlAllowed(url) {
        function matches(url) {
            return function(re) {
                return re.test(url);
            };
        }
        return allowedUrls.some(matches(url)) && !disallowedUrls.some(matches(url));
    }

    if (!isUrlAllowed(requestData.url)) {
        console.log("Aborting disallowed request (# " + requestData.id + ") to url: '" + requestData.url + "'");
        networkRequest.abort();
    }
};

I haven't discovered an effective method to achieve this with HtmlUnitDriver. The ScriptPreProcessor mentioned in How to filter javascript from specific urls in HtmlUnit, utilizes WebClient instead of HtmlUnitDriver. Any suggestions?

javascript selenium phantomjs htmlunit htmlunit-driver

Answer 1

Answer №1

Customize HtmlUnitDriver by extending it and creating a ScriptPreProcessor for editing content, as well as a HttpWebConnection for controlling access to URLs:

public class FilteringHtmlUnitDriver extends HtmlUnitDriver {

    private static final String[] ALLOWED_URLS = {
            "https?://localhost.*",
            "https?://.*\\.yes.yes/?.*",
    };
    private static final String[] DISALLOWED_URLS = {
            "https?://spam.nono.*"
    };

    public FilteringHtmlUnitDriver(DesiredCapabilities capabilities) {
        super(capabilities);
    }

    @Override
    protected WebClient modifyWebClient(WebClient client) {
        WebConnection connection = applyFilteringToWebConnection(client);
        ScriptPreProcessor preProcessor = applyFilteringToPreProcessor();

        client.setWebConnection(connection);
        client.setScriptPreProcessor(preProcessor);

        return client;
    }

    private ScriptPreProcessor applyFilteringToPreProcessor() {
        return (htmlPage, sourceCode, sourceName, lineNumber, htmlElement) -> manipulateContent(sourceCode);
    }

    private String manipulateContent(String sourceCode) {
        return sourceCode.replaceAll("foo", "bar");        
    }

    private WebConnection applyFilteringToWebConnection(WebClient client) {
        return new HttpWebConnection(client) {
            @Override
            public WebResponse getResponse(WebRequest request) throws IOException {
                String url = request.getUrl().toString();
                WebResponse emptyResponse = new WebResponse(
                        new WebResponseData("".getBytes(), SC_OK, "", new ArrayList<>()), request, 0);

                for (String disallowed : DISALLOWED_URLS) {
                    if (url.matches(disallowed)) {
                        return emptyResponse;
                    }
                }
                for (String allowed : ALLOWED_URLS) {
                    if (url.matches(allowed)) {
                        return super.getResponse(request);
                    }
                }
                return emptyResponse;
            }
        };
    }
}

This implementation allows for both modifying content and restricting access to specific URLs.

Answer 2

Customize HtmlUnitDriver by extending it and creating a ScriptPreProcessor for editing content, as well as a HttpWebConnection for controlling access to URLs:

public class FilteringHtmlUnitDriver extends HtmlUnitDriver {

    private static final String[] ALLOWED_URLS = {
            "https?://localhost.*",
            "https?://.*\\.yes.yes/?.*",
    };
    private static final String[] DISALLOWED_URLS = {
            "https?://spam.nono.*"
    };

    public FilteringHtmlUnitDriver(DesiredCapabilities capabilities) {
        super(capabilities);
    }

    @Override
    protected WebClient modifyWebClient(WebClient client) {
        WebConnection connection = applyFilteringToWebConnection(client);
        ScriptPreProcessor preProcessor = applyFilteringToPreProcessor();

        client.setWebConnection(connection);
        client.setScriptPreProcessor(preProcessor);

        return client;
    }

    private ScriptPreProcessor applyFilteringToPreProcessor() {
        return (htmlPage, sourceCode, sourceName, lineNumber, htmlElement) -> manipulateContent(sourceCode);
    }

    private String manipulateContent(String sourceCode) {
        return sourceCode.replaceAll("foo", "bar");        
    }

    private WebConnection applyFilteringToWebConnection(WebClient client) {
        return new HttpWebConnection(client) {
            @Override
            public WebResponse getResponse(WebRequest request) throws IOException {
                String url = request.getUrl().toString();
                WebResponse emptyResponse = new WebResponse(
                        new WebResponseData("".getBytes(), SC_OK, "", new ArrayList<>()), request, 0);

                for (String disallowed : DISALLOWED_URLS) {
                    if (url.matches(disallowed)) {
                        return emptyResponse;
                    }
                }
                for (String allowed : ALLOWED_URLS) {
                    if (url.matches(allowed)) {
                        return super.getResponse(request);
                    }
                }
                return emptyResponse;
            }
        };
    }
}

This implementation allows for both modifying content and restricting access to specific URLs.

Managing URLs in HtmlUnitDriver to either block or allow access.It's important

Answer №1

Similar questions

Transmitting various pieces of information using AJAX

Leveraging server-side functionality with JavaScript

Ways to extract particular items from a JSON array and store them in a JavaScript array

Quickest method for skimming through an extremely lengthy document beginning at any specified line X

Choosing a language - Sending dropdown selection from HTML5 to PHP

Dealing with a Node and Express server can be tricky, especially when trying to proxy a POST request along with parameters. You might encounter the error

Utilizing Firebase login to integrate with Facebook API

Error message: Unable to locate Bootstrap call in standalone Angular project after executing 'ng add @angular/pwa' command

What is the best way to implement a calendar view for selecting time periods on an HTML page?

A guide on how to export column headers along with table data from a website to Excel using a webdriver

What is the process for creating a callback wrapper?

Disable the swipe feature on a Bootstrap carousel to prevent users from navigating through slides on mobile devices. The attribute data-touch="

Unable to delete a JSON object containing an empty value

Interact with elements using Selenium with dynamic target IDs and AJAX

The appearance of Recaptcha buttons is unattractive and seemingly impossible to

Looking up a destination with the Google Places API

Right now, I am sending out 3 GET requests for JSON files using Axios. I wonder if they are being loaded simultaneously or one after the other

Tips for resolving the 'IHasInputDevices' warning in Selenium: Use the Actions or ActionsBuilder class for simulating keyboard and mouse input

Vue.js throws an error because it is unable to access the "title" property of an undefined value

Is there a way to have the span update even if the input stays the same? Currently, it only changes when the input is different