When attempting to access a webpage using a GET request, a string is returned but unfortunately I

As part of my project to create a custom Google Chrome extension, I encountered an interesting challenge. When I perform a GET request on the following web page URL:

https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION%5E27675&maxBedrooms=2&minBedrooms=2&sortType=6&propertyTypes=&mustHave=&dontShow=&furnishTypes=&keywords=
, I receive the HTML response from the webpage as expected. The website I am accessing does not provide an API, and web scraping is not an option due to certain constraints.

The issue arises when I try to split the string I received at a specific point, namely bis_skin_checked. Surprisingly, despite this term being present in the string, the split operation returns an array with only one element, indicating that no match was found. I have attempted various methods such as eliminating spaces and line breaks, but none have proved successful so far. Below is the code snippet for my GET request:

function getNewPage(url) {
    let returnedValue = fetch(url, {
        method: 'GET',
        headers: {
            'Content-Type': 'text/html',
        },
    })
    .then(response => response.text())
    .then(text => {
        return text
    })

    return returnedValue
}

Following this, I move on to resolve the promise associated with returnedValue:

let newURL = getHousePrices(currentUrl) // Obtain Promise object representing the new page content

newURL.then(function(value) { // Resolve the promise and perform desired actions
    console.log(value.split('bis_skin_checked').length)
})

I then proceed to manipulate the retrieved string, which resembles the data shown in the image accessed via the following link (as direct text extraction is not feasible):

Image Link To API Request result

Answer №1

If you're looking to retrieve home values based on specific search criteria, there's a more efficient method than scraping raw text data. By analyzing the site's network requests, you can make adjustments to extract the necessary data directly without resorting to HTML scraping.

I've developed a solution that enables you to dynamically input your desired parameters into the getHomes() function. You can utilize the default parameters as a starting point and customize the request to suit different scenarios.

To implement this solution, install it below and execute the getHomes() function from the service worker.

You can watch a concise video tutorial I created to understand how the solution works:

--- manifest.JSON ---
{
    "name": "UK Housing - Stackoverflow",
    "description": "Example for how to make network requests and mimic them in background.js to avoid web scraping raw text",
    "version": "1.0.0",
    "manifest_version": 3,
    "background": {
        "service_worker": "background.js"
    },
    "host_permissions": [
        "*://*.rightmove.co.uk/*"
    ]
}
--- background.js ---
async function getHomes(passedParams) {

    const newParams = passedParams ? passedParams : {}; // set to an empty object if no new params passed - avoid error in object.entries loop.

    var params = {
        "locationIdentifier": "REGION%5E27675",
        "maxBedrooms": "2",
        "minBedrooms": "1",
        "numberOfPropertiesPerPage": "25",
        "radius": "0.0",
        "sortType": "6",
        "index": "0",
        "viewType": "LIST",
        "channel": "BUY",
        "areaSizeUnit": "sqft",
        "currencyCode": "GBP",
        "isFetching": "false"
    }

    Object.entries(params).forEach(([key, value]) => {
        if (newParams[key]) params[key] = newParams[key];
    });

    const rightMoveAPISearch = `https://www.rightmove.co.uk/api/_search?
        locationIdentifier=${params['locationIdentifier']}
        &maxBedrooms=${params['maxBedrooms']}
        &minBedrooms=${params['minBedrooms']}
        &numberOfPropertiesPerPage=${params['numberOfPropertiesPerPage']}
        &radius=${params['radius']}
        &sortType=${params['sortType']}
        &index=${params['index']}
        &viewType=${params['viewType']}
        &channel=${params['channel']}
        &areaSizeUnit=${params['areaSizeUnit']}
        &currencyCode=${params['currencyCode']}
        &isFetching=${params['isFetching']}
    `.replace(/\s/g, '');

    const data = await
        fetch(rightMoveAPISearch, {
            "method": "GET",
        })
        .then(data => data.json())
        .then(res => { return res })
    
    if (data.resultCount) {
        console.log('\x1b[32m%s\x1b[0m', 'Request successful! Result count: ', parseInt(data.resultCount));
        console.log('All data: ', data);
        console.log('Properties: ', data.properties);
    }
    else console.log('\x1b[31m%s\x1b[0m', `Issue with the request:`, data)

    return data

}

I trust this explanation proves beneficial. Feel free to reach out if you have any additional inquiries.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Enhanced compatibility with Touch.radiusX feature on smartphone and tablet devices

Curious if anyone knows about the level of support for this property. I am currently using the PR0C0D1N6 app on an iPhone 4 and based on the specifications, when radiusX is not supported it should default to 1. However, in my case, radiusX is showing as un ...

Display an orange box (also known as a lightbox) when the page

My understanding of jquery and javascript is limited, so while I have come across solutions to similar problems, none have provided an explanation that allows me to adapt it to fit my version of a lightbox. I would like to use the orangebox (a jQuery plug ...

Modify button behavior on click after the initial press

Struggling to make this work the way I want. I have a button that should execute Javascript function A when clicked, and in function A, I need to change the onclick attribute to function B. This way, on the second tap of the button, it will trigger functio ...

I have been utilizing ESBuild to compile JavaScript code for browser usage. However, I encountered an issue when trying to import CSS as I received an error message stating "Unexpected '.'". Can anyone provide guidance on how to resolve this issue?

I am currently developing a JavaScript notebook that operates within the browser environment. To compile my code, I have chosen to utilize ESBuild. My primary objective is to enable the handling of CSS imports such as <import 'bulma/css/bulma.css&a ...

An AJAX request will only occur if there is an alert triggered on a particular computer

Here's an interesting issue I encountered with my company's CMS newsletter system. It seems that the AJAX call to send an email works flawlessly in all modern browsers and operating systems, except for one client. This particular client is using ...

Encountering difficulties in the installation of a package within Next JS

Can anyone assist me with this issue I'm facing while trying to install styled-components on Next JS? I keep getting an error after entering npm install --save styled-components: npm ERR! Unexpected end of JSON input while parsing near '...ry.np ...

Sending a custom `GET` request with multiple IDs and additional parameters using Restangular can be achieved by following

I'm trying to send multiple ids along with other parameters using my custom customGET method. However, the implementation seems to be incorrect: var selection = [2,10,20]; // Issue: GET /api/user/export/file?param1=test&ids=2,10,20 Restangular.a ...

dependency in useEffect hook not being properly updated

Currently, I am implementing an API call within the useEffect hook in the following manner: useEffect(() => { fetchPatientsStartAsync(patientListUrl); }, [patientListUrl]); Moreover, in my codebase I have two additional state variables and a method in ...

Uploading files with Vue.js Element-UI and axios triggers unwanted page refresh

I am utilizing the element-ui file upload component() to manage the frontend of image uploading. All functionalities work flawlessly (file is successfully sent to the server, etc). However, for some reason, after the axios's successful response code r ...

Develop a custom input field feature that utilizes both JavaScript and CSS

I really appreciate the feature that allows me to resize the textarea by simply clicking and dragging on the slanted lines in the lower right hand corner. However, I am looking for a way to apply CSS styles to text which is not possible with a textarea. ...

The issue with CSS filter invert not functioning properly in Mozilla Firefox is causing complications

I am currently developing a small Firefox add-on designed to make reading pages at night more comfortable. The goal is to invert page colors without affecting images. Here is the code snippet I am using: document.body.style.filter = "invert(100%)"; var i ...

What is the process of modifying the data in DataTables once it has already been initialized?

I am looking to dynamically populate a table using datatables through an ajax request. Here is the approach I have taken: $(function(e) { $('#CampaignMenu').change(function(e) { $('#ReportWrapper').hide(); ...

Ways to generate multiple elements using JavaScript

Is there a way to dynamically freeze columns in a table as I scroll it horizontally? I've achieved this statically using JavaScript, but is there a way to indicate the number of columns and achieve the desired style? This is what my JavaScript code c ...

What steps do I need to take in order to create functions that are

I have 10 functions with similar structures: function socialMedia_ajax(media){ return ajaxRequest('search/' + media + 'id', media + 'id').then( function(res) { var imgStatus = res[1].length > 0 ? "c ...

I'm not skilled in programming, so I'm not sure what the problem is with the code

While working on my Blogger blog, I encountered the need to add a fixed sidebar ad widget that would float along the screen. After trying multiple codes, I finally found one that worked. However, using the template's built-in variable functions led to ...

Choosing all components except for one and its descendants

I am searching for a method to choose all elements except for one specific element and its descendant, which may contain various levels of children/grandchildren or more. What I'm trying to accomplish is something like... $("*").not(".foo, .foo *").b ...

"Utilizing JavaScript to parse and extract data from a JSON

I just received a JSON object through a PHP get request and it has the following structure: data == {"user_id":"7","emp_type_id":[{"0":"11","1":"10"}]} My main focus right now is to retrieve the values within emp_type_id, which are "11" and "10" in this ...

AJAX request: No values are being returned by $_GET

After spending hours trying to figure this out... I've been working on using AJAX to grab values from a jQuery slider within an <input> tag. The AJAX request is not failing (see code below), and when I use console.log to check the variable I&ap ...

When swiping right with Swiper.js, the slides are jumping by all, skipping the following slide, but the left swipe functions correctly

Here is the code I used for my swiper element: new Swiper("#swiper-pricing", { slidesPerView: 1.3, spaceBetween: 30, centeredSlides: true, loop: true, keyboard: { enabled: true, }, autoplay: { delay: 50 ...

Using jQuery AJAX to send POST requests in CodeIgniter

I've been trying to configure the jQuery Ajax post URL, but it's not functioning as expected. I've searched extensively and found many solutions, but none of them seem to solve my issue. I've set the base URL in var baseurl = "<?php ...