My Objective
To efficiently process a large array using the .map method and interact with the Google Geocoder API through promises to get location data.
The goal is to utilize Promise.all to store results in a .json file upon completion of the operation.
This code is specifically designed for node, where the array has a structured format of [[key:value],[key:value],....]
Implementation Details
var foo = (function() {
let promises = airports.map(function(airport) {
return geocoder
.geocode(airport[1])
.then(function(res) {
let x = {
ident: airport[0],
address: res[0].formattedAddress,
lat: res[0].latitude,
long: res[0].longitude
};
return x;
})
.catch(function(err) {
console.log(err);
});
});
return Promise.all(promises)
.then(function(promises) {
console.log("promises resolved");
fs.writeFile(
path.join(__dirname, "/airports.json"),
JSON.stringify(promises),
"utf-8",
function(err) {
if (err) throw err;
console.log("Done!");
}
);
})
.catch(function(err) {
console.log(err);
});
})();
The Challenge
The current implementation works well with small arrays but encounters issues when dealing with larger datasets. It seems to be related to rate limiting from the api, resulting in many ETIMEDOUT errors after processing a few hundred records.
I am exploring ways to slow down the iteration process, perhaps by executing one request per second. How can I incorporate this delay while handling multiple promises effectively?
Implementing a delay could potentially reduce the occurrence of ETIMEDOUT errors during the execution.
Thank you for your assistance.