My current challenge involves fetching a large amount of data from a database in JavaScript using streaming to avoid loading all the data into memory at once. I am utilizing express as my server and a nodeJS client that retrieves the data using Axios. While I have successfully fetched the data with streaming, I am struggling to handle errors that occur during the streaming process.
Express Server:
app.get('/stream', async (req, res) => {
try {
const cursor = //fetch data with a limit & skip (MongoDB)
while(cursor.hasNext()) {
const data = await cursor.next()
const writeToStream = new Promise((resolve) => {
res.write(data, 'utf8', () => {
console.log("batch sent");
resolve()
})
})
await writeToStream
}
res.end()
} catch(error) {
console.log(`error: ${error.message}`)
//How do i send the status & error message to the requestor?
//return res.status(400).end(error.message) // <-- wanted behavior
})
Client:
try {
const res = await axios({
url: 'http://localhost:3000/test',
responseType: 'stream'
})
const { data } = res
data.pipe(someStream)
data.on('end', () => {
//stream finished
})
data.on('error', (error) => { // First Option
//error without a status or message
//res.status(error.status).send(error.message) // <-- wanted behavior
})
} catch(error) { // Second Option
//error without a status or message
//return res.status(error.status).send(error.message) // <-- wanted behavior
}
The error handling on the client is functioning properly but I am unsure how to communicate a status and message from the server to the client to indicate and specify an error.
Versions: "axios": "^1.5.1", "express": "^4.18.2"
Your assistance would be greatly appreciated. Thank you in advance!