Introduction
Typically, on the server side, there is a thread pool or event queue to handle incoming requests (referred to as the IO thread pool). The actual processing of these requests occurs on another thread (usually the main thread, in single-threaded scenarios), which we can call the processing thread pool. In this setup, the request is swiftly processed on the processing thread, and a response is then sent back to the IO thread pool. This simplified explanation provides an overview of the request-response mechanism.
---------- ----------------- ------------------ ---------------------
| client | <----> | load balancer | <---> | request thread | <---> | processing thread |
---------- ----------------- ------------------ ---------------------
When a client sends a request to the server, it is received by the request thread pool and may have to wait until the server processes the incoming request. Delays can occur due to various reasons such as being blocked by a lengthy database query or performing resource-intensive tasks like 3D rendering. However, more frequently, delays are caused by issues like unclosed resources depleting the processing thread pool.
Illustration
Let's explore an example with a simple node server using the express framework. In Node.js, there is only one thread for processing – the main thread. Let's observe the consequences when this thread gets blocked.
const express = require('express')
const app = express();
const port = 3000;
var sleep = function(seconds) {
var waitTill = new Date(new Date().getTime() + seconds * 1000);
while(waitTill > new Date()){}
};
app.get('/slow', (req, res) => {
// The sleep here signifies that process is taking a long time for whatever reason
sleep(1);
res.send('Slow query returns');
});
app.get('/fast', (req, res) => {
res.send('Fast query returns');
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
});
Let's create a basic client to test the application.
var unirest = require('unirest');
var i = 0;
var j = [];
var tot = 20;
for (var i=0; i < tot; i++) {
j = j.concat(i);
}
j.forEach(function(i) {
console.time("No: " + i);
unirest('GET', 'http://localhost:3000/fast')
.end(function (res) {
if (res.error) throw new Error(res.error);
console.timeEnd("No: " + i);
});
});
j.forEach(function(i) {
console.time("No: " + (i+tot));
unirest('GET', 'http://localhost:3000/slow')
.end(function (res) {
if (res.error) throw new Error(res.error);
console.timeEnd("No: " + (i+tot));
});
});
Output from the client:
No: 0: 31.406ms
No: 1: 31.203ms
No: 3: 31.171ms
No: 2: 31.628ms
No: 4: 31.662ms
No: 5: 31.835ms
No: 7: 31.703ms
No: 6: 32.288ms
No: 8: 32.295ms
No: 9: 32.459ms
No: 10: 1027.011ms
No: 11: 2027.326ms
No: 12: 3027.625ms
No: 13: 4027.925ms
No: 15: 5027.833ms
No: 14: 6028.303ms
No: 16: 7028.411ms
No: 18: 8030.380ms
No: 17: 9030.837ms
No: 19: 10030.692ms
The results show that the slow query significantly delays responses (e.g., the 10th request takes 10 seconds) and could potentially lead to prolonged waiting times.
Potential Issues
- A long-running database query blocking the processing thread and preventing timely responses
- An infinite loop causing a thread to be unresponsive, reducing the effective size of the processing thread pool
- Excessive object creation leading to excessive garbage collection, thereby obstructing the processing thread pool. This problem is common with lingering references and unclosed file pointers
- Deadlocks arising from threads contending for shared resources, resulting in multiple threads being blocked in the processing thread pool
This list is not exhaustive and highlights possible complications that may arise. It is crucial to address such issues promptly to ensure efficient server performance.
While older requests occupy the processing thread pool, new connections continue to be accepted by the request thread pool, potentially causing delays or even server crashes. Such scenarios indicate significant system inefficiencies.