I needed to go out yesterday and I did not worry about our server because everything was running fine before I left.
Then I went back home last night, plugged the computer and do my routine… I checked the server with a big surprise. I was terrified when I clicked an email and viewed a long list of reply history, in loop! I don’t know what happened but it’s probably because of the email configuration that I made the other day. It triggered an email loop that made the CPU load of our server terribly high this evening. The email loop inside was “piped” in an email system and 4 hours or endless loop, it resulted to 2.8 gigabytes of redundant data.
If you don’t know Piping is a technology that immediately query an email right inside the database for storage. In other words, it “capture” the email for instant logging or input. So in simpler words, in every loop, the email was stored repeatedly in our database as it snowballed.
I’m quite amazed that our Linux server still managed to survive this kind of stress test! It was able to withstand 100% CPU utilization for 4 hours without crashing. It proved that our server is powerful enough to host heavy loads without immediate risk of system failure.
This incident not only tested the power of our server but also my patience. Luckily we have the hardware to support this kind of abuse and I was not pushed to my limits.
