human frailty
![Post Post](./styles/prosilver/imageset/icon_post_target.gif)
Hey everyone - great to be back up and again, really sorry for the downtime!
In case you were wondering, I believe the immediate cause was a failing power supply which has now been replaced by our service provider.
That alone shouldn't have caused such a big problem though. Rather, the length of the downtime was due to two other issues -- 1) I had configured the main server so that the mail server did *not* start automatically on reboot. This I did over a decade ago when we were under some direct attacks and also when we had a *much* less efficient server configuration. We often had to deal with staggering backloads of email before restarting the mail server or the system would simply go down again.
We haven't had that problem for a very long time - since we moved to exim4 as the primary mail server and got better hardware -- and I knew the time had come long ago for me to use the standard configuration that would start the mail server automatically, but I didn't make the change. Instead, I simply paid close attention and started the mail server manually after each reboot.
But not this time - 2) I had a health issue that happened to occur at about the same time as a reboot that likely was caused by the power supply issue - I was simply not available for a few days. I'm much better now, and importantly, I made the configuration change so that if there is another reboot for some reason, everything will start working again, and likely no one will even notice.
That said, I have to acknowledge that this incident outlines the fact that it is probably time for the group of us to start thinking about ways to protect the ongoing continuity of the service, or to find an alternative.
I'm pretty much open to any consideration -- the service doesn't really make any money - it pretty much makes enough to pay the service provider fees and some of the domain registration fees (and that's basically all of the real costs), but definitely not enough to pay salaries for full time support personnel - not that we really need any.
To be sure - I don't have any plans to shut the service down, and it's likely it can continue to operate normally for many more years / decades - time is on our side. My experience in the last few days was sort of a 'wake up call' though - a reminder that I won't be around forever, and so I would love for us to take the opportunity to brainstorm possible approaches to reducing dependence on me.
In case you were wondering, I believe the immediate cause was a failing power supply which has now been replaced by our service provider.
That alone shouldn't have caused such a big problem though. Rather, the length of the downtime was due to two other issues -- 1) I had configured the main server so that the mail server did *not* start automatically on reboot. This I did over a decade ago when we were under some direct attacks and also when we had a *much* less efficient server configuration. We often had to deal with staggering backloads of email before restarting the mail server or the system would simply go down again.
We haven't had that problem for a very long time - since we moved to exim4 as the primary mail server and got better hardware -- and I knew the time had come long ago for me to use the standard configuration that would start the mail server automatically, but I didn't make the change. Instead, I simply paid close attention and started the mail server manually after each reboot.
But not this time - 2) I had a health issue that happened to occur at about the same time as a reboot that likely was caused by the power supply issue - I was simply not available for a few days. I'm much better now, and importantly, I made the configuration change so that if there is another reboot for some reason, everything will start working again, and likely no one will even notice.
That said, I have to acknowledge that this incident outlines the fact that it is probably time for the group of us to start thinking about ways to protect the ongoing continuity of the service, or to find an alternative.
I'm pretty much open to any consideration -- the service doesn't really make any money - it pretty much makes enough to pay the service provider fees and some of the domain registration fees (and that's basically all of the real costs), but definitely not enough to pay salaries for full time support personnel - not that we really need any.
To be sure - I don't have any plans to shut the service down, and it's likely it can continue to operate normally for many more years / decades - time is on our side. My experience in the last few days was sort of a 'wake up call' though - a reminder that I won't be around forever, and so I would love for us to take the opportunity to brainstorm possible approaches to reducing dependence on me.