Saturday, June 21, 2008
current status: normal
- POP3: normal
- SMTP: normal
- WWW: normal (under observation)
- DNS: normal (under observation)
- FTP: normal (under observation)
heat wave = server shutdown
another heat wave, another shutdown on one of the servers (the one that serves some of the websites and the nntp server).
the server is back now and i have once again requested an explanation from the datacenter. some of my drives were running 6 degrees Celsius over their safe temperature. not good although they seemed to have recovered. still, we might see more outages from this server.
the server is back now and i have once again requested an explanation from the datacenter. some of my drives were running 6 degrees Celsius over their safe temperature. not good although they seemed to have recovered. still, we might see more outages from this server.
Thursday, May 15, 2008
Here comes the heat
There's another heatwave in San Francisco and unfortunately the data center has heated up which is causing one of the servers to crash again as it did in mid April. This server really is just a finicky one - it's not nearly as hot in the cab as it was last time. I might put the server into some new hardware in the near future if things do not improve. In the meantime, expect some outages here and there. I will monitor and restart as needed.
Saturday, April 12, 2008
serious problems with www/ftp server
the server began crashing yesterday (2008.04.11) afternoon but was automatically recovered throughout the day. last night the machine crashed and would not restart. it looks very much like a trip to the colo is in order to fix it.
as this is a very old box running with very old disks in it, i am considering retiring it completely. the websites will all move to a new server but the news server will have to be retired as i do not have the bits to run it on other hardware.
more on that as things progress. i wont really know anything until i'm able to look at the machine in person.
update @ 13:22PDT: i'm in the colo now and the machine seems to have recovered after a lengthy disk repair task. it's extremely hot in here and i'm concerned that it's the heat in the cabinet that is causing problems with the hardware. the colo operators are working on bringing up another cooling unit as the place is definitely too warm for these machines. i'm leaving the server alone for now. if it crashes again i will take the disks out and bring them home and move all the data to a new machine. stay tuned.
update @ 14:33PDT: well that was wishful thinking. as soon as i got home the server went and died again. i will attempt a remote restart but i will not going back to the colo until tomorrow and then i'm not sure how long it will be before the www stuff is up again. note that the major www sites such as rideontwo.com are not at all effected by this outage.
updated @ 09:57PDT on Monday the 14th: i think the colo facilities temperature has dropped significantly and this server has been able to recover for the time being which points to an almost certain heat related issue with the hardware. I'm still planning on moving the data to a new machine but maybe it doesn't need to be so urgent.
as this is a very old box running with very old disks in it, i am considering retiring it completely. the websites will all move to a new server but the news server will have to be retired as i do not have the bits to run it on other hardware.
more on that as things progress. i wont really know anything until i'm able to look at the machine in person.
update @ 13:22PDT: i'm in the colo now and the machine seems to have recovered after a lengthy disk repair task. it's extremely hot in here and i'm concerned that it's the heat in the cabinet that is causing problems with the hardware. the colo operators are working on bringing up another cooling unit as the place is definitely too warm for these machines. i'm leaving the server alone for now. if it crashes again i will take the disks out and bring them home and move all the data to a new machine. stay tuned.
update @ 14:33PDT: well that was wishful thinking. as soon as i got home the server went and died again. i will attempt a remote restart but i will not going back to the colo until tomorrow and then i'm not sure how long it will be before the www stuff is up again. note that the major www sites such as rideontwo.com are not at all effected by this outage.
updated @ 09:57PDT on Monday the 14th: i think the colo facilities temperature has dropped significantly and this server has been able to recover for the time being which points to an almost certain heat related issue with the hardware. I'm still planning on moving the data to a new machine but maybe it doesn't need to be so urgent.
Sunday, January 27, 2008
mail server issues under control now
I've installed some new hardware in the ailing server and we've been running very stable for the last week. I'm declaring victory for the time being
Saturday, January 19, 2008
mail server issues continue
in addition to the unstable server hardware, which actually has been hanging on pretty well the last few weeks, now the database server on the machine is crashing at various times.
this is almost certainly a hardware issue, the problem is figuring out which hardware component is causing all the problems. i'm not yet ready to replace the whole system...
this is almost certainly a hardware issue, the problem is figuring out which hardware component is causing all the problems. i'm not yet ready to replace the whole system...
Saturday, December 29, 2007
mail server issues again
The mail server is hanging again. I ordered some new memory for it which will hopefully fix the issue (definitely feels like bad memory).
Wednesday, December 19, 2007
new certificates installed.
all SSL based services have had their certificates renewed. Unfortunately the signing authority has changed as well so you have to install a new root certificate as defined in this post.
Friday, December 14, 2007
mail server issues
The mail server is having problems since wednesdays. It started with a short outage and a reboot but now the machine is cycling from working to hung with regularity. I suspect bad memory and have ordered new memory chips. It could be a few days, however, before service is stable again.
Update: 2007.12.19 19:46PST: I'm pretty sure the server problems have been resolved. Things look stable at the moment.
Update: 2007.12.19 19:46PST: I'm pretty sure the server problems have been resolved. Things look stable at the moment.
Tuesday, September 04, 2007
new certificates
The SSL certificates have expired so for now just accept the existing ones when using mail/https. Updated certs will be available shortly. Your data is still encrypted it's just basically a timer reminding the admin to update the certs. Good security practice. This admin forgot to update them before they expired o.O
