Page 1 of 1
DSLReports down
Posted: Wed Apr 18, 2012 4:26 pm
by Philip
DSLReports has been down since Monday afternoon, caused by what started as a Net Access Corporation power outage, I'm sorry for Justin and hope they bring it back up with no further issues.
dslreports.com is offline
-------------------------
Update Wed Apr 18 09:54:54 EDT 2012
Well, the good news is we have a plan & I hope things will remain offline for no
longer than the balance of this week. Unfortunately, equipment must be sourced
installed, configured, and so on. Lacking a good explanation of what happened to
"redundant" disk storage, I am reluctantly faced with just replacing it with more
expensive gear. No certainty on times, yet. I will post a link to the gory details soon.
Update Tue Apr 17 04:14:29 EDT 2012
Things have turned out to be more complicated. Although no hardware was fried,
our large storage array has decided, probably as a result of going dark in mid write,
that our two sql servers can no longer have access to one of the small,
but important, partitions that makes up the larger SQL data area. All data
is present, but it refuses to offer it.
At this point all courses of action involve long delays. When I know more I'll post
-Justin
Update 15:35 EDT
Our data center experienced a power outage which affected our entire system. We are in the process of recovering everything
and turning a few cranky servers into happy ones.
Apologies for the inconvenience, we expect to be up in a few hours
Update 16:18 EDT
-------------------------
The power outage affected the entire data center and we are still awaiting our turn to power up some machines.
We will keep this page updated.
Update 19:20 EDT
-------------------------
Unfortunately, a database server doesn't take kindly to being taken down without proper shutdown procedures and the
restoration efforts are still under way. No ETA at this time.
http://www.dslreports.com/front/shutdown.html
Posted: Wed Apr 18, 2012 6:02 pm
by Ken
Totally helpless in that type of situation. You can never have enough back up power... Hopefully he will get it back up soon with out major data loss.
Posted: Wed Apr 18, 2012 6:03 pm
by nightowl
hope speedguide's server is not at the same data center....we have good UPS's on our speedguide servers right?????
Posted: Wed Apr 18, 2012 6:13 pm
by Ken
nightowl wrote:hope speedguide's server is not at the same data center....we have good UPS's on our speedguide servers right?????
SG servers are far away from there, however how long battery back ups can keep you on, depends... Keep in mind you would have no idea how long the power would be out...
It almost sounds like either the UPS was not there or not enough for the loads, or at minimum, not fully functioning, however it would either have needed to be configured to shut down automatically on an outage or someone would need to have been alert and manually shut it down...quickly...
Power loss is definitely a quick way to cause bad sectors on hdds...
Posted: Wed Apr 18, 2012 8:29 pm
by idavid4attdsl
as far as at&t dsl customers in the direct threads over there, I can take them on and still work with them. Just tell them to drop me an email. The QR code to the left of my name as a link to the contact page. They may not be back up before friday.
Posted: Wed Apr 18, 2012 8:29 pm
by Philip
The datacenter has diesel generators that are ready to kick in as necessary in case of power outages. We do have backups and should be able to bring the site back up fast in case of hardware failure, we have redundancy on most parts as feasible..
That said, as I was typing this one of the core routers in the data center went offline, their backup router did not pickup the load as it should, and it took them close to an hour to bring the network back up.. :/ Still, the colo facility, the network and the quality of service has been very good.
Here are some images of their cooling and backup power generators:
http://www.gorack.net/Colocation/facility.php
Posted: Wed Apr 18, 2012 9:14 pm
by nightowl
Philip wrote:The datacenter has diesel generators that are ready to kick in as necessary in case of power outages. We do have backups and should be able to bring the site back up fast in case of hardware failure, we have redundancy on most parts as feasible..
That said, as I was typing this one of the core routers in the data center went offline, their backup router did not pickup the load as it should, and it took them close to an hour to bring the network back up.. :/ Still, the colo facility, the network and the quality of service has been very good.
Here are some images of their cooling and backup power generators:
http://www.gorack.net/Colocation/facility.php
does speedguide have an off site backup?
Posted: Wed Apr 18, 2012 9:16 pm
by CableDude
People still visit that site?
Posted: Wed Apr 18, 2012 9:34 pm
by ace
CableDude wrote:People still visit that site?
Everyday. Way busier with alot more topics. I still visit SG everyday but its just really slow here.
Posted: Thu Apr 19, 2012 12:18 am
by Philip
nightowl wrote:does speedguide have an off site backup?
Yes, just not always current because of database sizes.
Posted: Thu Apr 19, 2012 6:20 pm
by CableDude
ace wrote:Everyday. Way busier with alot more topics. I still visit SG everyday but its just really slow here.
Maybe I should drop by.
Posted: Thu Apr 19, 2012 7:13 pm
by RoundEye
That’s a serious facility.
Posted: Thu Apr 19, 2012 7:30 pm
by idavid4attdsl
CableDude wrote:People still visit that site?
I still do for att dsl and uverse customer issues.
Posted: Fri Apr 20, 2012 7:19 pm
by hayc59
CableDude wrote:People still visit that site?
Oh yea!! not a bad forum
Posted: Sat Apr 21, 2012 11:25 pm
by Shinobi
Philip wrote:The datacenter has diesel generators that are ready to kick in as necessary in case of power outages.
The sign of a good B.C.P. :-)
Posted: Sun Apr 22, 2012 12:54 am
by thechemgeek
Meh.