A mail from Fergal Byrne on a mailing list I’m on this morning claimed that Hosting365 (Ireland’s largest hosting company) had “gone black”. It went on to say:
I’m getting alerts from my server monitors and cannot raise anything on hosting365’s network (where our servers are). Calling any of there numbers results in a tone which is neither ringing nor busy but sounds ominously like ‘no connection’
I checked the hosting365 site when I read the mail and it was back up so I can’t confirm it was offline - anyone?
If hosting365 and all their sites (and client’s sites) were offline it raises some serious questions:
- How can an entire hosting company go offline - don’t they have redundancy built in to all their systems so that they can survive power cuts, network outages etc.? Surely you host with a hosting company precisely because they never go dark.
- How many Hosting365-hosted ecommerce sites were affected? How much revenue did they lose? And do their SLA’s with Hosting365 mean they will be compensated or does the SLA allow Hosting365 to say “tough”?
- If you are hosting a site, how do you decide which hoster to go with? You would have thought that the largest hoster in Ireland would be a safe bet, no?
I tried to call Ed Byrne (Hosting365’s Marketing Director) on his mobile to get a comment but it rang out. Ed, you want to leave a comment here?
[Edited to add the following disclosure] - I do a little hosting myself (11 sites), reselling server space for my own hoster. Most of the hosting I do is for friends and very few of the sites I host have received any bill for the service.
If you enjoyed this post, make sure you subscribe to my RSS feed!
Probably shouldn’t be saying this (although I’m sure they’ll come clean on it): I heard there was a fire.
I saw intermittent connectivity on e-mail from 13:05 to 14:12 only. Other than that it has been fine all day. Argolon web-site was up any time I connected and I was able to use the Hosting365 management console to change settings. So if they did have a fire, I think they did alright for uptime considering we are on their cheapest shared-host Linux plan.
I heard the fire rumour too.
A lot of the Irish internet has been up-and-down since this morning. My bet is that it was some of the major routing infrastructure, and that Hosting365 was probably up and still transmitting on *some* of their links — but that the routes between Fergal Byrne’s site and Hosting365 were down — which may not be entirely H365’s fault.
My site, my client’s sites, boards.ie, hosting365.ie and more were completely unresponsive for about half an hour or so at about 1320. Steve from 365 says there will be an announcement on their forums shortly: http://www.boards.ie/vbulletin/showthread.php?t=2054956771.
tcal.net was dead and unpingable for a short time around 13.20 or so.
Yeh mine and all my reseller clients sites were down.
To be fair to Hosting365, this is the first major problem we’ve had with them in our 5 years of hosting hundreds of websites with them. I’m sure there is a good reason for this outage. Shit happens.
Why would you expect their marketing director to answer you in the middle of a crisis?
“I checked the hosting365 site when I read the mail and it was back up so I can’t confirm it was offline - anyone?”
Yeah, another site (angryPotato.net) hosted [sic] by hosting365 was down just before lunchtime (it was back up and running at some point during the lunch hour), with an error message indicating that a connection with the site’s servers (hosting365) was unable to be made.
Servers I mange (all the cheaper Rack365 brand) were all rebooted at lunch time, and available shortly before that. H365’s main ns1/2 servers were also unavailable during this time. So it would appear to be pretty widespread.
.cg
Peter - the point was, I didn’t know if there was a crisis. I know Ed. I have his mobile. I rang it. If he answered and said “I don’t know what you are on about Tom, let me ring that guy Fergal and ask him what is going on.” Then I would have known that there was no crisis.
As it happens there was no answer from his mobile - and, as it now appears that there was a crisis, I’m not surprised that he didn’t answer.
By the way Peter, “Shit happens” is not the response you give to potential customers when they want to know about your uptime. You talk about all the redundant systems you have in place to ensure that there is no downtime.
The DEG building, which Hosting365 moved out of, houses the INEX so you know there is massive network redundancy there and the DEG building is the only hosting facility in Ireland to have its own ESB substation. It also has 3 backup generators.
It looks like moving out of the DEG building to their own data centre was not the wisest of moves. It may have saved them some money in the short term, but outages like this do their reputation no end of harm.
I had a server re-booted in me earlier too … in fact at least two and although my third server is up sshd seems to have gone pear-shapped so I can’t log in and check what time it was rebooted at. The other two were rebooted at 12:41.
There were also serious problems with their mail server.
I’m most curious to hear what went belly-up and in particular what is being done to stop it happeneing again!
Official Statement:
http://forums.hosting365.ie/phpBB2/viewtopic.php?t=952
Robin,
I saw the official response but it leaves several questions unanswered. It doesn’t mention that they vanished from the BGP tables entirely.
The only way that can happen is if you either delete them which would be really retarded or your core network goes offline which is likely what happened to them.
It would appear they lost all power, which is more than they are admitting to, as loads of servers were down completely, not just one or two cabinets.
The DEG facility in Kilcarbery has 3 Generators? Dont always believe the hype.
A Anon - I don’t always believe hype, however I never believe anonymous commenters. As Scoble said:
Hey Tom and everyone,
First of all, I didn’t answer as I was off yesterday … not exactly the best day to take holidays but there you go!
Second of all, I actually take offence to you saying moving to our own data centre wasn’t the wisest move. We didn’t become Ireland’s largest hosting company (by a factor of about 4 last time I looked at the stats) by sitting in a competitors data centre that we don’t have full time engineers living in. This is the first issue is the 18 months that we have been here.
Next of all, of course we have redundant power supplies, it would be idiotic in the extreme to have a data centre running only off the ESB. We have multiple strings of UPS batteries and a diesel generator that’s monitored 24×7 … with SLA’s on diesel from nearby suppliers.
There will be a full report coming out in the next couple of days, and there’s no point me trying to go into the technical details and have an argument with people far more technical than I am about it.
Tom and I are actually friends, so I’m not trying to be harse here Tom … we treat this incredibly seriously, but please check the facts before commenting on our power systems or business decisions on moving to our own data centre. And also, DEG won’t be the only data centre with it’s own sub station for very long, we’ve had planning permission in for a while now and a contract agreed with a building company to get this built.
Finally, this is an open-invite to anyone that’s wants to come in and tour our data centre, if you don’t believe my ‘marketing speak comments’ then get our Technical Director to actually walk you around the place.
I’ve also posted some information in a comment on our own blog here:
http://www.hosting365.com/blog/2006/06/30/oh-dear-god-its-end-of-q2/#comments
[Edited to fix broken link]
Tom
First O’Reilly and now Ed - what is it about you and people on holidays?
des
Ed - welcome back!
I don’t want to split hairs and I’m sorry if you took offence but I didn’t say moving to your own data centre wasn’t the wisest move, I said “It looks like moving…” - there is a subtle but important difference between the two statements.
Ed, an outage like this couldn’t have happened in the DEG. They have had 100% power uptime since they opened their doors in early 2001. They run monthly power outtage tests to ensure that (for instance) no feed breakers trip! The tests are logged and available to clients. Hence my saying it looks like it was a bad move.
I’m delighted to hear you guys are building your own ESB Substation. Good stuff.
Des - LOL
Okay … well lets hope the ‘looks like’ is now cleared up and the very public success of moving (bar yesterday’s blip) speaks for itself.
On DEG …. I suggest you contact Smart Telecom and ask them if they have had 100% power uptime. I happen to know for definite that they had a 3 hour issue some time ago - so DEG have not infact had 100% power up time. We - hosting365 - were a customer of Smarts in 2005 in DEG and experienced the outage that took down their box from around midnight to 3:00 am on and off.
We also run monthly load tests on our generator, in fact only last week we did this and it worked perfectly. This was a totally freak occurence. Of course the ESB have gone down unexpectedly before and the building has not been affected.
If you ( or anyone ) want to come in and physically watch our next full test that is no problem.
Ed, that’s a pretty serious allgation to be making against a competitor in a public forum. Are you sure you don’t want to retract it?
*me looks around nervously for the lawyers*
Hey Tom … I didn’t say they had a total power outage, I said they had ‘power issues’ that I know affected a Smart Telecom box. It can happen to the best of us you know!
So Ed, just to get this straight in my head. Hosting365 were in the DEG in 2005. You guys were getting your network connectivity from Smart Telecom. Smart Telecom lost power to some of their networking kit and thus Hosting365 were offline for 3 hours or so.
If that is the scenario, I don’t see how you can blame the DEG for this. This was Smart Telecom’s fault, to my mind.
I have worked in the DEG when I was CTO for Chip eServices and I am very familiar with the floor there. All the cabinets receive power from two completely seperate circuits. Each unit in the cabinet should have two power cables coming in to them (redundant power supplies) from either side of the cabinet (the A and B circuits). If Smart weren’t using all the power being supplied to them (i.e. they were only plugged into one side of the cabinet), you can hardly blame the DEG.
I say it again - the DEG building has had 100% power uptime since it opened its doors in 2001
Sorry Tom … you got the wrong end of the stick … and to be honest I don’t really want to be publicly slating anyone … as little glitches can happen anywhere.
The point was that from our data centre we were connected to Smart through DEG and they experienced power issues - yes there is A and B circuits … but don’t believe that it’s impossible for human-error in connecting these … which could result in unexpected downtime even though there’s still power in the building.
Ah, gotcha Ed.
So what you are saying is that the DEG guys wired the cabinet incorrectly so that both sides of the cabinet were on the same circuit?
Interesting.
Something like that Tom … my point being that power issues are possibile no matter how many redundant systems you have built in!
DEG have had instances of power feeds labelled incorrectly in the past. They also only ran on 1 generator until less then 12months ago when a second was commissioned as part of their new floor expansion.
To be fair to DEG/Inflow - when they stated in the past that they had 2 generators, they were telling the truth. They just neglected to mention that one was sitting in a container in Dublin port.
Makes you wonder if this can happen to the largest out there, what about the company’s that are of lesser size.
and it’s just gone down again it seems!!!!
No phone responses either.
Has this happened again this morning (15th June 2007)? Our sites don’t respond, hosting365’s phones are dead, too. Is anybody else experiencing this?
Once again a major outage - we are all left in the dark!
How reliable are their systems??
Anyone know whats going on? having clients ringing every 5 mins
Ed Byrne contacted me about this - see my latest post on Hosting365 for more
365 have hosted my website for about 5 trouble free years, how and ever, for the last 1 1/2 years, I have expierenced endless problems. Gaining access to my e-mails, non recognition of passwords. I used simply to click on link, type in my password, and have instant access. Lately,I cant get into the system at all? Why has a user friendly system been changed for such a convaluted one? It takes for ever to get through by phone, then you end up talking to some one who knows less about it , than you do yourself, Help, Jack Clarke