I have been off the air for many hours due to this:

"4:45pm Monday: Due to heat in Perth we have lost a number of services and precautionary shut others down."

I know it's been very hot in Perth lately but for a major ISP based there not to be able to cope with it strikes me as not good enough.

We seem to have lost a lot of the old engineers & teachers with experience. The stupidity realised by all but the idiots doing it, in closing down the Victorian Technical School system (more or less the trades & engineering arm of education) left a major hole and a generation gap & similar stupidity in paying out the old guys again saw years of experience not passed on.

So we now have people building / supplying stuff for overseas temperatures with no clues on things like continuous ratings. So we have things like transformers that burn, power supplies, and motors that cook as they are incapable of dissipating heat for the required duty cycle. Nor in some cases is sufficient forced cooling provided to dissipate heat.

Two classic examples here are the control box of the Solar Hot water pump, which I believe was stopping as it is in a plastic box within a cover on the side of the tank. As I do not use a lot of HW the bottom of the tank gets extremely hot. I have not had any trouble since removing the cover. So that's on the list for modifications.

Then we have the phone company that knows little of customer service & maintenance. I have one of their 3G USB modems to which I have had to add an antenna. 4G (fourth go) does not work here, albeit that applies to most of the local phone system. This little 3G device has a large amount of SS inside that gets hot, so it has a solid plastic cover to ensure it gets hotter. So once the temp passes 25 degrees it starts to struggle & stop. Another design fail and engineering flop.

What I have done to it, is mounted it in a piece of 40mm Duct as it sits nicely in the slots; Removed the back & mounted a CPU fan (abt. 25mm square) to blow air onto the metal. So far so good it has not yet got back to the same high temps.


Marc

That is interesting. I am with iiNet, live in Perth and have had no outages at all. I am on line at least four times daily and have had no adverse problems here.

"Perth heat melts the internet":

http://www.smh.com.au/digital-life/digital-life-news/perth-heat-melts-the-internet-20150105-12iee2.html.

I remember the time that Bigpond users were offline for close to three full days due to a failed firmware upgrade on their Cable and ADSL network switches. Long time ago now but an absolute ballsup at the time.

I have to admit I was quite annoyed at the time.

I think in rural areas Telstra is waiting for NBN to fix up everthing they have let run down. And that is the message I have left with the local Federal MP. We have land line running on fences in places and they still do not seem to have fixed the corrosive joints issue.

Of late my landline has failed about six times in eighteen months and even when there is a planned power outage at 8am the whole RAX if it looses Utility power will be dead by morning tea time. Shame about Granny's landline based emergency response system, that can't dial out as all the lines are all dead in the whole area.

Marc

I remember back to when Telstra (Telecom) had something like 92,000 employees - totally distinct from the 34,000 they have now.

I can understand in the computer age that many administrative functions would become performed by computers instead of people however basic maintenance and repairs cannot. The trouble is, when a beancounter is looking for a pay rise and wants to demonstrate how irreplaceable he/she is the first people that get the chop are those working in maintenance departments.

One day they will learn that networks of any kind cannot be held together with chewing gum and that it is a false economy for any kind of infrastructure to exist without being correctly maintained. In my line of work, when ever an expensive and important piece of infrastructure is purchased and installed it comes with adequate warranty support for the first twelve months followed by a ten year service agreement with a service level guarantee where in case of breakdown we get a serviceman on site within 24 hours and spare parts on the supplier's shelves instead of waiting for bits and pieces to arrive from other states or overseas.

That is how Telstra and other ISPs should be running their businesses. In the long run it is far cheaper to maintain things and keep them going rather than react to breakdowns all the time.

I have yet to hear a decent explanation from iiNet as to what went wrong, but the bits and pieces I'm gleaning is that they had an air conditioning failure so turned off severs to drop the room temperature and did not bring them back up again until the aircon problem had been fixed. Following that their system could not cope with the surge of ADSL authentications and something else failed.

On top of that there are still areas with no iiNet access for yet other reasons.

All in all, a test of their disaster recovery processes that ought to give their management reason to review them thoroughly.

Not good enough for Australia's second or third largest ISP (depending in whose figures you use).

I'd agree that temperatures in server rooms rise quite quickly when air conditioning fails.

At my previous workplace we had a server room with roughly 55 boxes running across 7 racks. The room had two separate air conditioners each on its own power supply fed from different switch boards.

Two summers ago both units failed on high head pressure on one of the Sydney scorcher days and the only way to get them going again was to cool down the condenser coils with a standard garden hose - I spent about half an hour on the roof of the building in 47 degree heat cooling these things down. Before that I was in the server room watching the heat rise until the point that the IT department simply had to shut down any servers that were not absolutely vital to keep running. The normal temperature in this room was 16 degrees but once those A/C units fail the temperature rises to around 40 degrees in about five minutes. I could only imagine what happens at a major ISP with a few hundred of the same servers in the one room.

The thing that annoyed me about this fateful day in my workplace is that I proposed two things.

1. Either shift the two condenser coils so they were located away from direct afternoon sunlight or build a shelter to screen them from direct sunlight.
2. Install a third A/C system to take the load if the two main units fail. The condenser unit for this third system would be installed in the plantroom adjacent to the server room which had good crossflow ventilation even on super hot days. This third unit would only kick in after the server room temperature reached 28 degrees - keeping it in 'reserve' and thus less prone to failure and allowing the high temp alarm system to still work and notifying IT staff of the issue.

Big problem - brighter minds prevailed and my suggestion copped the kibosh. I said, "suit yourselves but I am not standing on the roof hosing the condensers again." I looked like a crayfish the next day and almost got sunstroke doing the firm that big favour.

Those suggestions of your are all very sensible and should be implemented by any company who takes infrastructure risk management and DRP seriously. Unfortunately, too few do.

I doubt that Perth is going to get any cooler anytime soon, so iiNet had better get its act together in the cooling department pronto. Another outage like this one would make it look mismanaged to put it mildly.

(PS: Brad, I used quotes in my thread title but the quoted part doesn't show in the thread listing, nor when I add posts to the thread.)

One of the things I love, is the way space is saved by putting air conditioners, in particular evaporative types, on the roof to ensure that it inhales the hottest air that is available and presents the greatest challenge to safety when servicing. The split system later had its compressor on the verandah with a cage that stopped leaves & vermin, no issues.

Perhaps it does not inhale so much rubbish up there, but the one at my previous residence needed the water dumping & hosing out frequently, particularly in drought, which it copped 15 years of. About every six or seven weeks was the cycle & because of what it inhaled apart from that, I ended up putting a screen type water filter in its pumping system.

Sometimes I think a heat exchanger system with a cooling tower is much more effective, but naturally you will whinging about the water consumption, which is only valid where there isn't a lot. One factory I know of cooled the emergency generator (marine engine) with a pond.

Marc

They put cooling towers on roofs to better distribute Legionella bacteria.

Automatic water sprayer monitoring liquid line temperature for air cooled condensers would do the job.

Too bad I can't box up some of our below freezing weather here in the USA and sent it to you guys in Perth....

One of the things I love, is the way space is saved by putting air conditioners, in particular evaporative types, on the roof...

Evaporative air conditioners are usually cheaper to install when on the roof though there are pitfalls. In winter the occupier has to get on the roof to put the cover on which for a lot of people is a hazardous task.

The other problem is for the humble electrician (people such as myself) putting their backsides on scalding hot roofs during summer to replace worn bearings in the fan motors when they seize. To stop respiratory issues the filters should be changed every two years and the calcium deposits hosed off every other year - again, not always a safe task to complete. These reasons usually lead owners of large buildings to install evaporative units on the ground beside the building and run a big duct up the wall.

(PS: Brad, I used quotes in my thread title but the quoted part doesn't show in the thread listing, nor when I add posts to the thread.)

Inverted commas should be changed by the site software to single commas automatically and displayed the same everywhere the title is shown. I will look into this over the weekend and fix as necessary.