Ericsson facing £100 million damages bill for network outage – report

With smartphones around the world being reduced to doorstops thanks to Ericsson’s software issues, the vendor is potentially facing a damages bill in excess of £100 million.

The full extent of the impact is a bit hazy for the moment, though it is rumoured to be much more than is officially known. What we do know is the O2 network was down for almost all of Thursday, Softbank’s customers were plunged back to the paper days and there are ‘several’ other customers who were impacted by the issue.

Ericsson has confirmed there were others, though it is remaining tight-lipped on who these operators actually are. That said, as you can see from the crowdsourced data below from wireless coverage mapping company Opensignal, it would probably be a fair assumption to add Vietnam’s Mobifone to the list.

Opensignal Graph

Looking at the bill, The Telegraph has reported Ericsson will be facing a global bill of £100 million to compensate customers for the oversight. Ericsson is yet to respond to a request for comment, though having admitted being the root cause of the data dessert there will certainly be conversations concerning compensation.

IWe understand O2 CEO Mark Evans is currently in discussion with the Ericsson management team regarding the whole issue, and the topic of damages will be included on the agenda. Ericsson UK and Ireland CEO Marielle Lindgren is certainly involved in these meetings, though we have not been able to confirm whether these specific discussions have been escalated all the way up to Group CEO Börje Ekholm.

In fairness to O2, it has not palmed off responsibility completely. Last week, it confirmed it would be reimbursing customers as a result of the outage. Pay Monthly, SMB business and mobile broadband customers will be credited with two additional days of monthly airtime subscription charges by the end of January. Pay-As-You-Go customers will be given a 10% credit on a top-up in the New Year, while PAYG Go mobile broadband customers will receive a 10% discount on a Bolt On purchase.

While some might suggest O2 is not completely blameless and should have had processes and systems in place to compensate for such instances, as the root cause of the disruption does lie elsewhere it does have some bargaining power in the talks. The rumoured £100 million bill would not all be credited towards O2, though as it seemed to be impacted the worst, it may well be seeking a large cheque to compensate for lost revenues associated with reimbursing customers.

Ericsson has narrowed the root cause of the issue down to two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). These nodes in the core of O2’s network caused the calamity, as the license for the software expired.

For Ericsson, there might well be no choice. Some are suggesting up to 20 operators were impacted by the expired software license and considering the importance of relationships ahead of the up-coming 5G bonanza, the vendor will need to do everything in its power to ensure favourable terms. It seems buying its way out of this mess might be the only sensible route to take.

O2 up and running but how much damage has been done to them and Ericsson?

With O2’s UK network back up and running, the 32 million Brits who depend on it have been returned to the digital era, but you have to wonder how big the fallout from this disaster will be.

With 3G data services restored late Thursday evening, and 4G getting the green-light early Friday morning, a stressful period comes to a close. Now the more difficult questions need to be asked to understand why this happened, why O2 is rumoured to have had to cancel its Christmas party last night and what the consequences of the chaos will actually be.

“We can now report that our 4G network has been restored,” said an O2 announcement. “Our technical teams will continue to monitor service performance closely over the next few days to ensure we remain stable. A review will be carried out with Ericsson to understand fully what happened. We’d like to thank our customers for their patience during the loss of service on Thursday 6 December and we’re sorry for any impact the issue may have caused.”

In fairness to O2, the simple thing to do here would have been to shift all of the bad press and finger pointing towards the root cause of the problem, Ericsson, but it has managed the saga as well as could be expected.

And while it might have taken a couple of hours for the Swedes to come clean, they finally did, as you can see below:

“The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers,” said Börje Ekholm, Ericsson CEO. “We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.”

While Ericsson is continuing to do root cause analysis on the fault, the issue has been narrowed down to two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). These nodes in the core of O2’s network caused the calamity, though it was not alone.

Softbank experienced the same issues over in Japan, while there are rumours several telcos in Asia also had network outages. Ericsson has confirmed it impacted other customers, but it is not its place to name said customers; O2 and Softbank made their own announcements, so it is up to the operators themselves. Mobifone in Vietnam could well be one of these customers, with its own network shutting down at 11.30am local time.

Before we move on, it’s worth drawing attention to the graph below from web performance and security company Cloudflare, just for context.

Cloudflare graph

As you can see the drop was incredibly pronounced, though the minor traffic which can be seen has been attributed to O2 customers who were roaming outside the UK. These customers do not seem to have been impacted by the data-doomsday.

One of the big questions which is now floating around concerns the fallout. With people and society on the whole relying so heavily on data networks, any fear of sub-par or non-existent performance will have a negative impact. Outages are something people should realistically expect to happen every now and then, but the severity and length of this one is certainly noteworthy.

“The disruption shows how much importance we place on a mobile device,” said telco and media analyst Paolo Pescatore. “Without connectivity, people are stranded, and businesses cannot compete. Furthermore, it underlines the need for continuous investment in the UK digital infrastructure, both fixed line and mobile networks to ensure growth and productivity.”

Just scrolling through Twitter, you can see how many people were impacted by the outage. This impact cannot just be restricted to personal activities, as there have also been several reports of sole-traders and critical services being unable to do their jobs. Whether we’re talking district nurses being unable to do house-calls because they are unable to rely on mapping apps or a plumber who can’t access his emails, the consequence is incredibly real and financial.

Earlier this year we had the chance to speak to various O2 executives, including CEO Mark Evans, over dinner, and the enterprise market was a segment targeted for growth at the firm. This is a lucrative market currently dominated by EE and Vodafone, though with this outage you have to wonder what the cost will be for O2. Joe Bloggs having to speak to someone on the bus is a minor inconvenience but shutting down a business which relies on mobile to work effectively is a completely different matter.

On one side of the coin, you have to feel a bit sorry for O2. This is a business which has been effectively shut down due to a fault from one of its network partners. Customers will not actually care what the problem is, they only deal with O2 therefore O2 should shoulder the blame. The negative impact on brand credibility and the company’s ability to offer basic services will certainly be questioned by some. Conversely, Ericsson’s share price has actually gone up a few percent since this news broke, perhaps reflecting the apparently swift resolution of the crisis.

However, you also have to wonder whether O2 is itself culpable. Should this be considered a due diligence or supply chain issue? As you can see from the tweet below, Heavy Reading’s Gabriel Brown doesn’t feel O2 is innocent through the saga:

And sticking with Twitter to finish, as you can imagine there were certainly a few people who had fun with the toil and torment of others. We’ve copied a couple of our favourite tweets from the last 24 hours below. Enjoy.

Ericsson rumoured to be behind O2 and Softbank outages

With O2 UK and Softbank customers being thrown back to the 90s after a network outage, reports have emerged suggest Ericsson was the root cause.

At roughly 5am this morning, O2 customers suffered the dreaded fate of a data-free diet, while Softbank also experienced similar problems. In Japan, the firm suggests 3G networks kicked in to partly compensate for the 4G issues, though the same could not be said for the UK. As a Giffgaff customer, your correspondent can confirm the data desert, at least in this part of East London at the time of writing (11.30am, UK)

“We’re aware that our customers are unable to use data this morning,” O2 has said in a statement. “One of our third-party suppliers has identified a global software issue in their system which has impacted us.

“We believe other mobile operators around the world are also affected. Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wifi wherever they can, and we apologise for the inconvenience caused.”

Although there is no official confirmation of which supplier this might be, the FT has sources which point the finger towards Ericsson. According to two people directly involved, an issue relating to software in the systems provided by Ericsson has been identified.

In an email to Telecoms.com, Ericsson said it is working with customers to correct the issue, though it did not explicitly confirm or deny whether it was the root case of the problem.

And while it hasn’t been heavily reported, O2 and Softbank are not alone with this issue. Several telcos around the world, mainly situated in Asia, are rumoured to be experiencing similar problems. There might be a common factor between the issues…

It’s unlikely the guilty party will take ownership for the fault until the PR madness has died down, especially considering telcos will be more stringently examining the track records of suppliers ahead of the 5G bonanza, and in light of recent issues at Huawei and ZTE.

Just so you are aware, O2’s primary suppliers for network infrastructure are Ericsson and Nokia.

For O2, the ‘double-edged sword’ metaphor is hitting home hard right now. Other telcos have experienced outages in the last couple of months, EE and Vodafone certainly did in October, though it wasn’t as widely reported or condemned on social media. In leading the market share rankings in the UK, an O2 outage would certainly impact more customers than other similar examples.