Ericsson facing £100 million damages bill for network outage – report

With smartphones around the world being reduced to doorstops thanks to Ericsson’s software issues, the vendor is potentially facing a damages bill in excess of £100 million.

The full extent of the impact is a bit hazy for the moment, though it is rumoured to be much more than is officially known. What we do know is the O2 network was down for almost all of Thursday, Softbank’s customers were plunged back to the paper days and there are ‘several’ other customers who were impacted by the issue.

Ericsson has confirmed there were others, though it is remaining tight-lipped on who these operators actually are. That said, as you can see from the crowdsourced data below from wireless coverage mapping company Opensignal, it would probably be a fair assumption to add Vietnam’s Mobifone to the list.

Opensignal Graph

Looking at the bill, The Telegraph has reported Ericsson will be facing a global bill of £100 million to compensate customers for the oversight. Ericsson is yet to respond to a request for comment, though having admitted being the root cause of the data dessert there will certainly be conversations concerning compensation.

IWe understand O2 CEO Mark Evans is currently in discussion with the Ericsson management team regarding the whole issue, and the topic of damages will be included on the agenda. Ericsson UK and Ireland CEO Marielle Lindgren is certainly involved in these meetings, though we have not been able to confirm whether these specific discussions have been escalated all the way up to Group CEO Börje Ekholm.

In fairness to O2, it has not palmed off responsibility completely. Last week, it confirmed it would be reimbursing customers as a result of the outage. Pay Monthly, SMB business and mobile broadband customers will be credited with two additional days of monthly airtime subscription charges by the end of January. Pay-As-You-Go customers will be given a 10% credit on a top-up in the New Year, while PAYG Go mobile broadband customers will receive a 10% discount on a Bolt On purchase.

While some might suggest O2 is not completely blameless and should have had processes and systems in place to compensate for such instances, as the root cause of the disruption does lie elsewhere it does have some bargaining power in the talks. The rumoured £100 million bill would not all be credited towards O2, though as it seemed to be impacted the worst, it may well be seeking a large cheque to compensate for lost revenues associated with reimbursing customers.

Ericsson has narrowed the root cause of the issue down to two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). These nodes in the core of O2’s network caused the calamity, as the license for the software expired.

For Ericsson, there might well be no choice. Some are suggesting up to 20 operators were impacted by the expired software license and considering the importance of relationships ahead of the up-coming 5G bonanza, the vendor will need to do everything in its power to ensure favourable terms. It seems buying its way out of this mess might be the only sensible route to take.

Nokia has a great week by simply not screwing anything up

Christmas has come early for Finnish kit vendor Nokia, with its competitors bestowing it with presents of breath-taking generosity.

Huawei was the first to smash through the secret Santa threshold, at least indirectly, when BT confirmed that Huawei would not be considered to contribute to its 5G core infrastructure and that it’s also being removed from legacy cores. This marked the first time a UK operator had directly addressed the matter of Huawei and 5G kit and would appear to be part of the move against Huawei by the US and its allies.

But that was just a stocking filler compared to what the US had under the tree. In a gesture of unprecedented aggression and hostility the US persuaded Canada to arrest Huawei’s CFO and daughter of its founder, Meng Wanzhou, and wants to extradite her to the US in order to face as yet unspecified charges.

This move is unrelated to all the stuff around 5G security and concerns suspected violations of US sanctions against certain countries. This is what drove ZTE so close to extinction earlier this year and would appear to be as much a front in the trade and political war between the US and China as it is any specific point of order. It also marks a significant escalation since no ZTE execs were subjected to this kind of legal rough treatment, and to add insult to injury it looks like some ZTE documents are being used in evidence against Huawei.

Huawei’s misfortunes alone must have had Nokia execs reaching for the breakfast bucks fizz, but when it emerged that Ericsson was responsible for major outages at some of its MNO core network partners they presumably ordered a whole new shipment of champers. The fact that Huawei is desperate to divert attention onto the Ericsson thing too must have resulted in port and brandy being added to the celebratory mix. Nokia’s resulting hangover may have been slightly exacerbated by the apparently effective damage limitation done by Ericsson, but on balance it can reflect on a pretty catastrophic week for the competition.

Nokia has sent out a few press releases this week but whatever incremental achievements they claim pale into insignificance compared to the misfortunes of Huawei and Ericsson. In the run up to Christmas Nokia would be well advised to focus all its resources on simply not screwing anything up and letting its competitive environment take care of itself.

O2 up and running but how much damage has been done to them and Ericsson?

With O2’s UK network back up and running, the 32 million Brits who depend on it have been returned to the digital era, but you have to wonder how big the fallout from this disaster will be.

With 3G data services restored late Thursday evening, and 4G getting the green-light early Friday morning, a stressful period comes to a close. Now the more difficult questions need to be asked to understand why this happened, why O2 is rumoured to have had to cancel its Christmas party last night and what the consequences of the chaos will actually be.

“We can now report that our 4G network has been restored,” said an O2 announcement. “Our technical teams will continue to monitor service performance closely over the next few days to ensure we remain stable. A review will be carried out with Ericsson to understand fully what happened. We’d like to thank our customers for their patience during the loss of service on Thursday 6 December and we’re sorry for any impact the issue may have caused.”

In fairness to O2, the simple thing to do here would have been to shift all of the bad press and finger pointing towards the root cause of the problem, Ericsson, but it has managed the saga as well as could be expected.

And while it might have taken a couple of hours for the Swedes to come clean, they finally did, as you can see below:

“The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers,” said Börje Ekholm, Ericsson CEO. “We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.”

While Ericsson is continuing to do root cause analysis on the fault, the issue has been narrowed down to two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). These nodes in the core of O2’s network caused the calamity, though it was not alone.

Softbank experienced the same issues over in Japan, while there are rumours several telcos in Asia also had network outages. Ericsson has confirmed it impacted other customers, but it is not its place to name said customers; O2 and Softbank made their own announcements, so it is up to the operators themselves. Mobifone in Vietnam could well be one of these customers, with its own network shutting down at 11.30am local time.

Before we move on, it’s worth drawing attention to the graph below from web performance and security company Cloudflare, just for context.

Cloudflare graph

As you can see the drop was incredibly pronounced, though the minor traffic which can be seen has been attributed to O2 customers who were roaming outside the UK. These customers do not seem to have been impacted by the data-doomsday.

One of the big questions which is now floating around concerns the fallout. With people and society on the whole relying so heavily on data networks, any fear of sub-par or non-existent performance will have a negative impact. Outages are something people should realistically expect to happen every now and then, but the severity and length of this one is certainly noteworthy.

“The disruption shows how much importance we place on a mobile device,” said telco and media analyst Paolo Pescatore. “Without connectivity, people are stranded, and businesses cannot compete. Furthermore, it underlines the need for continuous investment in the UK digital infrastructure, both fixed line and mobile networks to ensure growth and productivity.”

Just scrolling through Twitter, you can see how many people were impacted by the outage. This impact cannot just be restricted to personal activities, as there have also been several reports of sole-traders and critical services being unable to do their jobs. Whether we’re talking district nurses being unable to do house-calls because they are unable to rely on mapping apps or a plumber who can’t access his emails, the consequence is incredibly real and financial.

Earlier this year we had the chance to speak to various O2 executives, including CEO Mark Evans, over dinner, and the enterprise market was a segment targeted for growth at the firm. This is a lucrative market currently dominated by EE and Vodafone, though with this outage you have to wonder what the cost will be for O2. Joe Bloggs having to speak to someone on the bus is a minor inconvenience but shutting down a business which relies on mobile to work effectively is a completely different matter.

On one side of the coin, you have to feel a bit sorry for O2. This is a business which has been effectively shut down due to a fault from one of its network partners. Customers will not actually care what the problem is, they only deal with O2 therefore O2 should shoulder the blame. The negative impact on brand credibility and the company’s ability to offer basic services will certainly be questioned by some. Conversely, Ericsson’s share price has actually gone up a few percent since this news broke, perhaps reflecting the apparently swift resolution of the crisis.

However, you also have to wonder whether O2 is itself culpable. Should this be considered a due diligence or supply chain issue? As you can see from the tweet below, Heavy Reading’s Gabriel Brown doesn’t feel O2 is innocent through the saga:

And sticking with Twitter to finish, as you can imagine there were certainly a few people who had fun with the toil and torment of others. We’ve copied a couple of our favourite tweets from the last 24 hours below. Enjoy.

Ericsson rumoured to be behind O2 and Softbank outages

With O2 UK and Softbank customers being thrown back to the 90s after a network outage, reports have emerged suggest Ericsson was the root cause.

At roughly 5am this morning, O2 customers suffered the dreaded fate of a data-free diet, while Softbank also experienced similar problems. In Japan, the firm suggests 3G networks kicked in to partly compensate for the 4G issues, though the same could not be said for the UK. As a Giffgaff customer, your correspondent can confirm the data desert, at least in this part of East London at the time of writing (11.30am, UK)

“We’re aware that our customers are unable to use data this morning,” O2 has said in a statement. “One of our third-party suppliers has identified a global software issue in their system which has impacted us.

“We believe other mobile operators around the world are also affected. Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wifi wherever they can, and we apologise for the inconvenience caused.”

Although there is no official confirmation of which supplier this might be, the FT has sources which point the finger towards Ericsson. According to two people directly involved, an issue relating to software in the systems provided by Ericsson has been identified.

In an email to Telecoms.com, Ericsson said it is working with customers to correct the issue, though it did not explicitly confirm or deny whether it was the root case of the problem.

And while it hasn’t been heavily reported, O2 and Softbank are not alone with this issue. Several telcos around the world, mainly situated in Asia, are rumoured to be experiencing similar problems. There might be a common factor between the issues…

It’s unlikely the guilty party will take ownership for the fault until the PR madness has died down, especially considering telcos will be more stringently examining the track records of suppliers ahead of the 5G bonanza, and in light of recent issues at Huawei and ZTE.

Just so you are aware, O2’s primary suppliers for network infrastructure are Ericsson and Nokia.

For O2, the ‘double-edged sword’ metaphor is hitting home hard right now. Other telcos have experienced outages in the last couple of months, EE and Vodafone certainly did in October, though it wasn’t as widely reported or condemned on social media. In leading the market share rankings in the UK, an O2 outage would certainly impact more customers than other similar examples.

O2 bags Midlands driverless gig

O2 has announced a partnership with Wireless Infrastructure Group (WIG) to deliver connectivity for driverless vehicle test bed trials in the West Midlands.

With the UK government foolishly promising driverless cars will be on the roads by 2021, this is one of a number of trials located in the Midlands to make the wayward ambition a reality. As part of the agreement, O2 claims it will deliver Europe’s largest fibre connected small cell network along a 50-mile route that runs through Birmingham and Coventry.

“Following our successful deployment of the UK’s first centralised radio network (C-RAN) in Aberdeen, in partnership with WIG, we will be using this same cutting-edge technology across what we expect to be Europe’s largest fibre connected small cell network,” said Brendan O’Reilly, CTO at O2. “Mobile powers our modern world and 5G has even more potential to move Britain forward which is why we’re excited to be working together with WIG to continue to build this technology into the fabric of our cities and communities.”

“We are delighted to be working with O2 and WIG to test autonomous vehicles on roads here in the West Midlands,” said Andy Street, Mayor of the West Midlands. “As the heart of the UK’s future mobility research and development, and the UK’s first region-wide 5G testbed, the West Midlands is well-positioned to create thousands of new jobs in the industries of the future.”

With the UK desperate to prove it is at the forefront of the technology world, the UK government has promised driverless cars will be commercially available by 2021, a laughable ambition when you think of the work, both from a technological and administrative perspective, which will have to be done ahead of this point. The Midlands is certainly a beneficiary here, as the region has been attempted to create a regionalised technology hub around electric and driverless vehicles.

We’re sceptical driverless cars will be anywhere near a reality in the near future, though at least it is another usecase for 5G ROI.

Q3 validates O2 indifference towards convergence

Telefonica’s UK business O2 has continued a strong 2018 performance with a 7.9% increase in revenues in the third quarter, while it greedily captured an additional 120,000 subscribers.

The results perhaps justify the businesses decision not to enter into the convergence fight. Back in July, CEO Mark Evans confirmed the business would continue to focus on its mobile-only proposition, and wasn’t convinced entirely by the idea of bundled services. This statement is certainly contradictory to many telcos across the world, including its own cousin, Telefonica Germany, which plugged 5G FWA at Broadband World Forum. That said, the numbers speak for themselves.

Over the last three months, total revenues stood at £1.5 billion, up 7.9% year-on-year, while mobile service revenues grew by 3.4%, thanks to customers choosing larger bundles and MVNO growth. The O2 network now has 32.3 million customers, including MVNOs such as Lycamobile, making it the busiest network in the UK. Churn was also down to 1%, which O2 claims is the best in the UK.

“We continue to put the customer at the heart of our business, delivering leading propositions and unique customer experiences, as demonstrated by the launch of our revolutionary O2 Custom Plans, exclusively available in our direct channels,” said Evans. “O2 Custom Plan offers customers real choice, by giving them control, flexibility and transparency, and has once again driven the O2 point of difference in the market.

“Our on-going commitment to invest in our network includes enhancing 4G connectivity and preparing the ground for 5G. As champions of mobile we continue to build for the future, where mobile is one of the most powerful opportunities to strengthen the UK economy and enrich our society.”

This laser like focus on mobile is probably best for everyone involved. Despite O2 leading in the market share race, it has consistently been condemned for having the worst network in the UK. This has been confirmed quarter after quarter, by a variety of different sources. Some might come to the conclusion the consistency of poor performance simply suggests the management team does not care that much. However, efforts are being made to improve this record.

In the most recent spectrum auction, O2 claimed all the available 2.3 GHz spectrum to enhance its 4G offering. This spectrum has already been put to use, while most recently O2 suggested it was going to improve connectivity in 339 rural communities throughout the UK. The business is investing in its network, with the financial results indicating O2 spent £192 million on CAPEX over the quarter, which works out at roughly 12.5% of total revenues. This is not the highest around, but it is a healthy number.

O2 is the first of the UK MNOs to release its financial results for the third quarter, so there isn’t a fair comparison to make at the moment. However, 7.9% growth is going to be a very tough number to beat. Perhaps there is something in this ‘do what you know how to do’ mentality from O2.

O2 commits to plugging 339 farmer Johns into the digital economy

O2 has announced it will start pumping enhanced 4G into some of most notorious not-spots throughout the UK, with 339 communities set to receive the connectivity boost.

While the digital divide is clearly still present across the UK, it does seem O2 is attempting to make use of 4G spectrum acquired in the last auction to counter the imbalance. The last 12 months has certainly seen O2 up its game on the connectivity front, Ofcom confirmed O2 had delivered against its commitment to provide 98% indoor 4G coverage and 90% geographical coverage across the UK earlier this year, and this appears to be a continuation of the positive work. O2 has stated in intends to improve the 4G experience for an additional 339 communities of more than 100 people across the UK by the end of the year.

“We know mobile has the power to make a real, positive difference to people’s lives and businesses in rural communities across Britain,” said O2 UK COO Derek McManus. “That’s why we’re proud to be investing in 4G connectivity for more than 330 rural areas by the end of this year. Technology never stands still, which is why we are always looking for the right partners and investing in our future network. Whether trialling 5G to support a future-proof, mobile Britain, or ensuring the remotest parts of rural Britain can connect to 4G, for O2, this is about continuing to invest in all areas – not one at the cost of the other.”

“4G coverage is improving all the time, but there’s more to do, particularly in rural areas,” said Digital Minister Margot James. “We’ve already reformed planning laws to make it easier and cheaper to install and upgrade digital infrastructure, and it’s great to see O2 and the rest of industry responding to ensure more people in rural Britain can share the brilliant benefits of 4G connectivity.”

Thanks partly to an enhanced mobile experience, O2 has pointed to a report from Development Economics which suggests tourism, transport and manufacturing segments could receive a financial boost with improved connectivity. Perhaps this is one of the most notorious statistics associated with the digital divide, but Development Economics predicts connectivity parity could add as much as £141 million a year to the 14,000 rural businesses who are missing out on the full digital experience.

Although this is a positive step forward, let’s not forget O2 has a lot of catching up to do, it certainly isn’t uncommon to see the brand slumping at the bottom of the mobile performance rankings. Opensignal’s most recent report assessing the performance of the four UK MNOs had O2 sat in last place for all categories aside from latency (3G and 4G) and availability, where it was second behind EE. Back in August, Rootmetrics also had O2 at the bottom of the pile for almost every category.

The performance of the network is certainly a dent in the O2 pride, but it has still managed to claim top-spot in the market share leagues. Although MNOs should stride towards creating the best possible experience for customers, O2’s top and bottom standings demonstrate buying decisions are more than performance orientated. The big differentiator between O2 and rest of the UK MNOs is its loyalty programme, Priority, which does appear to be paying dividends, while Giffgaff is still proving a successful venture to attract a new SIM-only audience. Such is the success of Giffgaff over the last few years, Three has creating its own version in Smartie, while Vodafone has launched Voxi.

That said, building the customer experience in rural areas of the UK will only add to the success of the telco, creating a more interesting proposition for customers which might have ignored the brand in the past. O2 has been swimming against the tide when it comes to convergence trends, choosing to focus exclusively on mobile, a decision which is increasingly looking justified.

EE holds onto Opensignal MNO crown

EE has held onto its position as the best performing UK MNO according to the latest figures from Opensignal.

For 4G download performance, EE maintained its leadership position with average download speeds of 29 Mbps between June and August, while it also led in upload speed, latency and availability. This is not to say there weren’t improvements elsewhere, Vodafone grew its average 4G download to 21.9 Mbps, though Three’s dropped with the telco slipping down to third place in the performance rankings.

4G might not have been a fruitful playground for Three, but it did steal the top-spot for 3G speeds off EE. With average speeds of 7.8 Mbps it edged just ahead of EE at 7.2 Mbps, though this will come as little comfort as telcos increasingly look to re-farm 3G spectrum to bolster 4G performance.

Interestingly enough, O2 is still maintaining its position as the leading telco in terms of market share, despite a damning review of the telco from Opensignal. O2 sat in last place for all categories aside from latency (3G and 4G) and availability, where it was second behind EE. O2 might arguably have the weakest network in the UK, the power of promotions seems to counter this position. The Priority loyalty programme is perhaps proving its worth in gold here.

While many will preach the benefits of having the best network, these figures show it’s not always about being the fastest.

Opensignal Awards

Three and O2 put positive spin on sh*t connectivity

Three and O2 have signed a deal with SSE Enterprise which will enable the pair to access its fibre ring, part of which is located in the Thames Water waste water network, to improve connectivity backhaul capabilities.

With 5G on the horizon, and demands for improved 4G experience, partnerships like this will be key to not only improve backhaul but also enable further 4G and 5G deployment by connecting cell sites and masts. Robust aggregation of fronthaul and backhaul access is necessary in order to provide greater resiliency, increase capacity and reduce latency. In other words, fibre is king if you want to meet the demands of the data-craving consumers.

Just to put things in perspective, UK data usage is set to grow thirteen-fold between 2017 and 2025 according to data from Ofcom, with the first 5G offerings set to hit the market during 2019. To meet this demand, 5G is key, enhanced 4G experience is key and backhaul is key. Fibre rules the roost.

“Networks will fundamentally underpin the UK’s digital economy and will be essential to 5G services,” said Colin Sempill, MD of SSE Enterprise Telecoms. “With this high capacity core in the London sewers, Three UK and O2 are tapping into our unique, diverse connectivity and putting their networks in a strong position to trial 5G offerings, while enhancing existing services for their customers.”

“New and innovative models are essential to improving the customer experience of mobile networks by increasing the availability of dark fibre for mobile backhaul and driving competition in the market,” said Dave Dyson, CEO of Three. “Our partnership with SSE Enterprise Telecoms and O2 is one of the first examples of using existing infrastructure to improve connectivity in an urban area.”

“This kind of agreement is essential to allow for continued investment and improvement of services for our customers,” said Brendan O’Reilly, CTO at O2. “This partnership is a great example of SSE Enterprise Telecoms, Three UK and O2 coming together in a collaborative and innovative way to address the growing challenge and pressure of obtaining access to fibre for mobile backhaul in the UK.”

SSE has been running some interesting projects to improve the speed and reduce the cost in terms of laying fibre over the last couple of months. In this example, SSE is licensed to lay fibre optic cables throughout Thames Water’s waste water network which it claims can reduce network deployment costs by 60% and speed up deployment by up to ten times. As the sewers can be as deep as 10 metres, laying the fibre in this way can decrease accidental fibre breaks as the traditional means see the cables laid only 12 inches below the surface. The waste water network is also geographically very wide-spread, it is a creative solution to the challenge of laying fibre more efficiently, even if it is a bit of a smelly one.

The agreement with Three and O2 will see approximately 100 points of connectivity exit from this central London sewer network via two BT Exchanges. By partnering with SSE Enterprise Telecoms, Three and O2 can operate their own Central London Area (CLA) network, while also accessing spare fibre ducts for future initiatives in London.

The last couple of months have seen SSE ink numerous deals with the telcos, including a separate partnership with Three where it has begun facilitating fibre optic connections for the telcos 20 core data centres. Last October, SSE also won a competitive tender from advanced fibre broadband specialists Grain to deliver network connectivity to Countesswells, a new £800 million development in Aberdeen.

Ronan Dunne said to be in the running for the BT CEO gig

Current Verizon exec and former O2 UK boss Ronan Dunne is reported to be one of the people BT is considering to replace Gavin Patterson.

The rumour comes courtesy of the Telegraph, which has sources that claim Dunne was recently over here to speak to BT about the opportunity. That’s about it as far as the Telegraph story goes – no comment from either Dunne or BT and it could just be blind speculation. But it is also plausible as Dunne has a lot of relevant experience and could be keen on the opportunity having missed out on the top job at Verizon and an apparent sense of unfinished business in the UK.

Current BT CEO Gavin Patterson decided to throw in the towel earlier this year following a bunch of negative events afflicting BT under his tenure. It looks like a pretty exasperating gig, not least due to the tightrope the former state monopoly has to walk with regulators. Dunne’s move to Verizon seemed to be prompted by his own negative regulatory experiences following the blocking of O2’s merger with Three UK, but maybe a couple of years away has renewed his appetite for the fight.

Among the other frontrunners for the top job at BT are current consumer boss Marc Allera and various other BT senior execs as well as anyone else a headhunter might find when the Goole ‘UK telecoms CEO’. Informa’s Stephen Carter has apparently distanced himself from the position without entirely ruling it out and they could just stick the head of another utility in if they wanted to play safe.