For the third year in a row, the consulting, engineering and testing company P3 communications GmbH and connect magazine teamed up to evaluate the performance and service quality of the UK’s mobile networks. Based on the findings of an extensive and comprehensive series of tests, the 2016 P3 connect Mobile Benchmark UK produced some surprises including two of the candidates achieving exactly the same score.
Results in a Nutshell
P3‘s network benchmarks are widely accepted as being both highly authoritative and objective. The carefully designed methodology involves drive and walk tests in 13 large cities in the UK, as well as drive tests in smaller towns and along trunk roads and motorways. Great attention to detail includes the use of up-to-date LTE Cat 9 smartphones for the data tests and the use of the most comprehensive mobile plans available from each operator with adaptations to reflect real life conditions. The cities in which we tested accounted for more than 14 million people, or approximately 23% of the UK population.
After an extremly close race, EE and Vodafone came joint top, with both competitors achieving 803 out of a possible maximum of 1,000 points. EE scored highly with its very strong data performance, while Vodafone achieved the best results in the overall voice category. Vodafone also achieved the biggest improvement on the 2015 scores. Three came third overall and was also third in the individual voice and data disciplines. This operator scored best in the voice tests conducted on connecting roads and also achieved good results in smaller towns.
O2 also achieved a considerable improvement in comparison to its 2015 results. The O2 network scores best in the voice tests conducted in London and ranks second in the voice category overall, leaving room for improvement, particularly in the data category.
The most positive outcome of this year‘s mobile benchmark however, is that three out of four operators achieved “good“ ratings – a first in the UK. This is proof not only of the huge efforts the providers took in extending their networks, but is also evidence of the influence of the P3 connect Mobile Benchmark on the development of the country‘s networks.
The UK’s operators
With approximately 30 million customers, EE (formerly Everything Everywhere) is the biggest mobile network operator in the UK. Since January 2016, EE has been part of the British Telecom Group. EE started offering a 4G service in October 2012, that recently surpassed the 12 million subscriber landmark. The operator claims that its 4G coverage today reaches more than 95% of the UK population. The geographic 4G coverage currently is approximately 67%, with an ambition to hit 95% by 2020. EE operates its 4G network both at 1800 MHz and 2600 MHz. LTE The operator announced in September 2016 that it will support Category 9 (Cat 9) capabilities on its LTE Advanced network. So far, Cat 9 has only been activated in Wembley Stadium. Other sites in London, Birmingham and Manchester are to follow in 2016/2017.
O2 claims to be the second largest mobile network operator in the UK with approximately 23 million customers. Formerly a subsidiary of British Telecom, O2 plc was purchased by the Spanish telecommunications company Telefónica in 2006. Today, the company also owns half of the mobile virtual network operator Tesco Mobile which operates on the O2 network in the UK. O2 started providing its 4G/ LTE service in August 2013 and has expanded this service across the UK since. In September 2016, O2 claimed to cover over 70% of the UK population with its 4G service, striving to meet a regulatory requirement of covering 98% of the indoor population by the end of 2017. O2 operates its 4G network mainly on 800 MHz with limited additional 1800 MHz coverage in London. Concentrating on enlarging its 4G footprint, the company does not offer data speeds exceeding 100 Mbps (LTE Cat 3) at the time of writing.
Three UK is a subsidiary of Hutchison Whampoa and launched its mobile service in the UK in 2003. As a relatively young operator Three started as a 3G-only network supplemented by 2G via national roaming. In December 2013, Three began to roll out its 4G/LTE service and expanded it rapidly all over the UK. With about 8.8 million customers Three is the smallest mobile network operator in the UK but claims to carry over 40% of the nation‘s data traffic. Offering the cheapest price for 4G and unlimited data plans (excluding tethering) may well support this claim. Aiming for large 4G coverage In addition to 1800 MHz, Three offers 4G also at 800 MHz. Up to now, Three confines its 4G network to LTE Cat 3 or a maximum of 100 Mbps. The company claims to cover 98% of the UK‘s population with 3G. 4G coverage is somewhat lower, but should soon reach a comparable spread.
Vodafone UK is part of the Vodafone Group which is also headquartered in the UK. The Vodafone Group owns and operates networks in 21 countries. Vodafone UK launched 4G/LTE in 2013. With around 19.5 million subscribers, Vodafone is the third largest mobile network in the UK after EE and O2. In June 2012, Vodafone and O2 signed a deal to “pool“ their network technologies, creating a single national grid of 18,500 transmitter sites. Both networks however announced they would continue to use their own independent spectrum. Vodafone currently operates 4G/LTE at 800 and 2600 MHz and has started to refarm also some of its 3G spectrum at 2100 MHz to 4G. Carrier aggregation on its way Under its current 4G+ rollout, Vodafone is preparing to combine up to three frequency bands with “Carrier aggregation“ in high density areas. This technology could provide data rates of up to a maximum of 700 Mbps, with current Cat 9 devices making use of up to 450 Mbps.
A close look at UK networks
P3 communications GmbH, based in Aachen, Germany, is a world leader in mobile network testing. It is part of the P3 group, with over 3,000 employees worldwide and a turnover of more than €300 million. P3 is in partnership with the German telecommunications magazine connect, which has more than 20 years of editorial expertise and is one of the leading test authorities in Europe for telecommunications products and services.
Together, P3 and connect have been conducting the most important network benchmark test in Germany for nearly 15 years, extending it to Austria and Switzerland in 2009. Starting in 2014, P3 has also been conducting benchmarks in Australia and the UK, expanding its public mobile network benchmarks to cover the Netherlands and Spain last year.
In 2015 alone, P3 compiled more than 60,000 measurement hours in 47 countries, with its test vehicles covering more than 750,000 miles. As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality – examining both voice telephony that makes up 40% of the total result as well as data connectivity that accounts for 60% of the score. P3‘s network benchmarks are widely accepted as a completely objective authority.
Against this background, it was especially interesting to find out whether the UK carriers were able to improve on last year‘s results, bearing in mind that when compared with other European operators, all UK networks revealed room for improvement. See for yourself how they fared and get ready for some surprises.
In order to check out the voice quality of the UK‘s mobile networks, P3 conducted drive and walk tests in 13 cities with more than 100,000 inhabitants and also covered a large number of towns and connecting roads all over the UK. In total, the test vehicles have covered 4,456 miles (see detailed description of the methodology on page 9). Especially in the cities, Vodafone achieved the best results over its competitors, with O2 following at close distance. In the drive tests covering smaller towns, O2 excels, Three and Vodafone follow closely behind and EE comes last. Voice telephony on trunk roads and major highways obviously is a specialty of Three, with Vodafone ranking second and EE as well as O2 showing some weaknesses in this category.
When looking at the call set-up durations, last year‘s overall winner EE exhibits a 1.5- to-2-second deficit behind its competitors. All in all, O2 and Vodafone managed to steal last year‘ s top position in the voice category from Three. Success ratios and speech quality are mostly on similar levels, with a slight advantage for Three in rural areas as well as in terms of overall speech quality. While O2 is quite strong in smaller towns, it falls somewhat behind on connecting roads.
Voice results at a glance
In all tested voice scenarios, last year‘s overall winner EE is losing some ground. Three, which took the lead in the voice category in 2015, has to give way to Vodafone and O2 this year. 2016’s winner in the voice category across all scenarios is Vodafone – with both O2 and Three following closely.
While all UK mobile network providers nowadays offer good coverage with their 4G services, there are differences in their rollout strategies. EE and Vodafone chase each other with continually growing data rates that currently go up to 450 Mbps, based on the so-called carrier aggregation (the combination of multiple carrier frequencies). In contrast, O2 and Three stick with a solid 100 Mbps and mainly focus on enlarging their 4G footprint.
P3‘s testing takes both aspects into account – the benchmarking of web-page downloads as well as file downloads and uploads rewards fast throughputs. At the same time, it assesses the networks‘ availabilty and stability by examining success ratios.
The testing of YouTube playback recognizes that this popular video service has been introducing adaptive bit rates recently. This decision of the streaming provider aims at a better user experience, surrendering pixel resolution in favour of stable playback. As a consequence, besides success ratios, start-times and the absence of interruptions, the average value of the obtained video resolution became another important performance indicator.
In the drive tests and walk tests that P3 conducted in the 13 large cities of the UK, EE takes a clear lead in the data category. With this year‘s results EE maintains the high performance level it showed in our 2015 benchmark. Vodafone is following at a distance of only 3 to 4 percentage points, while Three ranks third and O2 last – the latter showing obvious weaknesses especially in the walk test scenarios.
Vodafone emerged narrowly ahead of its competitors in the results from drive tests in smaller towns, with EE and O2 a mere 1% behind in this category. In this discipline, Three is falling back a little, showing slightly worse success ratios, taking slightly more time to download the test web pages and achieving lower download and upload rates in some test cases. Still, all four operators show viable results when it comes to data connectivity in smaller towns.
On the major, connecting roads, EE again takes a clear lead, providing fast data rates and achieving convincing success ratios. Three also shows strong results in this category, while Vodafone and O2 lag a little behind – especially with regard to the success ratios of web page downloads and file transfers. Nonetheless, with all success ratios being well over 90% even on the relatively demanding connecting roads, all four mobile network providers of the UK improved considerably over last year‘s results.
Data results at a glance
Especially in the data categories, the 2016 mobile network benchmark shows considerable improvement in all four UK networks. EE takes a clear overall lead in data services, with Vodafone narrowly ahead in smaller towns and overall second in the data test.
Being a major business hub and by far the most populous area in the UK, London is an especially demanding terrain in which to deploy and maintain a mobile network. This is why we regularly take a closer look to see how coverage in the capital compares with the rest of the country. In the drive tests and walk tests measuring voice services, O2 scores best and takes a lead over a second-ranking Vodafone. EE and Three share the third spot in this discipline.
In the London-based data tests, EE reclaims the top position that it already earned in the nationwide data category. Vodafone follows closely, just five points behind. Combining the voice and data results achieved in London, Vodafone comes out 5 points ahead of EE in this discipline, due to a good performance in both categories.
London results at a glance
In comparison to 2015’s test, all four carriers improved their services in the London area. The overall winner in the capital is Vodafone, with EE not far be hind. In any case, definite improvements are visible in London as well as in the rest of the UK.
The P3 connect Mobile Benchmark took place throughout September 2016. All samples were collected du- ring the day, between 8am and 10pm. The network tests covered inner-city areas, outer metropolitan and suburban areas. Measurements were also taken in smaller towns and on trunk roads and motorways. The combination of test areas had been selected to provide a significant series of test results covering the UK population in England, Scotland, Wales and Northern Ireland. The areas chosen for the 2016 test account for more than 14 million people, or approximately 23% of the total population of the UK.
P3 conducted the tests with four drive-test cars, equipped with arrays of Samsung Galaxy S5 Cat 4 smart- phones (Voice) and Samsung Galaxy S7 Cat 9 smartphones (Data) for the simultaneous measurement of voice and data services. Additionally, two teams conducted the walk tests, measuring voice and data performance.
For the voice walk tests, one wal- king operator contacted a counterpart located in one of the drive test cars. Furthermore, two smartphones per operator in each car were used for voice tests from car to car. The voice quality was evaluated based on the HD-voice capable and ITU standar- dised so-called POLQA wideband algorithm. In order to account for typical smartphone use scenarios during the voice tests, background data traffic was generated in a con- trolled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40% of the total benchmark results.
Data performance was measured using one smartphone per operator in each car. The radio access technology was set to LTE preferred mode in order to reflect the customer experience. All the tests were conducted with the best-performing mobile plan available from each operator and in a full drive and walk test mode covering big cities, small towns and major connecting roads, including motorways. The data scores account for 60% of the total results.
The test routes are shown on page 1 of this report. In the big cities and smaller towns indicated, the cars had to follow predefined routes. Altogether more than 80,000 speech samples were logged per operator, about half of them in 4G-preferred-to-4G-preferred mode, while the other half were 4G-preferred-to-3G-preferred. For the data benchmark about 50,000 samples were obtained per operator. About 60 % of the samples were obtained in big cities while the remaining 40% were collected in towns and on connecting routes.
To reflect the critical importance of service access and stability, the percentage of these two KPIs was the most heavily weighted. We also measured performance indicators for voice such as call set-up time and speech quality. For data, performance indicators included the average session time to access web pages or transmit small files, throughput rates and average start times and resolution for YouTube videos.
The score weighting reflects both the geographical distribution of the UK‘s population and a ranking of usage scenarios. Therefore, 675 of the total of 1,000 maximum points were assigned to the cities – 525 maximum points referring to the drive test results (data: max. 315 points, voice: max. 210 points in voice) and 150 maximum points reflect walk test results (data: max. 90 points, voice: max. 60 points). In towns and on connecting roads, only drive tests were carried out. In the towns, a maximum of 150 points is available, with a maximum of 90 points in the data, and 60 points in the voice category. To reflect the importance of connectivity while driving on the road network, a total of 175 points has been allocated to this category, divided into a maximum of 105 points for the data re- sults and 70 points for the voice results. The table on page 2 shows the percentage of maximum points that each operator reached in each discipline.
The race was extremely close, and even our fine-grained scoring scheme ended up with a photo-finish deadheat between two winners: Scoring exactly 803 points out of the possible maximum of 1,000 points, EE and Vodafone share the winning position this year. EE achieves joint top-spot on the back of its very strong data performance, while Vodafone scores best in the voice category. O2 is nonetheless very close when considering voice-only but is held back from achieving a better total by its limited data performance. Interestingly, last year‘s distinct winner EE achieves the weakest score in the voice category. This however does not mean that EE is particularly bad in this discipline – but that its competitors successfully made inroads on their opponent in this field. Three ranks third with good overall results.
Probably the most interesting and positive outcome of 2016‘s P3 connect Mobile Benchmark UK is that three out of four operators were rated “good“ despite the use of a more demanding scoring scheme than last year. This is an unprecedented result in this market. A glance at the development of the scores achieved in the last three years shows the great improvements that the UK operators implemented in this period. P3 and connect take pride in the assumption that our scrutinising benchmark has very probably played an important role in this process.