THE MOBILE NETWORK TEST IN THE UNITED KINGDOM
Download test here
In 2015 the Mobile Network Test in the UK is conducted the second time. Did Three, O2 or Vodafone manage to improve their performance enough to overtake last years clear winner EE?
When it comes to mobile network testing P3 communications is probably the world leader. Being active in Germany, the USA, Australia, Austria, Switzerland, Spain, UK and many other countries, the mobile communication experts from Aachen have accumulated a lot of knowledge. No wonder connect, the biggest European magazine on telecommunication has chosen P3 communications as its partner for the mobile network benchmark in Germany, Austria and Switzerland since 2002. The awareness of the public regarding the quality of mobile networks has driven operators to improve their networks considerably to stay competitive.
Focus on United Kingdom
But how does the United Kingdom compare to other countries? In 2014 the mobile network test showed EE as the clear winner followed by Three with the best voice performance. O2 and Vodafone followed almost head-to-head. The test also displayed much better performance in metropolitan areas than in smaller cities and on connecting roads. That fact and the population of 8.6 million people are good reasons to look at the results of the largest city London separately. The results can be found on page 7. Compared to the leading operators in Europe and all over the world all UK networks revealed room for improvement. This headroom gives every operator the opportunity to make a big step forward and overtake formerly better competitors. It will be interesting to see which operator used the last year most effectively for its improvements.
LTE is a great technology for handling large amounts of data with high speed. This is because it was designed to handle the fast growing amount of data present since the introduction of smartphones. But it was not designed to handle voice calls. As a workaround a technique called circuit switched fall back (CSFB) changes the connection to UMTS or GSM whenever a call is made. This additional handover takes time and reduces reliability, so providers must step up efforts to keep the quality up to the level of the pre-LTE-era. Voice over LTE is the new alternative to CSFB. This technique allows the smartphone to place and receive voice calls without leaving the LTE network. But since LTE networks are all-IP networks and therefore not designed to handle real-time services like voice communication, VoLTE introduces new challenges to maintaining a good quality of service. All operators in the United Kingdom are experimenting with VoLTE, but Three is the first and only one to have launched it. As only a very small number of smartphones are able to use VoLTE, this service was not part of this years benchmark.
Voice in Big Cities
Nonetheless Three managed to show the best result in big cities. With 97.2 % call success ratio the youngest operator in the united kingdom obtained the best result. Vodafone ranked second at 96.7 % while Telefonica and EE had severe problems with one out of 20 calls. A call setup time between 5.3 (Three) and 7.3 seconds (EE) is typical for 4G-Networks and slightly higher than those measured in pre-LTE times. The speech quality is measured on the MOS scale, which goes from 1 (bad) to 5 (excellent). All operators accomplished MOS ratings between 3.3 and 3.6, with Three again being best. This makes Three the best operator for voice performance in big cities, with Vodafone in a close second place.
Voice in small cities and on connecting roads
It is easier to provide a stable network for speech in big cities with their high demand on network capabilities and a correspondingly dense network of base stations. Three only lost 2 % of reliability in small cities and on connecting roads and increased its advantage over the competition. EE with a 92.6 % call success ratio and Vodafone with 92 % followed at a distance while Telefonica lost more than one out of ten calls in rural areas. Telefonica‘s problem can also be seen at the speech quality. With a MOS value of 2.8 it is only slightly better than „merely understandable“. Call setup time stayed almost the same for all providers except of Three, who managed to reduce the time between dialing and an established call by another half a second.
Vodafone UK is part of the Vodafone Group which is also headquartered in the United Kingdom. The Vodafone Group owns and operates networks in 21 countries. Vodafone UK launched the first cellular network in 1985. The rollout of the first 3G network in the UK started in 2001. Now the company runs 2G at 900 and 1800 Mhz, 3G at 900 and 2100 MHz and LTE at 800 MHz and 2.6 GHz. During the last year Vodafone claims to have spent 1 billion pound on the networks. 200 Million have been spent in London alone to get the LTE population coverage above 99 %. VoLTE is currently on a trial with no release date yet. But the money is not only spent on LTE where Vodafone has licensed the largest bandwidth of all UK operators but also on extending 3G coverage in the formerly GSM-Band 900 MHz. This improves indoor reception for voice calls and older handsets without LTE and extends the reach of base stations. Population coverage across all mobile network classes is 98 % and 18 Million customers have access to this network.
With approximately 30 million customers EE is the biggest operator in the UK. The company‘s 4G network of EE was officially launched on 30 October 2012 and the operator has acquired 10 Million customers with a 4G plan since then. The population coverage is 93 % according to the operator. EE has started the rollout in a comparatively dense network at 1800 MHz in 11 big cities. This was followed with another layer at 2600 MHz at locations with high traffic. EE is the first in the UK to provide carrier aggregation (CA) to make the most out of modern smartphones. The deployment of 800 MHz LTE has recently started in rural areas to take advantage of the broader coverage of these lower frequencies. For the cities EE believes that 1800 MHz in a dense grid is perfect even for indoor use. EE‘s whole LTE network is VoLTE ready and a few hundred people are already using it in an expansive trial. But VoLTE will only be released to the public when the overall quality is as good as or better than HD voice over 3G.
Telefonica with its brand O2 claims to be the second largest mobile network operator in the UK with approximately 25 million customers. Telefonica has deployed 2G networks at 900 and 1800 Mhz, 3G networks at 900 and 2100 MHz. Currently the LTE roll-out is in progress. Until now the 4G networks serve 78 % of the outdoor population, which translates to 66 % indoor population coverage at 800 MHz. To extend its LTE bandwidth Telefonica has started to refarm part of its GSM 1800 spectrum to LTE and to carrier aggregate both frequencies. 98 % population coverage at the end of 2017 was promised by Telefonica as part of the licensing obligations. VoLTE trials are currently in preparation and is predicted to start in December 2015 with the launch planned for the first half of 2016.Telefonica is currently progressing with a big initiative to modernize its network. According to the operator as a part of this more than 3 billion pounds will be invested in 2G, 3G and 4G services during the next five years.
Three started its service Network on 2003 in the UK and five other countries. So it is a comparatively young operator starting in the upcoming age of mobile data with the aim to leave legacy behind. Consequently Three was a 3G-only network supplemented by 2G via national roaming. Nowadays 2G is only sparingly used for voice support. Three started its 4G rollout at the end of 2013 at 1800 MHz, which needs a denser network of base stations due to the high frequency. Now the 800 MHz rollout has started and population coverage of LTE is 50 % right now and is predicted to reach 65 % by the end of this year. This is a good foundation for the recently started VoLTE (Voice over LTE) service, which at this time is used where no other voice service is available. With about 8.8 Million customers Three is the smallest mobile network operator but claims to carry 42 % of the traffic in the UK. The modern network together with a no limit policy (excluding tethering) brings the average traffic to 4.65 GB per customer.
Although the LTE deployment started long before last year‘s mobile network benchmark, it is still in progress in 2015. With one year of ongoing LTE rollout we expect a considerable improvement in data speed. But with the shift from established legacy technology to the newer 4G standard, stability can always be an issue. Let´s see how the operators coped with the challenge of increasing speed while retaining a low error rate.
One of the most often used services is web surfing which was tested with a mixture of 9 popular live and one static web-pages. The winner of last year‘s benchmark EE managed to improve its session success ratio for live web sites from 97 to 99 %. The average session time to download a popular live web site decreased from 4.1 to 2.7 seconds. The other three operators showed almost identical session times between 3.1 and 3.2 seconds, but on reliability Vodafone clearly ranked second with a healthy success ratio of 97.4 %, while Telefonica followed with 96.8 %. With an error rate of more than 5 % Three still has some work to do.
When it comes to file up- and downloads the operators can show their best in speed and reliability. And with success ratios between 97,1 and 99.4 % the stability of the data sessions was indeed at a very high level. This is also true for the speed of the networks, although a little more differentiation can be seen. Therefore the mobile network benchmark measures the transfer rates in two different ways. With files of fixed size the overall speed of the network is tested including the time it takes to initiate a speedy data stream. This is important in cases where many small files will be transferred. On the other hand the „10-seconds-measurements“ show the peak performance once stable transmission is achieved. It gives insight on how the network behaves when large data files like high resolution photos or videos are up- or downloaded. The value given for „90 % faster“ is what a customer can reasonably expect most of the time. EE manages to increase its download performance by nearly 70 % and doubles the throughput of the next competitors. Vodafone und Telefonica still deliver more than 3 Mbit/s, which is enough for a good user experience, while Three at about 2 Mbit/s is a bit behind. The same superiority of EE is true for the upload performance. But this time the other operators are closer to EE and to each other and Telefonica comes closest to EE. The 10 seconds measurements show the same rank between operators. At more than 58 Mbit/s in download and more than 37 Mbit/s in upload in more than 10 % of the measurements EE highlights its high speed capabilities.
YouTube in standard and high definition
Video transmission was long regarded as impossible in a mobile network. But once the faster networks started to become reality, services like YouTube became the major source of traffic. Today YouTube amongst other video services is a commodity in the mobile world and every operator should deliver a highly reliable stream at least in standard definition (SD). This is true for EE and Vodafone with success ratios above 98 percent while Telefonica is slightly below and Three loses nearly 5 % to perfection at 95.1 %. With high definition (HD) Video demands are even more stringent on the network and only EE manages to stay above 98%.
DATA ACCESS IN SMALL CITIES
The cells in big city networks are usually densely located to fulfill the data demand of the public. In small cities a more widespread network of base stations can support the data demand, but coverage will suffer. In small cities EE loses nearly 6 % of its excellent success ratio and nonetheless stays on top with 93.2 % of successful live web site downloads. Three demonstrates good coverage in small cities and manages to be a close follower with success ratio of 91.3 %. Vodafone and Telefonica reached only around 80 %, so one out of five live web site downloads failed. Overall speed was good for all operators with session times between 3 and 4 seconds.
File Transfers and YouTube
The speed differences get bigger when it comes to file transfers. Customers interested in the best coverage should closely look at the speeds given, where 90 % of the samples are faster. Again EE succeeds in speed and stability, with Three as close follower in reliability. Vodafone and Telefonica show to a slightly lesser extent the same coverage problems also seen in the web-page download measurement. Looking at Youtube in standard definition EE and Three manage to be successfull in more than 90 %, while Vodafone is slightly below this mark.
DATA ON CONNECTING ROADS
Mobile cells covering roads reach only a comparatively small number of customers. This makes the deployment of a mobile network along motorways a relatively costly affair. Nonetheless EE and Three managed to deliver results in connecting roads performance which were remarkably close to the benchmark results shown in small cities. The key performance indicators in the web-page download sessions showed a success ratio of 92.4 % (EE) and 89.9 % (Three) which are slightly lower than in small cities. Vodafone with 75.8 and even more Telefonica with 75.6 % had noticeably more difficulties covering roads. The speed differences between big cities, small cities and connecting roads seem negligible by comparison. But speed differences become a major differentiator between the providers in the file transfer benchmarks. At the 3 MB download for instance EE reaches nearly 3 Mbit/s in 90 percent of the samples, while the other three operators stay shy of 1.5 Mbit/s. Vodafone and more so Telefonica again have issues with file transfer success ratios starting below 90 % and in many cases reaching around 80 %. On YouTube SD EE and Three show a 90 % success rate while Vodafone and Three stay below 80 %. Youtube HD makes things worse.
The drive test is a perfect tool to accumulate a large database, but a walk test provides an even more customer centric insight.
A trolley with a large battery pack, a compact measurement system and smartphones enabled a team from P3 communication to reach areas in big cities restricted for cars. This setup was used to measure in areas of high interest like railway stations and pedestrian areas. The routes between those were mainly bridged by walking, subways, buses or trams and were also part of the measurement.
EE lost a little bit of reliability in the telephony walk test compared to the drive test, while call setup time and the high speech quality stayed the same. Vodafone managed to increase its call success ratio from 96.7 to 97.1 % while Telefonica made an even higher jump from 95.5 to 97.1 percent. Nonetheless Three managed to prevail in the walk test with even slightly better results in the call success ratio (97.2 to 97.8 %) and consequently stayed on top. Three also reduced its overall call setup time and differences in speech quality were negligible.
While calls are handled in 2G (GSM) or 3G (UMTS) data usage should mainly rely on 4G (LTE). This is especially true for big cities with their high data demand. So it is interesting to see how a high percentage of LTE coverage enhances the results. At the web page download Vodafone slightly improved its result from the drive test and managed to show the highest reliability. EE ranked second with a slight reduction in success ratio but defended its first place in speed. Telefonica and Three again lost some ground. Speed was also the main advantage of EE at the file up- and downloads, the differences between drive- and walk test remained very low. This confirms the general validity of a sensibly executed drive test, even to judge the quality of networks for pedestrians. The success ratio of all operators was very close, except for some outliers. In Youtube performance with standard resolution Vodafone managed to reach the first place with an impressive success ratio of 99.3 %, while all other operators stayed slightly below 96 %. Placed overall first in the drive test, EE is also the first in the walk test. This is also true for the last place, while Vodafone and Telefonica although close together managed to change places.
More than 13 % of the population of the UK lives in its capital London, so clearly the city is very important for the operators. And with its very dense population roughly 20 times as high as on average in the UK it is a special terrain to deploy a mobile network. It is interesting to see if operators manage to cover London as well as or even better than the rest of the UK.
With success ratios of 95.9 % in the drive and 97.6 % in the walk test Three showed itself to be the best provider of voice services in London. But this time it was a head to head race against Vodafone with only one point difference to second place. Telefonica managed to win the third place this time with success ratios of nearly 96 %, while EE lost ground with 93 to 94 %.
In the data benchmarks the ranking changed again. With sucess ratios between 97.0 % (1 MB upload) and 98.7 (10 sec. download) and with generaly by far the highest speed in each category EE again proved to be best data provider in the test. But Vodafone was on par in stability and more than speedy enough for a good user experience. With a little higher error rate in popular categories like web browsing and Youtube access Telefonica ranked third not far behind Vodafone, while Three was left with the last place.
Conclusion for London
With good results in telephony and data performance Vodafone managed to win the network test in London. Judged on voice performance alone Three managed to be a tiny bit better, while by far best the data results were again obtained by EE.
As in Germany, Austria and Switzerland P3 communications, connect’s partner for executing network benchmarks, sent out two measurement cars to perform the tests. The cars were equipped with 8 Samsung Galaxy S5 and 4 Note 4 smartphones each. Eight of them where Cat4 devices and used for the voice test, the other four belonged in the Cat6 category and were used for data measurements.
As far as possible the smartphone’s firmware matched the network operator’s genuine ones. In cases where no operator specific firmware was available the most recent Samsung firmware was used.
Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions background data traffic was generated on the smartphones during a voice call. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. All network operators offer 4G capable subscriptions for everybody interested in a high performance data network. To take the high share of LTE into account speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch (“fall back”) to 2G or 3G because they camped on the LTE service when a phone call was initiated. This procedure is known as circuit-switched fall back (CSFB).
The testing of data services was done with four separate Samsung smartphones in the cars. As a first measurement task several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used. In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located in the Internet. In addition, the peak data performance was tested in upand downlink direction by assessing the amount of data that could be transferred within a 10 seconds time period. While 10 seconds peak throughput test focuses on network capability the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7MB, 30s) and high definition (HD, 720p, 11.9MB, 30s) were played using the YouTube player on the smartphones.
Route and Samples
The route is shown on page 1 of this report. In the big and small cities indicated the cars had to follow pre-defined routes to measure a choice. Altogether more than 90,000 speech samples were logged per operator, about half of them in 4G preferred to 4G preferred mode, while the other half 4G preferred to 3G preferred. For the data benchmark about 50,000 samples were obtained per operator.
About 60 % of the samples were obtained in big cities while the remaining 40 % were collected in small cities and on connecting routes.
A new approach to measure mobile networks
Previously it was the usual approach to conduct mobile network drive tests with the measurement devices mounted in the car and connected to antennas in a roof box mounted on the vehicle. In accordance with ETSI TR 102 581 an attenuation of the roof antenna signals of -12 decibel was used to compensate for the antenna gain (+ 5 dB) and to simulate a typical attenuation (- 7dB) of a mobile used indoor. With modern smartphones having their orthogonal MiMo antenna arrangements used for HSPA and LTE and with the use of more intelligent antennas changing their directivity and tuning in relation to the signals received, a new approach is better suited to reflect real life conditions.
This new approach is a highly optimized mounting case for the smartphones. It is designed with unique materials selected for their specific absorption rate at high frequencies between 800 MHz 2,6 GHz. By choosing the right selection of materials in the right places an attenuation rate typical for indoor measurements can be achieved without altering the antennas of the smartphones. The validity of this approach, for which a patent is pending, has been confirmed by comparing measurements between unaltered phones. The legacy approach with antennas in a roof box and attenuators and the so called P3 Antenuatr has been further verified by comparing TIS (total isotropic sensitivity) and TRP (total radiated power) according to the CTIA test plan for over the air measurements in the Testlab owned by WEKA Media Publishing, the parent company of connect. These measurements confirmed an attenuation rate very close to the wanted – 7 dB from LTE band 20 (800 MHz) to LTE Band 7 (2,6 GHz). The variation is even smaller than with external antennas connected over wideband attenuaters. Three of these antenna boxes were mounted in each of the two cars in the rear side and in the rear windows. One Antenuatr was responsible for 4G preferred to 3G preferred voice measurements, while the second was used for 4G preferred to 4G preferred voice measurements. The last Antenuatr was responsible for the data measurements. Real live testing confirmed the advantage of using the differently polarized original MiMo antennas, as this increases the likeliness of achieving maximum data rates due the the higher linear independence of the antenna signals.
In last year’s mobile network test it was mentioned that „compared with other leading operators in Europe and worldwide, all UK networks have significant potential to further improve their service quality and performance“. We see some improvement in this year’s benchmark, especially with EE in the data measurements. Here the best operator of last year’s benchmark managed to make a big step ahead in speed. No wonder EE defends its leading position and contends for „best in test“ seal again. But voice services from EE are handled only satisfactorily with success ratios slightly below 95 %. Other European operators constantly deliver above 98 %. Three also manages to offer a significantly better voice service and consequently reached second place in this year’s mobile network benchmark. But this goes along with a trade-off in data performance. It is Vodafone that delivers the best combination of voice and data services and ended up in second position for each category seperately. Looking at big cities alone, Vodafone’s results would have been even better, as the results for London on Page 7 show. Telefonica changed position with Vodafone and ranks fourth.