THE MOBILE NETWORK TEST IN THE NETHERLANDS 2016
mobile network test Netherlands 2016
Download mobile network test Netherlands here
For the second time now P3 communications and connect have carried out a mobile network benchmark in the Netherlands. Which of the four candidates takes the lead this time?
For mobile network operators, the Netherlands have quite favourable characteristics: The country is relatively flat and densely populated. Both properties prove helpful when it comes to providing dense and powerful mobile voice and data coverage. So the Dutch population has good reason to be happy about their mobile networks.
But how capable are the four mobile networks from KPN, T-Mobile, Vodafone and Tele2 in fact? And which one of these competitors actually delivers the best performance? These were the questions that P3 communications and connect wanted to answer in depth.
Second Dutch mobile benchmark
P3 communications, based in Aachen, Germany, is widely regarded as being the first port of call for professional and trustworthy mobile network testing in the world. The company regularly conducts tests in Germany, Austria, Switzerland, the USA, the United Kingdom, Australia, Spain and many other countries. This is why connect magazine, Europes biggest special interest magazine covering telecommunications, has chosen P3 communications as its partner for mobile network benchmarking all over Europe.
The mobile network test at hand is the second one that the two partners have conducted in the Netherlands. In the first Dutch mobile benchmark, published in mid-2015, the tested operators delivered convincing results. That’s why everybody involved was curious to see if they had improved since then. Read all detailed results on the following pages.
As welcome as LTE is for data communication – for telephony, it makes things a lot more complicated. But even if voice calls may become less important, customers still expect top telephony quality. Which Dutch operator makes the best out of these conditions? LTE has been developed for fast data communication. Telephony played a smaller role in the standard and was orginally only implemented by means of “circuit switched fallback“ (CSFB): In order to call someone or to receive a call, maybe right in the middle of a data session, the smartphone has to switch back to the older circuit-switched networks, UMTS or GSM. This process bears some technical challenges. The test procedure was designed considering this background: A total of four P3 teams drove through a large number of big and small Dutch cities as well as the roads connecting them. Each car was equipped with eight Samsung Galaxy S5 smartphones that would permanently call an identical counterpart in its “partner car“, thus generating more than 50,000 speech samples per operator. About half of the test calls took place between two phones that were set to “4G preferred“ mode, while the other half took place between one “4G preferred“ handset and one preferring 3G. So when at least one of the handsets was registered in a 4G/LTE network, this would invoke the complicated CSFB procedure.
Voice in big cities
All four Dutch networks delivered excellent test results in the voice discipline. In big cities, T-Mobile, KPN and Tele2 achieved stability results of over 99 percent, with Vodafone following closely at 98.8 percent. T-Mobile scores the win with very good call setup times and excellent speech quality, with Vodafone and KPN following narrowly behind. Tele2 is the only Dutch operator that lacks the beneficial HD voice feature in their own network and therefore achieves a slightly inferior speech quality. Even when Tele2 subcribers actually use the T-Mobile network by means of “national roaming“, they are still not admitted to using HD voice, as T-Mobile offers this feature voice only to their own customers and not to roaming Tele2 subscribers.
Voice in small cities and on connecting roads
The voice tests in smaller cities and on connecting roads show basically the same outcome that could be observed in the big cities – but on a slightly lower level. The most striking outcome is however, that even in rural areas with their typically lower network coverage, the differences to the big city scores remain small.
VoLTE approaching in 2016
As already mentioned, the fallback to the older 3G or 2G standards is complicated. But with “Voice over LTE“ (VoLTE) there is a more straightforward alternative. Right now, the Dutch operators are preparing to introduce this technology into their networks. It is expected to launch later in 2016, so the next test of the Dutch networks is likely to include it.
The Koninklijke PTT Nederland N.V. emerged from the privatisation of the formerly state-owned PTT in 1998. At the end of 2015, KPN is believed to have had about 8 million mobile customers. In its financial results for 2015, the company quotes a mobile market share in the Netherlands of 43 percent. Regardless of the precise numbers, KPN is the largest Dutch mobile operator. For 2015, the company reported a revenue of 1.5 billion Euros in the consumer mobile segment, 1.9 billion Euros in the consumer residential sector and another 2.7 billion Euros in the business segment. The company focuses on marketing its KPN brand, however with Simyo, Telfort and Ortel it also has offerings in the “no-frills“ segment. KPN operates 2G networks at 900 and 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 800, 1800 and 2600 MHz. The company claims a 99 percent 4G availability all over the country. Where KPN already offers carrier aggregation, the theoretical maximum speed of its LTE network is 225 Mbit/s (“4G+“). Otherwise it is at 150 Mbit/s.
Vodafone Libertel B.V., the Dutch subsidiary of the Vodafone Group that is operating all over Europe and beyond, acquired the formerly independent operator Libertel in 2003. Today, Vodafone is the second largest mobile operator in the Netherlands with a total of 5.3 million mobile customers or a mobile market share of about 32 percent. Its revenue in the 2014/2015 fiscal year was about 1.9 billion Euros. Vodafone Netherlands also offers landline internet access and IPTV to its Dutch customers. The company operates 2G networks (GSM, GPRS, EDGE) at 900 and 1800 MHz, 3G (UMTS up to HSPA+) at 2100 MHz and 4G/LTE at 800, 1800 and 2600 MHz. Vodafone claims to offer more than 95 percent coverage with 4G (up to 150 Mbit/s). In 110 communities the company offers “4G+“ with carrier aggration of its 800 and 1800 MHz bands. What is more, at the end of 2015 Vodafone Netherlands reported tests with aggregating its 1800 MHz spectrum with 20 MHz of the unlicensed 5 GHz band, reportedly resulting in a maximum data rate of 274 Mbit/s.
In 2000, the German company T-Mobile bought a minority of the Dutch mobile network operator Ben, which two years later was extended to a 100 percent acquisition. In 2003, Ben was renamed T-Mobile Netherlands, with the brand “Ben“ becoming a “no-frills“ offer within their portfolio. In 2007, T-Mobile Netherlands additionally acquired the operator Orange which until then had belonged to France Télécom. T-Mobile NL is a full subsidiary of the German Deutsche Telekom. At the end of its fiscal year 2014, the company reported 3.9 million customers and a revenue of 1.6 billion Euros. This equals a mobile market share of roughly 24 percent. T-Mobile Netherlands operates 2G networks at 900 and 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 900, 1800 and 2600 MHz. After a nationwide modernisation of its network, the company now claims a 4G coverage of more than 99 percent of the Dutch population and also claims to have most 4G antennas in the country. With carrier aggregation, T-Mobile’s LTE network offers a maximum speed of 225 MBit/s.
The Swedish telecommunications operator Tele2 acquired the former Versatel N.V. in 2005. The resulting Tele2 Netherlands Holding N.V. is a 75 percent subsidiary of its Swedish parent company. Originally acting as a MVNO (mobile virtual network operator), Tele2 nowadays operates its own infrastructure and has a portfolio of fixed telephony, data, internet and mobile telephony products. At the end of 2015 Tele2 reported 844,000 mobile, 344,000 fixed broadband and 55,000 fixed telephony subscribers. Tele2 operates its own LTE network at 800 and 2600 MHz with its coverage concentrated on larger Dutch communities. Tele2 plans to reach an almost complete 4G coverage of the Netherlands in 2016. Also, in some parts of the country, it already uses carrier aggregation with a maximum data rate of 225 Mbit/s. As the company had to pay considerably lower license fees for its 4G spectrum than its competitors, it can offer aggressive prices. Tele2 cooperates with T-Mobile NL by location sharing as well as national roaming for voice services.
DATA ACCESS IN BIG CITIES
LTE has been designed to offer mobile data access with very high speeds and a very high reliability. Can the Dutch network operators fulfill this promise? In big cities, each of the four Dutch network operators claims to offer almost complete 4G coverage. In order to measure the quality and reliability of mobile data services, the four P3 teams visited a total of ten Dutch cities each of which has more than 100,000 inhabitants. But in densely populated areas a large number of cells with comparably small diameters is necessary to provide high speed to every user. The real-life test scenarios applied by P3 (see also page 8) therefore concentrated on accessing both live and static web pages as well as downloading and uploading files and watching YouTube videos.
Web page access
In order to test the stability and speed of website access, the benchmark uses a mixture of ten popular live web pages as well as one static web page. In big cities all four operators showed excellent performance, with T-Mobile and KPN even accomplishing rare 100,0 percent success ratios when accessing the static test page. Further, average session times were pleasantly short, with a close lead by T-Mobile, a strong midfield consisting of KPN and Tele2 and a slight backlog for Vodafone.
File uploads and downloads
A similar picture could be seen in the upload and download disciplines. Success ratios of 100,0 percent or close are a clear indicator of the very high performance levels of the Dutch mobile networks. Although a casual observer might take for granted that almost every file transfer just works, this is in fact not so common – as can be seen when comparing the Dutch results to those of the benchmarks that P3 conducted in other countries. A little more deviation can however be seen when taking a closer look at the actual download and upload speeds. In order to point out these differences, transfer rates have been measured in two different ways: With files of fixed size the overall speed of the network was tested, including the time it takes to initiate a speedy data stream. This is important in cases where many small files will be transferred. Additionally, the “10 seconds measurements“ showed the sustained performance once a stable transmission was achieved. It gives insight on how the network behaves when large files like highres photos or videos are uploaded or downloaded. The value given for “90% faster“ is what data rate can be expected most of the time. Interestingly, when downloading small files, KPN and Tele2 achieved slightly higher maximum data rates (“10% faster“), while T-Mobile excelled with slightly higher sustainable speeds (“90% faster“). In the upload category, Tele2 can keep up with occasional top speeds, but the “90% faster“ value shows that its network does not constantly perform at such a high level. Also, in the Vodafone network the upload tests revealed somewhat slower data rates in comparison to the top duo T-Mobile and KPN.
YouTube in standard and high definition
As YouTube and similar video services are the major source of traffic in mobile networks today, the benchmark examines them separately. Again, the stability or success ratios and start times as well as successful playouts without interruptions were on a very high level. As might be expected they dropped a little when accessing the more demanding HD videos, but none of the four Dutch operators showed any weaknesses worth mentioning in this discipline.
DATA ACCESS IN SMALL CITIES
While providers usually perform superbly in metropolitan areas, they often lag behind in smaller cities. How noticeable is this effect in the Dutch networks? In addition to the ten large cities considered on the previous page, the four P3 measurement cars also visited a considerable number of smaller Dutch communities on their test routes. The map on the first page of this report shows which smaller cities were taken into account for this benchmark. In total, the test cars drove through cities that accomodate about 4.3 million people. Compared to the roughly 17 million inhabitants of the Netherlands, this gives the P3 connect Mobile Benchmark a high degree of statistical validity.
Web page access
In smaller towns and rural areas, mobile network coverage is typically less dense than in the metropolitan areas. So it could be expected that the measurement results and scores drop somewhat in comparison to the bigger cities. It is however striking, how close the values and scores turned out to be – especially when looking at the top three operators, T-Mobile, KPN and Vodafone. Their success ratios and average session times only rank fractions behind their results in the metropolises. And even Tele2, which by their own account is still working on extending its 4G network coverage to a nationwide footprint, scored convincingly well in this category.
File uploads and downloads
When examining file transfers outside of the big cities, the success ratios and data rates again remained at a very high level. While the maximum and average data rates of uploads and downloads in the KPN, Vodafone and Tele2 networks dropped to some extent in smaller cities, T-Mobile manages to keep them at almost the same level as in the big cities. Also, when comparing the throughput rates, T-Mobile plays in a league of its own, followed by KPN and in third position by Vodafone. As already observed in the major cities, Tele2 manages to almost keep up with the three stronger candidates in the top 10 percent categories and even outruns Vodafone when downloading large files. But when the majority of cases (“90% faster“) is taken into account, the smallest Dutch operator falls behind. For example, when it comes to uploading smaller files, Tele2 only achieved an average throughput of 925 kbit/s, while in the T-Mobile network, files would complete their journey with an average of 7752 kbit/s. Similarly, download rates (which are even more important for the majority of applications) were in most cases considerably higher in the T-Mobile network than via Tele2. All Dutch operators (as far as they were considered in last year’s P3 connect Mobile Benchmark) including Tele 2 were however able to improve themselves in almost all disciplines.
YouTube in standard and high definition
The benchmark also indicates that watching videos on smartphones while staying in smaller cities is a very nice experience. As the upload and download results already suggested, video playback works extremely well in the Dutch networks. All four candidates accomplish top values and success ratios. In both of the video categories (SD as well as HD), T-Mobile and Vodafone take a narrow lead, with KPN and Tele2 following closely. With sucess ratios and “video playouts without interruptions“ all scoring at or over 99 percent – the only exception being Tele2 with a close 98.6% in the YouTube HD discipline – the results are top notch. This is especially remarkable as the more demanding high-definition (HD) videos obviously did not make much of a difference in comparison to the smaller and less bandwidth-consuming standard definition (SD) videos. All in all, the measurements and scores achieved in smaller cities are impressive evidence for the high performance of the Dutch mobile networks compared to those of many other European countries.
DATA ON CONNECTING ROADS
While driving we increasingly demand connectivity for navigation, messaging or just pure entertainment. How do the Dutch networks fulfill these expectations? A mobile network cell covering a road usually only reaches a relatively small number of customers. This makes the deployment of mobile networks along roads and motorways a costly venture. In order to shed more light on this aspect, the four measurement cars also covered a total of 5,200 kilometres on the connecting roads between Dutch cities – about 1,300 kilometres per car. Whilst driving there, the cars permanently examined the coverage, reliability and data performance of all four candidates’ networks.
Web page access
When comparing the results obtained on connecting roads with those of smaller cities, it does not come as a surprise that performance indicators drop to some extent. But once more it is amazing how close to each other these values actually are. When examining the access to live and static web pages, the resulting success rates and average session times were almost identical to their counterparts – the results obtained in small cities. So in the benchmarking it made almost no difference whether the test cars were located within a smaller city or driving on the road. Only Tele2 shows a minor gap between these two environments. But their 97.2 percent success ratio is still a good result – and this operator has not even finished its nationwide 4G deployment.
File uploads and downloads
This general picture is also confirmed regarding file transfers. Each of the four Dutch operators scores almost the same on the connecting roads as in the small cities. This of course means that T-Mobile once more takes the lead concerning data rates, KPN and Vodafone again rank in the midfield, and Tele2 trails behind. But in most of the cases within our benchmark, each of the four Dutch networks provided reliable connectivity on the road – a benefit that many other European mobile network providers can not claim to this extent.
YouTube in SD and (with a little surprise) HD
The observations of the web and file categories are also mostly true for the video tests. Generally the results were very good and did not rank considerably worse on the connecting roads than in small cities. The heavier load of the HD videos caused somewhat smaller success ratios in the Tele2 network and – somewhat surprisingly – to a lesser extent also in the Vodafone network. However, as long as video consumption on the road is restrained to standard definition (SD) content, all four Dutch networks once more showed a convincingly high stability and quality.
The network tests in the Netherlands were conducted from 28.01. until 09.02.2016. As in Germany, Austria, Switzerland, the UK and Spain, P3, connect’s partner for network benchmarks, sent out measurement cars to perform the tests. For the benchmark of the Dutch mobile networks, four cars covered a total distance of 7,600 kilometres through big and small Dutch cities and the connecting routes between them. The route is shown on the first page of this report. In total, the test cars drove through cities that accomodate about 4.3 million residents. Each car was equipped with eight Samsung Galaxy S5 and four Samsung Note 4 smartphones. The eight Galaxy S5 Cat 4 devices were used for the voice tests, the four Note 4 belonged in the Cat 6 category and were used for data measurements. Whenever possible, the smartphone’s firmware matched the network operators’ genuine ones. Where no operator-specific firmware was available the most recent Samsung firmware was used.
Smartphone telephony and data
Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions, background data traffic was generated on the smartphones during a voice call. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. All network operators offer 4G capable subscriptions for anyone interested in a high performance data network. To take the high share of LTE into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones needed to switch (“fall back”) to 2G or 3G when they were connected to the LTE service before the call could be completed using circuit switched fall back (CSFB).
Data services were tested with four separate Samsung smartphones in the cars. As a first measurement task, several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used. In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the internet.
In addition, the peak data performance was tested in uplink and downlink direction by assessing the amount of data that could be transferred within a 10 second time period. While the 10 seconds peak throughput test focuses on network capability, the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7 MB, 30 seconds) and high definition (HD, 720p, 11.9 MB, 30 seconds) were played using the YouTube player on the smartphones.
Route and Samples
In big and small cities the cars followed pre-defined routes. Altogether more than 50,000 speech samples were logged per operator. In order to collect them, a call of a typical duration of 70 seconds was established during which eight speech samples were exchanged alternately (four each from mobile A to mobile B and vice versa). About half of them were collected in 4G preferred to 4G preferred mode, while the other half in 4G preferred to 3G preferred. For the data benchmark about 38,000 sessions per operator were established. About 55 percent of the samples were obtained in big cities while the remaining 45 percent were collected in small cities and on connecting routes.
Benchmarking at the edge of the technically possible
Previously, the usual approach was to conduct mobile network drive tests with the measurement devices mounted in the car and connected to antennas in a roof box mounted on the vehicle. In accordance with ETSI TR 102 581 an attenuation of the roof antenna signals of -12 decibel was used to compensate for the antenna gain (+5 dB) and to simulate a typical attenuation (-7dB) of a mobile phone used indoor. With modern smartphones having their orthogonal MiMo antenna arrangements used for HSPA and LTE and with the use of more intelligent antennas changing their directivity and tuning in relation to the signals received, a new approach is better suited to reflect real life conditions. This new approach is a highly optimized mounting case for the smartphones – the so-called P3 Antenuatr. It is designed with unique materials selected for their specific absorption rate at high frequencies between 800 MHz and 2.6 GHz.
By choosing the right materials in the right places, an attenuation rate typical for indoor measurements can be achieved without altering the antennas of the smartphones. The validity of this approach, for which a patent is pending, has been confirmed by comparing measurements between unaltered phones. The legacy approach with antennas in a roof box and attenuators has been further verified by comparing TIS (total isotropic sensitivity) and TRP (total radiated power) according to the CTIA test plan for over the air measurements in the Testlab owned by WEKA Media Publishing, the parent company of connect. These measurements confirmed an attenuation rate very close to the wanted -7 dB from LTE band 20 (800 MHz) to LTE Band 7 (2.6 GHz). The variation is even smaller than with external antennas connected over wideband attenuators. Three of these antenna boxes were mounted in each of the cars in the rear side and in the rear windows. One Antenuatr was responsible for 4G preferred to 3G preferred voice measurements, while the second was used for 4G preferred to 4G preferred voice measurements. The last Antenuatr was responsible for the data measurements. Real live testing confirmed the advantage of using the differently polarized MiMo antennas, as this increases the likeliness of achieving maximum data rates due to the higher linear independence of the antenna signals.
Hannes Rügheimer, connect author
In the 2016 installment of the P3 connect Mobile Benchmark for the Netherlands, the overall winner is T-Mobile. This operator managed to achieve a clear lead in the voice category and a slight advantage over its pursuer KPN in the data scoring.
In the voice discipline both networks showed close to perfect stability, with more than 99 percent success ratios in all scenarios. T-Mobile wins thanks to faster call setup and a slightly better speech quality. Vodafone and Tele 2 score almost as well in this category, despite different weaknesses: As Tele2 lacks HD voice, it achieves a lower quality result, while Vodafone has slight disadvantages regarding call setup duration and stability.
Regarding data, T-Mobile leads closely, with KPN following close by, and Vodafone and Tele2 (which is still working on its 4G rollout) still doing very well. It must however be emphasized how close the total ratings of all four candidates are. This is a clear indicator of the superior quality of the Dutch networks in comparison to those in many other countries. We applied exactly the same scoring standards here as in our other network benchmarks. And while for example in the 2015 Germany benchmark the resulting grades varied from “Very good“ to “Sufficient“, all four candidates in the Netherlands were able to obtain the treasured top grade.
We asked the four Dutch mobile operators to comment on the results of the P3 connect mobile benchmark for the Netherlands. Here are their responses.
“We at T-Mobile Netherlands are very proud and happy with the results of the P3 Mobile Benchmark in the Netherlands. This remarkable outcome is the proof of our network strategy which has started back in 2012 with the spectrum auction and has continued with the successful deployment of a completely new SRAN network. T-Mobile Netherlands has been ranked the best tested network in all categories of service (voice, data) in a very competitive landscape, where all operators have developed high quality networks. We will continue focusing on best customer experience based on our network leadership.”
Lampros Iskos, Director Technology T-Mobile Netherlands
“The KPN mobile network covers more than 99% of the Dutch population and Dutch territory and therefore has the best degree of coverage in the Netherlands. This P3 test – carried out mainly in the big cities of the Netherlands – also shows that the KPN network is one of the best in the world. We are very proud on this position. But the KPN network is for the benefit of the whole of the Netherlands, not just for the cities. And that is not immediately visible in this test. Furthermore, every test is a snapshot; satisfied customers are the most important indicator for us. Overall our customers are very satisfied, as can be seen from recent independent customer satisfaction surveys and from our Net Promoter Score (NPS), which is higher than that of other operators. Above all, we aspire to provide our millions of customers with an exceptional user experience so that every day, thanks to KPN, they are free to do what they enjoy, anytime, anywhere.”
Jacob Groote, Executive Vice President Mobile Networks KPN
“Great to see the Dutch mobile networks belonging to the top echelon worldwide with only small differences between the networks. Vodafone continues to drive for excellence in all of its services. Room for improvement can be found indoors, especially with the enhanced focus on sustainable buildings which makes penetration for mobile signals a lot harder. In response to this trend, we introduce not only 4G calling this year (Voice over LTE or VoLTE), but will also be the first telecom provider in the Netherlands to launch WiFi calling (Voice over WiFi or VoWiFi). By combining a 4G signal (with better penetration capabilities for homes) and a WiFi signal (which originates indoors), our customers will experience much better quality of their indoor mobile calls, even if they live in a house where mobile signals are disrupted by, for example, dense construction materials.”
Eben Albertyn, Chief Technology Officer Vodafone Netherlands
“We are extremely proud that we have been able to close the network quality gap on our competitors so quickly, offering excellent speed and coverage, so our customers can enjoy their 4G mobile data carefree. However, we are not done yet, and we continue to build out our network at full pace, expecting to reach nationwide population coverage in just a couple of months. These fantastic results are a real credit to the team, and a true testament to their extraordinary efforts. We now can’t wait for the next test …”
Francois Mairey, Chief Technology and Information Officer Tele2 Netherlands