The Netherlands 2015

THE MOBILE NETWORK TEST IN THE NETHERLANDSDutch-flag_2

Download test here

 

The mobile network test conducted by P3 communications and connect since 1993 has become a standard in Germany, Austria and Switzerland. But how good are the providers of the Netherlands in comparison? A benchmark.

When it comes to mobile network testing P3 communications is probably the first address in the world. Being active in Germany, the United Kingdom, the USA, Australia, Austria, Switzerland and many other countries, the mobile communication experts from Aachen have accumulated a lot of knowledge. No wonder the biggest European magazine on telecommunication connect has chosen P3 communications as the partner for the mobile network benchmark in Germany, Austria and Switzerland since 1993.

Focus on the Netherlands

But how do the Netherlands compare to other countries? The territory is comparably flat and densely populated, both factors relaxing the task for the operator. There are four operators present in the Netherlands. With 7,6 million customers the biggest is the formerly state owned but now independent KPN. With 5,2 Million customers not much smaller and part of a group active in many countries is Vodafone NL. With 850000 customers Tele2 is a small but constantly growing operator which relies on its own LTE-Network and national roaming with T-Mobile NL for voice services.

We regret not to be able to publish any results of the T-Mobile network in this first P3 connect Mobile Benchmark NL. During the measurement period in MayJune 2015 T-Mobile was in the midst of a nationwide full modernization of their network. This externally contracted work could not be delayed and would have meant that the measurement results would not have been a fair representation of the usual network Performance.

Customers expect nothing but a great experience in mobile telephony. But LTE introduces high hurdles on perfect voice services. Which provider brings stability and speech quality to the highest level?

A blade computer array in the the car is used to steer nine Samsung S5 used for the measurements.

 

TELEPHONY

LTE is a great technology for handling large amounts of data with high speed. This is because it was designed to offload the fast growing amount of data present since the introduction of smartphones.But it was not designed to handle voice calls. As a workaround a technique called circuit switched fall back (CSFB) changes the connection to UMTS or GSM whenever a call is done. This additional handover takes time and reduces reliability, so providers must step up efforts to keep the quality up to the level of the pre-LTE-era.

Voice over LTE (VoLTE) is the new alternative to CSFB. All operators in the Netherlands are experimenting with the technique which allows the smartphone to place and receive voice calls without leaving the LTE network. But the potentially more stable VoLTE is not available to the customer and consequently it was not part of the mobile network test.

 

OPERATOR VODAFONE KPN TELE2
TELEPHONY (BIG CITIES OUTDOOR)
Call Success Ratio (%) 99,0 98,7 98,1
Call Setup Time (s) 6,3 6,4 6,5
Speech Quality (MOS LQ0) 3,5 3,5 2,8
TELEPHONY (SMALL CITIES AND CONNECTION ROADS)
Call Success Ratio (%) 98,3 99,2 94,0
Call Setup Time (s) 6,3 6,3 6,9
Speech Quality (MOS LQO) 3,3 3,6 2,8

 

Voice in Big Cities

Despite the good LTE coverage found in the big cities for all operators and despite the fact, that VoLTE was not used for this benchmark the call success ratios of Vodafone and KPN were very good. Only 1, respectively 1,3 percent of the calls faced problems like failures to establish the connection, longer parts of very low quality of sound or no sound at all or early abortions of the call. Tele2 reached a call success ratio of 98,1 percent which is slightly lower but still good. It is worth noting that Tele2 is a LTE only Operator with no UMTS or GSM network. Because of this voice calls are not only handled by CSFB but must also include national roaming to T-Mobile to use their legacy networks. Overall all three providers compare favorably to the German operators measured at the end of last year with success rates between 95,7 and 98,5 percent. The call setup times of approximately 6,4 seconds are typical for 4G-Networks and slightly higher than those measured in times before LTE. The speech quality is measured on the MOS scale, which goes from 1 (bad) to 5 (excellent). Vodafone and KPN reached an average of 3,5, which translates to HD voice (High Definition). Tele2 is reaching 2,8 on average, a score associated with normal speech Quality.

Voice in Small Cities and on Connection Roads

Of course it is easier to provide a stable network for speech in big cities with their high demand of network capabilities and an accordingly dense network of mobile cells. Nonetheless Vodafone only loses 0,7 % in call success ratio, while KPN can even gain half a percent. No wonder they are both again rated very good. Tele2 reaches a call success ratio of 94 %, still above the average of the connect test conducted in Germany, Austria and Switzerland under similar conditions. Call setup time and speech quality nearly stay the same for all providers.

The promise of LTE is high speed data access everywhere. Can the providers of the Netherlands keep this promise?

„Vodafone and KPN are head-to-head on the highest level reached in international mobile network test so far“

 

DATA ACCESS IN BIG CITIES

With 400 inhabitants per square kilometer the Netherlands are populated higher-than-average. With only 50 percent of the land exceeding one meter above sea level and the highest point of the mainland only reaching about 322 meters the country is very flat.

This is easy territory for providers to reach a large number of people with a small number of mobile radio cells. But in big cities a large number of cells is needed to provide high speed to everyone. The mobile network benchmark will show which provider fares best in this regard.

Web-page download

One of the most often used services is web surfing which was tested with a mixture of 10 popular live and static web-pages. Vodafone and KPN fight head-to-head with sucess ratios of around 98 to 100 percent and only two tenths of a second difference in average download time at maximum with a very good average of 3,6 seconds for popular live sites. Tele2 delivers the same speed and a little bit lower success ratio, which shows that the network of the newest provider of the Netherlands is still in the buildup but strives for top performance.

 

OPERATOR VODAFONE KPN TELE2
WEB-PAGE DOWNLOAD(LIVE/STATIC)
Success Ratio(%/%) 97,8/99,9 97,9/100,0 96,4/98,1
Avg. Session Time (s/s) 3,7/1,3 3,5/1,3 3,6/1,4
FILE DOWNLOAD (3 MB)
Sucess Ratio/Avg. Session Time (%/s) 99,9/1,4 100/1,4 99,4/2,2
90 % faster than (kbit/s) 10261 10806 5311
10 % faster than (kbit/s) 50847 42328 51948
FILE-UPLOAD (1 MB)
Sucess Ratio/Avg. Session Time (%/s) 99,9/1,1 99,7/1,2 98,8/2,3
90 % faster than (kbit/s) 4582 4890 1740
10 % faster than (kbit/s) 15717 11730 13136
FILE DOWNLOAD (10 SECONDS)
Success Ratio (%) 99,7 99,9 99,5
Avg. Throughput (kbit/s) 37327 39664 32086
90 % faster than (kbit/s) 10304 11524 5819
10 % faster than (kbit/s) 76640 81173 74003
FILE UPLOAD (10 SECONDS)
Success Ratio (%) 99,8 99,7 99,2
Avg. Throughput (kbit/s) 22577 19743 13070
90 % faster than (kbit/s) 5869 5976 1736
10 % faster than (kbit/s) 42680 38645 35292
YOUTUBE SD
Success Ratio/Start Time (%/s) 99,5/1,1 96,1/1,0 99,2/1,1
Video playouts without interruptions (%) 99,9 100 99,8
YOUTUBE HD
Success Ratio/Start Time (%/s) 99,5/1,3 96,5/1,4 97,5/1,5
Video playouts without interruptions (%) 99,6 99,6 98,9

 

File up- and download

When it comes to file up- and downloads the operators can show their best in speed and reliability. And with between 98,8 and 100 % success ratio the stability of the data sessions was indeed on a very high level.

This is also true for the speed of the networks, although things can be seen a little bit more differentiated here. Therefore the mobile network benchmark measures the transfer rates in two different ways. With files of fixed size the overall speed of the network is tested including the time it takes to initiate a speedy data stream. This is important in cases where many small files will be transferred. On the other hand the „10-seconds-measurements“ show the peak performance once a stable transmission is achieved. It gives insight on how the network behaves when large data files like high resolution photos or videos are up- or downloaded. The value given for „90 % faster“ is what a customer can reasonably expect most of the time.

Vodafone and KPN delivered themselves a head-to-head-race on a high level in this category, while Tele2 was clear behind, but still delivered good performance. The newcomer-status of Tele2 can be seen in the fastest 10 % of the measurements, where this provider is on a par with Vodafone and KPN. This means in some areas Tele2 must have bridged the gap to Vodafone and KPN, but there still a lot of work to do, to roll out a high performance network all over the Netherlands.

YouTube in Standard and high definition

Video transmission was long regarded as impossible in a mobile network. But once the faster networks started to become reality, services like YouTube became the major source of traffic. Today YouTube amongst other video services is a commodity in the mobile world and every operator should deliver a highly reliable stream at least in standard definition (SD). This is true for Vodafone and Tele2 with success ratios above 99 percent while KPN loses nearly 4 % to perfection. With high definition (HD) Video demands are even more stringent on the network, but Vodafone can keep its performance notwithstanding the higher data rates. Curiously KPN is a little bit better on HD than on SD but still loses ground to Vodafone. And Tele2 again falls in between, this time nearer to KPN.

Conclusion

Vodafone is the winner of the data benchmark in big cities, as the operator reveals good results across all key performance indicators (KPI). KPN is a close second with only minor difficulties in YouTube reliability while Tele2 is still in the roll out phase but showing ambitions.

 

DATA ACCESS IN SMALL CITIES

While providers take great care in covering the metropolitan areas, small cities often lag behind. Is this true for the Netherlands, too?

 

OPERATOR Vodafone KPN TELE2
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Sucess Ratio (%/%) 95,2/99,5 97,7/100 92,2/94,2
Avg. Session Time (s/s) 3,9/1,5 3,5/1,4 4,0/2,3
FILE DOWNLOAD (3 MB)
Sucess Ratio/Avg. Session Time (%/s) 99,4/2,6 100/1,9 98,7/4,4
90 % faster than (kbit/s) 4934 7757 2858
10 % faster than (kbit/s) 37559 30341 39280
FILE-UPLOAD (1 MB)
Sucess Ratio/Avg. Session Time (%/s) 98,8/1,6 99,8/1,5 97,4/4,2
90 % faster than (kbit/s) 2982 3740 1195
10 % faster than (kbit/s) 13559 10959 9780
FILE DOWNLOAD (10 SECONDS)
Success Ratio (%) 99,8 100 99,6
Avg. Throughput (kbit/s) 22073 21777 14945
90 % faster than (kbit/s) 5610 7963 2689
10 % faster than (kbit/s) 49473 39410 42857
FILE UPLOAD (10 SECONDS)
Success Ratio (%) 100 100 98,2
Avg. Throughput (kbit/s) 15539 12833 5652
90 % faster than (kbit/s) 2934 4486 1096
10 % faster than (kbit/s) 29144 19408 16534
YOUTUBE SD
Success Ratio/Start Time (%/s) 99,8/1,2 97,0/1,1 98,4/1,8
Video playouts without interruptions
(%)
99,8 100 100
YOUTUBE HD
Success Ratio/Start Time (%/s) 98,8/1,5 95,7/1,5 91,6/2,4
Video playouts without interruptions (%) 99,2 100  95,7

 

While KPN managed to be a close second to Vodafone in the big cities, the biggest provider of the Netherlands shines in small cities. In the web-page download benchmark KPN delivers nearly the same reliability and speed as in big cities. A success ratio of 97,7 % for live web sites is remarkable. With 95.2 % Vodafone loses 2,5 % in this category while Tele2 is another 3 % behind. Concerning the speed of the surfing sessions KPN is again first, while Vodafone and then Tele2 are closely following.

File Transfers and YouTube

The speed differences get bigger when it comes to file transfers. Customers interested in the best coverage should closely look at the speeds given, where 90 % of the samples are faster. KPN is again the clear leader in this categorie.

In comparison the 10 % of the fastest values are comparatively close together. This is an indicator, that areas of good deployment make use of the potential given by modern LTE networks in nearly the same way.

While until now KPN dominated the benchmark in small cities, Vodafone again comes first with YouTube in SD and HD. The success ratio and the lack of interruptions is top notch. KPN and Tele2 in comparison are still good, with the better results with videos in standard Definition.

 

DATA ON CONNECTING ROADS

Perfect coverage of roads is a difficult task for providers. Nonetheless customers have high expectations. Will they be disappointed?

 

OPERATOR Vodafone KPN TELE2
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Sucess Ratio (%/%) 94,5/99,4 98,0/100 86,0/90,4
Avg. Session Time (s/s) 3,9/1,4 3,6/1,4 4,0/2,3
FILE DOWNLOAD (3 MB)
Sucess Ratio/Avg. Session Time (%/s) 99,3/2,6 100/1,8 94,6/5,3
90 % faster than (kbit/s) 5248 8400 2070
10 % faster than (kbit/s) 35088 32609 36866
FILE-UPLOAD (1 MB)
Sucess Ratio/Avg. Session Time (%/s) 99,3/1,7 99,6/1,3 93,6/4,3
90 % faster than (kbit/s) 2571 4376 1002
10 % faster than (kbit/s) 13722 11065 9975
FILE DOWNLOAD (10 SECONDS)
Success Ratio (%) 100 100 96,2
Avg. Throughput (kbit/s) 18980 22651 15675
90 % faster than (kbit/s) 3791 7218 2223
10 % faster than (kbit/s) 35599 43451 43592
FILE UPLOAD (10 SECONDS)
Success Ratio (%) 98,5 99,6 91,4
Avg. Throughput (kbit/s) 14438 14665 6822
90 % faster than (kbit/s) 1761 3313 1096
10 % faster than (kbit/s) 22205 20277 18788
YOUTUBE SD
Success Ratio/Start Time (%/s) 97,8/1,3 96,8/1,1 95,0/1,9
Video playouts without interruptions (%) 100 100 97,8
YOUTUBE HD
Success Ratio/Start Time (%/s) 95,6/1,8 97,1/1,5 83,6/2,9
Video playouts without interruptions (%) 96,5 99,3  92,8

 

A mobile cell covering the road normally reaches only a comparatively small number of customers. This makes the deployment of a mobile network along motorways a costly affair. Nonetheless KPN managed to deliver results in connecting roads performance which were remarkably close to the benchmark results shown in cities.

The key performance indicators in the web-page download sessions showed a success ratio of 98 % which is even slightly higher than in small cities. Vodafone with 94,5 % and even more Tele 2 with 86 % had noticeably more difficulties covering the roads. The speed differences between big cities, small cities and connecting roads seem negligible by comparison.

But speed differences become a major differentiator between the providers in the file transfer benchmarks. At the 3 MB download for instance KPN is faster than 8400 kB/s in 90 percent of the samples, while Tele2 only exceeds 2070 kBit/s and Vodafone falls in between the two. The same ranking applies for the 1 MB download and the 10 second transfers showing again an outstanding good coverage from KPN, good performance from Vodafone and a network in the roll out phase from Tele 2.

On YouTube KPN manages to beat Vodafone on High Definition while Vodafone is best on SD.

 

 

TEST METHODOLOGY

 

As in Germany, Austria and Switzerland P3 communications, connect’s partner for executing network benchmarks, sent out two measurement cars, to perform the tests. The cars were equipped with 9 Samsung Galaxy S5 smartphones each. Six of them where Cat4 devices (SMG900F) and used for the voice test, the other three belonged in the Cat6 category (SM-901F) and were used for data measurements.

As far as possible, the smartphone’s firmware matched the network operator’s genuine ones. In cases when no operator specific firmware was available the most recent Samsung firmware was used.

Smartphone-Telephony

Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions background data traffic was generated on the smartphones during a voice call.

The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm.

All network operators offer 4G capable subscriptions for everybody interested in a high performance data network. To take the high share of LTE into account speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch (“fall back”) to 2G or 3G because they camped on the LTE service when a phone call was initiated. This procedure is known as circuit-switched fall back (CSFB).

Smartphone-Data

The testing of data services was done with three separate Samsung smartphones in the cars. As a first measurement task several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used.

In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located in the Internet. In addition, the peak data performance was tested in up and downlink direction by assessing the amount of data that could be transferred within a 10 seconds time period. While 10 seconds peak throughput test focuses on network capability the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7MB, 30s) and high definition (HD, 720p, 11.9MB, 30s) were played using the YouTube player on the smartphones.

Route and Samples

The route is shown on page 1 of this report. In the big and small cities indicated the cars had to follow pre-defined routes to measure a choice.
Measurement of one of an early Antenuatr prototype in the connect Testlab. The OTA-measurements (Over The Air) were used to confirm the predicted attenuation of the specialized high performance material in the P3 design.

 

CONCLUSION Bernd Theiss, connect editor

The first mobile network test of connect and P3 communications in the Netherlands has shown three providers which fare very well in an international comparison. This is even true for the newcomer Tele2 with its LTE-OnlyNetwork and national roaming for voice services and 3G access in areas without LTE. The Tele2 network is still in the roll out phase, which is indicated by peak performance on par with the other two operators and slightly below average performance. It remains interesting where Tele2 will be once the roll out is completed and Voice over LTE becomes a standard. KPN and Vodafone deliver exceptional performance, especially when compared to mobile networks in other countries. KPN is best in small cities and on connecting roads, indicating a very broad coverage throughout the Netherlands. For people travelling and taking the internet everywhere paradigm seriously KPN certainly is the top choice. But Vodafone strikes back with a slightly better voice service in the big cities where most of the people live and with more reliability in the very popular YouTube video streaming. Vodafone manages to stay in front in this headto-head race. At the end the photo finish reveals a 2 points advance for Vodafone, which is best in test.
Pieter Waasdorp (left, editor in chief Connexie) and Cees de Zeeuw (right, P3 communications NL) in front of the measuring car.
Altogether more than 6000 speech samples were logged per operator, about half of them in 4G preferred to 4G preferred mode, while the other half 4G preferred to 3G preferred. For the data benchmark about 29760 samples were obtained per operator.

About 61 % of the samples were obtained in big cities while the remaining 39 % were collected in small cities and on connecting routes.

A new approach to measure mobile networks

In former days it was the usual approach to conduct mobile network drive tests with the measurement devices mounted in the car and connected to antennas in a roof box mounted on the vehicle. In accordance with ETSI TR 102 581 an attenuation of the roof antenna signals of -12 decibel was used to compensate for the antenna gain (+ 5 dB) and to simulate a typical attenuation (- 7dB) of a mobile used indoor. With modern smartphones with their orthogonal MiMo antenna arrangements used for HSPA and LTE and with the use of more intelligent antennas changing their directivity and tuning in relation to the signals received, a new approach is better suited to reflect real life conditions.
This new approach is a highly optimized mounting case for the smartphones. It is designed with unique materials selected for their specific absorption rate at high frequencies between all last 800 MHz und 2,6 GHz. By choosing the right selection of materials in the right places an attenuation rate typical for indoor measurements can be achieved without altering the antennas of the smartphones. The validity of this approach, for which a patent is pending, has been confirmed by comparing measurements between unaltered phones. The legacy approach with antennas in a roof box and attenuators and the P3 Antenuatr has been further verified by comparing TIS (total isotropic sensitivity) and TRP (total radiated power) acording to the CTIA test plan for over the air measurements in the Testlab owned by WEKA Media Publishing, the parent company of connect. These measurements confirmed an attenuation rate very close to the wanted – 7 dB from LTE band 20 (800 MHz) to LTE Band 7 (2,6 GHz). The variation is even smaller than with external antennas connected over wideband attenuaters.
Three of these antenna boxes have been mounted in each of the two cars in the rear side and in the rear windows. One Antenuatr was responsible for 4G preferred to 3G preferred voice measurements, while the second was used for 4G preferred to 4G preferred voice measurements. The last Antenuatr was responsible for the data measurements.
Real live testing confirmed the advantage of using the differently polarized original MiMo antennas, as this increases the likeliness of achieving maximum data rates due the the higher linear independence of the antenna signals.

Up to three P3 Antenuatr-Boxes can be mounted into the back and side windows of each measuring car and up to four smartphones can be used to test up to four providers simultaneous.

OPERATOR Vodafone KPN TELE2
Telephony max. 400 353 354 309
Big cities 260 88% 86% 79%
Small cities and connecting roads 140 89% 93% 74%
Data max. 600 528 525 457
Big cities 390 90% 87% 83%
Small cities 90 83% 84% 65%
Connecting roads 120 86% 92% 64%
Connect RATING max. 1000 881
VERY GOOD
879
VERY GOOD
766
GOOD

„Vodafone and KPN crossed the line in a photo finish which put our measuring accuracy to the test and with a score clearly ahead of providers in other countries“