THE MOBILE NETWORK TEST IN THE NETHERLANDS
Download test here
The mobile network test conducted by P3 communications and connect since 1993 has become a standard in Germany, Austria and Switzerland. But how good are the providers of the Netherlands in comparison? A benchmark.
When it comes to mobile network testing P3 communications is probably the first address in the world. Being active in Germany, the United Kingdom, the USA, Australia, Austria, Switzerland and many other countries, the mobile communication experts from Aachen have accumulated a lot of knowledge. No wonder the biggest European magazine on telecommunication connect has chosen P3 communications as the partner for the mobile network benchmark in Germany, Austria and Switzerland since 1993.
Focus on the Netherlands
But how do the Netherlands compare to other countries? The territory is comparably flat and densely populated, both factors relaxing the task for the operator. There are four operators present in the Netherlands. With 7,6 million customers the biggest is the formerly state owned but now independent KPN. With 5,2 Million customers not much smaller and part of a group active in many countries is Vodafone NL. With 850000 customers Tele2 is a small but constantly growing operator which relies on its own LTE-Network and national roaming with T-Mobile NL for voice services.
We regret not to be able to publish any results of the T-Mobile network in this first P3 connect Mobile Benchmark NL. During the measurement period in MayJune 2015 T-Mobile was in the midst of a nationwide full modernization of their network. This externally contracted work could not be delayed and would have meant that the measurement results would not have been a fair representation of the usual network Performance.
Customers expect nothing but a great experience in mobile telephony. But LTE introduces high hurdles on perfect voice services. Which provider brings stability and speech quality to the highest level?
A blade computer array in the the car is used to steer nine Samsung S5 used for the measurements.
LTE is a great technology for handling large amounts of data with high speed. This is because it was designed to offload the fast growing amount of data present since the introduction of smartphones.But it was not designed to handle voice calls. As a workaround a technique called circuit switched fall back (CSFB) changes the connection to UMTS or GSM whenever a call is done. This additional handover takes time and reduces reliability, so providers must step up efforts to keep the quality up to the level of the pre-LTE-era.
Voice over LTE (VoLTE) is the new alternative to CSFB. All operators in the Netherlands are experimenting with the technique which allows the smartphone to place and receive voice calls without leaving the LTE network. But the potentially more stable VoLTE is not available to the customer and consequently it was not part of the mobile network test.
|TELEPHONY (BIG CITIES OUTDOOR)|
|Call Success Ratio (%)||99,0||98,7||98,1|
|Call Setup Time (s)||6,3||6,4||6,5|
|Speech Quality (MOS LQ0)||3,5||3,5||2,8|
|TELEPHONY (SMALL CITIES AND CONNECTION ROADS)|
|Call Success Ratio (%)||98,3||99,2||94,0|
|Call Setup Time (s)||6,3||6,3||6,9|
|Speech Quality (MOS LQO)||3,3||3,6||2,8|
Voice in Big Cities
Despite the good LTE coverage found in the big cities for all operators and despite the fact, that VoLTE was not used for this benchmark the call success ratios of Vodafone and KPN were very good. Only 1, respectively 1,3 percent of the calls faced problems like failures to establish the connection, longer parts of very low quality of sound or no sound at all or early abortions of the call. Tele2 reached a call success ratio of 98,1 percent which is slightly lower but still good. It is worth noting that Tele2 is a LTE only Operator with no UMTS or GSM network. Because of this voice calls are not only handled by CSFB but must also include national roaming to T-Mobile to use their legacy networks. Overall all three providers compare favorably to the German operators measured at the end of last year with success rates between 95,7 and 98,5 percent. The call setup times of approximately 6,4 seconds are typical for 4G-Networks and slightly higher than those measured in times before LTE. The speech quality is measured on the MOS scale, which goes from 1 (bad) to 5 (excellent). Vodafone and KPN reached an average of 3,5, which translates to HD voice (High Definition). Tele2 is reaching 2,8 on average, a score associated with normal speech Quality.
Voice in Small Cities and on Connection Roads
Of course it is easier to provide a stable network for speech in big cities with their high demand of network capabilities and an accordingly dense network of mobile cells. Nonetheless Vodafone only loses 0,7 % in call success ratio, while KPN can even gain half a percent. No wonder they are both again rated very good. Tele2 reaches a call success ratio of 94 %, still above the average of the connect test conducted in Germany, Austria and Switzerland under similar conditions. Call setup time and speech quality nearly stay the same for all providers.
The promise of LTE is high speed data access everywhere. Can the providers of the Netherlands keep this promise?
„Vodafone and KPN are head-to-head on the highest level reached in international mobile network test so far“
DATA ACCESS IN BIG CITIES
With 400 inhabitants per square kilometer the Netherlands are populated higher-than-average. With only 50 percent of the land exceeding one meter above sea level and the highest point of the mainland only reaching about 322 meters the country is very flat.
This is easy territory for providers to reach a large number of people with a small number of mobile radio cells. But in big cities a large number of cells is needed to provide high speed to everyone. The mobile network benchmark will show which provider fares best in this regard.
One of the most often used services is web surfing which was tested with a mixture of 10 popular live and static web-pages. Vodafone and KPN fight head-to-head with sucess ratios of around 98 to 100 percent and only two tenths of a second difference in average download time at maximum with a very good average of 3,6 seconds for popular live sites. Tele2 delivers the same speed and a little bit lower success ratio, which shows that the network of the newest provider of the Netherlands is still in the buildup but strives for top performance.
|Avg. Session Time (s/s)||3,7/1,3||3,5/1,3||3,6/1,4|
|FILE DOWNLOAD (3 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||99,9/1,4||100/1,4||99,4/2,2|
|90 % faster than (kbit/s)||10261||10806||5311|
|10 % faster than (kbit/s)||50847||42328||51948|
|FILE-UPLOAD (1 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||99,9/1,1||99,7/1,2||98,8/2,3|
|90 % faster than (kbit/s)||4582||4890||1740|
|10 % faster than (kbit/s)||15717||11730||13136|
|FILE DOWNLOAD (10 SECONDS)|
|Success Ratio (%)||99,7||99,9||99,5|
|Avg. Throughput (kbit/s)||37327||39664||32086|
|90 % faster than (kbit/s)||10304||11524||5819|
|10 % faster than (kbit/s)||76640||81173||74003|
|FILE UPLOAD (10 SECONDS)|
|Success Ratio (%)||99,8||99,7||99,2|
|Avg. Throughput (kbit/s)||22577||19743||13070|
|90 % faster than (kbit/s)||5869||5976||1736|
|10 % faster than (kbit/s)||42680||38645||35292|
|Success Ratio/Start Time (%/s)||99,5/1,1||96,1/1,0||99,2/1,1|
|Video playouts without interruptions (%)||99,9||100||99,8|
|Success Ratio/Start Time (%/s)||99,5/1,3||96,5/1,4||97,5/1,5|
|Video playouts without interruptions (%)||99,6||99,6||98,9|
File up- and download
When it comes to file up- and downloads the operators can show their best in speed and reliability. And with between 98,8 and 100 % success ratio the stability of the data sessions was indeed on a very high level.
This is also true for the speed of the networks, although things can be seen a little bit more differentiated here. Therefore the mobile network benchmark measures the transfer rates in two different ways. With files of fixed size the overall speed of the network is tested including the time it takes to initiate a speedy data stream. This is important in cases where many small files will be transferred. On the other hand the „10-seconds-measurements“ show the peak performance once a stable transmission is achieved. It gives insight on how the network behaves when large data files like high resolution photos or videos are up- or downloaded. The value given for „90 % faster“ is what a customer can reasonably expect most of the time.
Vodafone and KPN delivered themselves a head-to-head-race on a high level in this category, while Tele2 was clear behind, but still delivered good performance. The newcomer-status of Tele2 can be seen in the fastest 10 % of the measurements, where this provider is on a par with Vodafone and KPN. This means in some areas Tele2 must have bridged the gap to Vodafone and KPN, but there still a lot of work to do, to roll out a high performance network all over the Netherlands.
YouTube in Standard and high definition
Video transmission was long regarded as impossible in a mobile network. But once the faster networks started to become reality, services like YouTube became the major source of traffic. Today YouTube amongst other video services is a commodity in the mobile world and every operator should deliver a highly reliable stream at least in standard definition (SD). This is true for Vodafone and Tele2 with success ratios above 99 percent while KPN loses nearly 4 % to perfection. With high definition (HD) Video demands are even more stringent on the network, but Vodafone can keep its performance notwithstanding the higher data rates. Curiously KPN is a little bit better on HD than on SD but still loses ground to Vodafone. And Tele2 again falls in between, this time nearer to KPN.
Vodafone is the winner of the data benchmark in big cities, as the operator reveals good results across all key performance indicators (KPI). KPN is a close second with only minor difficulties in YouTube reliability while Tele2 is still in the roll out phase but showing ambitions.
DATA ACCESS IN SMALL CITIES
While providers take great care in covering the metropolitan areas, small cities often lag behind. Is this true for the Netherlands, too?
|WEB-PAGE DOWNLOAD (LIVE/STATIC)|
|Sucess Ratio (%/%)||95,2/99,5||97,7/100||92,2/94,2|
|Avg. Session Time (s/s)||3,9/1,5||3,5/1,4||4,0/2,3|
|FILE DOWNLOAD (3 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||99,4/2,6||100/1,9||98,7/4,4|
|90 % faster than (kbit/s)||4934||7757||2858|
|10 % faster than (kbit/s)||37559||30341||39280|
|FILE-UPLOAD (1 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||98,8/1,6||99,8/1,5||97,4/4,2|
|90 % faster than (kbit/s)||2982||3740||1195|
|10 % faster than (kbit/s)||13559||10959||9780|
|FILE DOWNLOAD (10 SECONDS)|
|Success Ratio (%)||99,8||100||99,6|
|Avg. Throughput (kbit/s)||22073||21777||14945|
|90 % faster than (kbit/s)||5610||7963||2689|
|10 % faster than (kbit/s)||49473||39410||42857|
|FILE UPLOAD (10 SECONDS)|
|Success Ratio (%)||100||100||98,2|
|Avg. Throughput (kbit/s)||15539||12833||5652|
|90 % faster than (kbit/s)||2934||4486||1096|
|10 % faster than (kbit/s)||29144||19408||16534|
|Success Ratio/Start Time (%/s)||99,8/1,2||97,0/1,1||98,4/1,8|
|Video playouts without interruptions
|Success Ratio/Start Time (%/s)||98,8/1,5||95,7/1,5||91,6/2,4|
|Video playouts without interruptions (%)||99,2||100||95,7|
While KPN managed to be a close second to Vodafone in the big cities, the biggest provider of the Netherlands shines in small cities. In the web-page download benchmark KPN delivers nearly the same reliability and speed as in big cities. A success ratio of 97,7 % for live web sites is remarkable. With 95.2 % Vodafone loses 2,5 % in this category while Tele2 is another 3 % behind. Concerning the speed of the surfing sessions KPN is again first, while Vodafone and then Tele2 are closely following.
File Transfers and YouTube
The speed differences get bigger when it comes to file transfers. Customers interested in the best coverage should closely look at the speeds given, where 90 % of the samples are faster. KPN is again the clear leader in this categorie.
In comparison the 10 % of the fastest values are comparatively close together. This is an indicator, that areas of good deployment make use of the potential given by modern LTE networks in nearly the same way.
While until now KPN dominated the benchmark in small cities, Vodafone again comes first with YouTube in SD and HD. The success ratio and the lack of interruptions is top notch. KPN and Tele2 in comparison are still good, with the better results with videos in standard Definition.
DATA ON CONNECTING ROADS
Perfect coverage of roads is a difficult task for providers. Nonetheless customers have high expectations. Will they be disappointed?
|WEB-PAGE DOWNLOAD (LIVE/STATIC)|
|Sucess Ratio (%/%)||94,5/99,4||98,0/100||86,0/90,4|
|Avg. Session Time (s/s)||3,9/1,4||3,6/1,4||4,0/2,3|
|FILE DOWNLOAD (3 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||99,3/2,6||100/1,8||94,6/5,3|
|90 % faster than (kbit/s)||5248||8400||2070|
|10 % faster than (kbit/s)||35088||32609||36866|
|FILE-UPLOAD (1 MB)|
|Sucess Ratio/Avg. Session Time (%/s)||99,3/1,7||99,6/1,3||93,6/4,3|
|90 % faster than (kbit/s)||2571||4376||1002|
|10 % faster than (kbit/s)||13722||11065||9975|
|FILE DOWNLOAD (10 SECONDS)|
|Success Ratio (%)||100||100||96,2|
|Avg. Throughput (kbit/s)||18980||22651||15675|
|90 % faster than (kbit/s)||3791||7218||2223|
|10 % faster than (kbit/s)||35599||43451||43592|
|FILE UPLOAD (10 SECONDS)|
|Success Ratio (%)||98,5||99,6||91,4|
|Avg. Throughput (kbit/s)||14438||14665||6822|
|90 % faster than (kbit/s)||1761||3313||1096|
|10 % faster than (kbit/s)||22205||20277||18788|
|Success Ratio/Start Time (%/s)||97,8/1,3||96,8/1,1||95,0/1,9|
|Video playouts without interruptions (%)||100||100||97,8|
|Success Ratio/Start Time (%/s)||95,6/1,8||97,1/1,5||83,6/2,9|
|Video playouts without interruptions (%)||96,5||99,3||92,8|
A mobile cell covering the road normally reaches only a comparatively small number of customers. This makes the deployment of a mobile network along motorways a costly affair. Nonetheless KPN managed to deliver results in connecting roads performance which were remarkably close to the benchmark results shown in cities.
The key performance indicators in the web-page download sessions showed a success ratio of 98 % which is even slightly higher than in small cities. Vodafone with 94,5 % and even more Tele 2 with 86 % had noticeably more difficulties covering the roads. The speed differences between big cities, small cities and connecting roads seem negligible by comparison.
But speed differences become a major differentiator between the providers in the file transfer benchmarks. At the 3 MB download for instance KPN is faster than 8400 kB/s in 90 percent of the samples, while Tele2 only exceeds 2070 kBit/s and Vodafone falls in between the two. The same ranking applies for the 1 MB download and the 10 second transfers showing again an outstanding good coverage from KPN, good performance from Vodafone and a network in the roll out phase from Tele 2.
On YouTube KPN manages to beat Vodafone on High Definition while Vodafone is best on SD.
As in Germany, Austria and Switzerland P3 communications, connect’s partner for executing network benchmarks, sent out two measurement cars, to perform the tests. The cars were equipped with 9 Samsung Galaxy S5 smartphones each. Six of them where Cat4 devices (SMG900F) and used for the voice test, the other three belonged in the Cat6 category (SM-901F) and were used for data measurements.
As far as possible, the smartphone’s firmware matched the network operator’s genuine ones. In cases when no operator specific firmware was available the most recent Samsung firmware was used.
Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions background data traffic was generated on the smartphones during a voice call.
The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm.
All network operators offer 4G capable subscriptions for everybody interested in a high performance data network. To take the high share of LTE into account speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch (“fall back”) to 2G or 3G because they camped on the LTE service when a phone call was initiated. This procedure is known as circuit-switched fall back (CSFB).
The testing of data services was done with three separate Samsung smartphones in the cars. As a first measurement task several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used.
In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located in the Internet. In addition, the peak data performance was tested in up and downlink direction by assessing the amount of data that could be transferred within a 10 seconds time period. While 10 seconds peak throughput test focuses on network capability the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7MB, 30s) and high definition (HD, 720p, 11.9MB, 30s) were played using the YouTube player on the smartphones.
Route and Samples
The route is shown on page 1 of this report. In the big and small cities indicated the cars had to follow pre-defined routes to measure a choice.
Measurement of one of an early Antenuatr prototype in the connect Testlab. The OTA-measurements (Over The Air) were used to confirm the predicted attenuation of the specialized high performance material in the P3 design.
CONCLUSION Bernd Theiss, connect editor
The first mobile network test of connect and P3 communications in the Netherlands has shown three providers which fare very well in an international comparison. This is even true for the newcomer Tele2 with its LTE-OnlyNetwork and national roaming for voice services and 3G access in areas without LTE. The Tele2 network is still in the roll out phase, which is indicated by peak performance on par with the other two operators and slightly below average performance. It remains interesting where Tele2 will be once the roll out is completed and Voice over LTE becomes a standard. KPN and Vodafone deliver exceptional performance, especially when compared to mobile networks in other countries. KPN is best in small cities and on connecting roads, indicating a very broad coverage throughout the Netherlands. For people travelling and taking the internet everywhere paradigm seriously KPN certainly is the top choice. But Vodafone strikes back with a slightly better voice service in the big cities where most of the people live and with more reliability in the very popular YouTube video streaming. Vodafone manages to stay in front in this headto-head race. At the end the photo finish reveals a 2 points advance for Vodafone, which is best in test.
Pieter Waasdorp (left, editor in chief Connexie) and Cees de Zeeuw (right, P3 communications NL) in front of the measuring car.
Altogether more than 6000 speech samples were logged per operator, about half of them in 4G preferred to 4G preferred mode, while the other half 4G preferred to 3G preferred. For the data benchmark about 29760 samples were obtained per operator.
About 61 % of the samples were obtained in big cities while the remaining 39 % were collected in small cities and on connecting routes.
A new approach to measure mobile networks
In former days it was the usual approach to conduct mobile network drive tests with the measurement devices mounted in the car and connected to antennas in a roof box mounted on the vehicle. In accordance with ETSI TR 102 581 an attenuation of the roof antenna signals of -12 decibel was used to compensate for the antenna gain (+ 5 dB) and to simulate a typical attenuation (- 7dB) of a mobile used indoor. With modern smartphones with their orthogonal MiMo antenna arrangements used for HSPA and LTE and with the use of more intelligent antennas changing their directivity and tuning in relation to the signals received, a new approach is better suited to reflect real life conditions.
This new approach is a highly optimized mounting case for the smartphones. It is designed with unique materials selected for their specific absorption rate at high frequencies between all last 800 MHz und 2,6 GHz. By choosing the right selection of materials in the right places an attenuation rate typical for indoor measurements can be achieved without altering the antennas of the smartphones. The validity of this approach, for which a patent is pending, has been confirmed by comparing measurements between unaltered phones. The legacy approach with antennas in a roof box and attenuators and the P3 Antenuatr has been further verified by comparing TIS (total isotropic sensitivity) and TRP (total radiated power) acording to the CTIA test plan for over the air measurements in the Testlab owned by WEKA Media Publishing, the parent company of connect. These measurements confirmed an attenuation rate very close to the wanted – 7 dB from LTE band 20 (800 MHz) to LTE Band 7 (2,6 GHz). The variation is even smaller than with external antennas connected over wideband attenuaters.
Three of these antenna boxes have been mounted in each of the two cars in the rear side and in the rear windows. One Antenuatr was responsible for 4G preferred to 3G preferred voice measurements, while the second was used for 4G preferred to 4G preferred voice measurements. The last Antenuatr was responsible for the data measurements.
Real live testing confirmed the advantage of using the differently polarized original MiMo antennas, as this increases the likeliness of achieving maximum data rates due the the higher linear independence of the antenna signals.
Up to three P3 Antenuatr-Boxes can be mounted into the back and side windows of each measuring car and up to four smartphones can be used to test up to four providers simultaneous.
|Telephony max. 400||353||354||309|
|Big cities 260||88%||86%||79%|
|Small cities and connecting roads 140||89%||93%||74%|
|Data max. 600||528||525||457|
|Big cities 390||90%||87%||83%|
|Small cities 90||83%||84%||65%|
|Connecting roads 120||86%||92%||64%|
|Connect RATING max. 1000||881
„Vodafone and KPN crossed the line in a photo finish which put our measuring accuracy to the test and with a score clearly ahead of providers in other countries“