Germany 2014

THE GREAT MOBILE NETWORK TESTdtl. flagge

Download test here

 

Conducted since 1993, the connect mobile network test with its neutral, customer-oriented testing has become an industry standard by which big corporations choose their operator. Who’s leading in Germany, Austria and Switzerland in 2014?

As Rudolf Augstein, founder of Germanys biggest news magazine „Der Spiegel“, said as guidance to young journalists: „Just being straight“. This motto should also hold true for a mobile network test, but it is not always so.
One German magazine simply focuses on easy to sell sensations and presents their benchmark as “Germany’s hardest network test” – despite the fact that two of the four German operators ended up third for the second year in a row and with minimal distance to the winner.
Another big publication completely lost track of their benchmark data, rating an operator with less than 3% 4G market share higher than one with more than 30% of the LTE market but slightly less throughput.

The proven Standard – since 1993

Connect refrains from superlatives and sends the proven test team of P3 communications on a trip through Germany, Austria and Switzerland. Two measurement cars per country, equipped with two measurement smartphones per operator, analyzing voice and data services in big cities, small cities and on the connecting highways.
In addition, two walk test systems per country, one for voice and one for data, are used for the same tests as in the measurement cars, with focus on indoor locations and railway connections. A third measurement car is used as counterpart station for the walk test voice system. The detailed methodology description can be found at the end of this page. The results of the German tests can be found on the following pages. How the Austrian and Swiss operators perform this year, and whether they outperform the Germans again, can be found in the menue point “Network Tests”.

GERMANY

Voice Telephony

An almost perfect voice service should be mandatory. Nevertheless, some operators showed some weaknesses last year. Will that happen again, this year?
LTE was a perfect technology to help the operators to offload some data traffic from their heavily used 3G networks in last year’s network test. But it put a lot of challenge on the voice telephony. Since LTE did not support voice calls, as a workaround all smartphones had to switch to UMTS- or GSM-networks – so- called “circuit switched fall back” (CSFB). This complex handover took time and reliability and especially Vodafone, with its large LTE area coverage and incomplete optimization, suffered from this effect.
Voice over LTE (VoLTE) is currently being trialed by all operators (Connect 12/14), but not yet implemented. So normal voice calls are still using UMTS and GSM networks.

Voice in Cities

In big cities, Telekom, Vodafone and O2 have an LTE share between 80% and up to more than 90%, leading to many of those CSFB handovers.. E-Plus needs less CSFB due to an LTE share below 50%.
It is exciting to see how Vodafone has improved compared to last year, and what impacts we will identify from O2’s increased LTE share.
The outdoor drive test results show Vodafone on the same level as Deutsche Telekom with exactly the same performance figures and a reliability improvement of 4.4% points compared to last year. Their speech quality MOS value (mean opinion score) of 3.4 is even slightly better.
To ensure nothing can go wrong, engineers and technicans prepare the measurement setup painstakinngly.
The success rates of those two operators are again on a good level of 98.5%, which is sufficient to fulfill the requirements of demanding users.
O2 and E-Plus are dropping back, by only achieving success rates around 96%, plus O2 has additional weaknesses in call set up times. O2’s CSFB still needs some optimization. Missing HD voice features lead to worse speech quality in the O2 network.
E-Plus’ indoor voice reliability, with its smaller LTE share, comes very close to Telekom and to the even better performing Vodafone. O2 remains distant.

Voice on Highways and Railways

Telekom has increased the distance on highways and delivers an excellent 98% success ratio. Vodafone caught up by 4% points against last year and ends up 2% behind. O2 and E-Plus lost additional roughly 3%. O2 shows the same problems with call setup times and speech quality as in the cities.
All operators have problems with railway coverage. Telekom has 13% unsuccessful calls, closely followed by O2 and Vodafone, while EPlus lost up to 21% of their calls.
In addition, 2 – 3% of the calls suffer from interruptions or noise. All this leads to the impression that all operators have significant improvement potential on railways.

 

SUMMARY

Telekom achieved 175 of 210 points in voice telephony and is the best German operator in this category. Vodafone caught up tremendously and is able to beat Telekom in big cities with a slightly better score, improvement potential can be found on highways and in trains.
In big cities, E-Plus is not far away from the leaders and delivers a satisfactory result. Whereas O2 seems to be suffering from the fast LTE rollout and related challenges in reliability and call setup times.

Telephony

OPERATOR Deutsche
Telekom
Vodafone O2 E-Plus
TELEPHONY (BIG CITIES OUTDOOR)
Call Success Ratio (%) 98,5 98,5 95,7 96,0
Call Setup Time (s) 6,3 6,3 9,1 6,3
Speech Quality (MOS-LQO) 3,3 3,4 2,6 3,3
Call Sustainability (%) 99,6 99,6 99,2 98,8
TELEPHONY (BIG CITIES INDOOR)
Call Success Ratio (%) 98,0 98,3 96,1 97,4
Call Setup Time (s) 6,3 6,0 9,2 5,9
Speech Quality (MOS-LQO) 3,4 3,4 2,6 3,4
Call Sustainability (%) 99,7 99,6 99,8 98,7
TELEPHONY (HIGHWAYS)
Call Success Ratio (%) 97,6 95,4 93,1 92,5
Call Setup Time (s) 7,1 7,1 10,2 6,9
Speech Quality (MOS-LQO) 3,1 3,0 2,6 3,0
Call Sustainability (%) 98,9 99,2 98,9 98,1
TELEPHONY (TRAINS)
Call Success Ratio (%) 86,7 83,9 85,5 78,6
Call Setup Time (s) 7,6 7,3 9,7 8,0
Speech Quality (MOS-LQO) 2,7 2,7 2,5 2,6
Call Sustainability (%) 96,5 97,9 96,5  96,5


 

Smartphone Data usage in Cities

The LTE roll out has been driven quite far by three German network operators in the big cities.The questions is: will this provide top performance as regards data service usage?
Apart from some small exceptions, the data service success ratios are really on a very high level in the big city Scenario.

Data service in big cities

For example regarding drive testing the success ratios are in the range above 99%. Only E-Plus showed some weaker results for the static web browsing with 98,4 % and for the HTTP file download with 98,6 %.
Regarding the YouTube HD video assessment some minor (O2 and Vodafone) and some increased failure ratios (EPlus) have been observed.
Nevertheless the reliability of all operators is at a really high level in the city outdoor scenario. For anybody not interested in HD but just SD, video streaming service is sufficiently served by the EPlus network in terms of internet access. Anyhow for many users it’s not only the reliability that counts but also the transfer speed. Apart from some rare scenarios the order in terms of speed is always Telekom, Vodafone, O2 and EPlus. The live web browsing and the frequently used file upload performance of Vodafone is quite close to the leader Telekom, followed by O2 and E-Plus again.
The difference between Telekom and the other network is increased as regards file downloads. Telekom network is up to 3 times faster than EPlus. This operator achieves average speeds around 40 Mbit/s and speeds higher than 11 Mbit/s in 90% of all samples. Telkom achieves a similar good level as last year taking into account an increase in traffic volume. The additional indoor measurements confirm the outdoor results with very high accuracy. Only the upload results show a slightly weaker performance indoors. Telefonica shows in general slightly weaker results indoor and regarding YouTube HD Vodafone and O2 show weaker results indoor and end up behind E-Plus.
In general one could say that all operators in Germany provide stable conditions in big cities. Telekom takes the lead here due to its sufficient LTE 1800 MHz coverage.

Data service in small cities

Compared to the well deployed big cities, the smaller towns don’t show sufficient coverage in most cases. Nevertheless the Telekom network shows an impressive performance in this scenario. Based on the same good overall reliability observed in the big cities, web browsing performance at a similar level and a slightly decreased file transfer performance the Telekom network is the first choice for this scenario. Vodafone is placed behind Telekom and shows a decreased performance in terms of reliability and speed. Anyhow the Vodafone network is still able to satisfy user expectations. O2 and E-Plus show at least adequate performance for the web browsing service. However transfer of files or watching YouTube videos in HD quality is demanding for users.

SUMMARY

Regarding data performance in urban areas Telekom leads the field again in 2014. Vodafone and O2 offer good performance and high reliability with minor degradations in this scenario. Looking at small towns, O2 suffers from the as yet unfinished LTE expansion. This network is placed behind Vodafone in this category. But the Reds cannot manage to outdo the speed of Telekom with their wide-area broadband LTE-1800 expansion. E-Plus on the other hand clearly suffers from the delayed LTE rollout. The reliability is top, the speed a flop.

Data in big and small cities

OPERATOR Deutsche
Telekom
Vodafone O2 E-Plus
DATA (BIG CITIES OUTDOOR)
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Success Ratio (%/%) 99,8 / 99,9 99,9 / 99,7 99,7 / 99,0 99,4 / 98,4
Avg. Session Time (s/s) 3,2 / 0,8 3,8 / 1,0 4,0 / 1,1 5,1 / 2,8
FILE-DOWNLOAD (3 MB)
Success Ratio/Avg. Session Time (%/s) 99,8 / 1,6 99,8 / 2,5 99,9 / 2,6 98,6 / 4,5
90 % faster than (kbit/s)  11734  5475 5428 3340
FILE-UPLOAD (1 MB)
Success Ratio/Avg. Session Time (%/s) 99,9 / 1,8 99,8 / 2,4 99,9 / 3,4 99,4 / 7,0
90 % faster than (kbit/s) 1864 1822 1129 471
FILE-DOWNLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 37314 19707 17565 17432
90 % faster than (kbit/s)  11525  5411  5963  3710
FILE-UPLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 21134 12815 8875 6296
90 % faster than (kbit/s) 1579 1868 1291 382
YOUTUBE SD
Success Ratio / Start Time (%/s) 99,9 / 0,7 99,9 / 1 100 / 1,1 99,3 / 1,5
Video playouts without interruptions (%)  99,8  99,8  99,9  98,8
YOUTUBE HD
Success Ratio / Start Time (%/s)  99,5 / 0,9  98,7 / 1,4  98,9 / 1,5  96,8 / 2,1
Video playouts without interruptions (%)  98,9  97,0  98,2  94,3
DATA (BIG CITIES INDOOR)
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Success Ratio (%/%)  99,4 / 99,6  99,4 / 99,7  99,2 / 99,5  98,9 / 98,8
Avg. Session Time (s/s)  3,5 / 0,8  4,4 / 1,1  5,3 / 1,3  5,5 / 1,7
FILE-DOWNLOAD (3 MB)
Success Ratio/Avg. Session Time (%/s)  99,6 / 2,1  99,7 / 4,3  99,2 / 4,5  98,0 / 3,9
90 % faster than (kbit/s)  10729  4006  3023  3450
FILE-UPLOAD (1 MB)
Success Ratio/Avg. Session Time (%/s)  99,7 / 2,5 9  99,5 / 3,6  99,6 / 6,1  99,4 / 8,3
90 % faster than (kbit/s)  1558  1236  715  449
FILE-DOWNLOAD (10 SECONDS)
Avg. Throughput (kbit/s)  38570  23388  14233  16079
90 % faster than (kbit/s)  10930  4464  3199  3761
FILE-UPLOAD (10 SECONDS)
Avg. Throughput (kbit/s)  19299  12135  6260  4351
90 % faster than (kbit/s)  1441  1251  736 363
YOUTUBE SD
Success Ratio / Start Time (%/s) 100 / 0,7 99,9 / 1,1 99,6 / 1,6 99,6 / 1,6
Video playouts without interruptions (%) 100  99,6 99,6 99,1
YOUTUBE HD
Success Ratio / Start Time (%/s) 99,7 / 0,9 97,3 / 1,5 96,2 / 2,1 96,9 / 2,1
Video playouts without interruptions (%) 99,3 93,9 91,4 94,4
DATA (SMALL CITIES OUTDOOR)
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Success Ratio (%/%) 99,6 / 99,7 98,9 / 96,8 98,6 / 98,0 99,3 / 97,8
Avg. Session Time (s/s) 3,4 / 0,9 4,6 / 1,9 5,5 / 2,2 6,1 / 3,3
FILE-DOWNLOAD (3 MB)
Success Ratio/Avg. Session Time (%/s) 99,5 / 4,1 98,4 / 4,8 99,4 / 8,2 98,9 / 6,3
90 % faster than (kbit/s) 8386 2975  1956  2626
FILE-UPLOAD (1 MB)
Success Ratio/Avg. Session Time (%/s)  100 / 2,6  100 / 6,7  100 / 7,9  98,3 / 8,9
90 % faster than (kbit/s)  1611  541  700  431
FILE-DOWNLOAD (10 SECONDS)
Avg. Throughput (kbit/s)  29782  11311  17519  8870
90 % faster than (kbit/s)  7255  2739  2253  2152
FILE-UPLOAD (10 SECONDS)
Avg. Throughput (kbit/s)  13695  5031  6109  1980
90 % faster than (kbit/s)  1302  529  600  322
YOUTUBE SD
Success Ratio / Start Time (%/s)  100 / 0,8  100 / 1,5  100 / 1,5  96,7 / 2,3
Video playouts without interruptions (%)  99,4  99,5  98,3  96,7
YOUTUBE HD
Success Ratio / Start Time (%/s)  99,4 / 1,1  97,2 / 2,1  91,3 / 2,6  95,5 / 3,0
Video playouts without interruptions (%)  97,7  90,9  88,4  89,8

 

Data on Highways

OPERATOR Deutsche
Telekom
Vodafone O2 E-Plus
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Success Ratio (%/%) 99,1 / 99,4 98,3 / 98,0 93,3 / 91,3 94,5 / 91,4
Avg. Session Time (s/s) 3,8 / 1,5 4,4 / 1,5 5,8 / 2,4 6,8 / 4,1
FILE-DOWNLOAD (3 MB)
Succ. Rat./Avg. Ses. Time (%/s) 98,4 / 3,5 98,2 / 4,6 91 / 12,2 92,1 / 13,1
90 % faster than (kbit/s) 5572 3836 269 593
DATEI-UPLOAD (1 MB)
Succ. Rat./Avg. Ses. Time (%/s) 99,9 / 3,9 98,6 / 4,1 93,5 / 11 94,1 / 14,1
90 % faster than (kbit/s) 979 941 298 271
DATEI-DOWNLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 21741 15213 14793 8347
90 % faster than (kbit/s) 5264 3461 1291 383
DATEI-UPLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 10685 9790 6950 1909
90 % faster than (kbit/s) 1047 577 79 77
YOUTUBE SD
Success Ratio / Start Time (%/s) 99,2 / 0,9 99,4 / 1,2 99,1 / 1,5 98,2 / 1,9
Interruption free (%) 98,9 98,8 98,6 97,4
YOUTUBE HD
Success Ratio / Start Time (%/s) 98,1 / 1,4 96,1 / 1,8 94,6 / 2,1 89,9 / 3,1
Interruption free (%) 96,8 92,4 91,8 85,1

 

With a measurement setup in a trolley, the complete test routine can also be performed in Trains.

Data in Trains

Customers frequently facing low quality of service in trains may ask themselves: Could a change improve my service level?
The train scenario in general is a nightmare for mobile communication companies. Most of the time the base stations along the tracks don’t serve any customer and just consume power. But then for a few minutes they have to server a hundred or more customers passing by at high speeds. And additionally these customers are shielded by a case of metal and aluminized glass. Only some repeaters which amplify GSM and LTE 1800 indoor improve this situation.
This scenario is clearly reflected by the results observed on train routes. Telekom offers the lowest failure ratios with up to 11% but suffers from low speeds regarding all data services. But this is still the good case. E-Plus, Vodafone and O2 show huge gaps with 38% in terms of success ratios and really low throughputs. Based on these results normal Internet usage is not possible. As a result of this overall low service Level the Telkom network is in this case the one-eyed man among the blind.
The network operators and Deutsche Bahn have to improve this situation jointly in the future.

Data on Trains

OPERATOR Deutsche
Telekom
Vodafone O2 E-Plus
WEB-PAGE DOWNLOAD (LIVE/STATIC)
Success Ratio (%/%) 89,0 / 90,6 83,6 / 82,1 80,9 / 72,7 87,6 / 83,9
Avg. Session Time (s/s) 5,7 / 1,1 7,4 / 2,2 8,1 / 2,6 7,9 / 2,4
FILE-DOWNLOAD (3 MB)
Succ. Rat./Avg. Ses. Time (%/s) 86,1 / 7,1 72,8 / 18,9 61,9 / 23,9 77,6 / 12,8
90 % faster than (kbit/s) 320 111 100 144
FILE-UPLOAD (1 MB)
Succ. Rat./Avg. Ses. Time (%/s) 90,2 / 7,3 76,9 / 11,8 69,8 / 12,1 82,3 / 12,8
90 % faster than (kbit/s) 706 229 301 250
FILE-DOWNLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 17937 6426 5836 5190
90 % faster than (kbit/s) 97 40 27 59
FILE-UPLOAD (10 SECONDS)
Avg. Throughput (kbit/s) 12172 2854 2134 1985
90 % faster than (kbit/s) 45 16 7 12
YOUTUBE SD
Success Ratio / Start Time (%/s) 96,5 / 1,3 98,4 / 2,7 95,9 / 2,7 95,7 / 2,4
Interruption free (%) 95,3 97,6 93,8 94,9
YOUTUBE HD
Success Ratio / Start Time (%/s) 93,4 / 1,4 87,5 / 3,6 90,2 / 4,4 85,3 / 4,0
Interruption free (%) 92,7 76,8 74,4  78,9

 

Individual reviews

Overview of strengths and weaknesses for German network Operators.

Deutsche Telekom

Deutsche Telekom sets the benchmark in Germany for the fourth time in a row.
Deutsche Telekom outperforms the other operators with regards to railway service, however their performance is still is not considered good. This is especially clear when looking at the direct comparison with international competitors in neighbouring countries.
For voice telephony Telekom shows highest success ratios and is on the same high level with Vodafone for speech quality and call setup times.
In terms of data services Telekom achieves average download data rates between 8,4 and 30 Mbit/s, which is significantly higher than the competitors. Also in the other disciplines the operator from Bonn defines the performance that customers can expect. Due to this it is an excellent choice for customers that intend to use mobile Internet beyond just in the big cities.
For anyone looking for very good voice telephony and excellent data service quality in big cities, small cities and transfer routes, Deutsche Telekom is a good fit. After a dip to third rank in 2010 Deutsche Telekom now achieved 1st place for the fourth time in a row. Congratulations to Bonn!
connect-Score
very good (436 points)

Vodafone

Despite harder test procedures Vodafone is back as premium network operator.
As with the other network operators, covering the railways is challenging for Vodafone. For voice Vodafone has made a big step ahead after suffering from high failure ratios and long call setup times last year due to complex Circuit Switched Fallback (CSFB) that is needed in LTE. Success ratios of more than 98% in cities at best speech quality and fast call setup times show that the company from Düsseldorf is back on track.
This is also confirmed by data service measurements where Vodafone does not show gaps to Telekom with regards to reliability in big cities. The throughput levels are not always as high as for Telekom, however this only impacts very challenging data-intensive activities. In small cities there are specific measurement spots with improvement potential. The transfer routes show a high quality service.
With a good overall performance Vodafone made a comeback in fighting for demanding customers and achieved an indisputable 2nd rank.
connect-Score
good (400 points)

O2

O2 has progressed well in its LTE rollout, now is the time to follow-up with optimizations
Due to the acquisition of E-Plus, the O2 network can be considered as work in progress. The merger of both operator´s networks is expected soon. However this has not blocked the LTE rollout of O2 during the last year, and for the meantime the big cities in Germany are already covered. The ramp-up status of the O2 LTE network causes relatively weak voice telephony performance as seen for other operators before. CSFB is the reason behind reduced success ratios of around 96% and even lower outside big cities, as well as increased call setup times in the range of 9-10 seconds. Here O2 needs to quickly execute the respective optimization activities.
With respect to data connections in big cities O2 has significantly improved reliability compared to last year, and partly also data rates. In the case of railways O2 is dealing with coverage issues, while on transfer routes data services are perceived better. Overall O2 has a decent data performance.
connect-Score
satisfactory (335 Points)

E-Plus

With still lower LTE ratio the E-Plus data performance is showing gaps.
After optimizing the UMTS network E-Plus started to build LTE in large cities. Just below 50% of the data measurements there are executed in the fast fourth generation network, in small cities and on transfer routes the ratio was below 10%.
Also reliability of the EPlus voice service seems to suffer from the complexity associated with CSFB handover. The stability has reduced compared to last year. Despite this E-Plus managed to keep the call setup time at a low level and to improve speech quality. This is sufficient to achieve better voice performance than O2.
The reliability as well as speed of data connections on transfer routes and railways looks less promising. Also in cities E-Plus lacks throughput due to small LTE deployment. This is acceptable for inhabitants of the cities as long as large uploads or HD videos are not relevant, since the reliability is good and the speed is definitely enough for web browsing.. >>
connect-Score
satisfactory (327 points)

 

 

 

RULES OF CONNECT TEST

In order to assure fair competition, high transparency is essential.
The first meeting to setup the test for 2014 was held on the 24th of January this year, involving experts form P3 communication and Connect magazine. This meeting aimed to review the 2013 connect test, introducing LTE for the first time, and also to discuss the framework for 2014. The goal was to establish a high level schedule, choosing the smartphones and to define the test locations for the three countries under test.

Transparency for Equal Opportunities

After the basic alignment of the test framework detailed information was sent to all CTOs of the participating network operators. This information contained a specific time schedule and defined the concept in terms of logistics, locations and services under test. After sharing this information all CTOs were asked to provide their feedback. This approach was established to avoid any situation whereby major network changes were scheduled during the test period. Furthermore the technical experts of the mobile communication providers had the chance to provide their feedback and to challenge the test methodology. This feedback was introduced into a second and final version of the information presentation, which was shared with the technical management at the beginning of August. This document is available via this link: www.connect.de/netztestplanung.
These beneficial discussions are really appreciated by Connect magazine and also P3 communications. Nevertheless Connect magazine always has the last word regarding the measurement and scoring methodology.
Further Information shared with the operators contains start and end dates and pre-shared data about the results before the official release. This additional information avoids major network changes during the test period and gives sufficient preparation time for the post connect test public communication of each operator.

Transfer Routes

The mobile communication deployment on German highways is supposed to be quite good. Which network offers the best service?
Transfer routes between the cities are a challenge for all network operators under test. This was already revealed during past years’ tests, with success ration around 80% and below.
This year increased reliability was observed for all participating network operators, represented by a success level around 90% for most services. Regarding the live web browsing activity success ratios are located between 93.3% for Telefonica and 99.4% for Telekom. Vodafone shows values around the leading Telekom and E-Plus even better than Blue (O2).
A huge step forward was also observed for the speeds. The results show 81% LTE share for Telekom, 84% for Vodafone and O2 showed up with 72% LTE coverage. Only E-plus was observed behind with 7% less than O2. This dense deployment results in high throughputs, especially for Vodafone and Telekom, showing significantly increased data rates compared to 2013. Only O2 could not clearly benefit from the LTE share. 10% of all measure samples show less than 1.3 Mbit/s. Regarding E-plus performance, 10% of the downloaded samples show less than 384kbit/s. In terms of uplink speed both, O2 and eplus show weak results.

 

TEST METHODOLOGY CONNECT

As in previous years P3 communications, Connect’s partner for executing network benchmarks, sent out two measurement cars to perform the tests. The cars were equipped with ten Samsung Galaxy S4 LTE+ smartphones each – six to eight of them used for testing depending on the country. As far as possible, the smartphone’s firmware matched the network operator’s genuine ones. In cases when no operator specific firmware was available the most recent Samsung firmware was used. In addition to the cars a separate test team executed walk tests indoors or in trains using a portable battery powered measurement system.

Smartphone-Telephony

Data and voice services were executed with separate smartphones. Telephony was tested mobile-to-mobile from one car to the other. In order to ensure realistic testing conditions background data traffic was generated on the smartphones during a voice call.
The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm.
This year, all network operators offer LTE capable subscriptions. Therefore all smartphones were set to “LTE preferred” mode. As a consequence, the phones needed to switch (“fall back”) to 2G or 3G when they camped on the LTE service and a phone call was initiated. This procedure is known as circuit-switched fall back (CSFB).

Smartphone-Data

The testing of data services was done with three to four separate Samsung smartphones in the cars. As a first measurement task several popular live web pages were browsed using the built-in smartphone browser. The web pages had been selected previously according to the Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for such testing purposes was used.
In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located in the Internet. In addition, the peak data performance was tested in up- and downlink direction by assessing the amount of data that could be transferred within a 10 seconds time period.
While 10 seconds peak throughput test focuses on network capability the YouTube tests add a more end user centric view to the test curriculum. Here, videos with standard definition (SD, 360p, 2.7MB, 30s) and high definition (HD, 720p, 11.9MB, 30s) were played using the YouTube player on the smartphones.
In order to simulate in-car or indoor coverage a 12dB attenuation was inserted in the path between the smartphones in the car and the antennae mounted on the car’s roof top. Due to the usage of MIMO technology (Multiple Input Multiple Output) in LTE two antennas were used per each smartphone.

Walk tests indoors and in trains

The same test program executed for voice and data services in the measurement cars was also executed by the walk test teams.
Four smartphones on the left-hand side are used for testing telephony while the phones on the right serve for testing the data Services

Logistics

During the measurements, the two cars were in the same city, but not at the same location. This was to avoid disturbances from one car to the other. Both cars visited defined destinations. For stationary measurements they stopped at selected “areas of interest”, typically crowded locations such as airports, train stations, recreation areas or densely populated residential areas.
Especially at those locations, the measurement systems have to share the available networks capacities with other normal users. This is according to the usual competition between users for the more or less scarce bandwidth.
The selected cities should be representative for a large proportion of the population. Therefore the measurements were executed in 21 of the bigger cities in Germany, from Osnabrück with 156,000 inhabitants up to Berlin with 3,4 million inhabitants. Smaller cities were visited by one measurement car during the transfers from one large city to the next one, along the routes. 27 smaller cities were visited, covering sizes from 20,000 inhabitants, like Witten or Nauen, up to almost 90,000 inhabitants in cities like Esslingen. In Switzerland and Austria 16 of the biggest 25 cities were covered, including approximately 20 smaller cities in each country. Between the cities highways were mainly used by the measurement cars.

Effort and output

During 45,000 measurement kilometers – a circumnavigation of the world – 150,000 measurement samples were collected, almost half of them in Germany. This effort is necessary to provide statistically relevant network quality Statements.


FINAL CONCLUSION Bernd Theiss, connect editor

The 2014 network benchmark reveals that building and engineering an LTEnetwork for data combined with a reliable 2G/3G network for voice is not a trivial exercise. In Germany, Telekom was successful in doing so. The network operator provides telephony services of the highest performance in both small and big cities. But Telekon is also ahead of the competition on rural routes despite some weaknesses in trains. However Vodafone has made a great leap ahead and is now – in contrast to previous years – on Telekom’s heels and showed the best indoor coverage in the city centres during the test. Furthermore, Vodafone provides reliable and fast data services though Telekom is still ahead in terms of speed. Therefore, Telekom remains the test winner in Germany. O2 having introduced LTE late shows signs of weakness in voice service quality while E-Plus falls behind in data service performance due to a lack of LTE coverage. It will be exciting to see whether there will be a performance boost in the future after last year’s merger of the two operators seemed to have slowed down the development of their networks.
The Swiss networks Sunrise and Orange show how to exploit room for improvement and outpoint Germany’s number one Telekom this year. A1 Telekom Austria scored even slightly better but had to make way for the winner of the three country test: Swisscom.

The measurements took place on public transport and in particular in publicly accessible buildings like cafés, museums, train stations or airport terminal buildings. As far as possible the testers took regional trains to travel between cities and for taking measurements during the journey.

COUNTRY GERMANY AUSTRIA SWITZERLAND
Netzbetreiber Deutsche
Telekom
Vodafone O2 E-Plus A1 Telekom
Austria
T-Mobile Drei Swisscom Sunrise Orange  
Telephony max.210 175 171 128 145 188 175 183 181 189 182
Big Cities (outdoor) 95 85% 85% 58% 69% 90% 86% 87% 84% 91% 84%
Big Cities (indoor) 35 82% 84% 62% 78% 91% 89% 94% 89% 95% 93%
Highways 70 85% 79% 66% 68% 91% 80% 87% 88% 89% 87%
Trains 10 61% 57% 54% 43% 78% 66% 59% 82% 77% 89%
Data max. 290 261 229 207 182 265 242 230 275 253 259
Big Cities (Outdoor) 135 91% 83% 80% 68% 96% 87% 83% 94% 87% 88%
Big Cities (Indoor) 50 90% 78% 67% 65% 94% 91% 74% 94% 90% 92%
Small Cities 35 86% 65% 59% 53% 77% 72% 71% 96% 81% 86%
Highways 50 95% 88% 70% 58% 90% 77% 85% 97% 91% 90%
Trains 20 76% 54% 46% 55% 86% 77% 68% 95% 85% 96%
connect Rating max. 500 436
very good
400
good
335
satisfactory
327
satisfactory
453
very good
417
good
413
good
456
very good
442
very good
 441
very good