THE GREAT MOBILE NETWORK TEST 2015 IN GERMANY, AUSTRIA AND SWITZERLAND
Download mobile network test here
Going into its 22nd year, the connect mobile network test is still regarded as the industry standard: With great effort, as well as objective and customer-orientated testing methods, we have once again determined the leading mobile networks in Germany, Austria and Switzerland in 2015.
During the last five years, Deutsche Telekom alone has invested about 23 billion Euros in its fixed and wireless networks. The company plans to invest a similar amount within the next five years. Vodafone bought Kabel Deutschland for 7.7 billion Euros and has invested another four billion Euros into the expansion of its mobile networks in 2014 and 2015. Telefonica is also poised to spend billions of Euros in order to keep up with Deutsche Telekom and Vodafone.
Austria and Switzerland are no different: the network operators have invested several billions into their networks. And this doesn’t even include license fees for new wireless spectrum. These are impressive numbers, and the operators spend this amount of money in order to compete to offer the best mobile radio networks for their customers. It is thus very interesting to see which operator is really ahead of its competition.
Quality standard over many years
In order to answer this question, connect has been partnering for many years with the mobile radio network performance experts of P3 communications. Their test teams put great effort into carefully examining what network performance the users really experience in Germany, Austria and Switzerland. You can read about the test methodology on page 48. The primary goal for these measurements is objectivity and statistical relevance. The best proof for the quality of the measurements is the high regard with which the connect network test is received in the industry. Every year, the operators look forward to the relentless verdict of the connect test with much eagerness and excitement – and take our results much more seriously than those of some of our competitors. They might claim to conduct the „most severe“ or „most time-consuming“ network tests, but then they back manipulable user-reports, or let themselves be blinded by rare top data rates instead of significant average values. On the following pages, you will find credible answers to interesting questions such as: Was Deutsche Telekom able to keep its lead in Germany? How do the networks in Austria and Switzerland compare – are they still ahead of the German networks? And thus: are the billions of Euros of carrier investments paying off?
Voice calls may become less important compared to other communication channels. But anyone who makes a call expects reliability. Where to find it?
Both cars utilized for drive testing throughout 13 large cities and many smaller cities were equipped with eight Samsung Galaxy S5 each in order to assess voice service quality. The devices set up phone calls to their counterparts in the other car. In order to simulate normal smartphone behaviour there was continual background data service usage. The same setup was used for the walk test backpacks, which had their counterpart in a third car. The test devices and SIM cards supported LTE and for operation in networks offering the new 4G-voice service and also for “Voice over LTE” (VoLTE). Respectively for Vodafone and O2 a relevant proportion of the voice calls used VoLTE – making the connect network test 2015 the first VoLTE test with a big footprint. However, it was also considered that many subscribers still utilize older phones. The evaluation considered both, calls within the 4G networks (and with VoLTE where applicable) as well as calls from 4G to 3G over the usual “Circuit-Switched Fallback” technology.
VoLTE pays off
Call setup times and speech quality are clear assets of VoLTE. Vodafone even realized a small advantage in voice service compared to overall winner Telekom by using this technology. On the other hand it is by no means easier to achieve better results with VoLTE since the implementation of 4G-voice mode is complex. Telekom is still using circuit switched technology and loses only 2 out of 200 points against Vodafone. Also O2 was able to improve on last year´s result – in absolute numbers the biggest step. However, it was not sufficient to allow O2 to catch up with the leading two operators. E-Plus ended up with slightly better voice results than O2 and improvement compared to 2014 – but only to a small extent.
Best in big cities
Not surprising at all: In well covered big cities the success ratios were higher, call setup times shorter and speech quality slightly better than in smaller cities and on transfer routes. The small voice advantage for Vodafone materializes mainly in big cities – in other scenarios, Telekom led with a small advantage. While there was a small difference in quality level between the scenarios for the leading duo, O2 and E-Plus showed higher gaps.
Weaknesses in trains
Compared to last year there was not much progress on voice service in trains. Even with best-in-test Telekom ten per cent of the calls did not succeed. Even when the railway coaches are equipped with repeaters they are not capable of improving LTE signals – therefore almost no VoLTE was utilized. Clear degradation was visible for O2: Almost 30 per cent of the test calls failed. And even when call setup in the train worked for O2 or E-Plus, the speech quality was below expectations.
Despite a tight win on points for Vodafone, Telekom and Vodafone are on same level regarding telephony. E-Plus is behind with a significant gap but still having voice performance ahead of O2 – although the Telefónica network clearly improved compared to last year.
Mobile internet and services like audio streaming are booming, making mobile networks face even higher demands. Will they meet the increasing requirements?
To test data performance, the measurement cars and walk test backpacks were equipped with four Samsung Galaxy Note 4 each performing automated web browsing, file transfers and YouTube video playouts in SD- and HD-quality. The success ratios and the completeness as well as duration of the data transfers were evaluated. For YouTube the occurrence of video interruptions was considered as an additional performance metric.
Data service performance in big cities
Although all German network operators heavily invest in the development – and in case of O2 and E-Plus also in the merger – of their networks, only Deutsche Telekom managed to keep its performance on last year’s level. This indicates the high traffic load in the mobile networks caused by increasing usage of data services. However, especially the test results from the big cities are encouraging. Compared to 2014, at least slight enhancements in download data rates can be observed. In that respect, the simultaneous use of multiple LTE frequency bands by one device, called “carrier aggregation”, is a technology that could pay off for Deutsche Telekom and Vodafone. Deutsche Telekom is clearly one step ahead, showing download speeds nearly twice as fast as Vodafone. Furthermore, Deutsche Telekom achieves higher peak data rates than Vodafone. O2 and E-Plus achieve just about two thirds of what Vodafone provides on average – thus just about one third of the average speeds in the network of Deutsche Telekom. A slight decrease in data service performance compared to 2014 can be observed with respect to success ratios for web page download, both in the Vodafone and the O2 network as well as E-Plus, where the decrease is significantly higher. The performance decrease is also noticeable for video playouts in the O2 network and even worse in the E-Plus network. Telekom and Vodafone deliver videos with about the same performance as last year. This is confirmed by the walk test in big cities. However, compared to the test results collected on streets and drivable locations the success ratios for web browsing, video playouts as well as file up- and downloads are consistently worse in pedestrian areas and public buildings.
In smaller cities a slight decrease in success ratio and data rates can be observed while Telekom keeps its competitors at bay. However, the results show that even Telekom customers need to live with just half of the data rates they get in big cities. Anyway, all four operators slightly worsened over the course of time since last year’s test. Remarkably, O2 and Vodafone show about the same performance level in small cities and O2 outperforms Vodafone, although just by one point. E-Plus is clear bottom of the league although the networks shows significantly better web browsing performance than last year.
In 2015 Telekom clearly leads the field with respect to data service performance in German cities. Vodafone, and with some minor deductions also O2, provide good performance and overall high reliability. In small cities, O2 even catches up with Vodafone. Far behind on the fourth place for this discipline remains E-Plus.
O2 and E-Plus: On the way to a joint network
In the medium term O2 and E-Plus will merge into one single operator. First steps were already taken in 2015.
In mid-2013 it was announced that Telefónica would take over the network operator E-Plus. The European Commission gave its approval in June 2014 and the merger was officially announced 1st October 2014. After that it was clear that O2 and E-Plus would create a single, third mobile operator with one common network. Mid-2015 the two companies announced that they would sell to Telekom those mobile sites that had previously been used by both O2 and E-Plus. The complete integration of both networks is about to start in January 2016, and it has been announced that a combined LTE network will be ready by mid-2016. So far, E-Plus has held off building an LTE network on large scale.
As a first step towards a combined network O2 and E-Plus started so-called „national roaming“ in May 2015 – all customers of one network can also connect to the UMTS network of the other network. This option is limited to 3G-network only. It does not apply to either LTE (4G) or GSM (2G).
Observations in the test
As both providers still market their separate brands it was clear that for this year’s test O2 and E-Plus should still be treated as separate networks. In fact, the significantly different results emphasise the fact that customers are still dealing with two separate networks. However, it was observed that about 20%-30% of all evaluated voice calls were made via national roaming to the respective other network. Consequently, we have assigned such connections to the provider of the respective SIM card and the corresponding contract. The data tests were performed in LTE preferred mode. Here, a detailed analysis of the test data revealed that national roaming happened only rarely in practice.
Communication and data connection on the go are becoming more and more important – so what is the network performance on German streets?
To assess the network quality on the German transfer routes between cities, our two measurement cars drove about 4,500 kilometres. Here, the results confirmed what we saw in small and big cities: even between cities, Telekom performs best, with Vodafone on a close second place. O2 is already distinctly behind on place three, E-Plus even further on place four. This two-tiered result becomes very apparent e.g. when looking at the download success ratios: about 99% for Telekom and 96-98% for Vodafone while the percentage of successful downloads with O2 is at a much lower 85%, meaning that 15 out of 100 downloads are not successful. Even worse for E-Plus: depending on the download file size, only 73% to 80% success ratios; 20-27% of unsuccessful downloads are not great results on connecting roads in the age of the connected car. A similar picture for video: if your kids want to watch YouTube videos while driving, they (and you) are better off with choosing the Telekom network (98% success ratios with SD videos) or — with slight degradation– Vodafone (93%). Because quite often, there’s not much in terms of mobile entertainment in the O2 and E-Plus networks (82 and 74%, respectively).
Data performance on trains
The test teams also spent quite some time riding trains – not only on the high-speed ICE inter-city trains, but also on the regional connections of Deutsche Bahn and other carriers. Their learning: the trend seen for in-train voice calls was also observed for data connections. It was very noticeable that Telekom was distinctly ahead of their competition here as well, but also its network performance still has quite some room for improvement. But with regards to in-train data connections, Vodafone, O2 and E-Plus are clearly behind Telekom with only minor differences. One possible reason: the repeaters currently installed in trains – if deployed at all – only support GSM and the Telekom 1800 LTE band. In last year’s mobile network test, ‚connect‘ called Telekom with regards to their in-train network performance „almost like a one-eyed in the kingdom of the blind“. The situation has hardly improved in 2015 – and thus train and network operators should still work very hard to increase the mobile radio network performance in trains.
Overview: Strengths and weaknesses of the German operators.
For the fifth time in a row Telekom wins over the other German mobile operators.
After a decline to third place in 2010, Deutsche Telekom secured first place in the connect network test for the fifth time. The strong performance shows through all tested disciplines: Only in the category „Voice in cities“ Deutsche Telekom was barely outpaced by Vodafone. While Vodafone achieved this result with the cutting-edge VoLTE-technology, Telekom by contrast uses the older but proven circuit-switched technology – thereby achieving almost the same level. Deutsche Telekom is clearly ahead when it comes to data, whereas the lead in smaller cities and connecting routes is much clearer than in big cities. Also for train passengers, the Telekom network is currently the best choice – although, this success is at a rather modest level: The test winner shows room for improvement regarding mobile services on trains. Despite clearly tightened evaluation criteria, Telekom was able to hold its level in the overall ranking compared to 2014. Therefore, the Deutsche Telekom deserves congratulations.
very good (433 points)
A steady second place shows that, Vodafone is amongst Germany’s top networks.
Since Telekom deposed it from first place on the podium in 2011, Vodafone ranks on a stable second place among the German operators. This would not be possible without massive efforts regarding network expansion, which Vodafone has made for many years. The introduction of „Voice over LTE“ (VoLTE) into its network as well as the convincing implementation of this technology gives Vodafone a small head start in the voice discipline. But on the bottom line and because of clearly tightened evaluation criteria, Vodafone cannot escape unscathed. The improvement in the voice discipline compared to last year contrasts with a slight deterioration in data connections: Vodafone loses valuable points particularly in smaller cities and on trains. Also with data in big cities, Telekom is a little better. But the connect network test 2015 shows one thing very clearly: Germany currently has two top mobile operators. And the clear runner up is Vodafone.
good (396 points)
Despite the development of LTE and the introduction of VoLTE, O2 still couldn’t improve in 2015.
Although the integration between the networks of O2 and E-Plus is almost there, O2 parent company Telefónica has continued to invest in development of the O2 network, focusing particularly on LTE expansion in big cities as well as several smaller cities. Last year connect commented that speech quality would suffer during voice calls from the inevitable fall-back to 3G/2G („Circuit-Switched Fallback“). O2 quickly responded to this criticism by introducing the cutting-edge VoLTE technology in 2015. Unlike the competitor Vodafone, O2 does not yet benefit quite as much from the transition. Also „National Roaming“ for UMTS, which was offered in preparation for the network integration, does not have a great impact. O2 owes its significantly strong placement in the data discipline mainly to its LTE coverage. Overall, this leads to a stable third position this year.
We are now looking forward to the results of the interconnection of O2’s and E-Plus’ networks, which should really take off in 2016.
Selective improvement is not yet sufficient for E-Plus to avoid last place.
Though the merging with their new owner O2 was ahead, E-Plus had still started to build up its LTE-network in big cities in 2014. The network’s expansion continued in 2015, although the focus was still on UMTS. Through selective improvement, such as the clear reduction in loading times for websites, it is clear that E-Plus’ technical staff has not been inactive. Overall, E-Plus’ data-performance is slightly worse than last year. E-Plus often declares that it will focus on functions that are the most important for their customers. This may be true in the fields of voice calls and surfing – but whoever wants to transmit a greater amount of data or use streaming-platforms will not be perfectly served by this current network performance. „National Roaming“, also used by O2, is limited to UMTS and does not show any significant advantages in the test. So E-Plus customers will be interested to see whether and how the upcoming merger will improve their services.
The last few years saw a stable ranking at the top of the Austrian operators. In 2015 though, there is a surprise champion.
For several years now, everything has worked well in the Austrian mobile market. From their 3 high-quality network operators, the 8.5 Million inhabitants enjoy better value for money than those in Germany. In contrast to the German two-tier-society of mobile networks, A1 Telekom Austria, Hutchison Three Austria and T-Mobile Austria, three very good networks, compete closely for the top rank. Despite difficult topography, the Austrian operators stand out with excellent coverage and top-performance. Recent years were dominated by LTE roll-out in all 3 networks. A1 aims for full area coverage by the end of 2015, whilst Three had already reported this achievement in August. T-Mobile Austria currently serves 90 per cent of the population and plans to reach full coverage in 2016. A similar level of quality has been offered by these networks’ UMTS layer for some years. The customers reward these efforts; for quite some time, there are more SIM cards than population in Austria. At the end of 2014, the count was almost 13 Million. All this is also clearly visible in the three-country-competition, which connect has performed since 2009. Looking at the whole DACH region (Germany, Austria, Switzerland) the Austrian operators regularly finished among the top-runners, even though the overall lead changed several times. No Austrian provider ever received a verdict worse than “good”.
Again this year, the fight for the top position in Austria took place at the highest levels of quality. In the voice discipline, Three achieves the most visible improvement over last year. Also A1 and T-Mobile managed slight improvements compared to 2014. In that context, A1 relied on “Voice over LTE” (VoLTE) and appears to manage this new technology well. Telephony on the LTE network allows for shorter call setup times and best speech quality – provided you are in LTE coverage, have a SIM card that is enabled for the service and you own a VoLTE capable smart phone. Remarkably, Three managed to stay ahead on speech quality in the big cities, relying purely on the proven circuit-switched telephony. The walk test as well as the smaller city and connecting roads drive tests confirmed this result: With a minimal lead over its competitors, telephony scores best in the Hutchison Three network. When making calls on the train, all the alpine stars have their problems. Success ratios and Mean Opinion Score (MOS) values for speech quality drop significantly. But in contrast to the situation in German trains, all the Austrian operators end up on similar level.
In the increasingly important data segment, the three Austrian competitors are in a neck-and-neck race. Again it was Three that managed the most visible improvement compared to last year. With its data performance, it comes very close to the reigning top of the class, A1 – albeit not quite enough to overtake. Success ratios and loading times in both big and smaller cities are on a very high level for all three operators. A look at the achieved data rates reveals that even among such strong rivals, there are still performance differences:. They are part of the reason why T-Mobile slightly lags behind the leading duo of A1 and Three in the data discipline.
Data on connecting routes
A similar picture to that in the cities can be observed on the connecting routes between them. Again, Three and A1 are neck-and-neck in data performance, whereas T-Mobile is close on their heels by only a small distance. The same holds true on trains, even though the achieved success ratios and data speeds leave room for improvement on all three networks. That said, Three still achieved a remarkable improvement: from a clear third place last year and managed to catch up to the same level of performance as A1 in 2015.
Thanks to a significant improvement over its 2014 results, Hutchison manages to beat last year’s champion A1 to the line in a tight race. But also T-Mobile Austria proves its positioning as a premium provider and finished with an overall score only a fraction behind German winner Telekom. The biggest winners however are the Austrian customers since only here did all network operators improve over last year in the data discipline.
After a two-year break, Hutchison Three wins again in Austria.
After winning the connect network test in 2011 and 2012, Austrian operator Three had to give up first place to competitor A1 during the last years. But Hutchison technicians would apparently not stand for that, and their efforts were apparently worthwhile: In 2015, Three is on top of the podium again. Compared to last year, a significant increase was gained in the area of voice calls, There, Three manages to outstrip A1, which is very strong in this area. This also shows that the circuit-switched technology currently used by Three can still deliver top quality, whereas the VoLTE technology already introduced by A1 does not guarantee automatic success in voice services. The decision was made in the data-discipline. Here, although Hutchison still can’t reach previous year’s winner A1 in absolute numbers, the significance of its improvement compared to the 2014 results is so strong that the Hutchison-network eventually won overall. Whether in big or small cities or on transfer routes Three is equally strong as the market leader A1. Especially remarkable is the improvement made on trains. Though all of Austria’s three network operators show potential for improvement, last year’s third place in this category, Three, performed just as well as A1. The investment the operator has made in UMTS but mostly LTE-expansion was therefore definitely worth it. In 2015 Three achieved its best test result in Austria.
very good (457 points)
A1 Telekom Austria
Only pipped to the post by Hutchison Three, market leader A1 gains a very good second place in Austria.
It was an extremely close finish. A1 missed by only three points to this year’s winner Three. The tight outcome shows the high standard of the race being run by Austria’s mobile operators. Thus, the customers of Austria’s market leader can be sure of phoning and surfing in a top network. A1 accelerated in terms of LTE-expansion. By the end of this year, the operator wants to be able to provide a nearly nationwide 4G-network. Matching that, A1 is the first operator in Austria to offer modern „Voice-over-LTE“ (VoLTE) for phone calls. This partly contributes to the enhancement that was made in voice services last year. And this fact, unlike in the German market, also counts for data transmission as much in big and small cities as on transfer routes. Because A1 improved in this area, despite strong pressure from its competitors, the market leader defended its pole position in the data segment. Potential for improvement is mostly visible on trains. But this applies to all three Austrian operators. Even though A1 barely missed victory this time, the operator is once again certified with an exceedingly convincing performance by the connect network test 2015.
very good (454 points)
Third place in Austria is still a top position compared to the entire DACH region.
The fact that the third placed operator in Austria still achieves a „very good“, doesn’t by any means signify that this year’s assessment scheme is not rigorous enough. The height of the requirements set by connect and P3 communications is shown by looking at neighbour Germany. Plus P3’s analysis of measurement results from other countries underline the fact that the two alpine countries have an extraordinary high standard. Therefore, T-Mobile Austria although last placed in Austria still only misses a few points on the German test winner. There is a clear gap to Three and A1, which is even greater in the data-discipline than in voice-calls. But also T-Mobile Austria may consider itself as a top-provider by all means. The good result is not least owed to the committed expansion of LTE. By the end of 2015, according to its own figures, the operator reached 90% of the Austrian population with its 4G-network, with the remainder to be provided with LTE during 2016. Currently, the low success rates and mediocre data-rates cause the gap to the leading duo in Austria. Room for improvement for all three providers can be found in the network-performance on trains. But as a result of its placement in the three countries’ overall rating, customers of T-Mobile Austria can feel in good hands.
very good (427 points)
Despite ever growing competition, Swisscom was always the winner of previous benchmarks. Will it succeed again in 2015?
If you still need proof of the very high performance levels in the Swiss market, here it is. For the second time in a row all three Swiss competitors completed this benchmark with a top grade (“A”), despite the fact that the scoring schema was much tougher this year. Success like this does not come from nowhere but is the result of continuous investment into the networks. Market leader Swisscom today reaches about 98% of the Swiss population with LTE, competitor Sunrise aims to reach 95% by the end of the year, and “Salt”, the third Swiss operator, aims for 94%. After the takeover of the operator formerly known as “ORANGE” by the French investor NJJ Capital the third operator had to change its name due to brand licensing issues.
The extensive coverage of Switzerland is supplemented by very attractive offerings. After Swisscom launched its “Infinity” tariffs in 2012, the first with truly unlimited data usage, Sunrise and Salt followed quickly. German customers can only dream about something similar, and only T-Mobile has a similar product in Austria.
This leaves us with the open question of network quality in the highly acclaimed wireless offerings in Switzerland. In fact it’s extremely good, as the drive and walk tests in larger and smaller cities and their connecting roads prove. The biggest improvement regarding voice calls compared to last year was seen with Swisscom. This is clearly due to the launch of VoLTE, which delivers extremely fast call setup times and highest call quality to the customers of the Swiss market leader. But in this discipline even Swisscom is outperformed by Sunrise that is not only top in Switzerland but in the whole DACH region, and delivering this with the “legacy” circuit switched voice technology. Salt trails the field and does not offer VoLTE yet either. But both competitors are already well ahead in their VoLTE planning. Also impressive was the performance of voice calls on Swiss railroads, even though there is a small degradation compared to cities and roads. But the performance is significantly higher than in Germany or Austria, and about the same for all three competitors. This is a showcase for railroad networks in neighbouring countries.
Mobile Internet in the cities is also top notch in Switzerland. Good coverage in the bigger cities results in a head-to-head race between Swisscom and Salt, trailed by Sunrise. Also including walk-test results in the smaller cities results in a razor-thin advantage for Swisscom. Compared to last year, the biggest performance leap in this category was seen with Salt. Not quite enough to overtake Swisscom, but nonetheless commendable. One small surprise when comparing the results to the previous years was that whereas the Swiss always won the crown in this discipline, this year the Austrian operators A1 and Three outperformed the Swiss by just a notch.
Data on Connecting Roads
On the connecting roads between the cities the head-to-head race seen between Swisscom and Salt was repeated. Swiss market leader Swisscom won with a small lead and Sunrise followed just behind in third place. Overall, Swiss drivers enjoy good and reliable network performance, and this is even more the case for train users. As with voice, data performance for all three operators in Switzerland is top notch. While all three were similar for in-train voice performance, there is a small advantage for Swisscom and Salt compared to Sunrise for mobile data in trains.
… and the winner is: Swisscom – in Switzerland as well as in the Three-Country-Benchmark with Germany and Austria. Sunrise is a strong second, with slight improvement compared to last year. Even with this top performing competition in Switzerland Salt achieves the biggest performance gain, being a very good third.
The 2015 Switzerland test victory is the seventh in a row for Swisscom.
Seven wins in a row tell their own tale. Despite high network rollout speed and aggressive offers by its competitors, the Swiss market leader was able to keep its top-position again in 2015. This is not an easy achievement and requires high investments and significant effort. The main driver for the excellent test result is the voice service improvement. Swisscom benefits from its well-managed voice technology VoLTE, which is currently only offered by them in Switzerland. However, Sunrise was able to achieve a slightly better score in this section and the test victory was finally secured by Swisscom’s high quality data performance. Although strong rival Salt presents itself as a tough competitor in this section, Swisscom collected decisive scores for overall victory in big cities, small cities and on connecting roads. In doing so, Swisscom managed to maintain the high level of last year’s test results. But even in those areas, where the competitors are catching up – e.g. voice services in trains or mobile services in public buildings – Swisscom still maintains its overall leadership. Customers of this operator can be sure to get the best of the best mobile services. Congratulations for the 2015 victory in Switzerland and for the highest score in the three countries.
very good (458 points)
Sunrise finishes again with the best voice result and a very good data performance
In the context of the very high Swiss performance level even the stable second place of Sunrise represents an excellent result. Such a position would not be achievable without continuous network expansion and enhancements. Sunrise delivers the absolute best voice service result – in Switzerland as well as in the three-country comparison. This was not achieved with the new VoLTE technology, but with the legacy circuit switched voice service. The lead in voice is the main contributor for the second place in Switzerland. Notwithstanding its high performances in mobile internet, Sunrise only achieved third place in this category. In big cities, small cities and on connecting roads the data services of the competitors perform slightly better. This is particularly noticeable in the slightly lower average data rates. The reliability is still premium class. Only in trains some smaller degradation is visible compared to Swisscom and Salt. But German and Austrian train travellers can only dream of a data performance on the level of Sunrise. Therefore Sunrise customers can be happy about the well-deserved 2015 test verdict “very good” and the top position of their operator in the DACH-region.
very good (448 points)
The third ranked Swiss operator improved significantly and achieved second position in data services.
The rebranded third Swiss network operator again provides its customers with services at a high level of performance. The test results clearly reveal Salt’s improvements in voice as well as in data services compared with last year. Although Salt’s voice performance lies somewhat behind its competitors, it is worth bearing in mind the high level at which this race is being executed, and that Salt achieves the same level in voice as the market leaders Deutsche Telekom and Vodafone in Germany. The highest improvement versus 2014 was achieved by Salt in the data section. Even if it is insufficient to overtake the test winner Swisscom in this category or in total, it is sufficient for Salt to achieve second position in data in Switzerland. The distance to the test winner is very close in big cities and on connecting roads. In trains it operates on a level playing field. In the remaining data categories like small cities or indoor walk tests, Salt achieves at least the same level as the second ranked operator Sunrise. Thus Salt customers experience mobile services at a very good level. To better understand this high performance, it is worth having a look at the total score board. The score of the third place in Switzerland is clearly higher than the one of the German test winner Deutsche Telekom.
very good (441 points)
As in previous years connect’s partner in the network measurements, P3 communicactions, used two vehicles to test drive the chosen cities and highways. In Germany each car carried 8 Samsung Galaxy S5 smartphones to measures voice services and four Samsung Galaxy Note 4 performing the data service tests. This setup reflects 4 network operators in Germany. In Austria and Switzerland the vehicles were equipped with six Galaxy S5 and three Note 4 to reflect three competing networks in these countries. The same setup of devices was used in the walk tests. For this effort, the smartphones were installed in trolleys and backpacks with additional batteries.
The devices’ firmware was each operator’s current firmware version. If such software was not available the most current firmware from Samsung was used.
Voice services were measured with the Samsung Galaxy S5 performing calls alternating between the two measurement cars (mobile-to-mobile). Background data traffic was present to one of the parties in each call to reflect a realistic smartphone usage scenario. Audio quality was assessed by using POLQA (Perceptual Objective Listening Quality Assessment) wide band scoring, suitable for HD voice. All devices were configured in “LTE preferred” mode. For Vodafone, o2, A1 and Swisscom the new Voice-over-LTE (VoLTE) service could be used. Within networks currently not supporting VoLTE, the smartphones were forced to switch to 3G- or 2G-technology, the so-called circuit-switched-fall-back (CSFB).
To assess cellular data performance a sequence of tests were performed. As a dynamic web-browsing test, each country’s top web sites (according to Alexa ranking) were downloaded in the so-called live web-browsing test. Additionally a static web site was tested, the industry standard ETSI (European Telecommunications Standards Institute) reference page. HTTP down- and uploads were performed with 3 MB and 1 MB files, simulating small file transfers. Networks’ peak performance was tested with a ten second down- and upload of a single, very large file. Simulating further user behaviour, YouTube tests were executed for Standard Definition (SD, 360p, 2.7 MB, 30 sec.) and High Definition (HD, 720p, 11.9 MB, 30 sec.) videos.
P3’s “Antenuatr” attenuation enclosure was used for the measurements, so the devices had comparable radio conditions to in-car and indoor scenarios. The reason for this new attenuation solution is the sophisticated multi-antenna setup of modern smartphones. Today’s devices are tuned for best performance with internal antennas and lack sufficient external antenna connectors in most cases.
By foot: Indoor and train measurements
The so-called walk test consisted of the same tests as were performed in the cars. For this effort two teams measured in public transport and in public places, like coffee shops, museums, train stations and airport terminals. Travelling from city to city by public transport allowed the assessment of cellular network quality within the local railway’s trains.
Tests were performed in Austria, Germany and Switzerland around the same period of time (Germany: October 7 – 23; Austria: October 7 -23; Switzerland: October 7 -24). All measurements were done between 8 AM and 10 PM. Both cars were always in the same large cities, but on different routes to avoid any interference of one car’s measurement by the other car’s. Both vehicles followed a given route, including fixed location measurements at “areas of interest” such as well-visited public places. Measurements there lasted one hour. Locations such as train stations, airports, much-frequented public parks or high-density urban areas typically demonstrate how networks respond when a high number of users “fight” for their share of bandwidth within the network’s available radio frequencies.
The measurements included 13 larger cities of more than 100,000 inhabitants in Germany. In Switzerland the route included 12 bigger cities and in Austria five cities. Additionally several smaller cities and towns were included in the route planning. For these tests the measurement vehicles visited two smaller cities at the same time, enlarging the covered area of the connect test. Travel between the cities mainly used highways, but smaller state- and county roads were driven as well.
For each connect test P3 communications followed a well-defined process to generate four independent and representative city and route plans. The connect editors choose randomly one of these four alternatives.
Overall 20,000 km were driven for the connect test in 2015. In Germany the 10,000 km of driven routes alongside the cities and areas visited represent 12.6 million inhabitants, equalling around 15.7% of Germany’s population. Austria was measured by driving 4,700 km covering 3.1 million inhabitants (approx. 36.2% of the Austrian population). The test teams drove 5,200 km within Switzerland, covering 1.9 million people representing around 23.5% of the Swiss population.
The result of the connect test is based upon more than 8,500 voice calls including more than 60,000 speech samples, 20,000 downloaded websites (live pages) and more than 10,000 file transfers via HTTP per network operator.
Certainly a huge effort, but necessary to gain the required statistical relevance and confidence in the test results.
FAIRNESS AND TRANSPARENCY
In order to achieve an unbiased comparison, P3 communications and connect assure fair and transparent implementation of the network tests.
The design of the connect network test is prepared jointly by specialists from P3 communications and editors of connect. In January 2015 an initial meeting between P3 communications and the responsible individuals from connect took place in order to review last year´s tests and align the basics for the network test in 2015. This covered schedule, test criteria and evaluation scheme, as well as the test devices and many more details.
After these internal alignments completed by about May 2015, an information presentation was distributed to the CTOs of the network operators together with a request for feedback. This step was intended to avoid, for example, running the network test at a point in time where central network elements are being modernized. Furthermore the selection of the test devices should not impact any of the networks under test. So the technical specialists from all candidates had an opportunity to provide input about the test methodology since they have excellent expertise in that area. P3 communications and connect appreciated these fruitful discussions in advance of the tests. Full ownership of the process and evaluation remained with connect throughout.
The input was consolidated into a second and final information presentation, which was distributed during early August to the operator CTOs. Further information shared with the network operators addressed the exact start and stop date of the tests so that they were not influenced by significant network changes, as well as pre-reading of the test results a few days before publication to allow the network operators to prepare their communications.
OVERALL CONCLUSION: MOBILE NETWORK TEST
Hannes Rügheimer, connect author
All mobile network operators feel the mobile data boom – they can hardly manage to keep up with the pace of their network rollout. Thus, it is quite an achievement when a provider like German winner Telekom can equal last year’s results. The pressure seems even harder on Vodafone who, thanks to VoLTE, offer top voice call performance but lose points on data performance. O2 and E-Plus clearly fall back behind the top duo. Let’s see if the integrated offering of those two networks can close the gap and how quickly it can rise into premium class. How close competition between three almost equally strong operators can look like, is impressively demonstrated by the Austrian and Swiss markets. All networks tested in both countries receive the “very good” verdict, not because of slack scoring standards but thanks to an extremely high level of performance. In Austria, Three managed to get ahead of last year’s winner A1 in a tight neck-and-neck race. Swiss market leader Swisscom snatched victory in Switzerland for the 6th time in a row – despite high pressure from Sunrise with its convincing voice results and the strong data results from competitor Salt (formerly Orange).
Mobile network test Germany, Austria and Switzerland final results: