ISPs in Philippines - the real facts and why you should care

sarisari

Newbie Gamer
Jun 20, 2018
18
0
1
tl;dr version: (not in order) Ping is not reliable, PLDT sucks, Garena needs to spend more on connectivity, use WinMTR instead of tracert, government needs public IX.

Hi all, I'm a foreigner living in Manila that has spent the last 15 years building large scale networks in Europe, with a heavy focus on northern Europe. I apologize in advance that I do not speak Tagalog (yet) so I'm only able to write this in English.

I've read a lot of posts here on the forums, and although they are mostly well meant, most are factually wrong. I contribute this to lack of knowledge in terms of carrier grade networking so it's completely understandable so let's start by defining what lag (latency) is. Also this is focused at our specific situation - playing LoL in Philippines.

For a game such as LoL you have 4 layers of possible latency:

1) Consumer end (PC/Router/Bandwidth)
2) ISP
3) Interconnect/Transit
4) Server end (Servers, load balancers etc.)

To measure latency end-to-end ICMP echo (PING) packets are not a reliable tool. There are many reasons for this but for us the main reason is that ICMP is a down prioritized packet type. Historically the ping tool was meant to see if a host was alive, not to measure connectivity - the reason it does have response times were to identify possible routing errors or DNS IP mixups. Remember this tool is older than most people playing LoL.

On Windows 'tracert' uses ICMP echo requests to probe the route of the connection to your destination. If you do want to trace the route using ICMP echo I can recommend WinMTR as a replacement for tracert that is a lot faster and more accurate. WinMTR can be found here. The UNIX version has an option to do UDP based packets which is more relevant for us as UDP is usually used in online gaming because of reduced latency and overhead as compared to TCP.

In Europe, the infrastructure and government laws are usually very good and the laws enforced. This means that the most common causes of latency is on the consumer end. People mostly have ADSL connections and forget they have something running in the background (seeding is the main culprit in most cases) as with ADSL when you saturate your uplink you obliterate your downlink. But here in the Philippines, there are a lot of other causes for latency for us gamers - the worst being PLDT - yes you read that right, PLDT is intentionally disrupting online gaming in the Philippines. I will explain in greater detail below.

PLDT (and SMART) are actively fighting to interconnect laws here in the Philippines. This is done mostly because of the mobile market but the internet (and thus gaming) suffers as a result. Because SMART (and with the purchase of Sun) has the dominant market share, they are not interested in interconnecting with other Telcos, but specifically they do not want Globe subscribers to be satisfied with Globe so they will switch to SMART - this is why they are limiting the interconnect between PLDT and Globe, they are essentially throttling the data exchange between the companies. This is against the law here in the Philippines but as with a lot of other things here, money talks - morale walks. I assume you've experienced long delays on texting between Globe/SMART?

Here is a recent article on the problem. Globe’s interconnection woes
One of the most common complaints one hears in the past few months from subscribers of Globe is the alleged poor service of the Zobel-Singapore Telecom controlled telecom company.

But if you listen closely to what the Globe subscribers are griping about, you would find that their principal problem is the fact that text messages they send are received hours and sometimes even a day later or not at all. They also grumble about difficulties in making calls.

If you point out to disgruntled Globe subscriber that their problem is only with calls and text messages sent to the subscribers of the PLDT-controlled network–Smart and Sun – they begin to realize that the problem is not with Globe Telecom. Rather, the problem is rooted in the long-running interconnection problem of Globe with the former telecom monopoly.

The fact is that Globe Telecom service has been improving immensely as it nears completion of its massive $800-million network modernization program. The improved service, however, as far as the public is concerned, is negated by the continuing problem of delayed transmission of text messages and the difficulty of connecting to non-Globe networks.

The government agency that regulates the telecom industry, the National Telecommunications Commission (NTC), has given both Smart and Globe passing marks in its latest mobile network benchmarking for the parameters set by the commission.

The two major telecom networks were given passing marks in five parameters: blocked calls, dropped call rate, signal strength, signal quality and call set-up time.

So it would appear that the source of the problem of disgruntled Globe subscribers is not so much the services of Globe but rather the interconnection between Globe and the PLDT-controlled networks.

The issue came to a head when Globe Telecom, in a related issue, cited the long history of the refusal of PLDT to fully interconnect with it despite numerous orders from the NTC and court decisions ordering them to do so.

Globe head of National Carrier Relations Division Melvin Santos has in fact written a letter to NTC Commissioner Gamaliel Cordoba asking NTC to compel PLDT to fulfill interconnection commitments to Globe, particularly in the provinces.

In his letter, Santos said: “Local interconnection with PLDT has been pending for years now despite orders from the NTC and public inconvenience.” He stressed in his letter that only 11 out of the 32 candidate areas for interconnection have been accommodated by PLDT for activation.

The interconnection issue has been long the subject of legal dispute between Globe and PLDT as well as PLDT and other telecommunications companies.

Globe legal counsel Rogelio Salalima cited the Supreme Court cases Republic of the Philippines vs. Republic Telephone Co. in 1996 and Republic of the Philippines vs. Express Telecom Co. in 2002 as examples of PLDT either refusing interconnectivity agreements outright or connecting with them but using outlying linkages to block calls.

Salalima pointed out that in the Extelcom case, despite NTC’s order that PLDT interconnects with Extelcom, PLDT opposed the issuance of Extelcom’s license and claimed that the NTC abused its authority in mandating interconnection. Even after the Supreme Court ruled against and affirmed the NTC’s jurisdiction and mandate, the interconnection between PLDT and Extelcom did not happen.

PLDT/Smart Communications, of course, denies that there is a serious interconnection problem with Globe. PLDT/Smart Public Affairs head Mon Isberto cited the January 16, 2013 incident where Sun’s engineers detected a sudden rise of text messages that were being delayed or failing to arrive. The Sun engineers found no problem in the Sun-Globe interconnection and they alerted their Globe counterparts who said they will look into the problem which, instead of improving, worsened.

Isberto said that Smart and Sun have separate interconnection links with Globe and neither was having technical problems at that time. He also pointed out that SMS traffic within Sun and Smart and between the two PLDT controlled networks had no interruption at that time.

Of course, Globe spokesmen could have also pointed out that there is no problem in the transmission of text messages within the Globe network and the fact is it is the Globe interconnection with Smart and Sun that was and continues to be problematic.

Interconnection problem is not unique in the Philippines. This is a problem in many countries which have made a fundamental shift from monopolistic to a multi-operator environment,

The monopoly, or in the Philippine case the telecom oligopoly, will do anything and everything to stunt the growth of its competitors and one way that this can be done is to make interconnection difficult.

Much depends on NTC to make sure that the dominant telecom company does not suffocate its emerging rival.

Keep in mind that that interconnection is standard in other markets and I've never come across any other Telco being as manipulative towards their customers as PLDT/SMART.

So what does SMS delays have to do with LoL latency you ask? Well, the biggest problem here is that by default PLDT routes all domestic traffic via an IX in HK. This means if you send data between a PLDT DSL subscriber and any other major ISP here it goes via a Hong Kong Internet eXchange. There is a law passed in the Philippines that all ISPs have to interconnect freely via an Internet eXchange so data can flow freely between the customers of the ISPs. Mind you this does not mean that a Globe/Sky/Bayan/Eastern customer can access data/servers outside PLDTs network, through PLDTs network - this is entirely between customers of the ISPs. This is common practice all over the world, even in dictatorships - except in the Philippines.

ISPs buy transit traffic through backbone providers like PACNET, Level3, UUNet etc. - these are the services that connect your ISP to the world, it's too expensive for all ISPs to run their own international backbone fibers.

Because of PLDTs enormous power they have been able to defy the laws of interconnection and because of this keep prices up and bandwidth down as a whole in the Philippines. I'm not saying that the other Telcos here are angels but at least their obey the law of interconnection. Doesn't it seem odd that the Philippines is so far behind in terms of bandwidth/price for internet? It's literally 10 years behind Europe in its current state.

It's worth noting that Globe has a stake in the worlds most powerful sea fiber. This is why PLDT is scared that if they lose their stranglehold on the consumers here that they will realize that the competitor offers a superior product. Unfortunately, Globe has some of the least educated support staff/customer services in the Philippines so it's not all roses on the 'other side'.

I can give you a concrete example of how full PLDT is of themselves - I was involved in a project in Philippines (which is when I fell in love with the country and moved here afterwards) that also involved PACNET. I was dumbfounded why the routing in the Philippines was all over the place, I was trying to understand whom or what was the cause of this terrible infrastructure so I could fix it, eventually I narrowed it down to PLDT being incompetent. As I met with the senior technicians from PACNET regarding a multi-gigabit transit setup here in Manila I asked them why PLDTs routing was all over the place, as they are the major operator it was important for us to have good connectivity with PLDT. The PACNET technician and the senior sales manager explained to me that of all the countries they do business in, Philippines was the only place where a Telco willing and carelessly broke conventional international norms. PACNET had been dumbfounded when PLDT asked PACNET to pay PLDT when PLDT wanted to buy backbone capacity from PACNET. Their reasoning was that without PLDT you could not service the general public in the Philippines properly so that's why PLDT should get money for allowing their customers' direct access to PACNET. I'll let that sink in a little.

PACNET is a company who's main business is SELLING backbone capacity across South East Asia and PLDT wanted PACNET to pay so PLDT could get backbone capacity from PACNET. It is so stupid that it completely blows my mind.

Everything else they explain perfectly matched what I had seen examining the Philippines' internet infrastructure. Unfortunately to remedy this situation you need non-corrupt representatives that also understand technology or the need for improving the infrastructure as a whole in the Philippines - not being able to vote here was a relief as I doubt there are any.

To summarize:

Garena needs to supply us with an IP where we can do UDP based pings to have reliable connectivity diagnostics.

If Garena is relying solely on the standard connectivity at VITRO (ePLDT), then they need to invest in their own connectivity to Globe and Bayan/Sky.

The Filipino government needs to create a unified IX, that mandates you have to interconnect a certain bandwidth pr. subscriber with a minimum bandwidth of 1 Gigabit. Make it so that you lose the right to do Telco services if you are not connected to the public IX. (this would solve all connectivity issues for someone like Garena at zero cost to Garena and all other companies doing online services in the Philippines).

Feel free to ask any questions - I have a lot of knowledge in this field, and I've skipped some parts of it here as my post would end up too lengthy if it isn't already too long - in fact, I'll put a tl;dr in the top

Unfortunately, the tmcnet article has the same issue most articles on the subject has - lack of technological understanding.

On a national IX you announce your AS number (List of PH AS numbers here: http://bgp.he.net/country/PH) so other people can route traffic to those IPs via the IX. This happens automatically and the most common way of doing this is via BGP (https://en.wikipedia.org/wiki/Border_Gateway_Protocol).

Your IP ranges/subnets are assigned to your AS so when you announce you're AS via BGP the others listening will know they need to send the traffic bound for your IPs via the shortest route, in this case the national IX. IPs are assigned by the regional authorities, here in Asia it's APNIC - as an example, we can look up the IP of this forum:

Open: http://wq.apnic.net/apnic-bin/whois.pl
Search for: 125.5.3.65

You will see that it belongs to Garena and that Garena has their own AS number and that it's maintained by ePLDT/VITRO. In fact it looks like PLDT was the ones who set this up for Garena when they applied for this class c subnet 125.5.3.0 - 125.5.3.255 (125.5.3.0/24 in proper terms).

To further investigate it's easier to use my UNIX based utilities but I'll try to find web-based ones to make it easier to relay. Using a site like robtex we get this interesting information regarding Garena's AS number: http://as.robtex.com/as24071.html#peer

As you can see there those are the 3 peerings Garena has currently - Infocom (PLDT), Eastern Telecom and ePLDT/VITRO. But how Garena has setup routing is a best guess scenario only without actually doing the research from all the different ISPs. My guess is that customers outside PLDT are routed via Eastern, as Eastern peers with all the local ISPs (personal note: Eastern is a very good company, especially after the San Miguel investment in their nextgen infrastructure). This gives us the following:

LoL player using PLDT -> Garena
LoL player using Globe -> Eastern -> Garena
LoL player using Bayan -> Eastern -> Garena
LoL player using Skycable -> Bayan -> Eastern -> Garena

Well you get the picture - using Eastern as their transit for Skycable/Bayan and other smaller ISPs is the best economic solution - but having a direct peering with Bayan and Skycable would be even better.

In a perfect world, PLDT, Globe, Bayan, Skycable, Eastern etc. all had their own direct fiber to an IX so when Garena connects to the IX they would get direct transit to all their customers/players. This way the playing field is even and the ISPs can actually start competing on quality and service.

So I just did a test with SkyBroadband and they actually have a direct ePLDT peering.

LoL player using SkyBroadband -> ePLDT -> Garena

This means that the horrible latency between SkyBroadband users and Garena shouldn't be there - and based on personal experience I'll put that on Sky as the overall service and quality the last 6 months have been terrible.

I've also conducted a test via Eastern DSL and that is the best I've seen so far - average 10-15ms latency to Garena with a "spike" of 19ms. Not like Sky where the average latency is much higher and spikes of 200 with bonus DC's all the time.
 
Reply