Google’s services went offline for many users for nearly a half-hour on the evening of November 5, thanks to an erroneous routing message broadcast by Moratel, an Indonesian telecommunications company. The outage might have lasted even longer if it hadn’t been spotted by a network engineer at CloudFlare who had a friend in a position to fix the problem.

The root cause of the outage was a configuration change to routers by Moratel, apparently intended to block access to Google’s services from within Indonesia. The changes used the Border Gateway Protocol to “advertise” fake routes to Google servers, shunting traffic off to nowhere. But because of a misconfiguration, the BGP advertisements “leaked” through a peering connection in Singapore and spread to the wider Internet through Moratel’s connection to the network of Hong Kong-based backbone provider PCCW. Google was interrupted in a similar way in 2008, when Pakistan Telecom moved to block access to YouTube in Pakistan because of an order from the Pakistani government.

Tom Paskea, a networking engineer at the content distribution network and Web security provider Cloudflare, spotted the source of the outage. “When I figured out the problem,” Paseka wrote in CloudFlare’s blog this morning, “I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google’s services came back online.”

Read 2 remaining paragraphs | Comments

via Ars Technica » Technology Lab