As noticed by some in the Internet network operations community, at the beginning of May an odd event occurred as ICANN ended DNS service on the IP address formerly associated with L.ROOT-SERVERS.NET (“L-root”). Specifically, as ICANN turned off the DNS service at the address formerly used by the L-root, 198.32.64.12 (and the routing announcement by ICANN for 198.32.64.0/24), DNS root queries sent to that address instead of the new L-root address (199.7.83.42) continued to be answered.
At ICANN and for those who were aware of the situation this raised some eyebrows. Immediately after we turned off the old DNS server and noticed queries for the old address kept getting answers, we began looking into the issue by checking the DNS responses, reviewing routing system logs (including asking folks at Renesys and others for assistance), doing traceroutes from various points in the Internet, etc. Our investigation determined that the answers being provided by the machines now answering for the old L-root address were correct, returning proper referrals and answers including returning the correct address (199.7.83.42) for L-root, and as such, no immediate damage was being done. However, those answers should not have been returned period.
Why did the old L-root address keep answering?
As is well known, there are 13 root server IP addresses (and many DNS servers behind those addresses), referenced by a single letter (A through M) in the domain ROOT-SERVERS.NET. ICANN has operated L-root since 1997 or so. As part of an ongoing effort to improve L-root service, and after an extensive community consultation process, ICANN obtained a “critical infrastructure” block of addresses from ARIN and renumbered L-root out of a block that was originally intended for use at exchange points into the new block. Back in October 2007, ICANN announced this renumbering and continued to provide root name service on the old address for L-root while turning up a much beefier service on the new address. As part of the announced renumbering process, ICANN continued to answer DNS queries on the old address for 6 months, at which time root DNS service on 198.32.64.12 would cease.
On May 2, the 6 month clock expired, and we turned off root DNS service at the old address as planned. To our surprise, however, the registrant for the Exchange Point block, Bill Manning of “EP.NET, LLC.” (who under some relationship with the University of Southern California, Information Sciences Institute helps operate B.ROOT-SERVERS.NET) had entered into an agreement with a DNS hosting provider, CommunityDNS, to continue answering DNS root queries for the old L-root address.
As the folks at Renesys note, Bill Manning, as the current registrant for the superblock out of which for historical reasons the old L-root address was numbered, has the ability to re-provision the 198.32.64.0/24 address space as part of the Exchange Point block of address space. However, the continued availability of DNS root service on a “decommissioned” root server address violates the “Principle of Least Surprise”, at least for those who take a keen interest in the DNS infrastructure such as ICANN. Historically, when a root server has been renumbered, the root server operator maintains a DNS server at the old address for some period of time and then turns off service, with the address not being put back into use. The fact that queries to the old L-root address continued to be answered and ICANN, the operator of the L-root server, was unaware of agreements to continue service by a third party raises some significant areas of concern, not least of which is what should be done with IP addresses formerly used by root servers and who has the authority to make those sorts of decisions.
What did ICANN do?
After some discussion with the relevant parties, the DNS root service being provided at the old L-root address was discontinued. While undoubtedly a number of DNS queries were (and are) still being sent to the old L-root IP address, it is important to remember that the nature of the queries. When a typical DNS resolver first starts up, it uses information called the “root hints” as a list of known IP addresses for root servers. The DNS resolver will then ask one of those IP addresses (typically chosen at random) for an updated list, and then use that updated list from then on. This initial query is known as a “priming query” and only a fraction (1/13th to be precise) of priming queries from resolvers that had not updated their root hints file would have been affected — any queries following that priming query would not be affected since the answer to the priming query returns the correct root server address list.
ICANN has also been monitoring the results returned by these IP addresses through the entire time it was advertised, and believes it was always providing accurate root responses throughout its existence. ICANN continues to work with the root server operator community to improve monitoring and analysis of the root server system, aiming to ensure the continued security and stability of this critical component of the Internet.
So what does this mean for the Internet?
This incident highlights firstly how robust the DNS system is, that it continued to work flawlessly despite this issue. Given the existence of 13 root server IP addresses, only 1/13th of DNS resolvers that have not updated their root hints would have sent a priming query to the IP address formerly associated with the L-root server. However this incident also highlights how important it is for system administrators and DNS software developers to update their root hints as it becomes necessary. Thankfully, this usually only happens once every few years, and IANA sends announcements to the network operator communities when this information is updated.
ICANN, though its Root Server System Advisory Committee (RSSAC), is bringing some of the questions and issues raised by this incident to the root server community, who undoubtedly will want to consider this incident carefully.
