IPv6 in the Real World: Practical Multihoming
Late in 2010, it was finally time to multihome my network for IPv6. The Academ Network had been multihomed on IPv4 for a long time, but due to the lack of IPv6 transit support from most US providers (and the support is still limited), it had been hard for me to predict when I would be able to do the same over IPv6.
Identify Providers and Determine Their Technical Requirements
NTT America has provided IPv6 transit for my network (AS4050) for many years. It has been a good and reliable service. Because I wanted to diversely multihome, I needed to identify a second provider of IPv6 transit. After researching the providers available that met my technical needs, I selected Level(3). They were able to provide IPv6 transit on a schedule that allowed me to be operational shortly after Thanksgiving, 2010 and before the end of the year. When I talked with them about the requirements to multihome IPv6, I was told that I needed to advertise provider independent (PI) IPv6 addresses.
Contact the Regional Internet Registry for IPv6 Space
I contacted ARIN and they provided the PI IPv6 space to me at one-time cost of $1250 and a subsequent $100/year fee for maintenance. This is nothing new for those that have been using space allocated by ARIN, but it is new to me because my other Internet number resources (AS number and IPv4 space) was assigned to me before the RIRs came into existence. Being a one-person consultancy (that’s what Academ Consulting Services is), $1250 is not a small expense, but it is a required one to make this all work, so I signed the ARIN agreement and paid the money. ARIN issued me PI IPv6 space 2620:82::/48 I now use to route over both NTT and Level(3).
Determine Routing Policy and Document It
NTT America uses the IRR to track the routes it needs to accept from customers and advertise to peers. Early in 2010, NTT started supporting the use of the route6 RPSL objects documented in RFC 4012. To insure that NTT America would accept advertising for the new space, I submitted a route6 RPSL object to my routing registry (I use ALTDB) and waited for NTT America to pick the new route up(about 24 hours later). Fortunately, NTT America sends an email anytime they update their route filters related to a specific customer AS and I received notice I expected from them that the update is completed the next day.
Assign Addresses to Hosts, Renumber if Necessary
I needed a renumbering plan for my LAN so I could start using the new IPv6 space. I choose to duplicate the plan I have for the current IPv6 space I have from NTT and follow the renumbering recommendations outlined in RFC 4192. As a result of choosing this strategy, the change was relatively painless for users of the network and the resources hosted on it.
Configure Edge Router to Advertise Routing Information to One Provider
Next, I had to reconfigure my router to advertise the route and assigned an address from the new space to the router interface on the LAN segment. Both of these are simple tasks and I had them in place in a few minutes. The NTT America NOC confirmed they were now seeing my new route and I was able to start the renumbering process for the hosts on my LAN.
Setup DNS
First, DNS for the reverse lookup had to be configured. Since I already had set up reverse for the space originally allocated to me by NTT America for the current single homed deployment, I did the following work on the primary DNS server for the reverse zone: I duplicated the zone file for the old space, changed the origin to match the new addresses, updated the named.conf file and restarted that name server software. Then, I updated the named.conf on the secondary servers so they will properly slave the content from the master (primary) server for the reverse lookup of the new IPv6 space. All this was completed in a few minutes and I was able to verify that reverse lookups are properly working. I activated this immediately because reverse lookups are most often used to find the hostname of a device using the address.
Add Addresses to Hosts
To lessen the impact of the renumbering, I decided to add the new addresses to each host first and then wait a few weeks before stopping the route advertisement of the old numbers (those assigned to me by NTT America as part of the original IPv6 single-homed deployment). I was also adding the new AAAA records for these hosts only after I am sure that they are properly reachable on the new space. Essentially, this allows the host to remain reachable on the old space until it is clear to me that IPv6 traffic for the host is primarily coming through lookups for the new numbers. This is basically the same technique that should be use for any renumbering activity if it is important for the host to remain reachable during the transition.
Test Accessibility
With all this in place, I was able to test to see if these sites were accessible on the new address space via NTT America by verifying reachability from various points around the IPv6-enabled Internet. A great site for such testing is the IPv6 test site http://test-ipv6.com/. Another is the IPv6 Forum test page http://www.ipv6forum.com/test_ipv6.php. From the host you want to test for reachability, you can determine (via a browser to that URL) what IPv6 address (on the test host) you are using to reach either of these sites. The first site will also test if DNS is working properly. Once I was satisfied that the routing for the new addresses is working in the same single-homed manner routing for the old addresses, I was able to move on to the next step: getting the second IPv6 transit connection up and routing.
Configure Edge Router to Advertise Routing Information to Second Provider
Level(3) showed a few rough edges in their integration of IPv6 support into their standard provisioning process, but thanks to the watchful eye of the Level(3) sales engineer assigned to my account, those issues were worked through pretty quickly. Level(3) had the service properly operational after slightly more than a month from start to finish. Because of the work I had done with the existing connection to NTT America and Level(3)’s standard support for using the IRR, the key things I needed to do to commence using the service from Level(3) involved updating the aut-num RPSL object to reflect that AS4050 is exchanging routing with AS3561 and properly setting up the new BGP peers with Level(3) taking care to insure that there was one configured for IPv4 (using IPv4 addresses) and another for IPv6 (using IPv6 addresses). For those of you involved in multihoming, most of these steps look pretty routine. The only new wrinkle is the need for a second BGP session for the IPv6 routes. This is the most common approach for a dual-stack connection although technically, it is possible to have one BGP session handle both sets of routes.
Multihoming is Active
The Academ Network has been running without major incident multihomed via IPv6 over NTT America and Level(3) for a couple of months as this posting is being written and has been very stable. The only outages that have taken the Academ Network off the Internet have been due to planned work. There are four steps left to implement. The first one is to remove the old IPv6 addresses from the hosts so they are only using the PI IPv6 space. Next, remove the AAAA and reverse entries that reference this old space in the DNS. The final steps are to stop routing the original address space allocated by NTT America for the single-homed IPv6 connection that was in place before all this started and then return that space to them to reassign. I have started to remove the old addresses from the hosts and remove the DNS entries. I have been doing this things as part of other scheduled work that involves rebooting a host for other reasons (doing patching or OS upgrades) But, I have not yet determined a timetable for the final two items to close out this project.
I gave a talk about this at IEEE Globecom in 2010. The slides from that talk are available on this web site.
Ryan
April 20th, 2011 at 2:25 pm #
Very helpful in regards to the experience of implementing IPv6 multi-homing. Nice to know NTT and Level3 will both route PI IPv6 address blocks.
netsaint
April 28th, 2011 at 2:19 pm #
Luckily you started with NTT on your IPv6 service. They are connected well in the IPv6 world. If you would have started with Level3 first you would have notice that Level3 is not yet peering with everybody in the IPv6 world. When I started with Level3 there was no IPv6 connectivity to HE or Google. That also means no access to test-ipv6.com as this is hosted by HE. They are getting there, when I started with Level3, I had 3500 IPv6 routes vs ~5000 from NTT. Today I have 4121 IPv6 routes from Level3 and 5411 from NTT.
The problem with NTT right now in the US is that their IPv6 peering for example with Google is done in Japan. So, my latency to ipv6.google.com is 400ms. A regional peering for IPv6 with Google is needed.
Netsaint