One of my clients has had their web server exposed to the wild world of the internet now for several years. Up till about a year and a half ago many systems on their network actually had IP ANY ANY statements cut through from the Outside of their Firewall to the Inside. However it has been one of my many jobs since I started with them to eradicate these problems and start securing their infrastructure. The firewall changes have been easy for the most part and any problems that remain are policy issues that we are working to eliminate. However their web server sitting outside of the firewall has been an ongoing issue and due to some anomaly’s on the server they are deploying the recommended DMZ and migrating their web server there.
As I type this at 2:40am in the morning we are wrapping up the cutover. One of our biggest issues tonight was after moving the sever to a secure DMZ was the clients DNS. With a central site and 20 WAN Sites they have created what I think is a pretty basic yet elegant DNS structure. The central Data Center hosts the primary DNS server NS1. Each edge site then has a server responsible for DNS, DHCP, Directory Auth (via replica) and file and print services. The DNS on the local servers at the WAN sites hold DNS entries for all printers and server resources at those sites. They look to the central DNS server for any non-local name resolution. NS1 in turn does lookup then cache with their provider who does not allow zone file transfers.
Our issue was that we could not get the WAN site DNS servers to update to the new record entry. We tried doing a DNS Kill, flushing the cache then restarting the servers. We tried using dig @ns1.domain.com www.webhost.com axfr to force a zone transfer (this kept failing). And finally we found our solution. It is a bit brute force so bear with me. All of our DNS servers are OSX 10.4+. We modified the record on the root DNS server. We then restarted the root DNS server. Once the root DNS server was back up and we were able to locally resolve hosts we ssh’d to all remote DNS server and performed
and that was it.
In the end it worked. I am personally not happy with how sloppy it was but what I do know is that it reflects a lack of knowledge on my part as well as the customers when it comes to structured DNS. So this weekend once I get a few hours of sleep I think its time to break out my Oriely’s BIND/DNS book and spin up a few new VM’s on the home server and well and truly learn how DNS propagates and what we as administrators can do to optimize the process.
Once I get that done I plan on dumping a primer on DNS here to help others out. I know first hand that most of what is out there either deals with the RFC and its revision which tend to be overkill for troubleshooting or are a mixture trial and error tips from others who have fought the battle and one even if they are not sure why they won. The final note to this is that as a consultant I am expected to know what I am doing or how to quickly find the proper course. Thats why I get paid to come help people. I think often consultant, employees and more importantly bosses forget that we get paid to know things. This is a really pissy issue with me right now so look for a consulting article coming soon about industry expertise and our duty to produce quality services for our clients.