Moving dynamic zones to new master+slave pair without interruptions

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving dynamic zones to new master+slave pair without interruptions

Peter Rathlev
We currently have two internal DNS servers that are both authoritative
for a range of internal zones and caching resolvers for our clients. We
would like to split this so authorizative and caching roles exist on
different servers. And we would like to do this with as little down
time as possible, also for dynamic zones.

Moving static zones is of course trivial. Moving dynamic zones is what
I cannot quite wrap my head around.

I think I want to set up a new slave and AXFR from the existing master.
Then I can point delegations and "forwarders" at this new slave only,.
Together with having the configured "masters" pointing at a not yet
running master server this would make it "stand alone".

Next step in my head would be to re-create the master from this slave.
I thought that I could just copy the zone files from the slave, since
that slave would not have made any changes, seeing as it is only the
master that can do that. (I am fine with rejecting changes to the
dynamic zones during the move exercise.)

However, I see that the current slave also has ".jnl" files for the
dynamic zones and "rndc freeze <zone>" is invalid except on the zone
master. With journal files present I guess that I cannot trust the zone
files to actually be valid/complete.

So... What do I do then? Is there another way of committing the journal
to disk on a slave? Is there a "best practice" for re-creating a lost
master when dealing dynamic zones?

I may of course have started out completely wrong. If there are better
ways to acheive what I want then I am all ears! :-)

This is all a thought exercise right now, I have not actually tried to
move anything yet.

If BIND versions are relevant then we plan on using the CentOS 6
default which is BIND 9.8.2 (with some patches, so it's bind-9.8.2-
0.37.rc1.el6_7.5.x86_64) on the new servers. Building from sources is a
hassle we would rather avoid, but since we are already doing this with
ISC DHCP we could also do it with BIND if necessary.

Current master is _quite_ old, BIND 9.3.6 (bind-9.3.6-25.P1.el5_11.5).
So the setup is really in need of a refresh. :-)

Thank you in advance!

-- 
Peter Rathlev

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Tony Finch
Peter Rathlev <[hidden email]> wrote:

> We currently have two internal DNS servers that are both authoritative
> for a range of internal zones and caching resolvers for our clients. We
> would like to split this so authorizative and caching roles exist on
> different servers. And we would like to do this with as little down
> time as possible, also for dynamic zones.
>
> Moving static zones is of course trivial. Moving dynamic zones is what
> I cannot quite wrap my head around.

I suggest the following process:

* Set up a new hidden master, with copies of your zones. (See below)

* Change your existing servers to slave from the new hidden master instead
  of the old master. Reconfigure the old master to be a slave of the new
  one.

* Add new slaves which will be your new authoritative-only servers.

* Change your delegations to point to your new authoritative-only servers.


You don't need to worry about the data on disk on your existing slaves.
They will continue to serve the same data, they will just xfer changes
from a different master.


My program nsdiff (http://dotat.at/prog/nsdiff) is useful for copying
dynamic zones from from an existing master to a new master without
faffing around with `rndc freeze`. On the new master, run

        nsdiff -m oldmaster -s localhost myzone | nsupdate -l

and it will axfr the zone from the oldmaster and copy it into the new
master using dynamic updates.

(If you are changing your DNS infrastructure then nsdiff can be useful for
verifying that the zone data is consistent between old and new.)


Tony.
--
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
Southwest Forties, Cromarty, Forth: Southeasterly 6 to gale 8, occasionally
severe gale 9 later. Rough or very rough, occasionally high later. Rain at
times. Moderate, occasionally poor.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

RE: Moving dynamic zones to new master+slave pair without interruptions

Kevin Darcy
In reply to this post by Peter Rathlev
I'd just like to note in passing that the "separate authoritative and recursive" herd mentality reaches the ultimate point of absurdity when you only have 2 servers and you're going to create single points of failure (apparently, unless I'm misinterpreting "stand alone") to conform to this so-called "best practice".

Needless to say, I don't subscribe to the (apparently popular) notion that the roles need to exist on separate *hardware*. View-level separation is, in my opinion, sufficient to meet the security requirements. (Bear in mind, views can be matched by TSIG key, if one doesn’t consider match-clients or match-destinations to be sufficiently rigorous; while this may not be practical for typical stub-resolver-to-BIND-instance communication, it is something to consider at or near the apex of a forwarding hierarchy).

If match-clients-based or TSIG-based view-level separation isn't considered rigorous enough, then you could spin up additional IP addresses and run authoritative on one set, and recursive on another set. Even the eponymous Mr. Bernstein, one of the leading proponents of auth/recursive separation (in his DNS software package, they are totally separate programs), takes care to say to not run auth and recursive "on the same IP address". See https://cr.yp.to/djbdns/separation.html. Never does he say -- as others do -- that the roles have to be on separate *hardware* (or, in the modern era, these might actually be separate virtual instances). Now, whether you actually run separate named processes, with specific listen-on's, for those IPs, or take the view approach, with match-destinations, is, again, dependent on how much rigor you want to apply to your separation.

But, hopefully, I've given you some other options to consider besides the most extreme, hardware-based separation approach. Remember that "availability" is one of the pillars of information security, and if you sacrifice availability to conform to a "best practice", you might not be improving your overall information security.

Speaking of availability, as your network evolves, you might want to consider running recursive service on Anycast addresses (see http://ddiguru.com/blog/118-introduction-to-anycast-dns or Cricket's informative video). When implemented, this largely moots the whole "recursive versus authoritative" debate, because recursive service now runs on IP addresses that are "virtual", at a network-routing level, and do not intersect with the IP addresses used for authoritative service (if one wants to implement Anycast for *authoritative* service, like the Public Internet does, those would typically be a *separate* set of Anycast addresses from the recursive ones).

                                                                                                - Kevin


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Rathlev
Sent: Wednesday, January 06, 2016 8:17 AM
To: [hidden email]
Subject: Moving dynamic zones to new master+slave pair without interruptions

We currently have two internal DNS servers that are both authoritative for a range of internal zones and caching resolvers for our clients. We would like to split this so authorizative and caching roles exist on different servers. And we would like to do this with as little down time as possible, also for dynamic zones.

Moving static zones is of course trivial. Moving dynamic zones is what I cannot quite wrap my head around.

I think I want to set up a new slave and AXFR from the existing master.
Then I can point delegations and "forwarders" at this new slave only,.
Together with having the configured "masters" pointing at a not yet running master server this would make it "stand alone".

Next step in my head would be to re-create the master from this slave.
I thought that I could just copy the zone files from the slave, since that slave would not have made any changes, seeing as it is only the master that can do that. (I am fine with rejecting changes to the dynamic zones during the move exercise.)

However, I see that the current slave also has ".jnl" files for the dynamic zones and "rndc freeze <zone>" is invalid except on the zone master. With journal files present I guess that I cannot trust the zone files to actually be valid/complete.

So... What do I do then? Is there another way of committing the journal to disk on a slave? Is there a "best practice" for re-creating a lost master when dealing dynamic zones?

I may of course have started out completely wrong. If there are better ways to acheive what I want then I am all ears! :-)

This is all a thought exercise right now, I have not actually tried to move anything yet.

If BIND versions are relevant then we plan on using the CentOS 6 default which is BIND 9.8.2 (with some patches, so it's bind-9.8.2-
0.37.rc1.el6_7.5.x86_64) on the new servers. Building from sources is a hassle we would rather avoid, but since we are already doing this with ISC DHCP we could also do it with BIND if necessary.

Current master is _quite_ old, BIND 9.3.6 (bind-9.3.6-25.P1.el5_11.5).
So the setup is really in need of a refresh. :-)

Thank you in advance!

--
Peter Rathlev

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Peter Rathlev
In reply to this post by Tony Finch
Hi Tony,

Thank you for the suggestions!

On Wed, 2016-01-06 at 16:05 +0000, Tony Finch wrote:
> * Set up a new hidden master, with copies of your zones. (See below)
>
> * Change your existing servers to slave from the new hidden master
> instead of the old master. Reconfigure the old master to be a slave  
> of the new one.

Wouldn't this ruin dynamic updates from the DHCP servers? These updates
need to be sent to the master. I could of course configur™e "allow-
update-forwarding". Manually specifying the hidden master in the DHCP
configuration seems clumsy.

> You don't need to worry about the data on disk on your existing
> slaves. They will continue to serve the same data, they will just
> xfer changes from a different master.

This made my think... Maybe I could just AXFR from the running slave
and use the output as zone files on the master. As far as I can see
this should Just Work™.

> My program nsdiff (http://dotat.at/prog/nsdiff) is useful for copying
> dynamic zones from from an existing master to a new master without
> faffing around with `rndc freeze`.

Nice. :-) Perfect for copying changes without touching the files. I'll
take a thorough look at it.

-- 
Peter Rathlev

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Peter Rathlev
In reply to this post by Kevin Darcy
On Wed, 2016-01-06 at 18:04 +0000, Darcy Kevin (FCA) wrote:
> I'd just like to note in passing that the "separate authoritative and
> recursive" herd mentality reaches the ultimate point of absurdity
> when you only have 2 servers and you're going to create single points
> of failure (apparently, unless I'm misinterpreting "stand alone") to
> conform to this so-called "best practice".
[...]

I'm not religious about either model, but in this case the load on the
recursive caching servers merits them being their own instances. We are
not splitting the functions based on security concerns.

> Needless to say, I don't subscribe to the (apparently popular) notion
> that the roles need to exist on separate *hardware*. [...]

One of two authoritative servers and two of three recursing will be
virtual servers. So it's not as much a waste of hardware as it could
be. :-)

>  View-level separation is, in my opinion, sufficient to meet the
> security requirements. [...]

Certainly. We use views on the resolvers for our public "guest" network
and have had not concerns about this.

[...]
> Speaking of availability, as your network evolves, you might want to
> consider running recursive service on Anycast addresses [...]

We already use anycasting on the recursive servers and would prefer a
simple configuration that can easily be replicated to new instances. As
part of this pending transition we will introduce an extra recursing
server.

Keeping things simple, even if that means running more servers, helps
me sleep at night. It helps my colleagues handling things without
having to call me. :-)

-- 
Peter Rathlev
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Evan Hunt
In reply to this post by Peter Rathlev
On Thu, Jan 07, 2016 at 08:41:13AM +0100, Peter Rathlev wrote:
> This made my think... Maybe I could just AXFR from the running slave
> and use the output as zone files on the master. As far as I can see
> this should Just Work™.

Yes, it will. Relatively recent versions of dig (since 9.8) have a +onesoa
option to suppress the extra SOA record that turns up at the end of the
AXFR, for exactly this purpose.

$ dig +nocmd +onesoa @<server> axfr <zone>

--
Evan Hunt -- [hidden email]
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Niall O'Reilly
In reply to this post by Peter Rathlev
On Wed, 06 Jan 2016 13:16:39 +0000,
Peter Rathlev wrote:

>
> We currently have two internal DNS servers that are both authoritative
> for a range of internal zones and caching resolvers for our clients. We
> would like to split this so authorizative and caching roles exist on
> different servers. And we would like to do this with as little down
> time as possible, also for dynamic zones.
>
> Moving static zones is of course trivial. Moving dynamic zones is what
> I cannot quite wrap my head around.
>
> I think I want to set up a new slave and AXFR from the existing master.

  After reading other messages in this thread, it occurs to me that if
  enough of your client systems use DHCP to configure their stub
  resolvers, you could approach the problem from the other end by just
  adding more caching resolvers (perhaps on a new set of anycast
  addresses) and then change the option (perhaps using a gradual
  per-network roll-out) in the DHCP configuration(s).

  This way, you don't have to worry about moving zones around, and
  still relieve your combined-mode servers of the client query load.

  YMMV
  Niall O'Reilly
 

 
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Peter Rathlev
On Thu, 2016-01-07 at 09:30 +0000, Niall O'Reilly wrote:
> After reading other messages in this thread, it occurs to me that if
> enough of your client systems use DHCP to configure their stub
> resolvers, you could approach the problem from the other end by just
> adding more caching resolvers (perhaps on a new set of anycast
> addresses) and then change the option (perhaps using a gradual
> per-network roll-out) in the DHCP configuration(s).

We're also moving the recursive resolver function away from these
servers, but since the hardware is rather old (from 2006) we want to
move everything, including the authoritative role.

Generally our clients only query the anycast addresses and those can
easily be moved elsewhere. The IP addresses of the authoritative name
servers can thus be changed without incident.

-- 
Peter Rathlev

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: Moving dynamic zones to new master+slave pair without interruptions

Tony Finch
In reply to this post by Peter Rathlev
Peter Rathlev <[hidden email]> wrote:
> On Wed, 2016-01-06 at 16:05 +0000, Tony Finch wrote:
> > * Set up a new hidden master, with copies of your zones. (See below)
> >
> > * Change your existing servers to slave from the new hidden master
> > instead of the old master. Reconfigure the old master to be a slave
> > of the new one.
>
> Wouldn't this ruin dynamic updates from the DHCP servers? These updates
> need to be sent to the master.

The master name in the zone's SOA record should point to the real master,
i.e. the "hidden" master in your new setup. Updates are normally sent to
the SOA master.

> This made my think... Maybe I could just AXFR from the running slave
> and use the output as zone files on the master. As far as I can see
> this should Just Work™.

Yes, though you should also ensure there isn't any stray journal file on
the new master. (This is essentially what nsdiff does, but it uses
nsupdate to write the zone data to the master instead of just dumping to a
file.)

Tony.
--
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
Forth, Tyne, West Dogger: Southeasterly veering northwesterly, 7 to severe
gale 9, decreasing 5 or 6 later. Rough or very rough. Rain. Moderate,
occasionally poor.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users