BIND not loading into memory on first transfer

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

BIND not loading into memory on first transfer

Frank Even
The subject is about the only way I can think to describe a situation
we've run into recently.  First here is the system:

[root@dns]# cat /etc/redhat-release
CentOS release 6.6 (Final)
[root@dns]# rpm -q bind
bind-9.8.2-0.30.rc1.el6_6.2.x86_64

So, we got bit by a chroot permissions issue (unsure exactly how it
got introduced), where the chroot was owned by root, but had named as
the group owner.  Perms were 750 on the dir (rwxr-x---)

Zone files were in place for the necessary domains, but were outdated
(assuming one of our updates broke something somewhere, they were all
on average 3 months old).

We updated some of the boxes, and on restart, named started.

It initially started loading the 3 month old zone (one frequently
updated I might add).  The boxes then did a zone transfer from the
master.  Failing to be able to write the tmp file to the working
directory, it moved on.

Here is where the issue is.  I've generally found if BIND fails to
write the zone, it generally loads it into memory (masking the fact
that there is an issue here for an extended period of time).  In this
particular instance, the masters ended up under maintenance shortly
after these boxes rebooted, so they were unable to transfer the zone
from them for another 2 hours.  These boxes were serving stale data
after boot despite being able to accomplish one zone transfer after
boot.  It seems that failing the first zone transfer did NOT load the
zone into memory (but subsequent zone transfers while still failing to
write the tmp file DID load the zone into memory).

I guess the question really is, is this expected behavior or a bug?

Thanks,
Frank
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Reindl Harald

Am 26.03.2015 um 19:34 schrieb Frank Even:
> Zone files were in place for the necessary domains, but were outdated
> (assuming one of our updates broke something somewhere, they were all
> on average 3 months old)
>
> I guess the question really is, is this expected behavior or a bug?

after 3 months the zones may even be expired and not loaded at all, been
there after a dumb ISP closed port 53 incoming for weird reasons without
notify us (because the router in front was vulerable on port 53)

i guess you are in the area of undefined behavior here at all


_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Matus UHLAR - fantomas
In reply to this post by Frank Even
On 26.03.15 11:34, Frank Even wrote:
>Zone files were in place for the necessary domains, but were outdated
>(assuming one of our updates broke something somewhere, they were all
>on average 3 months old).

>Here is where the issue is.  I've generally found if BIND fails to
>write the zone, it generally loads it into memory (masking the fact
>that there is an issue here for an extended period of time).  In this
>particular instance, the masters ended up under maintenance shortly
>after these boxes rebooted, so they were unable to transfer the zone
>from them for another 2 hours.  These boxes were serving stale data
>after boot despite being able to accomplish one zone transfer after
>boot.  It seems that failing the first zone transfer did NOT load the
>zone into memory (but subsequent zone transfers while still failing to
>write the tmp file DID load the zone into memory).
>
>I guess the question really is, is this expected behavior or a bug?

What's the SOA? It's possible that the zones were not expired, so they were
provided as saved on disk. Since BIND wasn't able to transfer newer
versions, it continued providing old versions.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Have you got anything without Spam in it?
- Well, there's Spam egg sausage and Spam, that's not got much Spam in it.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Frank Even
On Thu, Mar 26, 2015 at 12:17 PM, Matus UHLAR - fantomas
<[hidden email]> wrote:

> On 26.03.15 11:34, Frank Even wrote:
>>
>> Zone files were in place for the necessary domains, but were outdated
>> (assuming one of our updates broke something somewhere, they were all
>> on average 3 months old).
>
>
>> Here is where the issue is.  I've generally found if BIND fails to
>> write the zone, it generally loads it into memory (masking the fact
>> that there is an issue here for an extended period of time).  In this
>> particular instance, the masters ended up under maintenance shortly
>> after these boxes rebooted, so they were unable to transfer the zone
>> from them for another 2 hours.  These boxes were serving stale data
>> after boot despite being able to accomplish one zone transfer after
>> boot.  It seems that failing the first zone transfer did NOT load the
>> zone into memory (but subsequent zone transfers while still failing to
>> write the tmp file DID load the zone into memory).
>>
>> I guess the question really is, is this expected behavior or a bug?
>
>
> What's the SOA? It's possible that the zones were not expired, so they were
> provided as saved on disk. Since BIND wasn't able to transfer newer
> versions, it continued providing old versions.
>

Yes, the old versions were provided on disk on initial load.  But that
was then followed up with  a SUCCESSFUL zone transfer minutes later,
but the server was unable to save the tmp file in the working
directory and served stale content until about 2 hours later when the
server was able to get another successful zone transfer from the
master and then loaded the new zone in memory (despite being unable to
write the tmp file to update the local copy of the zone).
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Matus UHLAR - fantomas
>On Thu, Mar 26, 2015 at 12:17 PM, Matus UHLAR - fantomas
><[hidden email]> wrote:
>> What's the SOA? It's possible that the zones were not expired, so they were
>> provided as saved on disk. Since BIND wasn't able to transfer newer
>> versions, it continued providing old versions.

On 26.03.15 12:48, Frank Even wrote:
>Yes, the old versions were provided on disk on initial load.  But that
>was then followed up with  a SUCCESSFUL zone transfer minutes later,
>but the server was unable to save the tmp file in the working
>directory and served stale content until about 2 hours later when the
>server was able to get another successful zone transfer from the
>master and then loaded the new zone in memory (despite being unable to
>write the tmp file to update the local copy of the zone).

what didthe logs say?
Looks like the first transfer wasn't really successful (or the zone was
rejected)
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I'm not interested in your website anymore.
If you need cookies, bake them yourself.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Frank Even
On Thu, Mar 26, 2015 at 1:00 PM, Matus UHLAR - fantomas
<[hidden email]> wrote:

>> On Thu, Mar 26, 2015 at 12:17 PM, Matus UHLAR - fantomas
>> <[hidden email]> wrote:
>>>
>>> What's the SOA? It's possible that the zones were not expired, so they
>>> were
>>> provided as saved on disk. Since BIND wasn't able to transfer newer
>>> versions, it continued providing old versions.
>
>
> On 26.03.15 12:48, Frank Even wrote:
>>
>> Yes, the old versions were provided on disk on initial load.  But that
>> was then followed up with  a SUCCESSFUL zone transfer minutes later,
>> but the server was unable to save the tmp file in the working
>> directory and served stale content until about 2 hours later when the
>> server was able to get another successful zone transfer from the
>> master and then loaded the new zone in memory (despite being unable to
>> write the tmp file to update the local copy of the zone).
>
>
> what didthe logs say?
> Looks like the first transfer wasn't really successful (or the zone was
> rejected)

Logs indicated successful transfer, permission denied writing the
tmp-xxxxx file that happens prior to writing it out to the zone file
itself.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Matus UHLAR - fantomas
>>>> What's the SOA? It's possible that the zones were not expired, so they
>>>> were provided as saved on disk.  Since BIND wasn't able to transfer
>>>> newer versions, it continued providing old versions.

>> On 26.03.15 12:48, Frank Even wrote:
>>> Yes, the old versions were provided on disk on initial load.  But that
>>> was then followed up with  a SUCCESSFUL zone transfer minutes later,
>>> but the server was unable to save the tmp file in the working
>>> directory and served stale content until about 2 hours later when the
>>> server was able to get another successful zone transfer from the
>>> master and then loaded the new zone in memory (despite being unable to
>>> write the tmp file to update the local copy of the zone).

>> what didthe logs say?
>> Looks like the first transfer wasn't really successful (or the zone was
>> rejected)

On 26.03.15 14:48, Frank Even wrote:
>Logs indicated successful transfer, permission denied writing the
>tmp-xxxxx file that happens prior to writing it out to the zone file
>itself.

and how do hey differ from the second transfer?
If they don't itmay be a bug (or a "bug") in named that it behaves
differently after first and other transfers...

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
M$ Win's are shit, do not use it !
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

/dev/rob0
In reply to this post by Frank Even
On Thu, Mar 26, 2015 at 11:34:42AM -0700, Frank Even wrote:

> The subject is about the only way I can think to describe a
> situation we've run into recently.  First here is the system:
>
> [root@dns]# cat /etc/redhat-release
> CentOS release 6.6 (Final)
> [root@dns]# rpm -q bind
> bind-9.8.2-0.30.rc1.el6_6.2.x86_64
>
> So, we got bit by a chroot permissions issue (unsure exactly how it
> got introduced), where the chroot was owned by root, but had named
> as the group owner.  Perms were 750 on the dir (rwxr-x---)
>
> Zone files were in place for the necessary domains, but were
> outdated (assuming one of our updates broke something somewhere,
> they were all on average 3 months old).
>
> We updated some of the boxes, and on restart, named started.
>
> It initially started loading the 3 month old zone (one frequently
> updated I might add).  The boxes then did a zone transfer from the
> master.  Failing to be able to write the tmp file to the working
> directory, it moved on.

Slave and other dynamic zones do require write privilege in the
working directory.  Have you fixed this problem yet?

If you're running as user "named", that's the user which must have
write privilege.  If running as root, root must have explicit
privilege to write, because named drops superuser capabilities.

I suspect the problem might be SELinux.  Check "getenforce", and
perhaps restore the context to the working directory (see "man
restorecon") or disable SELinux if you prefer.

> Here is where the issue is.  I've generally found if BIND fails to
> write the zone, it generally loads it into memory (masking the fact
> that there is an issue here for an extended period of time).

named makes a best effort to get up and running, which it ought to
do, IMO.  It's not masking anything; the inability to write to the
working directory has been logged.

> In this particular instance, the masters ended up under maintenance
> shortly after these boxes rebooted, so they were unable to transfer
> the zone from them for another 2 hours.  These boxes were serving
> stale data after boot despite being able to accomplish one zone
> transfer after boot.  It seems that failing the first zone transfer
> did NOT load the zone into memory (but subsequent zone transfers
> while still failing to write the tmp file DID load the zone into
> memory).
>
> I guess the question really is, is this expected behavior or a bug?

The bug is a misconfiguration bug, where contrary to documented
requirements, you have not given named write privilege in its
directory.

I think you're saying named should fail to load the stale zones,
which at startup it cannot know are stale.  That does not sound
correct to me.

Perhaps you're suggesting that named should SERVFAIL the zone when
the first zone transfer has happened, and while this sounds more
reasonable, I am not sure that the zone transfer actually does take
place if named is unable to open a temporary file to write.  (What
would be the point in talking to the master when you know you are
unable to handle the data?)
--
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Barry Margolin
In reply to this post by Frank Even
In article <[hidden email]>,
 /dev/rob0 <[hidden email]> wrote:

> On Thu, Mar 26, 2015 at 11:34:42AM -0700, Frank Even wrote:
> > In this particular instance, the masters ended up under maintenance
> > shortly after these boxes rebooted, so they were unable to transfer
> > the zone from them for another 2 hours.  These boxes were serving
> > stale data after boot despite being able to accomplish one zone
> > transfer after boot.  It seems that failing the first zone transfer
> > did NOT load the zone into memory (but subsequent zone transfers
> > while still failing to write the tmp file DID load the zone into
> > memory).
> >
> > I guess the question really is, is this expected behavior or a bug?
>
> The bug is a misconfiguration bug, where contrary to documented
> requirements, you have not given named write privilege in its
> directory.
>
> I think you're saying named should fail to load the stale zones,
> which at startup it cannot know are stale.  That does not sound
> correct to me.
>
> Perhaps you're suggesting that named should SERVFAIL the zone when
> the first zone transfer has happened, and while this sounds more
> reasonable, I am not sure that the zone transfer actually does take
> place if named is unable to open a temporary file to write.  (What
> would be the point in talking to the master when you know you are
> unable to handle the data?)

He's not suggesting either of these. He's saying that when it
successfully transferred the zone, it should update the in-memory
version, and serve that, even though it wasn't able to save it to disk.
That's what it does on the SECOND transfer, it just doesn't do it on the
FIRST transfer.

--
Barry Margolin
Arlington, MA
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND not loading into memory on first transfer

Frank Even
On Fri, Mar 27, 2015 at 8:25 AM, Barry Margolin <[hidden email]> wrote:

> In article <[hidden email]>,
>  /dev/rob0 <[hidden email]> wrote:
>
>> On Thu, Mar 26, 2015 at 11:34:42AM -0700, Frank Even wrote:
>> > In this particular instance, the masters ended up under maintenance
>> > shortly after these boxes rebooted, so they were unable to transfer
>> > the zone from them for another 2 hours.  These boxes were serving
>> > stale data after boot despite being able to accomplish one zone
>> > transfer after boot.  It seems that failing the first zone transfer
>> > did NOT load the zone into memory (but subsequent zone transfers
>> > while still failing to write the tmp file DID load the zone into
>> > memory).
>> >
>> > I guess the question really is, is this expected behavior or a bug?
>>
>> The bug is a misconfiguration bug, where contrary to documented
>> requirements, you have not given named write privilege in its
>> directory.
>>
>> I think you're saying named should fail to load the stale zones,
>> which at startup it cannot know are stale.  That does not sound
>> correct to me.
>>
>> Perhaps you're suggesting that named should SERVFAIL the zone when
>> the first zone transfer has happened, and while this sounds more
>> reasonable, I am not sure that the zone transfer actually does take
>> place if named is unable to open a temporary file to write.  (What
>> would be the point in talking to the master when you know you are
>> unable to handle the data?)
>
> He's not suggesting either of these. He's saying that when it
> successfully transferred the zone, it should update the in-memory
> version, and serve that, even though it wasn't able to save it to disk.
> That's what it does on the SECOND transfer, it just doesn't do it on the
> FIRST transfer.

^^^  THIS!  Exactly!  I REALIZE I had a configuration problem that
prevented writing the zone file locally that snuck in as it turns out
on the bind-chroot package update.  That's irrelevant to the issue I
noticed after that. It DOES load up in memory and serve it up
generally.  It's just that what I've seen in this particular instance
is that it failed to do this on the first successful zone transfer,
then loaded it up in memory on the 2nd try (which sadly in this
instance, was 2 hours later due to other issues, which of course
caused it to be noticed in this instance where it might not have been
in previous instances).

Thanks.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users