BIND-9.16.1 memory leak?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

BIND-9.16.1 memory leak?

sthaug
We have what appears to be a significant memory leak in BIND-9.16.1.

Environment:
 FreeBSD 12.1-STABLE.
 BIND-9.16.1 installed from packages.
 Also uses libuv-1.35.0 installed from packages.
 Authoritative only.
 Around 800 zones of varying sizes. DNSSEC in use.

Running a ps command for the named process every minute and logging
the result, I see the named virtual memory size (VSZ) increasing at
around 1.2 Mbyte/minute, and the resident size (RSS) increasing at
around 0.85 Mbyte/minute. No problems due to this so far, but pretty
obviously it's not viable in the long run.

I tried reading the CHANGES from 9.16.2, and didn't see anything which
suggested a fix for a memory leak problem.

Any suggestions?

Steinar Haug, Nethelp consulting, [hidden email]
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

Karl Pielorz


--On 17 April 2020 at 15:45:16 +0200 [hidden email] wrote:

> We have what appears to be a significant memory leak in BIND-9.16.1.
...

> Running a ps command for the named process every minute and logging
> the result, I see the named virtual memory size (VSZ) increasing at
> around 1.2 Mbyte/minute, and the resident size (RSS) increasing at
> around 0.85 Mbyte/minute. No problems due to this so far, but pretty
> obviously it's not viable in the long run.
>
> I tried reading the CHANGES from 9.16.2, and didn't see anything which
> suggested a fix for a memory leak problem.
>
> Any suggestions?

Hi,

I seem to remember we got 'bitten' by large memory use when moving from a
previous version of bind - do you have 'max-cache-size' set in your config?

As far as I can remember, the 'default' is to take 90% of the memory on the
machine. This is great, unless the machine has lots of other stuff going on
etc.  I think we noticed this during bind startup (i.e. from syslog output).

On our boxes we set it to "something sensible" - rather than using the
default.

Might not be your problem - but thought it was worth mentioning / checking.

-Karl
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

sthaug
>> We have what appears to be a significant memory leak in BIND-9.16.1.
...
> I seem to remember we got 'bitten' by large memory use when moving
> from a previous version of bind - do you have 'max-cache-size' set in
> your config?

Yes. Set to 1G. In reality it shouldn't need a cache at all, since
this is a purely authoritative server (recursion no).

So this doesn't appear to be the problem. But thanks for the
suggestion!

Steinar Haug, Nethelp consulting, [hidden email]
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

Anand Buddhdev
In reply to this post by Karl Pielorz
On 17/04/2020 17:02, Karl Pielorz wrote:

Hi Karl,

> I seem to remember we got 'bitten' by large memory use when moving from
> a previous version of bind - do you have 'max-cache-size' set in your
> config?

It's an authoritative-only server, so there is (almost) no caching involved.

Anand
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

sthaug
In reply to this post by sthaug
Followup:

> We have what appears to be a significant memory leak in BIND-9.16.1.
>
> Environment:
>  FreeBSD 12.1-STABLE.
>  BIND-9.16.1 installed from packages.
>  Also uses libuv-1.35.0 installed from packages.
>  Authoritative only.
>  Around 800 zones of varying sizes. DNSSEC in use.
>
> Running a ps command for the named process every minute and logging
> the result, I see the named virtual memory size (VSZ) increasing at
> around 1.2 Mbyte/minute, and the resident size (RSS) increasing at
> around 0.85 Mbyte/minute. No problems due to this so far, but pretty
> obviously it's not viable in the long run.
>
> I tried reading the CHANGES from 9.16.2, and didn't see anything which
> suggested a fix for a memory leak problem.

I now have a pcap file of queries that I can replay with the "drool"
application, and I'm consistently seeing similar memory leak problems
(i.e. the problems are reproducible). The memory leak rate seems to be
very approximately linear with the query rate - so replaying at 10
times the original speed means we also leak around 10 times as much per
minute.

Upgrading to 9.16.2 (and also libuv 1.36.0) makes no difference - the
same memory leak is observed.

Steinar Haug, Nethelp consulting, [hidden email]
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

Evan Hunt
On Sun, Apr 19, 2020 at 02:55:53PM +0200, [hidden email] wrote:
> I now have a pcap file of queries that I can replay with the "drool"
> application, and I'm consistently seeing similar memory leak problems
> (i.e. the problems are reproducible). The memory leak rate seems to be
> very approximately linear with the query rate - so replaying at 10
> times the original speed means we also leak around 10 times as much per
> minute.
>
> Upgrading to 9.16.2 (and also libuv 1.36.0) makes no difference - the
> same memory leak is observed.

Is there anything unusual in your server configuration?

--
Evan Hunt -- [hidden email]
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

Michael Sinatra
In reply to this post by sthaug
On 2020-04-17 06:45, [hidden email] wrote:
> We have what appears to be a significant memory leak in BIND-9.16.1.
>
> Environment:
>  FreeBSD 12.1-STABLE.
>  BIND-9.16.1 installed from packages.
>  Also uses libuv-1.35.0 installed from packages.
>  Authoritative only.
>  Around 800 zones of varying sizes. DNSSEC in use.

Additional datum, as I am seeing the same thing:

- FreeBSD 12.1-RELEASE-p3
- BIND-9.16.1 compiled from ports/poudriere via a local package build
server (no options changed, though, so it likely could have been
installed from the FreeBSD package repo).
- Authoritative only
- `rndc status` reports 1058 zones (69 automatic)
- Host is a VM with 16GiB allocated and 4 CPU cores
- named running for approx 2.5 weeks (wall-clock)

Current BIND status (from `top`):

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU
COMMAND
 1707 bind         14  52    0  5312M  5260M sigwai   2  34.4H   5.79% named

A recursive-only server, running the same versions of all software, on
an identically-provisioned VM, running for the same amount of wall-clock
time (approximately 2.5 weeks) looks like this:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU
COMMAND
 1485 bind         14  52    0   927M   890M sigwai   3  89.6H  32.86% named

The recursive memory footprint looks normal.

Contrast that with a separate server:

- FreeBSD 11.3-RELEASE-p7
- BIND 9.14.11 compiled from ports/poudriere via a local package build
server (no options changed, though, so it likely could have been
installed from the FreeBSD package repo).
- Authoritative only + recursive only running in a separate jail
- Same configuration as above, only a bit busier
- Host is standalone with 96GiB RAM and 8 cores

In the `top` output below, both the jailed named processes are shown.
The busier one is the authoritative-only:

  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU
COMMAND
  896 bind           18  52    0   956M   927M sigwai  0  99.2H  30.03%
named
 1584 bind           18  52    0  1171M  1080M sigwai  2 166.2H  13.47%
named

It definitely looks like a memory leak in 9.16.1 when configured as
authoritative-only.  The leak seems slow enough as to be manageable, but
the footprint does appear to growing monotonically (and is still
growing--by another 4M as I wrote this email).

michael
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: BIND-9.16.1 memory leak?

Greg Rivers
In reply to this post by sthaug
On Friday, 17 April 2020 08:45:16 CDT Steinar Haug wrote:
> We have what appears to be a significant memory leak in BIND-9.16.1.
>
> Environment:
>  FreeBSD 12.1-STABLE.
>  BIND-9.16.1 installed from packages.
>  Also uses libuv-1.35.0 installed from packages.
>  Authoritative only.
>  Around 800 zones of varying sizes. DNSSEC in use.
>
https://gitlab.isc.org/isc-projects/bind9/-/issues/1893

--
Greg


_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users