random dispatch error after reconfig with bind ESV 9.11

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

random dispatch error after reconfig with bind ESV 9.11

Laurent Frigault
Hi,

OS: FreeBSD 12.1 / AMD64
56 cores CPU
64G RAM

bind ESV 9.11.19 from standard freebsd pkg

We are runng an hiddenmaster bind server with DNSSEC zone "bookmyname.be" {
zone "bookmyname.be" {
    type master;
    file "custom/b/o/bookmyname.be/bookmyname.be";
    notify explicit;
    also-notify { 213.36.252.135; 62.210.98.15; 213.36.253.14; };
    auto-dnssec maintain;
    inline-signing yes;
    key-directory "custom/b/o/bookmyname.be";
};

We have about 73000 zones , most signed.

We periodically regenerate our configuration to add/update/remove zones.

when needed, we use "rndc reconfig"

Every few weeks we get the following error in the log  :

Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: ./server.c:3881: unexpected error:
Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: unable to obtain neither an IPv4 nor an IPv6 dispatch
Jun 18 11:02:42 nsmaster named[50196]: 18-Jun-2020 11:02:42.728 general: error: reloading configuration failed: unexpected error

And after this, it stops responding for some zone but not all of them
and not always the recently added.

Of course the  ipv4 and v6 where not changed/added/remove  on the server and th configuration was correct it is generated by a script from a database).

After this, the only solution is to stop and restart bind. An other rndc reconfig produce the same error.

Someone tells me this may be a socket issue.
The freebsd pkg/port is build with --with-tuning=default

% /usr/local/sbin/named -V
BIND 9.11.19 (Extended Support Version) <id:905ec64>
running on FreeBSD amd64 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC
built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=default' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' '
 LDFLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with libjson-c version: 0.13.1
linked to libjson-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled

default paths:
  named configuration:  /usr/local/etc/namedb/named.conf
  rndc configuration:   /usr/local/etc/namedb/rndc.conf
  DNSSEC root key:      /usr/local/etc/namedb/bind.keys
  nsupdate session key: /var/run/named/session.key
  named PID file:       /var/run/named/pid
  named lock file:      /var/run/named/named.lock


I upgraded bind to 9.11.20, and restarted it with -U 21000 .
After looking at the source, I added query-source and query-source-v6 in
the configuration even if the server has "recursion no;" and has no slave.

Today, same problem:

Jun 24 08:52:39 nsmaster named[46662]: general: error: could not get query source dispatcher (213.36.252.194#0)
Jun 24 08:52:39 nsmaster named[46662]: general: error: reloading configuration failed: out of memory

top show no memory issue:

last pid: 16247;  load averages:  1.08,  1.80,  6.80   up 154+20:15:48 13:02:17
40 processes:  1 running, 39 sleeping
CPU:  1.6% user,  0.0% nice,  0.5% system,  0.1% interrupt, 97.8% idle
Mem: 4568M Active, 4978M Inact, 32G Wired, 580M Buf, 21G Free
ARC: 16G Total, 8811M MFU, 5160M MRU, 20M Anon, 471M Header, 1751M Other
     12G Compressed, 38G Uncompressed, 3.31:1 Ratio
Swap: 64G Total, 64G Free

...
14164 bind         59  52    0  5320M  5200M sigwai  32  21.1H   0.31% named
...

% limits
Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize             33554432 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked               64 kB
  maxprocesses            63709
  openfiles             1883583
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

# rndc status
version: BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
running on nsmaster.free.org: FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
boot time: Wed, 24 Jun 2020 08:11:43 GMT
last configured: Wed, 24 Jun 2020 11:02:40 GMT
configuration file: /usr/local/etc/namedb/named-custom.conf
CPUs found: 56
worker threads: 56
UDP listeners per interface: 20
number of zones: 144218 (0 automatic)
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/900/1000
tcp clients: 6/1000
TCP high-water: 40
server is up and running

We have only about 70000 zones but bind semmes to count them twice when
they are signed.

I have rebuild bind with tuning=large :

# /usr/local/sbin/named -V
BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
running on FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=large' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' 'LD
 FLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
linked to OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
compiled with libxml2 version: 2.9.10
linked to libxml2 version: 20910
compiled with libjson-c version: 0.13.1
linked to libjson-c version: 0.13.1
compiled with zlib version: 1.2.11
linked to zlib version: 1.2.11
threads support is enabled

default paths:
  named configuration:  /usr/local/etc/namedb/named.conf
  rndc configuration:   /usr/local/etc/namedb/rndc.conf
  DNSSEC root key:      /usr/local/etc/namedb/bind.keys
  nsupdate session key: /var/run/named/session.key
  named PID file:       /var/run/named/pid
  named lock file:      /var/run/named/named.lock

What can I do more to avoid this bug?

Is there a parameter or build option for such "big" server ?


Regards,
--
Laurent Frigault | Free.org - BookMyName.com - ONLINE SAS - Registar ID 74
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: random dispatch error after reconfig with bind ESV 9.11

Laurent Frigault
On Wed, Jun 24, 2020 at 01:11:25PM +0200, Laurent Frigault wrote:
> Hi,
>
> OS: FreeBSD 12.1 / AMD64
> 56 cores CPU
> 64G RAM
>
> bind ESV 9.11.19 from standard freebsd pkg
>
> We are runng an hiddenmaster bind server with DNSSEC zone "bookmyname.be" {

Ooops,

I wanted to write :

We are runing an hidden master bind server with DNSSEC zones all LIKE
this one:

> zone "bookmyname.be" {
>     type master;
>     file "custom/b/o/bookmyname.be/bookmyname.be";
>     notify explicit;
>     also-notify { 213.36.252.135; 62.210.98.15; 213.36.253.14; };
>     auto-dnssec maintain;
>     inline-signing yes;
>     key-directory "custom/b/o/bookmyname.be";
> };
>
> We have about 73000 zones , most signed.
>
> We periodically regenerate our configuration to add/update/remove zones.
>
> when needed, we use "rndc reconfig"
>
> Every few weeks we get the following error in the log  :
>
> Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: ./server.c:3881: unexpected error:
> Jun 18 11:02:41 nsmaster named[50196]: 18-Jun-2020 11:02:41.989 general: error: unable to obtain neither an IPv4 nor an IPv6 dispatch
> Jun 18 11:02:42 nsmaster named[50196]: 18-Jun-2020 11:02:42.728 general: error: reloading configuration failed: unexpected error
>
> And after this, it stops responding for some zone but not all of them
> and not always the recently added.
>
> Of course the  ipv4 and v6 where not changed/added/remove  on the server and th configuration was correct it is generated by a script from a database).
>
> After this, the only solution is to stop and restart bind. An other rndc reconfig produce the same error.
>
> Someone tells me this may be a socket issue.
> The freebsd pkg/port is build with --with-tuning=default
>
> % /usr/local/sbin/named -V
> BIND 9.11.19 (Extended Support Version) <id:905ec64>
> running on FreeBSD amd64 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC
> built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=default' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing '
  '

>  LDFLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
> compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
> compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
> linked to OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
> compiled with libxml2 version: 2.9.10
> linked to libxml2 version: 20910
> compiled with libjson-c version: 0.13.1
> linked to libjson-c version: 0.13.1
> compiled with zlib version: 1.2.11
> linked to zlib version: 1.2.11
> threads support is enabled
>
> default paths:
>   named configuration:  /usr/local/etc/namedb/named.conf
>   rndc configuration:   /usr/local/etc/namedb/rndc.conf
>   DNSSEC root key:      /usr/local/etc/namedb/bind.keys
>   nsupdate session key: /var/run/named/session.key
>   named PID file:       /var/run/named/pid
>   named lock file:      /var/run/named/named.lock
>
>
> I upgraded bind to 9.11.20, and restarted it with -U 21000 .
> After looking at the source, I added query-source and query-source-v6 in
> the configuration even if the server has "recursion no;" and has no slave.
>
> Today, same problem:
>
> Jun 24 08:52:39 nsmaster named[46662]: general: error: could not get query source dispatcher (213.36.252.194#0)
> Jun 24 08:52:39 nsmaster named[46662]: general: error: reloading configuration failed: out of memory
>
> top show no memory issue:
>
> last pid: 16247;  load averages:  1.08,  1.80,  6.80   up 154+20:15:48 13:02:17
> 40 processes:  1 running, 39 sleeping
> CPU:  1.6% user,  0.0% nice,  0.5% system,  0.1% interrupt, 97.8% idle
> Mem: 4568M Active, 4978M Inact, 32G Wired, 580M Buf, 21G Free
> ARC: 16G Total, 8811M MFU, 5160M MRU, 20M Anon, 471M Header, 1751M Other
>      12G Compressed, 38G Uncompressed, 3.31:1 Ratio
> Swap: 64G Total, 64G Free
>
> ...
> 14164 bind         59  52    0  5320M  5200M sigwai  32  21.1H   0.31% named
> ...
>
> % limits
> Resource limits (current):
>   cputime              infinity secs
>   filesize             infinity kB
>   datasize             33554432 kB
>   stacksize              524288 kB
>   coredumpsize         infinity kB
>   memoryuse            infinity kB
>   memorylocked               64 kB
>   maxprocesses            63709
>   openfiles             1883583
>   sbsize               infinity bytes
>   vmemoryuse           infinity kB
>   pseudo-terminals     infinity
>   swapuse              infinity kB
>   kqueues              infinity
>   umtxp                infinity
>
> # rndc status
> version: BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
> running on nsmaster.free.org: FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
> boot time: Wed, 24 Jun 2020 08:11:43 GMT
> last configured: Wed, 24 Jun 2020 11:02:40 GMT
> configuration file: /usr/local/etc/namedb/named-custom.conf
> CPUs found: 56
> worker threads: 56
> UDP listeners per interface: 20
> number of zones: 144218 (0 automatic)
> debug level: 0
> xfers running: 0
> xfers deferred: 0
> soa queries in progress: 0
> query logging is ON
> recursive clients: 0/900/1000
> tcp clients: 6/1000
> TCP high-water: 40
> server is up and running
>
> We have only about 70000 zones but bind semmes to count them twice when
> they are signed.
>
> I have rebuild bind with tuning=large :
>
> # /usr/local/sbin/named -V
> BIND 9.11.20 (Extended Support Version) <id:f3d1d66>
> running on FreeBSD amd64 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
> built by make with '--localstatedir=/var' '--disable-linux-caps' '--with-randomdev=/dev/random' '--with-libxml2=/usr/local' '--with-readline=-L/usr/local/lib -ledit' '--with-dlopen=yes' '--with-gost=no' '--without-python' '--sysconfdir=/usr/local/etc/namedb' '--with-dlz-filesystem=yes' '--disable-dnstap' '--enable-filter-aaaa' '--disable-fixed-rrset' '--without-geoip2' '--without-gssapi' '--with-libidn2=/usr/local' '--enable-ipv6' '--with-libjson=/usr/local' '--disable-largefile' '--with-lmdb=/usr/local' '--disable-native-pkcs11' '--disable-querytrace' '--enable-rpz-nsdname' '--enable-rpz-nsip' 'STD_CDEFINES=-DDIG_SIGCHASE=1' '--with-openssl=/usr' '--enable-threads' '--with-tuning=large' '--disable-symtable' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd12.1' 'build_alias=amd64-portbld-freebsd12.1' 'CC=cc' 'CFLAGS=-O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing ' '
 LD

>  FLAGS= -fstack-protector-strong ' 'LIBS=-L/usr/local/lib' 'CPPFLAGS=-DLIBICONV_PLUG -isystem /usr/local/include' 'CPP=cpp' 'PKG_CONFIG=pkgconf'
> compiled by CLANG 4.2.1 Compatible FreeBSD Clang 8.0.1 (tags/RELEASE_801/final 366581)
> compiled with OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
> linked to OpenSSL version: OpenSSL 1.1.1d-freebsd  10 Sep 2019
> compiled with libxml2 version: 2.9.10
> linked to libxml2 version: 20910
> compiled with libjson-c version: 0.13.1
> linked to libjson-c version: 0.13.1
> compiled with zlib version: 1.2.11
> linked to zlib version: 1.2.11
> threads support is enabled
>
> default paths:
>   named configuration:  /usr/local/etc/namedb/named.conf
>   rndc configuration:   /usr/local/etc/namedb/rndc.conf
>   DNSSEC root key:      /usr/local/etc/namedb/bind.keys
>   nsupdate session key: /var/run/named/session.key
>   named PID file:       /var/run/named/pid
>   named lock file:      /var/run/named/named.lock
>
> What can I do more to avoid this bug?
>
> Is there a parameter or build option for such "big" server ?

--
Laurent Frigault | Free.org - BookMyName.com - ONLINE SAS - Registar ID 74
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users