NSEC3 salt change - temporary performance decline

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

NSEC3 salt change - temporary performance decline

Bind-Users forum mailing list

Hello BIND users

 

Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).

 

Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 – 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.

 

Other nodes running NSD and Knot are not affected.

 

What could be the reason for the performance decline?

 

Best regards

 

Niels Haarbo

DK Hostmaster A/S


_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Ondřej Surý
Hi Niels,

> On 21 Jan 2020, at 15:43, Niels Haarbo via bind-users <[hidden email]> wrote:
>
> Hello BIND users
>  
> Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).

Just don’t do that, there’s no sensible reason to change salt that often (or ever).  I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.

> Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 – 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
>  
> Other nodes running NSD and Knot are not affected.
>  
> What could be the reason for the performance decline?

We are currently investigating performance degradation related to big IXFRs.  Do you use ixfr-from-differences in your BIND configuration?  You could try enforcing AFRX on salt change.

This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447

and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515

Ondrej
--
Ondřej Surý
[hidden email]

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Daniel Stirnimann
> Just don’t do that, there’s no sensible reason to change salt that often (or ever).  I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.

I agree that re-salting is kind of pointless (we still do it for .ch
though because so far I've been to lazy to change the code) but here is
one reference where it is recommended.

   The salt SHOULD be changed periodically to prevent pre-computation
   using a single salt.  It is RECOMMENDED that the salt be changed for
   every re-signing.

https://tools.ietf.org/html/rfc5155#appendix-C.1


>> What could be the reason for the performance decline?
>
> We are currently investigating performance degradation related to big IXFRs.  Do you use ixfr-from-differences in your BIND configuration?  You could try enforcing AFRX on salt change.

I use "max-journal-size" to force AXFR on big changes. A good value
depends on your zone size.

Daniel
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Jim Reid


> On 21 Jan 2020, at 15:59, Daniel Stirnimann <[hidden email]> wrote:
>
> I agree that re-salting is kind of pointless

So, just like NSEC3 then? :-)

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Ondřej Surý
NSEC3 is like a toilet window. You want it translucent, not transparent. For that purpose, it serves well.

--
Ondřej Surý — ISC

> On 21 Jan 2020, at 17:05, Jim Reid <[hidden email]> wrote:
>
> 
>
>> On 21 Jan 2020, at 15:59, Daniel Stirnimann <[hidden email]> wrote:
>>
>> I agree that re-salting is kind of pointless
>
> So, just like NSEC3 then? :-)
>
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/bind-users

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

RE: NSEC3 salt change - temporary performance decline

Bind-Users forum mailing list
In reply to this post by Ondřej Surý
Thank you all for the answers.

We do not use ixfr-from-differences on the actual zone, but on several others on the same server. Not sure how a BIND handles that scenario.

I will try to solve the problem by changing the max-journal-size. According to the docs https://kb.isc.org/docs/aa-01641 it cannot 'hurt' integrity to set a low value - but a value too low will affect performance.

If I can't find a solution by lowering the max-journal-size, I will disable NSEC3 salt changes.

Best regards

Niels Haarbo
DK Hostmaster A/S


-----Original Message-----
From: Ondřej Surý <[hidden email]>
Sent: Tuesday, January 21, 2020 4:41 PM
To: Niels Haarbo <[hidden email]>
Cc: [hidden email]
Subject: Re: NSEC3 salt change - temporary performance decline

Hi Niels,

> On 21 Jan 2020, at 15:43, Niels Haarbo via bind-users <[hidden email]> wrote:
>
> Hello BIND users
>  
> Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).

Just don’t do that, there’s no sensible reason to change salt that often (or ever).  I don’t know where the advice to change salt often comes from, but the advice has been wrong for so many years.

> Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 – 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
>  
> Other nodes running NSD and Knot are not affected.
>  
> What could be the reason for the performance decline?

We are currently investigating performance degradation related to big IXFRs.  Do you use ixfr-from-differences in your BIND configuration?  You could try enforcing AFRX on salt change.

This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447

and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515

Ondrej
--
Ondřej Surý
[hidden email]

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Klaus Darilion
In reply to this post by Bind-Users forum mailing list
Hello Niels!

Thanks for bringing this to attention. I have reported it before [1][2]
without response.

We see this regulary. AFAIS it happens actually always, but if the IXFR
is small, the performance decline is so short that you usually won't
notice it.

The bigger the zonechange ie NSEC3 change, full resigning ....* the
longer is the performance decline and you will notice it more often.

*we don't resalt or resign completele - but this is what several of our
TLD customers do.

I hope it will be fixed soon, we already test other software.

regards
Klaus


[1] https://lists.isc.org/pipermail/bind-users/2018-March/099814.html
[2] https://lists.isc.org/pipermail/bind-users/2019-March/101579.html


Am 21.01.2020 um 15:43 schrieb Niels Haarbo via bind-users:

> Hello BIND users
>
> Our DNSSEC signer changes NSEC3 salt every 30 days. The signer resigns all the relevant records and the zone is transferred using IXFR to the authoritative servers (6 nodes).
>
> Two of the 6 authoritative servers (BIND 9.11.13 and 9.11.14) are affected by a performance decline shortly after the change of salt. This has happened after the last 3 changes of salt and the period of performance decline is within 30 - 90 minutes. Most queries are dropped by the affected nodes during the period. The normal rate is between 1.000 and 1.500 queries/second.
>
> Other nodes running NSD and Knot are not affected.
>
> What could be the reason for the performance decline?
>
> Best regards
>
> Niels Haarbo
> DK Hostmaster A/S
>
>
> _______________________________________________
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
>
> bind-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/bind-users
>
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Klaus Darilion
In reply to this post by Ondřej Surý
Am 21.01.2020 um 16:40 schrieb Ondřej Surý:
> We are currently investigating performance degradation related to big IXFRs.  Do you use ixfr-from-differences in your BIND configuration?  You could try enforcing AFRX on salt change.
>
> This is currently tracked as https://gitlab.isc.org/isc-projects/bind9/issues/1447
>
> and associated feature request: https://gitlab.isc.org/isc-projects/bind9/issues/1515

thanks for giving priority to this issue.

Regards
Klaus
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

Re: NSEC3 salt change - temporary performance decline

Cathy Almond
In reply to this post by Klaus Darilion
On 29/01/2020 11:50, Klaus Darilion wrote:

> Hello Niels!
>
> Thanks for bringing this to attention. I have reported it before [1][2]
> without response.
>
> We see this regulary. AFAIS it happens actually always, but if the IXFR
> is small, the performance decline is so short that you usually won't
> notice it.
>
> The bigger the zonechange ie NSEC3 change, full resigning ....* the
> longer is the performance decline and you will notice it more often.
>
> *we don't resalt or resign completele - but this is what several of our
> TLD customers do.
>
> I hope it will be fixed soon, we already test other software.
>
> regards
> Klaus
>
>
> [1] https://lists.isc.org/pipermail/bind-users/2018-March/099814.html
> [2] https://lists.isc.org/pipermail/bind-users/2019-March/101579.html

FYI this will be fixed in the June 2020 BIND releases (in 9.11.20,
9.16.4 and 9.17.2):

https://gitlab.isc.org/isc-projects/bind9/-/issues/1834

Cathy
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users
Reply | Threaded
Open this post in threaded view
|

AW: NSEC3 salt change - temporary performance decline

Klaus Darilion-2
> -----Ursprüngliche Nachricht-----
> Von: bind-users <[hidden email]> Im Auftrag von Cathy
> Almond
> Gesendet: Dienstag, 9. Juni 2020 14:30
> An: [hidden email]
> Betreff: Re: NSEC3 salt change - temporary performance decline
...
>
> FYI this will be fixed in the June 2020 BIND releases (in 9.11.20,
> 9.16.4 and 9.17.2):
>
> https://gitlab.isc.org/isc-projects/bind9/-/issues/1834

Great news. Thank's for the work.
Klaus
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/bind-users