pdns_recursor does not perform recursive lookups on domain specific forwarders
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	squeeby
	Apr 13 2019, 9:19 PM

Description

Since dnsmasq was replaced with PowerDNS Recursor, recursive domain name resolution for explicitly configured forwarders for a particular domain does not function as expected.
That is, it doesn't behave how dnsmasq does.

Example:

With the following configuration, we see the expected recursive resolver behaviour:

VyOS gateway:

service {
    dns {
        forwarding {
             cache-size 1024
             name-server 2606:4700:4700::1111
             name-server 2606:4700:4700::1001
             name-server 1.1.1.1
             name-server 1.0.0.1
        }
    }
}

Host:

$ dig +short foo.com
23.23.86.44
$ dig +short www.foo.com
newdigiadmin-1201528726.us-east-1.elb.amazonaws.com.
54.165.87.12
52.73.176.251

But with explicit forwarders set for foo.com, only the origin and none subdomain records can be resolved:

VyOS gateway:

service {
    dns {
        forwarding {
             cache-size 1024
             name-server 2606:4700:4700::1111
             name-server 2606:4700:4700::1001
             name-server 1.1.1.1
             name-server 1.0.0.1
             domain foo.com {
                 server 172.23.1.25
                 server 172.23.1.28
             }
        }
    }
}

Host:

$ dig +short foo.com
1.2.3.4
$ dig +short www.foo.com
foo.com.
$ dig +short www.subdomain.foo.com
$ dig +short some.host.subdomain.foo.com
$

The old behaviour in dnsmasq was to use the specified forwarder for all recursive requests associated with the domain.
For example:

Host:

$ dig +short foo.com
1.2.3.4
$ dig +short www.foo.com
foo.com.
$ dig +short www.subdomain.foo.com
1.2.3.4
$ dig +short some.host.subdomain.foo.com
3.4.5.6

Details

Version: VyOS 1.2.0-rolling+201903090337

Related Objects

Mentioned In: rVYOSONEXc9787fb53c06: [dns-forwarding] T1333: handle domain forward and general recursion in one…
rVYOSONEX5886dd27cbc6: [dns-forwarding] T1333: handle domain forward and general recursion in one…
Mentioned Here: T1469: Create forward-zones-recurse entry instead of forward-zones when setting service dns forwarding

Event Timeline

squeeby created this task.Apr 13 2019, 9:19 PM

syncer changed the task status from Open to In progress.Apr 17 2019, 7:33 PM

syncer triaged this task as Low priority.

syncer edited projects, added VyOS 1.2 Crux (VyOS 1.2.3); removed VyOS 1.2 Crux.

pasik subscribed.Apr 22 2019, 3:11 PM

For completeness, this was discussed in slack:

dmbaturin [12:14 PM]
@squeeb Hhm. I'll check if there's a way to make pdns change its behaviour.

squeeb [12:14 PM]
I think it’s just forward_zones_recurse instead of forward_zones

dmbaturin [12:15 PM]
Could you confirm it by modifying the config by hand?

squeeb [12:15 PM]
I’ll give it a shot
Do I just kill and run the pdns_recursor process by hand?
or will it accept SIGHUP or something

dmbaturin [12:15 PM]
You can restart it via systemd

squeeb [12:15 PM]
oh! handy :slightly_smiling_face:

dmbaturin [12:16 PM]
We have been eliminating hand-controlled daemon restarts steadily, there still are instances, but pdns is not one of them.

squeeb [12:24 PM]
Yes that appears to work

# name-server
forward-zones-recurse=.=1.1.1.1;[2606:4700:4700::1001]
forward-zones-recurse=otbeach.com=172.23.1.25;172.23.1.28

Means I can resolve anything .otbeach.com, regardless to how deep the heirachy goes,
IE some.switch.somewhere.in.a.datacentre.otbeach.com resolves
Just doublechecking it’s still using one.one.one.one for all other queries
hmm, it’s not.
I should probably read the pdns-recursor manual
ah, syntax was wrong, apparently you separate each zone with a ,

# name-server
forward-zones-recurse=.=1.1.1.1;[2606:4700:4700::1001],otbeach.com=172.23.1.25;172.23.1.28

this appears to work as expected
So I guess, adding a keyword to the VyOS config, perhaps set service dns forwarding domain foo.com server <IP> recurse would move it to the forward-zones-recursive line instead of forward-zones

dmbaturin [12:37 PM]
Do you think making it non-recusrive by default has merits, or making it recursive can do any harm?
To be fair, I never noticed because domain-specific servers I used are always non-recursive.

squeeb [12:38 PM]
I can’t think of a usecase where it wouldn’t be more useful by having it recursive by default

dmbaturin [12:38 PM]
I think we should make it the default then.
So should we just replace forward-zones with forward-zones-recurse then?

squeeb [12:39 PM]
I guess so, you’d have to drop it on the same line as the default forwarders though, separated by commas
I’m no powerdns expert mind
with the latter example (with the two foward-zones-recurse lines), it appears that the last line takes precedence and it defaults to using root servers to perform lookups against zones not listed (anything else)

in vyos-1.2.0-rolling+201906250337 dns forwarding domain doesn't work. Only the nameserver from 'name-server' are used.
In /etc/powerdns/recursor.conf it lokks like this now:

# domain ... server ...
forward-zones-recurse=intern=172.16.1.1,internaldomain.com=172.16.1.1

...
...

# name-server
forward-zones-recurse=.=1.1.1.1

So there are 2lines 'forward-zones-recurse' and only the last one wins. I merged both lines manually in one comma separated, restarted pdsn-recursor and everything works.
So I can confirm what squeeb said.
Is there a workaround at the moment beside manually editing resolver.conf?

in vyos-1.2.0-rolling+201907031336 now the dns forwarding domain entries are completely missing in recursor.conf:

# ignore-hosts-file
export-etc-hosts=no

# listen-on
local-address=127.0.0.1,172.16.100.10

# domain ... server ...

# dnssec
dnssec=process-no-validate

I suggest to rise the priority of this task.

Please check wirh upcomming rolling release. There was a fix in T1469

I supposed something like that :-) I will test and report

Build complete

I just tested with a split dns config (so I can exlude the internal DNS server has resolved the external domain name I used for the test). It works again!

recursor.conf look like this now:

# name-server
forward-zones-recurse=.=1.1.1.1

# domain ... server ...
forward-zones-recurse=intern=172.16.10.1

So now the 'domain forwarding' line is after the 'name-server' line.

In my tests this task is resolved in this rolling. Thanks @c-po

Fixed by T1469 both in rolling and crux

c-po edited projects, added VyOS 1.3 Equuleus, VyOS 1.2 Crux (VyOS 1.2.2); removed VyOS 1.2 Crux (VyOS 1.2.3).Jul 3 2019, 10:13 PM

c-po moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.

c-po moved this task from Needs Triage to Finished on the VyOS 1.2 Crux (VyOS 1.2.2) board.

The problem is back in 1.2.0-rolling+201907110337 :-(
The line with the forwarders in recursor.conf is missing again.
@c-po
How can I reopen this task here?

c-po reopened this task as Open.Jul 11 2019, 3:23 PM

recursor.conf looks like this:

# name-server
forward-zones-recurse=.=1.1.1.1

# domain ... server ...

after that is end of file

How should the config look like?

There are no modifications to dns_forwarding.py after T1469 was implemented

strange

It worked as it looked like this:

# name-server
forward-zones-recurse=.=1.1.1.1

# domain ... server ...
forward-zones-recurse=intern=172.16.10.1

So the 'domain forwarding' line was after the 'name-server' line. It's still so but there is no 'domain forwarding' line anymore. The config in confif tree is still there

Can you share your configuration and the actual /etc/powerdns/recursor.conf file? And an expected /etc/powerdns/recursor.conf?

From the slack discussion, we found there should be a single forward-zones-recurse line as the last one takes precedence

Upstream resolvers are configured for each specific zone (including root (.) separated by commas, but on the same line.

test config looks like this:

service {
    dns {
        forwarding {
            cache-size 0
            domain intern {
                server 172.16.10.1
            }
            ignore-hosts-file
            listen-address 127.0.0.1
            listen-address 172.16.100.10
            name-server 1.1.1.1
        }
    }

resulting not working resolv.conf in 1.2.0-rolling+201907110337:

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=0

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=no

# listen-on
local-address=127.0.0.1,172.16.100.10

# dnssec
dnssec=process-no-validate

# name-server
forward-zones-recurse=.=1.1.1.1

# domain ... server ...

I don't know exactly the syntax of powerdns. But safest would be to have all in one line as @squeeby wrote:

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=0

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=no

# listen-on
local-address=127.0.0.1,172.16.100.10

# dnssec
dnssec=process-no-validate

# name-server
forward-zones-recurse=.=1.1.1.1,intern=172.16.10.1

After reading the PowerDNS docs this should be correct: forward-zones-recurse=.=1.1.1.1,intern=172.16.10.1 I will take a look.

c-po moved this task from Finished to In Progress on the VyOS 1.3 Equuleus board.Jul 14 2019, 7:58 AM

c-po moved this task from Finished to In Progress on the VyOS 1.2 Crux (VyOS 1.2.2) board.

c-po edited projects, added VyOS 1.2 Crux (VyOS 1.2.3); removed VyOS 1.2 Crux (VyOS 1.2.2).

c-po moved this task from Needs Triage to In Progress on the VyOS 1.2 Crux (VyOS 1.2.3) board.

The new implementation will behave as follows:

Catch-all forward to specific nameserver

vyos@vyos# show service dns forwarding
 listen-address 172.18.254.201
 name-server 1.1.1.1
 name-server 9.9.9.9

vyos@vyos# cat /etc/powerdns/recursor.conf | tail -n 7
# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=.=1.1.1.1;9.9.9.9

Multiple forwarding domains

vyos@vyos# # show service dns forwarding
 domain extern {
     server 172.18.2.2
 }
 domain intern {
     server 172.16.1.1
     server 172.16.1.2
 }
 listen-address 172.18.254.201

vyos@vyos# cat /etc/powerdns/recursor.conf | tail -n 7
# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=extern=172.18.2.2, intern=172.16.1.1;172.16.1.2

Multiple forwarding domains and catch-all forward to specific nameserver

vyos@vyos# show service dns forwarding
 domain extern {
     server 172.18.2.2
 }
 domain intern {
     server 172.16.1.1
     server 172.16.1.2
 }
 listen-address 172.18.254.201
 name-server 1.1.1.1
 name-server 9.9.9.9

vyos@vyos# cat /etc/powerdns/recursor.conf | tail -n 7
# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=extern=172.18.2.2, intern=172.16.1.1;172.16.1.2, .=1.1.1.1;9.9.9.9

c-po closed this task as Unknown Status.Jul 14 2019, 9:01 AM

c-po raised the priority of this task from Low to Normal.

c-po edited a custom field.

c-po mentioned this in rVYOSONEX5886dd27cbc6: [dns-forwarding] T1333: handle domain forward and general recursion in one….Jul 14 2019, 9:01 AM

I tested again with vyos-1.2.0-rolling+201907141109 but in recursor.conf the domain forwarding is missing (VyOS config is the same as in tests before):

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=0

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=no

# listen-on
local-address=127.0.0.1,172.16.100.10

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=.=1.1.1.1

strange, I just have to reboot again with same software version and now it looks good!!

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=0

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=no

# listen-on
local-address=127.0.0.1,172.16.100.10

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=intern=172.16.10.1, .=1.1.1.1

thanks @c-po

Is this what you expect?

Woah hang on.... we actually consulted the documentation on this?! That’s cheating

"cheat link here"
I tested and it works for me as expected. I also have a test record on the internal DNS server not in name space '.intern', so I checked everything not '.intern' goes to the DNS server 1.1.1.1. It seems the more specific wins.

fix backported to crux

c-po changed the task status from Unknown Status to Resolved.Jul 21 2019, 3:48 PM

c-po moved this task from In Progress to Finished on the VyOS 1.3 Equuleus board.

c-po moved this task from In Progress to Finished on the VyOS 1.2 Crux (VyOS 1.2.3) board.

c-po mentioned this in rVYOSONEXc9787fb53c06: [dns-forwarding] T1333: handle domain forward and general recursion in one….Jul 21 2019, 3:49 PM

@c-po
I have some new findings. When I upgrade to a new rolling release (tested with the one from 11.08.2019 and 12.08.2019), I do 'add system image ...', then reboot, then the forwarders are missing in recursor.conf:

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=.=1.1.1.1

then another reboot and the domain forwarders are there again:

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=intern=172.16.10.1, .=1.1.1.1

I just tested this 2 times with the same results.

c-po reopened this task as Open.Aug 12 2019, 1:33 PM

Using 1.2-rolling-201908171107 I configured:

set service dns forwarding domain example.com server '172.16.10.1'
set service dns forwarding listen-address '172.18.201.10'
set service dns forwarding name-server '1.1.1.1'

Which resulted in:

vyos@vyos# cat /etc/powerdns/recursor.conf

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=10000

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=yes

# listen-on
local-address=172.18.201.10

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=example.com=172.16.10.1, .=1.1.1.1

then I upgraded to: 1.2-rolling-201908171224

and after the first boot the file /etc/powerdns/recursor.conf looked like:

vyos@vyos# cat /etc/powerdns/recursor.conf

### Autogenerated by dns_forwarding.py ###

# Non-configurable defaults
daemon=yes
threads=1
allow-from=0.0.0.0/0, ::/0
log-common-errors=yes
non-local-bind=yes
query-local-address=0.0.0.0
query-local-address6=::

# cache-size
max-cache-entries=10000

# negative TTL for NXDOMAIN
max-negative-ttl=3600

# ignore-hosts-file
export-etc-hosts=yes

# listen-on
local-address=172.18.201.10

# dnssec
dnssec=process-no-validate

# forward-zones / recursion
#
# statement is only inserted if either one forwarding domain or nameserver is configured
# if nothing is given at all, powerdns will act as a real recursor and resolve all requests by its own
#
forward-zones-recurse=example.com=172.16.10.1, .=1.1.1.1

@Line2 can you please retest with the latest rolling ISOs? as I can not reproduce this

I just tested again.
Running on rolling VyOS-1.2-201908130337, same config as before, recursor.conf is ok.
Updated VyOS to rolling VyOS-1.2-201908190337, reboot, check recursor.conf. The domain forwarding is missing.
Reboot again, it's back in the recursor.conf.

Unfortunately I can not reproduce this issue

If this issue can be reproduced by someone else please open a new bug report. THX!

c-po closed this task as Resolved.Aug 25 2019, 5:22 PM

with vyos-1.2-rolling-201908270337-amd64.iso also fixed for me. Thanks

dmbaturin removed a project: VyOS 1.3 Equuleus.Sep 10 2021, 2:24 PM

pdns_recursor does not perform recursive lookups on domain specific forwardersClosed, ResolvedPublicBUGActions

Description

Details

Related Objects

Event Timeline

Catch-all forward to specific nameserver

Multiple forwarding domains

Multiple forwarding domains and catch-all forward to specific nameserver

pdns_recursor does not perform recursive lookups on domain specific forwarders
Closed, ResolvedPublicBUG
Actions