When /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, Mullvad doesn't detect that it cannot modify system DNS settings

[Mikaela]

Issue report

Operating system: Debian Testing

App version: 2020.5

Issue description

If /etc/resolv.conf is a symlink pointing to /run/systemd/resolve/stub-resolv.conf which is systemd-resolved's file not including search domains, Mullvad doesn't alert about it being unable to modify system DNS settings and connects normally while /etc/resolv.conf is unmodified. This doesn't happen when resolv.conf is either a normal file or a symlink somewhere else such as /usr/lib/systemd/resolv.conf which doesn't include search domains in which case Mullvad blocks connection saying that it couldn't manage system's DNS settings.

Steps to reproduce:

1. Disconnect from Mullvad, so resolv.conf can be modified.
2. sudo rm /etc/resolv.conf && sudo ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
3. Either connect Mullvad or reboot.
4. ln -l /etc/resolv.conf && cat /etc/resolv.conf
   • Expected result: /etc/resolv.conf should probably be a normal file and contain Mullvad's DNS resolvers.
   • What happens instead: the file is a symlink and contains nameserver 127.0.0.53

I hope this report is beneficial to you as I am unsure whether I should leave this unreported as from my point of view this is a workaround to #473 while #473 (comment) gives me an impression that you would consider this as a DNS leak bug.

[pinkisemils]

Have you checked if systemd-resolved ends up using the correct DNS servers? The daemon is able to manage DNS on Linux in 4 ways - by just writing to /etc/resolv.conf, by using the resolvconf binary, via the DBus interface for the NetworkManager and via the DBus interface for systemd-resolved.

[Mikaela]

I hadn't thought of that, but resolvectl status seems to have my configuration from /etc/systemd/resolved.conf.d/

Global
       LLMNR setting: yes                 
MulticastDNS setting: yes                 
  DNSOverTLS setting: yes                 
      DNSSEC setting: yes                 
    DNSSEC supported: yes                 
  Current DNS Server: 149.112.112.112     
         DNS Servers: 2620:fe::9          
                      149.112.112.112     
                      2620:fe::fe         
                      9.9.9.9             
          DNS Domain: ~.                  
          DNSSEC NTA: 10.in-addr.arpa     
                      16.172.in-addr.arpa 
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa 
                      18.172.in-addr.arpa 
                      19.172.in-addr.arpa 
                      20.172.in-addr.arpa 
                      21.172.in-addr.arpa 
                      22.172.in-addr.arpa 
                      23.172.in-addr.arpa 
                      24.172.in-addr.arpa 
                      25.172.in-addr.arpa 
                      26.172.in-addr.arpa 
                      27.172.in-addr.arpa 
                      28.172.in-addr.arpa 
                      29.172.in-addr.arpa 
                      30.172.in-addr.arpa 
                      31.172.in-addr.arpa 
                      corp                
                      d.f.ip6.arpa        
                      home                
                      internal            
                      intranet            
                      lan                 
                      local               
                      private             
                      test                

Link 7 (tun0)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: no                   
       LLMNR setting: yes                  
MulticastDNS setting: no                   
  DNSOverTLS setting: yes                  
      DNSSEC setting: yes                  
    DNSSEC supported: yes                  

Link 6 (yggdrasil)
      Current Scopes: LLMNR/IPv6
DefaultRoute setting: no        
       LLMNR setting: yes       
MulticastDNS setting: no        
  DNSOverTLS setting: yes       
      DNSSEC setting: yes       
    DNSSEC supported: yes       

Link 5 (wlan0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: yes                      
      DNSSEC setting: yes                      
    DNSSEC supported: yes                      
  Current DNS Server: 2001:998:20::20          
         DNS Servers: 2001:998:20::20          
                      2001:998:20::40          
                      192.168.5.1              
          DNS Domain: ~.                       
                      mhs                      

Link 4 (wwan0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: yes 
      DNSSEC setting: yes 
    DNSSEC supported: yes 

Link 2 (eth0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: yes                      
      DNSSEC setting: yes                      
    DNSSEC supported: yes                      
  Current DNS Server: 2001:998:20::40          
         DNS Servers: 192.168.5.1              
                      2001:998:20::20          
                      2001:998:20::40          
          DNS Domain: ~.                       
                      mhs                      

---

Edit: here are my configs

└┌(%:~)┌- tail -n +1 /etc/systemd/resolved.conf.d/*
==> /etc/systemd/resolved.conf.d/everywhere.conf <==
[Resolve]
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
Cache=true

==> /etc/systemd/resolved.conf.d/quad9-strict.conf <==
[Resolve]
DNS=2620:fe::9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 9.9.9.9#dns.quad9.net
Domains=~.
DNSSEC=true
DNSOverTLS=true
Cache=true

(and now I unrelatedly realise that I need to rename everywhere.conf to 00-everywhere.conf, because the later file probably takes priority)

[pinkisemils]

Oh, and this is whilst you are connected, right? This seems bad - does DNS actually work? Because it shouldn't. You could send a problem report via the app so that we may read the daemon logs, otherwise I can try and reproduce the issue on my own.

[Mikaela]

Yes, this is when connected and https://am.i.mullvad.net/ gives a warning on failing to check for DNS leaks (and this particular IP being blacklisted apparently) and DNS works. I sent a problem report referencing this issue with optional email as my name at myname dot info, but I don't know if it would be more helpful to reproduce the issue locally.

[pinkisemils]

Would you mind testing to see whether any of the global DNS servers in your config are reachable?
dig example.com @149.112.112.112 should time out. Whilst we still fail to set DNS, I want to make sure that hostname lookup works on that machine only because you're allowing LAN traffic and thus the resolvers on your local network are being used. Otherwise, you'd be leaking DNS packets out to the internet.

[Mikaela]

dig example.com @149.112.112.112 should time out.

Times out, however 149.112.112.112:853 works as far as I can tell, I don't know any dig for DNS-over-TLS equivalent or if dig supports it with some flags.

[pinkisemils]

We currently do not block DNS-over-TLS, so that is why everything continues to work normally. I will however investigate why the daemon silently fails to set the appropriate DNS config for tun0.

[pinkisemils]

With a default install with Gnome and systemd-resolved enabled and configured as above, and being connected to a tunnel, I get to a point where resolvectl status returns:

Global
       LLMNR setting: yes                 
MulticastDNS setting: yes                 
  DNSOverTLS setting: yes                 
      DNSSEC setting: yes                 
    DNSSEC supported: yes                 
  Current DNS Server: 149.112.112.112     
         DNS Servers: 2620:fe::9          
                      149.112.112.112     
                      2620:fe::fe         
                      9.9.9.9             
          DNS Domain: ~.                  
          DNSSEC NTA: 10.in-addr.arpa     
                      16.172.in-addr.arpa 
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa 
                      18.172.in-addr.arpa 
                      19.172.in-addr.arpa 
                      20.172.in-addr.arpa 
                      21.172.in-addr.arpa 
                      22.172.in-addr.arpa 
                      23.172.in-addr.arpa 
                      24.172.in-addr.arpa 
                      25.172.in-addr.arpa 
                      26.172.in-addr.arpa 
                      27.172.in-addr.arpa 
                      28.172.in-addr.arpa 
                      29.172.in-addr.arpa 
                      30.172.in-addr.arpa 
                      31.172.in-addr.arpa 
                      corp                
                      d.f.ip6.arpa        
                      home                
                      internal            
                      intranet            
                      lan                 
                      local               
                      private             
                      test                

Link 4 (tun0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes                      
       LLMNR setting: yes                      
MulticastDNS setting: no                       
  DNSOverTLS setting: yes                      
      DNSSEC setting: yes                      
    DNSSEC supported: yes                      
  Current DNS Server: 10.10.0.1                
         DNS Servers: 10.10.0.1                
          DNS Domain: ~.                       

Link 2 (enp1s0)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: no                   
       LLMNR setting: yes                  
MulticastDNS setting: no                   
  DNSOverTLS setting: yes                  
      DNSSEC setting: yes                  
    DNSSEC supported: yes  

And resolvectl query example.com returns the following:

example.com: 93.184.216.34                     -- link: tun0
             2606:2800:220:1:248:1893:25c8:1946 -- link: tun0

-- Information acquired via protocol DNS in 387.5ms.
-- Data is authenticated: yes

So I cannot reproduce the reported issue. I tried this with both NetworkManager and just a plain DHCPCD service. Are you using systemd's networkd by any chance? Is there any other config that I should be aware of? I can imagine that there's some config option that would force DNS over TLS, then the daemon's RPCs to add plain DNS servers to tun0 might get ignored.

It might be the case that the daemon should verify if the config it's trying to set is actually set, via DBus. But for now, I still can't reproduce the issue.

[Mikaela]

I am using NetworkManager. So you don't get a DNS leak on https://am.i.mullvad.net/ detected? What does your /etc/resolv.conf look like, it should be a symlink to /run/systemd/resolve/stub-resolv.conf ?

I get the same result with resolvectl query example.com, but it doesn't tell me what DNS server it's using to get that result.

[Mikaela]

I thought about this a bit more and remembered/found https://powerdns.org/useful-names/ which tells where they are receiving the DNS queries from and I see addresses belonging to Woodynet aka Quad9.

└┌(%:~)┌- resolvectl query whoami.lua.powerdns.org -t TXT 
whoami.lua.powerdns.org IN TXT "2620:171:f9:f0::2"          -- link: tun0

-- Information acquired via protocol DNS in 118.5ms.
-- Data is authenticated: yes

└┌(%:~)┌- resolvectl query whoami.v4.powerdns.org -t TXT
whoami.v4.powerdns.org IN TXT "74.63.25.249"                -- link: tun0

-- Information acquired via protocol DNS in 823.6ms.
-- Data is authenticated: no


└┌(%:~)┌- resolvectl query whoami.v6.powerdns.org -t TXT
whoami.v6.powerdns.org IN TXT "2620:171:f9:f0::9"           -- link: tun0

-- Information acquired via protocol DNS in 276.5ms.
-- Data is authenticated: no

[pinkisemils]

You are correct, mullvad's DNS servers will not be used. I presume this is due to DNSOverTLS being set to yes. If I run resolvectl dnsovertls tun0 off, resolvectl query -t TXT whoami.lua.powerdns.org returns whoami.lua.powerdns.org IN TXT "2a03:1b20:5:f011::a09d" -- link: tun0, which is a Mullvad owned IP address. And yes, I've symlinked /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf.

This is probably the culprit here. As dirty as it may feel, I think disabling that setting until we support DNS over TLS in our tunnels (I'm not sure if we have any plans to do so). But this will still not fix the DNS leaks entirely. In this case, we'd probably need to either do what always ends in bloosdhed - record the current DNS config to a file and configure systemd-resolved exactly the way we want it, or we could update our firewall to block DNS over TLS. This still allows for DNS over HTTPS, which can still cause leaks (but it's far harder to block).

Technically, we've designed the DNS config management systems to only apply the correct DNS config so that DNS works, not to prevent leaks from happening, the firewall is supposed to help with that. Of course, with DNS over TLS and DNS over HTTPS, that is no longer enough, so we'll have to investigate how best to deal with this. Personally, I feel this isn't as bad as leaking plain text DNS packets outside of the tunnel, and in this case, the encrypted DNS requests are still sent through the tunnel (otherwise DNS would time out). The leak of course here is that we aren't ensuring that your DNS requests are being sent to our servers, which is bad.

Perhaps you personally have an opinion on where the line should be between enforcing privacy focused configuration and respecting what the user has personally configured?

[Mikaela]

I am not sure on blocking DNS-over-TLS while it would be consistent with blocking normal DNS and the dedicated port helps. What I want to do is to have DNSSEC validation locally, so I don't have to trust Mullvad's DNS servers to do that and use trust-ad in resolv.conf as it was intented. Regarding DNS-over-HTTPS, I am currently not aware of any implementation of it that performs DNSSEC validation locally (without chaining either systemd-resolved or unbound to do that) so on computers I see it kind of as a step backwards.

I guess the best case scenario for me would be having custom DNS (#473) and if I opted to use Mullvad's DNS servers, the app could possibly perform a DNS test and show a warning about Mullvad's DNS not being used and giving an option to not block connectivity.

[chandradeepdey]

Now broken on Arch too. archlinux/svntogit-packages@e3dba1b

Is there any workaround other than building systemd without the flag? On my PC I just get "Failed to set system DNS" and Mullvad goes into blocking state. There is no tun0 to do resolvectl dnsovertls tun0 off on.

[chandradeepdey]

Oh and trying to report the error from the app fails as well. It asks me to hit disconnect on the main screen, but I already disconnected and disabled "Always require VPN".

[Mikaela]

I didn't understand your comments earlier, but now updating systemd from 245.6-2 (when I opened the issue) to 245.7-1 (which adds the trust-ad option to resolv.conf) I started getting the same "Failed to set system DNS".

I wonder if that is a different issue from this though and I am not sure if I should close this issue for it being upstream or somehow fixed by a newer systemd version, but that fix probably isn't in too many stable releases of distributions with systemd.

[chandradeepdey]

Oh, you are probably right about my issue being different then. I saw in logs that Mullvad tried to parse the resolved symlinked /etc/resolv.conf as a static /etc/resolv.conf and failed to parse the file. Then I found this issue and saw Arch's changelog so I thought it might be the same issue.

[pinkisemils]

It seems like a different issue. I'll have a look at it. I presume you have configured your system to use systemd-resolved? Are there any other alterations that I should be aware of?

[pinkisemils]

Can reproduce it, will look into it. Looks like the systemd-resolved interface broke and we're falling back to managing DNS via writing directly to /etc/resolv.conf. I'll try and fix the issues with systemd-resolved.

[cljoly]

If it helps, I’ve got a similar problem of the client failing to change dns servers (on Archlinux, with /etc/resolv.conf symlinked to /run/systemd/resolve/stub-resolv.conf as well). Logs from

[talpid_core::tunnel_state_machine::connected_state][ERROR] Error: Failed to set DNS
Caused by: Error in static /etc/resolv.conf DNS monitor
Caused by: resolv.conf at /etc/resolv.conf could not be parsed
Caused by: option at line 17 is not recognized

And content from /etc/resolv.conf (from line 16):

nameserver 127.0.0.53
options edns0 trust-ad
search default

After removing the last two lines, it worked fine. Removing only of options edns0 trust-ad or search default was not enough though.

[Mikaela]

And as what changed between the versions was trust-ad being added there, I guess the Mullvad client cannot handle it being there instead of just options edns0 and just removing that word would make it work at the cost of breaking things like OpenSSH's VerifyHostKeyDNS?

[cljoly]

Removing trust-ad works indeed. Thanks!

[1player]

Removing trust-ad works indeed. Thanks!

Not for me, as any change to /etc/resolv.conf gets immediately reverted as soon as (or just before) the VPN is started, so still fails with that same error.

[cljoly]

Well, if I enter disconnected state (mullvad disconnect), then write the file without trust-ad and quickly run mullvad connect, it works for me.

Edit: This small one-liner works for me:

mullvad disconnect; sudo sed -i 's/options edns0 trust-ad/options edns0/' /etc/resolv.conf; mullvad connect

But granted, it is a bit of a hack, this should get better with the next release. 🤞

[Flat]

Can reproduce it, will look into it. Looks like the systemd-resolved interface broke and we're falling back to managing DNS via writing directly to /etc/resolv.conf. I'll try and fix the issues with systemd-resolved.

I believe it's falling back to managing DNS via /etc/resolv.conf due to this call https://github.com/mullvad/mullvadvpn-app/blob/master/talpid-core/src/dns/linux/systemd_resolved.rs#L97 which calls resolv_conf::Config::parse(resolv_conf_contents) and fails due to options edns0 trust-ad not being a 'valid' option in the resolv-conf crate. hickory-dns/resolv-conf#21

Is it really necessary to parse resolv.conf for systemd-resolved if you already check for the symlink and can interact through dbus? Additionally it only checks ipv4 and not ipv6.

[Flat]

I've created a PR in the upstream resolv-conf library that fixes the issue of trust-ad, I've tested a build with my PR and mullvad-daemon shows that systemd-resolved is being used to manage DNS. hickory-dns/resolv-conf#22

Although this doesn't address the original bug of this thread.