I need to mess with my local DNS resolver for unrelated reasons. Is there a way, in 2026, to setup DNS in a federated way?
I could point to ISP DNS, or 1.1.1.1 / 8.8.8.8 / etc but those are all centralized, and are all subject to surveillance and censorship laws.
DNSSEC, etc, are something, but they rely on CAs. I mostly just want some sort of system that's outside centralized control, but that can be externally audited for trustworthiness.
Noting this article skips over any discussion about why, or potentially (in the eyes of the law at least) legitimate reason an ISP might do this.
Many ISPs in many countries are forced to do this for certain enforcement action that gets put upon them by their respective states for things such as age gating, censorship, monitoring, it varies state by state.
And to be clear this is in no way an endorsement of the practice, and I am confident no ISP WANTS to do this.
I am speaking here as the owner of a UK ISP (who does not currently do this).
If you're a UK ISP owner, you're almost certainly a RIPE NCC member, given the UK is within its regional scope. Therefore, you've probably spoken to other RIPE NCC members via mailing lists or meetings at least once. Since I also represent a provider that is a RIPE NCC member, I can object that not everybody shares the same ideology you noted. There are some providers that actually speak in favor of such over-regulation and censorship. Everybody can read and/or participate in the RIPE Discussion List, though, and you will be amazed by some responses from ISPs that speak in favor of such measures.
But the good news is that at least the majority (for now) is against it.
I speak for another RIPE NCC member when I say that that ideology is limited to a very small minority.
Even outside ideology, plain business sense makes ISPs not want to deal with censorship & co. It's work to do that, hence costs money. But there are also ISPs resisting out of ideology and civic duty.
I hijack DNS in my LAN and block popular DoH servers so they would have to fall back to my DNS. It is a good way to block ads for smart TVs and services that try to use DoH to bypass DNS level ad-blocking.
There were certainly a number of ISPs in the UK that tested ways to inject ads in content in the 2000s, there was quite a hoo-hah about it in some forums. That was usually by modifying HTML content in transit via transparent proxies though, not DNS poisoning, but it shows that the intent exist{ed|s}. This is one of the reasons that HTTPS-everywhere is a good idea even if you aren't concerned about snooping in general.
AT&T still does this, if you're using their (intentionally terrible) standard gateway device the default DNS will return a "search page" instead of NXDOMAIN which predictably has all sorts of negative impacts on non-browser devices but the wastes of oxygen responsible for this don't care as long as they see that delicious ad revenue.
Some mobile phone network carriers would inject certain PII (such as your mobile number) into HTTP request headers sent to ad networks. I'm sure they still do it, but by requiring the site to have some extra request to be made instea.
Some carriers also passed info in these headers to be used to make billing calls to the Telco, such as subscribing to premium services with minimal (if any) consent from the user.
I got stung by this years ago. An ad for mobile ringtones showed up on some news site I visited on my phone, I scrolled past without clicking. Next minute I get a text telling me I've subscribed signed up to a (iirc) $25/month ringtone plan and that it'd show up on my phone bill. It took an hour on the phone to my carrier to block the charges and get a refund. The ringtone company kept messaging me for a year that I needed to contact them and "fix" the issue.
I dunno what was in a ringtone subscription. I used to work at a ringtone company, and it was $1-$3 per ringtone ($1 for monophonic, $2 for polyphonic and $3 for wav/mp3/whatever; which yeah, is more than a high quality download from iTunes... But $25/month is a ton of money.
Even with free ringtones, I'd send myself about three if I got a new phone. These days, I just keep a copy of the Nokia dubstep ringtone on hand in case I turn the ringer on, but it doesn't happen.
As someone who was sometimes using http as a back end transport for "edi" data (hey at the time it was usually ftp in that industry so no worse, but at least http got around nat problems than even pasv didn't), one time in the mid 00's I was in a coffee shop and just speechless flabbergasted outraged to discover the contents of a random web page I happened to know the contents of get some extra html and js injected into it from the isp (the isp not the coffee shop).
"WHO THE F gave them the right to modify the data in my tcp session!!!!".
Spying on it I assumed but modifying one byte of it? A naively written edi that did not expect that would be completely destroyed.
Probably shouldn't have been so stupid for 15 years already by that point. There is no such thing as the right, there is only the ability.
If someone has the ability to do something, they will. Reputation, right & wrong, none of that actually exists or matters.
I didn't track down if the mechanism was based on dns. Could have been but I don't see why it would have had to be. But I was also not attempting to use any non default dns on that device at that time anyway. Obviously since they are trasporting unencrypted data they could modify it any time simply because it's in their hands.
dns would just be one handy low-hanging-fruit way to detour the data to a convenient dedicated purpose server to do the work. But not special or necessary at all. If they want to do the same thing without hijacking dns, they can with almost not even any difference in total effort.
Last I checked Verizon FIOS still has the opt-out. You have to go into your account preferences and manually disable it because spying is on by default.
This is a good reminder to login and take a look, I wouldn't put it past them to change the setting name and reset the default.
edit: apparently Verizon has zero record of the username I've used for my FIOS account for years (and have saved in my password manager). Their account reset also fails with an error. What a dumpster fire of a company.
Yep, I've been in Verizon online account purgatory for awhile now. They merged my mom's with mine and nothing works to reset or restore online account access for even bill pay. I have to push their payment out via billpay. I really need to move that phone number over to voip, I kept the classic POTS line as a type 1 diabetic I like that it stayed on during both NYC blackouts in 2003 and 2012. But now I'm in a fiber only building and that redundancy is mostly moot.
Thank you for this. Just opted out. They certainly made that page as hard as possible to find. Figuring out how to change my routers DNS server was also made needlessly complex.
I don't use their resolver at all. Their opt-out was more of an all-encompassing "let us take whatever data we can manage to get" than DNS specifically.
AT&T has been well-known to do this, since they are violently opposed to net neutrality and they want to force you to only use their approved paid partners for everything. They do this on all their internet services.
Netflix has always listed AT&T as their number one worst offender when it comes to net neutrality.
IIRC, early into ATT’s US fiber / gigabit rollout, they wanted an additional $40 / month to forgo collecting and selling your browsing history (DNS queries).
this is valid, in NL most DNS is also heavy monitored and intercepted to do blocks if mandated etc. if you want to get around it u can tunnel ur tlsdns thingy to another place and they essentially cannot intercept it, they can still block the IPs it resolves to so its hit and miss depending on how the other end is implemented.
its usually not nefarious, that is just the simpelest (and usually, not always) wrong explanation.
people who think this is not normal probarbly didnt experience what kind of shit u could find on the pre google pre monitored internet. for sure shit u do not want your kids to find. if u dont understand, then likely u dont have kids and very weird fetishes...
One massive reason is they redirect requests for videos to closer (or local) caches of videos for netflix (directly from youtube placing it with a large enough ISP), or local caching at the ISP level so less traffic hits the internet.
If you've ever been hit with a "webpage isn't updating" issue when someone else can see it, often it's different DNS servers.
It's best to never use the ISP's DNS servers, regardless of the reason why they might do what they do.
A reminder that the average consumer device doesn't actually validate DNSSEC in its default configuration: it relies on its recursive resolver to set a bit on the response saying "I validated DNSSEC, things are all good". In that configuration, DNSSEC gets you no protection against this.
One of the reasons a variety of providers have been pushing for DoH is because DoH fully closes the risk of ISP MITM without needing individual site operators to do anything. This is why you're seeing folks like Firefox and Chrome enable DoH by default.
> One of the reasons a variety of providers have been pushing for DoH is because DoH fully closes the risk of ISP MITM without needing individual site operators to do anything.
... by replacing it with Google/Cloudflare MITM with whom the user doesn't even have a business relationship or even shared jurisdiction.
Introducing what ISPs do with DNS information is helpful to understand how encompassing DNS self-hosting needs to be thought about - DNS requests are important to consider the appropriate encryption for.
From the homelab side, increasingly with the automated attacks on systems, it seems wise to keep all services off the internet directly and put it behind one or more service to keep them safe. From that perspective, setting up something like Tailscale can also help mitigate this kind of issue.
They don't mess with DNS to do that. The video provider points them to the appropriate cache. And it's usually only for video content, not the whole web app.
Why do you assume the others have same ethical compass as you? I hope everyone has the same compass but we have seen again and again trust being broken.
ISPs talk to each other, and people move jobs between them. It's not a uniform community but you certainly get a sense of the general spread of opinions after a while.
There are of course outliers but most people I know in the field care as much about personal rights & freedom as anyone else. Maybe even a small bit more.
My workplace is messing with DNS as well, probably to filter specific websites, and I would not mind it since for the context of my usage that is perfectly fair.
But their DNS responses have latency spikes every couple of minutes of 1-2 seconds that was driving me crazy. I simply enabled DNS over HTTPS in Firefox and everything feels so smooth now.
I have no idea how my coworkers seem to be so insensitive to it.
For companies with ISO-27001, Web Filtering is a control recently introduced. Protective DNS can tick few boxes and genuinely keep your network a little bit more secure (and ad free).
As an employee I'd be against it, but company device so what can you do. As the one managing the security and ISO process, it made my life a lot easier from both auditing and "why is my computer acting funny lately" perspectives. Showing my own user account having triggered an ad-based block, and the company-wide malware block rate was enough to satisfy the auditor that we has appropriate controls in place for a small company and move on to the next item.
Provided you take the stance of everyone is trusted to do their jobs, so we only need to block ads, piracy and malware lists and don't retain logging except where a block has occurred for auditing, I think it strikes a reasonable middle ground.
If you're not working on something highly secretive and valuable (with a salary to match) then they're doing you a favor by flipping out over something so stupid.
They might be proxying all your traffic through a "WAF". I've noticed that even quite a lot of regular websites linked on HN on "non standard" domains are simply unresolvable on my work network.
Ok, free tech support I guess. It could be shitty DNS provisioning, that's pretty common. But are you using wireless? Are you familiar with Buffer Bloat generally, let alone some of the other implicit shenanigans in "traffic shaping"?
A few years ago the cognitive dissonance of people saying "DoH is faster" to the same DNS server(s) started really bugging me: is it really DoH, or is it simply TCP vs UDP? So I wrote a little thing I can install on my laptop to do DNS forwarding over plain old TCP (originally no encryption): https://github.com/m3047/tcp_only_forwarder
Golly, on my wifi TCP was certainly more reliable than UDP. It's due to how wifi devices prioritize and buffer traffic. Maybe they should implement / enable DoH on their own servers.
On the other hand if all of your access from the workplace network to web resources is proxied (and since DoH is a web service), they may not care.
Things are crazy now. Our TVs snapshot the stuff we watch; our telephones, watches, and TVs are constantly listening what we say; our PCs have undocumented "remote administration" devices that we can't opt-out; every internet service out there is stealing every data that it physically can and selling it to a long list of organizations, including some military ones...
Hello fellow time traveller, I also come to you from 5 - 10 years ago.
Back in this time children and families playing AR games are actually just data collection for GPS-denied military drones [1]. ~70% of the world's population voluntarily carry continuously connected GPS trackers, multi-camera, multi-microphone devices with fairly advanced sensor suites and send this into a plethora of 'social' platforms. This includes when they're jogging/tracing the outline of aircraft carriers [2] and military bases [3]. Vehicles are internet connected, laughably insecure and record every movement, and the movement of everything else in their vicinity [4]. And, for some reason, everything in your house is 'smart' and the building you work in is filled with 'advanced IoT' sensors and high precision wifi-tralateration that provides a firehouse of data to countless third parties.
We've been steadily descending for a long time and seem intent of speed-running dystopia.
That is regulation! You do actually have to win the argument for liberalism in public if you don't want your fellow citizens to keep trying to stop you doing this.
Telecom Italia locks my DNS server (I cannot change it).
I ended up using PIHole as DHCP, to drive home devices to PIHole, and control my privacy.
The trick is simple: disable DHCP on your router, and activate it on PIHole.
As a bonus, PIHole on my RasperryPI2 is way faster than the little chip on my provider Router, and can be patched automatically with unattended upgrades.
And if you can't disable DHCP or set the router in bridge mode, you can just add another router (that you control) between your ISP device and the rest of your network. I've been running this setup for over a decade and didn't run into any noteworthy issues. Leave the ISP-infested router outside your network.
The whole point of ad companies like google pushing DoH is to bypass things like pinhole and take power away from you, the owner, and give it to them, the trillion dollar advertising company.
Note that there are ISPs that actively hijack traffic to UDP port 53 further inside their network and transparently maps that onto their internal DNS servers.
Does anyone know if T-Mobile does this too? I'm considering moving fiber to them instead of AT&T, but have stayed mostly due to "better the devil you know..."
I'll throw my $.02 in here since this is getting some traction: DNSSEC validation should be used, no exceptions, no excuses. If you're running a local resolver, turn on stub validation for DNSSEC - it prevents anyone (even a DOH/DOT upstream forwarding cache) from lying to you without your knowledge. If you run a domain name: turn on DNSSEC. It's 2026 - not 2012. DNSSEC works great, we validate it on 100% of our queries, and is one of the tools needed to stop people from meddling with the DNS. https://circleid.com/posts/common-beliefs-about-dnssec-put-t...
Also, encrypt everything. DOT/DOH/DOH3/DOQ - leave no packet available for observation. Criminals, nosy network snoopers, and governments either want to know what you're doing or they want to break the integrity of the internet for profit or political reasons, and it is work to keep them out of your business.
context: I work at Quad9, one of the larger public resolvers.
The point of running a local unbound is indeed to turn on DNSsec validation. After that, it's the domains' fault for not using DNSsec. But at least your own stuff won't be tampered with, because you use DNSsec on your own domains, right? Right?
:-(
Coincidentally, a bunch of free wifi stuff blocks the root key queries unbound does at startup, resulting in nothing working until you get those records in there somehow (e.g. load_cache).
DNSSEC as designed is pretty useless. It was designed in an era where computers could not be relied upon to do cryptographic validation of something like DNS records, so that duty was delegated to DNS servers like the ones ISPs ran, where records could be cached.
There are also potential downsides (like someone being able to enumerate your subdomains, if you care about such a thing) and it's often switched off by default on operating systems consumers use.
However, it's the only thing that validates DNS records at this moment. That alone makes it worth enabling, in my opinion. For many domain owners, you only need to tick the checkbox in your registrar's domain control pane. If you use a modern DNS server, it's also one config flag and maybe copy/pasting a handful of fields away.
I'll gladly turn off DNSSEC the moment we come up with something better. DoT/DoH doesn't solve the same problem and no real solutions seem to have been proposed. That said, modernisation efforts are still being done. Smaller keys and modern cryptography rather than the massive RSA ones for instance.
Another approach that we analysed in more detail is the distribution of the host key fingerprints
via a special resource record. This removes any user interaction, potential human error, or
presumed configuration of a user’s device because OpenSSH-client can query the DNS and do the
comparison. It only requires the administrator to publish a host’s fingerprints as a set of SSHFP
records, which can be securely obtained (DNSSEC). If the latter holds, MITM attacks are mitigated,
and overall security is improved. [1]
I have never been able to get my head around the idea that the DNS should have anything whatsoever to do with my SSH fleet. The DNS exists to solve the problem of introductions between unaffiliated network endpoints. To a first approximation nobody SSH's to a random server. You don't need any third-party PKI to get optimal SSH security!
SSHFPS is what I should be using in my homelab if it weren't for the awful split-DNS setup I have!
I personally prefer the theoretical advantages of SSH certificates, but those don't have CRL support (let alone an ACME alternative) as far as I know, so that's not exactly perfect either.
I want DNSSEC('s goals and ambitions) to succeed, but having a "trusted" ISP server that sets a "trust me bro" bit is the design flaw that kills the original design.
There's a patch for openssh to use the ldns library and do its own DNSsec validation. A few distros apply it to their packages, though I'm not sure what the current status is. It never got accepted into openssh sadly.
I can't speak about the strength or weaknesses of the whole concept.
Out of curiosity I enabled DNSSEC checking beginning this year. I mostly noted nothing in my browsing. However, with 2 services I noted obscure problems that worked only partially. Some Jira plugins and Swedish railway SJ. I needed to excempt some of their domains from being checked to use their services. Of course you waste a couple of hours every time it happens because you don't think of DNSEC first when some web-based services behaves weirdly. There might be more requests not served to my browser, but I just never noticed or bothered to debug.
So not fit for general use. And even as an experienced developer it's not easy to understand why an exception would be needed. A bug in systemd? Incorrect data in DNS? A real attack?
Edit: I don't use standard DNS on port 53 at all. Blocked outgoing requests on that port. Everything DoH or DoT. Of course I need to trust those endpoints at least in repect to confidentiality. And most domains are unsigned. At least for those unsigned ones I need to just trust also in respect to integrity.
DNSSEC failures need to be louder. If websites that fucked up their DNSSEC would get the same error messages as websites that fucked up other parts of their security, people wouldn't be so eager to turn off security festures as workarounds.
Give SJ the option to turn off DNSSEC and their website will stay broken. Give people a "someone might be tampering with your connection" popup and they would start caring.
There are also other issues with DNSSEC that need working out, but I still believe it does much more good than it does bad. Leaving everything in the background with an easy-to-disable setting makes it next to useless.
Full ACK. As a part of this, DNSsec validation needs to be as close to the client app as possible and reasonable; definitely never off-host. (Caches can do additional validation, but that makes the error reporting complicated.)
corollary: the AD bit should never be relied on on anything actually on the network. Maybe on localhost, but even there it's questionable in UX results.
It's about time to question the whole off-host DNS resolver thing entirely. Fine if you want to have a cache, but the cache should return the whole record chain starting from the root in that case, so the client can validate it. Also fine to have an on-host cache that caches validation status.
Can and should! Unbound is very easy to set up, if you know how to administer Linux systems at all. It also acts as a local cache and I noticed a small improvement to page loading time because of having the cache. Otherwise there is no local cache - every single call to getaddrinfo sends a network request to your configured DNS server.
People should just say what they mean here. DNSSEC is in fact a server-to-server protocol, and your response is "yes, you should just run your own servers".
Automatic Certificate Management Environment (ACME), RFC 8555, recommends certificate issuers use DNSSEC when using DNS to validate control of the DNS zone:
As noted above, DNS forgery attacks against the ACME server can
result in the server making incorrect decisions about domain control
and thus mis-issuing certificates. Servers SHOULD perform DNS
queries over TCP, which provides better resistance to some forgery
attacks than DNS over UDP.
An ACME-based CA will often need to make DNS queries, e.g., to [1]
validate control of DNS names. Because the security of such
validations ultimately depends on the authenticity of DNS data, every
possible precaution should be taken to secure DNS queries done by the
CA. Therefore, it is RECOMMENDED that ACME-based CAs make all DNS
queries via DNSSEC-validating stub or recursive resolvers. This
provides additional protection to domains that choose to make use of
DNSSEC.
An ACME-based CA must only use a resolver if it trusts the resolver
and every component of the network route by which it is accessed.
Therefore, it is RECOMMENDED that ACME-based CAs operate their own
DNSSEC-validating resolvers within their trusted network and use
these resolvers both for CAA record lookups and all record lookups in
furtherance of a challenge scheme (A, AAAA, TXT, etc.).
That's saying less than it appears to. What the RFC is saying is that CAs should be validating DNSSEC where it exists. It was written at a time when CAs didn't uniformly validate DNSSEC at all (because DNSSEC causes outages and pisses off customers). Even I think CAs should validate DNSSEC where it exists! It is not a general recommendation that everyone sign their zones, though.
This is literally a presentation about Ukraine using its control over the DNS to manipulate queries from clients! Like, I like and support Ukraine and all, but this is the problem with top-down PKIs: states use them as policy instruments!
...99% of the internet runs on DV certificates; that's what ACME & Let's Encrypt are at the end of the day.
(You might also note I didn't actually speak in favor of DNSsec, I'm just arguing that running unbound locally without having DNSsec validation enabled isn't particularly useful. I am, generally, in favor of rolling out DNSsec, but it certainly has its problems. It doesn't help against ISP tampering on unsecured domains, for example. DoT and DoH are better against that, if [and only if] you have a trustworthy cache to point them at.)
DNSSEC adoption rate means that this would be insufficient to detect (let alone prevent) that type of tampering for most domains. The correct solution is identified the the first paragraph of the post, which is to use a tamper resistant transport to the dns server (eg DoT, or DoH if you like everything being wrapped in http)
Which DNS server? Your client talks to a central cache (of your choice) which talks to the chain of servers for all relevant domains, most of which don't support DoT or DoH.
Adoption at the TLD is high as it is expected of the TLD operators but the second level is very low. So .org itself is signed but .org second level domains are just below 5% adoption overall. .com second level is around 4.2%. .net second level are a little higher at around 5.2%. So most people's answer would be a solid no. HN audience may be some of that 5%.
There's a difference between what you're likely referring to, where the ISP's own DNS, which most people are using by default, returning particular results, and what the author is showing here where the ISP is actively tampering with outside DNS responses to insert incorrect responses.
Until now, I was not aware of any ISP doing the latter.
Only one ISP I've used ever did this (VodafoneZiggo) and that was over a decade ago, and only to redirect domains that couldn't be found to ad-filled "search" pages.
It's not universal, the occurance or this type of abuse seems to vary wildly from country to country
Illegal, where exactly? In various western countries ISPs are by law required to block some websites. In other countries, ISPs ar eby law required to have escape hatches to be able to shut down the internet. In yet others, everything runs behind a firewall. Take your pick, I guess?
Some countries have laws around Net Neutrality or Open Internet that can protect you from this, but even those countries have other legislations that can superseed that in specific cases. (e.g. hijacking DNS can be completely legal for some child protection laws, or to uphold laws that makes some web sites illegal)
>"That's when I realised my ISP was hijacking my DNS queries and returning a tampered IP address.
[...]
Now here's the catch. DNS resolution doesn't happen over a single connection. Your resolver might have to talk to multiple nameservers to figure out an IP address, and these queries are traditionally sent over plaintext DNS, usually UDP. Which means, you guessed it, your ISP can snoop on or even tamper with your DNS traffic."
This article raises several excellent points.
Basically, unencrypted DNS requests are not private.
Even encrypted DNS, once it gets to the DNS Server wherever that is, is no longer private to the owner of that DNS server, because it is unencrypted there... it may go nowhere from there, but then again, there could be logging and/or monitoring...
And DNS servers that you think don't belong to your upstream ISP -- can be proxied/rerouted by your upstream ISP...
If those points are understood by consumers, then an average person can clearly see why there is such a market (and that's the magic word, "market") for privacy solutions, i.e., encrypted VPN's, encrypted 3rd-party DNS, IP source address anonymization, etc., etc., here in the U.S. ...
In other words, a savvy privacy company or companies could sell to all U.S. Internet customers regardless of who their upstream ISP is, or what their upstream ISP promises in terms of privacy, if (and it's a big if!) they truly understand these issues and could explain them from the ground up, one-to-one, to all such potential customers...
Yes, there are a lot of VPN and other such companies, and prior customers already (not to mention free alternatives, such as Tor (but I still have questions about the exact quality of privacy Tor actually provides given browser fingerprinting and other related technologies), but that market is still segmented, has a relatively low bar for entry, and is still "up-for-grabs" for new small companies (i.e., no need to invest billions in a datacenter of GB300's to get your first AI model out...)...
Anyway, great article.
I never considered some of these issues (or their implications!) before reading it!
For various reason, I also setup an Unbound who proxy to quad9 and is only accessible through DNS over HTTPS/QUIC and is behind a reverse proxy with auth (because being an open dns resolver on the internet is still a bad idea). This is nice because I can then have a dnsproxy or similar on my various devices, who all point to it and be pretty sure nothing can hijack my DNS queries. I also come with the benefit of forcing thing like a minimum TTL to avoid dumb services who decide to put a 10 seconds TTL to all their DNS entry...
I’m using NextDNS over https and frequently experience connection freezes with both Spectrum and T-Mobile. I don’t know what they’re trying to do, but I’m not happy with the way they’re trying to block basic services.
Recently went down a little rabbit hole to get my house in order when I noticed weird things showing up at https://dnscheck.tools/ . (I don't really understand how these checks work though)
It shows that my DNS resolvers are my (public edge) gateway server and then declares that the DNS resolver belongs to my ISP although the PTR is correctly (more or less, let's not get into it) set to an FQDN which is under my domain. In actuality my resolvers don't have public addresses at all (and all of the DNS traffic does of course go out through the gateway server).
I do think ISPs should take a more active role is filtering DNS for their Customers, by default, but opt-out-able, against known-bad domains, or partner with public DNS providers for said service, and include options for blocking additional specific topics (ads, adult sites, gambling, social media, the usual suspects). It could be a value-add differentiator.
It should only be done as a Customer Service, however, and disclosed in a normal sized font and, as I said above, opt-out-able.
I would also extend this to say that ISPs should do the same for known-bad ASNs, and network ranges. Following on from yesterday's discussion about (p)residential proxies, blocking access to known C&C server domains and addresses at an ISP level would be a great solution.
I am, of course, looking at this through a utopian lens where ISPs aren't in bed with big tech and data brokers and other sketchy industries where profit can be made from selling out your customers' data. A 'real' ISP, you know, like 'no true scotsman'. It's all a question of trust and responsibility.
I pretty severely 'mess' with DNS on my home network and, damn, it makes using the Internet a much nicer experience. In fact, it makes using the Internet tolerable, where otherwise it is decidedly not. It could be classified as 'insecure' to not mess with DNS on any network[0].
I wouldn't publicize that you have Podman containers connecting to 1337x.cc. You have flagged yourself as probably an advanced pirate, the kind that's worth pursuing for damages.
Reminds me of the Xfinity "Your personalized WiFi summary" emails. Despite returning their router, using a third party almost-layer-2 modem, my own openwrt router (with its own wifi, thank you very much), their emails seem spooky.
They have a counter of all of my devices, somehow (i mean... i guess their count is wrong, but it's infuriating that they try to keep track).
All of that could be done without "ISP was hijacking my DNS". DPI/SNI sniffing is more accurate and allows you to get more detailed statistics (eg. "20% of your bandwidth goes to youtube"
Unlike OP, I have all my DNS route completely to my home DNS server, and only make changes to that container and my VM host during a "change window" meaning not during work hours. I block port 53 outbound entirely and have host entries for the two domains that my DNS server uses for DoH. If I had a better firewall on the edge, I would leave 53 open for my DNS container, but this workaround seems suitable enough for the home network.
"DoT and DoH rely on you trusting the DNS resolver you're sending your queries to. They encrypt the connection, so your ISP can't casually snoop on or tamper with your DNS traffic in transit."
But _their_ ISP can casually snoop or tamper with their DNS traffic in transit
Because these third party caches (generally) still retrieve DNS data over unencrypted connections to authoritative nameservers
Cloudflare is one of the internet's largest authoritative DNS providers
This no doubt reduces the number of remote queries their cache has to send, but it doesn't account for domains that are not using Clouflare nameservers
It's worth considering why authortative DNS providers (generally) do not offer encrypted DNS
If this service exists then the www user does not need to use third party caches in order to have "encrypted DNS"
As a general rule of thumb, the first AS immediately outside the edge device be that a laptop with a 5G WWAN card or the router of the LAN must not be able to see anything other than QUIC or TLSv1.3, either of with encrypted hello, or IPsec or Wireguard packets.
And if the remote end is available over the local region's mesh network so to not use any commercially operated AS, even better.
No. That's not within the design function of the roots.
It's very much part of the design that roots and caches are separated. The security properties necessary between root and cache (authenticity) are fulfilled by DNSsec, not DoT/DoH (primarily privacy). And most TLDs are DNSsec signed, so this part is in fact OK. (Both give some level of tamper protection, in a different manner though.)
You're probably saying this because you run a local cache directly against the roots. If you need DoT/DoH, you shouldn't be doing that and rather forward your queries to a cache somewhere else.
(To make sure there's no misunderstanding here: DoT and DoH do absolutely nothing to guarantee the data you're getting is what the roots/TLD/owner published. It only ensures you're talking to the cache you think you're talking to. The cache can serve to you whatever it wants.)
I agree with your reply - root servers don't do DOT or DOH for recursive queries. However, there are experiments already under way for a few root servers to support DOT for authoritative results to recursive resolvers, and that is really encouraging for this kind of discussion of interception/blocking. See https://dnsprivacy.org/adox_status_and_deployment/ for more details.
Isn’t that the same as no vpn with extra steps? You are tunneling traffic from a computer you own to a computer you also still own. Third party vpn the whole point is I don’t run it.
If you set up a VPN endpoint on a Hetzner box, you're trusting Hetzner not to MITM your DNS queries. They're likely more trustworthy than your local ISP (although that leaves you more identifiable to folks running the servers or snooping on server-side traffic, and Cloudflare will hate you).
That threat model tradeoff is good for many people.
Not if they are only using it to route the DNS traffic (though in that case a cheaper VPS might be better), which will work if the ISP is only poisoning DNS replies and not (which is often the case) blocking traffic to the target servers entirely.
The problem with this is you are adding a moving part to your DNS lookups. It might be worth adding two on separate VPS hosts and have your resolver fall back to the second if the first goes down.
Another, usually more minor, problem is that some services (like CDNs) use the source of your DNS query to modify the response so you get sent to a topologically near-by end-point. This means you might notice such services being slower as you are sent to a PoP near your VPN host not one closer to your actual location.
Not for DNS lookups. Hetzner forwards complaints from copyright holders, they don't proactively snoop traffic to make sure you don't access 1337x at all. It's legal to access the site.
Oh well OP was doing shady stuff ... then yeah they should continue with using VPN provider. VPN providers are shady AF, I don't think there is VPN provider I would trust more than my local internet provider.
Well that wouldn’t be running your own vpn. Hetzner is. They’d let you do this? Seedbox is against their tos hard to imagine they’d be a good choice to run a vpn for your linux isos.
Can't speak for Hetzner, but considering VPN use in Russia and China - I'd imagine most EU\US vps providers have someone's 3x-ui, remnawave or naked v2ray installed.
Plus - how are they going to detect your remnawave node sitting on 443 port? For them it's just some traffict.
And if you are worried about the amount (torrents) - you don't really need to torrent though them. You just need an access to a torrect tracker to get your magnet link and meta info.
Most default configs for 'ray have torrents locks for this very reason - you communicate with other people (seeds\peers) directly.
Nothing. They'll only act if they receive a complaint. If you're downloading or seeding 1337x torrents from your Hetzner IP address, they will receive a complaint and act on it.
What about it? Unless they've hijacked your local cert store somehow (which corporate machines do), they can't see inside the TLS. They can detect that you're connecting to somewhere and probably infer that it's some VPN, but only the most repressive countries actually block VPNs altogether.
A VPN just moves the attack vector from the ISP to the VPN provider, it doesn't protect you from this in any way at all. Better to use an ISP you trust, though that's difficult to find.
I don't trust any public VPN provider because they all start off by lying about what their product does.
Switching too is funny. You are at the mercy of what isp
ran a line to you. Starlink will fully cooperate with government too. They are a military contractor after all and also at the mercy of musk deciding dumb stuff.
I need to mess with my local DNS resolver for unrelated reasons. Is there a way, in 2026, to setup DNS in a federated way?
I could point to ISP DNS, or 1.1.1.1 / 8.8.8.8 / etc but those are all centralized, and are all subject to surveillance and censorship laws.
DNSSEC, etc, are something, but they rely on CAs. I mostly just want some sort of system that's outside centralized control, but that can be externally audited for trustworthiness.
Noting this article skips over any discussion about why, or potentially (in the eyes of the law at least) legitimate reason an ISP might do this.
Many ISPs in many countries are forced to do this for certain enforcement action that gets put upon them by their respective states for things such as age gating, censorship, monitoring, it varies state by state.
And to be clear this is in no way an endorsement of the practice, and I am confident no ISP WANTS to do this.
I am speaking here as the owner of a UK ISP (who does not currently do this).
If you're a UK ISP owner, you're almost certainly a RIPE NCC member, given the UK is within its regional scope. Therefore, you've probably spoken to other RIPE NCC members via mailing lists or meetings at least once. Since I also represent a provider that is a RIPE NCC member, I can object that not everybody shares the same ideology you noted. There are some providers that actually speak in favor of such over-regulation and censorship. Everybody can read and/or participate in the RIPE Discussion List, though, and you will be amazed by some responses from ISPs that speak in favor of such measures.
But the good news is that at least the majority (for now) is against it.
I speak for another RIPE NCC member when I say that that ideology is limited to a very small minority.
Even outside ideology, plain business sense makes ISPs not want to deal with censorship & co. It's work to do that, hence costs money. But there are also ISPs resisting out of ideology and civic duty.
I'm pretty confident ISPs do want hijack DNS since they've been doing it for decades in order to push ads to their customers.
I hijack DNS in my LAN and block popular DoH servers so they would have to fall back to my DNS. It is a good way to block ads for smart TVs and services that try to use DoH to bypass DNS level ad-blocking.
Do you use a dynamic list of IP addresses to block ?
...
Examples?
There were certainly a number of ISPs in the UK that tested ways to inject ads in content in the 2000s, there was quite a hoo-hah about it in some forums. That was usually by modifying HTML content in transit via transparent proxies though, not DNS poisoning, but it shows that the intent exist{ed|s}. This is one of the reasons that HTTPS-everywhere is a good idea even if you aren't concerned about snooping in general.
An example I remember is when Virgin Media replaced any DNS queries returning NXDOMAIN with their own "search page" plastered with ads.
https://shkspr.mobi/blog/2021/03/advanced-network-error-sear...
Ah, yes, I forgot the NXDOMAIN hijacking.
I was thinking about interfering with existing sites.
AT&T definitely used to do this ~14 years ago.
Verisign tried this trick too.
https://dn.org/the-sitefinder-debacle-that-shook-the-interne...
AT&T still does this, if you're using their (intentionally terrible) standard gateway device the default DNS will return a "search page" instead of NXDOMAIN which predictably has all sorts of negative impacts on non-browser devices but the wastes of oxygen responsible for this don't care as long as they see that delicious ad revenue.
Some mobile phone network carriers would inject certain PII (such as your mobile number) into HTTP request headers sent to ad networks. I'm sure they still do it, but by requiring the site to have some extra request to be made instea.
Some carriers also passed info in these headers to be used to make billing calls to the Telco, such as subscribing to premium services with minimal (if any) consent from the user.
I got stung by this years ago. An ad for mobile ringtones showed up on some news site I visited on my phone, I scrolled past without clicking. Next minute I get a text telling me I've subscribed signed up to a (iirc) $25/month ringtone plan and that it'd show up on my phone bill. It took an hour on the phone to my carrier to block the charges and get a refund. The ringtone company kept messaging me for a year that I needed to contact them and "fix" the issue.
Insanity that people used to pay more for goofy little midi files than they do now for Netflix.
I dunno what was in a ringtone subscription. I used to work at a ringtone company, and it was $1-$3 per ringtone ($1 for monophonic, $2 for polyphonic and $3 for wav/mp3/whatever; which yeah, is more than a high quality download from iTunes... But $25/month is a ton of money.
Even with free ringtones, I'd send myself about three if I got a new phone. These days, I just keep a copy of the Nokia dubstep ringtone on hand in case I turn the ringer on, but it doesn't happen.
As someone who was sometimes using http as a back end transport for "edi" data (hey at the time it was usually ftp in that industry so no worse, but at least http got around nat problems than even pasv didn't), one time in the mid 00's I was in a coffee shop and just speechless flabbergasted outraged to discover the contents of a random web page I happened to know the contents of get some extra html and js injected into it from the isp (the isp not the coffee shop).
"WHO THE F gave them the right to modify the data in my tcp session!!!!".
Spying on it I assumed but modifying one byte of it? A naively written edi that did not expect that would be completely destroyed.
Probably shouldn't have been so stupid for 15 years already by that point. There is no such thing as the right, there is only the ability.
If someone has the ability to do something, they will. Reputation, right & wrong, none of that actually exists or matters.
I didn't track down if the mechanism was based on dns. Could have been but I don't see why it would have had to be. But I was also not attempting to use any non default dns on that device at that time anyway. Obviously since they are trasporting unencrypted data they could modify it any time simply because it's in their hands.
dns would just be one handy low-hanging-fruit way to detour the data to a convenient dedicated purpose server to do the work. But not special or necessary at all. If they want to do the same thing without hijacking dns, they can with almost not even any difference in total effort.
That's one of the reasons we now have HTTPS in most places.
Comcast and/or verizon did it in the 2010s, there was an opt-out in the account dashboard.
It's sort of an absurd question, the prevalence of such things by ISPs was the literal impetus for DoH and DoT.
Last I checked Verizon FIOS still has the opt-out. You have to go into your account preferences and manually disable it because spying is on by default.
This is a good reminder to login and take a look, I wouldn't put it past them to change the setting name and reset the default.
edit: apparently Verizon has zero record of the username I've used for my FIOS account for years (and have saved in my password manager). Their account reset also fails with an error. What a dumpster fire of a company.
Yep, I've been in Verizon online account purgatory for awhile now. They merged my mom's with mine and nothing works to reset or restore online account access for even bill pay. I have to push their payment out via billpay. I really need to move that phone number over to voip, I kept the classic POTS line as a type 1 diabetic I like that it stayed on during both NYC blackouts in 2003 and 2012. But now I'm in a fiber only building and that redundancy is mostly moot.
Thank you for this. Just opted out. They certainly made that page as hard as possible to find. Figuring out how to change my routers DNS server was also made needlessly complex.
You should be NATing outbound 53 to a local resolver that upstreams to DoH/DoT. Relying on such an opt-out is rather optimistic.
I don't use their resolver at all. Their opt-out was more of an all-encompassing "let us take whatever data we can manage to get" than DNS specifically.
AT&T has been well-known to do this, since they are violently opposed to net neutrality and they want to force you to only use their approved paid partners for everything. They do this on all their internet services.
Netflix has always listed AT&T as their number one worst offender when it comes to net neutrality.
So, I always run VPNs that I can trust.
Time Warner Cable used to do it. I haven't checked since they became Spectrum.
IIRC, early into ATT’s US fiber / gigabit rollout, they wanted an additional $40 / month to forgo collecting and selling your browsing history (DNS queries).
RCN definitely was doing it for a while, so was Comcast. I learned quickly to override this garbage
Deutsche Telekom (largest ISP in Germany) did this, not sure if they still do or were ordered to stop by court yet
this is valid, in NL most DNS is also heavy monitored and intercepted to do blocks if mandated etc. if you want to get around it u can tunnel ur tlsdns thingy to another place and they essentially cannot intercept it, they can still block the IPs it resolves to so its hit and miss depending on how the other end is implemented.
its usually not nefarious, that is just the simpelest (and usually, not always) wrong explanation.
people who think this is not normal probarbly didnt experience what kind of shit u could find on the pre google pre monitored internet. for sure shit u do not want your kids to find. if u dont understand, then likely u dont have kids and very weird fetishes...
One massive reason is they redirect requests for videos to closer (or local) caches of videos for netflix (directly from youtube placing it with a large enough ISP), or local caching at the ISP level so less traffic hits the internet.
If you've ever been hit with a "webpage isn't updating" issue when someone else can see it, often it's different DNS servers.
It's best to never use the ISP's DNS servers, regardless of the reason why they might do what they do.
> It's best to never use the ISP's DNS servers, regardless of the reason why they might do what they do.
But providers overwriting DNS packets regardless – this is the topic of the article :)
P.S.: And yes, there are ways around this like DNSsec etc.
A reminder that the average consumer device doesn't actually validate DNSSEC in its default configuration: it relies on its recursive resolver to set a bit on the response saying "I validated DNSSEC, things are all good". In that configuration, DNSSEC gets you no protection against this.
One of the reasons a variety of providers have been pushing for DoH is because DoH fully closes the risk of ISP MITM without needing individual site operators to do anything. This is why you're seeing folks like Firefox and Chrome enable DoH by default.
DoH is fine if a bit complicated. The problem is who the DoH provider is. Usually it's Cloudflare.
> One of the reasons a variety of providers have been pushing for DoH is because DoH fully closes the risk of ISP MITM without needing individual site operators to do anything.
... by replacing it with Google/Cloudflare MITM with whom the user doesn't even have a business relationship or even shared jurisdiction.
The business relationship most home users have with their ISP explicitly permits that ISP to harvest data from their DNS requests.
I'm glad you know everyones ISP terms of services as well as the local laws that restrict which parts of them are worth a damn.
Do you actually disagree with my claim, or is your concern purely that you don’t like my generalization?
For sure, I read the article.
Introducing what ISPs do with DNS information is helpful to understand how encompassing DNS self-hosting needs to be thought about - DNS requests are important to consider the appropriate encryption for.
From the homelab side, increasingly with the automated attacks on systems, it seems wise to keep all services off the internet directly and put it behind one or more service to keep them safe. From that perspective, setting up something like Tailscale can also help mitigate this kind of issue.
They don't mess with DNS to do that. The video provider points them to the appropriate cache. And it's usually only for video content, not the whole web app.
Why do you assume the others have same ethical compass as you? I hope everyone has the same compass but we have seen again and again trust being broken.
ISPs talk to each other, and people move jobs between them. It's not a uniform community but you certainly get a sense of the general spread of opinions after a while.
There are of course outliers but most people I know in the field care as much about personal rights & freedom as anyone else. Maybe even a small bit more.
This is a complete non sequitur
My workplace is messing with DNS as well, probably to filter specific websites, and I would not mind it since for the context of my usage that is perfectly fair.
But their DNS responses have latency spikes every couple of minutes of 1-2 seconds that was driving me crazy. I simply enabled DNS over HTTPS in Firefox and everything feels so smooth now.
I have no idea how my coworkers seem to be so insensitive to it.
For companies with ISO-27001, Web Filtering is a control recently introduced. Protective DNS can tick few boxes and genuinely keep your network a little bit more secure (and ad free).
As an employee I'd be against it, but company device so what can you do. As the one managing the security and ISO process, it made my life a lot easier from both auditing and "why is my computer acting funny lately" perspectives. Showing my own user account having triggered an ad-based block, and the company-wide malware block rate was enough to satisfy the auditor that we has appropriate controls in place for a small company and move on to the next item.
Provided you take the stance of everyone is trusted to do their jobs, so we only need to block ads, piracy and malware lists and don't retain logging except where a block has occurred for auditing, I think it strikes a reasonable middle ground.
Enabling DoH on a locked down workplace network sounds like a way to get fired.
But I guess that if they let you do it they're probably not competent enough to catch you.
It’s a university network so things are not as strict as other work places.
They also do not pay for my device, so it’s only fair I can configure things my own way.
I’m pretty sure the worst that could happen is they would ask me to turn it off.
No way you would get fired for this, come on man. Maybe someone from IT would ping you to turn it off, worst case.
Happened to me once, my computer had Little Snitch block fortinet. Got a nice message to look into why. Suddenly started calling home.
Depends on the workplace, but I've definitely seen people fired for less.
If you're not working on something highly secretive and valuable (with a salary to match) then they're doing you a favor by flipping out over something so stupid.
They might be proxying all your traffic through a "WAF". I've noticed that even quite a lot of regular websites linked on HN on "non standard" domains are simply unresolvable on my work network.
Have you bothered asking your IT to look into it?
Ok, free tech support I guess. It could be shitty DNS provisioning, that's pretty common. But are you using wireless? Are you familiar with Buffer Bloat generally, let alone some of the other implicit shenanigans in "traffic shaping"?
A few years ago the cognitive dissonance of people saying "DoH is faster" to the same DNS server(s) started really bugging me: is it really DoH, or is it simply TCP vs UDP? So I wrote a little thing I can install on my laptop to do DNS forwarding over plain old TCP (originally no encryption): https://github.com/m3047/tcp_only_forwarder
Golly, on my wifi TCP was certainly more reliable than UDP. It's due to how wifi devices prioritize and buffer traffic. Maybe they should implement / enable DoH on their own servers.
On the other hand if all of your access from the workplace network to web resources is proxied (and since DoH is a web service), they may not care.
I fear a future where pretty much everything is trying to hack you or doing something which is indistinguishable from the meddling of a hostile actor.
I suppose the answer is politics rather than technology: Get enough people together to make a rule against it, and at least it will become niche.
Hello from 9 hours in the future.
Things are crazy now. Our TVs snapshot the stuff we watch; our telephones, watches, and TVs are constantly listening what we say; our PCs have undocumented "remote administration" devices that we can't opt-out; every internet service out there is stealing every data that it physically can and selling it to a long list of organizations, including some military ones...
Hello fellow time traveller, I also come to you from 5 - 10 years ago.
Back in this time children and families playing AR games are actually just data collection for GPS-denied military drones [1]. ~70% of the world's population voluntarily carry continuously connected GPS trackers, multi-camera, multi-microphone devices with fairly advanced sensor suites and send this into a plethora of 'social' platforms. This includes when they're jogging/tracing the outline of aircraft carriers [2] and military bases [3]. Vehicles are internet connected, laughably insecure and record every movement, and the movement of everything else in their vicinity [4]. And, for some reason, everything in your house is 'smart' and the building you work in is filled with 'advanced IoT' sensors and high precision wifi-tralateration that provides a firehouse of data to countless third parties.
We've been steadily descending for a long time and seem intent of speed-running dystopia.
[1] https://www.nianticspatial.com/defense-and-intelligence
[2] https://www.bbc.com/news/articles/cd9vdel17wqo
[3] https://www.theguardian.com/world/2018/jan/28/fitness-tracki...
[4] https://www.abc.net.au/news/science/2024-10-09/car-brands-ar...
People are still wearing tinfoil hats about Intel's IPMI that's not even usable unless you have compatible hardware and purposely set it up?
I'm not sure regulation alone would solve it. Depending on the country, governments themselves sometimes require ISPs to interfere with DNS
That is regulation! You do actually have to win the argument for liberalism in public if you don't want your fellow citizens to keep trying to stop you doing this.
Telecom Italia locks my DNS server (I cannot change it).
I ended up using PIHole as DHCP, to drive home devices to PIHole, and control my privacy.
The trick is simple: disable DHCP on your router, and activate it on PIHole. As a bonus, PIHole on my RasperryPI2 is way faster than the little chip on my provider Router, and can be patched automatically with unattended upgrades.
And if you can't disable DHCP or set the router in bridge mode, you can just add another router (that you control) between your ISP device and the rest of your network. I've been running this setup for over a decade and didn't run into any noteworthy issues. Leave the ISP-infested router outside your network.
The whole point of ad companies like google pushing DoH is to bypass things like pinhole and take power away from you, the owner, and give it to them, the trillion dollar advertising company.
Note that there are ISPs that actively hijack traffic to UDP port 53 further inside their network and transparently maps that onto their internal DNS servers.
This is why you would want to setup DNS-over-TLS or DNS-over-HTTPS to, say, 1.1.1.1. They can’t hijack DNS traffic that looks like HTTPS traffic.
Cloudflare knowing every single DNS resolution is just as bad.
Pretty sure that you can use DHCP and still override the DNS servers with most DHCP clients.
Yes but you need to do it MANUALLY on every client. Using the PIHOLE DHCP automate this and simplify the setup.
I think they're saying that their ISP's router does not allow this, thus the need in their case to use third-party DHCP.
AT&T in the US does this. Traditionally they did this so that failed lookups were routed to their search engine, which of course fed users ads.
At least they no longer insert ads straight into web pages. Thank goodness for https.
Does anyone know if T-Mobile does this too? I'm considering moving fiber to them instead of AT&T, but have stayed mostly due to "better the devil you know..."
They don't at least on cellular. Home Internet; not sure.
I'll throw my $.02 in here since this is getting some traction: DNSSEC validation should be used, no exceptions, no excuses. If you're running a local resolver, turn on stub validation for DNSSEC - it prevents anyone (even a DOH/DOT upstream forwarding cache) from lying to you without your knowledge. If you run a domain name: turn on DNSSEC. It's 2026 - not 2012. DNSSEC works great, we validate it on 100% of our queries, and is one of the tools needed to stop people from meddling with the DNS. https://circleid.com/posts/common-beliefs-about-dnssec-put-t...
Also, encrypt everything. DOT/DOH/DOH3/DOQ - leave no packet available for observation. Criminals, nosy network snoopers, and governments either want to know what you're doing or they want to break the integrity of the internet for profit or political reasons, and it is work to keep them out of your business.
context: I work at Quad9, one of the larger public resolvers.
The point of running a local unbound is indeed to turn on DNSsec validation. After that, it's the domains' fault for not using DNSsec. But at least your own stuff won't be tampered with, because you use DNSsec on your own domains, right? Right?
:-(
Coincidentally, a bunch of free wifi stuff blocks the root key queries unbound does at startup, resulting in nothing working until you get those records in there somehow (e.g. load_cache).
I was under the impression that dnssec's entire value was rather controversial? I'm not a crypographer so I had to skim https://sockpuppet.org/blog/2015/01/15/against-dnssec/ and I still don't understand it, but if
> In fact, it does nothing for any of the “last mile” of DNS lookups: the link between software and DNS servers. It’s a server-to-server protocol.
is true, then I'm ... pretty unsold on its usefulness. Anyone wanna ELI5 whether that's untrue or anything's changed in the last decade?
DNSSEC as designed is pretty useless. It was designed in an era where computers could not be relied upon to do cryptographic validation of something like DNS records, so that duty was delegated to DNS servers like the ones ISPs ran, where records could be cached.
There are also potential downsides (like someone being able to enumerate your subdomains, if you care about such a thing) and it's often switched off by default on operating systems consumers use.
However, it's the only thing that validates DNS records at this moment. That alone makes it worth enabling, in my opinion. For many domain owners, you only need to tick the checkbox in your registrar's domain control pane. If you use a modern DNS server, it's also one config flag and maybe copy/pasting a handful of fields away.
I'll gladly turn off DNSSEC the moment we come up with something better. DoT/DoH doesn't solve the same problem and no real solutions seem to have been proposed. That said, modernisation efforts are still being done. Smaller keys and modern cryptography rather than the massive RSA ones for instance.
> DNSSEC as designed is pretty useless.
Another approach that we analysed in more detail is the distribution of the host key fingerprints via a special resource record. This removes any user interaction, potential human error, or presumed configuration of a user’s device because OpenSSH-client can query the DNS and do the comparison. It only requires the administrator to publish a host’s fingerprints as a set of SSHFP records, which can be securely obtained (DNSSEC). If the latter holds, MITM attacks are mitigated, and overall security is improved. [1]
[1]: "Improving SSH’s security with SSHFP DNS records" - https://blog.apnic.net/2022/12/02/improving-sshs-security-wi...
I have never been able to get my head around the idea that the DNS should have anything whatsoever to do with my SSH fleet. The DNS exists to solve the problem of introductions between unaffiliated network endpoints. To a first approximation nobody SSH's to a random server. You don't need any third-party PKI to get optimal SSH security!
SSHFPS is what I should be using in my homelab if it weren't for the awful split-DNS setup I have!
I personally prefer the theoretical advantages of SSH certificates, but those don't have CRL support (let alone an ACME alternative) as far as I know, so that's not exactly perfect either.
I want DNSSEC('s goals and ambitions) to succeed, but having a "trusted" ISP server that sets a "trust me bro" bit is the design flaw that kills the original design.
There's a patch for openssh to use the ldns library and do its own DNSsec validation. A few distros apply it to their packages, though I'm not sure what the current status is. It never got accepted into openssh sadly.
Rebuttal: <https://easydns.com/blog/2015/08/06/for-dnssec/>
General DNSSEC rationale: <https://www.icann.org/resources/pages/dnssec-what-is-it-why-...>
DNSSEC adoption graph: <https://www.verisign.com/resources/dnssec-tools/dnssec-score...>
Sometimes that rebuttal is summarily dismissed for being old, so here is a modern rebuttal, from a few weeks ago in fact: <https://circleid.com/posts/common-beliefs-about-dnssec-put-t...>
This is the funniest DNSSEC thing I've read, by the way. Apparently the market failure of DNSSEC is because of... me?
Somewhere out there, a Linux users group is acting out mock trials to charge you with murder of a crucial security standard.
I can't speak about the strength or weaknesses of the whole concept.
Out of curiosity I enabled DNSSEC checking beginning this year. I mostly noted nothing in my browsing. However, with 2 services I noted obscure problems that worked only partially. Some Jira plugins and Swedish railway SJ. I needed to excempt some of their domains from being checked to use their services. Of course you waste a couple of hours every time it happens because you don't think of DNSEC first when some web-based services behaves weirdly. There might be more requests not served to my browser, but I just never noticed or bothered to debug.
So not fit for general use. And even as an experienced developer it's not easy to understand why an exception would be needed. A bug in systemd? Incorrect data in DNS? A real attack?
Edit: I don't use standard DNS on port 53 at all. Blocked outgoing requests on that port. Everything DoH or DoT. Of course I need to trust those endpoints at least in repect to confidentiality. And most domains are unsigned. At least for those unsigned ones I need to just trust also in respect to integrity.
DNSSEC failures need to be louder. If websites that fucked up their DNSSEC would get the same error messages as websites that fucked up other parts of their security, people wouldn't be so eager to turn off security festures as workarounds.
Give SJ the option to turn off DNSSEC and their website will stay broken. Give people a "someone might be tampering with your connection" popup and they would start caring.
There are also other issues with DNSSEC that need working out, but I still believe it does much more good than it does bad. Leaving everything in the background with an easy-to-disable setting makes it next to useless.
> DNSSEC failures need to be louder.
Full ACK. As a part of this, DNSsec validation needs to be as close to the client app as possible and reasonable; definitely never off-host. (Caches can do additional validation, but that makes the error reporting complicated.)
corollary: the AD bit should never be relied on on anything actually on the network. Maybe on localhost, but even there it's questionable in UX results.
It's about time to question the whole off-host DNS resolver thing entirely. Fine if you want to have a cache, but the cache should return the whole record chain starting from the root in that case, so the client can validate it. Also fine to have an on-host cache that caches validation status.
> It’s a server-to-server protocol is true, then I'm ... pretty unsold on its usefulness.
You can run a DNSSEC-aware resolver like unbound on your laptop if you wanted to. It comes with a bunch of Linux distros.
Can and should! Unbound is very easy to set up, if you know how to administer Linux systems at all. It also acts as a local cache and I noticed a small improvement to page loading time because of having the cache. Otherwise there is no local cache - every single call to getaddrinfo sends a network request to your configured DNS server.
People should just say what they mean here. DNSSEC is in fact a server-to-server protocol, and your response is "yes, you should just run your own servers".
Automatic Certificate Management Environment (ACME), RFC 8555, recommends certificate issuers use DNSSEC when using DNS to validate control of the DNS zone:
[1]: https://www.rfc-editor.org/info/rfc8555/#section-11.2That's saying less than it appears to. What the RFC is saying is that CAs should be validating DNSSEC where it exists. It was written at a time when CAs didn't uniformly validate DNSSEC at all (because DNSSEC causes outages and pisses off customers). Even I think CAs should validate DNSSEC where it exists! It is not a general recommendation that everyone sign their zones, though.
Just yesterday, there was a talk by representatives from Ukraine and Germany about how and why they use DNSSEC:
<https://www.icann.org/en/engagement-calendar/details/icann-t...>
This is literally a presentation about Ukraine using its control over the DNS to manipulate queries from clients! Like, I like and support Ukraine and all, but this is the problem with top-down PKIs: states use them as policy instruments!
Note the linked post is from 2015. But also:
> domain-validated certificates remain insecure
...99% of the internet runs on DV certificates; that's what ACME & Let's Encrypt are at the end of the day.
(You might also note I didn't actually speak in favor of DNSsec, I'm just arguing that running unbound locally without having DNSsec validation enabled isn't particularly useful. I am, generally, in favor of rolling out DNSsec, but it certainly has its problems. It doesn't help against ISP tampering on unsecured domains, for example. DoT and DoH are better against that, if [and only if] you have a trustworthy cache to point them at.)
DNSSEC adoption rate means that this would be insufficient to detect (let alone prevent) that type of tampering for most domains. The correct solution is identified the the first paragraph of the post, which is to use a tamper resistant transport to the dns server (eg DoT, or DoH if you like everything being wrapped in http)
Which DNS server? Your client talks to a central cache (of your choice) which talks to the chain of servers for all relevant domains, most of which don't support DoT or DoH.
Adoption at the TLD is high as it is expected of the TLD operators but the second level is very low. So .org itself is signed but .org second level domains are just below 5% adoption overall. .com second level is around 4.2%. .net second level are a little higher at around 5.2%. So most people's answer would be a solid no. HN audience may be some of that 5%.
Which ISP was doing this? It's worth calling out the bad actors.
It's pretty much most Indian ISPs and in this case it was ACT Fibernet.
Wouldn't the list of ISPs not doing this be easier to make?
No.
There's a difference between what you're likely referring to, where the ISP's own DNS, which most people are using by default, returning particular results, and what the author is showing here where the ISP is actively tampering with outside DNS responses to insert incorrect responses.
Until now, I was not aware of any ISP doing the latter.
Only one ISP I've used ever did this (VodafoneZiggo) and that was over a decade ago, and only to redirect domains that couldn't be found to ad-filled "search" pages.
It's not universal, the occurance or this type of abuse seems to vary wildly from country to country
Want to ISP-less in India? :D At least ACT doesn't blanet throttle/fail uplaods of any sort like Jio does on its 5G – even LLM calls fail.
How is this not illegal... isn't what the ISP doing "hacking"?
Illegal, where exactly? In various western countries ISPs are by law required to block some websites. In other countries, ISPs ar eby law required to have escape hatches to be able to shut down the internet. In yet others, everything runs behind a firewall. Take your pick, I guess?
Very few, if any, countries makes this illegal.
Some countries have laws around Net Neutrality or Open Internet that can protect you from this, but even those countries have other legislations that can superseed that in specific cases. (e.g. hijacking DNS can be completely legal for some child protection laws, or to uphold laws that makes some web sites illegal)
The hijack is probably legally mandated. The site he was trying to reach is a torrent site.
Interfering in communication. If I understand correctly the ISP rewrote the author's communication with an external DNS/third party.
That's not a crime
>"That's when I realised my ISP was hijacking my DNS queries and returning a tampered IP address.
[...]
Now here's the catch. DNS resolution doesn't happen over a single connection. Your resolver might have to talk to multiple nameservers to figure out an IP address, and these queries are traditionally sent over plaintext DNS, usually UDP. Which means, you guessed it, your ISP can snoop on or even tamper with your DNS traffic."
This article raises several excellent points.
Basically, unencrypted DNS requests are not private.
Even encrypted DNS, once it gets to the DNS Server wherever that is, is no longer private to the owner of that DNS server, because it is unencrypted there... it may go nowhere from there, but then again, there could be logging and/or monitoring...
And DNS servers that you think don't belong to your upstream ISP -- can be proxied/rerouted by your upstream ISP...
If those points are understood by consumers, then an average person can clearly see why there is such a market (and that's the magic word, "market") for privacy solutions, i.e., encrypted VPN's, encrypted 3rd-party DNS, IP source address anonymization, etc., etc., here in the U.S. ...
In other words, a savvy privacy company or companies could sell to all U.S. Internet customers regardless of who their upstream ISP is, or what their upstream ISP promises in terms of privacy, if (and it's a big if!) they truly understand these issues and could explain them from the ground up, one-to-one, to all such potential customers...
Yes, there are a lot of VPN and other such companies, and prior customers already (not to mention free alternatives, such as Tor (but I still have questions about the exact quality of privacy Tor actually provides given browser fingerprinting and other related technologies), but that market is still segmented, has a relatively low bar for entry, and is still "up-for-grabs" for new small companies (i.e., no need to invest billions in a datacenter of GB300's to get your first AI model out...)...
Anyway, great article.
I never considered some of these issues (or their implications!) before reading it!
For various reason, I also setup an Unbound who proxy to quad9 and is only accessible through DNS over HTTPS/QUIC and is behind a reverse proxy with auth (because being an open dns resolver on the internet is still a bad idea). This is nice because I can then have a dnsproxy or similar on my various devices, who all point to it and be pretty sure nothing can hijack my DNS queries. I also come with the benefit of forcing thing like a minimum TTL to avoid dumb services who decide to put a 10 seconds TTL to all their DNS entry...
I’m using NextDNS over https and frequently experience connection freezes with both Spectrum and T-Mobile. I don’t know what they’re trying to do, but I’m not happy with the way they’re trying to block basic services.
Some day we will find aliens, and I wonder if we will be well prepared against their hacking or find we are naively vulnerable
Recently went down a little rabbit hole to get my house in order when I noticed weird things showing up at https://dnscheck.tools/ . (I don't really understand how these checks work though)
It shows that my DNS resolvers are my (public edge) gateway server and then declares that the DNS resolver belongs to my ISP although the PTR is correctly (more or less, let's not get into it) set to an FQDN which is under my domain. In actuality my resolvers don't have public addresses at all (and all of the DNS traffic does of course go out through the gateway server).
> Although your regular internet traffic is encrypted with TLS, your ISP can see the hostname during the TLS handshake (SNI).
https://support.mozilla.org/en-US/kb/understand-encrypted-cl... not anymore, with many (not all) sites.
Very few of the top 10k sites support ECH
https://divested.dev/misc/ech.txt
I would argue that knowing you are visiting a top 10k site is less valuable than... All other sites.
Also that was last updated 6 months ago...
Feel free to find a better source and to actually back up your initial claim
Sounds familiar to a Chinese
Hardly a new thing.
Oh and btw, it's not "DNS Hijacking" it's a "Domain Helper", see: https://feross.org/shame-on-you-comcast/
I do think ISPs should take a more active role is filtering DNS for their Customers, by default, but opt-out-able, against known-bad domains, or partner with public DNS providers for said service, and include options for blocking additional specific topics (ads, adult sites, gambling, social media, the usual suspects). It could be a value-add differentiator.
It should only be done as a Customer Service, however, and disclosed in a normal sized font and, as I said above, opt-out-able.
I would also extend this to say that ISPs should do the same for known-bad ASNs, and network ranges. Following on from yesterday's discussion about (p)residential proxies, blocking access to known C&C server domains and addresses at an ISP level would be a great solution.
I am, of course, looking at this through a utopian lens where ISPs aren't in bed with big tech and data brokers and other sketchy industries where profit can be made from selling out your customers' data. A 'real' ISP, you know, like 'no true scotsman'. It's all a question of trust and responsibility.
I pretty severely 'mess' with DNS on my home network and, damn, it makes using the Internet a much nicer experience. In fact, it makes using the Internet tolerable, where otherwise it is decidedly not. It could be classified as 'insecure' to not mess with DNS on any network[0].
[0]: https://www.pcmag.com/news/fbi-recommends-installing-an-ad-b...
I wouldn't publicize that you have Podman containers connecting to 1337x.cc. You have flagged yourself as probably an advanced pirate, the kind that's worth pursuing for damages.
Isn't that just where you can find a whole lot of Linux ISOs?
Probably not a single one. It's curated, unlike Pirate Bay.
Deutsche Telekom did that too on my DSL line once, took some arguing to them suddenly 'do nothing' but it suddenly 'worked'
Reminds me of the Xfinity "Your personalized WiFi summary" emails. Despite returning their router, using a third party almost-layer-2 modem, my own openwrt router (with its own wifi, thank you very much), their emails seem spooky.
They have a counter of all of my devices, somehow (i mean... i guess their count is wrong, but it's infuriating that they try to keep track).
All of that could be done without "ISP was hijacking my DNS". DPI/SNI sniffing is more accurate and allows you to get more detailed statistics (eg. "20% of your bandwidth goes to youtube"
Here is my self-documentation for home DNS setup: https://docs.zeropolis.net/doku.php/tech:homedns
Unlike OP, I have all my DNS route completely to my home DNS server, and only make changes to that container and my VM host during a "change window" meaning not during work hours. I block port 53 outbound entirely and have host entries for the two domains that my DNS server uses for DoH. If I had a better firewall on the edge, I would leave 53 open for my DNS container, but this workaround seems suitable enough for the home network.
It's best to treat your ISP's DNS as not being in your best interest, and actively against you getting your full internet speed and responsiveness.
Set an external DNS server on your router, or learn to run your own locally.
my mobile network carrier blocks quad9
In France copyright law says they have to
"DoT and DoH rely on you trusting the DNS resolver you're sending your queries to. They encrypt the connection, so your ISP can't casually snoop on or tamper with your DNS traffic in transit."
But _their_ ISP can casually snoop or tamper with their DNS traffic in transit
Because these third party caches (generally) still retrieve DNS data over unencrypted connections to authoritative nameservers
Cloudflare is one of the internet's largest authoritative DNS providers
This no doubt reduces the number of remote queries their cache has to send, but it doesn't account for domains that are not using Clouflare nameservers
It's worth considering why authortative DNS providers (generally) do not offer encrypted DNS
If this service exists then the www user does not need to use third party caches in order to have "encrypted DNS"
When the www user does not need to use third party caches (recursive resolution), then there's no need for DNSSEC
Iterative resolution can substitute for recursive resolution
As a general rule of thumb, the first AS immediately outside the edge device be that a laptop with a 5G WWAN card or the router of the LAN must not be able to see anything other than QUIC or TLSv1.3, either of with encrypted hello, or IPsec or Wireguard packets.
And if the remote end is available over the local region's mesh network so to not use any commercially operated AS, even better.
current mood: https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...
We need root DNS servers that support DoH/DoT.
No. That's not within the design function of the roots.
It's very much part of the design that roots and caches are separated. The security properties necessary between root and cache (authenticity) are fulfilled by DNSsec, not DoT/DoH (primarily privacy). And most TLDs are DNSsec signed, so this part is in fact OK. (Both give some level of tamper protection, in a different manner though.)
You're probably saying this because you run a local cache directly against the roots. If you need DoT/DoH, you shouldn't be doing that and rather forward your queries to a cache somewhere else.
(To make sure there's no misunderstanding here: DoT and DoH do absolutely nothing to guarantee the data you're getting is what the roots/TLD/owner published. It only ensures you're talking to the cache you think you're talking to. The cache can serve to you whatever it wants.)
I agree with your reply - root servers don't do DOT or DOH for recursive queries. However, there are experiments already under way for a few root servers to support DOT for authoritative results to recursive resolvers, and that is really encouraging for this kind of discussion of interception/blocking. See https://dnsprivacy.org/adox_status_and_deployment/ for more details.
sigh
The end result of this will be ISPs dropping all traffic to port 853.
comcast/xfinity has been doing this for years, its called security edge
Yum, screenshots of terminal windows, which is just text...
Edit: oh wait, it finally loaded, 3x3MB images of text with scenery in the background!
Another reason to use a VPN you can trust, across your entire network and at all times.
So a VPN you set up and run yourself.
Isn’t that the same as no vpn with extra steps? You are tunneling traffic from a computer you own to a computer you also still own. Third party vpn the whole point is I don’t run it.
If you set up a VPN endpoint on a Hetzner box, you're trusting Hetzner not to MITM your DNS queries. They're likely more trustworthy than your local ISP (although that leaves you more identifiable to folks running the servers or snooping on server-side traffic, and Cloudflare will hate you). That threat model tradeoff is good for many people.
Given the OP use case, you are going to get a letter from Hetzner at some point.
Not if they are only using it to route the DNS traffic (though in that case a cheaper VPS might be better), which will work if the ISP is only poisoning DNS replies and not (which is often the case) blocking traffic to the target servers entirely.
The problem with this is you are adding a moving part to your DNS lookups. It might be worth adding two on separate VPS hosts and have your resolver fall back to the second if the first goes down.
Another, usually more minor, problem is that some services (like CDNs) use the source of your DNS query to modify the response so you get sent to a topologically near-by end-point. This means you might notice such services being slower as you are sent to a PoP near your VPN host not one closer to your actual location.
Not for DNS lookups. Hetzner forwards complaints from copyright holders, they don't proactively snoop traffic to make sure you don't access 1337x at all. It's legal to access the site.
Oh well OP was doing shady stuff ... then yeah they should continue with using VPN provider. VPN providers are shady AF, I don't think there is VPN provider I would trust more than my local internet provider.
Well that wouldn’t be running your own vpn. Hetzner is. They’d let you do this? Seedbox is against their tos hard to imagine they’d be a good choice to run a vpn for your linux isos.
> They’d let you do this?
Can't speak for Hetzner, but considering VPN use in Russia and China - I'd imagine most EU\US vps providers have someone's 3x-ui, remnawave or naked v2ray installed.
Plus - how are they going to detect your remnawave node sitting on 443 port? For them it's just some traffict.
And if you are worried about the amount (torrents) - you don't really need to torrent though them. You just need an access to a torrect tracker to get your magnet link and meta info.
Most default configs for 'ray have torrents locks for this very reason - you communicate with other people (seeds\peers) directly.
>Plus - how are they going to detect your remnawave node sitting on 443 port? For them it's just some traffict.
What about all the outgoing traffic? That's going to be hard to mask unless you have another VPN for outbound
But what's wrong with my VPN talking to, say, youtube (blocked in Russia) or some torrent site (for example to fetch RSS feed of new releases).
Fedora, for example, provides torrent access (including RSS feed) for its releases: https://fedoraproject.org/torrents/45/
Nothing. They'll only act if they receive a complaint. If you're downloading or seeding 1337x torrents from your Hetzner IP address, they will receive a complaint and act on it.
But the isp sits between that no?
All traffic to the VPN is encrypted, including DNS queries.
All your ISP can tell is you are using a VPN.
What about TLS inspection?
What about it? Unless they've hijacked your local cert store somehow (which corporate machines do), they can't see inside the TLS. They can detect that you're connecting to somewhere and probably infer that it's some VPN, but only the most repressive countries actually block VPNs altogether.
Not unless they can break your VPN's encryption. Which, if your ISP can do that, you have bigger problems.
A VPN just moves the attack vector from the ISP to the VPN provider, it doesn't protect you from this in any way at all. Better to use an ISP you trust, though that's difficult to find.
I don't trust any public VPN provider because they all start off by lying about what their product does.
ISP can still see exit IP unless you VPN
Cloudflare Warp is handy, complete tunnel even if not as private as real VPN
since 80% of the web is on cloudflare it accidentally makes it faster anyway
and you don't have to use the cloudflare client, you can use Wireguard
The story ended before it even started... What are the machine with the wrong IP doing? Is it just forwarding the request, or what is happening?
Why not start by
1. contact the ISP and ask wtf they are doing
2. switch ISPs
How do you think that conversation will go? You phone them, get put through to the call center and surprise surprise its not covered by their script.
Switching too is funny. You are at the mercy of what isp ran a line to you. Starlink will fully cooperate with government too. They are a military contractor after all and also at the mercy of musk deciding dumb stuff.
well after going down a few levels of tech support you do get to someone that 'does nothing' while it the magically works
You make it sound like switching ISP is an option in every part of the world.