HN Explorer

DuckDuckGo Lite now requires a User-Agent header

10 points by 1vuio0pswjnm7 ↗ HN

    # Test:
    
    x=$(echo x|tr x '\r');
        
    # Without UA header
    
    sed "s/$/&$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof 
    POST /lite/ HTTP/1.1
    host: lite.duckduckgo.com
    content-length: 6
    content-type: application/x-www-form-urlencoded
    connection: close
    
    q=test
    EOF
    
    # With UA header
    
    sed "s/$/&$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof 
    POST /lite/ HTTP/1.1
    host: lite.duckduckgo.com
    content-length: 6
    content-type: application/x-www-form-urlencoded
    user-agent: test
    connection: close
    
    q=test
    EOF

8 comments

[ 2.4 ms ] story [ 36.6 ms ] thread
solarmist
Is that supposed to imply something to me?
pYroStack0
What are the exact consequences of this for the end-user compared to no user agent header? Does it only reveal your used browser and version no.?
opardalis
Yeah, but your 2nd example (with the UA header included) also returns a 400 response code. I am able to POST a search query to lite.duckduckgo.com just fine using another tool while also completely omitting the user agent request header (not empty string, completely omit the UA header field). I think your openssl s_client syntax is wrong, and thats why your getting a 400 status code, 400 is usually malformed request syntax.
1vuio0pswjnm7
That's not what causes the malformed request, but thanks for the heads up. Apologies for the inadvertence.

Try this:

   addcr <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
   POST /lite/ HTTP/1.1
   host: lite.duckduckgo.com
   content-length: 6
   content-type: application/x-www-form-urlencoded
   user-agent: test
   connection: close
   
   q=test
   EOF
addcr is from djb's daemontools.

If do not want to compile/install daemontools, try something like:

   x=$(printf '\r\n');
   sed "s/$/$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
   POST /lite/ HTTP/1.1
   host: lite.duckduckgo.com
   content-length: 6
   content-type: application/x-www-form-urlencoded
   user-agent: test
   connection: close
       
   q=test
   EOF
opardalis
Ok... The above example works now, and with user-agent omitted, I still get search results. My point to others watching/following this thread is that the initial claim is false, lite.duckduckgo.com does indeed return search results if your request header does not have a user agent field.

So to be clear, there is no issue here.

[deleted]
(comment deleted)
1vuio0pswjnm7
NB. If using HTTP/2, then no UA header is needed. The examples I provided use HTTP/1.1

    curl --http1.1 --resolve lite.duckduckgo.com:443:52.142.124.215 -d "q=test" -A "" https://lite.duckduckgo.com/lite/
"Several characteristics of HTTP/2 provide an observer an opportunity to correlate actions of a single client or server over time. These include the value of settings, the manner in which flow-control windows are managed, the way priorities are allocated to streams, the timing of reactions to stimulus, and the handling of any features that are controlled by settings."

HTTP/2 has some fingerprinting enablers that HTTP/1.1 does not. The above is from the HTTP/2 RFC.

Further reading: https://www.blackhat.com/docs/eu-17/materials/eu-17-Shuster-...

1vuio0pswjnm7
There are a variety of uses that a website operator might make of a User-Agent header.

Today, many of those uses are associated with identifying computer users, usually for tracking and/or advertising purposes, sometimes for access control, e.g., limiting access to only "acceptable" clients that run Javascript.

In this case, lite.duckduckgo.com, which can be used to make DDG search requests via POST instead of GET, has returned search results without requiring a User-Agent header since 2021. This is common. Fun fact: The majority of websites on the internet will return requested pages absent a User-Agent header.^1

Given that DDG is in the business of marketing itself as a privacy solution, it is curious to me why DDG is choosing now to collect more data from its users, expecially an item of data that is frequently (mis)used for purposes that contravene "privacy".

DDG was allegedly founded to "respect users' privacy" and to offer an alternative to "filter bubble" search results. The web search providers that DDG is supposedly competing against make heavy use of the User-Agent header for so-called fingerprinting. Fingerprinting can be used for a variety of purposes, but "respecting user privacy" and avoiding "filter bubbles" are not among them.^2

1. I have been using the web this way for many years now, sending only the minimal HTTP headers. I use non-browser clients and a text-only browser, so I have little need for graphical design, "responsive" or otherwise. As such, I send comparitively less data to web servers than if I was using a popular graphical browser. I avoid graphical browsers not for "privacy" reasons, but due to personal preference, although I welcome any "privacy" benefits that come as a result of this choice.

For folks who are concerned about graphics (cf. text), sending a UA header might be useful. YMMV.

Some amount of "fingerprinting" is inevitable if using the internet. Every remote access, regardless of the software used, has a fingerprint. However not all fingerprints are created equal. Some might be composed of less or more data, and thereby less or detail. Arguably, a more detailed fingerprint is generally higher fidelity. Therefore the more details the fingerprinter can collect, the more useful the fingerprint, e.g., for commercial purposes, and consequently, the larger the loss of "internet privacy".

IMHO, it would be a mistake to conclude that the inability to avoid any amount of fingerprinting (something that has never been possible) means all software is sending the same amount of "fingerpint" data to remote servers and thereby all TCP/HTTP clients, including but not limited to web browsers, are the same with respect to their "fingerprintability". They are not. See, e.g., https://arxiv.org/pdf/1703.05066.pdf

2. The UA header can of course be used for "browser sniffing", but keep in mind that in this case lite.duckduckgo.com is very basic HTML. IMHO, it is not a highly stylised web page. Consider also that UA "spoofing" has been common since the early days of the www and today Google is actvely trying to "deprecate" use of this header in favour of "Client Hints". See RFC 8942.