Sudo lsof -i not showing certain ip address, but FF developer tools do?

[humpbacktwale]

So usually, if I wanted to view the connections currently being made by my mac, I would use sudo lsof -i to get a list of processes and ip addresses. However, I have noticed something odd, especially with this site, macrumors. While the network tab of the developer tools for firefox list the ip addresses and domains, for some reason these sometimes do not appear when I run sudo lsof -i.

For macrumours especially, I can see all the loaded assets and their domain and ip addresses using the developer tools. However, these ip addresses never appear when I run sudo lsof -i, or sudo lsof -ni. However, for as far as I can tell, this does happen for every other site.

So is there a reason why this is occurring? As far as I am aware, there should be no reason why the command would ignore or not display certain ip addresses or domains. Before anyone asks, yes I am sure it is not caching them, as they are not listed as cached in the firefox developer tools, and I have cleared my browser cache.

 

[svenmany]

Pardon my ignorance...

I thought lsof -i shows sockets (listening and established), while Firefox shows things that have been accessed. The sockets aren't guaranteed to remain open after Firefox retrieves the resources. Have you checked if the sockets are being closed in Wireshark?

 

[appltech]

If I got correctly what you mean
lsof shows current established connections entries, for deeper or for packet analyze you need something as above mentioned (or check all the ports with connections but that could be madness because TCP and UDP often utilizes the same ports but "barking" separately, so every port would have TCP and UDP separate connections)

 

[svenmany]

I did a little test using Firefox and hitting MacRumors. lsof showed the established connection for approximately 2 minutes after the retrieval of resources. At that point in time, as Wireshark shows, Firefox closed the socket. Firefox sent a "FIN", which triggers a graceful close.

Any chance you aren't checking lsof soon enough? If you are, then something else must be closing the socket. it would be very unexpected, but the server could be closing it. Maybe more likely, the socket is being reset for some reason. Anyway, it's all guesswork on my part.

 

[humpbacktwale]

Pardon my ignorance...

I thought lsof -i shows sockets (listening and established), while Firefox shows things that have been accessed. The sockets aren't guaranteed to remain open after Firefox retrieves the resources. Have you checked if the sockets are being closed in Wireshark?

I believe you are correct. I've just never noticed another site that closes a connection so quickly that I cannot observe it using lsof before

 

[humpbacktwale]

If I got correctly what you mean
lsof shows current established connections entries, for deeper or for packet analyze you need something as above mentioned (or check all the ports with connections but that could be madness because TCP and UDP often utilizes the same ports but "barking" separately, so every port would have TCP and UDP separate connections)

So I understand it shows established connections. I just thought it was odd that I can't observe it's domain or ip using lsof at all, but for pretty much every single other site that comes to mind I can.

 

[humpbacktwale]

I did a little test using Firefox and hitting MacRumors. lsof showed the established connection for approximately 2 minutes after the retrieval of resources. At that point in time, as Wireshark shows, Firefox closed the socket. Firefox sent a "FIN", which triggers a graceful close.

Any chance you aren't checking lsof soon enough? If you are, then something else must be closing the socket. it would be very unexpected, but the server could be closing it. Maybe more likely, the socket is being reset for some reason. Anyway, it's all guesswork on my part.

Hmm, see the 2 minutes is what I would expect to be honest, or at least 10-20 secs for a socket closure. I'm literally checking it as soon as I connect to the site, and whenever a new request appears in the developer tools, and I get nothing :/

 

[svenmany]

Hmm, see the 2 minutes is what I would expect to be honest, or at least 10-20 secs for a socket closure. I'm literally checking it as soon as I connect to the site, and whenever a new request appears in the developer tools, and I get nothing :/

It's a mystery. I'm sure you know that HTTP uses persistent socket connections normally, so I agree that you should see the connections right after the retrieval.

Something is closing those sockets - either Firefox, the server, or some intermediary is resetting it (VPN, router, proxy?). It's atypical for productions servers to close sockets because of TIME_WAIT issues. (Here's a good article https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux). But servers definitely do that sometime. MacRumors didn't do that to me when I tested.

Wireshark would show if the socket is being closed or reset. It would also show which side is initiating that.

 

[humpbacktwale]

The weird thing is, I can see other connections. For example, macrumours uses gravatar, and I can see it in my list of established connections. But the connections to forums.macrumours, both before, and after, do not appear. No VPN's or proxies set up. No clue why my router would randomly do it for this once site. The provider is cloudflare, using 104.22.3.106, so perhaps that has something to do with it? Though inspecting the request header, it says the connection is keep-alive :/

Unless ublock origin or privacy badger can close connections early, I am stumped.

 

[svenmany]

I did a more thorough test this morning - need to start work soon. I'm seeing the same thing as you. I believe the answer is here:

https://en.wikipedia.org/wiki/QUIC

I really don't know anything about it yet. I'm looking forward to studying it over the weekend.

In a nutshell - Wireshark shows no HTTP (or even TCP) traffic as Firefox accesses 104.22.3.106. I have Firefox configured to NOT use the DNS over HTTPS, so I'm able to see the initial DNS query - nothing special there. There is a standard where DNS can return extra information to allow the browser to know certain things and proceed differently when first interacting with the server. Since you mentioned Cloudflare, their blog post on this topic is super interesting. But, I don't see that going on.

The tendency for browsers to keep sockets open after HTTP traffic has passed is part of HTTP; the server and client agree to not close the socket so that it can be reused on subsequent HTTP requests. But, when talking to 104.22.3.106 all traffic is being wrapped in QUIC UDP packets. My intuition tells me that it's completely reasonable that no sockets are left open after this traffic has finished.

Anyway, that might be enough information to satisfy you with respect to this mystery. I don't know if you're interested in more detail after I've studied the situation. If so, then I'll post that here after I've learned some.

 

[svenmany]

My closing remark...

I do see the socket in lsof - it's a UDP socket and it lasts for 30 seconds. QUIC is a UDP protocol. I saw this for 30 seconds:

IPv4 0x5ba9220ea48383cb 0t0 UDP *:56022

Unless you look at something like Wireshark or just tcpdump output, you won't know that 56022 was the port used (in my last test) for communicating with MacRumors. UDP sockets are connectionless, so there's no evidence of the remote host.

If you're interested in QUIC or HTTP/3 and Firefox, check out

https://hacks.mozilla.org/2021/04/quic-and-http-3-support-now-in-firefox-nightly-and-beta/

 

[humpbacktwale]

My closing remark...

I do see the socket in lsof - it's a UDP socket and it lasts for 30 seconds. QUIC is a UDP protocol. I saw this for 30 seconds:



Unless you look at something like Wireshark or just tcpdump output, you won't know that 56022 was the port used (in my last test) for communicating with MacRumors. UDP sockets are connectionless, so there's no evidence of the remote host.

If you're interested in QUIC or HTTP/3 and Firefox, check out

https://hacks.mozilla.org/2021/04/quic-and-http-3-support-now-in-firefox-nightly-and-beta/

Well I can confidently say this is one of the most in-depth responses I have gotten for a query on this forum, ever. I appreciate you taking the time to delve into this. Yes I can also see a UDP port being used when connected to macrumours. So strange I have never heard of this before.

So its perfectly reasonable that we would see nothing given it uses QUIC? How come in your initial reply, you stated you were able to see the traffic? Or were you referring to something else?

Appreciate all the help

 

[unrigestered]

for quick real time monitoring i'm mostly using

nettop -m tcp

change tcp to udp if you want to monitor those connections

 

[svenmany]

Appreciate all the help

The appreciation goes both ways. I love questions in topics that interest me since they provide a great opportunity to learn something. Before this thread, I'd never heard of QUIC or even investigated what HTTP/3 is.

How come in your initial reply, you stated you were able to see the traffic?

It's a bit of a mystery why I saw the TCP sockets when I experimented earlier. I have a guess. In my earlier posts I chose not to include these details in case they would cause some confusion.

For a client and server to communicate using QUIC, the server has to tell the client that it's capable of that. It seems there are two ways for a client and server to do QUIC. One uses extra information returned when the client does a DNS query to figure out the server's IP address. In that case the client queries for a different kind of DNS record; not the A or AAAA types, but rather the HTTPS type. The DNS response tells the client that QUIC is an option at a different endpoint. The client then can choose to use that endpoint. I did confirm that Firefox was not doing that, even though QUIC was subsequently used.

The other way it can happen is that the client, during the initial HTTP / TCP connection, is given a special alt-svc header that tells it that QUIC is available. At that point the client can switch to QUIC. During my later testing, I saw no TCP traffic (therefore no TCP socket using lsof) that would have been required for the original HTTP/2 request.

So, why did Firefox just immediately use QUIC in my later tests? My guess is that Firefox had cached an earlier answer that the server supported QUIC and that answer had not reached its max age. Then, why did I see a TCP socket in my earlier tests? My guess is that the socket was the result of the preliminary TCP connection used to determine that QUIC was supported via the alt-svc header.

Assuming my guesses are correct, how can we explain the coincidence that in my earlier tests I did not have a cached answer already available for MacRumors. My guess is that's because I never use Firefox for MacRumors; I fired it up to access MacRumors precisely to run the test to see if lsof showed the sockets. Since that was the first time in quite a while that Firefox accessed MacRumors, I would expect to see the TCP socket that was used to acquire the alt-svc header during the original HTTP/2 interaction.

So many guesses. That's why I didn't introduce all this mess until you asked.

 

[svenmany]

Not sure why my last post made me investigate a bit more.

At the shell I ran "curl http://www.macrumor.com". Notice the "http" rather than "https". I needed to see the actual content of the initial call in Wireshark. I did get a redirect to the SSL page, but before that happened I did see the response from the server.

Hypertext Transfer Protocol
HTTP/1.1 301 Moved Permanently\r\n
Date: Thu, 05 Oct 2023 17:26:37 GMT\r\n
Content-Type: text/html\r\n
Transfer-Encoding: chunked\r\n
Connection: keep-alive\r\n
Location: https://www.macrumors.com/\r\n
CF-Ray: 81175ee2bd1cf9cc-SJC\r\n
CF-Cache-Status: HIT\r\n
Age: 84\r\n
Cache-Control: max-age=300\r\n
Vary: Accept-Encoding\r\n
x-content-type-options: nosniff\r\n
x-frame-options: *\r\n
x-turbo-charged-by: LiteSpeed\r\n
x-xss-protection: 1; mode=block\r\n
Server: cloudflare\r\n
alt-svc: h3=":443"; ma=86400\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.090013000 seconds]
[Request in frame: 4]
[Request URI: https://www.macrumors.com/]
HTTP chunked response
File Data: 694 bytes
Line-based text data: text/html (13 lines)

The 301 and the Location header is what would send me off to the SSL page (assuming the browser would continue with HTTP/2). Safari does HTTP/2 with MacRumors.

The alt-svc header is what told Firefox that QUIC was available. "ma" is the max age of the response in seconds (in this case 24 hours), which allowed Firefox to cache that information. "h3" means HTTP/3. Firefox does HTTP/3 with MacRumors.

 

[humpbacktwale]

Not sure why my last post made me investigate a bit more.

At the shell I ran "curl http://www.macrumor.com". Notice the "http" rather than "https". I needed to see the actual content of the initial call in Wireshark. I did get a redirect to the SSL page, but before that happened I did see the response from the server.




The 301 and the Location header is what would send me off to the SSL page (assuming the browser would continue with HTTP/2). Safari does HTTP/2 with MacRumors.

The alt-svc header is what told Firefox that QUIC was available. "ma" is the max age of the response in seconds (in this case 24 hours), which allowed Firefox to cache that information. "h3" means HTTP/3. Firefox does HTTP/3 with MacRumors.

An even more in depth reply XD

So you're saying that, for whatever reason, firefox switches to the use of QUIC, which is also identified as HTTP/3, but safari doesn't make this choice? And due to QUIC, udp is utilised instead of tcp, do it wasn't showing up when I used lsof?

 

[svenmany]

An even more in depth reply XD

So you're saying that, for whatever reason, firefox switches to the use of QUIC, which is also identified as HTTP/3, but safari doesn't make this choice? And due to QUIC, udp is utilised instead of tcp, do it wasn't showing up when I used lsof?

Exactly right. You were probably looking for established TCP sockets, to the websites you see accessed by Firefox, in the output of lsof. They wouldn't be there for HTTP/3 connections since QUIC uses UDP.

Firefox has an advanced option to disable HTTP/3, but it's enabled by default (on my machine running 118.0.2). Here is the setting.




I'm not sure why Safari doesn't use HTTP/3. Here is a forum post from 5 months ago discussing that. https://forums.macrumors.com/thread...ing-over-quic-it-uses-http-2-instead.2389022/