Fixing HTTPS issues on old versions of OS X

[maverick28]

Address Book started giving an SSL error lately (the error repeats many times in Console). All my contact pictures are gone in Mac OS X Lion, mails are in place. apple_ID in the path to an image (the contact image) is my Apple ID that I've anonymized. However, the photos in question disappeared from the messages of Google accounts in Apple Mail as well

Address Book    [CardDAVPlugin-ERROR] error retrieving image for url:https://apple_ID%40icloud.com@gateway.icloud.com/contacts/16463060523/ck/card/d38d25a0fbaa31a5cad9a289a2defeb6
Error Domain=NSURLErrorDomain Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made." UserInfo=0x7f95ee2cf3a0 {NSUnderlyingError=0x7f95ee2d4bc0
"An SSL error has occurred and a secure connection to the server cannot be made.", NSErrorFailingURLStringKey=https://apple_ID%40icloud.com@gateway.icloud.com/contacts/16463060523
/ck/card/d38d25a0fbaa31a5cad9a289a2defeb6, NSErrorFailingURLKey=https://apple_ID%40icloud.com@gateway.icloud.com/contacts/16463060523/ck/card/d38d25a0fbaa31a5cad9a289a2defeb6,
NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, NSLocalizedDescription=An SSL error has occurred and a secure connection to the server cannot be made.}

Are there any means to re-route these connections with Squid? Apple domains are on the exclusion list but here we're.

 

[Wowfunhappy]

Apple domains are on the exclusion list but here we're.

I made the built-in exclusion list very specific, it only matches the following regex patterns:

ess\.apple\.com$
^sw.*\.apple\.com$
^iphone-services\.apple\.com

Since none of the URLs in your log match those, I don't think they're getting excluded unless you added additional rules in System Preferences.

Does Contacts just ignore the proxy like Dictionary? You could potentially try running in Terminal:

DYLD_INSERT_LIBRARIES=/Applications/Dictionary.app/Contents/Frameworks/ProxyFix.dylib /Applications/Contacts.app/Contents/MacOS/Contacts

 

[maverick28]

Hi, Johnathan, thanks for clearing that out for me. With my limited knowledge, is this the much-talked "dylib injection"?
How would I undo this command in the event of something going wrong?

 

[Wowfunhappy]

The dylib will only be injected during that single run of the Contacts application. No permanent changes will be made. The goal is to test to see if it does anything—I frankly suspect it won't, unfortunately.

 

[maverick28]

Unfortunately, you were right.

 

[RobJos]

A short while ago I changed providers. This caused some trouble that lasted a few days, until I figured out that the new provider was tackling DNS server settings in a different way than the old one.

In the process I found out that for some reason I hadn't been aware of the 2022 update of our Squid proxy. Its use, for my purposes, has been limited to 1. downloading podcasts in iTunes and 2. having embedded images show in my antique mail application. But after updating and straightening up the issues with the provider, none of it was working anymore.

I ended up literally spending days trying to get it to do its thing again, uninstalling Squid (with the Uninstall script) and reinstalling it, reinstalling the previous version from an archive, following all the precepts to the letter (erasing the Squid certificate, updating cacert.pem etc., as well as flushing the DNS cache, PRAM reset, Safe Boot and what more). All the while rebooting countless times.

To no avail: errors 9838 and 503 in iTunes, and no images, except — occasionally — in emails from Amazon (in this case, strangely, inside the same mail some images would show up and others not, yet all of them were hosted on the same Amazon server.)

All this in OS 10.6.8, Intel machine.

So this morning I was gearing up to come here and post my despair, when, flabbergastingly, everything was back to normal upon booting up. Not a clue why. In 40 years of computing one is bound to see a few miracles, but one of this size?

Robert

Note: Jonathan, when using the Uninstall.command, the output consisted of what looked to me like very curious lines. If you don't mind, I'm PM'ing them to you, in case you have an idea what should be done about them.

 

[Wowfunhappy]

@RobJos I'm glad it's working now. I'm not convinced this had anything to do with the proxy.

The uninstaller log you PM'd me looks normal in terms of the proxy package. The uninstaller tries to remove multiple bundle id's for different components/versions of the proxy; if one isn't found, no harm done.

However, it does look like you have other software on your machine which does not have a bundle identifier, or at least does not list one in the receipt visible to pkgutil. I'm not sure of the details which would cause this to happen, but my guess would be either (A) the software itself was poorly made or (B) at some point you attempted to remove this software, and did so incompletely. Regardless, pkgutil appears to be complaining about these receipts every time it's run. This is probably harmless.

 

[RobJos]

Thanks for the feedback, I do appreciate your technical explanations very much. The first four receipts are related to software installed between 2006 and 2011, which was probably never used and the existence of which I didn't even remember. The text editor I use daily, but in an updated version. So, I assume clutter-hating me can safely delete those receipts?

 

[RobJos]

UPDATE. Today, nothing was working anymore. Back to fuddling about, until I remembered which one of the provider's servers I had been connected to when I got Squid to work the other day. And, lo and behold, podcasts and mails got back in town. I then tested with several other servers (not their whole range), and again things weren't working. How in the world is this possible?

It's a bit ackward to have to make sure to be connected to one specific server in order to have one's mails appear as they should. And I'm pretty sure that trying to explain this to the provider would be useless, as most support people were born after 10.6.8 and tend to get rid of you by telling you to update/change your computer/OS/browser etc.

Is there a place in Squid where IP addresses can be inserted manually?

 

[f54da]

I don't think squid proxy would be useful for mail, since that would be pop3/imap over tls, which wouldn't go through the http proxy. Don't know why one specific server is working for you, it's possible it would work even without squid.

 

[RobJos]

I don't think squid proxy would be useful for mail, since that would be pop3/imap over tls, which wouldn't go through the http proxy. Don't know why one specific server is working for you, it's possible it would work even without squid.

Just tested again to make sure: on that same server, the moment I uncheck Secure Web Proxy (HTTPS) in the Network panel, podcasts downloads are blocked and email images don't show up.

 

[f54da]

Oh my mistake, if you're talking about email resources (embedded images) and podcasts downloads, those are different, they're HTTP requests so the proxy will apply to those. I was assuming you were referring to fetching email itself, since you mentioned provider and I thought it was mail provider (as opposed to ISP).

I'm still not sure what you mean by
>which one of the provider's servers I had been connected to
though, I've never heard of a consumer ISP that offers multihomed connection.

 

[Wowfunhappy]

Is there a place in Squid where IP addresses can be inserted manually?

The answer is probably yes, you can do lots of different routing stuff by editing squid.conf, but echoing @f54da that I don't understand your network setup at all.

By "servers" are you still talking about DNS servers?

Edit: Oh! By "provider", do you mean a VPN provider? In other words, are you using a VPN? IME, using a VPN causes the proxy to be bypassed entirely, but maybe that's dependent on the VPN client (I use Viscocity).

What VPN software do you use? There's probably a way to make this work but I'm not sure what it is?

 

[RobJos]

Oh my mistake, if you're talking about email resources (embedded images) and podcasts downloads, those are different, they're HTTP requests so the proxy will apply to those. I was assuming you were referring to fetching email itself, since you mentioned provider and I thought it was mail provider (as opposed to ISP).

Yes, this is correct. But it's maybe me who wasn't specific enough, in which case: apologies — my “expertise” is made of lots of bits accumulated throughout the years, not always with the necessary precision.

I'm still not sure what you mean by
>which one of the provider's servers I had been connected to
though, I've never heard of a consumer ISP that offers multihomed connection.

I'm not sure what you mean by “consumer ISP” and “multihomed connection”, but @‪Wowfunhappy‬'s post and my reply are probably the answer.

 

[RobJos]

The answer is probably yes, you can do lots of different routing stuff by editing squid.conf, but echoing @f54da that I don't understand your network setup at all.

By "servers" are you still talking about DNS servers?

Edit: Oh! By "provider", do you mean a VPN provider? In other words, are you using a VPN? IME, using a VPN causes the proxy to be bypassed entirely, but maybe that's dependent on the VPN client (I use Viscocity).

What VPN software do you use? There's probably a way to make this work but I'm not sure what it is?

VPN, yes, absolutely, with servers to connect to in many different cities and countries. Sorry if that wasn't evident from the start, I never connect otherwise than through a VPN and have done so sort of forever. I have used Viscosity earlier, but it was giving some problems, can't remember which, so I moved to Tunnelblick a good many years ago. It has worked perfectly well during all that time, also with Squid.

To try and summarise: with my previous provider the two issues I use Squid for were taken care of, regardless of which of their many servers worldwide I was using. With the new provider the two issues came back when connecting to all of the servers I have tested (couldn't do them all), except one. But in the case of this specific server, unchecking Secure Web Proxy (HTTPS) revives the issues, while checking it again cancels them immediately — all this seems to indicate that VPN and Squid do not necessarily exclude each other, not?

I've just taken a look at squid.conf, but even if manually entering the single servers' IP addresses (I have a need for access points in different places) would be an easy task, I wouldn't know where to begin. To someone like me, entries such as http_access allow localhost and http_access deny all are a bit puzzling.

Incidentally, activating the provider's Socks Proxy Server (as I do in browsers and applications that allow for it) at the system level, in that same Network pannel, doesn't seem to have any effect on these particular issues. I say "seem", because there appear to be some caches here and there that are playing games and making it difficult to test things consistently. Plus the fact that you can test downloading a podcast if there isn't a new one there to download.

 

[Wowfunhappy]

Editing squid.conf isn't going to help if nothing is being sent to Squid in the first place. As best as I understand it, traffic needs to go squid → VPN → website in that order, because once it has gone through the VPN it's encrypted and Squid wouldn't be able to do anything.

But in the case of this specific server, unchecking Secure Web Proxy (HTTPS) revives the issues, while checking it again cancels them immediately — all this seems to indicate that VPN and Squid do not necessarily exclude each other, not?

Yes except I have no clue how that could be! Unless there is some openvpn configuration that changes how clients should handle this? Are there any differences in the openvpn configuration file of the working server vs the nonworking servers that stand out?

Another option may be to set up the VPN at the router level, if that's possible for you.

 

[f54da]

vpn should be layer 3 (tun/tap) though, while theoretically any http level proxy is layer 7 so should be transparently on top. Now I don't know how osx in particular handles it though, it might see that a vpn is being used and choose to ignore proxy settings.

 

[Wowfunhappy]

Note that OS X doesn't natively support openvpn, which is what RobJos is using. So the question is what does RobJos's client, Tunnelblick, do.

 

[f54da]

I'd think that anything acting as a tun/tap device would be operating at a layer below the http proxy layer though. It's the application itself that makes the HTTP GET to the proxy (or HTTP CONNECT if tls), this is basically all just l7 traffic that should be transparently forwarded to the layer 3/layer 2 relay.

The only case I can think of in which wouldn't apply is when osx sees that it's using a vpn and somehow bypasses the proxy, which would probably only happen if you're using native vpn (unless there's some other way for OpenVPN to signal as such).

Although now that I think about it, perhaps it's possible that the localhost part of the hop to the squid proxy server is being routed via OpenVPN as well? I assume that this shouldn't happen though.

 

[Wowfunhappy]

Although now that I think about it, perhaps it's possible that the localhost part of the hop to the squid proxy server is being routed via OpenVPN as well?

I don't think so. I use the built-in Apache server that ships with OS X. When I navigate to http://localhost/~myUserName while the VPN is connected, I see the contents of my Sites folder, as expected.

The only case I can think of in which wouldn't apply is when osx sees that it's using a vpn and somehow bypasses the proxy, which would probably only happen if you're using native vpn (unless there's some other way for OpenVPN to signal as such).

...and yet, this does seem to be the case.

Doing some testing, Cocoa apps like Dictionary and Mail (external resources) are indeed not hitting the proxy at all when I'm connected to a VPN (OpenVPN, via the third party Viscosity). These apps are using the proxy because it's set in System Preferences.

However, curl continues to use the proxy, websites download properly and I can see it going through Squid in access.log. Notably, curl doesn't respect System Preferences, it's using the proxy because I set the HTTPS_PROXY environment variable.



And so then... why in @RobJos's case do certain VPN servers continue to work with the proxy? I am wondering again if there's some OpenVPN configuration setting (which might be set for one server and not another) which tells clients whether they are supposed to ignore https proxies. I did some Googling yesterday on that and couldn't find anything.

 

[RobJos]

Some of the points I know how to reply to. I had been wondering about some possible differences too, but as far as I can see the OpenVPN config files are exactly the same, the only difference being of course the IP address (and consequently the local provider at the other end). The settings in Tunnelblick are also, as far as I can judge, exactly the same for all config files.

A couple of things after experimenting a bit more. Geographical proximity of the server has no effect, neither does using TCP instead of UDP. And: on the one "good" server, images embedded in a mail turn up immediately, but with the "bad" ones it takes around 20 seconds before a "non-image":

shows up. I assume that that's the time it takes Squid to try and reach the server before figuring out that it cannot do it. I say Squid, because there's a similar lapse of time before the message shows up that a podcast cannot be downloaded.

Talking of podcasts, there could be a useful indication from the use of http vs. https, but I'd have to experiment a bit more, which is when the next podcast turns up (tomorrow, I believe).

The router — there are other computers in this house, and I cannot risk messing things up for them too. Besides it doesn't belong to me as such. But thanks for the suggestion.

Regarding some of the notions put forward by ‪f54da‬ and discussed further by both of you, I'm totally in the dark, but nonetheless more than thankful for your interest and efforts. I have noticed the expressions “tun/tap” in Tunnelblick's settings (Load “tun/tap” driver automatically/always/never) but I have never touched them.

If you all think it's of any use, I could post screenshots of the settings tabs, and maybe even the 20 lines or so of a config file.

 

[f54da]

@Wowfunhappy

>...and yet, this does seem to be the case.

Hm so that'd probably be the part to investigate. The fact that curl still hits the proxy is reassuring, so it must be somewhere in Apple's Foundation frameworks for http requests that either directly or indirectly bypasses proxy upon use of a vpn. I found https://stackoverflow.com/questions/34794255/check-whether-device-is-connected-to-a-vpn

which seems to indicate that you can check for whether tun/tap is active in output of CFNetworkCopySystemProxySettings(). I wonder if you could run that snippet and see if there's any difference? (I don't know if it works with the setup on older osx though, I recall older osx versions needed a kext for tun/tap support, although googling now it seems native utun support has been there since 10.6.8, so idk). Tbh I haven't used VPNs on osx at all so I don't really know how it affects the network as perceived by osx.

@RobJos yes posting the configs for a working vs not working server would be helpful, as well as screenshots of system preferences network pane and openvpn settings.

 

[RobJos]

@RobJos yes posting the configs for a working vs not working server would be helpful, as well as screenshots of system preferences network pane and openvpn settings.

OK, here we go…

Working:
client
dev tun
proto udp
remote [IP address 1] 1194
auth-user-pass
resolv-retry infinite
nobind
persist-tun
persist-key
persist-remote-ip
cipher AES-256-CBC
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-DSS-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
remote-cert-tls server
verify-x509-name pt name-prefix
key-direction 1
comp-lzo no
verb 3
;ca ca.crt
<ca>
[Certificate]
</ca>
<tls-auth>
[Certificate]
</tls-auth>

Not working (no difference that I can see…):
client
dev tun
proto udp
remote [IP address 2] 1194
auth-user-pass
resolv-retry infinite
nobind
persist-tun
persist-key
persist-remote-ip
cipher AES-256-CBC
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-DSS-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
remote-cert-tls server
verify-x509-name pt name-prefix
key-direction 1
comp-lzo no
verb 3
;ca ca.crt
<ca>
[Certificate]
</ca>
<tls-auth>
[Certificate]
</tls-auth>

Tunnelblick main:

​

Tunnelblick advanced:

​

Network CP:

​

(note that having SOCKS Proxy — the provider's — checked or not seems to make no difference…)

Just tell me if you need more.

 

[RobJos]

iTunes test, now. I realised that one of the podcast's servers had an https:// address, whereas most if not all the other ones were http://. So I duplicated it and changed the https to http. The result:

“Bad server”
https: Service unavailable
http: Service unavailable, but shows title of episode (apparently accesse something more) then stops, error 503.

“Good server”: episode downloads fine, both with https and http.

 

[f54da]

Yeah the config looks identical modulo ip, and there's not a separate interface for utun in network preferences so I'd assume from the system's perspective the proxy settings should still be applied. It's still not clear to me why it would work with one server address but not another.

Probably next step would be to confirm whether or not requests made with vpn are hitting the proxy server or not, I assume you can check squid logs for this.

And @Wowfunhappy, if you have free time can you dump the output of CFNetworkCopySystemProxySettings() with vpn enabled/disabled to see why the system skips the proxy in the former in your case?