Fixing HTTPS issues on old versions of OS X

[Wowfunhappy]

I originally posted about this in the Mavericks section, where it never belonged, because it also works on 10.6–10.8. Here's a new thread for our fancy section!

If you're currently running an old version of OS X, and it's connected to the internet, you probably already have this line somewhere in the Console:

CFNetwork SSLHandshake failed (-9836)

No need to go looking for it. If you're on OS X 10.9 or below, just open the Dictionary app and try searching for something on Wikipedia. Dictionary will tell you that Wikipedia's servers are unavailable—an obvious lie, since it loads elsewhere. And that little SSLHandshake error will appear in the console.

Here's what's going on: Your version of OS X does not support newer versions of https. Wikipedia, like many modern websites, does not support older versions of https. This creates an impasse where neither can talk to each other.

To fix this problem, you need an interpreter—a piece of software which sits between you and the server, intercepts your traffic, and modifies it to be compatible. The technical term for this (which you've probably heard somewhere) is a "proxy server", and more specifically a "MitM" or "man-in-the-middle" proxy which decrypts and re-encrypts your traffic before sending it on its way.

After a bit of work, I was able to configure the open source Squid to do what was needed. And I discovered it fixed a lot of problems—Apple Mail is no longer filled with broken images, dashboard widgets like Deliveries Status no longer fail to connect, and NetNewsWire is able to load all of my RSS feeds. The Dictionary app initially refused to use the system proxy, but then a user on StackExchange told me how to patch it, and now it works too.

I ended up putting the whole thing together into an easy-to-install package. If you're using OS X 10.6 (Snow Leopard), 10.7 (Lion), 10.8 (Mountain Lion), or 10.9 (Mavericks), I encourage you to give it a try. It won't work magic—a couple of people who emailed me seemingly expected it to fix every website in existence—but it does help.

https://jonathanalland.com/downloads/legacy-mac-proxy.dmg

P.S. Chromium does not play nicely with this proxy. If you use Chromium Legacy, please download it via this prefpane, which will set up Chromium to ignore the system proxy. It doesn't need it anyway.

 

[RogerWilco6502]

I originally posted about this in the Mavericks section, where it never belonged, because it also works on 10.6–10.8. Here's a new thread for our fancy section!

If you're currently running an old version of OS X, and it's connected to the internet, you probably already have this message printed somewhere in the Console:

CFNetwork SSLHandshake failed (-9836)

No need to go looking for it. If you're on OS X 10.9 (Mavericks) or below, just open the Dictionary app and try to search for something on Wikipedia. Dictionary will tell you that Wikipedia's servers are unavailable—an obvious lie. And that little SSLHandshake error will appear in the console.

Here's what's going on: Your Mac does not support newer versions of https. Wikipedia, like many modern web servers, does not support older versions of https. This creates an impasse where neither can talk to each other, with the result being that your Mac thinks the server is down.

To fix this problem, you need an interpreter—a piece of software which sits between you and the server, intercepts your traffic, and modifies it to be compatible. The technical term for this (which you've probably heard somewhere) is a "proxy server", and specifically an "MitM" or "man-in-the-middle" proxy server which decrypts, modifies, and re-encrypts your traffic before sending it on its way.

After a bit of work, I was able to configure the open source proxy software Squid to do what I needed. And I discovered it fixed a lot of software—Apple Mail is no longer filled with broken images, dashboard widgets like Deliveries no longer fail to connect, and NetNewsWire is able to load all of my RSS feeds. The Dictionary app initially refused to use the system proxy, but then a user on StackExchange told me how to patch the Dictionary app, and now that works too†.

I ended up putting the whole thing together into a little easy-to-install package. If you're using OS X 10.6 (Snow Leopard), 10.7 (Lion), 10.8 (Mountain Lion), or 10.9 (Mavericks), I'd encourage you to give it a try. It won't work magic—a couple of people who emailed me seemingly expected it to fix every website in existence, which it did not—but it helps a lot!

https://jonathanalland.com/downloads/wowfunhappy-https-proxy.dmg

----------

† But this part works best on Snow Leopard and Mavericks. On Lion and Mountain Lion, the installer needs to perform a diff patch on the binary; I haven't tested this extensively and it may fail.

Epic! I'll have to try this out. Just one question: How's the security? Can you give the path of any information transmitted through the proxy and does unencrypted information ever leave your computer in a way that could compromise security? I ask because it deals with Mail and such and I'm just curious how it might handle potentially sensitive information.

Thanks~

 

[Wowfunhappy]

Epic! I'll have to try this out. Just one question: How's the security? Can you give the path of any information transmitted through the proxy and does unencrypted information ever leave your computer in a way that could compromise security? I ask because it deals with Mail and such and I'm just curious how it might handle potentially sensitive information.

Thanks~

Oh, yes, it's a local proxy server running on your own Mac. All of the decryption and re-encryption happens locally, before anything is sent outside of your computer/network.

If/when you run the installer, it will ask you to add a certificate to Keychain Access (I very purposefully avoided automating any of this, so that it's transparent), and this certificate is what gives Squid the special ability to decrypt your traffic. The certificate itself is also generated on your own computer by the installer, because I wanted to make sure it was different for every person.

If you can't tell, I'm trying to be very explicit about how all of this works! Installing hidden man-in-the-middle proxies is a common malware tactic, for injecting ads and spying on you and whatnot. This package is not doing any of that, and I'm trying to go out of my way to ensure everything happens with the user's knowledge.

Please note that I'm not a security expert, or even a professional software developer, so it's always possible I've configured something terribly wrong! I also have no plans to keep the underlying version of Squid up to date, although since it's set up to only accept connections from localhost, the attack surface should be pretty small.

On the other hand, the reason so many servers don't support older cipher suites and TLS implementations is because those systems are hackable, so using Squid should make you more secure in that regard.

Anyway, I hope that helps you make an informed decision. Please do always feel free to ask me these types of questions!

 

[RogerWilco6502]

Since the proxy is running on your own machine, all of the decryption and re-encryption happens locally, before anything is sent outside of your computer/network. So it goes: software on your Mac (encrypted) → Squid running on your Mac (decrypt traffic, then re-encrypt it) → outside world. And the reverse for incoming data.

If/when you run the installer, it will ask you to add a certificate to Keychain Access (I very purposefully avoided automating any of this, so that it's transparent), and this certificate is what gives Squid the special ability to decrypt your traffic. The certificate is also generated by the installer, because I wanted to make sure it was different on every computer.

If you can't tell, I'm trying to be very explicit about how all of this works! Installing hidden man-in-the-middle proxies is a common malware tactic, for injecting ads and spying on you and whatnot. This package is not doing any of that—it doesn't modify the unencrypted data at all—but I want everyone who uses it to understand what's going on.

Please note also that I'm not a security expert, or even a professional software developer, so it's always possible I've configured something terribly wrong! I also have no plans to keep the underlying version of Squid up to date, although since it only accepts connections from localhost, the attack surface should be pretty damn small.

Of course, do also remember that the reason so many servers don't support old cipher suites is because they're insecure, and using Squid fixes that. So it could make you more secure in that regard.

Anyway, I hope that helps you make an informed decision. Please do always feel free to ask me these types of questions!

Ok, thanks for the clarification and the transparency!

 

[maverick28]

For those who want to install it on Lion: it works perfectly too and in some cases even smoother than with Mavericks. Lion doesn't have a bug when some login items, including agents and demons for processes - Squid is among them, won't load on logging into your account. In Mavericks that leads to a workaround which is that Squid's manual launch has to be simulated by a 15-sec delay, while in Lion it kicks in right away. So you're safe to go.

 

[wicknix]

Nice. Thanks. No problem setting it up in snow leopard.

 

[RobJos]

Sounds interesting. 2 quick questions. is there a risk that this local proxy gets into conflict with proxy used by my VPN service? And: in case of trouble, is it easy to uninstall?

 

[Wowfunhappy]

Sounds interesting. 2 quick questions. is there a risk that this local proxy gets into conflict with proxy used by my VPN service? And: in case of trouble, is it easy to uninstall?

I know when I enable Viscocity (an OpenVPN client), the proxy doesn't get used. It doesn't cause problems per se, but you're stuck with stock HTTPS support until you disconnect from the VPN. I imagine whatever you're using would be similar.

 

[RobJos]

I know when I enable Viscocity (an OpenVPN client), the proxy doesn't get used. It doesn't cause problems per se, but you're stuck with stock HTTPS support until you disconnect from the VPN. I imagine whatever you're using would be similar.

Thanks, but honestly I don't quite understand this part, probably due to my limited understanding. I used to use Viscosity, but had to move to Tunnelblick some time ago. The browsers' proxy settings are set to “SOCKS v.5” + "Use proxy to perform DNS queries", whereas in SnL's Network panel only the "SOCKS Proxy" is checked, without any other specification. Proxy server and port are set by my VPN provider. Does your reply imply that I'd have to check either one, or both, off, for your application to do what it's meant to do? Or that they would be automatically disabled while your local proxy is running?

 

[Wowfunhappy]

With Viscocity, the proxy stops being used automatically whenever the VPN is enabled, you don't have to do anything. But for your particular setup, you'll just need to experiment and find out I'm afraid!

 

[maverick28]

In my particular case, I want to route all iPhone connections through my Mac to make it use the same DNS resolution scheme written into the local hosts file to disable iOS update notifications and get rid of the red badge. I read on StackExchange that you need to run a server utility and tell iPhone to use its local IP instead of that of the router, so I installed dnsmasq and don't know what the next move should be. The goal is to keep Squid intact, and dnsmasq requires extensive networking knowledge to understand its frightening abundance of parameters. I'm stuck for a year now.

 

[RobJos]

With Viscocity, the proxy stops being used automatically whenever the VPN is enabled, you don't have to do anything. But for your particular setup, you'll just need to experiment and find out I'm afraid!

OK, I think that one of the reasons I wasn't understanding is that my VPN is always on, no exceptions. Actually, it's a bit beyond me why anyone would use a VPN and not have it on all the time (I know, there can be country-determined exceptions.) But I'll give it a try with Tunnelblick and see what goes. Please correct me if I'm still not with.

 

[Project Alice]

What exactly do you have squid configured to do?
I have squid already set up on my router, so all the Macs are already using squid. But it’s just set up for web filtering and clamAV.
I still have SLL errors on 10.4-10.6, when trying to do something as simple as install TigerBrew.

 

[Wowfunhappy]

What exactly do you have squid configured to do?
I have squid already set up on my router, so all the Macs are already using squid. But it’s just set up for web filtering and clamAV.
I still have SLL errors on 10.4-10.6, when trying to do something as simple as install TigerBrew.

You need to enable a feature that Squid calls "SSL Bump" in order for it to reencrypt https traffic.

Here's the full config file that gets included with the installer. You should be able to adapt this to your router, with a couple of changes—for instance, you will want to allow connections that come from computers other than localhost!

http_port 3128 ssl-bump generate-host-certificates=on cert=/Library/Squid/Certificates/squid.pem key=/Library/Squid/Certificates/squid-key.pem

tls_outgoing_options cafile=/Library/Squid/Certificates/cacert.pem
sslcrtd_program /Library/Squid/security_file_certgen

acl excluded_domains ssl::server_name .pypi.org .pythonhosted.org
acl apple_domains ssl::server_name_regex ess\.apple\.com$  ^sw.*\.apple\.com$
acl local_addresses ssl::server_name_regex ^192\.[0-9]+\.[0-9]+\.[0-9]+$ ^10\.[0-9]+\.[0-9]+\.[0-9]+$ ^172\.(1[6-9]|2[0-9]|3[01])\.[0-9]+\.[0-9]+$
acl loopback_addresses ssl::server_name_regex ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ^::1$
acl excluded any-of excluded_domains apple_domains local_addresses loopback_addresses
ssl_bump splice excluded
ssl_bump bump all

acl fetched_certificate transaction_initiator certificate-fetching
cache allow fetched_certificate
http_access allow fetched_certificate
sslproxy_cert_error deny all

http_access allow localhost
http_access deny to_localhost
http_access deny all

The sets of excluded addresses are "optional", although if you leave them out some software (e.g. iMessage) will break.

Note, you probably don't want Squid to intercept the https traffic of every device on your network! Whatever root certificate Squid is configured to use needs to be trusted by the device whose traffic you're modifying. Otherwise, the device will assume it's being hacked (because it is, in a way) and freak out.

And, a lot of modern devices don't let you add your own certificates!

 

[Project Alice]

You need to enable a feature that Squid calls "SSL Bump" in order for it to reencrypt https traffic.

Here's the full config file that gets included with the installer. You should be able to adapt this to your router, with a couple of changes—for instance, you will want to allow connections that come from computers other than localhost!

http_port 3128 ssl-bump generate-host-certificates=on cert=/Library/Squid/Certificates/squid.pem key=/Library/Squid/Certificates/squid-key.pem

tls_outgoing_options cafile=/Library/Squid/Certificates/cacert.pem
sslcrtd_program /Library/Squid/security_file_certgen

acl excluded_domains ssl::server_name .pypi.org .pythonhosted.org
acl apple_domains ssl::server_name_regex ess\.apple\.com$  ^sw.*\.apple\.com$
acl local_addresses ssl::server_name_regex ^192\.[0-9]+\.[0-9]+\.[0-9]+$ ^10\.[0-9]+\.[0-9]+\.[0-9]+$ ^172\.(1[6-9]|2[0-9]|3[01])\.[0-9]+\.[0-9]+$
acl loopback_addresses ssl::server_name_regex ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ^::1$
acl excluded any-of excluded_domains apple_domains local_addresses loopback_addresses
ssl_bump splice excluded
ssl_bump bump all

acl fetched_certificate transaction_initiator certificate-fetching
cache allow fetched_certificate
http_access allow fetched_certificate
sslproxy_cert_error deny all

http_access allow localhost
http_access deny to_localhost
http_access deny all

The sets of excluded addresses are "optional", although if you leave them out some software (e.g. iMessage) will break.

Note, you probably don't want Squid to intercept the https traffic of every device on your network! Whatever root certificate Squid is configured to use needs to be trusted by the device whose traffic you're modifying. Otherwise, the device will assume it's being hacked (because it is, in a way) and freak out.

And, a lot of modern devices don't let you add your own certificates!

Ah yes, I was looking into this actually and I have it set on SSL splice. It looked like too much of a hassle to get certs on every device that connects.

Perhaps I should just try it out on specific macs on it’s own.

 

[Wowfunhappy]

Heads up for Chromium Legacy users—Chromium doesn't like this proxy for some reason. As of right now, the only fix I know of is to launch Chromium Legacy with --no-proxy-server in the Terminal (or to temporarily disable the proxy in System Preferences), which is obviously annoying/untenable. I'm investigating...

 

[RobJos]

Reporting after installing and testing for a few days...

Browsers: I found it preferable to keep on using my VPN provider's extremely strict security and privacy rules — that's for those 10.6 browsers that allow for setting the proxy settings (not Chromium, for example, although I'm not sure whether this relates or not to the previous post). However, the error code “CFNetwork SSLHandshake failed (-9836)” was nowhere to be found in the Console (logs over a year's time), either because I don't know where to look, or because my browsers (Arctic Fox and SpiderWeb) have spared me from it.

Podcasts: a number of podcasts that a year or two ago had suddenly stopped downloading in iTunes (the provider had no idea why this had happened) are now again accessible for download. Excellent, it only takes telling LittleSnitch (LS) to allow a few Squid connections.

Mailing: on the SMTP side, also a good while ago, Sending Mails using ports 465 or 587 + SSl stopped working altogether in my aging mail application, and I've had to use port 2525 ever since. Also in this case, the provider couldn't explain. Squid hasn't changed anything to this situation. On the Receiving side, on the other hand, a few mails with images that used not to show now appear correctly after authorizing Squid to contact the servers. The problem is that there are at least as many servers as there are senders of mails with embedded images, which makes it really tedious having to tell LS to allow Squid connections to port 443 for every single mail sender. Trouble is, if I told LS to allow connections to port 443 for any server, this would be set across the system. In other words, another application might connect to some port 443 without my being aware of it.

So the question is: is there a way to tell certain applications that use the system's proxy (because they don't have their own proxy settings) to keep on doing so, and tell other applications to use Squid instead? I hope I'm making myself understandable.

Other applications: my anti-virus software, which calls its servers at boot-up, now does so through Squid, and so does Apple's Help Viewer. Other apps such as VBox VMs, Spotify, torrent download, appear to be unaffected by the installation of Squid and function as usual.

Apologies if I'm not using the correct terminology, and of course, if there are things I haven't understood.

 

[Wowfunhappy]

So the question is: is there a way to tell certain applications that use the system's proxy (because they don't have their own proxy settings) to keep on doing so, and tell other applications to use Squid instead?

No, not unless a "please bypass the proxy" feature is built into the application itself, as it is in Firefox. OS X doesn't offer a way to allow specific apps through the proxy, and Squid has no way to know which app is sending it traffic.

What OS X can do is disable the proxy for specific domains. Open System Preferences > Network > Advanced > Proxies, and look for the field at the bottom which reads "Bypass proxy settings for these Hosts & Domains". But I don't think this will work for your use case.

Edit: It really sounds like Little Snitch is the problem here. I'm disappointed it isn't smarter.

Edit2: I'm actually a little confused as to what's going on with Little Snitch. It's been a little while since I last used it, but I thought it usually sets rules per app? Why can't you set Little Snitch to let Squid make connections to anywhere, and then set up rules for the individual apps?

Or does Little Snitch think those apps are just trying to connect to localhost (the proxy address), rather than their real destination? If so, that's dumb, because it means that Little Snitch is just plain broken with proxies—a feature built into OS X!

 

[Wowfunhappy]

The Dictionary patch has been disabled on Lion and Mountain Lion. This is the one part of this package that I would consider a "hack"—everything else is just making clever use of the functionality Apple built into OS X—and I don't want to do it unless I can be sure it's reliable. And, it's not reliable—some users have been reporting problems.

The fix is still present on Snow Leopard and Mavericks. Snow Leopard does not require modifying the binary, and Mavericks does so via a different, safer method.

 

[maverick28]

Wikipedia in Dictionary on Lion, Feb 9 2021.

 

[Wowfunhappy]

Yep, I know it works for you, and it will keep working for you! But I've removed it from the installer going forwards. (Remember, the patch for Lion was based on a binary you sent me—it would seem that somehow, not all binaries are exactly the same.)

If anyone really wants to make the Wikipedia portion of the dictionary app work on Lion or Mountain Lion, just ask and I'll post manual instructions. You will need to use a second Mac on 10.9 or above to do the patch.

 

[RobK88]

Yes, when you get a chance, please post manual instructions to create an updated Dictionary app for Lion and Mountain Lion!

Many thanks!

 

[RobK88]

Has anyone been able to use the squid https proxy in a regular (non administrator) account?

I successfully installed the squid https proxy in an Administrator account on a Mac running Lion. It works great!
But when I switch to a regular (non-administrator) account, I am unable to connect to the squid https proxy.

I checked the keychain. The certificate in the https proxy package is imported for all users and set-up properly.

Any ideas?

 

[Wowfunhappy]

Yes, when you get a chance, please post manual instructions to create an updated Dictionary app for Lion and Mountain Lion!

Many thanks!

Sure thing! (Wow, that didn't take long.) So, basically, you need to do two things:

1. Tell the Dictionary app to load ProxyFix.dylib when it launches.
2. Remove the code signature from the Dictionary binary, so the system won't kill it when you do #1.

The first part is easy. Copy ProxyFix.dylib (in attached) to Dictionary.app/Contents/Frameworks, then run in Terminal:

defaults write /Applications/Dictionary.app/Contents/Info LSEnvironment -dict DYLD_INSERT_LIBRARIES @executable_path/../Frameworks/ProxyFix.dylib

To remove the code signature, copy the binary file Dictionary.app/Contents/MacOS/Dictionary to another Mac running OS X 10.9 (Mavericks) or newer, and download optool (in attached). Then, run in Terminal:

/path/to/optool strip --target /path/to/copied/Dictionary

Lastly, copy your modified Dictionary binary back to your older Mac, replacing the original copy.

Once all of that is in place, the Dictionary should launch correctly and use the system proxy, fixing Wikipedia if Squid has also been set up.

I have not actually run through these instructions on a clean machine to check for gaps, but that's the general process. Definitely make a Time Machine backup, and let me know if you run into problems.

(And if anyone knows of a Lion-compatible way to strip code signatures, please let me know so I can add it to the installer and make this whole thing automatic.)

 

[Wowfunhappy]

But when I switch to a regular (non-administrator) account, I am unable to connect to the squid https proxy.

...interesting. I didn't test that particular scenario on 10.6–10.8, only on 10.9 (where it does work).

Questions:

1. Have you rebooted at least once since running the installer? (If not, please do so and try again.)
2. Is Squid running according to Activity Monitor?