Fixing Mavericks's Outdated HTTPS

[maverick28]

What I was talking about was what happened to me when I first installed Squid: it screened all traffic, including stock apps but lifted some restrictions on iTunes. As a result, though iTunes Store auth and all the frills got back to life (except downloading and updating apps), iMessage and Facetime fell down. However, by looking into access.log I discovered some domains to which something was trying to connect and excluded them. I noticed that .configuration and .ess subdomains were responsible for iMessage and Facetime, and apparently that worked for @otetzone too. The gist of the further conversation was that he thought it's possible to recover apps downloading/update function which, unfortunately, is gone for good unless updated to iTunes 12.6 which both of us not willing to do.

 

[Wowfunhappy]

Okay, maybe the reason I was confused was because I don't use iTunes.

But, the stock configuration that I include in the download—and use myself—should not break any built-in apps other than iTunes, and it should fix all HTTPS problems on any site other than apple.com. This is not what was happening.

I would make sure everything else works before attempting to get iTunes working.

---

On a seperate note, maverick28, if you can provide a config that fixes (most of) iTunes without breaking any other functionality, I'd love to make that the default. The key principle it needs to follow is "first, do no harm"—something that originally worked without the proxy should never stop working with the proxy. I wasn't clear if that applied to the configs you shared previously.

 

[maverick28]

On a separate note, if you can provide a config that fixes (most of) iTunes without breaking any other functionality, I'd love to make that the default. The key principle it needs to follow is "first, do no harm"—something that originally worked without the proxy should never stop working with the proxy.

With iTunes you don't have to do anything, leaving as is. Your proxy works well where deeper issues aren't involved (I'm pretty sure iTunes updates are quite complex because they install new frameworks and commerce processing methods: that's why you don't have any problems in iTunes 12 - Apple pushes their new apps on you for a particular reason). In other cases, any proxy isn't able to change the situation radically and work magic. Talking about the Squid certificate I went full throttle and allowed it to be trusted with regard to every policy, as a safety net: I don't know whether that made a difference but I did that.
I have included only a few of other subdomains in addition to the default ones: .ess.apple.com, .configuration.apple.com, .icloud.com.

I now recall starting to have a minor issue in MAS (macOS 10.9) when clicking on "Updates" would bring an error pull-down. I then tried to use access.log to exclude su.itunes.apple.com (and smth else) which I noticed appearing in the log, but that cut me off of iTunes Store again, so I dropped that.

Can be that the problems with connecting to your site are related to certificates on both ends? I also noticed that entering or clicking on wowfunhappy.com redirects to a different domain name and the access log on my machine contains only entries with jonathanalland.com but not wowfunhappy.com as opposed to otetzone's access log where I saw the latter rather than the former. Could the culprit lie in the faulty redirection? If yes, then why?

 

[Wowfunhappy]

I have included only a few of other subdomains other than the default ones included by you: .ess.apple.com, .configuration.apple.com.

This had an effect? Did you remove .apple.com first?

The reason I'm super surprised is because .apple.com (with a . in front) is supposed to implicitly include every Apple subdomain. This comes straight from the Squid documentation, and it's what I see in my own testing too.

Good to know iTunes works out of the box! (Not that I'm surprised—it should work out of the box—I guess I misunderstood your earlier posts.)

Talking about the Squid certificate I went full throttle and allowed it to be trusted with regard to every policy, as a safety net: I don't know whether that made a difference but I did that.

It definitely doesn't.

(I'm pretty sure iTunes updates are quite complex because they install new frameworks and commerce processing methods: that's why you don't have any problems in iTunes 12 - Apple pushes their new apps on you for a particular reason). In other cases, any proxy isn't able to change the situation radically and work magic.

Just to be clear, it isn't about complexity—the proxy works for any software that actually uses the certificates in Keychain Access. Apps that use certificate pinning ignore the certificate we added to Keychain Access, so if we try to intercept their traffic, the apps (correctly) notice the dependency and freak out, because they think they're being hacked!

 

[maverick28]

This had an effect? Did you remove .apple.com first?

The reason I'm super surprised is because .apple.com (with a . in front) is supposed to implicitly include every Apple subdomain. This comes straight from the Squid documentation, and it's what I see in my own testing too.

Yes, I replaced the top-level domain .apple.com with some of its subdomains. It had an effect, and, frankly, I wasn't surprised since it's sensible: you just narrowing down the domain area of the application of your rules. And because of that, we exclude only specific domains from proxying. Now, how the Squid's implementation of a proxy technology works is another topic.
If certificate pinning plays such an important role then no wonder that such apps as Facetime and iMessage crumble as soon as we try intercepting their traffic, and that's why you can't log into Facetime in Lion (aside of plethora of other reasons). Another wonder is that Squid revived iTunes Store, although not without some 'peculiarities': as evidenced by examining accept.log both audio- and video-previews apply the same URL scheme, however, the latter is playable, while the former isn't. So, is iTunes 'pinned' or not? (Rhetorical question).

 

[otetzone]

Wait, now I'm confused! So Facetime originally worked before you installed Squid, but not after you installed Squid but disabled the proxy? That doesn't make any sense!

Yessir, exactly! iMessage and Facetime worked just fine on Mavericks, had absolutely no problem with that. The reason I went for Squid was to revive iTunes App Store and to become being able to download and update apps. This was the one and only reason. Except for that one thing everything worked as it should have. Actually nothing had changed since I installed Squid. Issues started to show up as I played with the entries in the .conf file.
I got home recently, what do you suggest - remove everything installed and try it on a new user account, make a clean install or just forget it altogether? Damn, I just can't put up with inability to manage the apps the way I used to. I'm beyond frustrated with that.
I'm also surprised to hear that people had problems with Facetime and iMessage on Mavericks, it has always been fine with me. I had problems logging into those two on ML though. That was the reason I moved to Mavericks.

 

[Wowfunhappy]

Yes, I replaced the top-level domain .apple.com with some of its subdomains.

Oookay, that makes much more sense! When you initially just said you "added" domains that threw me for a loop!

Another wonder is that Squid revived iTunes Store, although not without some 'peculiarities': as evidenced by examining accept.log both audio- and video-previews apply the same URL scheme, however, the latter is playable, while the former isn't. So, is iTunes 'pinned' or not?

Some requests user certificate pinning; others don't.

Yessir, exactly! iMessage and Facetime worked just fine on Mavericks, had absolutely no problem with that.

Okay! When you disabled the proxy as a test, did you also click "Apply"? I think this all might have all just been a big miscommunication.

 

[otetzone]

When you disabled the proxy as a test, did you also click "Apply"? I think this all might have all just been a big miscommunication.

Of course.

I ended up removing everything (Squid) and reinstalling the system over (not a clean install) to find that Facetime would be able to call only my own number. However incoming audio Facetime calls from my phone wouldn't have an "Accept" button on my laptop though, only "Decline" and instead of real number it would report "null". Video calls come through fine both directions though. As I tried to call someone else from my laptop, no matter video/audio, their iphones just don't receive the call at all. However, when someone calls ME, I can accept calls both video and audio just fine as a receiving end. Accept button is there and their numbers are seen. I'm not sure if I made myself clear here, English is my second language. The situation is so complicated.

iMessage works fine though.

 

[Wowfunhappy]

@otetzone

On one hand, if you've disabled the proxy in System Preferences, then OS X should not be sending any traffic through the proxy, and any problems you're encountering should be unrelated to the proxy.

On the other hand, I can't explain why Facetime was working for you before you installed Squid, but is not working now, even when Squid is disabled. That just doesn't make any sense.

So, I don't know what you should do. My gut is there's some type of weird server thing going on with Facetime. It might just fix itself within a few days.

 

[Wowfunhappy]

On a separate note, I spent some time this morning trying to narrow down the list of excluded Apple subdomains to the absolute minimum required. As far as I can tell, there are only two (patterns of) of subdomains which need to be excluded:

ess.apple.com — this is the subdomain Apple uses for encryption keys, so I can understand why they decided to protect it. https://blog.quarkslab.com/imessage-privacy.html. This exemption fixes Facetime and iMessage.

sw*.apple.com – these are the subdomains that distribute software updates. This exemption fixes the App Store and iTunes.

Excluding config.apple.com and icloud.com did not seem to have any effect in my testing.

I've attached the configuration I'm using. With this:

• iTunes works, including logging in, purchasing songs from the store, and downloading purchased songs. Downloading iOS apps and iOS app updates is not possible, but this does not appear to be an SSL problem—the requests are going through without errors—and it also happens without the proxy.
• Game Center (another app I don't normally use) gives a warning the first time you log in, but this also happens without the proxy. It also seems to still work fine?
• iMessage, Facetime, and iCloud work.
• Location services work.
• The Mac App Store and software updates work.

I've attached the configuration. If no problems come up in the next few days, I'll update the download in the first post to use this as well. I'm also considering creating a more polished/streamlined installer.

 

[maverick28]

Sorry for my 2 cents again, but in order to exlude domains, one could do without tweaking that conf file. All you have to do is include the same domains at the GUI level, by typing them in the text box at the bottom of the Proxy view, like so (blurred part of the image).

Speaking of issues with iMessage and Facebook, it seems that Squid has either undocumented or poorly documented characteristics of what it does and how it does its job. Also, it's the way it was compiled using MacPorts or the source code. Something's definitely overlooked here. For example, can it inadvertently cause other connections to bog down. Theoretically, it shouldn't, but how does macOS co-operate with proxy such as Squid?
Here's a support article by Apple which lists ports that should be opened in the event that firewall and other security software are enabled. It doesn't list port 3128, though, but is Squid not coopting it occasionally?

If you use FaceTime and iMessage behind a firewall - Apple Support

 

[Wowfunhappy]

Sorry for my 2 cents again, but in order to exlude domains, one could do without tweaking that conf file.

Yep, I've actually started using this for some other things! System Preferences doesn't allow me to use regex patterns, however, and I don't have a way to "build in" those addresses by default.

The way I think of it is, squid.conf is for addresses that I've decided should always be excluded from the proxy for technical reasons, whereas System Preferences gives users a way to add additional domains as they see fit.

Here's a support article by Apple which lists ports that should be opened in the event that firewall and other security software are enabled. It doesn't list port 3128, though, but is Squid not coopting it occasionally?

Squid isn't a firewall, and most firewalls don't man-in-the-middle SSL traffic. 3128 is just the default listening port for Squid, it's not something used by Apple.

Squid's documentation is indeed somewhat poor, but it is very robust, reliable software. It's been around for a long time, and it's used by a lot of large servers/businesses/organizations. I initially tried some alternate proxies (mitmproxy, Charles), and they were slower and buggy.

Intercepting requests does have an inherent performance impact, but it should be minimal, both because Squid is lightweight and because you're running the proxy on your own machine. There's no other way we can practically fix SSL.

(I did initially want to modify Apple's Security Framework instead, to give the OS itself support for newer cipher suites. Unfortunately, just getting the thing to compile ended up being a complete nightmare! I don't know why Apple even bothers releasing source code, when they leave out so much it can't be used for anything... https://apple.stackexchange.com/a/399650/150839 )

 

[otetzone]

Huh..
Let me put it this way. With no Squid installed iMessage works fine as well as updates (got one offered immediately after reinstalling OSX) and the Mac App Store works and iCloud is fine. Facetime has a couple of unimportant (to me) glitches and it's still impossible to download apps through iTunes - then is there any reason for me to install Squid?

On the other hand, may I be missing something?

 

[maverick28]

(I did initially want to modify Apple's Security Framework instead, to give the OS itself support for newer cipher suites. Unfortunately, just getting the thing to compile ended up being a complete nightmare! I don't know why Apple even bothers releasing source code, when they leave out so much it can't be used for anything... https://apple.stackexchange.com/a/399650/150839 )

That's because Apple is not a subject of economy, it's an esoteric company just like Ahnenerbe and Black Sun Brotherhood.

 

[Wowfunhappy]

Huh..
Let me put it this way. With no Squid installed iMessage works fine as well as updates (got one offered immediately after reinstalling OSX) and the Mac App Store works and iCloud is fine. Facetime has a couple of unimportant (to me) glitches and it's still impossible to download apps through iTunes - then is there any reason for me to install Squid?

On the other hand, may I be missing something?

In Mavericks, any app that use Apple's network library (most of them) will be unable to connect to HTTPS servers that don't support older cipher suites. This is going to become increasingly common as time goes on.

Here's a non-exhaustive list of problems I encountered in Mavericks, that have been fixed since I started using Squid.

• In Apple Mail, lots of emails would have broken images. Now, they all appear correctly.
• My RSS reader (NetNewsWire) was unable to load feeds from certain websites, such as daringfireball.net. With Squid, they work fine.
• I couldn't sync the Delivery Status dashboard widget with my Junecloud account. Now, I can log in without problems.
• Lots of third party apps that used Sparkle were unable to check for updates. Now they can. (Other Sparkle apps were fine, it was dependent on the server.)
• Transmission was unable to connect to certain trackers, which it's now able to use.

Squid isn't a magic bullet—it only fixes https problems. If a server or API has straight up gone offline, Squid won't do anything. But, I don't think there are any downsides to using it.

 

[otetzone]

Squid isn't a magic bullet—it only fixes https problems. If a server or API has straight up gone offline, Squid won't do anything. But, I don't think there are any downsides to using it.

Since I'm not that good at figuring out the issues, would that be a good idea to hold on until the thing is finished the way I wouldn't be bothering you or Maverick28 with stupid questions? Or it's worth to try again with the new .conf you attached a couple of messages above?

Did I get it right that there's absolutely no hope to fix download apps in iTunes app store, no matter which version of iTunes I'd be using?

 

[Wowfunhappy]

I absolutely don't mind you "bothering" me with questions—it helps make the overall setup more robust. I do feel guilty when I don't know what's wrong!

Did I get it right that there's absolutely no hope to fix download apps in iTunes app store, no matter which version of iTunes I'd be using?

I don't use iTunes generally, and I only tested iTunes 11. I'd be happy to take a quick look at what happens in iTunes 12, if that would be an option for you.

Just out of pure curiosity, can I ask why iOS app downloads in iTunes are important for you? I find it so much easier to manage iOS apps on my device itself.

 

[otetzone]

Thank you!

I, on the other side, feel guilty when the only thing I can say is that "this thing doesn't work here and here" without even providing a slightest clue myself. I don't consider myself the typical "press to play" computer user however I'm not as advanced as, say, writing some code.

Anyway, thank you for your work and for your kind words and for your patience. I'll do what' I can on my side. I'm going to backup the Macbook as it is now and then try Squid again with the new confing and report here. If there are questions that are unsolvable then I'd just fresh install Mavericks and add iTunes 12.6 and we'll try from there. I hope this may help.

My best bet would be using iTunes 10 (or lower) or 11 on Mavericks, however we'll see how it goes with 12 and maybe that would help to get to the point. I'll be back home at night (it's morning here now) and see what I can do. Please let me know if it's a necessary move to make a fresh install of Mavericks to try Squid from there. I'll do it if it could help you to understand what's going on there.

Worst case scenario, I'll restore from the backup.

 

[otetzone]

@Wowfunhappy
Okay, installed Squid according to instructions with the new squid.conf (The same system, haven't yet clean installed, just trying on the current one so far)

Results so far -
- iMessage works fine
- iTunes 11 shows all parts of the store (Music, Video, Apps etc) however still unable to download or update the apps.
- Facetime works only if I'm receiving calls, both audio and video. And it doesn't work if I try to call someone, no matter video or audio. The calls just don't come through. I mean I can hear the calling sound on my machine but the receiving end doesn't get any calls as if I hadn't called them at all.

Squid definitely is working, there are two instances of it in Activity Monitor just like it was before. I think I published the screenshot.

 

[otetzone]

Just out of pure curiosity, can I ask why iOS app downloads in iTunes are important for you? I find it so much easier to manage iOS apps on my device itself.

The reason behind it is that I usually back up my apps somwhere in case the updated version breaks something. This way I can go back and install the older version that I had backed up. If I update apps on the device itself and some functionality is broken then there's nothing I can do, there's no option to get the older version of the app back.

I as well have several devices running older iOS. If I download an app already updated for the recent versions of iOS, then when I download it on my older devices after the app is already on my iTunes account, instead of getting that "This app is incompatible with your iOS version, you need to update" message, it would give me the option to download last compatible version for the older iOS that I have.

If there are other ways to reach these goals I'd be happy to know.

By the way, do you want me to install iTunes 12.6 in order to see if it would be downloading apps or we could start with outgoing Facetime calls first? Is iTunes 10 even an option or we should go straight to 12?

 

[maverick28]

Did I get it right that there's absolutely no hope to fix download apps in iTunes app store, no matter which version of iTunes I'd be using?

I know for sure iTunes 12.6 works in full, I have checked it myself at some point. Managing apps is definitely a lot more comfortable within iTunes Store rather than having to pick you device every time (especially if you're spending long hours in front of your desktop).

 

[otetzone]

I know for sure iTunes 12.6 works in full, I have checked it myself at some point. Managing apps is definitely a lot more comfortable within iTunes Store rather than having to pick you device every time (especially if you're spending long hours in front of your desktop).

Oh, that's great to hear! You mean, you can download and update apps? This is unbelievably good news, I think I should try it then, though I really hate the circle icons and overall UI of iTunes 12. I'd rather have it work on another machine with High Sierra in case it's possible. At least it looks consistent with 10.13 UI.

Just in case, did you try the same with iTunes 10 on Mavericks machine?

I would also like to ask you whether outgoing Facetime calls work for you. So far I can only receive calls.

ps. You were right, it was my fault that you couldn't send me PM. I have checked my account settings and now everything should be okay.

 

[Wowfunhappy]

I haven't tested outgoing Facetime calls with the configuration I posted yesterday—that will have to wait until I have an opportunity to call someone, as I don't want to bother them. :3 So it's possible I'll need to add an additional subdomain.

I think you should go ahead and try different versions of iTunes.

Edit: realized I can just call myself... should have thought of that! Let me see what's going on...

 

[otetzone]

Edit: realized I can just call myself... should have thought of that! Let me see what's going on...

I tried that and that worked for some reason, however it wouldn't work for anyone else's number that I tried. Weird.

I'll try iTunes 10 and will let you know.

 

[maverick28]

Just in case, did you try the same with iTunes 10 on Mavericks machine?

I use iTunes 10 only in Lion, not Mavericks. In Lion the behaviour is similar to that in Mavericks: Genius works, iTunes Store icons load, audio-previews - no, video-previews - yes, podcasts - yes, app downloads/purchases/updates - no. I'm pretty confident iTunes 10 in Mavericks won't do what it doesn't do in Lion and vice versa.