Let’s Encrypt vs. iTunes: anatomy of an error delivering Fatal Error
Update, 2016-12-18: FeedPress reports that Apple has silently added support for Let’s Encrypt certificates on podcast feeds.
Soroush and I released a new episode of Fatal Error, our podcast about iOS development, today. During my commute, I noticed that Overcast wasn’t picking up the new episode quicky, which is unusual; usually Overcast notifies me of new episodes unbelievably quickly.
A quick check in the iTunes UI revealed that both this latest episode and our previous one seemed to be missing. So I logged into Podcasts Connect to see if I could find anything useful. There appeared this semi-helpful error message (formatting mine):
Can’t read your feed.
sun.security.validator.ValidatorException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Ah-ha! Just after the release of our last episode, I noticed that Squarespace — where Fatal Error is hosted — had recently added free TLS support via Let’s Encrypt, a truly awesome project (whose crowdfunding campaign I encourage you to contribute to). We decided, “hey, this is cool and we should enable it for the Fatal Error site! Encrypt everything!”
But as it turns out, this broke iTunes. Why?
Chain of trust
The SSL/TLS ecosystem functions based a principle called certificate chains. Briefly, this works like so1:
Our site presents a security certificate claiming, “this server really is fatalerror.fm.” This certificate is digitally signed by Let’s Encrypt. Our site also presents Let’s Encrypt’s security certificate, which your web browser can verify was used to sign our site’s certificate.
But how does the browser (or in this case, iTunes’ servers) know Let’s Encrypt is a trustworthy authority? One of two cases can apply:
- The client might have included Let’s Encrypt in its list of authorities that it considers trustworthy. Browsers and operating systems ship with a list of authorities that are considered trustworthy and can vouch for the security certificates that websites present.
- Alternatively, another certificate authority has used their security certificate to digitally sign Let’s Encrypt’s certificate, effectively vouching for each certificate that Let’s Encrypt has issued. Most browsers and other software already consider this “DST Root CA X3” trustworthy, and thus by extension Let’s Encrypt.
You can see this relationship in Safari:
The problem, as it turns out is that neither of those cases apply to Java: Let’s Encrypt certificates are not trusted by default by Java clients. It appears that iTunes is using Java to fetch podcast feeds, and Apple hasn’t configured Java on the relevant iTunes servers to trust Let’s Encrypt security certificates or the IdenTrust “DST Root CA X3” certificate. This is a problem3 for Fatal Error and any other podcasters who’d like to distribute their show via Squarespace, using TLS.
If you’re interested in further reading, the Let’s Encrypt website has details on their chain of trust.
Fixing the issue
In the immediate term, my goal was to get iTunes to pick up our feed again, ideally while continuing to serve our content via TLS.
Luckily, I control another site which uses a security certificate from a different certificate authority! dzombak.com uses a security certificate issued by Comodo2, whose certificates are widely trusted, even — crucially — by Java, by default.
A simple nginx configuration tweak allows me to proxy Fatal Error content via my own server and security certificate:
server {
listen 443 ssl;
server_name www.dzombak.com;
# snip ...
location /fatalerror-proxy/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_pass https://fatalerror.fm/;
}
}
💥, suddenly https://www.dzombak.com/fatalerror-proxy/episodes?format=RSS is a usable Fatal Error RSS feed, served via TLS with a certificate iTunes can trust. I pointed iTunes Connect to that feed URL, and it appears that iTunes has already picked up the missing episodes.
Longer-term solutions
Squarespace is a popular host which specifically caters to podcasters, among other groups; and we’re not the first podcast to discover that iTunes doesn’t play well with Let’s Encrypt.
It’s surprising that Squarespace doesn’t mention this incompatibility in their support article about SSL support. In the near term, I would like them to note this as a warning to others.
Over the longer term, I hope Apple will configure iTunes’s Java stack to trust Let’s Encrypt directly, or at least to trust the IdenTrust certificate which cross-signs Let’s Encrypt. OS X and iOS already trust Let’s Encrypt sites, so this wouldn’t represent a policy shift for Apple; it seems more like an oversight in configuration of the Java stack underlying the iTunes server infrastructure.
I’d hope Squarespace, by virtue of being a popular podcast host, can encourage Apple to take this step.
Bug reports: Radar 29139592; Squarespace case #2341469.
1: This explanation and my language choices are somewhat simplified for the purposes of this blog post. Don’t hate me.
2: You can get very inexpensive Comodo certificates via Namecheap.
3: One could even say it’s a fatal error 😝