Lua-resty-auto-ssl: OCSP stapling: Network is unreachable

Did you specify the resolver?
resolver 8.8.8.8;

yes

this looks like a infrastructure issue, like youre not listening on port 80. are you using docker?

yes

you could try my own image and see if it works: https://hub.docker.com/r/pushtospace/nginx/

I will try if I have time.
Mine:

FROM openresty/openresty:latest-xenial
RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-auto-ssl
RUN openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 \
      -subj '/CN=sni-support-required-for-valid-ssl' \
      -keyout /etc/ssl/resty-auto-ssl-fallback.key \
      -out /etc/ssl/resty-auto-ssl-fallback.crt
ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
RUN mkdir -p /etc/resty-auto-ssl/storage/file/
VOLUME ["/etc/resty-auto-ssl/storage/file/"]

@serathius: This looks like it's failing when your server is trying to make an outbound request to Let's Encrypt's OCSP server. From this same server where you have lua-resty-auto-ssl installed, are you able to manually establish a connection to the default Let's Encrypt OCSP server? You can test that with this command:

curl -v "http://ocsp.int-x3.letsencrypt.org/"

You should see a response with a "200 OK" status. If you don't see that, can you post the output? Or is there anything on your network you can think of that would be preventing this outbound request?

It's also worth noting that in this case, you should still have a valid SSL certificate, despite the error in the log file. This simply means that OCSP stapling has failed, so it's up to the client to perform any OCSP validation.

Maybe it has something to do with IPv6? I get this

2016/08/31 04:58:27 [error] 31119#0: unexpected response for ocsp.int-x3.letsencrypt.org
2016/08/31 04:58:28 [error] 31119#0: connect() to [2001:428:4402:108::419e:2f9a]:80 failed (101: Network is unreachable), context: ssl_certificate_by_lua*, client: 50.4.134.47, server: 0.0.0.0:443
2016/08/31 04:58:28 [error] 31119#0: [lua] ssl_certificate.lua:203: set_cert(): auto-ssl: failed to set ocsp stapling for staging.example.com - continuing anyway - failed to get ocsp response: OCSP responder query failed (http://ocsp.int-x3.letsencrypt.org/): network is unreachable, context: ssl_certificate_by_lua*, client: snip, server: 0.0.0.0:443

After I refresh the page once it comes up with a working https page.

Same here, any idea whats wrong?

2016/10/18 18:38:30 [error] 18084#18084: *24710 [lua] ssl_certificate.lua:203: set_cert(): auto-ssl: failed to set ocsp stapling for www.franklpharma.cz - continuing anyway - failed to get ocsp response: OCSP responder query failed (http://ocsp.int-x3.letsencrypt.org/): network is unreachable, context: ssl_certificate_by_lua*, client: 10.135.30.111, server: 0.0.0.0:443
2016/10/18 18:38:54 [error] 18084#18084: *24729 connect() to [2a02:26f0:122::215:f618]:80 failed (101: Network is unreachable), context: ssl_certificate_by_lua*, client: 10.135.30.111, server: 0.0.0.0:443

We are experiencing Nginx/Openresty freeze (nginx_status page not loading, logs are empty) every aprox. 10 hours. Is this connected? Any idea how to fix it? Thanks

P.S. I don't recognize those IPv6 addresses.

@GUI curl is working, any other idea? Certs are working fine, however my log has this error for every page load. Thanks

[root@realm0-ssl1 logs]# curl -v "http://ocsp.int-x3.letsencrypt.org/"
*   Trying 2.22.8.114...
* Connected to ocsp.int-x3.letsencrypt.org (2.22.8.114) port 80 (#0)
> GET / HTTP/1.1
> Host: ocsp.int-x3.letsencrypt.org
> User-Agent: curl/7.47.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Cache-Control: max-age=33645
< Expires: Fri, 28 Oct 2016 23:29:12 GMT
< Date: Fri, 28 Oct 2016 14:08:27 GMT
< Connection: keep-alive
< 
* Connection #0 to host ocsp.int-x3.letsencrypt.org left intact

@fibigerg: Ah, interesting. It looks like curl is resolving the domain using IPv4, but the connection inside nginx is trying to use IPv6 (and failing). What resolver setting have you configured in nginx? Are you using Google's DNS with resolver 8.8.8.8? And similarly, what DNS servers is your system using by default? On a linux system, you should be able to find these by cat /etc/resolv.conf (look for the nameserver entries).

I suspect what's happening is that you're using different DNS servers between nginx and the default system ones. Unfortunately, nginx doesn't pick up the default system DNS servers, so that's why our README uses the Google DNS entries as an example. Usually this is okay, but I think what might be happening is that Google's DNS is returning IPv6 addresses to nginx, but you might be on a network that's not fully IPv6 compatible. So after nginx receives the IPv6 addresses and attempts to connect to it, the connection fails.

If that's what's happening, then I think this should probably be solvable if you make the nginx resolver setting match whatever servers your system uses by default (which presumably won't return any IPv6 addresses).

As you noted, the SSL certificates will still work when this aspect fails, it's just that the certificates won't be returned with OCSP stapling (and nginx will keep trying to request the stapling requests, rather than caching the success).

I found a resolution to only resolve IPv4 address by configuring Nginx with the ipv6=off flag:

resolver 8.8.8.8 ipv6=off;

can help resolve this. I run this on AWS and the resolver IP that should be used is different:

resolver 172.16.0.23 ipv6=off;

(This IP can be found by running cat /etc/resolv.conf and looking for the nameserver value)

@GUI @berzniz Thank you for the solutions! We have enabled IPv6 on our Digital Ocean VPSs and the error is gone.

Thanks for all the sleuthing and debugging, everyone!

Since this issue does seem to stem from your server's network environment (whether it's IPv6 compatible) and DNS server choices (whether they return any IPv6 results), it doesn't seem like there's much we can do from a coding perspective to handle this. But I've added some comments to the README's example to hopefully clarify and explain this: https://github.com/GUI/lua-resty-auto-ssl/commit/856f52fb096c29f950dda83b3201faa5557a239b So I'm going to go ahead and close this issue, but give a shout if anyone's still running into issues or has other suggestions.

I'm seeing "OCSP response not successful (6: unauthorized)", i suspect it might be related to this issue and i want to double check before create a new one.

127.0.0.1 - - [03/Jan/2017:19:18:19 +0000] "POST /deploy-challenge HTTP/1.1" 200 5 "-" "curl/7.47.0"
10.255.0.3 - - [03/Jan/2017:19:18:20 +0000] "GET /.well-known/acme-challenge/i64vxEpJN-wUE-OvK7tKh0M3o842VcXSSEoyxtCd7Wk HTTP/1.1" 200 99 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1 - - [03/Jan/2017:19:18:22 +0000] "POST /clean-challenge HTTP/1.1" 200 5 "-" "curl/7.47.0"
127.0.0.1 - - [03/Jan/2017:19:18:25 +0000] "POST /deploy-cert HTTP/1.1" 200 30 "-" "curl/7.47.0"
2017/01/03 19:18:26 [error] 16#16: 3 [lua] ssl_certificate.lua:212: set_cert(): auto-ssl: failed to set ocsp stapling for search-api.platforg.com - continuing anyway - failed to get ocsp response: failed to validate OCSP response (http://ocsp.int-x3.letsencrypt.org/): OCSP response not successful (6: unauthorized), context: ssl_certificate_by_lua, client: 10.255.0.3, server: 0.0.0.0:443

why i get the unauthorized response ?

thanks

@faguirre1: This "unauthorized" error looks like a slightly different issue than the earlier "network unreachable" error in this thread. But in any case, if you make another request to your domain, do you get the same OCSP error in your nginx logs? In a couple of other cases where Let's Encrypt OCSP was returning "unauthorized" (#1, #2), it looks like this was a temporary server issue on Let's Encrypt's end.

If you are still seeing the same error in your logs, what output do you get when you run curl -v "http://ocsp.int-x3.letsencrypt.org/" from the server?

(And just to clarify, despite this error, your SSL certificate should be completely valid as-is, it's just that OCSP stapling isn't working, which is an optimization so clients can skip OCSP lookups themselves.)

Hi,

thanks for the answer, I got the same error in all request. but after checking today the issue is gone. seems you were right about it was an issue on let's encrypt end.

Thanks again

Adding ipv6=off solved this issue for me too. I first tried using the DNS servers as listed in resolv.conf but that had no effect.

BTW @GUI, I'm loving how easy lua-resty-auto-ssl is making the SSL process! Well done! Do you have a facility to accept donations?

Just ran into the same issue. Seems like ipv6=off solved it as well.

I'm not sure how closely related this may be, so I'll post here before creating a new issue.

I upgraded to the newest version yesterday, ahead of our certificates expiring, as it hadnt been able to re-issue (same issue as #192), and today, it still hasnt managed to issue new certificates.

Looking into the logs I find:

2019/11/24 12:50:45 [crit] 17#17: *3 connect() to [2600:1415:2000::17ce:f212]:80 failed (99: Address not available), context: ssl_certificate_by_lua*, client: 1.158.52.47, server: 0.0.0.0:443
2019/11/24 12:50:45 [error] 17#17: *3 [lua] ssl_certificate.lua:260: set_response_cert(): auto-ssl: failed to set ocsp stapling for <domain> - continuing anyway - failed to get ocsp response: OCSP responder query failed (http://ocsp.int-x3.letsencrypt.org): address not available, context: ssl_certificate_by_lua*, client: 1.158.52.47, server: 0.0.0.0:443

This is interesting as it appears to be attempting to reach an IPv6 address, despite the resolver instruction includingipv6=off, and as this is running within a docker conatiner with no ipv6 network it fails (resulting in the 99: Address not available).

I've tried all the things that I could immediately think of, but I've likely missed some obvious options as its now early in the morning here in Australia.

Does anyone have any suggestions as to what may be causing it to resolve a IPv6 address in this case? and what configuration I may need to change, either in the nginx config, or in the docker image and its associated stack to prevent this issue?

Thank you for any assistance you can provide.

OK, new morning, brain actually working, did the first thing I should have tried and removed the offending certificates in the storage directory. There was no issue issuing new certificates. so on the upside, everything is fixed for now.

I'm still curious as to what would have been causing this issue? both for my own understanding and in the event this rears its head again later, so input would still be appreciated.