Where did this 404 come from?

Recently, a friend reached out to me to help him debug a 404 error that didn't make any sense in his Kubernetes cluster.

He had a rather simple setup:

• a pod for his application
• a nginx ingress used as a load balancer
• a nginx ingress that pointed to the application pod
• cert-manager for automatically generating certificates from Let's Encrypt

After talking through his setup, I put together the following diagram:

Rather than the 404 error, he expected a 308 redirect from the root domain to the www subdomain, e.g. https://example.com redirecting to https://www.example.com.

With this information, I traced the 404 through the logs to the ingress. So I cracked open the existing ingress configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: my-app-ns
  annotations:
    nginx.ingress.kubernetes.io/from-to-www-redirect: true
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          serviceName: my-app
          servicePort: 80
        path: /
      - backend:
          serviceName: cm-acme-http-solver-4k39l
          servicePort: 8089
        path: /.well-known/acme-challenge/d34db33fde4db3ef
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-7l45p
          servicePort: 8089
        path: /.well-known/acme-challenge/d3adb33fd34dbe3f
  tls:
  - hosts:
    - www.example.com
    - example.com
    secretName: my-ingress-cert

Within this configuration, I noticed 2 things:

1. The nginx ingress annotation which sets up the www redirect.
2. The "/.well-known/acme-challenge/" paths indicated that the cert-manager had been set up to use the HTTP01 Challenge for Let's Encrypt

Perhaps the host stanza for "example.com" was the problem? I removed it and applied the configuration to the cluster, watching the logs. Once the configuration was applied, cert-manager stepped in and updated the configuration to re-add the missing stanza, at which point the nginx-ingress refused to create the redirect to the www subdomain. There was the problem!

I spoke with my friend about switching cert-manager to use the Let's Encrypt DNS01 Challenge. He pointed out that cert-manager didn't have an integration for his DNS provider, which is why he hadn't used it in the first place. Until that integration existed, we needed a solution that kept the existing cert-manager configuration and didn't use the "from-to-www-redirect" annotation.

After some trial and error, and some help from Stack Overflow and the DigitalOcean community, we ended up using the following annotation:

nginx.ingress.kubernetes.io/configuration-snippet: |
  if ($host = "example.com") {
      set $test "1";
  }
  if ($request_uri !~* "^/.well-known/") {
      set $test "${test}1";
  }
  if ($test = "11") {
      return 302 https://www.example.com$request_uri;
  }

Because nginx does not allow the if directive to handle multiple conditions, we had to use three conditions. The first two test the actual properties we wanted to check, while the third tests the outcome of those two previous checks. This workaround was rather gnarly yet ensured that cert-manager could renew certificates for the root domain while redirecting all other traffic to the www subdomain.