The wildcard certificate that stopped renewing: finding a cert-manager bug in logs nobody read

Purpose

On 2026-10-12 at 00:35Z the cluster’s wildcard certificate got stuck renewing. The old certificate kept working, so nothing broke. It was due to expire on 2026-11-11, and every HTTPS route would have gone with it. A review of the lab’s monitoring agents found the problem on 2026-10-22, about 20 days before expiry.

This post is the troubleshooting and monitoring follow-up. The root cause is already covered in one paragraph in the wildcard certificate post, which also has the TLSStore design. Here I cover where the evidence was, how the upgrade fixed it, how to force a renewal without cmctl, and the check that now makes a stuck renewal visible.

Deployment

The cluster has one Certificate, in the default namespace. A ClusterIssuer issues it from Let’s Encrypt production using DNS-01 through Cloudflare. The Cloudflare API token is stored as a SealedSecret in the cert-manager namespace.

cert-manager runs v1.18.6 for the controller, cainjector and webhook. It is managed by an ArgoCD application that is Synced and Healthy (see the ArgoCD post and the base cert-manager post). The k3s server is Kubernetes v1.30.

What happened, in order:

flowchart TD
    A["Controller sends DNS cleanup<br/>with an empty zone ID"]
    B["Cloudflare answers error 7003<br/>Could not route"]
    C["Challenge never completes"]
    D["Order stays pending"]
    E["Certificate stays Issuing"]
    F["Old certificate keeps serving"]
    G["Errors repeat in the controller log<br/>nobody reads it"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    B --> G

Configuration

A certificate that still worked

kubectl describe certificate on the Certificate showed Issuing=True since 2026-10-12, reason Renewing, with a renewalTime already in the past.

kubectl get certificaterequest,order,challenge -A showed the rest of the chain: the CertificateRequest had READY=False, an Order had been pending for 10 days, and the Challenge had no state.

The evidence in the logs

The controller log had the answer:

kubectl -n cert-manager logs deploy/cert-manager | grep dns_records

It showed a DNS-record DELETE request with an empty zone ID, DELETE "/zones//dns_records/…", and Cloudflare replying with error 7003, “Could not route”. That happened about 63 times per 24 hours on one pod.

The same search in the central log server (namespace:cert-manager AND message:dns_records) showed 248 matching lines in 24 hours and 1,256 since 2026-10-16, the oldest day the server retains. No stream, alert or report showed them. The lines were there for days, in logs nobody read.

Root cause

Cloudflare stopped returning zone_id on DNS record objects. cert-manager releases before 1.17.1 build the cleanup URL from that field (upstream issue 7540, fixed in 1.17.1). With the field gone, the zone ID in the URL was empty, and the cleanup call failed every time.

The upgrade

cert-manager upgrades one minor version at a time. On 2026-10-22 I went from 1.16.1 to 1.17.4. The CRDs are installed separately (installCRDs: false), so they go first, then the Helm chart with the exported values:

kubectl apply -f cert-manager.crds.yaml
helm upgrade cert-manager jetstack/cert-manager --version v1.17.4 -f values.yaml

Later the install moved into ArgoCD, and on 2026-10-24 it went to 1.18.6. The renewed certificate is valid until 2027-01-20.

Forcing a renewal without cmctl

cmctl renew marks the Certificate as issuing. You can do the same with a status patch: set the Issuing condition to True with reason ManuallyTriggered, using kubectl patch --subresource=status --type=json on the Certificate.

Unsticking an Order

If an Order has an authorization pending and no Challenge was created, delete the Order. cert-manager re-creates it from the same CertificateRequest, and the new one validated in seconds.

Storage

The Cloudflare token is stored as a SealedSecret in the cert-manager namespace.

Don’t print a Secret’s metadata while debugging. The last-applied-configuration annotation holds the original stringData in plain text. Storing a token as a SealedSecret avoids that annotation.

Access

Making the failure visible took a rule in the daily ops digest (the ops digest post). Its certificate check now flags a stuck renewal when a Certificate has been Issuing=True for more than 24 hours, or when renewalTime is overdue by more than 24 hours. The window is configurable as stuck_hours. Severity is crit when expiry is under 14 days away, otherwise warn. A unit test covers the rule.

flowchart LR
    A["Certificate status"]
    B["Daily digest check"]
    C["Issuing more than 24 h"]
    D["renewalTime overdue<br/>more than 24 h"]
    E["warn, or crit under 14 days"]
    A --> B
    B --> C
    B --> D
    C --> E
    D --> E

The central log server is described in the Graylog post.

Notes

• A renewal failure is quiet because the old certificate keeps working for weeks. Having logs is not the same as monitoring them.
• Two Certificates for the same names, ordered at the same time, raced on the shared apex authorization at Let’s Encrypt. One stalled until its Order was deleted. That was the reason for the later single-certificate design.
• Let’s Encrypt allows 5 duplicate certificates per week for the same names. Three copies used that budget three times faster.
• 1.18 is the newest cert-manager minor for Kubernetes 1.30. Version 1.19 needs Kubernetes 1.31 or later, 1.20 needs 1.32 and 1.21 needs 1.33. The next step is a k3s upgrade, stepping through 1.31 to 1.33 up to 1.34 this month. cert-manager then follows one minor at a time.