Troubleshooting
Diagnose worker registration, identity, RBAC, connectivity, image, Secret, and scratch-storage failures.
Loading documentation…
Powered by heyo
Start with the customer-visible symptom, then inspect the agent Pod and the affected restore Job. Do not change RBAC, disable NetworkPolicy, or expose Secret values as a first response.
Record status, last seen, configured capacity, advertised capacity, effective capacity, and Accepting jobs. Open the Logs tab when the worker is connected.
Do not decode or print the identity Secret.
For a failed or stuck check:
Compare Kubernetes events with the check's General and Logs pages in RestoreTrust.
Correct the generated resource, StorageClass, DNS path, firewall rule, or local Secret that matches the evidence. Re-run a small manual check before restoring normal schedules.
| Symptom | Likely cause | What to do |
|---|---|---|
| Worker remains Created | Agent never connected | Confirm the Deployment exists, inspect Pod logs, and allow the generated worker target |
Agent Pod is CreateContainerConfigError | Identity Secret is missing or malformed | Reapply the original identity manifest; if it is unavailable, create a replacement worker |
Agent is in CrashLoopBackOff immediately | Invalid namespace, capacity, scratch, CIDR, identity, RBAC, or egress-floor configuration | Read the first error in current or previous agent logs and restore the generated settings |
| Logs say another agent uses the identity | Two Pods or Deployments share one worker identity | Return the generated Deployment to one replica and remove the duplicate; create another worker for scale-out |
| Worker is Online but not accepting jobs | Kubernetes cache not ready, job identity invalid, graceful drain, or temporary local backpressure | Describe the Pod, inspect logs, and verify the job ServiceAccount and API access |
| Worker is Offline | Pod stopped, session connectivity failed, or worker was removed | Check replicas, Pod state, DNS, outbound TLS, and whether the worker still exists |
| Console agent logs report forbidden | Current agent-namespace RBAC is missing | Reapply the current generated RBAC that grants get on pods and pods/log in the agent namespace |
| Agent image stays unknown after an update | The agent may not read its own Pod | Reapply the current generated RBAC that grants get on pods and pods/log in the agent namespace |
| Restore Pod cannot resolve its source | Egress floor does not allow the active cluster resolver | Add NodeLocal or custom resolver IPs to the floor while retaining the CoreDNS selector |
| Source address is denied | Address falls in a built-in, cluster, deny, or outside-allow range | Correct DNS or revise customer CIDRs; do not allow Kubernetes, metadata, loopback, or management ranges |
| Restore Pod stays Pending on its PVC | StorageClass, quota, topology, or attach limit | Describe Pod and PVC; fix provisioning or reduce parallelism |
Restore Pod stays Pending, Insufficient cpu or Insufficient memory; exit 49 | The schedule's database container requests more CPU or memory than any node has free | Lower the schedule's requests, reduce concurrent restores, or add node capacity |
| Restore reports insufficient storage | Scratch PVC filled or Pod was evicted | Increase the schedule's scratch size; inspect node disk pressure |
| Restore container reports missing Secret | Selected source or encryption Secret is absent from the job namespace | Create the expected Secret in the job namespace and verify its name and keys |
ImagePullBackOff or ErrImagePull | Registry egress, registry policy, or node credentials | Allow the registry referenced by the Pod and verify the exact image can be pulled |
| Job has no Pod | Admission rejected the Pod or restoretrust-job is missing/unsafe | Inspect Job events and restore the generated ServiceAccount without cloud identity or token automount |
| Check result or message | Meaning | What to do |
|---|---|---|
Start not confirmed after the attempt limit; exit 41 | Repeated claims could not become a confirmed start; the default limit is 10 | Check agent connectivity, admission, and capacity before a new manual run |
query exceeded the gate time limit | One query exceeded its timeout_seconds, not necessarily the overall run limit | Simplify the query or raise its limit within 1–3,600 seconds; for large MongoDB counts consider an estimated count |
| Duration gate failed after a reconnect | The server interval or worker duration did not satisfy the gate | Compare both values and the timing anomaly; a reconnect alone does not fail a measurable gate |
server-side timing not observed | A gated completed stage lacks a usable timing interval or worker duration | Inspect missing/out-of-order stage reports and agent connectivity |
MySQL/MariaDB sidecar hardening failure; exit 48 | The temporary database lacks required hardening settings; this is an infrastructure failure | Bring the API, validator release, and worker installation into the supported combination; do not loosen the database settings |
| Database container was OOM-killed | Kubernetes killed the database for memory pressure or at the schedule's memory limit | Inspect Pod and node events; raise the schedule's memory limit, reduce concurrent restores, or provide more memory |
| No matching S3 file after changing the path | Paths select one folder, not a recursive key prefix or a single object | Set the containing folder and put filename matching in the filename pattern |
For a finished check whose worker capacity is still reserved, inspect the retained Job and Pod and the agent's connection. Cleanup requires an evidence handoff; do not strip finalizers or delete scratch volumes as a routine fix. See Reconnection and capacity recovery.
Run the checks as the agent ServiceAccount:
The first five should return yes; Secret access should return no. If either agent-namespace check fails, apply the current self-read RBAC.
Also confirm that the job ServiceAccount has token automount disabled and no cloud identity annotations:
Read the target from the running Deployment:
Verify that cluster DNS and firewalls permit that exact host and port. Use the CA Secret and TLS settings generated by RestoreTrust; do not replace them with an insecure target.
Contact RestoreTrust support when:
Include the evidence listed in Worker health. Never include worker identity JSON, Secret values, source credentials, or decryption material.