CI/CD Pipelines

CI/CD (Continuous Integration / Continuous Delivery) automates the process of building, testing, and deploying software. A well-designed pipeline gives developers fast feedback, reduces manual errors, and enables reliable, frequent releases.

CI vs CD vs CD

Continuous Integration (CI)

Developers merge code frequently (multiple times per day). Each merge triggers an automated build and test suite. Goal: catch integration bugs early.

Continuous Delivery (CD)

Every change that passes CI is automatically prepared for release to production. Deployment is a manual, one-click action. Goal: always have a deployable artifact.

Continuous Deployment (CD)

Every change that passes all automated tests is automatically deployed to production — no human intervention. Goal: maximum release velocity.

GitHub Actions

Workflow Structure

# .github/workflows/ci.yml

name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  release:
    types: [published]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # ...stages defined here

Complete CI/CD Workflow

# .github/workflows/ci-cd.yml

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:

  # ── Stage 1: Lint & Test ───────────────────────────────────────
  test:
    name: Lint & Test
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: testuser
          POSTGRES_PASSWORD: testpass
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt -r requirements-dev.txt

      - name: Lint with flake8
        run: flake8 src/ tests/

      - name: Type check with mypy
        run: mypy src/

      - name: Run unit tests
        run: pytest tests/unit/ -v --cov=src --cov-report=xml
        env:
          DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb

      - name: Run integration tests
        run: pytest tests/integration/ -v
        env:
          DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb

      - name: Upload coverage report
        uses: codecov/codecov-action@v4
        with:
          file: coverage.xml
          token: ${{ secrets.CODECOV_TOKEN }}

  # ── Stage 2: Security Scan ────────────────────────────────────
  security:
    name: Security Scan
    runs-on: ubuntu-latest
    needs: test

    steps:
      - uses: actions/checkout@v4

      - name: Dependency vulnerability scan
        uses: snyk/actions/python@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

      - name: SAST with Semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          config: p/python p/security-audit

  # ── Stage 3: Build & Push Image ───────────────────────────────
  build:
    name: Build & Push Docker Image
    runs-on: ubuntu-latest
    needs: [test, security]
    if: github.ref == 'refs/heads/main'

    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
      image-digest: ${{ steps.build.outputs.digest }}

    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha,prefix=,suffix=,format=short
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan image for vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ghcr.io/${{ github.repository }}:latest
          format: sarif
          output: trivy-results.sarif
          severity: CRITICAL,HIGH

  # ── Stage 4: Deploy to Staging ────────────────────────────────
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: build
    environment: staging

    steps:
      - uses: actions/checkout@v4

      - name: Set up kubectl
        uses: azure/setup-kubectl@v4

      - name: Configure kubeconfig
        run: |
          echo "${{ secrets.STAGING_KUBECONFIG }}" | base64 -d > kubeconfig
          echo "KUBECONFIG=$PWD/kubeconfig" >> $GITHUB_ENV

      - name: Update image in Helm values
        run: |
          helm upgrade --install myapp ./helm/myapp \
            --namespace staging \
            --create-namespace \
            --set image.tag=${{ github.sha }} \
            --set env=staging \
            --values helm/myapp/values-staging.yaml \
            --wait --timeout 5m

      - name: Run smoke tests
        run: |
          kubectl run smoke-test \
            --image=curlimages/curl \
            --restart=Never \
            --rm -it \
            -- curl -f http://myapp.staging.svc.cluster.local/health

  # ── Stage 5: Deploy to Production ────────────────────────────
  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: deploy-staging
    environment: production    # Requires manual approval

    steps:
      - uses: actions/checkout@v4

      - name: Configure kubeconfig
        run: echo "${{ secrets.PROD_KUBECONFIG }}" | base64 -d > kubeconfig

      - name: Deploy to production
        env:
          KUBECONFIG: ./kubeconfig
        run: |
          helm upgrade --install myapp ./helm/myapp \
            --namespace production \
            --set image.tag=${{ github.sha }} \
            --set env=production \
            --values helm/myapp/values-production.yaml \
            --wait --timeout 10m

      - name: Notify Slack
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "✅ Deployed myapp ${{ github.sha }} to production"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

GitLab CI/CD

Complete .gitlab-ci.yml

# .gitlab-ci.yml

stages:
  - lint
  - test
  - build
  - scan
  - deploy-staging
  - deploy-production

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: "/certs"
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

# ── Lint ──────────────────────────────────────────────────────
lint:
  stage: lint
  image: python:3.11
  cache:
    key: pip-$CI_COMMIT_REF_SLUG
    paths: [.pip/]
  script:
    - pip install flake8 mypy --cache-dir .pip/
    - flake8 src/
    - mypy src/

# ── Tests ─────────────────────────────────────────────────────
unit-tests:
  stage: test
  image: python:3.11
  services:
    - name: postgres:15
      alias: postgres
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: testuser
    POSTGRES_PASSWORD: testpass
    DATABASE_URL: postgresql://testuser:testpass@postgres:5432/testdb
  script:
    - pip install -r requirements.txt -r requirements-dev.txt
    - pytest tests/unit/ -v --junitxml=report.xml --cov=src --cov-report=xml
  artifacts:
    when: always
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  coverage: '/TOTAL.*\s+(\d+%)$/'

# ── Build Docker image ────────────────────────────────────────
build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $IMAGE_TAG -t $CI_REGISTRY_IMAGE:latest .
    - docker push $IMAGE_TAG
    - docker push $CI_REGISTRY_IMAGE:latest
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

# ── Container scan ────────────────────────────────────────────
trivy-scan:
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image --exit-code 1 --severity CRITICAL $IMAGE_TAG
  allow_failure: true
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

# ── Deploy staging ────────────────────────────────────────────
deploy-staging:
  stage: deploy-staging
  image: alpine/helm:3.14
  environment:
    name: staging
    url: https://staging.myapp.example.com
  script:
    - echo "$STAGING_KUBECONFIG" | base64 -d > kubeconfig
    - export KUBECONFIG=$PWD/kubeconfig
    - helm upgrade --install myapp ./helm/myapp
        --namespace staging --create-namespace
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --values helm/myapp/values-staging.yaml
        --wait
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

# ── Deploy production ─────────────────────────────────────────
deploy-production:
  stage: deploy-production
  image: alpine/helm:3.14
  environment:
    name: production
    url: https://myapp.example.com
  script:
    - echo "$PROD_KUBECONFIG" | base64 -d > kubeconfig
    - export KUBECONFIG=$PWD/kubeconfig
    - helm upgrade --install myapp ./helm/myapp
        --namespace production
        --set image.tag=$CI_COMMIT_SHORT_SHA
        --values helm/myapp/values-production.yaml
        --wait --timeout 10m
  when: manual          # Requires manual trigger
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Dockerfile Best Practices

# ── Multi-stage build for Python application ──────────────────

# Stage 1: Build dependencies
FROM python:3.11-slim AS builder

WORKDIR /build

# Copy and install dependencies first (better layer caching)
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Stage 2: Production image
FROM python:3.11-slim AS production

# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser appuser

WORKDIR /app

# Copy installed packages from builder
COPY --from=builder /root/.local /home/appuser/.local

# Copy application code
COPY --chown=appuser:appuser src/ ./src/

# Switch to non-root user
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

EXPOSE 8080

CMD ["python", "-m", "uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080"]

✅ Dockerfile Best Practices:

Use multi-stage builds to keep images small
Copy dependency files first to maximize layer cache reuse
Run containers as non-root user
Use slim or distroless base images
Always set a HEALTHCHECK
Pin base image versions (avoid :latest)

Helm Chart for Kubernetes Deployment

# helm/myapp/values.yaml — default values

replicaCount: 2

image:
  repository: ghcr.io/myorg/myapp
  pullPolicy: IfNotPresent
  tag: "latest"

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  className: nginx
  host: myapp.example.com
  tls: true

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

env: production
logLevel: info

# helm/myapp/values-staging.yaml — override for staging

replicaCount: 1
ingress:
  host: myapp.staging.example.com
  tls: false
autoscaling:
  enabled: false
env: staging
logLevel: debug

ArgoCD — GitOps Deployment

# argocd-app.yaml — ArgoCD Application manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/myorg/myapp-config
    targetRevision: main
    path: helm/myapp
    helm:
      valueFiles:
        - values-production.yaml

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: true        # Delete removed resources
      selfHeal: true     # Auto-fix manual changes
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground

  # Health checks and rollback
  revisionHistoryLimit: 5

# ArgoCD CLI commands
# Install ArgoCD CLI
brew install argocd

# Login
argocd login argocd.example.com --username admin --password $ARGOCD_PASSWORD

# List applications
argocd app list

# Sync an application
argocd app sync myapp

# Check sync status
argocd app get myapp

# Manual rollback to previous version
argocd app rollback myapp

# Set image tag (triggers sync)
argocd app set myapp --helm-set image.tag=abc1234

Pipeline Best Practices

✅ Do:

Keep pipelines fast — slow pipelines get bypassed. Target <10 min for CI
Run tests in parallel to reduce total time
Use caching for dependencies (pip, npm, Maven)
Fail fast — run quick checks (lint, unit tests) before slow ones
Use environment protection rules with required reviewers for production
Store all secrets in CI/CD secret variables, never in code
Use pinned action versions (e.g., actions/checkout@v4)

⚠️ Avoid:

Deploying from developer machines — all deployments must go through the pipeline
Skipping tests to "speed up" a hotfix — it always comes back to bite you
Using :latest image tags in production manifests — use immutable SHA tags
Sharing secrets between environments — each environment has its own secrets

Next Steps

Monitoring & Logging — Observe your deployed applications
DevOps Overview — Back to DevOps fundamentals

DevOps