How do you implement automated rollback strategies in CI/CD pipelines?

In this blog post, we’ll walk through a real-world example of implementing automated rollback strategies in a CI/CD pipeline. We’ll build a simple web application, set up a CI/CD pipeline using GitHub Actions, and integrate rollback mechanisms using blue-green deployment, canary releases, and feature toggles. By the end, you’ll have a fully functional example project that demonstrates how to recover from failed deployments automatically.

Table of Contents

1. Project Overview
2. Setting Up the Example Application
3. CI/CD Pipeline Configuration
4. Implementing Rollback Strategies
   • Blue-Green Deployment
   • Canary Deployment
   • Feature Toggles
5. Testing Rollback Scenarios
6. Monitoring and Alerts

1. Project Overview

The Application

We’ll build a simple Node.js/Express API that returns a greeting message. The application will have two versions:

• Version 1: Returns Hello, World!
• Version 2: Returns Hello, Universe! (a new feature we’ll deploy and test).

Infrastructure

• GitHub Actions for CI/CD.
• Docker for containerization.
• Kubernetes (using kind for local testing) for orchestration.
• Prometheus and Grafana for monitoring.

Rollback Strategies

• Blue-Green Deployment: Switch traffic between two identical environments.
• Canary Deployment: Gradually roll out changes to a subset of users.
• Feature Toggles: Enable/disable features without redeploying code.

2. Setting Up the Example Application

Step 1: Create the Node.js API

Create a directory structure:

rollback-demo/
├── src/
│   ├── v1/
│   │   └── app.js      # Version 1 of the app
│   ├── v2/
│   │   └── app.js      # Version 2 of the app
│   └── feature-toggle.js
├── Dockerfile
└── kubernetes/
    ├── deployment-v1.yaml
    ├── deployment-v2.yaml
    └── service.yaml

Version 1 (src/v1/app.js):

const express = require('express');
const app = express();
app.get('/', (req, res) => res.send('Hello, World!'));
app.listen(3000, () => console.log('Version 1 running on port 3000'));

Version 2 (src/v2/app.js):

const express = require('express');
const app = express();
app.get('/', (req, res) => res.send('Hello, Universe!'));
app.listen(3000, () => console.log('Version 2 running on port 3000'));

Step 2: Dockerize the Application

Create a Dockerfile:

FROM node:14
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["node", "src/v1/app.js"]  # Default to Version 1

Step 3: Kubernetes Configuration

Define deployments and services in the kubernetes directory.

Deployment for Version 1 (deployment-v1.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: greeting-app
      version: v1
  template:
    metadata:
      labels:
        app: greeting-app
        version: v1
    spec:
      containers:
        - name: app
          image: your-docker-image:v1
          ports:
            - containerPort: 3000

Service (service.yaml):

apiVersion: v1
kind: Service
metadata:
  name: greeting-service
spec:
  selector:
    app: greeting-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: LoadBalancer

3. CI/CD Pipeline Configuration

We’ll use GitHub Actions to automate the build, test, and deployment process. Create a .github/workflows/pipeline.yaml file:

name: CI/CD Pipeline

on:
  push:
    branches: [ main ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Build Docker image
        run: |
          docker build -t your-docker-image:$GITHUB_SHA .

      - name: Run tests
        run: |
          # Add your test commands here (e.g., npm test)

  deploy:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to Kubernetes (Blue-Green)
        run: |
          kubectl apply -f kubernetes/deployment-v2.yaml  # Deploy new version (green)
          kubectl rollout status deployment/app-v2        # Wait for deployment to stabilize
          kubectl patch service greeting-service -p '{"spec":{"selector":{"version":"v2"}}}'  # Switch traffic to green

  rollback:
    needs: deploy
    if: failure()
    runs-on: ubuntu-latest
    steps:
      - name: Rollback to previous version
        run: |
          kubectl patch service greeting-service -p '{"spec":{"selector":{"version":"v1"}}}'  # Switch back to blue
          kubectl delete deployment app-v2  # Remove faulty deployment

4. Implementing Rollback Strategies

Strategy 1: Blue-Green Deployment

• How It Works: Deploy the new version (green) alongside the old version (blue). Switch traffic to green once validated.
• Example:
  • Deploy app-v2 and update the service selector to route traffic to version: v2.
  • If errors occur, revert the selector to version: v1.

Strategy 2: Canary Deployment

Modify the service to split traffic between versions:

# Update service.yaml to use Istio for traffic splitting
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: greeting-virtual-service
spec:
  hosts:
    - greeting-service
  http:
    - route:
        - destination:
            host: greeting-service
            subset: v1
          weight: 90  # 90% traffic to v1
        - destination:
            host: greeting-service
            subset: v2
          weight: 10  # 10% traffic to v2

Strategy 3: Feature Toggles

Add a feature toggle to enable/disable the new version dynamically:

src/feature-toggle.js:

const features = {
  newGreeting: process.env.FEATURE_GREETING === 'enabled'  // Toggle via environment variable
};

module.exports = features;

Update the API endpoint:

const features = require('./feature-toggle');

app.get('/', (req, res) => {
  if (features.newGreeting) {
    res.send('Hello, Universe!');
  } else {
    res.send('Hello, World!');
  }
});

5. Testing Rollback Scenarios

Simulating a Failed Deployment

1. Deploy Version 2:

   kubectl apply -f kubernetes/deployment-v2.yaml
   

2. Introduce a Bug: Modify src/v2/app.js to throw an error:

   app.get('/', (req, res) => {
     throw new Error('Intentional failure');
   });
   

3. Trigger the Rollback: The CI/CD pipeline detects the failure (via Prometheus alerts) and executes the rollback job, reverting to Version 1.

Testing Canary Deployment

1. Deploy the canary configuration with 10% traffic to v2.
2. Use a load testing tool (e.g., k6) to simulate traffic:

   k6 run --vus 10 --duration 30s script.js
   

3. If errors spike in v2, manually adjust the traffic weights or automate rollback using Prometheus alerts.

6. Monitoring and Alerts

Set Up Prometheus and Grafana

1. Deploy Prometheus:

   helm install prometheus prometheus-community/prometheus
   

2. Configure Alerts: Create a prometheus-alerts.yaml rule to trigger alerts when error rates exceed 5%:

   groups:
   - name: greeting-app-alerts
     rules:
     - alert: HighErrorRate
       expr: rate(http_requests_total{status="500"}[5m]) > 0.05
       for: 2m
       labels:
         severity: critical
       annotations:
         summary: "High error rate detected in greeting app"
   

Integrate Alerts with GitHub Actions

Update the pipeline to trigger rollbacks on alerts:

- name: Check Prometheus Alerts
  uses: example/prometheus-action@v1
  with:
    alert-name: HighErrorRate
  if: alerts.detected == 'true'
  run: |
    echo "Rolling back due to high error rate"
    kubectl rollout undo deployment/app-v2

Key Takeaways

• Automate Everything: Use CI/CD tools to automate rollbacks based on predefined criteria.
• Monitor Relentlessly: Implement real-time monitoring to detect issues early.
• Test Rollbacks: Regularly simulate failures to ensure your rollback mechanisms work.

Best Practices

1. Version Control Everything: Track infrastructure, configurations, and pipelines in Git.
2. Use Immutable Infrastructure: Avoid manual changes to running instances.
3. Leverage Feature Toggles: Decouple deployment from feature release.
4. Document Processes: Maintain clear documentation for rollback procedures.

GitHub Repository

For the complete code and configurations, check out the example repository:
github.com/your-username/rollback-demo