CI/CD Pipelines: GitHub Actions and GitLab CI for Python

CI/CD pipelines automate testing, building, and deployment of applications. GitHub Actions and GitLab CI are popular platforms for implementing these workflows.

GitHub Actions Fundamentals

Workflow Structure

GitHub Actions workflows are defined in .github/workflows/ directory as YAML files.

# .github/workflows/python-tests.yml
name: Python Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    strategy:
      matrix:
        python-version: ['3.9', '3.10', '3.11']
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install pytest pytest-cov
    
    - name: Run tests
      run: pytest tests/ --cov=src --cov-report=xml
    
    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        files: ./coverage.xml

Basic Workflow Example

name: Build and Test

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    
    - name: Cache pip packages
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install flake8 black isort
    
    - name: Lint with flake8
      run: flake8 src/ --count --select=E9,F63,F7,F82 --show-source --statistics
    
    - name: Format check with black
      run: black --check src/
    
    - name: Import sort check
      run: isort --check-only src/
    
    - name: Run tests
      run: pytest tests/ -v

Matrix Strategy

jobs:
  test:
    runs-on: ${{ matrix.os }}
    
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        python-version: ['3.9', '3.10', '3.11']
        exclude:
          - os: macos-latest
            python-version: '3.9'
    
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - run: pip install -r requirements.txt
    - run: pytest tests/

Conditional Steps

steps:
- name: Run tests
  run: pytest tests/
  if: github.event_name == 'pull_request'

- name: Deploy to production
  run: ./deploy.sh
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'

- name: Notify on failure
  if: failure()
  run: echo "Tests failed!"

Advanced GitHub Actions

Secrets and Environment Variables

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Login to registry
      uses: docker/login-action@v2
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    
    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest

Artifacts and Caching

steps:
- name: Run tests and generate report
  run: pytest tests/ --html=report.html

- name: Upload test report
  uses: actions/upload-artifact@v3
  if: always()
  with:
    name: test-report
    path: report.html
    retention-days: 30

- name: Cache dependencies
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

Notifications

- name: Notify Slack on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    webhook-url: ${{ secrets.SLACK_WEBHOOK }}
    payload: |
      {
        "text": "Build failed for ${{ github.repository }}",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "Build failed: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }
          }
        ]
      }

GitLab CI

Basic GitLab CI Configuration

# .gitlab-ci.yml
image: python:3.11

stages:
  - test
  - build
  - deploy

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip
    - venv/

before_script:
  - python -m venv venv
  - source venv/bin/activate
  - pip install -r requirements.txt

test:
  stage: test
  script:
    - pip install pytest pytest-cov
    - pytest tests/ --cov=src --cov-report=term --cov-report=html
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    paths:
      - htmlcov/
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

lint:
  stage: test
  script:
    - pip install flake8 black isort
    - flake8 src/
    - black --check src/
    - isort --check-only src/

build:
  stage: build
  script:
    - pip install build
    - python -m build
  artifacts:
    paths:
      - dist/

deploy:
  stage: deploy
  script:
    - pip install twine
    - twine upload dist/*
  only:
    - tags

GitLab CI with Docker

image: docker:latest

services:
  - docker:dind

stages:
  - build
  - test
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  REGISTRY_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $REGISTRY_IMAGE .
    - docker push $REGISTRY_IMAGE

test:
  stage: test
  script:
    - docker run $REGISTRY_IMAGE pytest tests/

deploy:
  stage: deploy
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $REGISTRY_IMAGE
    - docker tag $REGISTRY_IMAGE $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main

Python-Specific CI/CD Patterns

Testing Multiple Python Versions

# GitHub Actions
strategy:
  matrix:
    python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']

steps:
- uses: actions/setup-python@v4
  with:
    python-version: ${{ matrix.python-version }}

# GitLab CI
test:
  parallel:
    matrix:
      - PYTHON_VERSION: ['3.8', '3.9', '3.10', '3.11', '3.12']
  image: python:${PYTHON_VERSION}
  script:
    - pytest tests/

Code Quality Checks

# GitHub Actions
- name: Lint with flake8
  run: flake8 src/ --count --statistics

- name: Type check with mypy
  run: mypy src/

- name: Security check with bandit
  run: bandit -r src/

- name: Format check with black
  run: black --check src/

- name: Import sorting with isort
  run: isort --check-only src/

Database Testing

services:
  postgres:
    image: postgres:15
    env:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: test_db
    options: >-
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5
    ports:
      - 5432:5432

steps:
- name: Run tests with database
  env:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_db
  run: pytest tests/

Deployment Workflows

Deploy to PyPI

name: Publish to PyPI

on:
  release:
    types: [created]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    
    - name: Install dependencies
      run: |
        pip install build twine
    
    - name: Build distribution
      run: python -m build
    
    - name: Publish to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

Deploy to Cloud

- name: Deploy to AWS
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  run: |
    pip install awscli
    aws s3 cp dist/ s3://my-bucket/ --recursive

- name: Deploy to Heroku
  env:
    HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
  run: |
    pip install heroku
    heroku login
    git push heroku main

Best Practices

Fail fast: Run quick checks first (linting, type checking)
Parallel execution: Run independent jobs in parallel
Caching: Cache dependencies to speed up builds
Secrets management: Use secrets for sensitive data
Notifications: Alert on failures
Artifacts: Save test reports and coverage
Documentation: Document CI/CD configuration

Common Pitfalls

Bad Practice:

# Don't: Hardcode secrets
- name: Deploy
  run: aws s3 cp dist/ s3://bucket/ --access-key AKIAIOSFODNN7EXAMPLE

# Don't: No caching
- name: Install dependencies
  run: pip install -r requirements.txt  # Every time!

# Don't: Single long job
jobs:
  everything:
    steps:
      - run: lint
      - run: test
      - run: build
      - run: deploy

Good Practice:

# Do: Use secrets
- name: Deploy
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  run: aws s3 cp dist/ s3://bucket/

# Do: Cache dependencies
- uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

# Do: Separate jobs
jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [lint steps]
  
  test:
    runs-on: ubuntu-latest
    steps: [test steps]
  
  deploy:
    needs: [lint, test]
    steps: [deploy steps]

Conclusion

CI/CD pipelines automate testing and deployment, improving code quality and release velocity. GitHub Actions and GitLab CI provide powerful platforms for implementing these workflows. Master workflow configuration, testing strategies, and deployment patterns to build reliable automation pipelines.