Run Diffie Tests in Your GitHub CI Pipeline

Running AI end-to-end tests in CI/CD means connecting your AI testing agent to GitHub Actions so automated browser tests execute on every pull request. With two API calls and a single workflow file, your Diffie test suite runs automatically, and results appear as a green check or red X directly on the PR.

This 5-minute setup guide covers everything from API tokens to dynamic preview URLs.

The big picture

The integration boils down to two API calls inside a shell script:

1. Start the run — tell Diffie which suite to execute and what URL to test against.
2. Wait for results — check the run status every 10 seconds until it finishes.

If all tests pass, the workflow exits successfully and your PR gets a green check. If any test fails, the workflow fails and you get a link to the Diffie dashboard where you can see screenshots and failure details.

Before you start

You will need two things from your Diffie account:

• API Token — generate one in the Diffie dashboard under Settings → API Tokens. This is your DIFFIE_TOKEN.
• Suite ID — open the test suite you want to run and copy the ID from the URL or the settings page. This is your DIFFIE_SUITE_ID.

Add your secrets to GitHub

Never hardcode tokens in your workflow file. Instead, store them as GitHub secrets:

1. Open your repository on GitHub.
2. Go to Settings → Secrets and variables → Actions.
3. Click New repository secret and add:

• Name: DIFFIE_TOKEN — paste your API token
• Name: DIFFIE_SUITE_ID — paste your suite ID

Create the workflow file

Add this file to your repository at .github/workflows/diffie.yml:

name: Diffie Tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  diffie-tests:
    name: diffie-e2e-tests
    runs-on: ubuntu-latest

    steps:
      - name: Run Diffie Test Suite
        env:
          DIFFIE_TOKEN: ${{ secrets.DIFFIE_TOKEN }}
          DIFFIE_SUITE_ID: ${{ secrets.DIFFIE_SUITE_ID }}
          PREVIEW_URL: https://your-app.example.com
        run: |
          # Start the suite run
          RESPONSE=$(curl -s -X POST \
            "https://api.diffie.ai/ci/suites/$DIFFIE_SUITE_ID/execute" \
            -H "Authorization: Bearer $DIFFIE_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"baseUrl\": \"$PREVIEW_URL\"}")

          RUN_ID=$(echo $RESPONSE | jq -r '.suiteRunId')
          RUN_URL=$(echo $RESPONSE | jq -r '.url')

          if [ "$RUN_ID" = "null" ] || [ -z "$RUN_ID" ]; then
            echo "Failed to start suite run"
            echo "$RESPONSE"
            exit 1
          fi

          echo "Suite run started: $RUN_ID"
          echo "View results: $RUN_URL"

          # Poll until the run finishes
          while true; do
            STATUS_RESPONSE=$(curl -s \
              "https://api.diffie.ai/ci/suite-runs/$RUN_ID" \
              -H "Authorization: Bearer $DIFFIE_TOKEN")

            STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')
            PASSED=$(echo $STATUS_RESPONSE | jq -r '.passed_tests')
            TOTAL=$(echo $STATUS_RESPONSE | jq -r '.total_tests')

            echo "Status: $STATUS ($PASSED/$TOTAL passed)"

            if [ "$STATUS" = "passed" ]; then
              echo "All tests passed!"
              exit 0
            elif [ "$STATUS" = "failed" ]; then
              echo "Tests failed! Details: $RUN_URL"
              exit 1
            elif [ "$STATUS" = "cancelled" ]; then
              echo "Run was cancelled"
              exit 1
            fi

            sleep 10
          done

What this script does

The workflow runs on every pull request. Here is what happens inside the shell step:

1. It sends a POST request to Diffie with your suite ID and the URL you want to test. Diffie responds with a suiteRunId and a link to the live results.
2. It enters a loop, checking the run status every 10 seconds. Diffie returns one of four statuses: running, passed, failed, or cancelled.
3. When the run finishes, the script exits with code 0 (success) or 1 (failure). GitHub uses this to mark the PR check as passed or failed.

Using dynamic preview URLs

In the example above, PREVIEW_URL is hardcoded. That works if you always test against the same environment. But if your hosting platform creates a unique preview URL for each pull request (Vercel, Netlify, Cloudflare Pages, etc.), you can grab that URL dynamically.

Here is an example using Vercel:

steps:
  - name: Wait for Vercel preview
    uses: patrickedqvist/wait-for-vercel-preview@v1.3.2
    id: vercel
    with:
      token: ${{ secrets.GITHUB_TOKEN }}
      max_timeout: 300

  - name: Run Diffie Test Suite
    env:
      DIFFIE_TOKEN: ${{ secrets.DIFFIE_TOKEN }}
      DIFFIE_SUITE_ID: ${{ secrets.DIFFIE_SUITE_ID }}
      PREVIEW_URL: ${{ steps.vercel.outputs.url }}
    run: |
      # Same script as above

The first step waits for Vercel to finish deploying, then passes the preview URL to the Diffie step. This way your tests always run against the actual code in the pull request.

API quick reference

Start a run

POST https://api.diffie.ai/ci/suites/{suiteId}/execute
Authorization: Bearer <token>

Body: { "baseUrl": "https://your-app.example.com" }

Response: { "suiteRunId": "abc-123", "url": "https://app.diffie.ai/runs/abc-123" }

Check run status

GET https://api.diffie.ai/ci/suite-runs/{runId}
Authorization: Bearer <token>

Response: { "status": "passed", "passed_tests": 12, "failed_tests": 0, "total_tests": 12 }

That is the entire API surface. Two endpoints, one token.

Not using GitHub? No problem

Since the integration is just curl and jq in a shell script, it works in any CI system — GitLab CI, CircleCI, Jenkins, Bitbucket Pipelines, or anything else that can run a shell command. Copy the same script into your platform's equivalent of a “run” step and you are good to go. See our integration guides for step-by-step setup with each platform.

What your workflow looks like after this

1. A developer opens a pull request.
2. GitHub Actions triggers the Diffie workflow.
3. Diffie runs your tests in a real browser in the cloud.
4. Results appear as a check on the PR — green for pass, red for fail.
5. If anything fails, click through to the Diffie dashboard to see screenshots, traces, and error details.

No local browser setup. No flaky test infrastructure. Just write your tests in Diffie, add this workflow, and every PR is automatically tested.