DocsCI/CD Integration

CI/CD Integration Guide

Automatically upload test results from your CI/CD pipeline to Testably. Use our SDK reporters for Playwright and Cypress (Jest coming soon), or integrate manually via GitHub Actions, GitLab CI, and Python.

Prerequisites

Create an API Token

Go to Settings → API & Tokens in your Testably project and generate a new token. Copy it immediately — it will not be shown again.

Add the following environment variables to your CI/CD system:

TESTABLY_URL=https://api.testably.app

TESTABLY_TOKEN=your_api_token_here

Playwright Reporter

@testably.kr/playwright-reporter hooks into Playwright's native reporter API. No wrappers or config changes to your test files are required.

Install the package

Shell

npm install --save-dev @testably.kr/playwright-reporter

Configure in playwright.config.ts

TypeScriptplaywright.config.ts

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['html'],
    ['@testably.kr/playwright-reporter', {
      // Options (all optional — env vars are used by default)
      testCaseIdSource: 'annotation', // 'annotation' | 'tag' | 'title' | 'custom'
      failOnUploadError: false,
    }],
  ],
});

Map test cases to Testably TC IDs

TypeScripte2e/login.spec.ts

// e2e/login.spec.ts
import { test } from '@playwright/test';

// Method 1: Annotation (recommended)
test('login with valid credentials', async ({ page }) => {
  test.info().annotations.push({ type: 'testably', description: 'TC-001' });
  await page.goto('/login');
  // ...
});

// Method 2: Tag
test('checkout flow @TC-045', async ({ page }) => {
  // ...
});

// Method 3: Title pattern
test('[TC-010] dashboard loads correctly', async ({ page }) => {
  // ...
});

Set environment variables

Shell.env / CI secrets

TESTABLY_URL=https://your-project.supabase.co
TESTABLY_TOKEN=testably_your_ci_token_here
TESTABLY_RUN_ID=your-run-uuid-here

TC ID Mapping Strategies

annotation — test.info().annotations.push({ type: 'testably', description: 'TC-001' }) (default)
tag — @TC-001 in test title or tags
title — [TC-001] pattern in test title
custom — provide a mapTestCaseId(name, file) function

Troubleshooting

"No mapped test results to upload" — nothing gets uploaded

The reporter could not extract a TC ID from any test. Make sure your test titles contain a [TC-001] pattern, or that you've added the correct annotation/tag. Enable verbose: true in the reporter config to see which tests are being skipped.

401 Unauthorized

Check that TESTABLY_TOKEN is set correctly in your CI environment. Tokens start with testably_. Generate a new token in Settings → CI/CD.

403 Forbidden / Tier error

The CI/CD integration requires a Professional plan or higher. Upgrade your plan in Settings → Billing.

Run not found (404)

Verify that TESTABLY_RUN_ID is a valid run UUID for your project. The run must exist and be in an open state before uploading results.

429 Rate Limited

The SDK automatically retries with exponential backoff and respects the Retry-After header. If you're consistently hitting rate limits, consider batching fewer results per run or contacting support.

Common Reference

Status Values

Value	Description
passed	Test executed successfully
failed	Test did not meet expected result
blocked	Test could not be executed due to a dependency
retest	Test needs to be re-executed

How to get run_id

You can find the run ID in two ways:

Open a Test Run in Testably — the ID is shown in the URL: /runs/run_abc123
Use the Create Run API to create a run programmatically and receive the run_id in the response.

Test Case ID Format

Test case IDs follow the format TC-XXX where XXX is a sequential number. You can find the ID in the test case detail view or export your test cases via CSV export.

Next Steps

Webhooks

Get notified when runs complete

API Reference

Full endpoint documentation