Skip to content

ADR-003: Test Management Tooling

Status: Proposed

Last Updated: 2026-02-16

Target: Adopt a test management tool to centrally record, organise, and track test scenarios across the Petfolio platform

Overview

As the Petfolio platform grows, test knowledge lives in scattered locations: Playwright spec files, Jest tests, manual ad-hoc checks, and mental notes. There is no single place to define what we test, why we test it, or whether a scenario has been verified for a given release.

This ADR proposes adopting Zephyr Scale (free tier, 1--10 Jira users) as the test management tool for Petfolio. Zephyr Scale is a Jira Marketplace app by SmartBear that adds test case management, test cycle execution, and traceability directly inside Jira Cloud.

The solution will:

  • Provide a central repository of all manual and automated test scenarios
  • Link test cases to Jira stories and bugs for requirement traceability
  • Organise test cases into folders by feature area and test level
  • Support test cycle execution for release-level sign-off
  • Accept automated test results from CI via its REST API
  • Operate within Jira Cloud with no additional infrastructure

Architecture Decisions

  1. Tool Choice: Zephyr Scale (Jira Marketplace App)

    • Free for 1--10 Jira Cloud users (covers current team size)
    • Installs directly into Jira Cloud -- no separate application to host or maintain
    • Test cases, test cycles, and results are managed within the Jira UI
    • REST API for programmatic test result uploads from CI pipelines

  2. Test Case Organisation: Folder Hierarchy by Feature Area

    • Top-level folders per application: Petfolio Business, Component Library, Docs
    • Sub-folders by feature area (e.g., Authentication, Animal Management, Theme Switching)
    • Within each feature area, sub-folders by test level: Unit, Component, E2E, Manual
    • Labels for cross-cutting concerns: accessibility, security, regression

  3. Traceability: Jira Issue Links

    • Every test case linked to the Jira story or bug it verifies
    • Coverage reports show which stories have test cases and which do not
    • Defects raised during test execution automatically link back to the failing test case

  4. CI Integration: JUnit XML Upload via REST API

    • GitHub Actions uploads Jest and Playwright JUnit XML reports to Zephyr Scale after each CI run
    • Test cases in Zephyr Scale are matched by test name convention
    • Automated test cycles are created per CI run for historical tracking

Justification

Free tier covers our needs: Petfolio has fewer than 10 Jira users. Zephyr Scale's free tier includes test cases, test cycles, folders, parameters, and the automation API -- no features are gated behind the paid tier at this scale.

Lives inside Jira: The team already works in Jira for sprint planning and issue tracking. Zephyr Scale adds test management as a tab on Jira issues and a dedicated section in the project sidebar, eliminating context-switching to an external tool.

Traceability out of the box: Linking test cases to Jira issues provides requirement coverage visibility without manual bookkeeping. Sprint reviews can show which stories are fully tested and which have gaps.

CI automation path: The JUnit XML API means Jest and Playwright results can flow into Zephyr Scale automatically. This bridges the gap between "tests pass in CI" and "we know what scenarios are covered."

Low commitment: As a Jira Marketplace app, Zephyr Scale can be installed and uninstalled without affecting Jira data. Test cases are exportable. If the tool doesn't work out, migration cost is low.

Alternatives Considered

Jira-Native Test Management Tools

  • Alternative 1: Xray

    • What: The most-installed Jira test management app. Test cases are native Jira issue types, meaning they support JQL, dashboards, automation rules, and all standard Jira workflows. Supports manual, automated, and BDD (Gherkin/Cucumber) test types.
    • Why not:
      • No free tier -- starts at ~$10/month for up to 10 users
      • Test cases as Jira issues can clutter the backlog and issue counts
      • More powerful than needed for initial adoption; complexity can slow onboarding
    • When to reconsider:
      • If the team adopts BDD with Cucumber/Gherkin and wants native .feature file integration
      • If JQL-based test reporting becomes a hard requirement
      • If Zephyr Scale's data model (test cases are not Jira issues) proves too limiting

  • Alternative 2: AIO Tests

    • What: All-in-one test management built into Jira. Supports classic and BDD test cases with AI-assisted test creation. Lightweight and affordable.
    • Why not:
      • Smaller community and fewer integrations than Zephyr Scale or Xray
      • Less mature CI/CD automation API
      • Limited ecosystem of third-party reporters and plugins
    • When to reconsider:
      • If AI-assisted test case generation becomes a priority
      • If AIO's simpler model better fits a very small team

Standalone Tools with Jira Integration

  • Alternative 3: TestRail
    • What: Purpose-built standalone test management platform with bi-directional Jira sync. Industry-leading reporting and dashboards. Available as Cloud or self-hosted.
    • Why not:
      • Not free -- starts at ~$35/user/month (Professional Cloud)
      • Requires context-switching between two applications (TestRail and Jira)
      • Separate user management, separate login, separate permissions
      • Overkill for a small team that already lives in Jira
    • When to reconsider:
      • If the team grows significantly and needs advanced reporting dashboards
      • If a dedicated QA team joins who prefer a standalone testing platform
      • If cross-project test management across multiple Jira instances is needed

No Dedicated Tool

  • Alternative 4: Jira Issues + Confluence Pages
    • What: Use standard Jira tasks/sub-tasks to represent test cases, with Confluence pages documenting test plans and scenarios. No additional tooling.
    • Why not:
      • No structured test execution workflow (pass/fail/blocked per step)
      • No test cycle concept for grouping executions per release
      • No automated test result import
      • Manual traceability maintenance -- links between stories and "test tasks" break easily
      • Scales poorly as the number of test scenarios grows
    • When to reconsider: This is the fallback if the team decides no dedicated tool is worth the overhead

Initial Adoption Plan

Phase 1: Setup and Manual Test Scenarios (Immediate)

  1. Install Zephyr Scale from the Atlassian Marketplace (free tier)
  2. Create folder structure mirroring feature areas: 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    Petfolio Business/
  Authentication/
    Manual/
  Animal Management/
    Manual/
  Theme Switching/
    Manual/
Component Library/
  Atoms/
    Manual/
  Molecules/
    Manual/
  3. Write test cases for existing features that are currently only manually verified
  4. Link test cases to existing Jira stories for traceability
  5. Run a test cycle against the current release to establish a baseline

Phase 2: CI Integration (Short-term)

  1. Configure Jest to output JUnit XML reports (--reporters=jest-junit)
  2. Configure Playwright to output JUnit XML reports (reporter: 'junit')
  3. Add a GitHub Actions step to upload JUnit XML to Zephyr Scale via REST API: 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    - name: Upload test results to Zephyr Scale
  if: always()
  run: |
    curl -X POST \
      "https://api.zephyrscale.smartbear.com/v2/automations/executions/junit" \
      -H "Authorization: Bearer ${{ secrets.ZEPHYR_SCALE_API_TOKEN }}" \
      -H "Content-Type: multipart/form-data" \
      -F "projectKey=PET" \
      -F "autoCreateTestCases=true" \
      -F "file=@test-results/junit.xml"
  4. Map automated test names to Zephyr Scale test cases using the [PET-T123] naming convention in test titles
  5. Automated test cycles appear in Zephyr Scale alongside manual test cycles

Phase 3: Full Traceability (Medium-term)

  1. Require test case links on all new Jira stories before moving to "Done"
  2. Use Zephyr Scale's coverage reports to identify untested features
  3. Create regression test cycles for each release candidate
  4. Add data-driven test cases using Zephyr Scale parameters for scenarios with multiple input combinations
  5. Create reusable test steps (via "Call to Test") for common flows like login, navigation, and animal creation

Phase 4: Maturity (Future)

  1. Define test plans per epic or milestone, grouping related test cycles
  2. Track test case versioning as features evolve
  3. Use Zephyr Scale reports in sprint retrospectives to identify testing gaps
  4. Evaluate whether the free tier still meets needs as the team grows
  5. If the team exceeds 10 Jira users, evaluate paid tier vs. migrating to Xray

Considerations

Strengths of Zephyr Scale

  • Free for current team size with no feature gating
  • Lives inside Jira -- no context-switching, no separate credentials
  • Structured test execution with pass/fail/blocked per step
  • Folder-based organisation with labels for cross-cutting concerns
  • Parameters and data-driven testing for combinatorial scenarios
  • Reusable test steps via "Call to Test" reduce duplication
  • REST API accepts JUnit XML for CI integration
  • 70+ built-in reports for test coverage, execution trends, and traceability
  • Export capability (CSV, XML) for migration if needed

Limitations and Risks

  • Test cases are not Jira issues: Zephyr Scale uses its own data model. Test cases do not appear in JQL queries, Jira dashboards, or Jira automation rules. This limits integration with workflows that rely on Jira issue operations.
  • Pricing scales by Jira seat, not tester: If the team grows past 10 Jira users, the paid tier charges per Jira user (~$5.62/user/month), not per tester. Non-testing team members (designers, product owners) inflate the cost.
  • Vendor lock-in to SmartBear: Test case data is stored in Zephyr Scale's proprietary format. Migration to another tool requires export and re-import, which may lose execution history and step-level attachments.
  • Learning curve: While Zephyr Scale lives inside Jira, its concepts (test cases, test cycles, test plans, folders) are separate from Jira's issue model. Team members need onboarding to use it effectively.
  • API token management: CI integration requires a Zephyr Scale API token stored as a GitHub Actions secret. Token rotation and access control must be managed.

Impact on Existing Workflow

Current Practice Change with Zephyr Scale
Manual test knowledge in developers' heads Documented as test cases in Zephyr Scale
Playwright/Jest results only visible in CI logs Results also uploaded to Zephyr Scale for historical tracking
No formal link between stories and test scenarios Test cases linked to Jira stories with coverage visibility
Ad-hoc release testing Structured test cycles with pass/fail tracking per release
No way to identify untested features Coverage reports highlight stories without linked test cases

References