Skip to main content
Version: 3.0.1

STD Reconciliation Guide

Purpose​

This guide documents the process for reconciling STD (Software Test Documentation) test vectors with existing Behat tests to achieve:

  • Single source of truth: STD documents define what must be tested
  • Full traceability: Every test vector maps to a test scenario
  • Confident coverage: "All tests pass = version validated"

Overview​

STD (Source of Truth)     Existing Tests        Reconciled Output
┌─────────────────┐       ┌─────────────┐       ┌─────────────────┐
│ 388 REQs        │       │ 1217 tests  │       │ Coverage Matrix │
│ 1244 TVs        │──────▶│ Real fixtures│──────▶│ Tagged Tests    │
│ Decision Tables │       │ v3/v30 configs│      │ Gap-Fill Tests  │
└─────────────────┘       └─────────────┘       └─────────────────┘

Process Steps​

Step 1: Perform Coverage Mapping​

For each rule/domain STD file, run a coverage mapping pass with this pattern:

Task: Map [RULE-NAME] STD Test Vectors to Existing Tests

Sources:
- STD: docusaurus/docs/std/rules/std-rule-[name].md
- Catalogue: tests/catalogue/feature_catalogue.json
- Feature files: tests/exports/cucumber/*.feature

Query existing tests:
jq -r '.tests[] | select(.name | test("[search-pattern]"; "i"))' tests/catalogue/feature_catalogue.json

Output is captured by the unified traceability system (see Output section below).

Step 2: Coverage Matrix Structure​

Each coverage file should contain:

# [RULE] STD Test Vector Coverage Analysis

## Summary
| Metric | Count |
|--------|-------|
| Total Test Vectors | X |
| Covered | X |
| Partial | X |
| Gaps | X |

## Coverage Matrix
| TV ID | STD Description | Existing Test | Config | Status | Evidence |

## V2 Config Log
[Tests using non-v3 configs that were included]

## Gaps Summary
[TVs with no coverage]

## Recommendations
[Tests to tag, new tests needed]

Step 3: Status Definitions​

StatusMeaningAction
✅ CoveredTest verifies the TV condition/outcomeAdd TV tag to existing test
⚠️ PartialTest touches the area but missing assertionsEnhance existing test
❌ GapNo existing test coverageCreate new test from new-tests/features/ template

Step 4: Config Version Preferences​

Prefer (in order):

  1. v3.xlsx, v30.xlsx, v31.xlsx
  2. v30_roles_updated.xlsx, v30 pp based.xlsx
  3. Other versioned configs

Log but avoid:

  • Viracor_PROD.xlsx (v2-era)
  • 2.19.x, 2.25.x versions
  • Custom configs (aron.xlsx, alex.xlsx)

If a v2 test is the only coverage, log it in the "V2 Config Log" section for later review.

Catalogue Query Patterns​

Find tests by rule name​

jq -r '.tests[] | select(.name | test("RULE_NAME"; "i")) | "\(.jira_key): \(.name) | \(.config_file)"' tests/catalogue/feature_catalogue.json

Find v3 tests only​

jq -r '.tests[] | select(.config_file | test("v3|v30"; "i")) | "\(.jira_key): \(.name)"' tests/catalogue/feature_catalogue.json

Check support files for a test​

jq '.file_references["BT-XXXX"]' tests/catalogue/cross_references.json

List configs in use​

jq -r '[.file_references[].config_files[]] | unique | .[]' tests/catalogue/cross_references.json | sort

Parallel Processing​

For efficiency, run multiple coverage mappings in parallel (up to 10 at a time, limited by the DB pool):

# Example: 5 rules in parallel
Task 1: Map RULES-CONTROLFAIL coverage
Task 2: Map RULES-BPEC coverage
Task 3: Map RULES-NEC coverage
Task 4: Map RULES-NECINH coverage
Task 5: Map RULES-WC coverage

Each mapping task works independently with its own STD file and produces its own coverage output.

Exception Handling​

Missing Fixtures​

If a test exists but has no fixtures in support_files/:

  • Check tests/catalogue/cross_references.json under tests_without_support
  • Log as gap requiring fixture creation

Duplicate Coverage​

If multiple tests cover the same TV:

  • Prefer v3 config test
  • Prefer test with more comprehensive assertions
  • Note alternatives in "Evidence" column

Ambiguous Mapping​

If unsure whether a test covers a TV:

  • Mark as ⚠️ Partial
  • Add note in Evidence explaining uncertainty
  • Flag for human review in Recommendations

Output Location​

Coverage data is managed by the unified traceability system. After completing coverage mapping or updating STD/SDS/code changes, regenerate the traceability outputs:

python3 docusaurus/scripts/generate-unified-traceability.py --render-md

This produces:

  • tests/catalogue/unified-traceability.json — machine-readable traceability data
  • Four Markdown views under docusaurus/docs/std/ and docusaurus/docs/sds/traceability/

Use --dry-run for stats only, or --validate to check consistency without writing files.

Reconciliation Log​

Maintain a running log in the coverage file for:

  • V2 tests that were included (need review)
  • Tests with missing fixtures
  • Ambiguous mappings requiring human review
  • Any exceptions or anomalies

Normative Constraints​

The following constraints are normative and apply to all STD document content produced or updated during reconciliation. They originate from the STD Creation Plan and remain binding throughout the reconciliation process.

note

NC-2 applies to STD document content (the deliverables), not to this guide's process instructions. References to jq, Behat, or catalogue queries in the process steps above are permitted because they describe the reconciliation workflow, not the STD specification itself.

NC-1: Test Case Granularity​

A test case may cover multiple acceptance criteria only if they are verified by the same scenario and cannot fail independently. If two ACs can pass/fail independently, they require separate test cases or explicit separate verification steps within the same test case.

NC-2: No Tool References​

STD documents shall not reference specific automation frameworks, libraries, selectors, or scripts. STD specifies what to verify and how success is determined, not how automation is implemented. Tool selection is out of scope; this ensures STD remains stable when tooling changes.

  • Prohibited: "Use Selenium to click #submit-btn", "Assert with PHPUnit", "Playwright selector: data-testid=foo"
  • Permitted: "TM-UI", "TM-API", "Automated", "Manual"

NC-3: Rule Verification Format​

Rule verification uses data-driven decision tables rather than narrative test cases. Rules are pure logic with discrete input/output combinations; decision tables provide exhaustive condition coverage, auditable test vectors, and direct mapping to parameterized tests. Narrative prose test cases are prohibited for rule STDs.

Success Criteria​

Reconciliation is complete when:

  1. Every STD test vector has a mapping (Covered, Partial, or Gap)
  2. Gaps have recommendations for new tests
  3. Existing tests have TV tags recommended
  4. V2 dependencies are logged for future migration

See also: V3 Testing Dashboard for implementation status and gap tracking

Next Steps: Gap-Fill with Behat Tests​

After reconciliation identifies gaps (test vectors with no existing Behat coverage), the next step is creating tests to fill them.

For API test gaps (TM-API):

For browser test gaps (TM-UI):

For parallel execution:

  • Use the Agent Orchestration Guide for wave planning, DB pool management, and QR protocol
  • Proven pattern: 8-10 creation agents per wave, followed by QR pass

Integration with version updates: