Skip to main content

Testing Strategy

How OpenHuman tests its product. The definitive source for "where does my test go?" Companion TEST-COVERAGE-MATRIX.md.

Test Tiers​

TierLocationWhat It TestsDriven By
Rust Unit#[cfg(test)] mod tests within source *.rs files,同级 tests.rs, or tests/ subdirectory under a domainPure domain logic, schemas, RPC handler shapes, in-memory state machinescargo test
Rust Integrationtests/*.rs at repo rootFull domain connections with real Tokio runtime, mock external services, JSON-RPC end-to-endpnpm test:rust
Vitest UnitSource-collocated *.test.ts(x) or under app/src/**/__tests__/React components, hooks, store slices, pure utilities, service layer adapterspnpm test:unit
WDIO E2Eapp/test/e2e/specs/*.spec.tsFull desktop flow: UI → Tauri → core sidecar → JSON-RPC; user-visible behaviorLinux CI: tauri-driver (port 4444). macOS local: Appium Mac2 (port 4723)
Manual Smokedocs/RELEASE-MANUAL-SMOKE.mdOS-level surfaces the driver can't assert: TCC permission prompts, Gatekeeper, code signing, DMG install, OS-native toastsHuman confirmation at release time

Test Placement Decision Tree​

Is the change behind the JSON-RPC boundary (in `src/`)?
├─ Yes - Does it cross domains or talk to external services?
│   ├─ Yes → Rust Integration (tests/*.rs)
│   └─ No → Rust Unit (next to source)
└─ No - The change is in `app/`
    ├─ Is it a pure function, hook, slice, or isolated component?
    │   └─ Yes → Vitest Unit (*.test.tsx collocated)
    └─ Is it user-visible and crosses UI ⇄ Tauri ⇄ sidecar ⇄ JSON-RPC?
        ├─ Yes → WDIO E2E (app/test/e2e/specs/*.spec.ts)
        └─ Is it OS-level (TCC, Gatekeeper, install, OS toasts)?
            └─ Yes → Manual smoke checklist

If your change touches multiple tiers, write tests at each tier you touch. Don't substitute one tier for another.

Failure Path Requirements​

Every feature leaf in the coverage matrix must have at least one failure/edge assertion in addition to the happy path. Examples:

  • File write tool: happy path = write bytes; failure path = path restriction rejection
  • OAuth flow: happy path = issue token; edge = expired refresh token recovery
  • Memory storage: happy path = store + recall; edge = forget then recall returns empty

Mock Strategy​

  • No real network in unit/integration/E2E. Use the shared mock backend
  • External services (Telegram, Slack, Gmail, Notion, Ollama, OpenAI, etc.) are stubbed at the mock backend level
  • The only acceptable exception is documented manual smoke steps at release time

Determinism Rules​

  • Don't use wall-clock waits; use waitForApp, waitForAppReady, waitForWebView helpers or explicit element-ready predicates
  • Don't use shared filesystem state; each E2E spec runs in an isolated OPENHUMAN_WORKSPACE
  • Don't depend on ordering-related specs; each spec must pass when run alone
  • Don't depend on absolute coordinates or animation timing

Pre-Merge Checks​

# Rust core
cargo fmt --check
cargo check --manifest-path Cargo.toml
cargo clippy --manifest-path Cargo.toml -- -D warnings
cargo test --manifest-path Cargo.toml

# Tauri shell
cargo check --manifest-path app/src-tauri/Cargo.toml

# Frontend
pnpm typecheck
pnpm lint
pnpm format:check
pnpm test:unit

# Rust integration
pnpm test:rust

# E2E (slow — only run when behavior changes visibly)
pnpm test:e2e:build

What the Driver Cannot Automate​

Some surfaces cannot be driven by WDIO / Appium driver because they cross OS-level trust boundaries or hardware paths:

  • macOS TCC permission prompts (Accessibility, Input Monitoring, Screen Recording, Microphone)
  • Gatekeeper signature verification
  • Code signature integrity
  • DMG install / drag-to-Applications flow
  • Auto-update download + restart
  • OS-native notification toasts on Linux

Coverage Matrix as Contract​

Every feature leaf in the coverage matrix maps to:

  1. A test path, or
  2. A 🚫 with justification and a manual smoke entry

When you add/delete/rename a feature, update the matrix row in the same PR.

Next Steps​