qa-testing-ios

QA Testing (iOS)

High-signal iOS test execution and flake control for XCTest, XCUITest, Swift Testing, xcodebuild, xcresult, and simctl.

Pair this skill with software-ios-native when the work is part of a native iOS rewrite or a Codex / Claude Code implementation loop.

Core docs:

• https://developer.apple.com/documentation/xctest
• https://developer.apple.com/documentation/testing
• https://developer.apple.com/documentation/xcode/testing-your-apps-in-xcode
• https://developer.apple.com/documentation/xcode/simctl
• https://developer.apple.com/documentation/xcode-release-notes/xcode-26-release-notes

Quick Reference

Release Validation Boundary

• Do not treat simulator green or archive green as full iOS release proof when the feature depends on distribution signing, real hardware, APNs, camera, biometrics, purchases, or background execution.
• For those cases, require the release path that matches production behavior: signed archive, beta/distribution channel validation where relevant, and a real-device user-visible outcome.
• Keep project-specific rollout procedures, backend endpoints, tester-account cleanup, and internal runbooks out of this portable core. Put them in project docs or scoped references.
• Use references/ios-ci-general.md for release-CI and archive-path checks, and qa-testing-mobile/references/release-and-rollout.md for distribution-channel planning.

Defaults

• New unit and integration tests: prefer Swift Testing unless the project is already standardized on XCTest.
• UI and performance tests: keep using XCTest/XCUITest.
• PR gate: thin simulator smoke coverage with xcresult artifacts always enabled.
• Release confidence: add a real-device pass only where hardware behavior matters.
• Flake posture: prove the flake first, then fix isolation, waits, or environment drift; retries are a debugging aid, not a success criterion.
• Locale, region, timezone, permissions, and app state must be explicit in automation.
• XCTest interop with Swift Testing is off by default in Xcode 26. If CI shows unexpected test-count drops after upgrading, set SWIFT_TESTING_XCTEST_INTEROP_MODE=limited in your test plan environment.
• Snapshot baselines recorded before iOS 26 must be regenerated after adopting the iOS 26 SDK due to Liquid Glass visual changes. Review each diff before accepting.

Inputs to Gather

• Xcode entrypoint: -workspace or -project
• -scheme and optional -testPlan
• Destination strategy: simulator, real device, or both
• Required hooks: launch arguments, launch environment, test data, auth bypass, animation toggles
• Artifact needs: xcresult, coverage, screenshots, logs, diagnostics
• CI environment: local, GitHub Actions, Xcode Cloud, or self-hosted macOS
• Whether the task first needs fresh uninstall/install/launch proof before trusting UI observations
• Whether the task includes distribution-channel or hardware-specific behavior that requires a real-device validation pass

Quick Start

Workflow

• Resolve the build inputs first: workspace or project, scheme, test plan, destination, and required launch hooks.
• Make the environment repeatable: simulator boot, permissions, locale, region, and app state reset.
• If the task depends on whether the current binary is really on screen, do a fresh uninstall/install/launch smoke pass before interpreting screenshots or UI-test failures.
• If a simulator screenshot path is missing or expired, treat it as a tooling artifact, not as no evidence. Re-capture from the current simulator or use the reported visible symptom plus source inspection to choose the next focused check.
• If the task includes push, purchases, deep links, or other distribution-channel behavior, split transport proof from user-visible outcome proof and use a real-device pass when required.
• Run with artifacts enabled: -resultBundlePath, and add coverage or diagnostics only when they serve the task.
• Triage from xcresult first, then reproduce a single failing test with -only-testing.
• Treat rerun-pass as a flake that needs ownership and a root-cause fix.

Runtime Proof Boundary

• Use this skill for test execution, xcresult, destinations, and flake control after the app is buildable and installable.
• If the core problem is stale installs, simulator drift, malformed .app bundles, missing executables, or install/launch failures, route to software-ios-runtime-debugging.
• If the app cannot be installed or launched reliably, that is a runtime-debugging problem first and a test problem second.

xcodebuild Patterns

# Enumerate before an expensive run
xcodebuild test \
  -scheme MyApp \
  -testPlan Smoke \
  -destination 'platform=iOS Simulator,name=<simulator-name>,OS=latest' \
  -enumerate-tests \
  -test-enumeration-format json

# Target one test
xcodebuild test \
  -scheme MyApp \
  -destination 'platform=iOS Simulator,name=<simulator-name>,OS=latest' \
  -only-testing:MyAppUITests/LoginFlowTests/testHappyPath \
  -resultBundlePath TestResults.xcresult

# Parallelize only when the suite is isolation-safe
xcodebuild test \
  -scheme MyApp \
  -destination 'platform=iOS Simulator,name=<simulator-name>,OS=latest' \
  -parallel-testing-enabled YES \
  -maximum-parallel-testing-workers 4 \
  -resultBundlePath TestResults.xcresult

# Controlled retry for CI triage
xcodebuild test \
  -scheme MyApp \
  -destination 'platform=iOS Simulator,name=<simulator-name>,OS=latest' \
  -retry-tests-on-failure \
  -test-iterations 2 \
  -test-repetition-relaunch-enabled YES \
  -collect-test-diagnostics on-failure \
  -resultBundlePath TestResults.xcresult

# Prove a flake locally
xcodebuild test \
  -scheme MyApp \
  -destination 'platform=iOS Simulator,name=<simulator-name>,OS=latest' \
  -only-testing:MyAppUITests/LoginFlowTests/testHappyPath \
  -run-tests-until-failure \
  -test-iterations 25

Flake Control

• Prefer waitForExistence, expectations, and state-based assertions over sleeps.
• Disable or reduce animations in UI-test runs where the app allows it.
• Stub or redirect third-party boundaries; do not depend on live external services in UI tests.
• Reset permissions and app state between tests.
• Pin -testLanguage and -testRegion when locale affects assertions.
• Use test plans for matrix-style coverage across device classes, locales, and environments.
• Keep UI suites thin. Put most business logic coverage in lower layers.

Localization and Visual Regression Proof

• Missing-key crashes such as LocalizationStore.text / generated-catalog lookups are testable defects, not acceptable runtime assertions. Add or run catalog coverage before returning to UI polish.
• Verify both key presence and translated value quality. A locale file containing the English fallback is still a failed localization gate for user-visible copy.
• When new UI copy is added for a feature, run a focused key/value check for the new keys across every shipped locale, then run the broader static-key coverage suite.
• Pair locale coverage with narrow-width visual smoke for high-risk locales such as German, Russian, Japanese, and Arabic. Container width, wrapping, and overlay occlusion are part of the localization test, not a separate design nicety.
• For dense chart or diagram screens, include a targeted smoke pass that proves controls, help/info cards, and detail affordances do not cover the primary diagram and remain usable after zoom/filter changes.

Deterministic E2E Harness

For auth, onboarding, billing, referral, and other backend-coupled flows, prefer a fixture-backed launch-environment harness over live credentials. Keep user variants explicit, keep the fixture branch in the same function as the real path, and treat .accessibilityIdentifier() strings as part of the test contract.

Load references/e2e-harness-and-selectors.md for:

• the canonical launch-environment fixture pattern
• harness rules and reset-hook discipline
• selector rules for identifiers vs labels
• the atomic-commit pattern for landing a new E2E suite
• multi-account and backend-parity edge cases

Test Plan Organization

Xcode test plans (.xctestplan) control which tests run, with what configuration, and in which environment. Use them to manage matrix-style test execution:

• Smoke plan: thin critical-path tests for PR gates. Fast, reliable, minimal device matrix.
• Full plan: complete unit + integration + UI suite for nightly or release-candidate runs.
• Locale plan: same UI tests with different -testLanguage / -testRegion overrides per configuration.
• Keep test plans in the project directory alongside the scheme. Reference via -testPlan PlanName in xcodebuild.
• Each configuration within a plan can override launch arguments, environment variables, and enabled tests independently.
• Prefer separate plans over complex multi-configuration single plans — easier to run, triage, and maintain.

AI-Agent Testing for iOS

AI-native tools now generate, execute, and maintain iOS UI tests with minimal manual scripting. Use them as a complement to XCUITest, not a replacement.

Current positioning:

• Maestro + MaestroGPT + Maestro MCP: fast smoke and onboarding flow authoring
• TestSprite: self-healing AI-owned test generation and maintenance
• Drizz: visual validation for design-heavy iOS apps

When to use AI-native vs XCUITest

For fuller tool coverage, MCP integration patterns, and the decision framework, load qa-testing-mobile/references/ai-native-testing.md.

When To Use

• Choosing the right iOS test layer
• Running or debugging XCTest, XCUITest, or Swift Testing from CLI
• Stabilizing flaky UI tests
• Parsing xcresult bundles and deciding the next reproduction step
• Setting up or reviewing iOS test execution in CI
• Acting as the testing and evidence layer for a native iOS rewrite
• Proving a fresh uninstall/install/launch smoke loop before trusting UI-test observations
• Evaluating AI-native testing tools for iOS (Maestro MCP, TestSprite, Drizz)

When NOT To Use

Do / Avoid

Do

• Use waitForExistence and expectations over sleeps
• Pin -testLanguage and -testRegion when locale affects assertions
• Reset permissions and app state between tests
• Keep UI suites thin; push business logic coverage to lower layers
• Capture -resultBundlePath artifacts for every CI run
• Use generic/platform=iOS destination when simulator service is crashed or unavailable — validates Swift compilation without requiring a running simulator
• Test Codable decoders with hand-crafted JSON payloads for missing, null, and unexpected fields — especially for API response models where the backend has evolved. Use JSONDecoder().decode(T.self, from: json) with minimal payloads that omit optional fields and payloads that include unknown keys. Focus on fields the backend has changed or may omit.
• Add explicit localization coverage when the app ships multiple locales; use qa-testing-mobile/references/localization-testing.md for the layered pattern.
• Add a release-path real-device pass when the feature depends on APNs, purchases, camera, biometrics, or background execution.

Avoid

• Depending on live external services in UI tests
• Using retries as a success criterion; treat rerun-pass as a flake signal
• Inflating global timeouts instead of fixing root-cause flakiness
• Running full UI suites on every PR when smoke coverage is sufficient
• Do not retry xcodebuild with the same simulator destination when CoreSimulatorService returns "Connection refused" — switch to generic iOS destination instead
• Keeping project-specific release runbooks, backend QA endpoints, or tester-account cleanup steps in this shared skill

Backend-Coupled Edge Cases

• Exercise multi-account and backend-parity scenarios before calling a backend-heavy flow "stable".
• Typical checks: onboarding after account recreation, cache leakage across sign-out/sign-in, web-vs-API tier parity, OTP confirmation flow behavior, Apple Sign-In on a physical device, and hybrid email-template parity.
• Use references/e2e-harness-and-selectors.md for the concrete scenario checklist.

Resources

Scripts

Templates

ASCII Flow

iOS testing request
  -> Classify layer: Swift Testing, XCTest, XCUITest, snapshot, or release smoke
  -> Pick exact scheme, destination, simulator/device, OS, and runner mode
  -> Stabilize fixtures, accessibility identifiers, permissions, and time/locale
  -> Run the smallest targeted test or xcresult-producing command
  -> Parse artifacts, screenshots, logs, and xcresult failures
  -> Widen to CI, matrix, or release gate only after targeted evidence is green

Navigation

• ## Workflow, ## xcodebuild Patterns, and ## Flake Control for the baseline sequence
• ## Resources and ## Templates for deeper materials
• ## Related Skills for mobile QA and native-platform handoffs

Related Skills

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.