Test selection for diff packages

How validation, builds and releases choose the Apex tests of a diff package, and how to pick between RunRelevantTests, codev suggested and declared by the author.

A diff package contains only the components that changed since its baseline. Every deployment of it that runs tests needs a list of Apex tests, and the choice of that list decides how long validation takes and what a release proves. This page explains the three ways codev can choose it, what each one runs at each stage, and how to pick one.

For the settings and the pages, see Change validation. For how the tests reach the package artifact, see sfp: Deployment test suites.

What has to be decided

A deployment to a sandbox or production org that includes Apex must run tests, and each Apex class and trigger that changed must reach the coverage threshold (75% by default). Salesforce offers these test levels for that deployment:

Test level	What runs
`RunRelevantTests`	Salesforce picks the tests that it considers related to the deployed components
`RunSpecifiedTests`	Exactly the classes that are named, and only they count for coverage
`RunLocalTests`	Every test in the org except managed packages

A diff package that no test suite is bound to deploys with RunRelevantTests. That is the default, and it works without any list. Teams change it when Salesforce's choice runs more tests than they want, or fewer than they trust.

Test selection for diff packages in Workflow Settings sets how the list for RunSpecifiedTests is chosen. It applies to the whole project, not per target branch.

The three options

	RunRelevantTests (default)	codev suggested	Declared by the author
Who chooses the tests	Salesforce	codev suggests, people accept	The pull request's author
Test level in validation	`RunRelevantTests`	`RunSpecifiedTests` with the accepted list	`RunSpecifiedTests` with the declared list
Validation waits for a decision	No	Yes, until each list is accepted or rejected	No
Pull request with no list for a changed package	Validates	Waits, or runs with its own test level if the list is rejected	Fails, and says how to declare tests
Releases after merge	`RunRelevantTests`	The accepted lists since the baseline	The declared lists since the baseline

RunRelevantTests

Nothing is configured. The package keeps its own test level, or the suites that are bound to it in config/sfp_testsuites.yaml. Use it when you do not need to control which tests run.

codev suggested

On every pull request update, codev reads the source and suggests test classes for each diff package whose Apex the pull request changes. It follows the classes that use a changed class up to Depth steps (1 to 5, default 2) to find their tests. People accept, edit or reject each list, on the validation page or with /tests comments, and validation starts after the last decision.

A higher depth finds more tests and makes the list longer. Teams that find the suggested lists too long lower the depth, or declare the tests instead.

The decision flow, the page and the comments are described in Suggested tests for diff packages.

Declared by the author

The author states the tests with the change, so there is nothing to suggest and nobody to wait for. A declaration is either or both of:

a tests: footer on any commit of the pull request:

feat(orders): recalc discount on update

tests: diff-lifecycle-pkg:AccountServiceTest,OrderTriggerTest

a file under .codev/tests/ with any name and the extension .yaml or .yml, added or changed by the pull request:
```
diff-lifecycle-pkg:
  - AccountServiceTest
  - OrderTriggerTest
```

The list that runs is the union of the footers, the files and any /tests add or /tests remove comment. Name the file after something known before the pull request exists, such as .codev/tests/JIRA-123.yaml, so that it can go in the first commit. The rules for finding the files and for editing an existing file are in Tests declared by the author.

Which package types it applies to

The setting chooses tests for diff packages only. The other package types get their tests in other ways, and the setting does not change them:

Package type	How tests are chosen	Test selection setting
Diff	`RunRelevantTests`, the bound suites, or the list from this page	Applies
Source	Its own test level, or the suites that are bound to it in `config/sfp_testsuites.yaml`. With `isOptimizedDeployment: false` it deploys with `RunLocalTests`	Does not apply. A suite can still be bound
Unlocked	Salesforce runs the package's tests when the package version is created, and the version needs 75% coverage. sfp checks that coverage during the build, and during validation when you enable it	Does not apply. A package version installs without a test level, so nothing can be bound
Org-dependent unlocked	Salesforce does not enforce coverage when the version is created. sfp checks coverage during validation by default, and the check can be switched off in `sfdx-project.json`	Does not apply
Data	Loads records and runs no Apex tests	Does not apply

Only source and diff packages can be bound to a suite, because only they deploy through the Metadata API with a test level. A binding only applies when the deployed package contains Apex. See sfp: Which packages can be bound.

A pull request can change several packages of different types. Each diff package gets its own list, and the packages of other types follow their own rules in the same validation run. A package with skipTesting skips its tests whatever its type.

What runs at each stage

Validation

A package with a list deploys with RunSpecifiedTests and the listed classes, plus the classes of any suite that is already bound to it. Each Apex class and trigger that the pull request changes must reach the coverage threshold from those tests alone. So a missing test fails the pull request instead of the release.

Validation fails, and names what is wrong, when:

(declared by the author) a diff package whose Apex the pull request changes has no tests declared;
a listed class or package does not exist in the project;
the listed tests do not cover a changed class or trigger to the threshold.

A package with skipTesting, a package whose Apex the pull request does not change, and a package with isOptimizedDeployment: false need no list.

Merged pull requests that are not released yet

A diff package deploys every change since its baseline. After another pull request is merged, and until it reaches the baseline, a pull request's validation also deploys that pull request's Apex in the same package. If validation ran only its own list, the 75% coverage rule would fail on classes the pull request did not change.

So validation also runs the lists of the merged pull requests in that package since its baseline. They show as Source Declared with merged PR #<number>, and they are not added to the pull request's own list. Once the merged pull request is in the baseline, its list drops out.

Merge, build and release

After merge, builds bind to each diff package the lists that codev recorded for the pull requests merged since the package's baseline, joined with the suites bound in config/sfp_testsuites.yaml. Releases deploy the package with RunSpecifiedTests and that list in every environment. Builds read the recorded lists, not the commit footers or the files, so a squash merge that drops the footers changes nothing.

A list applies until every environment's baseline for the package has passed its merge. After that, later builds no longer run its tests.

	codev suggested	Declared by the author
A merged pull request with no recorded list, or a commit pushed without a pull request	The build binds no list and the package keeps its own test level. The validation result warns about it	Adds nothing. The build binds the lists it finds, with no warning
A pending or rejected list in the range	The build binds no list for the package	Not applicable. A list is accepted when it is declared

An artifact override on a release takes precedence over the bound list.

Choosing an option

If	Use
Validation time is acceptable and nobody wants to manage lists	RunRelevantTests
Authors do not know which tests cover their change, and a reviewer can judge a suggestion	codev suggested
Authors know their tests and suggested lists are too long	Declared by the author

Changing the option applies to pull requests validated after the change. Pull requests that were merged before the change have no decision, so builds use the package's own test level until the baselines pass them.