Deployment test suites

A package can be bound to one or more test suites in config/sfp_testsuites.yaml. When a bound package deploys Apex, it runs exactly the tests of those suites as RunSpecifiedTests, in every environment and on every path that deploys it, except a diff-deploy sync. An explicit test level, skipTesting or isOptimizedDeployment: false overrides the binding, as described in Test-level precedence.

A binding is for a package whose covering tests live somewhere else, for example a service package tested by a separate test package, or a diff package whose classes are exercised by tests the diff does not touch. The default test levels do not cover this case:

• the package's own tests leave classes under 75%
• RunRelevantTests finds no tests for a newly added class
• RunLocalTests runs every test in the org to deploy one package

Binding a package to a suite

A suite in config/sfp_testsuites.yaml selects tests with classes, packages (the test classes in those packages) and domains (the test classes in every package of the release config whose releaseName matches). Add a deploy_with list to bind the suite to packages:

test_suites:
  - core-regression:
      packages: [core-tests]                   # where the tests come from
      classes: [AccountServiceTest]
      deploy_with: [core-crm, core-service]    # which packages deploy with the suite
  - sales-smoke:
      domains: [sales]
      deploy_with: [sales-diff, core-crm]

A package named in any suite's deploy_with is bound. A package that a suite only takes tests from is not bound. A bound package runs the union of every suite that names it:

deploy_with is not a test selector, so a suite still needs at least one of classes, packages or domains. sfdx-project.json does not change.

Bindings can also be edited from the Apex test suites page in codev. See Apex tests.

Which packages can be bound

Only source and diff packages can be bound. Those two types deploy through the Metadata API with a test level. An unlocked package installs a package version and a data package loads records, so neither has a test level for a binding to set.

A binding only applies when the deployed package contains Apex. A deployment without Apex keeps its usual no-test level.

Diff-deploy syncs

sfp sync org --diff-deploy, which a pull request's sfp sync <environment> comment runs, deploys the changed files as one synthetic package. No deploy_with entry names that package, so bindings don't apply to it, and the build doesn't check them. To run a suite's tests in a diff-deploy, name it with --testsuite: its test classes, plus any --apextests, run with RunSpecifiedTests. See Syncing to a named environment.

The bound list replaces the package's own tests

A bound package runs its bound list instead of its default tests. The list is not added to them:

• a bound source package does not run the test classes detected inside it, unless a bound suite selects them
• a bound diff package does not use RunRelevantTests

Packages that no suite binds deploy as described in Optimized installation. The build still detects and stores a bound package's own tests, but the deployment does not use them.

Test-level precedence

For a source or diff package with Apex, the first matching row sets the test level:

• skipTesting on production. A release to a production org does not honour skipTesting, because Salesforce requires tests for an Apex deployment to production. The release continues down the table, so a bound package runs its bound list there. Validation still honours skipTesting.
• isOptimizedDeployment: false takes precedence over a binding. It applies to source and diff packages alike. The build warns when a bound package also sets it, because the binding then has no effect.
• isOptimizedDeployment: false with sfp install --use-relevant-tests. A source package deploys with RunRelevantTests, not RunLocalTests, whether it is bound or not. A diff package gets RunLocalTests.
• A binding takes precedence over sfp install --use-relevant-tests while isOptimizedDeployment is not false. The flag then only changes the default of unbound source packages.

The deployment log shows the outcome for each package. A bound package prints Bound Test Suites: <suites> (<n> test classes). When a row above it overrides the binding, the line ends with the reason, for example — not applied, isOptimizedDeployment is false.

The coverage rule

With RunSpecifiedTests, Salesforce requires every Apex class and trigger in the deployment to reach at least 75% coverage from the specified tests alone. RunLocalTests instead needs 75% across the org. Because the bound list replaces the package's own tests, a suite that does not cover part of the package fails the release.

sfp validate applies the same rule to bound packages, so the gap fails the pull request instead of the release:

For a bound package the scope is the payload actually deployed, so a payload pruned by the component checksum skip narrows it.

Each class must reach the --coveragepercent threshold, and never less than 75%. This applies to every validation mode: tests run after the deployment, tests embedded in it with --deploywithtests, and check-only deployments. The failure names the classes under the threshold and the suites that ran:

Code coverage validation failed: 1 class(es) below 75% coverage from bound test suite sales-coverage [QuoteBuilder: 40%]. Add a test that covers it to a bound suite

--skipCoverage and the package's skipCoverageValidation turn this check off in validation, as they do for unbound packages. Salesforce still applies its own rule when the package is released.

Each package is a separate Metadata API deployment. A suite bound to several packages runs once per package in a release or validation, and each package must meet coverage on its own classes from its own run. Coverage from another package's run does not count.

Component checksum skip keeps the bound level

The component checksum skip still prunes unchanged components from a bound package's payload. Unlike an unbound package, a bound package's test level is not switched to RunRelevantTests on a pruned deployment. It keeps RunSpecifiedTests with its bound list.

The list is fixed at build time

sfp build expands a bound package's suites into test classes. It stores the list, and the names of the suites it came from, in the artifact's artifact_metadata.json as boundTestSuites:

"boundTestSuites": {
    "suites": ["sales-coverage"],
    "testClasses": ["OpportunityPricingServiceTest", "QuoteBuilderTest", "SalesSmokeTest"]
}

sfp release and sfp install deploy this list and do not read the suites file. As a result:

• An artifact version runs the same tests in every environment it reaches.
• A change to a suite or a binding takes effect from the next build of the packages it binds. An artifact that is already built keeps its list. Until every bound package is rebuilt, two packages bound to the same suite can carry different versions of it.
• sfp validate builds from the pull request head, so a pull request that changes a suite or a binding is validated with that change.

Build-time checks

A suites file without any deploy_with key is not checked, so projects that do not use bindings build as before. Once the file contains deploy_with, the build fails, naming the suite and the package, when:

None of these failures falls back to the package default.

The build also warns, without failing, when:

• a packages entry in a bound suite is not in sfdx-project.json or its directory doesn't exist
• a domains entry matches no release config in config/
• a classes entry could not be parsed, so the build cannot confirm it is a test class. The class stays in the list
• a bound package sets isOptimizedDeployment: false

A successful resolution is logged per package, for example Package 'sales-diff' deploys with 3 test classes from bound test suites 'sales-coverage'.

Example: a diff package covered by another package's tests

sales-diff is a diff package holding unpackaged sales metadata. Its tests live in a separate source package, sales-tests:

{
    "packageDirectories": [
        { "path": "src/sales-tests", "package": "sales-tests", "type": "source", "versionNumber": "1.0.0.NEXT" },
        { "path": "src/sales", "package": "sales-diff", "type": "diff", "versionNumber": "1.0.0.NEXT", "default": true }
    ]
}

A pull request adds QuoteBuilder to sales-diff. Unbound, the diff deploys with RunRelevantTests. QuoteBuilder is new to the org, so no existing coverage relationship selects QuoteBuilderTest. The class reaches 0% and the release fails.

Bind sales-diff to a suite that takes the tests from sales-tests and adds the smoke test from the diff itself:

test_suites:
  - sales-coverage:
      packages: [sales-tests]
      classes: [SalesSmokeTest]
      deploy_with: [sales-diff]

Suggested tests for a diff package

sfp can suggest which test classes a diff package should deploy with. It finds the test classes that exercise the Apex a pull request changes in each diff package, so a diff package does not have to rely on RunRelevantTests.

How the suggestion is used depends on where it runs:

• With codev. The project setting Test selection for diff packages set to codev suggested turns the suggestion into a test list that people accept or reject on the pull request. Validation runs the accepted list, and builds bind the accepted lists to the package, so releases run them too. See Lists decided in codev and Change validation.
• Without codev. sfp validate org --testselection codev-suggested publishes the suggestion with the validation result. It changes no test level: to run the list, add it to a suite as described in Pasting the list into the suites file.

How the list is built

For each diff package with changed Apex:

1. The changed Apex classes and triggers are the ones the pull request's own commits change, from where its branch left the base branch to the commit being checked. Without a base branch, no suggestion is made.
2. sfp reads the package's Apex dependency graph and follows the classes that use a changed class, up to --testselection-depth hops (default 2, from 1 to 5). Every test class of the package on that path is suggested. sfp validate suggest-tests, which codev runs, reads the package's full source at that commit, so it also finds tests the pull request does not touch. sfp validate org reads only the source in the package's diff. An @IsTest class without a test method, such as a test data factory, is followed but never suggested, because RunSpecifiedTests cannot name it.
3. When no test reaches a changed class within the depth, the test classes named for it are suggested: XTest, X_Test, XTests, X_Tests, TestX or Test_X. A test that reaches Apex only through DML and triggers does not reference it.
4. A changed class that only declares constants adds a single test that loads it, preferring one a bound suite already runs. It adds none when another suggested test already loads it.
5. Changed Apex classes and triggers that neither a reached test nor a named test covers are listed as Not mapped to tests. Other metadata, such as fields, flows or permission sets, is not listed.

A change to a class's -meta.xml file, such as an API version bump, counts as a change to the class.

Each suggested class carries the reason it was picked:

sfp validate suggest-tests --basebranch <branch> prints the suggestion for the checked-out commit. It needs no org and deploys nothing. codev runs it on every pull request update before validation, and publishes the lists to the pull request with --pr-number and --commit-sha. See the sfp validate reference.

sfp build of a diff package also logs the test classes its changes reach, with their reasons, for example Impacted test classes: 2 QuoteBuilderTest (changed), OpportunityPricingServiceTest (uses QuoteBuilder). The log on its own does not change the artifact.

Lists decided in codev

With codev suggested on, each diff package's list is accepted, rejected or edited on the pull request in codev. A package whose pull request changes no Apex has nothing to decide.

Validation runs the pull request's list

sfp validate reads the pull request's lists from sfp server for the commit it validates:

For an accepted list, validation applies the coverage rule to the Apex classes and triggers the pull request changes: each must reach the threshold from the listed tests alone. A class added to the list by hand that does not exist in the project fails the validation with Test class <name>, added to the list for <package>, doesn't exist in the project. skipTesting still skips the tests, and the result says the list did not run.

Builds bind the accepted lists

After the pull request merges, sfp build binds the accepted lists to the diff package, so sfp release and sfp install deploy it with RunSpecifiedTests. For each diff package with Apex, the build takes the commits on the branch's first-parent history that change the package's Apex, from the package's baseline to the commit being built. It finds the pull request of each commit, by its merge commit or from the Git provider, so squash, merge-commit and rebase merges all count.

When every one of those pull requests has an accepted or not-needed list for the package, the build binds the union of the accepted lists under the reserved suite name codev-suggested, joined with any suites bound in config/sfp_testsuites.yaml:

"boundTestSuites": {
    "suites": ["sales-coverage", "codev-suggested"],
    "testClasses": ["OpportunityPricingServiceTest", "PricingRuleTest", "QuoteBuilderTest", "SalesSmokeTest"],
    "codevSuggested": {
        "testClasses": ["PricingRuleTest", "QuoteBuilderTest"],
        "pullRequests": [41, 42]
    }
}

The build logs the result per package, for example Package 'sales-diff' deploys with 2 codev-suggested test classes from #41, #42: PricingRuleTest, QuoteBuilderTest. A class on a list that no longer exists in the project is left out with a warning.

A list applies to builds until every environment's baseline for the package has passed its merge. After that, the commits are no longer in the range and the next build binds only the newer lists.

The build does not bind a list, and the package keeps its own test level, when any commit in the range is not covered. The log names each one, for example codev-suggested tests not bound to sales-diff: #40 has no test decision:

Until the range is covered, an artifact override can set the test classes for a release. An override takes precedence over the bound list, as row 1 of Test-level precedence shows.

sfp build --testselection run-relevant builds without binding any lists, whatever the project setting. A suite in config/sfp_testsuites.yaml cannot be named codev-suggested: the name is reserved, and the build fails until the suite is renamed.

Pasting the list into the suites file

Without lists decided in codev, the pull request comment and the validation result show a Test selection section for each package. It lists the classes to add with their reasons, the suggested classes a bound suite already runs, and the changed Apex not mapped to tests. It then says what to change in config/sfp_testsuites.yaml:

For an unbound package sales-diff, the snippet goes under test_suites::

- sales-diff-deploy-tests:
    classes:
      - QuoteBuilderTest
      - OpportunityPricingServiceTest
    deploy_with:
      - sales-diff

Committed in the same pull request, the suite binds the package from the next validation. That validation deploys the package with RunSpecifiedTests and the listed classes, and applies the coverage rule. A bound suite applies to every later build of the package until someone edits it, so take classes out once they no longer need to run.

When no test of the package reaches the changed Apex within the depth, the section says so and has no snippet. When the dependency graph cannot be read, the section says the suggested tests could not be computed and gives the reason. Validation continues in both cases.

Minimum versions

The suites schema rejects unknown keys. A version that predates deploy_with rejects a suites file that contains it, including for codev's own suite runs. Upgrade every consumer before adding the key: