Deployment test suites

A package can be bound to one or more test suites in config/sfp_testsuites.yaml. When a bound package deploys Apex, it runs exactly the tests of those suites as RunSpecifiedTests, in every environment and on every path that deploys it. An explicit test level, skipTesting or isOptimizedDeployment: false overrides the binding, as described in Test-level precedence.

A binding is for a package whose covering tests live somewhere else, for example a service package tested by a separate test package, or a diff package whose classes are exercised by tests the diff does not touch. The default test levels do not cover this case:

• the package's own tests leave classes under 75%
• RunRelevantTests finds no tests for a newly added class
• RunLocalTests runs every test in the org to deploy one package

Binding a package to a suite

A suite in config/sfp_testsuites.yaml selects tests with classes, packages (the test classes in those packages) and domains (the test classes in every package of the release config whose releaseName matches). Add a deploy_with list to bind the suite to packages:

test_suites:
  - core-regression:
      packages: [core-tests]                   # where the tests come from
      classes: [AccountServiceTest]
      deploy_with: [core-crm, core-service]    # which packages deploy with the suite
  - sales-smoke:
      domains: [sales]
      deploy_with: [sales-diff, core-crm]

A package named in any suite's deploy_with is bound. A package that a suite only takes tests from is not bound. A bound package runs the union of every suite that names it:

deploy_with is not a test selector, so a suite still needs at least one of classes, packages or domains. sfdx-project.json does not change.

Bindings can also be edited from the Apex test suites page in codev. See Apex tests.

Which packages can be bound

Only source and diff packages can be bound. Those two types deploy through the Metadata API with a test level. An unlocked package installs a package version and a data package loads records, so neither has a test level for a binding to set.

A binding only applies when the deployed package contains Apex. A deployment without Apex keeps its usual no-test level.

The bound list replaces the package's own tests

A bound package runs its bound list instead of its default tests. The list is not added to them:

• a bound source package does not run the test classes detected inside it, unless a bound suite selects them
• a bound diff package does not use RunRelevantTests

Packages that no suite binds deploy as described in Optimized installation. The build still detects and stores a bound package's own tests, but the deployment does not use them.

Test-level precedence

For a source or diff package with Apex, the first matching row sets the test level:

• skipTesting on production. A release to a production org does not honour skipTesting, because Salesforce requires tests for an Apex deployment to production. The release continues down the table, so a bound package runs its bound list there. Validation still honours skipTesting.
• isOptimizedDeployment: false takes precedence over a binding. It applies to source and diff packages alike. The build warns when a bound package also sets it, because the binding then has no effect.
• isOptimizedDeployment: false with sfp install --use-relevant-tests. A source package deploys with RunRelevantTests, not RunLocalTests, whether it is bound or not. A diff package gets RunLocalTests.
• A binding takes precedence over sfp install --use-relevant-tests while isOptimizedDeployment is not false. The flag then only changes the default of unbound source packages.

The deployment log shows the outcome for each package. A bound package prints Bound Test Suites: <suites> (<n> test classes). When a row above it overrides the binding, the line ends with the reason, for example — not applied, isOptimizedDeployment is false.

The coverage rule

With RunSpecifiedTests, Salesforce requires every Apex class and trigger in the deployment to reach at least 75% coverage from the specified tests alone. RunLocalTests instead needs 75% across the org. Because the bound list replaces the package's own tests, a suite that does not cover part of the package fails the release.

sfp validate applies the same rule to bound packages, so the gap fails the pull request instead of the release:

For a bound package the scope is the payload actually deployed, so a payload pruned by the component checksum skip narrows it.

Each class must reach the --coveragepercent threshold, and never less than 75%. This applies to every validation mode: tests run after the deployment, tests embedded in it with --deploywithtests, and check-only deployments. The failure names the classes under the threshold and the suites that ran:

Code coverage validation failed: 1 class(es) below 75% coverage from bound test suite sales-coverage [QuoteBuilder: 40%]. Add a test that covers it to a bound suite

--skipCoverage and the package's skipCoverageValidation turn this check off in validation, as they do for unbound packages. Salesforce still applies its own rule when the package is released.

Each package is a separate Metadata API deployment. A suite bound to several packages runs once per package in a release or validation, and each package must meet coverage on its own classes from its own run. Coverage from another package's run does not count.

Component checksum skip keeps the bound level

The component checksum skip still prunes unchanged components from a bound package's payload. Unlike an unbound package, a bound package's test level is not switched to RunRelevantTests on a pruned deployment. It keeps RunSpecifiedTests with its bound list.

The list is fixed at build time

sfp build expands a bound package's suites into test classes. It stores the list, and the names of the suites it came from, in the artifact's artifact_metadata.json as boundTestSuites:

"boundTestSuites": {
    "suites": ["sales-coverage"],
    "testClasses": ["OpportunityPricingServiceTest", "QuoteBuilderTest", "SalesSmokeTest"]
}

sfp release and sfp install deploy this list and do not read the suites file. As a result:

• An artifact version runs the same tests in every environment it reaches.
• A change to a suite or a binding takes effect from the next build of the packages it binds. An artifact that is already built keeps its list. Until every bound package is rebuilt, two packages bound to the same suite can carry different versions of it.
• sfp validate builds from the pull request head, so a pull request that changes a suite or a binding is validated with that change.

Build-time checks

A suites file without any deploy_with key is not checked, so projects that do not use bindings build as before. Once the file contains deploy_with, the build fails, naming the suite and the package, when:

None of these failures falls back to the package default.

The build also warns, without failing, when:

• a packages entry in a bound suite is not in sfdx-project.json or its directory doesn't exist
• a domains entry matches no release config in config/
• a classes entry could not be parsed, so the build cannot confirm it is a test class. The class stays in the list
• a bound package sets isOptimizedDeployment: false

A successful resolution is logged per package, for example Package 'sales-diff' deploys with 3 test classes from bound test suites 'sales-coverage'.

Example: a diff package covered by another package's tests

sales-diff is a diff package holding unpackaged sales metadata. Its tests live in a separate source package, sales-tests:

{
    "packageDirectories": [
        { "path": "src/sales-tests", "package": "sales-tests", "type": "source", "versionNumber": "1.0.0.NEXT" },
        { "path": "src/sales", "package": "sales-diff", "type": "diff", "versionNumber": "1.0.0.NEXT", "default": true }
    ]
}

A pull request adds QuoteBuilder to sales-diff. Unbound, the diff deploys with RunRelevantTests. QuoteBuilder is new to the org, so no existing coverage relationship selects QuoteBuilderTest. The class reaches 0% and the release fails.

Bind sales-diff to a suite that takes the tests from sales-tests and adds the smoke test from the diff itself:

test_suites:
  - sales-coverage:
      packages: [sales-tests]
      classes: [SalesSmokeTest]
      deploy_with: [sales-diff]

Minimum versions

The suites schema rejects unknown keys. A version that predates deploy_with rejects a suites file that contains it, including for codev's own suite runs. Upgrade every consumer before adding the key: