When developers take on the task of creating their own local environments, the process, although tailored, can introduce significant inefficiencies, particularly in terms of spent time and resources. Dissecting the setup into its fundamental steps reveals why this method can be more tedious than beneficial:

1. Data Imports: The process of importing configurations and test data is not only crucial but also time-intensive.

2. User Configurations: Assigning the correct permissions requires careful consideration, further slowing down the setup.

3. Deploy Metadata: Adjusting and deploying metadata to fit the environment involves dealing with intricacies like endpoints and user integrations.

4. Install Package Dependencies: Vital for functionality, the installation of necessary packages consumes additional time.

5. Org Shape and Scratch Org Definition: Establishing a scratch org involves creating or utilizing a definition file, a step foundational yet time-consuming.

Scratch Orgs, lauded for their ability to provide a fresh, customizable development environment, also come with a significant drawback—the massive amount of setup time before they're ready for use. Starting with only the standard features, they require a thorough, often lengthy process to incorporate project-specific requirements. While Scratch Orgs offer unmatched flexibility and customization, it's essential to weigh these benefits against the considerable time investment required for their setup.

Scratch org pools represent an indispensable asset for development teams aiming to refine their operational efficiencies and enhance their workflow optimizations. A pool of pre preared scratch orgs facilitate a more rapid and straightforward setup and management of scratch orgs, thereby bolstering team productivity. By adopting scratch org pools, development entities are empowered to access new environments on-demand, circumventing the delays inherent to manual configurations.

prepare command has inbuilt capability to orchestrate installation of external package dependencies. Package dependencies are defined in the sfdx-project.json. More information on defining package dependencies can be found in the Salesforce docs.

{
  "packageDirectories": [
    {
      "path": "util",
      "default": true,
      "package": "Expense-Manager-Util",
      "versionName": "Winter ‘22",
      "versionDescription": "Welcome to Winter 2022 Release of Expense Manager Util Package",
      "versionNumber": "4.7.0.NEXT"
    },
    {
      "path": "exp-core",
      "default": false,
      "package": "ExpenseManager",
      "versionName": "v 3.2",
      "versionDescription": "Winter 2022 Release",
      "versionNumber": "3.2.0.NEXT",
      "dependencies": [
        {
          "package": "ExpenseManager-Util",
          "versionNumber": "4.7.0.LATEST"
        },
          {
          "package": "TriggerFramework",
          "versionNumber": "1.7.0.LATEST"
        },
        {
          "package": "External Apex Library - 1.0.0.4"
        }
      ]
    }
  ],
  "sourceApiVersion": "59.0",
  "packageAliases": {
    "TriggerFramework": "0HoB00000004RFpLAM",
    "Expense Manager - Util": "0HoB00000004CFpKAM",
    "External Apex Library@1.0.0.4": "04tB0000000IB1EIAW",
    "Expense Manager": "0HoB00000004CFuKAM"
  }
}

Let's unpack the concepts utilizing the above example:

• There are two unlocked packages

  • Expense Manager - Util is an unlocked package in your DevHub, identifiable by 0H in the packageAlias

  • Expense Manager - another unlocked package which is dependent on ' Expense Manager - Util', 'TriggerFramework' and 'External Apex Library - 1.0.0.4'

• External Apex Library is an external dependency, It could be a managed package or any unlocked package released on a different Dev Hub. All external package dependencies have to be defined with a 04t ID, which can be determined from the installation URL from AppExchange or by contacting your vendor.

• sfpowerscripts parses sfdx-project.json and does the following in order

  • Skips Expense manager - Util as it doesn't have any dependencies

  • For Expense manager

    • Checks whether any of the package is part of the same repo, in this example 'Expense Manager-Util' is part of the same repository and will not be installed as a dependency

    • Installs the latest version of TriggerFramework ( with major, minor and patch versions matching 1.7.0) to the scratch org

    • Install the 'External Apex Library - 1.0.0.4' by utilizing the 04t id provided in the packageAliases

If any of the managed package has keys, it can be provided as an argument to the prepare command. Check the command's flags for more information.

Key Support for Managed Packages

The format for the 'keys' parameter is a string of key-value pairs separated by spaces - where the key is the name of the package, the value is the protection key of the package, and the key-value pair itself is delimited by a colon .

e.g. --keys "packageA:12345 packageB:pw356 packageC:pw777"

To keep the pools up to date, a nightly scheduled job can be utilised which can delete all the scratch orgs in each pool. In the case of developer pools (pools with source tracking and used by developers as opposed to CI pools), care must be taken to delete only the unassigned pools by omitting the --allscratchorgs flag.

Deleting Orphaned Scratch orgs

sfpowerscripts during prepare attempts to create multiple scratch orgs in parallel, while respecting the timeout as provided in Pool configuration. At times when the Salesforce infrastructure is stretched such as during release windows or during maintenance windows, it could be often seen some of the scratch orgs requested results in being timed out and are discarded by the rest of the pool activities. However, some of these scratch orgs are in fact created by Salesforce without respecting the timeout parameter, and counted against the active scratch org limits.

These scratch orgs titled orphaned scratch orgs in sfpowerscripts lingo can be reclaimed by issuing the pool: delete command with --orphans flag. This will delete these scratch orgs and hence allow to recover your active scratch org limits

// Example command & output demonstrating delete of orphaned scratch orgs
sfp pool:delete -o -v flxbl    

sfp's prepare command helps you to build a pool of pre-built scratch orgs which can include managed packages as well as packages in your repository. This process allows you to cut down time in re-creating a scratch org during validation process when a scratch org is used as just-in-time environment or when being used as a developer environment. As you integrate more automated processes into Salesforce, incorporating third-party managed packages into your repository's configuration metadata and code becomes necessary. This integration increases the setup time for CI scratch orgs or developer environments for various tasks, such as running data loading scripts or assigning permission sets.

Steps undertaken by prepare command

The prepare command does the following sequence of activities:

1. Calculate the number of scratch orgs to be allocated (Based on your requested number of scratch orgs and your org limits, we calculate what is the number of scratch orgs to be allocated at this point in time)

2. Fetch the artifacts from registry if "fetchArtifacts" is defined in config, otherwise build all artifacts

3. Create the scratch orgs in parallel (respecting the timeout requested in the config), and update Allocation_status_c of each these orgs to "In Progress"

4. On each scratch org, in parallel, do the following activities:

   • Install SFPOWERSCRIPTS_ARTIFACT_PACKAGE (04t1P000000ka9mQAA) for keeping track of all the packages which will be installed in the org. You can set an environment variable SFPOWERSCRIPTS_ARTIFACT_PACKAGE to override the installation with your own package id (the source code is available here)

   • Install all the dependencies of your packages, such as managed packages that are marked as dependencies in your sfdx-project.json

   • Install all the artifacts that is either built/fetched

   • If enableSourceTracking is specified in the configuration, the command will create and deploy "sourceTrackingFiles" using sfpowerscripts artifacts to the scratch org. To store local source tracking files, we re-create it when fetching a scratch org from a pool, using the @salesforce/source-tracking library. Checkout the commit from which each sfpowerscripts artifact was created, and update the local source tracking using the package directory. The files are retrieved to the local ".sf" directory, when using sfpowerscripts:pool:fetch to fetch a scratch org, and allows users to deploy their changes only, through source tracking. Refer to the decision log for more details.

5. Mark each completed scratch org as "Available", depending on the pool config `succeedOnDeploymentErrors` is true, else scratch orgs are deleted

Install sfpowerscripts Scratch Org Pooling Unlocked Package in DevHub

The Scratch Org Pooling Unlocked Package adds additional custom fields, validation rules, and workflow to the standard object "ScratchOrgInfo" in the DevHub to enable associated scratch org pool commands to work for the pipeline.

sf package install -p 04t1P000000katQQAQ -o <Your_DevHub_Username> -r -a package -s AdminsOnly -w 30

Generate SFDX auth URL for Pipeline Authentication

In order for pools command to work effectively, ensure that you have authenticated to DevHub using SFDX Auth URL instead of other authentication strategies where you are executing the pool operations

sf org display -o <orgAlias> --verbose --json > authFile.json
cat authFile.json
> {
  "status": 0,
  "result": {
    "id": "XXXXYYY",
    "accessToken": "00D8G0000009g7h!uhuRfGKbvPeubTZKztmFWgrykDuuVdxbffzjjVTqjMyRcV{wb+2JtxsevgKfGiGXRz02jY83uNBsD4CuWHwv.b21KZdFxbTi",
    "instanceUrl": "https://your.salesforce.com",
    "username": "vu.ha@dxatscale.io.dxatscale.shareddev",
    "clientId": "PlatformCLI",
    "connectedStatus": "Connected",
    "sfdxAuthUrl": "force://PlatformCLI::Cq$QLeQvDxpvUoNKgiDkoTqyVHdeoMupiZvkgHYcdVHsfMaDpqKJNbg#8ZtUpfBuIdVaUD0B21cFav5X2Pzv5X2@yoursalesforce.com",
    "alias": "SharedDev"
  }
}

Public Group and Sharing Rules Creation for Developer Access to Scratch Org Pools

For developers (who are on limited access license) to access scratch orgs created by the CI service user, for their local development, a sharing setting needs to be created on the ScratchOrgInfo object. The sharing setting should grant read/write access to the ScratchOrgInfo records owned by a public group consisting of the CI service user and a public group consisting of the developer users.

1. Create Public Groups (Setup > Users > Public Groups)

   • CI Users (Admin users/ CI users who creates scratch orgs in pool)

   • Developers (developers who are allowed to fetch scratch orgs from pool)
2. Create Sharing Rule "ScratchOrgInfo RW to Developers" (Setup > Security > Sharing Settings)

   • Grant Read/Write access to the ScratchOrgInfos records owned by the CI Users to Developers

1. Assign Users to Public Groups (Setup > Security > Sharing Settings)

   • CI Users

   • Developers

J. Permission Set Creation for Developer Access to ScratchOrgInfo Object

The developers must also have object-level and FLS permissions on the ScratchOrgInfo object. One way to achieve this is to assign a permission set that has Read, Create, Edit and Delete access on ScratchOrgInfos, as well as Read and Edit access to the custom fields used for scratch org pooling: Allocation_status__c, Password__c, Pooltag__c and SfdxAuthUrl__c

Permission Set Name: "Scratch Org Developer"

Object: Scratch Org Info

1. Object Permissions

   • Read, Create, Edit and Delete

2. Field Permissions

   • Read, Edit for Custom Fields

     • Allocation_status__c

     • Password__c

     • Pooltag__c

     • SfdxAuthUrl__c

System Permissions:

• API Enabled = True

• API Only User = False

• Create and Update Second-Generation Packages = True

K. Profile and Permission Set Assignment for Developers in Production

To onboard new developers, the following Profiles and Permission Set will need to be assigned to the new Developer User Account in Salesforce.

Profile (Choose 1 Only)

1. Minimum Access - Salesforce

   • Licence Type - Salesforce

2. Limited Access User

   • Licence Type - Salesforce Limited Access - Free

Permission Set

• Scratch Org Developer

Public Groups

• Developers - Add Users to "Developers" Group

Pools are defined by creating a pool definition file. The file can be placed in a directory within your repository.

The below configuration details a pool with tag 'DEV-POOL' allocated with 20 scratch orgs. Read on below to understand other configurations that are available

{
    "tag": "DEV-POOL",
    "maxAllocation": 20,
    "expiry": 10,
    "batchSize": 10,
    "configFilePath": "config/project-scratch-def.json",
    "relaxAllIPRanges": true,
    "installAll": true,
    "enableSourceTracking": true,
    "retryOnFailure": true,
    "succeedOnDeploymentErrors": true,
    "fetchArtifacts": {
        "npm": {
            "scope": "myproject",
    }
}

The latest JSON schema for scratch org pool configuration can be found here.​ The pool prepare command accepts a JSON configuration file that defines settings and attributes of the scratch org pool. The properties accepted by the configuration file are shown in the table below.

List all the active scratch orgs with the given pool tag, only 'available' and 'in progress' scratch orgs are displayed

The sfp pool sandbox init command is used to create and initialize a pool of Salesforce sandboxes.

Usage

Flags

• -f, --poolconfig=<path>: Path to the sandbox pool configuration file (default: 'config/sandbox-pool-config.json')

• -v, --targetdevhubusername=<devhub-alias>: Alias of the target Dev Hub org

• -r, --repo=<owner/repo>: GitHub repository in the format owner/repo

Configuration File

The configuration file should be a JSON file containing an array of pool configurations. Each configuration should include:

• pool: Name of the sandbox pool (will be converted to uppercase)

• count: Number of sandboxes to create for this pool

• sourceSB: Source sandbox name (use 'production' for creating from scratch)

• branch: Git branch associated with this pool

• defaultExpirationHours: (Optional) Default expiration time in hours (default: 24)

• extendedExpirationHours: (Optional) Extended expiration time in hours (default: 24)

• averageOrgCreationTime: (Optional) Average time in hours for sandbox creation (default: 2)

Process

1. Reads the configuration file

2. Authenticates with GitHub

3. Creates sandboxes for each pool configuration

4. Sets up GitHub repository variables for tracking sandboxes

Example

This command initializes sandbox pools as defined in my-sandbox-pools.json, using the Dev Hub 'my-devhub', and creates corresponding variables in the GitHub repository 'myorg/myrepo'.

The sfp pool sandbox fetch command is used to fetch an available sandbox from a pool and assign it to a specific issue or pull request.

Usage

Flags

• --repository, -r: The repository path (e.g., owner/repo). Default is the GITHUB_REPOSITORY environment variable.

• --pool, -p: The name of the pool to fetch the sandbox from (required).

• --branch, -b: The branch of the pool from where the environment is to be fetched (required).

• --issue, -i: Issue number to be associated with the sandbox (required).

• --devhubalias: The DevHub alias associated with the pool (default: 'devhub').

• --wait: Time in minutes to wait for an available sandbox (default: 20).

• --leasefor: Time in minutes to lease the sandbox for the current task (default: 15).

Process

1. Checks for existing sandbox assignments to the issue

2. If no assignment, acquires a lock on the pool

3. Fetches an available sandbox

4. Assigns the sandbox to the specified issue

5. Sets GitHub Actions output variable (if in GitHub Actions environment)

6. Releases the lock on the pool

Leasing

The --leasefor parameter specifies how long the sandbox is reserved for the current task before it can be reassigned within the same issue context. This is different from the overall expiration time of the sandbox in the pool.

Example

This fetches a sandbox from 'dev-pool' for the branch 'feature/new-feature', assigns it to issue 1234, and leases it for 60 minutes.

The following section details on how to operate a pool of scratch orgs

Sandbox pools in sfp are designed to provide instantly available Salesforce environments for development, testing, and review processes. By maintaining a pool of pre-created sandboxes, teams can significantly reduce wait times and streamline their development workflows.

Key Benefits

1. Immediate Availability: Eliminate waiting times for sandbox creation, allowing developers to start work instantly.

2. Reduced Overhead: Minimize the administrative burden of creating and managing individual sandboxes for each task.

3. Consistent Environment: Ensure all team members work with standardized, pre-configured sandbox environments.

4. Seamless Integration: Easily incorporate sandbox allocation into automated CI/CD pipelines and development workflows.

Sandbox Lifecycle

The following diagram illustrates the lifecycle of a sandbox within a pool:

Commands Overview

1. Sandbox Pool Initialization

Initializes sandbox pools based on configuration files.

2. Fetch a Sandbox

Fetches an available sandbox from a pool and assigns it to an issue.

3. Monitor Sandbox Pools

Monitors sandbox status, handles activations, expirations, and deletions.

Continuous Monitoring

The sfp sandbox monitor command is designed to be run as a continuous cron job. This ensures that:

1. Newly created sandboxes are activated promptly.

2. Expired sandboxes are identified and marked for deletion.

3. Marked sandboxes are deleted, freeing up resources.

4. The pool is kept in a healthy state, always ready for use.

It's recommended to set up this command to run at regular intervals (e.g., every 15-30 minutes) to maintain an up-to-date and efficient sandbox pool.

For detailed information on each command and the expiration process, please refer to the individual command documentation.

A developer can fetch a scratch org with all the dependencies installed and the latest code base has been pushed from a pool for their feature development. the developer can start developing new features without spending a few hours preparing the development environment.

The sfp sandbox monitor command is used to monitor the status of sandboxes in pools, activate new sandboxes, handle expirations, and manage deletions. This command is designed to be run as a continuous cron job.

Usage

sfp sandbox monitor -v <devhub-alias> -r <owner/repo> -f <path/to/config-file> [<path/to/config-file>...]

Flags

• -v, --targetdevhubusername: Alias of the target Dev Hub org (required)

• -r, --repo: Repository in the format owner/repo (required)

• -f, --configfile: Path to the sandbox pool configuration file(s) (required, can be specified multiple times)

Continuous Monitoring

This command should be set up as a cron job to run at regular intervals (e.g., every 15-30 minutes). This ensures:

1. Prompt activation of newly created sandboxes

2. Timely identification and marking of expired sandboxes

3. Deletion of marked sandboxes to free up resources

4. Maintenance of a healthy and efficient sandbox pool

Processes

Sandbox Activation

• Identifies sandboxes in 'InProgress' state

• Attempts to activate them

• Updates their status in GitHub variables

Expiration Checking

• Checks sandboxes against their expiration times

• Considers default expiration, extended expiration, and average creation time

• Marks expired sandboxes in GitHub variables

Sandbox Deletion

• Identifies sandboxes marked as 'Expired'

• Attempts to delete them from the Salesforce org

• Removes corresponding GitHub variables

• Logs results of the deletion process

Expiration Times

• Default Expiration: 24 hours (configurable)

• Extended Expiration: Additional 24 hours (configurable)

• Average Creation Time: 2 hours (configurable)

These times can be customized in the pool configuration file.

Example

sfp sandbox monitor -v my-devhub -r myorg/myrepo -f config/pool-config-1.json config/pool-config-2.json

This command monitors sandbox pools defined in both configuration files, using the Dev Hub 'my-devhub', and updates statuses in the 'myorg/myrepo' GitHub repository.