When developers take on the task of creating their own local environments, the process, although tailored, can introduce significant inefficiencies, particularly in terms of spent time and resources. Dissecting the setup into its fundamental steps reveals why this method can be more tedious than beneficial:

1. Data Imports: The process of importing configurations and test data is not only crucial but also time-intensive.

2. User Configurations: Assigning the correct permissions requires careful consideration, further slowing down the setup.

3. Deploy Metadata: Adjusting and deploying metadata to fit the environment involves dealing with intricacies like endpoints and user integrations.

4. Install Package Dependencies: Vital for functionality, the installation of necessary packages consumes additional time.

5. Org Shape and Scratch Org Definition: Establishing a scratch org involves creating or utilizing a definition file, a step foundational yet time-consuming.

Scratch Orgs, lauded for their ability to provide a fresh, customizable development environment, also come with a significant drawback—the massive amount of setup time before they're ready for use. Starting with only the standard features, they require a thorough, often lengthy process to incorporate project-specific requirements. While Scratch Orgs offer unmatched flexibility and customization, it's essential to weigh these benefits against the considerable time investment required for their setup.

Scratch org pools represent an indispensable asset for development teams aiming to refine their operational efficiencies and enhance their workflow optimizations. A pool of pre preared scratch orgs facilitate a more rapid and straightforward setup and management of scratch orgs, thereby bolstering team productivity. By adopting scratch org pools, development entities are empowered to access new environments on-demand, circumventing the delays inherent to manual configurations.

To keep the pools up to date, a nightly scheduled job can be utilised which can delete all the scratch orgs in each pool. In the case of developer pools (pools with source tracking and used by developers as opposed to CI pools), care must be taken to delete only the unassigned pools by omitting the --allscratchorgs flag.

Deleting Orphaned Scratch orgs

sfpowerscripts during prepare attempts to create multiple scratch orgs in parallel, while respecting the timeout as provided in Pool configuration. At times when the Salesforce infrastructure is stretched such as during release windows or during maintenance windows, it could be often seen some of the scratch orgs requested results in being timed out and are discarded by the rest of the pool activities. However, some of these scratch orgs are in fact created by Salesforce without respecting the timeout parameter, and counted against the active scratch org limits.

These scratch orgs titled orphaned scratch orgs in sfpowerscripts lingo can be reclaimed by issuing the pool: delete command with --orphans flag. This will delete these scratch orgs and hence allow to recover your active scratch org limits

// Example command & output demonstrating delete of orphaned scratch orgs
sfp pool:delete -o -v flxbl    

Install sfpowerscripts Scratch Org Pooling Unlocked Package in DevHub

The Scratch Org Pooling Unlocked Package adds additional custom fields, validation rules, and workflow to the standard object "ScratchOrgInfo" in the DevHub to enable associated scratch org pool commands to work for the pipeline.

sf package install -p 04t1P000000katQQAQ -o <Your_DevHub_Username> -r -a package -s AdminsOnly -w 30

Generate SFDX auth URL for Pipeline Authentication

In order for pools command to work effectively, ensure that you have authenticated to DevHub using SFDX Auth URL instead of other authentication strategies where you are executing the pool operations

sf org display -o <orgAlias> --verbose --json > authFile.json
cat authFile.json
> {
  "status": 0,
  "result": {
    "id": "XXXXYYY",
    "accessToken": "00D8G0000009g7h!uhuRfGKbvPeubTZKztmFWgrykDuuVdxbffzjjVTqjMyRcV{wb+2JtxsevgKfGiGXRz02jY83uNBsD4CuWHwv.b21KZdFxbTi",
    "instanceUrl": "https://your.salesforce.com",
    "username": "vu.ha@dxatscale.io.dxatscale.shareddev",
    "clientId": "PlatformCLI",
    "connectedStatus": "Connected",
    "sfdxAuthUrl": "force://PlatformCLI::Cq$QLeQvDxpvUoNKgiDkoTqyVHdeoMupiZvkgHYcdVHsfMaDpqKJNbg#8ZtUpfBuIdVaUD0B21cFav5X2Pzv5X2@yoursalesforce.com",
    "alias": "SharedDev"
  }
}

Public Group and Sharing Rules Creation for Developer Access to Scratch Org Pools

For developers (who are on limited access license) to access scratch orgs created by the CI service user, for their local development, a sharing setting needs to be created on the ScratchOrgInfo object. The sharing setting should grant read/write access to the ScratchOrgInfo records owned by a public group consisting of the CI service user and a public group consisting of the developer users.

1. Create Public Groups (Setup > Users > Public Groups)

   • CI Users (Admin users/ CI users who creates scratch orgs in pool)

   • Developers (developers who are allowed to fetch scratch orgs from pool)
2. Create Sharing Rule "ScratchOrgInfo RW to Developers" (Setup > Security > Sharing Settings)

   • Grant Read/Write access to the ScratchOrgInfos records owned by the CI Users to Developers

1. Assign Users to Public Groups (Setup > Security > Sharing Settings)

   • CI Users

   • Developers

J. Permission Set Creation for Developer Access to ScratchOrgInfo Object

The developers must also have object-level and FLS permissions on the ScratchOrgInfo object. One way to achieve this is to assign a permission set that has Read, Create, Edit and Delete access on ScratchOrgInfos, as well as Read and Edit access to the custom fields used for scratch org pooling: Allocation_status__c, Password__c, Pooltag__c and SfdxAuthUrl__c

Permission Set Name: "Scratch Org Developer"

Object: Scratch Org Info

1. Object Permissions

   • Read, Create, Edit and Delete

2. Field Permissions

   • Read, Edit for Custom Fields

     • Allocation_status__c

     • Password__c

     • Pooltag__c

     • SfdxAuthUrl__c

System Permissions:

• API Enabled = True

• API Only User = False

• Create and Update Second-Generation Packages = True

K. Profile and Permission Set Assignment for Developers in Production

To onboard new developers, the following Profiles and Permission Set will need to be assigned to the new Developer User Account in Salesforce.

Profile (Choose 1 Only)

1. Minimum Access - Salesforce

   • Licence Type - Salesforce

2. Limited Access User

   • Licence Type - Salesforce Limited Access - Free

Permission Set

• Scratch Org Developer

Public Groups

• Developers - Add Users to "Developers" Group

Pools are defined by creating a pool definition file. The file can be placed in a directory within your repository.

The below configuration details a pool with tag 'DEV-POOL' allocated with 20 scratch orgs. Read on below to understand other configurations that are available

{
    "tag": "DEV-POOL",
    "maxAllocation": 20,
    "expiry": 10,
    "batchSize": 10,
    "configFilePath": "config/project-scratch-def.json",
    "relaxAllIPRanges": true,
    "installAll": true,
    "enableSourceTracking": true,
    "retryOnFailure": true,
    "succeedOnDeploymentErrors": true,
    "fetchArtifacts": {
        "npm": {
            "scope": "myproject",
    }
}

The latest JSON schema for scratch org pool configuration can be found here.​ The pool prepare command accepts a JSON configuration file that defines settings and attributes of the scratch org pool. The properties accepted by the configuration file are shown in the table below.

sfp's prepare command helps you to build a pool of pre-built scratch orgs which can include managed packages as well as packages in your repository. This process allows you to cut down time in re-creating a scratch org during validation process when a scratch org is used as just-in-time environment or when being used as a developer environment. As you integrate more automated processes into Salesforce, incorporating third-party managed packages into your repository's configuration metadata and code becomes necessary. This integration increases the setup time for CI scratch orgs or developer environments for various tasks, such as running data loading scripts or assigning permission sets.

Steps undertaken by prepare command

The prepare command does the following sequence of activities:

1. Calculate the number of scratch orgs to be allocated (Based on your requested number of scratch orgs and your org limits, we calculate what is the number of scratch orgs to be allocated at this point in time)

2. Fetch the artifacts from registry if "fetchArtifacts" is defined in config, otherwise build all artifacts

3. Create the scratch orgs in parallel (respecting the timeout requested in the config), and update Allocation_status_c of each these orgs to "In Progress"

4. On each scratch org, in parallel, do the following activities:

   • Install SFPOWERSCRIPTS_ARTIFACT_PACKAGE (04t1P000000ka9mQAA) for keeping track of all the packages which will be installed in the org. You can set an environment variable SFPOWERSCRIPTS_ARTIFACT_PACKAGE to override the installation with your own package id (the source code is available )

   • Install all the dependencies of your packages, such as managed packages that are marked as dependencies in your sfdx-project.json

   • Install all the artifacts that is either built/fetched

   • If enableSourceTracking is specified in the configuration, the command will create and deploy "sourceTrackingFiles" using sfpowerscripts artifacts to the scratch org. To store local source tracking files, we re-create it when fetching a scratch org from a pool, using the library. Checkout the commit from which each sfpowerscripts artifact was created, and update the local source tracking using the package directory. The files are retrieved to the local ".sf" directory, when using sfpowerscripts:pool:fetch to fetch a scratch org, and allows users to deploy their changes only, through source tracking. Refer to the for more details.

5. Mark each completed scratch org as "Available", depending on the pool config `succeedOnDeploymentErrors` is true, else scratch orgs are deleted

prepare command has inbuilt capability to orchestrate installation of external package dependencies. Package dependencies are defined in the sfdx-project.json. More information on defining package dependencies can be found in the Salesforce .

Let's unpack the concepts utilizing the above example:

• There are two unlocked packages

  • Expense Manager - Util is an unlocked package in your DevHub, identifiable by 0H in the packageAlias

  • Expense Manager - another unlocked package which is dependent on ' Expense Manager - Util', 'TriggerFramework' and 'External Apex Library - 1.0.0.4'

• External Apex Library is an external dependency, It could be a managed package or any unlocked package released on a different Dev Hub. All external package dependencies have to be defined with a 04t ID, which can be determined from the installation URL from AppExchange or by contacting your vendor.

• sfpowerscripts parses sfdx-project.json and does the following in order

  • Skips Expense manager - Util as it doesn't have any dependencies

  • For Expense manager

    • Checks whether any of the package is part of the same repo, in this example 'Expense Manager-Util' is part of the same repository and will not be installed as a dependency

    • Installs the latest version of TriggerFramework ( with major, minor and patch versions matching 1.7.0) to the scratch org

    • Install the 'External Apex Library - 1.0.0.4' by utilizing the 04t id provided in the packageAliases

If any of the managed package has keys, it can be provided as an argument to the prepare command. Check the command's flags for more information.

Key Support for Managed Packages

The format for the 'keys' parameter is a string of key-value pairs separated by spaces - where the key is the name of the package, the value is the protection key of the package, and the key-value pair itself is delimited by a colon .

e.g. --keys "packageA:12345 packageB:pw356 packageC:pw777"

List all the active scratch orgs with the given pool tag, only 'available' and 'in progress' scratch orgs are displayed

The following section details on how to operate a pool of scratch orgs

A developer can fetch a scratch org with all the dependencies installed and the latest code base has been pushed from a pool for their feature development. the developer can start developing new features without spending a few hours preparing the development environment.

sfp pool fetch -t dev-pool -v devhub

Fetching a scratch org from pool dev-pool in Org 00D6F000002Xfaixxx
Enabling source tracking, this will take a bit of time, please hang on
Copying the repository to /var/folders/0n/qs0txqqj6p7gkjlztgmf224h0000gn/T/tmp-11226-pTuia24bl8Gs
Successfully created temporary repository at /var/folders/0n/qs0txqqj6p7gkjlztgmf224h0000gn/T/tmp-11226-pTuia24bl8Gs with commit HEAD
Total Artifacts to Analyze: 1

Installed managed packages:
Packages installed in org:
                                                              
 Package             Version       Type      isOrgDependent 
 FSL                 238.0.35.1    Managed   false          
 Conga Composer      8.174.0.15    Managed   false          
 Conga Batch         8.22.0.3      Managed   false          
 Vlocity Insurance   890.333.0.1   Managed   false          
                                                              
Artifacts installed in org:
                                                                                          
 Artifact                      Version         Commit Id                                
 utils                         1.2.2.1        f79efddfd7f57e52d63222f6d9c23fb1e6a47152 

                                                                                          
======== Scratch org details ========
 KEY         VALUE                                                                                                                                               
 ─────────── ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 
 Id          2SR5m0000001d0vxxx                                                                                                                                  
 orgId       00D2O000000Hxxx                                                                                                                                     
 loginURL    https://efficiency-data-0000.my.salesforce.com/                                                                                                     
 username    test-zey1higmxxxx@example.com                                                                                                                       
 password    **********                                                                                                                                       
 expiryDate  2023-03-03                                                                                                                                          
 sfdxAuthUrl force://PlatformCLI::*******@efficiency-data-0000.my.salesforce.com 
 status      Assigned                                                                                                                                            
----------------------------------------------------------------------------------------------------
Succesfully fetched a scratch org and enabled source tracking  in 00:00:11.291
----------------------------------------------------------------------------------------------------