flxbl docs
slackGitHub
  • flxbl
  • sfp
  • sfops
  • Overview
  • Getting Started
    • Pre-Requisites
    • Install sfp
    • Configure Your Project
    • Build & Install an Artifact
    • Congratulations!
    • Docker Images
      • sfp-pro
  • CONCEPTS
    • Overview
    • SF CLI vs. SFP
    • Domains
    • Packages
    • Supported package types
      • Unlocked Packages
      • Org-Dependent Unlocked Packages
      • Source Packages
      • Diff Package
      • Data Packages
    • Artifacts
    • Package vs Artifacts
    • Identifying types of a package
    • Dependency management
    • Transitive Dependency Resolution
    • Destructive Changes
  • configuring a project
    • Project structure
    • Setup Salesforce Org
    • Creating a package
    • Defining a domain
    • Release Config
  • BUILDING ARTIFACTS
    • Overview
    • Determining whether an artifact need to be built
    • Building a domain
    • Building an artifact for package individually
    • Limiting artifacts to be built
    • Controlling aspects of the build command
      • Ignoring packages from being built
      • Building a collection of packages together
      • Selective ignoring of components from being built
      • Use of multiple config file in build command
    • Configuring installation behaviour of a package
      • Always deploy a package
      • Skip Install on Certain Orgs
      • Optimized Installation
      • Pre/Post Deployment Script
      • Reconciling Profiles
      • PermissionSet Assignment
      • Updating Picklist
      • Entitlement Deployment Helper
      • Field History & Feed Tracking
      • Aliasfy Packages
        • Aliasfy Packages - Merge Mode
      • State management for Flows
  • Installing an artifact
    • Overview
    • Controlling Aspects of Installation
    • Applying attributes of an artifact
    • BuiltIn Deployment Helpers
      • PermissionSet Group Awaiter
  • publishing and fetching artifacts
    • Publish Artifact
    • Fetching Artifacts
  • Releasing artifacts
    • Overview
    • Release Definitions
    • Generating a release definition
    • Generating a changelog
  • Validating a change
    • Overview
    • Different types of validation
    • Limiting Validation by Domain
    • Controlling validation attributes of a package
      • Skip Testing
      • Skip Coverage Validation
      • Test Synchronously
  • Analysing a Project
    • Overview
    • Duplicate Check
  • Environment Management
    • Pools
      • Scratch Org Pools
        • Defining a pool
        • Setting up your Salesforce Org for Scratch Org Pools
        • Pool Operations
          • Preparing pools
            • Handling dependencies
          • List Scratch Orgs in a pool
          • Fetch a scratch org
          • Delete Pools
      • Sandbox Pools
        • Sandbox Pool Initialization
        • Fetch a Sandbox from Pool
        • Monitor Sandbox Pools
    • Review Environments
      • Commands
        • Fetch a Review Environment
        • Check Review Environment Status
        • Extend a Review Environment
        • Transition Review Environment Status
        • Unassign a Review Environment
      • Considerations
    • Sandbox
      • Create Sandbox
      • Delete Sandbox
      • List Sandbox
      • Login to Sandbox
      • Update Sandbox
  • Development
    • Development Environment
    • Pull Changes from your org
    • Push Changes to your org
    • Dependency Management
      • Expand Dependencies
      • Shrink Dependencies
      • Explain Dependencies
  • Running sfp as a server
    • Introduction
    • sfp-pro-server: Architecture Overview (Alpha)
      • Task Processing System
      • Authentication & Security Architecture
      • Authentication System: Deep Dive
      • Database Architecture
      • Network Architecture and Integration System
      • Integration Architecture: Building Extensions
    • Installing SFP Server
    • Initializing SFP server
  • API Reference
    • Health
    • Authentication
    • Token
    • Salesforce
    • Team
    • Users
    • Tasks
    • Key Value
    • Repository
    • WebHooks
  • Metrics
    • Available Metrics
    • Custom Metrics
    • Configuring Collectors
      • Datadog
      • Splunk
      • New Relic
      • StatsD
  • Helpers
    • Managing Shared Resources
  • Command Guide
    • Core
      • Build
      • Quickbuild
      • Publish
      • Install
      • Release
    • Advanced
      • Validate
      • Artifacts
      • Changelog
      • Impact
      • Pool
      • Metrics
      • Repo
    • Utilities
      • Apex Tests
      • Flow
      • Dependency
      • Profile
  • FAQs
    • Common Errors
      • Org Shapes
      • Troubleshooting Unlocked Packages Build Failure Due to Code Coverage
    • Common Questions
      • Email Templates Deployment: Classic vs Lightning
      • Dealing with Long Build Times in Salesforce
      • Standard ValueSets and unlocked packages
      • Common Issues encountered with aliasfied packages
      • API Version
      • Understanding alwaysDeploy and skipIfAlreadyInstalled in Deployment Pipelines
    • sfp versioning and upgrade Process
  • References
  • Legal
    • Terms of Service for sfp
    • Terms of Service for 'sfp-pro' Software
  • LLMs.txt
Powered by GitBook
On this page
  • Task Lifecycle
  • Task Classification and Priority
  • Error Handling and Recovery
  1. Running sfp as a server
  2. sfp-pro-server: Architecture Overview (Alpha)

Task Processing System

Last updated 3 months ago

The task processing system in sfp pro server is designed to handle operations ranging from quick metadata validations to long-running deployments. Understanding how this system works is crucial because it forms the core of how work gets done in the platform.

Task Lifecycle

When a user or CI/CD system initiates an operation, it begins a journey through several stages of processing. Let's examine this journey in detail:

Task Classification and Priority

Every task in sfp pro server is assigned to one of three processing queues based on its characteristics:

  1. Critical Queue These tasks demand immediate attention and quick processing. They typically involve operations that developers are actively waiting on. For example:

    • Code linting operations

    • Quick metadata validations

    • Configuration checks

    Critical tasks receive dedicated worker capacity and are processed ahead of other queues. This ensures developers get immediate feedback for operations that block their work.

  2. Normal Queue The standard processing queue handles most day-to-day development operations. These tasks include:

    • Package installations

    • Deployment validations

    • Environment preparations

    • Source tracking operations

    Normal queue tasks are processed with fair scheduling, ensuring all teams get reasonable response times while managing system resources efficiently.

  3. Batch Queue This queue is designed for resource-intensive, long-running operations that don't require immediate completion. Examples include:

    • Full test suite executions

    • Bulk data operations

    • Nightly builds

    • Complete org validations

    Batch tasks can be preempted by higher priority work and are often scheduled during off-peak hours to maximize resource utilization.

Worker Lifecycle

Each worker in sfp pro server follows a strict lifecycle that ensures security and reliability:

  1. Initialization Phase

    • Worker process spawns in a clean environment

    • No inherited state or resources

    • Fresh memory space allocation

    • Temporary workspace creation

  2. Secret Loading The worker securely loads required credentials:

    • Salesforce org credentials

    • GitHub tokens

    • Other necessary secrets All secrets are loaded just-in-time and held only in memory.

  3. Environment Setup

    • Prepares necessary tools and dependencies

    • Configures connection parameters

    • Sets up logging channels

    • Initializes progress reporting

  4. Task Execution During execution, the worker:

    • Processes the assigned task

    • Reports progress through WebSocket channels

    • Manages resource utilization

    • Handles any necessary retries

  5. Completion and Cleanup After task completion:

    • All secrets are cleared from memory

    • Temporary files are securely removed

    • Resources are released

    • Worker process terminates completely

This ephemeral worker model provides several key benefits:

  • Complete isolation between tasks

  • No credential persistence

  • Clean state for each operation

  • Predictable resource cleanup

Resource Management

The task processing system includes sophisticated resource management to ensure system stability and fair resource allocation:

  1. Worker Pool Management Each queue type has its own worker pool:

    • Critical pool maintains minimum available workers

    • Normal pool scales based on demand

    • Batch pool uses excess capacity

  2. Resource Quotas The system enforces quotas at multiple levels:

    • Per-tenant resource limits

    • Queue-specific allocations

    • Individual task resource boundaries

  3. Dynamic Scaling Worker pools scale based on:

    • Current workload

    • Queue depths

    • Resource availability

    • Priority requirements

Error Handling and Recovery

The task processing system implements robust error handling to maintain reliability:

  1. Task Failure Handling When a task fails, the system:

    • Captures detailed error information

    • Provides error context through WebSocket updates

    • Maintains failure state for analysis

    • Cleans up any partial changes

  2. Worker Failure Recovery If a worker fails unexpectedly:

    • The task is marked as failed

    • Resources are forcefully cleaned up

    • Client is notified of the failure

    • System maintains audit trail

  3. Queue Recovery During system restarts or failures:

    • Queue state is preserved

    • Incomplete tasks are requeued

    • Progress information is maintained

    • Clients can reconnect to existing tasks

The error handling mechanism ensures that system stability is maintained even during unexpected failures, while providing clear feedback to users about any issues that arise.