Workflow Process Engine System Design

 J. Rogers, SE Ohio

System Overview

A generic workflow orchestration engine that routes XML documents through versioned process definitions. Each process instance maintains its complete state and history as it moves through configured steps, with the system handling routing, validation, and version management.

Core Concepts

The Event Stack

Each process instance is an XML document that accumulates data as it progresses:

• Contains all historical decisions and data
• Carries its process definition version
• Includes current execution state
• Self-contained audit trail

Process Definitions

Versioned configurations that define:

• Sequence of steps
• Input/output schemas for each step
• Routing logic based on outputs
• Inbox assignments

Inbox Types

Generic processors that handle steps:

• Human Inboxes: Present forms to users based on role
• AI Agents: Execute narrow processing tasks
• System Processors: Automated validations and checks
• External Systems: API integrations

Organizational Structure

Role Hierarchy

<organization>
  <roles>
    <role id="employee" level="1"/>
    <role id="team_lead" level="2" reports_to="department_manager"/>
    <role id="department_manager" level="3" reports_to="director"/>
    <role id="director" level="4" reports_to="vp"/>
    <role id="vp" level="5" reports_to="ceo"/>
    <role id="ceo" level="6"/>
    
    <role id="hr_specialist" level="2" department="hr"/>
    <role id="hr_manager" level="3" department="hr"/>
    <role id="finance_analyst" level="2" department="finance"/>
    <role id="finance_manager" level="3" department="finance"/>
  </roles>
  
  <people>
    <person id="12345" name="Jane Smith" role="employee" manager_role="team_lead" department="engineering"/>
    <person id="12346" name="Bob Jones" role="team_lead" manager_role="department_manager" department="engineering"/>
    <person id="12347" name="Alice Chen" role="department_manager" manager_role="director" department="engineering"/>
    <person id="54321" name="Carol White" role="hr_specialist" manager_role="hr_manager" department="hr"/>
  </people>
</organization>

Dynamic Role Resolution

The system resolves roles at runtime:

• submitter.manager → looks up submitter's manager_role, finds person in that role
• submitter.department.hr_specialist → finds HR specialist for submitter's department
• submitter.manager.manager → walks up org chart two levels

Process Definition Schema

<process_definition name="vacation_request" version="2">
  <metadata>
    <description>Employee vacation time request and approval</description>
    <created>2024-11-20</created>
    <active>true</active>
  </metadata>
  
  <steps>
    <step id="1" name="submission" type="human">
      <inbox>submitter</inbox>
      <form>
        <field name="start_date" type="date" required="true" label="Vacation Start Date"/>
        <field name="end_date" type="date" required="true" label="Vacation End Date"/>
        <field name="reason" type="text" required="false" label="Reason (optional)" max_length="500"/>
      </form>
      <output_schema>
        <step1_submission>
          <employee_id type="string"/>
          <start_date type="date"/>
          <end_date type="date"/>
          <days_requested type="integer"/>
          <reason type="text"/>
          <submitted_timestamp type="datetime"/>
        </step1_submission>
      </output_schema>
      <routing>
        <next>2</next>
      </routing>
    </step>
    
    <step id="2" name="policy_validation" type="ai">
      <inbox>ai_policy_validator</inbox>
      <input_fields>
        <field>step1_submission.employee_id</field>
        <field>step1_submission.start_date</field>
        <field>step1_submission.end_date</field>
        <field>step1_submission.days_requested</field>
      </input_fields>
      <task>
        <description>Validate vacation request against company policies</description>
        <checks>
          - Verify employee has sufficient PTO balance
          - Check team coverage (no more than 50% of team off simultaneously)
          - Verify not during company blackout periods
          - Calculate remaining balance after approval
        </checks>
      </task>
      <output_schema>
        <step2_policy_validation>
          <current_balance type="integer"/>
          <requested_days type="integer"/>
          <remaining_balance type="integer"/>
          <team_coverage_status type="enum" values="ok,conflict"/>
          <blackout_conflict type="boolean"/>
          <validation_result type="enum" values="approved,denied"/>
          <denial_reason type="text"/>
          <processed_timestamp type="datetime"/>
        </step2_policy_validation>
      </output_schema>
      <routing>
        <condition>
          <if field="step2_policy_validation.validation_result" equals="denied">
            <next>99</next> <!-- denial notification step -->
          </if>
          <else>
            <next>3</next>
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="3" name="manager_approval" type="human">
      <inbox>submitter.manager</inbox> <!-- resolved dynamically -->
      <form>
        <field name="decision" type="enum" values="approved,denied" required="true" label="Decision"/>
        <field name="notes" type="text" required="false" label="Notes" max_length="1000"/>
      </form>
      <output_schema>
        <step3_manager_approval>
          <approver_id type="string"/>
          <approver_role type="string"/>
          <decision type="enum" values="approved,denied"/>
          <notes type="text"/>
          <processed_timestamp type="datetime"/>
        </step3_manager_approval>
      </output_schema>
      <routing>
        <condition>
          <if field="step3_manager_approval.decision" equals="denied">
            <next>99</next>
          </if>
          <else>
            <next>4</next>
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="4" name="hr_approval" type="human">
      <inbox>submitter.department.hr_specialist</inbox>
      <form>
        <field name="decision" type="enum" values="approved,denied" required="true" label="Decision"/>
        <field name="notes" type="text" required="false" label="Notes" max_length="1000"/>
      </form>
      <output_schema>
        <step4_hr_approval>
          <approver_id type="string"/>
          <approver_role type="string"/>
          <decision type="enum" values="approved,denied"/>
          <notes type="text"/>
          <processed_timestamp type="datetime"/>
        </step4_hr_approval>
      </output_schema>
      <routing>
        <condition>
          <if field="step4_hr_approval.decision" equals="denied">
            <next>99</next>
          </if>
          <else>
            <next>5</next>
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="5" name="system_logging" type="system">
      <inbox>pto_system_logger</inbox>
      <input_fields>
        <field>step1_submission.employee_id</field>
        <field>step1_submission.start_date</field>
        <field>step1_submission.end_date</field>
        <field>step1_submission.days_requested</field>
        <field>step2_policy_validation.remaining_balance</field>
      </input_fields>
      <output_schema>
        <step5_system_logging>
          <pto_record_id type="string"/>
          <calendar_updated type="boolean"/>
          <balance_updated type="boolean"/>
          <processed_timestamp type="datetime"/>
        </step5_system_logging>
      </output_schema>
      <routing>
        <next>6</next>
      </routing>
    </step>
    
    <step id="6" name="confirmation" type="system">
      <inbox>email_notifier</inbox>
      <input_fields>
        <field>step1_submission.employee_id</field>
        <field>step1_submission.start_date</field>
        <field>step1_submission.end_date</field>
      </input_fields>
      <output_schema>
        <step6_confirmation>
          <email_sent type="boolean"/>
          <notification_id type="string"/>
          <processed_timestamp type="datetime"/>
        </step6_confirmation>
      </output_schema>
      <routing>
        <complete/>
      </routing>
    </step>
    
    <step id="99" name="denial_notification" type="system">
      <inbox>email_notifier</inbox>
      <input_fields>
        <field>step1_submission.employee_id</field>
        <field>step2_policy_validation.denial_reason</field>
        <field>step3_manager_approval.notes</field>
        <field>step4_hr_approval.notes</field>
      </input_fields>
      <output_schema>
        <step99_denial_notification>
          <email_sent type="boolean"/>
          <notification_id type="string"/>
          <processed_timestamp type="datetime"/>
        </step99_denial_notification>
      </output_schema>
      <routing>
        <complete status="denied"/>
      </routing>
    </step>
  </steps>
</process_definition>

Inbox Type Implementations

1. Human Inbox

Purpose: Present forms to users and collect responses

Configuration:

<inbox_type name="human">
  <handler>human_task_handler</handler>
  <capabilities>
    - Render forms based on field definitions
    - Validate input against schema
    - Support role-based routing
    - Queue tasks by priority
  </capabilities>
</inbox_type>

Behavior:

• Receives XML document with current state
• Looks up user by role (using org chart)
• Presents form with specified fields
• Validates responses match output schema
• Appends response to XML document
• Returns to orchestrator for routing

2. AI Agent Inbox

Purpose: Execute narrow AI processing tasks

Configuration:

<inbox_type name="ai">
  <handler>ai_agent_handler</handler>
  <capabilities>
    - Execute specific AI tasks
    - Extract structured data from text
    - Validate against business rules
    - Classify and categorize
    - Generate summaries
  </capabilities>
</inbox_type>

Example AI Agents:

ai_policy_validator:

Task: Validate vacation request against policies
Inputs: employee_id, start_date, end_date, days_requested
Process:
  1. Query HR database for PTO balance
  2. Query team calendar for coverage
  3. Check blackout period calendar
  4. Calculate remaining balance
  5. Determine approval/denial
Output: structured validation result

ai_expense_categorizer:

Task: Categorize expense from receipt and description
Inputs: receipt_image, description, amount
Process:
  1. Extract text from receipt image (OCR)
  2. Identify vendor and items
  3. Classify expense type
  4. Flag policy violations (limits, prohibited vendors)
Output: categorization and validation

ai_invoice_extractor:

Task: Extract structured data from invoice
Inputs: invoice_pdf, vendor_info
Process:
  1. Extract line items
  2. Identify totals and taxes
  3. Match to purchase order if available
  4. Flag discrepancies
Output: structured invoice data

3. System Processor Inbox

Purpose: Automated system operations

Configuration:

<inbox_type name="system">
  <handler>system_processor_handler</handler>
  <capabilities>
    - Database queries and updates
    - Business rule evaluation
    - Data transformations
    - System integrations
  </capabilities>
</inbox_type>

Example System Processors:

pto_system_logger:

Process: Log approved vacation to HR system
Inputs: employee_id, start_date, end_date, days_requested
Actions:
  1. Update PTO balance in HR database
  2. Add entries to team calendar
  3. Generate PTO record ID
  4. Update employee profile
Output: confirmation of updates

budget_validator:

Process: Check if expense within budget
Inputs: department, category, amount
Actions:
  1. Query current budget for department/category
  2. Calculate spent vs. allocated
  3. Check if amount would exceed budget
  4. Flag for review if over threshold
Output: validation result with budget info

email_notifier:

Process: Send notification emails
Inputs: recipient_id, template, data
Actions:
  1. Look up recipient email
  2. Render email from template with data
  3. Send via email service
  4. Log notification
Output: confirmation with notification ID

4. External System Inbox

Purpose: Integration with third-party services

Configuration:

<inbox_type name="external">
  <handler>external_system_handler</handler>
  <capabilities>
    - API calls to external services
    - Authentication management
    - Response transformation
    - Error handling and retry
  </capabilities>
</inbox_type>

Example External Systems:

credit_bureau_api:

Process: Request credit check
Inputs: applicant_ssn, applicant_name, loan_amount
Actions:
  1. Authenticate with credit bureau API
  2. Submit credit check request
  3. Wait for response (may be async)
  4. Transform response to internal schema
  5. Store report ID for compliance
Output: credit score, risk level, report ID

payment_processor:

Process: Process payment transaction
Inputs: amount, payment_method, invoice_id
Actions:
  1. Connect to payment gateway
  2. Submit payment request
  3. Handle success/failure
  4. Store transaction ID
Output: transaction status and ID

background_check_service:

python

Process: Initiate background check
Inputs: candidate_info, check_types
Actions:
  1. Submit to background check vendor
  2. Poll for completion (async)
  3. Retrieve results when ready
  4. Transform to standard format
Output: background check results

Additional Process Definition Examples

Invoice Approval Process

<process_definition name="invoice_approval" version="1">
  <steps>
    <step id="1" name="submission" type="human">
      <inbox>submitter</inbox>
      <form>
        <field name="vendor_name" type="text" required="true"/>
        <field name="invoice_number" type="text" required="true"/>
        <field name="invoice_pdf" type="file" required="true" accept="application/pdf"/>
        <field name="amount" type="currency" required="true"/>
        <field name="department" type="enum" required="true"/>
      </form>
    </step>
    
    <step id="2" name="data_extraction" type="ai">
      <inbox>ai_invoice_extractor</inbox>
      <task>Extract line items, verify totals, match to PO if available</task>
    </step>
    
    <step id="3" name="budget_check" type="system">
      <inbox>budget_validator</inbox>
    </step>
    
    <step id="4" name="manager_approval" type="human">
      <inbox>submitter.manager</inbox>
      <routing>
        <condition>
          <if field="step1_submission.amount" greater_than="5000">
            <next>5</next> <!-- requires director approval -->
          </if>
          <else>
            <next>6</next> <!-- skip to finance -->
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="5" name="director_approval" type="human">
      <inbox>submitter.manager.manager</inbox>
    </step>
    
    <step id="6" name="finance_review" type="human">
      <inbox>role.finance_analyst</inbox>
    </step>
    
    <step id="7" name="payment_processing" type="external">
      <inbox>payment_processor</inbox>
    </step>
    
    <step id="8" name="confirmation" type="system">
      <inbox>email_notifier</inbox>
    </step>
  </steps>
</process_definition>

New Hire Onboarding Process

<process_definition name="new_hire_onboarding" version="1">
  <steps>
    <step id="1" name="hr_submission" type="human">
      <inbox>role.hr_specialist</inbox>
      <form>
        <field name="employee_name" type="text" required="true"/>
        <field name="start_date" type="date" required="true"/>
        <field name="department" type="enum" required="true"/>
        <field name="role_title" type="text" required="true"/>
        <field name="is_remote" type="boolean" required="true"/>
      </form>
    </step>
    
    <step id="2" name="background_check" type="external">
      <inbox>background_check_service</inbox>
      <routing>
        <condition>
          <if field="step2_background_check.status" equals="failed">
            <next>99</next> <!-- rejection path -->
          </if>
          <else>
            <next>3</next>
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="3" name="it_provisioning" type="system">
      <inbox>it_provisioning_system</inbox>
      <!-- Creates email, assigns laptop, grants system access -->
    </step>
    
    <step id="4" name="buddy_assignment" type="human">
      <inbox>step1_hr_submission.department.manager</inbox>
      <form>
        <field name="buddy" type="employee_selector" required="true"/>
        <field name="onboarding_plan" type="text"/>
      </form>
    </step>
    
    <step id="5" name="facilities_setup" type="system">
      <inbox>facilities_system</inbox>
      <routing>
        <condition>
          <if field="step1_hr_submission.is_remote" equals="true">
            <next>7</next> <!-- skip desk assignment -->
          </if>
          <else>
            <next>6</next>
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="6" name="desk_assignment" type="human">
      <inbox>role.facilities_manager</inbox>
      <form>
        <field name="desk_number" type="text" required="true"/>
        <field name="parking_spot" type="text" required="false"/>
      </form>
    </step>
    
    <step id="7" name="welcome_email" type="system">
      <inbox>email_notifier</inbox>
      <wait_until>step1_hr_submission.start_date</wait_until>
    </step>
    
    <step id="8" name="30_day_checkin" type="human">
      <inbox>step4_buddy_assignment.buddy</inbox>
      <wait_for_days>30</wait_for_days>
      <form>
        <field name="feedback" type="text" required="true"/>
        <field name="concerns" type="text" required="false"/>
      </form>
    </step>
  </steps>
</process_definition>

Expense Report Process

<process_definition name="expense_report" version="1">
  <steps>
    <step id="1" name="submission" type="human">
      <inbox>submitter</inbox>
      <form>
        <field name="purpose" type="text" required="true"/>
        <field name="receipts" type="file" multiple="true" required="true"/>
      </form>
    </step>
    
    <step id="2" name="expense_categorization" type="ai">
      <inbox>ai_expense_categorizer</inbox>
      <task>Extract expenses from receipts, categorize each, flag policy violations</task>
    </step>
    
    <step id="3" name="policy_validation" type="system">
      <inbox>expense_policy_validator</inbox>
      <routing>
        <condition>
          <if field="step3_policy_validation.violations_found" equals="true">
            <next>4</next> <!-- manager must review violations -->
          </if>
          <else>
            <next>5</next> <!-- auto-approve if under limit -->
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="4" name="manager_review" type="human">
      <inbox>submitter.manager</inbox>
    </step>
    
    <step id="5" name="finance_processing" type="human">
      <inbox>role.finance_analyst</inbox>
      <routing>
        <condition>
          <if field="step1_submission.total_amount" greater_than="1000">
            <next>6</next> <!-- requires manager approval -->
          </if>
          <else>
            <next>7</next> <!-- direct to reimbursement -->
          </else>
        </condition>
      </routing>
    </step>
    
    <step id="6" name="finance_manager_approval" type="human">
      <inbox>role.finance_manager</inbox>
    </step>
    
    <step id="7" name="reimbursement" type="external">
      <inbox>payment_processor</inbox>
    </step>
  </steps>
</process_definition>

Example Process Instance (Running Vacation Request)

<process_instance id="vac-2024-00482" process="vacation_request" version="2" status="active">
  <metadata>
    <created>2024-11-20T09:15:00Z</created>
    <last_updated>2024-11-21T14:30:00Z</last_updated>
  </metadata>
  
  <execution_state>
    <current_step>3</current_step>
    <awaiting_inbox>human:john.doe@company.com</awaiting_inbox>
    <awaiting_role>team_lead</awaiting_role>
  </execution_state>
  
  <step1_submission>
    <employee_id>12345</employee_id>
    <employee_name>Jane Smith</employee_name>
    <start_date>2024-12-20</start_date>
    <end_date>2024-12-27</end_date>
    <days_requested>5</days_requested>
    <reason>Family holiday vacation</reason>
    <submitted_timestamp>2024-11-20T09:15:00Z</submitted_timestamp>
  </step1_submission>
  
  <step2_policy_validation>
    <current_balance>12</current_balance>
    <requested_days>5</requested_days>
    <remaining_balance>7</remaining_balance>
    <team_coverage_status>ok</team_coverage_status>
    <team_coverage_details>
      <total_team_members>8</total_team_members>
      <others_on_vacation>1</others_on_vacation>
      <coverage_percentage>87.5</coverage_percentage>
    </team_coverage_details>
    <blackout_conflict>false</blackout_conflict>
    <validation_result>approved</validation_result>
    <denial_reason/>
    <processed_timestamp>2024-11-20T09:15:30Z</processed_timestamp>
    <processed_by>ai_policy_validator</processed_by>
  </step2_policy_validation>
  
  <!-- Currently waiting at step 3 for manager approval -->
  <!-- Step 3 will be added when manager responds -->
</process_instance>

System Architecture

Orchestration Engine

class OrchestrationEngine:
    """
    Core engine that routes process instances through their definitions
    """
    
    def process_event(self, process_instance_xml):
        # 1. Parse current state
        instance = parse_xml(process_instance_xml)
        process_def = load_process_definition(
            instance.process_name, 
            instance.version
        )
        
        # 2. Get current step definition
        current_step = process_def.get_step(instance.current_step)
        
        # 3. Route to appropriate inbox
        inbox = resolve_inbox(current_step.inbox, instance)
        inbox_handler = get_inbox_handler(current_step.type)
        
        # 4. Send to inbox and wait for response
        response = inbox_handler.process(
            process_instance=instance,
            step_definition=current_step
        )
        
        # 5. Validate response against schema
        validate_schema(response, current_step.output_schema)
        
        # 6. Append response to instance
        instance.append_step_data(response)
        
        # 7. Determine next step based on routing rules
        next_step = evaluate_routing(
            current_step.routing,
            instance
        )
        
        # 8. Update execution state
        if next_step == "complete":
            instance.status = "complete"
        else:
            instance.current_step = next_step
            instance.awaiting_inbox = resolve_inbox(
                process_def.get_step(next_step).inbox,
                instance
            )
        
        # 9. Persist updated instance
        save_process_instance(instance)
        
        # 10. Continue to next step if not waiting for human
        if not requires_human_input(next_step):
            self.process_event(instance.to_xml())

Role Resolution

class RoleResolver:
    """
    Resolves dynamic role references to actual people
    """
    
    def resolve(self, role_reference, process_instance, org_chart):
        # Examples:
        # "submitter.manager" -> find submitter's manager
        # "submitter.department.hr_specialist" -> find HR for dept
        # "role.finance_analyst" -> any finance analyst
        
        parts = role_reference.split('.')
        
        if parts[0] == "submitter":
            person = org_chart.get_person(
                process_instance.step1_submission.employee_id
            )
            return self._walk_reference(person, parts[1:], org_chart)
        
        elif parts[0] == "role":
            # Find any person with this role
            return org_chart.get_person_by_role(parts[1])
        
        else:
            # Direct reference to a system inbox
            return role_reference
    
    def _walk_reference(self, person, path, org_chart):
        if not path:
            return person
        
        next_hop = path[0]
        
        if next_hop == "manager":
            manager = org_chart.get_manager(person)
            return self._walk_reference(manager, path[1:], org_chart)
        
        elif next_hop == "department":
            # Next part should be a role within the department
            role = path[1]
            return org_chart.get_person_by_role_and_department(
                role, 
                person.department
            )

Key Features

Version Management

• New process instances use latest active version
• In-flight instances complete on their original version
• Allows gradual rollout and A/B testing
• Version history maintained for compliance

Audit Trail

• Complete history in XML document
• Every decision and data point timestamped
• Who processed each step (person or system)
• Immutable once step completed

Extensibility

• New inbox types can be added
• Process definitions are pure configuration
• AI agents can be added/updated independently
• External systems integrated via adapters

Error Handling

• Failed steps can retry with exponential backoff
• Timeouts can escalate to manager or alternative path
• Validation errors return to previous step
• System maintains dead letter queue for unprocessable instances

This architecture provides a complete SAP-level workflow engine that's generic enough to handle any business process while being specific enough to integrate AI agents, human approvals, and system automations seamlessly.