aishastr.live

Agentic Design Pattern: Simulation Testing

Written by

admin

in


    """
Pattern 092: Simulation Testing

Description:
    Simulation Testing creates controlled, reproducible environments to test agent behavior
    under various scenarios, conditions, and edge cases. Unlike production testing,
    simulations allow you to test dangerous, expensive, or rare scenarios safely and
    repeatedly. This pattern is essential for validating agent behavior before deployment
    and for stress testing system limits.

    Simulation testing enables:
    - Safe testing of dangerous scenarios
    - Reproducible test conditions
    - Stress and load testing
    - Edge case exploration
    - Multi-agent interaction testing
    - Cost-effective experimentation

Components:
    1. Simulation Environment
       - Mock external services (APIs, databases)
       - Configurable conditions (delays, errors, loads)
       - State management and reset
       - Recording and replay capabilities
       - Deterministic execution

    2. Scenario Definition
       - Initial state setup
       - Event sequences
       - Expected outcomes
       - Success criteria
       - Failure conditions

    3. Test Harness
       - Scenario execution engine
       - Result validation
       - Metrics collection
       - Logging and tracing
       - Report generation

    4. Mock Components
       - LLM response mocking
       - Tool/API simulation
       - User interaction simulation
       - External system simulation
       - Network condition simulation

Use Cases:
    1. Safety Testing
       - Dangerous scenarios without risk
       - Edge case handling
       - Failure mode analysis
       - Recovery testing
       - Adversarial inputs

    2. Performance Testing
       - Load testing (many concurrent requests)
       - Stress testing (beyond capacity)
       - Endurance testing (long duration)
       - Spike testing (sudden load changes)
       - Scalability validation

    3. Integration Testing
       - Multi-component interaction
       - External service integration
       - Data flow validation
       - Error propagation
       - State consistency

    4. Regression Testing
       - Validate fixes don't break features
       - Ensure consistent behavior
       - Track performance changes
       - Verify improvements
       - Document behavior changes

    5. Exploratory Testing
       - Discover edge cases
       - Test rare scenarios
       - Validate assumptions
       - Find unexpected behaviors
       - Stress system boundaries

LangChain Implementation:
    LangChain supports simulation through:
    - Mock LLM implementations
    - Custom callbacks for control
    - Test chain compositions
    - Fake embeddings for testing
    - Memory simulation

Key Features:
    1. Controllable Environment
       - Deterministic responses
       - Configurable delays
       - Error injection
       - State manipulation
       - Reproducible conditions

    2. Comprehensive Scenarios
       - Normal operations
       - Error conditions
       - Edge cases
       - Load patterns
       - Attack scenarios

    3. Detailed Observation
       - Action logging
       - State tracking
       - Performance metrics
       - Error recording
       - Decision paths

    4. Flexible Validation
       - Multiple success criteria
       - Partial success handling
       - Performance thresholds
       - State assertions
       - Behavior verification

Best Practices:
    1. Scenario Design
       - Start simple, add complexity
       - Cover happy path first
       - Add error scenarios
       - Include edge cases
       - Test boundaries

    2. Mock Fidelity
       - Balance realism vs simplicity
       - Accurate error simulation
       - Realistic timing
       - State consistency
       - Clear documentation

    3. Test Independence
       - Each test isolated
       - Clean state between tests
       - No shared dependencies
       - Deterministic execution
       - Parallel-safe

    4. Performance
       - Fast execution ( str:
        """
        Mock invoke method.
        
        Args:
            input: Input string
            
        Returns:
            Mock response
        """
        self.call_count += 1
        self.call_history.append({
            "input": input,
            "timestamp": datetime.now()
        })
        
        # Find matching response
        response = self.responses.get(input, self.default_response)
        
        # Simulate delay
        if response.delay > 0:
            time.sleep(response.delay)
        
        # Simulate error
        if response.should_error:
            raise Exception(response.error_message or "Mock error")
        
        return response.content
    
    def add_response(self, input: str, response: MockResponse):
        """Add a response pattern"""
        self.responses[input] = response
    
    def reset(self):
        """Reset mock state"""
        self.call_count = 0
        self.call_history.clear()


class MockTool:
    """
    Mock tool for testing agent tool use.
    
    Simulates external tools without actual execution.
    """
    
    def __init__(
        self,
        name: str,
        behavior: Callable[[str], str],
        should_fail: bool = False,
        failure_rate: float = 0.0
    ):
        """
        Initialize mock tool.
        
        Args:
            name: Tool name
            behavior: Function defining tool behavior
            should_fail: Whether tool should always fail
            failure_rate: Probability of failure (0-1)
        """
        self.name = name
        self.behavior = behavior
        self.should_fail = should_fail
        self.failure_rate = failure_rate
        self.call_count = 0
    
    def execute(self, input: str) -> str:
        """
        Execute tool with mock behavior.
        
        Args:
            input: Tool input
            
        Returns:
            Tool output
        """
        self.call_count += 1
        
        # Simulate failure
        if self.should_fail or random.random()  Any:
        """Get environment state"""
        return self.state.get(key)
    
    def log_event(self, event_type: str, details: Dict[str, Any]):
        """Log an event"""
        self.event_log.append({
            "timestamp": datetime.now(),
            "type": event_type,
            "details": details
        })
    
    def reset(self):
        """Reset environment to initial state"""
        self.mock_llm.reset()
        for tool in self.mock_tools.values():
            tool.call_count = 0
        self.state.clear()
        self.event_log.clear()


class SimulationTester:
    """
    Executes and validates simulation tests.
    
    Runs scenarios in controlled environment and collects results.
    """
    
    def __init__(self, environment: SimulationEnvironment):
        """
        Initialize tester.
        
        Args:
            environment: Simulation environment
        """
        self.environment = environment
    
    def run_scenario(self, scenario: SimulationScenario) -> SimulationResult:
        """
        Run a simulation scenario.
        
        Args:
            scenario: Scenario to run
            
        Returns:
            Simulation result
        """
        # Reset environment
        self.environment.reset()
        
        # Setup initial state
        for key, value in scenario.initial_state.items():
            self.environment.set_state(key, value)
        
        start_time = time.time()
        observations = []
        errors = []
        events_executed = 0
        
        try:
            # Execute events
            for event in scenario.events:
                event_type = event.get("type")
                
                if event_type == "llm_call":
                    input_text = event.get("input", "")
                    try:
                        response = self.environment.mock_llm.invoke(input_text)
                        observations.append(f"LLM called: {input_text[:50]}...")
                        observations.append(f"LLM response: {response[:50]}...")
                        events_executed += 1
                    except Exception as e:
                        errors.append(f"LLM error: {str(e)}")
                
                elif event_type == "tool_call":
                    tool_name = event.get("tool")
                    tool_input = event.get("input", "")
                    
                    if tool_name in self.environment.mock_tools:
                        try:
                            result = self.environment.mock_tools[tool_name].execute(tool_input)
                            observations.append(f"Tool {tool_name} called")
                            observations.append(f"Tool result: {result[:50]}...")
                            events_executed += 1
                        except Exception as e:
                            errors.append(f"Tool error: {str(e)}")
                
                elif event_type == "state_update":
                    key = event.get("key")
                    value = event.get("value")
                    self.environment.set_state(key, value)
                    observations.append(f"State updated: {key}={value}")
                    events_executed += 1
                
                # Log event
                self.environment.log_event(event_type, event)
            
            # Evaluate success criteria
            success = scenario.success_criteria(self.environment.state)
            
            # Determine outcome
            if errors:
                outcome = OutcomeType.ERROR
            elif success:
                outcome = OutcomeType.SUCCESS
            elif events_executed  List[SimulationResult]:
        """
        Run multiple scenarios.
        
        Args:
            scenarios: List of scenarios
            
        Returns:
            List of results
        """
        results = []
        for scenario in scenarios:
            result = self.run_scenario(scenario)
            results.append(result)
        return results


def demonstrate_simulation_testing():
    """Demonstrate simulation testing"""
    print("=" * 80)
    print("SIMULATION TESTING DEMONSTRATION")
    print("=" * 80)
    
    # Example 1: Setup Simulation Environment
    print("\n" + "=" * 80)
    print("EXAMPLE 1: Creating Simulation Environment")
    print("=" * 80)
    
    env = SimulationEnvironment()
    
    # Configure mock LLM responses
    env.setup_mock_llm({
        "What is 2+2?": MockResponse(content="4", delay=0.1),
        "What is the capital of France?": MockResponse(content="Paris", delay=0.1),
        "Error test": MockResponse(
            content="",
            should_error=True,
            error_message="Simulated LLM error"
        )
    })
    
    # Add mock tools
    calculator = MockTool(
        name="calculator",
        behavior=lambda x: str(eval(x)),  # Simple eval for demo
        failure_rate=0.0
    )
    env.add_mock_tool(calculator)
    
    search = MockTool(
        name="search",
        behavior=lambda x: f"Search results for: {x}",
        failure_rate=0.0
    )
    env.add_mock_tool(search)
    
    print("\nSimulation environment configured:")
    print(f"  Mock LLM: {len(env.mock_llm.responses)} responses")
    print(f"  Mock Tools: {len(env.mock_tools)} tools")
    print(f"    - calculator")
    print(f"    - search")
    
    # Example 2: Normal Operation Scenario
    print("\n" + "=" * 80)
    print("EXAMPLE 2: Normal Operation Scenario")
    print("=" * 80)
    
    normal_scenario = SimulationScenario(
        id="scenario_001",
        name="Normal Q&A Flow",
        description="Test normal question answering",
        scenario_type=ScenarioType.NORMAL,
        initial_state={"question_count": 0},
        events=[
            {"type": "llm_call", "input": "What is 2+2?"},
            {"type": "state_update", "key": "question_count", "value": 1},
            {"type": "llm_call", "input": "What is the capital of France?"},
            {"type": "state_update", "key": "question_count", "value": 2}
        ],
        success_criteria=lambda state: state.get("question_count", 0) == 2
    )
    
    tester = SimulationTester(env)
    result = tester.run_scenario(normal_scenario)
    
    print(f"\nScenario: {normal_scenario.name}")
    print(f"Type: {normal_scenario.scenario_type.value}")
    print(f"Outcome: {result.outcome.value}")
    print(f"Duration: {result.duration:.3f}s")
    print(f"Events executed: {result.events_executed}/{len(normal_scenario.events)}")
    print(f"Success criteria met: {result.success_criteria_met}")
    print(f"\nMetrics:")
    for key, value in result.metrics.items():
        print(f"  {key}: {value}")
    
    # Example 3: Error Handling Scenario
    print("\n" + "=" * 80)
    print("EXAMPLE 3: Error Handling Scenario")
    print("=" * 80)
    
    error_scenario = SimulationScenario(
        id="scenario_002",
        name="LLM Error Recovery",
        description="Test error handling when LLM fails",
        scenario_type=ScenarioType.ERROR,
        initial_state={"error_handled": False},
        events=[
            {"type": "llm_call", "input": "Error test"},
            {"type": "state_update", "key": "error_handled", "value": True}
        ],
        success_criteria=lambda state: state.get("error_handled", False)
    )
    
    error_result = tester.run_scenario(error_scenario)
    
    print(f"\nScenario: {error_scenario.name}")
    print(f"Outcome: {error_result.outcome.value}")
    print(f"Errors encountered: {len(error_result.errors)}")
    for error in error_result.errors:
        print(f"  - {error}")
    
    # Example 4: Tool Usage Scenario
    print("\n" + "=" * 80)
    print("EXAMPLE 4: Tool Usage Scenario")
    print("=" * 80)
    
    tool_scenario = SimulationScenario(
        id="scenario_003",
        name="Calculator Tool Usage",
        description="Test agent using calculator tool",
        scenario_type=ScenarioType.NORMAL,
        initial_state={"calculations_done": 0},
        events=[
            {"type": "tool_call", "tool": "calculator", "input": "10 + 5"},
            {"type": "state_update", "key": "calculations_done", "value": 1},
            {"type": "tool_call", "tool": "calculator", "input": "20 * 3"},
            {"type": "state_update", "key": "calculations_done", "value": 2}
        ],
        success_criteria=lambda state: state.get("calculations_done", 0) == 2
    )
    
    tool_result = tester.run_scenario(tool_scenario)
    
    print(f"\nScenario: {tool_scenario.name}")
    print(f"Outcome: {tool_result.outcome.value}")
    print(f"Tool calls: {tool_result.metrics['tool_calls']}")
    print(f"Success: {tool_result.success_criteria_met}")
    
    print("\nObservations:")
    for obs in tool_result.observations:
        print(f"  {obs}")
    
    # Example 5: Edge Case Scenario
    print("\n" + "=" * 80)
    print("EXAMPLE 5: Edge Case Scenario")
    print("=" * 80)
    
    edge_scenario = SimulationScenario(
        id="scenario_004",
        name="Empty Input Handling",
        description="Test handling of empty inputs",
        scenario_type=ScenarioType.EDGE_CASE,
        initial_state={"edge_case_handled": False},
        events=[
            {"type": "llm_call", "input": ""},
            {"type": "state_update", "key": "edge_case_handled", "value": True}
        ],
        success_criteria=lambda state: state.get("edge_case_handled", False)
    )
    
    edge_result = tester.run_scenario(edge_scenario)
    
    print(f"\nScenario: {edge_scenario.name}")
    print(f"Type: {edge_scenario.scenario_type.value}")
    print(f"Outcome: {edge_result.outcome.value}")
    print(f"Success: {edge_result.success_criteria_met}")
    
    # Example 6: Load Testing Simulation
    print("\n" + "=" * 80)
    print("EXAMPLE 6: Load Testing Simulation")
    print("=" * 80)
    
    print("\nSimulating high load scenario...")
    
    load_scenarios = []
    for i in range(10):
        scenario = SimulationScenario(
            id=f"load_test_{i}",
            name=f"Load Test {i}",
            description="Concurrent request simulation",
            scenario_type=ScenarioType.LOAD,
            initial_state={},
            events=[
                {"type": "llm_call", "input": f"Query {i}"},
            ],
            success_criteria=lambda state: True
        )
        load_scenarios.append(scenario)
    
    start_time = time.time()
    load_results = tester.run_scenarios(load_scenarios)
    total_time = time.time() - start_time
    
    successful = sum(1 for r in load_results if r.outcome == OutcomeType.SUCCESS)
    
    print(f"\nLoad test completed:")
    print(f"  Total scenarios: {len(load_scenarios)}")
    print(f"  Successful: {successful}")
    print(f"  Failed: {len(load_scenarios) - successful}")
    print(f"  Total time: {total_time:.2f}s")
    print(f"  Avg time per scenario: {total_time/len(load_scenarios):.3f}s")
    
    # Example 7: Comprehensive Test Suite
    print("\n" + "=" * 80)
    print("EXAMPLE 7: Running Comprehensive Test Suite")
    print("=" * 80)
    
    test_suite = [normal_scenario, error_scenario, tool_scenario, edge_scenario]
    
    print(f"\nRunning {len(test_suite)} scenarios...\n")
    
    suite_results = tester.run_scenarios(test_suite)
    
    # Analyze results
    by_outcome = {}
    for result in suite_results:
        outcome = result.outcome.value
        by_outcome[outcome] = by_outcome.get(outcome, 0) + 1
    
    print("Test Suite Results:")
    print(f"  Total: {len(suite_results)}")
    for outcome, count in by_outcome.items():
        print(f"  {outcome.upper()}: {count}")
    
    success_rate = (by_outcome.get("success", 0) / len(suite_results)) * 100
    print(f"\nSuccess Rate: {success_rate:.1f}%")
    
    print("\nDetailed Results:")
    for result in suite_results:
        status = "✓" if result.outcome == OutcomeType.SUCCESS else "✗"
        print(f"  {status} {result.scenario_id}: {result.outcome.value}")
        print(f"     Duration: {result.duration:.3f}s")
        print(f"     Events: {result.events_executed}")
        if result.errors:
            print(f"     Errors: {len(result.errors)}")
    
    # Summary
    print("\n" + "=" * 80)
    print("SIMULATION TESTING SUMMARY")
    print("=" * 80)
    print("""
Simulation Testing Benefits:
1. Safety: Test dangerous scenarios without risk
2. Reproducibility: Consistent test conditions
3. Speed: Fast execution without API calls
4. Cost: No real API costs during testing
5. Control: Complete control over conditions
6. Coverage: Test rare and edge cases easily

Simulation Components:
1. Mock LLM
   - Pre-configured responses
   - Error simulation
   - Delay simulation
   - Call tracking
   - Deterministic behavior

2. Mock Tools
   - Simulated tool behavior
   - Failure injection
   - Performance simulation
   - Call counting
   - State management

3. Simulation Environment
   - State management
   - Event logging
   - Component coordination
   - Reset capabilities
   - Observation collection

Scenario Types:
1. Normal Operation
   - Happy path testing
   - Expected workflows
   - Common use cases
   - Standard behavior
   - Success criteria validation

2. Error Handling
   - LLM failures
   - Tool failures
   - Network errors
   - Timeout scenarios
   - Recovery mechanisms

3. Edge Cases
   - Boundary conditions
   - Empty inputs
   - Large inputs
   - Unusual patterns
   - Corner cases

4. Load Testing
   - Concurrent requests
   - High volume
   - Resource limits
   - Performance degradation
   - Scalability validation

5. Adversarial
   - Malicious inputs
   - Attack scenarios
   - Security testing
   - Abuse detection
   - Safety boundaries

Best Practices:
1. Scenario Design
   - Clear objectives
   - Measurable outcomes
   - Realistic conditions
   - Comprehensive coverage
   - Maintainable tests

2. Mock Fidelity
   - Accurate behavior
   - Realistic timing
   - Error patterns
   - State consistency
   - Documentation

3. Test Organization
   - Logical grouping
   - Clear naming
   - Independent tests
   - Reusable components
   - Version control

4. Continuous Testing
   - CI/CD integration
   - Automated runs
   - Fast feedback
   - Regression detection
   - Quality gates

When to Use Simulation:
✓ Development phase testing
✓ Dangerous scenario validation
✓ Rare case exploration
✓ Cost-sensitive testing
✓ Performance testing
✓ Edge case discovery
✗ Production validation (use real testing)
✗ API behavior verification (use integration tests)

Limitations:
- Mock/reality gap
- May miss real-world issues
- Requires maintenance
- Setup complexity
- False confidence risk

Complementary Approaches:
1. Simulation (Development)
   - Fast feedback
   - Safe exploration
   - Cost-effective

2. Integration Tests (Pre-production)
   - Real API calls
   - Actual behavior
   - Environment validation

3. Shadow Testing (Production)
   - Real traffic
   - No user impact
   - Behavior comparison

4. A/B Testing (Production)
   - User feedback
   - Real performance
   - Gradual rollout

Production Tips:
- Use simulation for development
- Combine with integration tests
- Validate mocks against reality
- Update scenarios regularly
- Monitor real vs simulated gaps
- Document assumptions
- Review test coverage
- Fast test execution (< 5s per scenario)
- Parallel execution for large suites
- Clear failure messages
""")
    
    print("\n" + "=" * 80)
    print("Pattern 092 (Simulation Testing) demonstration complete!")
    print("=" * 80)


if __name__ == "__main__":
    demonstrate_simulation_testing()

Agentic Design Pattern: Simulation Testing

Pattern 092: Simulation Testing – Line-by-Line Explanation

Overview

This document provides a detailed line-by-line explanation of the Simulation Testing pattern implementation.

Pattern Description

Pattern 092: Simulation Testing

Description: Simulation Testing creates controlled, reproducible environments to test agent behavior under various scenarios, conditions, and edge cases. Unlike production testing, simulations allow you to test dangerous, expensive, or rare scenarios safely and repeatedly. This pattern is essential for validating agent behavior before deployment and for stress testing system limits.

Simulation testing enables:
- Safe testing of dangerous scenarios
- Reproducible test conditions
- Stress and load testing
- Edge case exploration
- Multi-agent interaction testing
- Cost-effective experimentation

Components: 1. Simulation Environment – Mock external services (APIs, databases) – Configurable conditions (delays, errors, loads) – State management and reset – Recording and replay capabilities – Deterministic execution

2. Scenario Definition
   - Initial state setup
   - Event sequences
   - Expected outcomes
   - Success criteria
   - Failure conditions

3. Test Harness
   - Scenario execution engine
   - Result validation
   - Metrics collection
   - Logging and tracing
   - Report generation

4. Mock Components
   - LLM response mocking
   - Tool/API simulation
   - User interaction simulation
   - External system simulation
   - Network condition simulation

Use Cases: 1. Safety Testing – Dangerous scenarios without risk – Edge case handling – Failure mode analysis – Recovery testing – Adversarial inputs

2. Performance Testing
   - Load testing (many concurrent requests)
   - Stress testing (beyond capacity)
   - Endurance testing (long duration)
   - Spike testing (sudden load changes)
   - Scalability validation

3. Integration Testing
   - Multi-component interaction
   - External service integration
   - Data flow validation
   - Error propagation
   - State consistency

4. Regression Testing
   - Validate fixes don't break features
   - Ensure consistent behavior
   - Track performance changes
   - Verify improvements
   - Document behavior changes

5. Exploratory Testing
   - Discover edge cases
   - Test rare scenarios
   - Validate assumptions
   - Find unexpected behaviors
   - Stress system boundaries

LangChain Implementation: LangChain supports simulation through: – Mock LLM implementations – Custom callbacks for control – Test chain compositions – Fake embeddings for testing – Memory simulation

Key Features: 1. Controllable Environment – Deterministic responses – Configurable delays – Error injection – State manipulation – Reproducible conditions

2. Comprehensive Scenarios
   - Normal operations
   - Error conditions
   - Edge cases
   - Load patterns
   - Attack scenarios

3. Detailed Observation
   - Action logging
   - State tracking
   - Performance metrics
   - Error recording
   - Decision paths

4. Flexible Validation
   - Multiple success criteria
   - Partial success handling
   - Performance thresholds
   - State assertions
   - Behavior verification

Best Practices: 1. Scenario Design – Start simple, add complexity – Cover happy path first – Add error scenarios – Include edge cases – Test boundaries

2. Mock Fidelity
   - Balance realism vs simplicity
   - Accurate error simulation
   - Realistic timing
   - State consistency
   - Clear documentation

3. Test Independence
   - Each test isolated
   - Clean state between tests
   - No shared dependencies
   - Deterministic execution
   - Parallel-safe

4. Performance
   - Fast execution (< 1s per test)
   - Efficient mocking
   - Minimal overhead
   - Parallel execution
   - Resource cleanup

Trade-offs: Advantages: – Safe testing of dangerous scenarios – Reproducible conditions – Cost-effective (no real API calls) – Fast execution – Comprehensive coverage – Early issue detection

Disadvantages:
- Mock/reality gap
- Maintenance overhead
- Complexity in setup
- May miss real-world issues
- Requires good scenario design
- Can give false confidence

Production Considerations: 1. Test Coverage – Normal operation paths – Common error scenarios – Known edge cases – Performance limits – Security boundaries

2. Realism Balance
   - Accurate enough for confidence
   - Simple enough to maintain
   - Fast enough for CI/CD
   - Flexible for experimentation
   - Close to production behavior

3. Maintenance
   - Update with code changes
   - Sync with production behavior
   - Regular review
   - Refactor as needed
   - Document assumptions

4. Integration with CI/CD
   - Automated test runs
   - Fast feedback loops
   - Quality gates
   - Regression prevention
   - Performance baselines

5. Hybrid Approach
   - Simulation for development
   - Integration tests for validation
   - Shadow testing in production
   - A/B testing for changes
   - Monitoring in production

File Statistics

  • Total Lines: 948
  • Classes: 9
  • Functions: 1
  • Constants: 0

Module Documentation

"""
Pattern 092: Simulation Testing

Description:
    Simulation Testing creates controlled, reproducible environments to test agent behavior
    under various scenarios, conditions, and edge cases. Unlike production testing,
    simulations allow you to test dangerous, expensive, or rare scenarios safely and
    repeatedly. This pattern is essential for validating agent behavior before deployment
    and for stress testing system limits.

    Simulation testing enables:
    - Safe testing of dangerous scenarios
    - Reproducible test conditions
    - Stress and load testing
    - Edge case exploration
    - Multi-agent interaction testing
    - Cost-effective experimentation

Components:
    1. Simulation Environment
       - Mock external services (APIs, databases)
       - Configurable conditions (delays, errors, loads)
       - State management and reset
       - Recording and replay capabilities
       - Deterministic execution

    2. Scenario Definition
       - Initial state setup
       - Event sequences
       - Expected outcomes
       - Success criteria
       - Failure conditions

    3. Test Harness
       - Scenario execution engine
       - Result validation
       - Metrics collection
       - Logging and tracing
       - Report generation

    4. Mock Components
       - LLM response mocking
       - Tool/API simulation
       - User interaction simulation
       - External system simulation
       - Network condition simulation

Use Cases:
    1. Safety Testing
       - Dangerous scenarios without risk
       - Edge case handling
       - Failure mode analysis
       - Recovery testing
       - Adversarial inputs

    2. Performance Testing
       - Load testing (many concurrent requests)
       - Stress testing (beyond capacity)
       - Endurance testing (long duration)
       - Spike testing (sudden load changes)
       - Scalability validation

    3. Integration Testing
       - Multi-component interaction
       - External service integration
       - Data flow validation
       - Error propagation
       - State consistency

    4. Regression Testing
       - Validate fixes don't break features
       - Ensure consistent behavior
       - Track performance changes
       - Verify improvements
       - Document behavior changes

    5. Exploratory Testing
       - Discover edge cases
       - Test rare scenarios
       - Validate assumptions
       - Find unexpected behaviors
       - Stress system boundaries

LangChain Implementation:
    LangChain supports simulation through:
    - Mock LLM implementations
    - Custom callbacks for control
    - Test chain compositions
    - Fake embeddings for testing
    - Memory simulation

Key Features:
    1. Controllable Environment
       - Deterministic responses
       - Configurable delays
       - Error injection
       - State manipulation
       - Reproducible conditions

    2. Comprehensive Scenarios
       - Normal operations
       - Error conditions
       - Edge cases
       - Load patterns
       - Attack scenarios

    3. Detailed Observation
       - Action logging
       - State tracking
       - Performance metrics
       - Error recording
       - Decision paths

    4. Flexible Validation
       - Multiple success criteria
       - Partial success handling
       - Performance thresholds
       - State assertions
       - Behavior verification

Best Practices:
    1. Scenario Design
       - Start simple, add complexity
       - Cover happy path first
       - Add error scenarios
       - Include edge cases
       - Test boundaries

    2. Mock Fidelity
       - Balance realism vs simplicity
       - Accurate error simulation
       - Realistic timing
       - State consistency
       - Clear documentation

    3. Test Independence
       - Each test isolated
       - Clean state between tests
       - No shared dependencies
       - Deterministic execution
       - Parallel-safe

    4. Performance
       - Fast execution (< 1s per test)
       - Efficient mocking
       - Minimal overhead
       - Parallel execution
       - Resource cleanup

Trade-offs:
    Advantages:
    - Safe testing of dangerous scenarios
    - Reproducible conditions
    - Cost-effective (no real API calls)
    - Fast execution
    - Comprehensive coverage
    - Early issue detection

    Disadvantages:
    - Mock/reality gap
    - Maintenance overhead
    - Complexity in setup
    - May miss real-world issues
    - Requires good scenario design
    - Can give false confidence

Production Considerations:
    1. Test Coverage
       - Normal operation paths
       - Common error scenarios
       - Known edge cases
       - Performance limits
       - Security boundaries

    2. Realism Balance
       - Accurate enough for confidence
       - Simple enough to maintain
       - Fast enough for CI/CD
       - Flexible for experimentation
       - Close to production behavior

    3. Maintenance
       - Update with code changes
       - Sync with production behavior
       - Regular review
       - Refactor as needed
       - Document assumptions

    4. Integration with CI/CD
       - Automated test runs
       - Fast feedback loops
       - Quality gates
       - Regression prevention
       - Performance baselines

    5. Hybrid Approach
       - Simulation for development
       - Integration tests for validation
       - Shadow testing in production
       - A/B testing for changes
       - Monitoring in production
"""

Purpose: This module implements the Simulation Testing pattern, providing a structured approach to agent design and implementation.

Import Statements

Standard Library Imports

import dataclasses.dataclass
import dataclasses.field
import datetime.datetime
import enum.Enum
import langchain_core.output_parsers.StrOutputParser
import langchain_core.prompts.ChatPromptTemplate
import os
import random
import time
import typing.Any
import typing.Callable
import typing.Dict
import typing.List
import typing.Optional

Purpose: Core Python functionality: – dataclasses.dataclass: Simplified class definitions with @dataclass – dataclasses.field: Simplified class definitions with @dataclass – datetime.datetime: Date and time handling – enum.Enum: Enumeration support – langchain_core.output_parsers.StrOutputParser: Regular expression operations – langchain_core.prompts.ChatPromptTemplate: Regular expression operations – os: Environment variable access and system operations – random: Random number generation – time: Time-related functions – typing.Any: Type hints for better code documentation (List, Dict, Any, Optional) – typing.Callable: Type hints for better code documentation (List, Dict, Any, Optional) – typing.Dict: Type hints for better code documentation (List, Dict, Any, Optional) – typing.List: Type hints for better code documentation (List, Dict, Any, Optional) – typing.Optional: Type hints for better code documentation (List, Dict, Any, Optional)

Other Imports

from dotenv.load_dotenv

Purpose: Additional dependencies for specialized functionality.

Classes

Class: ScenarioType

Lines 219-225

Inherits from: Enum

Purpose:

Types of test scenarios

Class: OutcomeType

Lines 228-233

Inherits from: Enum

Purpose:

Test outcome types

Class: MockResponse

Lines 237-242

Purpose:

Represents a mock LLM response

Class: SimulationScenario

Lines 246-255

Purpose:

Defines a simulation test scenario

Class: SimulationResult

Lines 259-268

Purpose:

Result of a simulation test

Class: MockLLM

Lines 271-329

Purpose:

Mock LLM for testing that returns pre-configured responses.

Allows testing without making actual API calls.

Methods:

__init__(self, responses)

Lines 278-291

Purpose:

Initialize mock LLM.

Args:
    responses: Dictionary mapping inputs to mock responses

Parameters: – responses: Responses

invoke(self, input)

Lines 293-320

Purpose:

Mock invoke method.

Args:
    input: Input string

Returns:
    Mock response

Parameters: – input: Input data

Returns: str

add_response(self, input, response)

Lines 322-324

Purpose:

Add a response pattern

Parameters: – input: Input data – response: Response

reset(self)

Lines 326-329

Purpose:

Reset mock state

Parameters:

Class: MockTool

Lines 332-377

Purpose:

Mock tool for testing agent tool use.

Simulates external tools without actual execution.

Methods:

__init__(self, name, behavior, should_fail, failure_rate)

Lines 339-359

Purpose:

Initialize mock tool.

Args:
    name: Tool name
    behavior: Function defining tool behavior
    should_fail: Whether tool should always fail
    failure_rate: Probability of failure (0-1)

Parameters: – name: Name – behavior: Behavior – should_fail: Should fail – failure_rate: Failure rate

execute(self, input)

Lines 361-377

Purpose:

Execute tool with mock behavior.

Args:
    input: Tool input

Returns:
    Tool output

Parameters: – input: Input data

Returns: str

Class: SimulationEnvironment

Lines 380-424

Purpose:

Controlled environment for simulation testing.

Provides mock components and state management.

Methods:

__init__(self)

Lines 387-392

Purpose:

Initialize simulation environment

Parameters:

setup_mock_llm(self, responses)

Lines 394-396

Purpose:

Configure mock LLM responses

Parameters: – responses: Responses

add_mock_tool(self, tool)

Lines 398-400

Purpose:

Add a mock tool

Parameters: – tool: Tool

set_state(self, key, value)

Lines 402-404

Purpose:

Set environment state

Parameters: – key: Key – value: Value

get_state(self, key)

Lines 406-408

Purpose:

Get environment state

Parameters: – key: Key

Returns: Any

log_event(self, event_type, details)

Lines 410-416

Purpose:

Log an event

Parameters: – event_type: Event type – details: Details

reset(self)

Lines 418-424

Purpose:

Reset environment to initial state

Parameters:

Class: SimulationTester

Lines 427-558

Purpose:

Executes and validates simulation tests.

Runs scenarios in controlled environment and collects results.

Methods:

__init__(self, environment)

Lines 434-441

Purpose:

Initialize tester.

Args:
    environment: Simulation environment

Parameters: – environment: Environment

run_scenario(self, scenario)

Lines 443-539

Purpose:

Run a simulation scenario.

Args:
    scenario: Scenario to run

Returns:
    Simulation result

Parameters: – scenario: Scenario

Returns: SimulationResult

run_scenarios(self, scenarios)

Lines 541-558

Purpose:

Run multiple scenarios.

Args:
    scenarios: List of scenarios

Returns:
    List of results

Parameters: – scenarios: Scenarios

Returns: <ast.Subscript object at 0x000001A06D60A650>

Functions

demonstrate_simulation_testing()

Lines 561-943

Purpose:

Demonstrate simulation testing

Key Concepts Summary

Pattern Overview

The Simulation Testing pattern provides a structured approach to building AI agents with specific capabilities and behaviors. This implementation leverages LangChain’s composable components to create a robust and maintainable solution.

Core Components

  1. Agent Class: Main implementation of the pattern logic
  2. Prompt Engineering: Carefully crafted prompts to guide LLM behavior
  3. Tool Integration: Optional tools for extended functionality
  4. Memory Management: State handling across interactions
  5. Output Parsing: Structured response processing

When to Use

✅ Good for: – Scenarios requiring this specific pattern’s strengths – Applications needing structured agent behavior – Systems where explainability is important – Production environments requiring reliability

❌ Not ideal for: – Simple single-prompt tasks – Scenarios requiring different pattern characteristics – Resource-constrained environments (if computationally intensive)

Best Practices Demonstrated

1. Environment Configuration

  • ✅ Use .env files for API keys
  • ✅ Never hardcode credentials
  • ✅ Validate environment variables on startup

2. Error Handling

  • ✅ Graceful degradation on failures
  • ✅ Informative error messages
  • ✅ Logging for debugging

3. Prompt Engineering

  • ✅ Clear, specific instructions
  • ✅ Structured output formats
  • ✅ Few-shot examples when beneficial

4. Code Organization

  • ✅ Separation of concerns (class-based design)
  • ✅ Type hints for clarity
  • ✅ Docstrings for documentation

5. Performance Optimization

  • ✅ Appropriate temperature settings
  • ✅ Token budget management
  • ✅ Caching where applicable

LangChain Components Used

  • Core LangChain framework components

Why LangChain? – Composable abstractions for LLM applications – Rich ecosystem of integrations – Production-ready patterns – Active community and development

Execution Flow

1. Initialize agent with configuration
   ↓
2. Load environment and dependencies
   ↓
3. Prepare prompt with input data
   ↓
4. Invoke LLM through LangChain
   ↓
5. Parse and validate response
   ↓
6. Return formatted result

Key Decision Points: – Model selection (temperature, model type) – Prompt structure (few-shot vs zero-shot) – Error handling strategy – Output format validation

Performance Considerations

Computational Cost

  • LLM Calls: Primary cost factor (based on tokens)
  • Latency: Depends on model and input size
  • Memory: Minimal for most patterns

Optimization Strategies

  1. Prompt Optimization: Minimize tokens while maintaining clarity
  2. Caching: Cache repeated queries/responses
  3. Model Selection: Use smaller models when appropriate
  4. Batch Processing: Group similar requests
  5. Streaming: For long responses, stream tokens

Cost Estimates (Approximate)

  • GPT-3.5-turbo: ~$0.002 per request
  • GPT-4: ~$0.03 per request
  • GPT-4-turbo: ~$0.01 per request

Actual costs vary based on input/output token counts

Common Pitfalls to Avoid

1. Prompt Design Issues

❌ Vague or ambiguous instructions
✅ Clear, specific, testable prompts

2. Error Handling

❌ Ignoring API errors or rate limits
✅ Implement retry logic and graceful degradation

3. Token Management

❌ Exceeding model token limits
✅ Truncate inputs, chunk processing

4. Security

❌ Exposing API keys in code
✅ Use environment variables

5. Testing

❌ Only testing happy paths
✅ Test edge cases, failures, malformed inputs

6. Cost Management

❌ Unmonitored API usage
✅ Implement budgets, alerts, and monitoring

Further Improvements

Production Enhancements

  1. Monitoring: Add logging, metrics, tracing
  2. Validation: Input/output schema validation
  3. Rate Limiting: Implement request throttling
  4. Caching: Redis or in-memory cache for responses
  5. Async Support: Non-blocking operations for scalability

Feature Extensions

  1. Multi-Model Support: Switch between providers
  2. Fallback Models: Graceful degradation strategy
  3. Custom Tools: Domain-specific function calling
  4. Advanced Memory: Vector stores for long-term memory
  5. Evaluation: Automated testing and benchmarking

Code Quality

  1. Testing: Unit tests, integration tests
  2. Documentation: API docs, usage examples
  3. Type Safety: Stricter type hints
  4. Configuration: Externalized config files

Related Patterns

  • Pattern 091
  • Pattern 093

Pattern Families: – Reasoning Patterns (001-011) – Multi-Agent Patterns (012-019) – Tool Use Patterns (020-025) – Memory Patterns (026-031) – And many more…

Summary

The Simulation Testing pattern provides a powerful approach to building AI agents with specific capabilities. This implementation demonstrates:

✅ Key Strengths: – Clean, maintainable code structure – LangChain integration for composability – Type hints and documentation – Production-ready patterns

🎯 Best Use Cases: – Applications requiring this pattern’s characteristics – Systems needing structured agent behavior – Production environments requiring reliability

📚 Learning Resources: – LangChain Documentation: https://python.langchain.com/ – Pattern Documentation: See related patterns – OpenAI API Reference: https://platform.openai.com/docs/

File: 092_simulation_testing.py
Total Lines: 948
Pattern Category: AI Agent Design Pattern #092

Generated from actual implementation code
For educational and reference purposes

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts