"""
Pattern 093: A/B Testing Pattern
Description:
A/B Testing (also called split testing) compares two or more variants of an agent,
prompt, model, or configuration to determine which performs better. This pattern is
essential for data-driven optimization of AI systems, enabling evidence-based decisions
about changes rather than relying on intuition. A/B testing helps identify improvements,
validate hypotheses, and continuously optimize agent performance in production.
A/B testing enables:
- Objective comparison of variants
- Statistical significance validation
- Risk mitigation through gradual rollout
- Continuous improvement culture
- Data-driven decision making
- ROI measurement for changes
Components:
1. Variant Definition
- Control (baseline/current version)
- Treatment (new version/s)
- Configuration differences
- Hypothesis statement
- Success metrics
2. Traffic Splitting
- Random assignment
- Consistent user experience
- Traffic allocation ratios
- Segment-based splitting
- Gradual rollout
3. Metrics Collection
- Primary metrics (key objectives)
- Secondary metrics (side effects)
- Guardrail metrics (safety)
- Performance metrics (cost, latency)
- User satisfaction metrics
4. Statistical Analysis
- Sample size calculation
- Significance testing
- Confidence intervals
- Effect size measurement
- Winner determination
Use Cases:
1. Prompt Optimization
- Compare prompt variations
- Test different instructions
- Optimize few-shot examples
- Refine system messages
- Improve output formatting
2. Model Selection
- Compare different models (GPT-4 vs GPT-3.5)
- Test model parameters (temperature, etc.)
- Evaluate fine-tuned models
- Assess cost/performance trade-offs
- Validate model upgrades
3. Feature Testing
- New capabilities evaluation
- Tool addition impact
- Memory strategy comparison
- Retrieval method testing
- Chain architecture optimization
4. Configuration Tuning
- Response length optimization
- Timeout adjustment
- Retry logic tuning
- Cache strategy comparison
- Rate limit optimization
5. User Experience
- Response style testing
- Persona variations
- Output format preferences
- Interaction flow optimization
- Error message improvements
LangChain Implementation:
LangChain supports A/B testing through:
- Multiple chain configurations
- Custom callbacks for tracking
- LangSmith for experiment tracking
- Flexible chain composition
- Variant switching logic
Key Features:
1. Experiment Management
- Multiple variants support
- Traffic allocation control
- Consistent assignment
- Experiment lifecycle
- Result tracking
2. Metrics Framework
- Flexible metric definitions
- Automatic collection
- Real-time monitoring
- Historical comparison
- Statistical analysis
3. Safety Mechanisms
- Guardrail metrics
- Auto-stop on degradation
- Gradual rollout
- Easy rollback
- Error rate monitoring
4. Analysis Tools
- Statistical significance
- Confidence intervals
- Visualization support
- Winner determination
- ROI calculation
Best Practices:
1. Experiment Design
- Clear hypothesis
- Single variable change
- Appropriate metrics
- Sufficient sample size
- Realistic duration
2. Traffic Allocation
- Start small (5-10% treatment)
- Increase gradually
- Monitor continuously
- Keep control stable
- Random assignment
3. Metric Selection
- Primary metric (main goal)
- Secondary metrics (side effects)
- Guardrails (safety limits)
- Leading indicators
- Lagging indicators
4. Analysis Rigor
- Wait for significance
- Check for novelty effects
- Consider practical significance
- Review all metrics
- Document decisions
Trade-offs:
Advantages:
- Objective decision making
- Risk reduction
- Continuous optimization
- Evidence-based improvements
- Stakeholder confidence
- Measurable impact
Disadvantages:
- Requires traffic volume
- Time to statistical significance
- Complexity overhead
- May slow innovation
- Requires discipline
- Can be misinterpreted
Production Considerations:
1. Sample Size
- Calculate minimum required
- Consider baseline metrics
- Factor in expected lift
- Account for variance
- Plan for duration
2. Experiment Duration
- Minimum: 1-2 weeks
- Cover business cycles
- Avoid holidays/events
- Monitor stability
- Check for time effects
3. Monitoring
- Real-time dashboards
- Alert on anomalies
- Track guardrails
- Monitor costs
- Review regularly
4. Decision Criteria
- Statistical significance (p values
def get_mean(self, metric_name: str) -> float:
"""Calculate mean for a metric"""
values = self.metrics.get(metric_name, [])
return sum(values) / len(values) if values else 0.0
def get_std(self, metric_name: str) -> float:
"""Calculate standard deviation for a metric"""
values = self.metrics.get(metric_name, [])
if len(values) Variant:
"""
Select variant for a request.
Args:
user_id: Optional user ID for consistent assignment
Returns:
Selected variant
"""
# Use user_id for consistent assignment if provided
if user_id:
# Simple hash-based assignment for consistency
hash_value = hash(user_id) % 1000 / 1000.0
else:
hash_value = random.random()
# Select variant based on traffic allocation
cumulative = 0.0
for variant in self.variants:
if variant.status != VariantStatus.ACTIVE:
continue
cumulative += variant.traffic_allocation
if hash_value VariantMetrics:
"""Get metrics for a variant"""
return self.variant_metrics.get(variant_id)
def check_guardrails(self) -> Dict[str, bool]:
"""
Check if guardrail metrics are within thresholds.
Returns:
Dictionary of guardrail checks
"""
results = {}
for metric_name, metric in self.metrics.items():
if not metric.is_guardrail:
continue
for variant in self.variants:
if variant.is_control:
continue
variant_metrics = self.variant_metrics[variant.id]
mean_value = variant_metrics.get_mean(metric_name)
if metric.threshold is not None:
if metric.higher_is_better:
passed = mean_value >= metric.threshold
else:
passed = mean_value float:
"""
Calculate t-statistic for two samples.
Returns:
T-statistic
"""
if n1 < 2 or n2 float:
"""
Calculate p-value from t-statistic (simplified).
This is a simplified approximation for demonstration.
Production code should use scipy.stats.
Returns:
P-value
"""
# Simplified: convert to z-score approximation
# For large samples, t-distribution ≈ normal distribution
if df 2.0 else 0.5
else:
# Normal approximation for large samples
return 0.01 if abs(t_stat) > 2.58 else 0.05 if abs(t_stat) > 1.96 else 0.5
def compare_variants(
self,
experiment: ABExperiment,
control_id: str,
treatment_id: str,
metric_name: str
) -> ABTestResult:
"""
Compare two variants on a metric.
Args:
experiment: A/B experiment
control_id: Control variant ID
treatment_id: Treatment variant ID
metric_name: Metric to compare
Returns:
Test result
"""
control_metrics = experiment.get_variant_metrics(control_id)
treatment_metrics = experiment.get_variant_metrics(treatment_id)
control_mean = control_metrics.get_mean(metric_name)
control_std = control_metrics.get_std(metric_name)
control_n = control_metrics.sample_size
treatment_mean = treatment_metrics.get_mean(metric_name)
treatment_std = treatment_metrics.get_std(metric_name)
treatment_n = treatment_metrics.sample_size
# Calculate lift
if control_mean != 0:
lift = ((treatment_mean - control_mean) / control_mean) * 100
else:
lift = 0.0
# Calculate t-statistic
t_stat = self.calculate_t_statistic(
treatment_mean, treatment_std, treatment_n,
control_mean, control_std, control_n
)
# Calculate p-value
df = control_n + treatment_n - 2
p_value = self.calculate_p_value(t_stat, df)
# Determine significance
is_significant = p_value control_mean else control_id
else:
winner = treatment_id if treatment_mean 0 else "↓"
print(f" {result.metric_name}: {direction} {abs(result.lift):.1f}%")
print("\n" + "=" * 80)
print("RECOMMENDATION:")
print("=" * 80)
if primary_result.is_significant and all_passed and primary_result.lift > 0:
print("""
✓ DEPLOY TREATMENT
Justification:
- Primary metric shows significant improvement
- All guardrails passed
- No significant degradation in secondary metrics
- Safe to rollout to 100% of traffic
Next Steps:
1. Increase treatment traffic to 100%
2. Monitor for 1 week
3. Validate sustained improvement
4. Document learnings
""")
else:
print("""
✗ KEEP CONTROL
Justification:
- Treatment did not show sufficient improvement
- OR guardrail metrics were violated
- OR improvement not statistically significant
Next Steps:
1. Analyze why treatment underperformed
2. Iterate on design
3. Run new experiment
4. Consider alternative approaches
""")
# Example 7: Gradual Rollout Simulation
print("\n" + "=" * 80)
print("EXAMPLE 7: Gradual Rollout Strategy")
print("=" * 80)
print("\nRecommended rollout plan:\n")
rollout_stages = [
{"traffic": 5, "duration": "2 days", "action": "Initial validation"},
{"traffic": 10, "duration": "3 days", "action": "Monitor stability"},
{"traffic": 25, "duration": "1 week", "action": "Expand cautiously"},
{"traffic": 50, "duration": "1 week", "action": "Half traffic"},
{"traffic": 100, "duration": "Ongoing", "action": "Full rollout"}
]
for stage in rollout_stages:
print(f"Stage {stage['traffic']}%:")
print(f" Duration: {stage['duration']}")
print(f" Action: {stage['action']}")
print(f" If issues detected: Rollback to previous stage")
print()
# Summary
print("\n" + "=" * 80)
print("A/B TESTING SUMMARY")
print("=" * 80)
print("""
A/B Testing Benefits:
1. Objective Decision Making: Data over intuition
2. Risk Mitigation: Gradual rollout limits exposure
3. Continuous Improvement: Systematic optimization
4. Evidence-Based: Statistical confidence
5. Stakeholder Alignment: Clear success criteria
6. ROI Measurement: Quantifiable impact
Key Components:
1. Variants
- Control: Current/baseline version
- Treatment: New version to test
- Clear differences between variants
- Consistent user experience
2. Metrics
- Primary: Main success metric
- Secondary: Side effect monitoring
- Guardrails: Safety boundaries
- Leading & lagging indicators
3. Traffic Splitting
- Random assignment for unbiased results
- Consistent per-user (same user = same variant)
- Flexible allocation (50/50, 90/10, etc.)
- Gradual rollout capability
4. Statistical Analysis
- Significance testing (p-value < 0.05)
- Effect size (practical significance)
- Confidence intervals
- Sample size validation
Best Practices:
1. Experiment Design
- Clear hypothesis statement
- Single variable change
- Measurable success criteria
- Appropriate sample size
- Realistic timeline
2. Metric Selection
- One primary metric (main goal)
- Multiple secondary metrics
- Guardrail metrics (boundaries)
- Balance short/long term
- Track user sentiment
3. Running Experiments
- Minimum 1-2 weeks duration
- Wait for statistical significance
- Monitor guardrails continuously
- Check for novelty effects
- Consider seasonality
4. Making Decisions
- Require statistical significance (p < 0.05)
- Check practical significance (meaningful lift)
- Validate all guardrails passed
- Review all secondary metrics
- Document reasoning
5. Rollout Strategy
- Start small (5-10% traffic)
- Increase gradually
- Monitor at each stage
- Easy rollback mechanism
- Full rollout only after validation
Common Pitfalls to Avoid:
1. Stopping too early (not enough samples)
2. Multiple testing without correction
3. Ignoring guardrail metrics
4. Changing experiment mid-flight
5. Cherry-picking metrics
6. Insufficient traffic volume
7. Not considering novelty effects
8. Forgetting about opportunity cost
When to Use A/B Testing:
✓ Production systems with traffic
✓ Uncertain about impact
✓ Multiple competing approaches
✓ Optimization opportunities
✓ Risk-averse changes
✓ Stakeholder buy-in needed
✗ Low traffic applications
✗ Obviously correct changes
✗ Emergency fixes
✗ Insufficient time
Sample Size Guidelines:
- Minimum 100 samples per variant
- More for smaller expected differences
- Calculator: n = (Z * σ / E)²
where Z = z-score (1.96 for 95% confidence)
σ = standard deviation
E = margin of error
Statistical Significance:
- Standard: p-value < 0.05 (95% confidence)
- Conservative: p-value < 0.01 (99% confidence)
- Always report effect size, not just p-value
- Consider practical significance
Production Tips:
- Integrate with feature flags
- Real-time monitoring dashboards
- Automated guardrail alerts
- Easy rollback capability
- Experiment registry/documentation
- Post-experiment analysis
- Share learnings across team
- Iterate based on results
""")
print("\n" + "=" * 80)
print("Pattern 093 (A/B Testing) demonstration complete!")
print("=" * 80)
if __name__ == "__main__":
demonstrate_ab_testing()
Agentic Design Pattern: Ab Testing
Pattern 093: Ab Testing – Line-by-Line Explanation
Overview
This document provides a detailed line-by-line explanation of the Ab Testing pattern implementation.
Pattern Description
Pattern 093: A/B Testing Pattern
Description: A/B Testing (also called split testing) compares two or more variants of an agent, prompt, model, or configuration to determine which performs better. This pattern is essential for data-driven optimization of AI systems, enabling evidence-based decisions about changes rather than relying on intuition. A/B testing helps identify improvements, validate hypotheses, and continuously optimize agent performance in production.
A/B testing enables:
- Objective comparison of variants
- Statistical significance validation
- Risk mitigation through gradual rollout
- Continuous improvement culture
- Data-driven decision making
- ROI measurement for changes
Components: 1. Variant Definition – Control (baseline/current version) – Treatment (new version/s) – Configuration differences – Hypothesis statement – Success metrics
2. Traffic Splitting
- Random assignment
- Consistent user experience
- Traffic allocation ratios
- Segment-based splitting
- Gradual rollout
3. Metrics Collection
- Primary metrics (key objectives)
- Secondary metrics (side effects)
- Guardrail metrics (safety)
- Performance metrics (cost, latency)
- User satisfaction metrics
4. Statistical Analysis
- Sample size calculation
- Significance testing
- Confidence intervals
- Effect size measurement
- Winner determination
Use Cases: 1. Prompt Optimization – Compare prompt variations – Test different instructions – Optimize few-shot examples – Refine system messages – Improve output formatting
2. Model Selection
- Compare different models (GPT-4 vs GPT-3.5)
- Test model parameters (temperature, etc.)
- Evaluate fine-tuned models
- Assess cost/performance trade-offs
- Validate model upgrades
3. Feature Testing
- New capabilities evaluation
- Tool addition impact
- Memory strategy comparison
- Retrieval method testing
- Chain architecture optimization
4. Configuration Tuning
- Response length optimization
- Timeout adjustment
- Retry logic tuning
- Cache strategy comparison
- Rate limit optimization
5. User Experience
- Response style testing
- Persona variations
- Output format preferences
- Interaction flow optimization
- Error message improvements
LangChain Implementation: LangChain supports A/B testing through: – Multiple chain configurations – Custom callbacks for tracking – LangSmith for experiment tracking – Flexible chain composition – Variant switching logic
Key Features: 1. Experiment Management – Multiple variants support – Traffic allocation control – Consistent assignment – Experiment lifecycle – Result tracking
2. Metrics Framework
- Flexible metric definitions
- Automatic collection
- Real-time monitoring
- Historical comparison
- Statistical analysis
3. Safety Mechanisms
- Guardrail metrics
- Auto-stop on degradation
- Gradual rollout
- Easy rollback
- Error rate monitoring
4. Analysis Tools
- Statistical significance
- Confidence intervals
- Visualization support
- Winner determination
- ROI calculation
Best Practices: 1. Experiment Design – Clear hypothesis – Single variable change – Appropriate metrics – Sufficient sample size – Realistic duration
2. Traffic Allocation
- Start small (5-10% treatment)
- Increase gradually
- Monitor continuously
- Keep control stable
- Random assignment
3. Metric Selection
- Primary metric (main goal)
- Secondary metrics (side effects)
- Guardrails (safety limits)
- Leading indicators
- Lagging indicators
4. Analysis Rigor
- Wait for significance
- Check for novelty effects
- Consider practical significance
- Review all metrics
- Document decisions
Trade-offs: Advantages: – Objective decision making – Risk reduction – Continuous optimization – Evidence-based improvements – Stakeholder confidence – Measurable impact
Disadvantages:
- Requires traffic volume
- Time to statistical significance
- Complexity overhead
- May slow innovation
- Requires discipline
- Can be misinterpreted
Production Considerations: 1. Sample Size – Calculate minimum required – Consider baseline metrics – Factor in expected lift – Account for variance – Plan for duration
2. Experiment Duration
- Minimum: 1-2 weeks
- Cover business cycles
- Avoid holidays/events
- Monitor stability
- Check for time effects
3. Monitoring
- Real-time dashboards
- Alert on anomalies
- Track guardrails
- Monitor costs
- Review regularly
4. Decision Criteria
- Statistical significance (p < 0.05)
- Practical significance (effect size)
- Cost implications
- All metrics green
- Business alignment
5. Rollout Strategy
- Start with 5-10%
- Monitor for issues
- Increase to 50%
- Full rollout if successful
- Easy rollback plan
File Statistics
- Total Lines: 969
- Classes: 7
- Functions: 1
- Constants: 0
Module Documentation
"""
Pattern 093: A/B Testing Pattern
Description:
A/B Testing (also called split testing) compares two or more variants of an agent,
prompt, model, or configuration to determine which performs better. This pattern is
essential for data-driven optimization of AI systems, enabling evidence-based decisions
about changes rather than relying on intuition. A/B testing helps identify improvements,
validate hypotheses, and continuously optimize agent performance in production.
A/B testing enables:
- Objective comparison of variants
- Statistical significance validation
- Risk mitigation through gradual rollout
- Continuous improvement culture
- Data-driven decision making
- ROI measurement for changes
Components:
1. Variant Definition
- Control (baseline/current version)
- Treatment (new version/s)
- Configuration differences
- Hypothesis statement
- Success metrics
2. Traffic Splitting
- Random assignment
- Consistent user experience
- Traffic allocation ratios
- Segment-based splitting
- Gradual rollout
3. Metrics Collection
- Primary metrics (key objectives)
- Secondary metrics (side effects)
- Guardrail metrics (safety)
- Performance metrics (cost, latency)
- User satisfaction metrics
4. Statistical Analysis
- Sample size calculation
- Significance testing
- Confidence intervals
- Effect size measurement
- Winner determination
Use Cases:
1. Prompt Optimization
- Compare prompt variations
- Test different instructions
- Optimize few-shot examples
- Refine system messages
- Improve output formatting
2. Model Selection
- Compare different models (GPT-4 vs GPT-3.5)
- Test model parameters (temperature, etc.)
- Evaluate fine-tuned models
- Assess cost/performance trade-offs
- Validate model upgrades
3. Feature Testing
- New capabilities evaluation
- Tool addition impact
- Memory strategy comparison
- Retrieval method testing
- Chain architecture optimization
4. Configuration Tuning
- Response length optimization
- Timeout adjustment
- Retry logic tuning
- Cache strategy comparison
- Rate limit optimization
5. User Experience
- Response style testing
- Persona variations
- Output format preferences
- Interaction flow optimization
- Error message improvements
LangChain Implementation:
LangChain supports A/B testing through:
- Multiple chain configurations
- Custom callbacks for tracking
- LangSmith for experiment tracking
- Flexible chain composition
- Variant switching logic
Key Features:
1. Experiment Management
- Multiple variants support
- Traffic allocation control
- Consistent assignment
- Experiment lifecycle
- Result tracking
2. Metrics Framework
- Flexible metric definitions
- Automatic collection
- Real-time monitoring
- Historical comparison
- Statistical analysis
3. Safety Mechanisms
- Guardrail metrics
- Auto-stop on degradation
- Gradual rollout
- Easy rollback
- Error rate monitoring
4. Analysis Tools
- Statistical significance
- Confidence intervals
- Visualization support
- Winner determination
- ROI calculation
Best Practices:
1. Experiment Design
- Clear hypothesis
- Single variable change
- Appropriate metrics
- Sufficient sample size
- Realistic duration
2. Traffic Allocation
- Start small (5-10% treatment)
- Increase gradually
- Monitor continuously
- Keep control stable
- Random assignment
3. Metric Selection
- Primary metric (main goal)
- Secondary metrics (side effects)
- Guardrails (safety limits)
- Leading indicators
- Lagging indicators
4. Analysis Rigor
- Wait for significance
- Check for novelty effects
- Consider practical significance
- Review all metrics
- Document decisions
Trade-offs:
Advantages:
- Objective decision making
- Risk reduction
- Continuous optimization
- Evidence-based improvements
- Stakeholder confidence
- Measurable impact
Disadvantages:
- Requires traffic volume
- Time to statistical significance
- Complexity overhead
- May slow innovation
- Requires discipline
- Can be misinterpreted
Production Considerations:
1. Sample Size
- Calculate minimum required
- Consider baseline metrics
- Factor in expected lift
- Account for variance
- Plan for duration
2. Experiment Duration
- Minimum: 1-2 weeks
- Cover business cycles
- Avoid holidays/events
- Monitor stability
- Check for time effects
3. Monitoring
- Real-time dashboards
- Alert on anomalies
- Track guardrails
- Monitor costs
- Review regularly
4. Decision Criteria
- Statistical significance (p < 0.05)
- Practical significance (effect size)
- Cost implications
- All metrics green
- Business alignment
5. Rollout Strategy
- Start with 5-10%
- Monitor for issues
- Increase to 50%
- Full rollout if successful
- Easy rollback plan
"""
Purpose: This module implements the Ab Testing pattern, providing a structured approach to agent design and implementation.
Import Statements
Standard Library Imports
import dataclasses.dataclass
import dataclasses.field
import datetime.datetime
import enum.Enum
import langchain_core.output_parsers.StrOutputParser
import langchain_core.prompts.ChatPromptTemplate
import os
import random
import time
import typing.Any
import typing.Callable
import typing.Dict
import typing.List
import typing.Optional
Purpose: Core Python functionality:
– dataclasses.dataclass: Simplified class definitions with @dataclass
– dataclasses.field: Simplified class definitions with @dataclass
– datetime.datetime: Date and time handling
– enum.Enum: Enumeration support
– langchain_core.output_parsers.StrOutputParser: Regular expression operations
– langchain_core.prompts.ChatPromptTemplate: Regular expression operations
– os: Environment variable access and system operations
– random: Random number generation
– time: Time-related functions
– typing.Any: Type hints for better code documentation (List, Dict, Any, Optional)
– typing.Callable: Type hints for better code documentation (List, Dict, Any, Optional)
– typing.Dict: Type hints for better code documentation (List, Dict, Any, Optional)
– typing.List: Type hints for better code documentation (List, Dict, Any, Optional)
– typing.Optional: Type hints for better code documentation (List, Dict, Any, Optional)
LangChain Imports
from langchain_openai.ChatOpenAI
Purpose: LangChain framework components: – LLM Interface: Wrapper for OpenAI models – Chains: Composable building blocks for complex workflows
Other Imports
from dotenv.load_dotenv
from math
Purpose: Additional dependencies for specialized functionality.
Classes
Class: VariantStatus
Lines 221-226
Inherits from: Enum
Purpose:
Variant status in experiment
Class: Variant
Lines 230-238
Purpose:
Represents an experiment variant
Class: Metric
Lines 242-249
Purpose:
Represents a metric to track
Class: VariantMetrics
Lines 253-272
Purpose:
Metrics for a variant
Methods:
get_mean(self, metric_name)
Lines 259-262
Purpose:
Calculate mean for a metric
Parameters:
– metric_name: Metric name
Returns: float
get_std(self, metric_name)
Lines 264-272
Purpose:
Calculate standard deviation for a metric
Parameters:
– metric_name: Metric name
Returns: float
Class: ABTestResult
Lines 276-288
Purpose:
Result of A/B test analysis
Class: ABExperiment
Lines 291-418
Purpose:
A/B test experiment manager.
Manages variants, traffic allocation, and metrics collection.
Methods:
__init__(self, experiment_id, name, description, metrics)
Lines 298-320
Purpose:
Initialize A/B experiment.
Args:
experiment_id: Unique experiment identifier
name: Experiment name
description: Experiment description
metrics: Metrics to track
Parameters:
– experiment_id: Experiment id
– name: Name
– description: Description
– metrics: Metrics
add_variant(self, variant)
Lines 322-334
Purpose:
Add a variant to the experiment.
Args:
variant: Variant to add
Parameters:
– variant: Variant
select_variant(self, user_id)
Lines 336-364
Purpose:
Select variant for a request.
Args:
user_id: Optional user ID for consistent assignment
Returns:
Selected variant
Parameters:
– user_id: User id
Returns: Variant
record_metric(self, variant_id, metric_name, value)
Lines 366-384
Purpose:
Record a metric value for a variant.
Args:
variant_id: Variant identifier
metric_name: Metric name
value: Metric value
Parameters:
– variant_id: Variant id
– metric_name: Metric name
– value: Value
get_variant_metrics(self, variant_id)
Lines 386-388
Purpose:
Get metrics for a variant
Parameters:
– variant_id: Variant id
Returns: VariantMetrics
check_guardrails(self)
Lines 390-418
Purpose:
Check if guardrail metrics are within thresholds.
Returns:
Dictionary of guardrail checks
Parameters:
Returns: <ast.Subscript object at 0x000001A06D7BA250>
Class: ABTestAnalyzer
Lines 421-555
Purpose:
Analyzes A/B test results and determines winners.
Performs statistical tests and calculates significance.
Methods:
__init__(self, confidence_level)
Lines 428-436
Purpose:
Initialize analyzer.
Args:
confidence_level: Confidence level for significance (default 95%)
Parameters:
– confidence_level: Confidence level
calculate_t_statistic(self, mean1, std1, n1, mean2, std2, n2)
Lines 438-463
Purpose:
Calculate t-statistic for two samples.
Returns:
T-statistic
Parameters:
– mean1: Mean1
– std1: Std1
– n1: N1
– mean2: Mean2
– std2: Std2
– n2: N2
Returns: float
calculate_p_value(self, t_stat, df)
Lines 465-482
Purpose:
Calculate p-value from t-statistic (simplified).
This is a simplified approximation for demonstration.
Production code should use scipy.stats.
Returns:
P-value
Parameters:
– t_stat: T stat
– df: Df
Returns: float
compare_variants(self, experiment, control_id, treatment_id, metric_name)
Lines 484-555
Purpose:
Compare two variants on a metric.
Args:
experiment: A/B experiment
control_id: Control variant ID
treatment_id: Treatment variant ID
metric_name: Metric to compare
Returns:
Test result
Parameters:
– experiment: Experiment
– control_id: Control id
– treatment_id: Treatment id
– metric_name: Metric name
Returns: ABTestResult
Functions
demonstrate_ab_testing()
Lines 558-964
Purpose:
Demonstrate A/B testing pattern
Key Concepts Summary
Pattern Overview
The Ab Testing pattern provides a structured approach to building AI agents with specific capabilities and behaviors. This implementation leverages LangChain’s composable components to create a robust and maintainable solution.
Core Components
- Agent Class: Main implementation of the pattern logic
- Prompt Engineering: Carefully crafted prompts to guide LLM behavior
- Tool Integration: Optional tools for extended functionality
- Memory Management: State handling across interactions
- Output Parsing: Structured response processing
When to Use
✅ Good for: – Scenarios requiring this specific pattern’s strengths – Applications needing structured agent behavior – Systems where explainability is important – Production environments requiring reliability
❌ Not ideal for: – Simple single-prompt tasks – Scenarios requiring different pattern characteristics – Resource-constrained environments (if computationally intensive)
Best Practices Demonstrated
1. Environment Configuration
- ✅ Use
.envfiles for API keys - ✅ Never hardcode credentials
- ✅ Validate environment variables on startup
2. Error Handling
- ✅ Graceful degradation on failures
- ✅ Informative error messages
- ✅ Logging for debugging
3. Prompt Engineering
- ✅ Clear, specific instructions
- ✅ Structured output formats
- ✅ Few-shot examples when beneficial
4. Code Organization
- ✅ Separation of concerns (class-based design)
- ✅ Type hints for clarity
- ✅ Docstrings for documentation
5. Performance Optimization
- ✅ Appropriate temperature settings
- ✅ Token budget management
- ✅ Caching where applicable
LangChain Components Used
- ChatOpenAI: OpenAI model interface
- Chains: Sequential component composition
Why LangChain? – Composable abstractions for LLM applications – Rich ecosystem of integrations – Production-ready patterns – Active community and development
Execution Flow
1. Initialize agent with configuration
↓
2. Load environment and dependencies
↓
3. Prepare prompt with input data
↓
4. Invoke LLM through LangChain
↓
5. Parse and validate response
↓
6. Return formatted result
Key Decision Points: – Model selection (temperature, model type) – Prompt structure (few-shot vs zero-shot) – Error handling strategy – Output format validation
Performance Considerations
Computational Cost
- LLM Calls: Primary cost factor (based on tokens)
- Latency: Depends on model and input size
- Memory: Minimal for most patterns
Optimization Strategies
- Prompt Optimization: Minimize tokens while maintaining clarity
- Caching: Cache repeated queries/responses
- Model Selection: Use smaller models when appropriate
- Batch Processing: Group similar requests
- Streaming: For long responses, stream tokens
Cost Estimates (Approximate)
- GPT-3.5-turbo: ~$0.002 per request
- GPT-4: ~$0.03 per request
- GPT-4-turbo: ~$0.01 per request
Actual costs vary based on input/output token counts
Common Pitfalls to Avoid
1. Prompt Design Issues
❌ Vague or ambiguous instructions
✅ Clear, specific, testable prompts
2. Error Handling
❌ Ignoring API errors or rate limits
✅ Implement retry logic and graceful degradation
3. Token Management
❌ Exceeding model token limits
✅ Truncate inputs, chunk processing
4. Security
❌ Exposing API keys in code
✅ Use environment variables
5. Testing
❌ Only testing happy paths
✅ Test edge cases, failures, malformed inputs
6. Cost Management
❌ Unmonitored API usage
✅ Implement budgets, alerts, and monitoring
Further Improvements
Production Enhancements
- Monitoring: Add logging, metrics, tracing
- Validation: Input/output schema validation
- Rate Limiting: Implement request throttling
- Caching: Redis or in-memory cache for responses
- Async Support: Non-blocking operations for scalability
Feature Extensions
- Multi-Model Support: Switch between providers
- Fallback Models: Graceful degradation strategy
- Custom Tools: Domain-specific function calling
- Advanced Memory: Vector stores for long-term memory
- Evaluation: Automated testing and benchmarking
Code Quality
- Testing: Unit tests, integration tests
- Documentation: API docs, usage examples
- Type Safety: Stricter type hints
- Configuration: Externalized config files
Related Patterns
- Pattern 092
- Pattern 094
Pattern Families: – Reasoning Patterns (001-011) – Multi-Agent Patterns (012-019) – Tool Use Patterns (020-025) – Memory Patterns (026-031) – And many more…
Summary
The Ab Testing pattern provides a powerful approach to building AI agents with specific capabilities. This implementation demonstrates:
✅ Key Strengths: – Clean, maintainable code structure – LangChain integration for composability – Type hints and documentation – Production-ready patterns
🎯 Best Use Cases: – Applications requiring this pattern’s characteristics – Systems needing structured agent behavior – Production environments requiring reliability
📚 Learning Resources: – LangChain Documentation: https://python.langchain.com/ – Pattern Documentation: See related patterns – OpenAI API Reference: https://platform.openai.com/docs/
File: 093_ab_testing.py
Total Lines: 969
Pattern Category: AI Agent Design Pattern #093
Generated from actual implementation code
For educational and reference purposes
Leave a Reply