When to Use Manual Tracking
Choose manual tracking when you need to:- Custom Integration: Work with LLM providers not yet supported by automatic wrappers
- Granular Control: Track specific events or add custom metadata
- Complex Workflows: Monitor multi-step processes with custom logic
- Legacy Systems: Integrate with existing agent implementations
- Custom Events: Track domain-specific events beyond standard LLM interactions
Core Tracking Methods
Conversation Lifecycle
Track the complete lifecycle of agent conversations:Copy
import { AgentMonitor } from '@agent-governance/node';
const monitor = new AgentMonitor({
apiKey: 'your-api-key',
organizationId: 'your-org-id',
enableComplianceChecks: true
});
const agentId = 'customer-service-agent';
const sessionId = 'session-' + Date.now();
// 1. Start conversation
monitor.trackConversationStart(agentId, sessionId, 'customer-123');
// 2. Track user input
monitor.trackUserMessage(agentId, sessionId, 'I need help with my account', 'customer-123');
// 3. Track agent response
monitor.trackAgentResponse(agentId, sessionId, 'I can help you with your account. What specific assistance do you need?');
// 4. End conversation
monitor.trackConversationEnd(agentId, sessionId, {
duration: 45000, // 45 seconds
messageCount: 4,
userSatisfaction: 8, // 1-10 scale
resolutionStatus: 'resolved'
});
Tool and Function Calls
Track when your agent uses tools or external functions:Copy
// Track tool call and result together
monitor.trackToolCall(
agentId,
sessionId,
'get_account_balance',
{ accountId: 'acc_123', accountType: 'checking' }, // Parameters
{ balance: 2547.83, currency: 'USD', lastUpdated: '2024-01-15' }, // Result
320 // Execution time in milliseconds
);
// Or track them separately for more control
monitor.track(agentId, {
sessionId: sessionId,
interactionType: 'tool_call',
toolName: 'transfer_funds',
toolParameters: {
fromAccount: 'acc_123',
toAccount: 'acc_456',
amount: 500.00
}
});
// Track the result separately
monitor.track(agentId, {
sessionId: sessionId,
interactionType: 'tool_result',
toolName: 'transfer_funds',
toolResult: {
transactionId: 'txn_789',
status: 'completed',
confirmationNumber: 'CF123456'
},
metadata: {
toolExecutionTime: 1250
}
});
Error Tracking
Comprehensive error tracking with context:Copy
// Simple error tracking
monitor.trackError(agentId, sessionId, 'Database connection failed');
// Detailed error with metadata
monitor.trackError(agentId, sessionId, new Error('API timeout'), {
errorType: 'APITimeoutError',
errorCode: 'TIMEOUT_001',
severity: 'high',
recoverable: true,
userImpact: 'moderate',
systemImpact: 'minimal',
errorSource: 'external'
});
// Using the generic track method for full control
monitor.track(agentId, {
sessionId: sessionId,
interactionType: 'error',
content: 'Failed to process payment request',
errorType: 'PaymentProcessingError',
errorCode: 'PAY_ERR_001',
severity: 'critical',
metadata: {
recoverable: false,
userImpact: 'high',
systemImpact: 'moderate',
errorSource: 'server',
originalError: {
message: 'Payment gateway unreachable',
code: 502,
timestamp: Date.now()
}
}
});
Advanced Tracking Patterns
Banking Workflow Example
Here’s a comprehensive example tracking a complete banking interaction:Copy
async function trackBankingWorkflow(monitor) {
const agentId = 'personal-banking-agent';
const sessionId = `banking-${Date.now()}`;
const userId = 'customer-456';
try {
// Start the conversation
monitor.trackConversationStart(agentId, sessionId, userId, {
userAgent: 'Mozilla/5.0...',
ipAddress: '192.168.1.100',
referrer: 'https://mybank.com/dashboard'
});
// User asks about balance
monitor.trackUserMessage(
agentId,
sessionId,
'Can you show me my checking account balance?',
userId
);
// Agent decides to check account
monitor.trackToolCall(
agentId,
sessionId,
'get_account_summary',
{ accountType: 'checking', includeRecentTransactions: true },
{
balance: 3247.89,
currency: 'USD',
recentTransactions: [
{ date: '2024-01-14', amount: -45.67, description: 'Grocery Store' },
{ date: '2024-01-13', amount: 2500.00, description: 'Payroll Deposit' }
]
},
450 // Execution time
);
// Agent responds with balance information
monitor.trackAgentResponse(
agentId,
sessionId,
'Your checking account balance is $3,247.89. I can see your most recent transactions include a payroll deposit and a grocery purchase.',
{
llmLatency: 850,
tokensUsed: { input: 125, output: 67, total: 192 },
responseQuality: 92,
cost: 0.0034
}
);
// User asks about suspicious transaction
monitor.trackUserMessage(
agentId,
sessionId,
'I see a charge I don\'t recognize from last week. Can you help me check if it\'s fraudulent?',
userId
);
// Agent runs fraud check
monitor.trackToolCall(
agentId,
sessionId,
'check_transaction_fraud',
{
accountId: 'acc_123',
lookbackDays: 7,
flagSuspicious: true
},
{
suspiciousTransactions: [
{
id: 'txn_suspicious_001',
amount: 127.45,
merchant: 'Unknown Merchant LLC',
location: 'Los Angeles, CA',
riskScore: 75
}
],
recommendations: ['contact_customer', 'freeze_card']
},
1200
);
// Agent provides fraud analysis
monitor.trackAgentResponse(
agentId,
sessionId,
'I found one potentially suspicious transaction for $127.45 from "Unknown Merchant LLC" in Los Angeles. This has a risk score of 75/100. I recommend we freeze your card immediately and investigate this charge.',
{
llmLatency: 920,
tokensUsed: { input: 245, output: 89, total: 334 },
responseQuality: 95,
flagged: true, // High-risk response
requiresReview: true
}
);
// End conversation successfully
monitor.trackConversationEnd(agentId, sessionId, {
duration: 180000, // 3 minutes
messageCount: 6,
userSatisfaction: 9,
resolutionStatus: 'escalated',
endReason: 'user_initiated',
followUpRequired: true,
escalationReason: 'potential_fraud_detected'
});
} catch (error) {
// Track any errors that occur
monitor.trackError(agentId, sessionId, error, {
errorType: 'WorkflowError',
severity: 'high',
recoverable: false,
userImpact: 'high'
});
}
}
Custom Event Types
Track domain-specific events using the generictrack
method:
Copy
// Track compliance review trigger
monitor.track(agentId, {
sessionId: sessionId,
interactionType: 'compliance_violation',
metadata: {
violationType: 'fair_lending',
severity: 'warning',
description: 'Potential discriminatory language detected',
ruleId: 'fair_lending_001',
triggeredContent: 'people like you typically...',
recommendedActions: ['human_review', 'response_revision'],
requiresImmediateAction: false
}
});
// Track risk threshold exceeded
monitor.track(agentId, {
sessionId: sessionId,
interactionType: 'risk_threshold_exceeded',
metadata: {
riskScore: 85,
threshold: 75,
riskFactors: [
{ factor: 'suspicious_transaction_inquiry', weight: 40 },
{ factor: 'high_value_account_access', weight: 30 },
{ factor: 'unusual_time_of_day', weight: 15 }
],
recommendedActions: ['escalate_to_human', 'additional_verification']
}
});
Metadata and Context
Enhance your tracking with rich metadata:Performance Metrics
Copy
monitor.trackAgentResponse(agentId, sessionId, responseText, {
// LLM Performance
llmLatency: 1250,
tokensUsed: {
input: 156,
output: 89,
total: 245
},
cost: 0.0067,
temperature: 0.7,
maxTokens: 500,
// Quality Metrics
responseQuality: 88,
relevanceScore: 92,
coherenceScore: 85,
helpfulnessScore: 90,
// Compliance and Risk
complianceFlags: [
{
rule: 'pii_detection',
severity: 'info',
description: 'Account number mentioned in context',
context: { detected: 'account ending in 1234' }
}
],
riskScore: 25,
requiresReview: false
});
Business Context
Copy
monitor.trackUserMessage(agentId, sessionId, userMessage, userId, {
// Customer Context
customerTier: 'premium',
accountAge: 1825, // days
totalAccountValue: 125000,
// Interaction Context
messageId: 'msg_unique_123',
threadId: 'thread_abc',
channel: 'web_chat',
// Technical Context
userAgent: 'Mozilla/5.0...',
ipAddress: '192.168.1.100',
sessionDuration: 45000,
// Business Context
previousIssues: 2,
satisfactionHistory: [8, 9, 7, 9],
preferredLanguage: 'en-US'
});
Batch Operations and Performance
Efficient Batch Tracking
For high-volume applications, use efficient batching:Copy
// Configure for high-volume scenarios
const monitor = new AgentMonitor({
apiKey: 'your-api-key',
organizationId: 'your-org-id',
batchSize: 500, // Larger batches for efficiency
flushInterval: 10000, // 10 second intervals
enableLogging: false // Disable verbose logging
});
// Track multiple events efficiently
const events = [
{
sessionId: 'session-1',
interactionType: 'user_message',
content: 'Hello'
},
{
sessionId: 'session-1',
interactionType: 'agent_response',
content: 'Hi there!'
},
{
sessionId: 'session-2',
interactionType: 'user_message',
content: 'Help me'
}
];
// Track all events
events.forEach(event => monitor.track(agentId, event));
// Force flush when needed
await monitor.flush();
Conditional Tracking
Track selectively based on business logic:Copy
function trackInteractionConditionally(monitor, agentId, interaction) {
const { sessionId, userMessage, agentResponse, riskScore } = interaction;
// Always track high-risk interactions
if (riskScore > 70) {
monitor.trackUserMessage(agentId, sessionId, userMessage);
monitor.trackAgentResponse(agentId, sessionId, agentResponse, {
riskScore,
flagged: true,
requiresReview: true
});
return;
}
// Sample normal interactions (10% sampling)
if (Math.random() < 0.1) {
monitor.trackUserMessage(agentId, sessionId, userMessage);
monitor.trackAgentResponse(agentId, sessionId, agentResponse, { sampled: true });
}
// Always track errors and tool calls
if (interaction.hasError) {
monitor.trackError(agentId, sessionId, interaction.error);
}
if (interaction.toolCalls?.length > 0) {
interaction.toolCalls.forEach(tool => {
monitor.trackToolCall(agentId, sessionId, tool.name, tool.params, tool.result);
});
}
}
Integration Patterns
Express.js Middleware
Create middleware for automatic tracking in web applications:Copy
function createAgentTrackingMiddleware(monitor) {
return (req, res, next) => {
const sessionId = req.sessionID || `session-${Date.now()}`;
const userId = req.user?.id;
// Add tracking utilities to request
req.agentTracking = {
trackUserMessage: (agentId, message) => {
monitor.trackUserMessage(agentId, sessionId, message, userId, {
ipAddress: req.ip,
userAgent: req.get('User-Agent'),
referer: req.get('Referer')
});
},
trackAgentResponse: (agentId, response, metadata = {}) => {
monitor.trackAgentResponse(agentId, sessionId, response, {
...metadata,
endpoint: req.path,
method: req.method
});
},
trackError: (agentId, error) => {
monitor.trackError(agentId, sessionId, error, {
endpoint: req.path,
method: req.method,
statusCode: res.statusCode
});
}
};
next();
};
}
// Use in Express app
app.use(createAgentTrackingMiddleware(monitor));
app.post('/api/chat', (req, res) => {
const { message } = req.body;
// Track user message
req.agentTracking.trackUserMessage('web-agent', message);
// Process message and generate response
const response = processMessage(message);
// Track agent response
req.agentTracking.trackAgentResponse('web-agent', response);
res.json({ response });
});
Custom Agent Wrapper
Create a custom wrapper for your agent class:Copy
class MonitoredAgent {
constructor(agentImplementation, monitor, agentId) {
this.agent = agentImplementation;
this.monitor = monitor;
this.agentId = agentId;
}
async handleMessage(sessionId, userMessage, userId = null) {
const startTime = Date.now();
try {
// Track conversation start if needed
if (!this.activeSessions?.has(sessionId)) {
this.monitor.trackConversationStart(this.agentId, sessionId, userId);
this.activeSessions = this.activeSessions || new Set();
this.activeSessions.add(sessionId);
}
// Track user message
this.monitor.trackUserMessage(this.agentId, sessionId, userMessage, userId);
// Process with underlying agent
const result = await this.agent.processMessage(userMessage, { sessionId, userId });
// Track any tool calls
if (result.toolCalls) {
for (const tool of result.toolCalls) {
this.monitor.trackToolCall(
this.agentId,
sessionId,
tool.name,
tool.parameters,
tool.result,
tool.executionTime
);
}
}
// Track agent response
this.monitor.trackAgentResponse(this.agentId, sessionId, result.response, {
llmLatency: Date.now() - startTime,
tokensUsed: result.tokenUsage,
responseQuality: result.qualityScore,
cost: result.cost
});
return result;
} catch (error) {
// Track error
this.monitor.trackError(this.agentId, sessionId, error, {
errorType: error.constructor.name,
severity: 'high',
recoverable: error.recoverable || false
});
throw error;
}
}
async endSession(sessionId, metadata = {}) {
this.monitor.trackConversationEnd(this.agentId, sessionId, metadata);
this.activeSessions?.delete(sessionId);
}
}
// Usage
const monitoredAgent = new MonitoredAgent(myAgentImplementation, monitor, 'my-agent');
const response = await monitoredAgent.handleMessage('session-123', 'Hello!', 'user-456');
Compliance Integration
Manual tracking allows for sophisticated compliance monitoring:Real-time Compliance Checks
Copy
async function trackWithCompliance(monitor, agentId, sessionId, userMessage, agentResponse) {
// Track user message first
monitor.trackUserMessage(agentId, sessionId, userMessage);
// Pre-response compliance check
const complianceResult = await monitor.complianceEngine?.evaluateInteraction({
agentId,
agentCategory: 'tool_calling',
agentSpecialty: 'personal_banking',
userMessage,
agentResponse,
sessionId,
timestamp: Date.now()
});
// Track response with compliance data
monitor.trackAgentResponse(agentId, sessionId, agentResponse, {
complianceFlags: complianceResult?.violations || [],
riskScore: complianceResult?.riskScore || 0,
requiresReview: complianceResult?.requiresReview || false
});
// If compliance violations detected, track separately
if (complianceResult?.violations?.length > 0) {
for (const violation of complianceResult.violations) {
monitor.track(agentId, {
sessionId,
interactionType: 'compliance_violation',
metadata: {
violationType: violation.rule,
severity: violation.severity,
description: violation.description,
context: violation.context,
recommendation: violation.recommendation
}
});
}
}
}
Custom Compliance Rules
Copy
// Add custom compliance rule
if (monitor.complianceEngine) {
monitor.complianceEngine.addRule({
id: 'banking-hours-check',
name: 'Banking Hours Compliance',
description: 'Ensures agents mention banking hours appropriately',
category: 'consumer_protection',
severity: 'info',
isActive: true,
ruleFunction: (context) => {
const violations = [];
const response = context.agentResponse || '';
// Check if banking hours are mentioned without proper context
if (response.includes('banking hours') && !response.includes('Monday through Friday')) {
violations.push({
rule: 'banking-hours-check',
severity: 'info',
description: 'Banking hours mentioned without full schedule',
recommendation: 'Include complete banking hours schedule'
});
}
return {
isCompliant: violations.length === 0,
violations,
riskScore: violations.length * 5,
requiresReview: false
};
}
});
}
Analytics and Reporting
Track custom metrics for business intelligence:Custom Analytics Events
Copy
// Track business-specific metrics
function trackBusinessMetrics(monitor, agentId, sessionId, interaction) {
const { customerSegment, productInterest, conversionEvent } = interaction;
// Track customer segment interaction
monitor.track(agentId, {
sessionId,
interactionType: 'custom',
metadata: {
eventType: 'customer_segment_interaction',
segment: customerSegment,
products: productInterest,
timestamp: Date.now()
}
});
// Track conversion events
if (conversionEvent) {
monitor.track(agentId, {
sessionId,
interactionType: 'custom',
metadata: {
eventType: 'conversion',
conversionType: conversionEvent.type,
value: conversionEvent.value,
currency: conversionEvent.currency
}
});
}
}
// Usage in banking context
trackBusinessMetrics(monitor, 'loan-agent', 'session-123', {
customerSegment: 'high_value',
productInterest: ['mortgage', 'investment'],
conversionEvent: {
type: 'mortgage_application_started',
value: 450000,
currency: 'USD'
}
});
Session Analytics
Copy
class SessionAnalytics {
constructor(monitor, agentId) {
this.monitor = monitor;
this.agentId = agentId;
this.sessions = new Map();
}
startSession(sessionId, userId = null) {
this.sessions.set(sessionId, {
startTime: Date.now(),
messageCount: 0,
toolCallCount: 0,
errorCount: 0,
userId,
events: []
});
this.monitor.trackConversationStart(this.agentId, sessionId, userId);
}
trackMessage(sessionId, type, content, metadata = {}) {
const session = this.sessions.get(sessionId);
if (!session) return;
session.messageCount++;
session.events.push({
type,
timestamp: Date.now(),
content: content.substring(0, 100) // Store preview
});
if (type === 'user_message') {
this.monitor.trackUserMessage(this.agentId, sessionId, content, session.userId, metadata);
} else if (type === 'agent_response') {
this.monitor.trackAgentResponse(this.agentId, sessionId, content, metadata);
}
}
trackToolCall(sessionId, toolName, params, result, executionTime) {
const session = this.sessions.get(sessionId);
if (session) session.toolCallCount++;
this.monitor.trackToolCall(this.agentId, sessionId, toolName, params, result, executionTime);
}
trackError(sessionId, error) {
const session = this.sessions.get(sessionId);
if (session) session.errorCount++;
this.monitor.trackError(this.agentId, sessionId, error);
}
endSession(sessionId, userSatisfaction = null) {
const session = this.sessions.get(sessionId);
if (!session) return;
const duration = Date.now() - session.startTime;
const resolutionStatus = session.errorCount > 0 ? 'unresolved' : 'resolved';
this.monitor.trackConversationEnd(this.agentId, sessionId, {
duration,
messageCount: session.messageCount,
toolCallCount: session.toolCallCount,
errorCount: session.errorCount,
userSatisfaction,
resolutionStatus,
avgResponseTime: duration / Math.max(session.messageCount, 1)
});
this.sessions.delete(sessionId);
}
}
// Usage
const analytics = new SessionAnalytics(monitor, 'customer-service-agent');
analytics.startSession('session-456', 'customer-123');
analytics.trackMessage('session-456', 'user_message', 'I need help');
analytics.trackMessage('session-456', 'agent_response', 'How can I assist you?');
analytics.endSession('session-456', 8);
Error Handling and Resilience
Implement robust error handling for tracking operations:Resilient Tracking Wrapper
Copy
class ResilientTracker {
constructor(monitor, options = {}) {
this.monitor = monitor;
this.options = {
maxRetries: 3,
retryDelay: 1000,
fallbackToLocalStorage: true,
...options
};
this.failedEvents = [];
}
async safeTrack(agentId, event, retries = 0) {
try {
this.monitor.track(agentId, event);
// Process any previously failed events
if (this.failedEvents.length > 0) {
await this.retryFailedEvents();
}
} catch (error) {
console.warn('Tracking failed:', error.message);
if (retries < this.options.maxRetries) {
setTimeout(() => {
this.safeTrack(agentId, event, retries + 1);
}, this.options.retryDelay * Math.pow(2, retries));
} else {
// Store for later retry
this.failedEvents.push({ agentId, event, timestamp: Date.now() });
if (this.options.fallbackToLocalStorage && typeof localStorage !== 'undefined') {
this.saveToLocalStorage();
}
}
}
}
async retryFailedEvents() {
const eventsToRetry = [...this.failedEvents];
this.failedEvents = [];
for (const { agentId, event } of eventsToRetry) {
try {
this.monitor.track(agentId, event);
} catch (error) {
// Put back in failed events
this.failedEvents.push({ agentId, event, timestamp: Date.now() });
}
}
}
saveToLocalStorage() {
try {
localStorage.setItem('agent-tracking-failed-events', JSON.stringify(this.failedEvents));
} catch (error) {
console.warn('Failed to save to localStorage:', error.message);
}
}
loadFromLocalStorage() {
try {
const stored = localStorage.getItem('agent-tracking-failed-events');
if (stored) {
this.failedEvents = JSON.parse(stored);
localStorage.removeItem('agent-tracking-failed-events');
}
} catch (error) {
console.warn('Failed to load from localStorage:', error.message);
}
}
}
// Usage
const resilientTracker = new ResilientTracker(monitor);
resilientTracker.safeTrack('agent-id', {
sessionId: 'session-123',
interactionType: 'user_message',
content: 'Hello'
});
Performance Optimization
Optimize tracking for high-throughput scenarios:Debounced Tracking
Copy
class DebouncedTracker {
constructor(monitor, debounceMs = 100) {
this.monitor = monitor;
this.debounceMs = debounceMs;
this.pendingEvents = new Map();
this.timers = new Map();
}
track(agentId, event) {
const key = `${agentId}-${event.sessionId}`;
// Clear existing timer
if (this.timers.has(key)) {
clearTimeout(this.timers.get(key));
}
// Add to pending events
if (!this.pendingEvents.has(key)) {
this.pendingEvents.set(key, []);
}
this.pendingEvents.get(key).push(event);
// Set new timer
const timer = setTimeout(() => {
this.flushEvents(agentId, key);
}, this.debounceMs);
this.timers.set(key, timer);
}
flushEvents(agentId, key) {
const events = this.pendingEvents.get(key) || [];
// Process all pending events
events.forEach(event => {
this.monitor.track(agentId, event);
});
// Cleanup
this.pendingEvents.delete(key);
this.timers.delete(key);
}
async flush() {
// Flush all pending events immediately
for (const [key, timer] of this.timers) {
clearTimeout(timer);
const agentId = key.split('-')[0];
this.flushEvents(agentId, key);
}
await this.monitor.flush();
}
}
Best Practices
Troubleshooting
Events not appearing in dashboard
Events not appearing in dashboard
Check that you’re calling
monitor.flush()
or waiting for the automatic flush interval. Verify your API key and organization ID are correct.High memory usage
High memory usage
Reduce batch size and flush interval. Implement event sampling for high-volume scenarios. Check for event accumulation without proper flushing.
Missing event metadata
Missing event metadata
Ensure you’re passing metadata objects correctly. Check that metadata keys match expected field names. Verify data types are serializable.
Compliance violations not detected
Compliance violations not detected
Confirm that
enableComplianceChecks: true
is set. Verify that compliance checks are running on the correct event types (typically agent responses).Session tracking issues
Session tracking issues
Use consistent session IDs across related events. Ensure session IDs are unique per conversation. Check that conversation start events are tracked.