Common issues, solutions, and debugging procedures for the Add-on Services system

Troubleshooting Guide

This guide provides solutions for common issues encountered with the Add-on Services system.

Common Issues & Solutions

1. Credit Consumption Failures

Symptoms

Users report service usage blocked despite having credits
"Insufficient credits" errors when credits are available
Credit balance discrepancies between dashboard and actual usage

Diagnostic Steps

Check Credit Balance Consistency:

# Verify credit balances
php artisan addon-services:verify-credits --service=sms

# Check for database inconsistencies
php artisan addon-services:audit-credits --detailed

Analyze Recent Activity:

# Check recent credit transactions
php artisan addon-services:recent-activity --hours=24

# Look for held credits that may be stuck
php artisan addon-services:check-holds --expired

Solutions

Fix Credit Balance Discrepancies:

# Reconcile all credit balances
php artisan addon-services:reconcile-credits --fix

# Reset specific service balance
php artisan addon-services:reset-balance --service-id=1 --recalculate

Release Stuck Credit Holds:

# Clean up expired holds (older than 2 hours)
php artisan addon-services:cleanup-holds --force

# Release specific hold
php artisan addon-services:release-hold --usage-id=12345

2. Payment Processing Issues

MPesa Payment Failures

Symptoms:

STK push not received on mobile device
Payment shows as pending indefinitely
"Transaction failed" errors

Diagnostic Commands:

# Check MPesa transaction status
php artisan mpesa:check-transaction --reference=MPX123456789

# Verify MPesa configuration
php artisan mpesa:test-connection

# Review recent MPesa logs
tail -f storage/logs/payments.log | grep -i mpesa

Solutions:

# Retry failed MPesa payment
php artisan addon-services:retry-payment --payment-id=123

# Force payment verification
php artisan addon-services:verify-payment --reference=MPX123456789 --force

# Reset payment status
php artisan addon-services:reset-payment --payment-id=123 --status=pending

Card Payment Issues

Symptoms:

Redirect to payment gateway fails
Payment successful but credits not allocated
Duplicate payment charges

Diagnostic Steps:

# Check payment gateway logs
php artisan payment:check-gateway-status

# Verify webhook deliveries
php artisan payment:check-webhooks --hours=24

# Audit payment records
php artisan addon-services:audit-payments --suspicious

3. Invoice Generation Problems

Symptoms

PDF generation fails or times out
Invoice documents not created
Client notifications not sent

Diagnostic Commands

# Check PDF generation queue
php artisan queue:work --queue=invoices --timeout=300

# Monitor document storage usage
df -h /path/to/invoice/storage

# Check failed PDF generation jobs
php artisan failed:list --queue=invoices

# Test PDF generation manually
php artisan addon-services:test-pdf --invoice-id=12345

Solutions

# Retry failed invoice generation
php artisan addon-services:retry-invoice --invoice-id=12345

# Regenerate invoice PDF
php artisan addon-services:regenerate-pdf --invoice-id=12345

# Clear PDF generation cache
php artisan addon-services:clear-pdf-cache

# Fix storage permissions
sudo chown -R www-data:www-data storage/app/invoices
sudo chmod -R 755 storage/app/invoices

4. Client Synchronization Issues

Symptoms

Plans not appearing in client systems
Sync status stuck in "pending" or "in_progress"
Authentication failures with client APIs

Diagnostic Tools

# Check sync status for all clients
php artisan addon-services:sync-status

# Test client connectivity
php artisan addon-services:test-connection --client=123

# View sync error logs
php artisan addon-services:sync-errors --recent

# Retry failed synchronizations
php artisan addon-services:retry-sync --status=failed

Solutions

# Reset client sync status
php artisan addon-services:reset-sync --client-plan-id=456

# Force resync specific client
php artisan addon-services:force-sync --client=123

# Update client credentials
php artisan addon-services:update-credentials --client=123

# Test SAT library connectivity
php artisan sat:test-connection --client-sub-account=456

5. Threshold Notification Issues

Symptoms

Low credit alerts not being sent
Duplicate threshold notifications
Notifications sent to wrong recipients

Diagnostic Commands

# Check threshold configurations
php artisan addon-services:check-thresholds

# Test notification delivery
php artisan addon-services:test-notification --service-id=1

# View recent notification logs
php artisan addon-services:notification-history --days=7

# Check email queue status
php artisan queue:work --queue=notifications

Solutions

# Fix threshold settings
php artisan addon-services:fix-thresholds

# Resend failed notifications
php artisan addon-services:resend-notifications --failed

# Clear notification cache
php artisan addon-services:clear-notification-cache

# Update notification recipients
php artisan addon-services:update-recipients --service-id=1

Performance Issues

1. Slow Credit Balance Queries

Symptoms

Dashboard loading slowly
Timeout errors when checking credit balances
High database CPU usage

Optimization Commands

# Analyze slow queries
php artisan addon-services:analyze-queries --slow

# Rebuild credit balance indexes
php artisan addon-services:rebuild-indexes

# Optimize credit balance calculation
php artisan addon-services:optimize-balances

# Enable query caching
php artisan config:cache

Database Optimization

-- Add missing indexes
CREATE INDEX idx_credits_consumption_analytics 
ON billing_credits_usages(subscription_id, hold, created_at, quantity);

CREATE INDEX idx_payment_status_tracking 
ON billing_add_ons_payments(invoice_status, created_at, user_id);

-- Analyze table statistics
ANALYZE TABLE billing_add_ons_plans;
ANALYZE TABLE billing_credits_usages;
ANALYZE TABLE billing_add_ons_payments;

2. Queue Processing Bottlenecks

Symptoms

Jobs backing up in queues
Long processing times for credit operations
Memory exhaustion in queue workers

Diagnostic Commands

# Check queue status
php artisan queue:monitor

# View failed jobs
php artisan failed:list

# Monitor queue worker memory usage
ps aux | grep queue:work

# Check Redis memory usage
redis-cli info memory

Solutions

# Restart queue workers
sudo supervisorctl restart addon-services-worker:*

# Clear failed jobs
php artisan failed:flush

# Optimize queue configuration
php artisan queue:restart

# Scale up queue workers
sudo supervisorctl add addon-services-worker:3

Data Integrity Issues

1. Credit Balance Inconsistencies

Detection Commands

# Run comprehensive audit
php artisan addon-services:audit --comprehensive

# Check for orphaned records
php artisan addon-services:check-orphans

# Validate foreign key relationships
php artisan addon-services:validate-relationships

Correction Procedures

# Fix balance discrepancies
php artisan addon-services:fix-balances --dry-run
php artisan addon-services:fix-balances --execute

# Remove orphaned records
php artisan addon-services:cleanup-orphans --confirm

# Rebuild subscription balances
php artisan addon-services:rebuild-balances

2. Payment Record Inconsistencies

Detection and Correction

# Find duplicate payments
php artisan addon-services:find-duplicates --payments

# Check for unallocated payments
php artisan addon-services:check-unallocated

# Fix payment-credit mismatches
php artisan addon-services:fix-payment-credits

System Monitoring

Health Check Commands

# Overall system health
php artisan addon-services:health-check

# Check all services status
php artisan addon-services:status

# Verify external integrations
php artisan addon-services:check-integrations

# Database connectivity test
php artisan addon-services:test-db

Log Analysis

Important Log Files:

storage/logs/addon-services.log - General system logs
storage/logs/payments.log - Payment processing logs
storage/logs/threshold-alerts.log - Threshold notification logs
storage/logs/laravel.log - Application errors

Useful Log Commands:

# Follow addon services logs
tail -f storage/logs/addon-services.log

# Search for specific errors
grep -r "InsufficientCreditsException" storage/logs/

# View payment errors
grep -A 5 -B 5 "payment.*failed" storage/logs/payments.log

# Check threshold alerts
grep "threshold.*breach" storage/logs/threshold-alerts.log

Emergency Procedures

1. Service Outage Response

Immediate Actions:

# Disable problematic services temporarily
php artisan addon-services:disable --service=sms_messaging

# Enable maintenance mode if needed
php artisan down --message="Addon services maintenance"

# Check system resources
df -h
free -m
top

Recovery Steps:

# Restart all workers
sudo supervisorctl restart all

# Clear all caches
php artisan cache:clear
php artisan config:clear
php artisan route:clear

# Re-enable services
php artisan addon-services:enable --service=sms_messaging
php artisan up

2. Data Corruption Recovery

Backup Verification:

# Check recent backups
php artisan addon-services:check-backups

# Verify backup integrity
php artisan addon-services:verify-backup --file=backup.sql

Recovery Process:

# Stop all processing
php artisan queue:restart
sudo supervisorctl stop all

# Restore from backup (if needed)
mysql -u username -p database_name < backup.sql

# Rebuild indexes and caches
php artisan migrate:refresh --seed
php artisan addon-services:rebuild-all

# Restart services
sudo supervisorctl start all

Getting Help

Support Channels

Internal Documentation: Check this guide and API documentation
System Logs: Review relevant log files for error details
Development Team: Contact the addon services development team
Database Administrator: For database-specific issues

Escalation Procedures

Level 1 - Basic Issues:

Credit balance problems
Simple payment failures
Configuration issues

Level 2 - Complex Issues:

Performance problems
Integration failures
Data synchronization issues

Level 3 - Critical Issues:

System outages
Data corruption
Security incidents

Emergency Contacts

On-call Engineer: [Contact details]
Database Admin: [Contact details]
Infrastructure Team: [Contact details]
Security Team: [Contact details]

This troubleshooting guide should help resolve most common issues with the Add-on Services system. For issues not covered here, please escalate through the appropriate support channels.

Troubleshooting Guide

On this page