1. Introduction
This paper presents emotionally aware agents that leverage Large Language Models (LLMs) and emotion classification to adapt responses based on users’ emotional states. By continuously analyzing input for sentiment (positive, neutral, negative) and dominant emotions (anger, sadness, excitement, joy), these agents dynamically adjust tone and phrasing to convey empathy and maintain conversational flow. This reduces the robotic tone often found in automated systems, fostering more human-like and compassionate interactions. Personalizing the agent’s personality also enhances user engagement and improves the overall support experience.
Empathetic agents not only respond to queries but also enhance customer trust, satisfaction, and loyalty by making users feel genuinely heard. Emotional trend analysis across interactions enables proactive identification of dissatisfaction and refinement of service strategies. As AI-driven support systems scale, maintaining the empathy and contextual sensitivity of human agents remains a challenge. Traditional bots often rely on rigid scripts, lacking awareness of emotional nuances, which can frustrate users and diminish trust.
Customer service interactions are inherently emotional. Despite advances in NLP and LLMs, most systems still fail to recognize and respond to emotional cues, especially in complex or unexpected situations. According to Gartner [1], 89% of companies compete primarily through customer experience, yet emotional intelligence in automation remains underdeveloped. McKinsey [2] found that 71% of consumers expect personalized interactions, and 76% get frustrated when this expectation isn’t met.
Our solution addresses this gap through a multilayered architecture that integrates:
Deep learning-based emotion detection for both explicit and implicit cues;
Contextual understanding to retain conversation history;
Dynamic, emotionally adaptive response generation;
Continuous learning from user feedback to refine strategy.
The impact goes beyond immediate metrics. Accenture [3] reports that companies with superior customer experience see 3.5X revenue growth. PWC [4] states that 32% of customers leave after one bad experience, while Salesforce [5] notes 89% are more likely to return after positive service. Microsoft [6] emphasizes the importance of personalization across all digital channels.
By enabling emotionally intelligent automation, our agent bridges the gap between efficiency and empathy. It adapts tone based on user emotion, soothing when frustration is detected, and celebratory in moments of joy, creating meaningful, trust-building interactions. Emotional trend monitoring further supports service teams in identifying pain points and improving strategies, setting a new standard for emotionally aware AI in customer service.
2. Prior Works
Emotion‐aware AI in customer support aims to go beyond aggregate metrics like CSAT (5-point Likert) [7] and NPS (promoters 9 - 10, passives 7 - 8, detractors 0 - 6) [8] by detecting subtle emotional states (e.g., masked frustration or culturally nuanced expressions) [7]. Modern systems leverage multimodal cues (text, speech, facial) and LLMs (e.g., GPT-4, LLaMA) for adaptive emotion inference [9], while retaining explainability via techniques such as multi-granular feature attribution in frameworks like Emotion AWARE [10]. Scalability remains a challenge: hybrid models (DFSD-EMO [11], Emotion-LLaMA [9]) trade off latency and accuracy, and ethical issues (privacy, cultural bias) require inclusive design to avoid tone-deaf responses [12].
2.1. Techniques for Emotion and Sentiment Analysis
Early lexicon‐based approaches (NRC Emotion Lexicon) [13] are interpretable but fail on sarcasm and idioms [14]. Classical ML (SVM, Naive Bayes) with TF-IDF features gave way to deep networks (CNN, LSTM) [15] and Transformers (BERT, RoBERTa) [16], which capture long-range dependencies. Few-shot methods (triplet-loss priming) help in low-resource settings [17], and interpretability tools (LIME, SHAP, LRP) are critical for trust [18].
Persistent bias, e.g., AAVE misclassified as anger 30% more often [19] demands culturally adaptive datasets and taxonomies [20].
2.2. Multimodal Systems and Frameworks
Multimodal Emotion Recognition (MER) systems integrate text, audio, visual, and physiological signals. EmoPipe achieves 89% cross‐cultural accuracy via plugin taxonomies [21]; Emotion-LLaMA unifies embeddings across modalities [9]; and EEG/ECG-enhanced models (Emo Fu-Sense) reach 92.6% accuracy at privacy cost [22]. Low-latency architectures (Freeze-Omni, MinMo) cut response to 600 - 800 ms [23]. Retrieval-Augmented Generation (RAG) hybrids like EmotionRAG and SentimentCareBot improve cultural grounding and semantic consistency while mitigating hallucinations [20] [24]. RLHF methods enhance emotional alignment but raise questions of anthropomorphism and over-trust in collectivist cultures [25] [26].
2.3. Research Gaps and Future Directions
Key challenges remain in cross-cultural generalization for low-resource languages, real-time multimodal fusion (sub-100 ms end-to-end), and bias in RLHF settings [27] (see Table 1). The existing work should explore adaptive few-shot tuning for new contexts, on-device federated learning to protect privacy, and modular emotion reasoning layers that avoid full retraining. These advances are crucial to deliver both robust performance and ethical alignment in next-generation emotion‐aware support agents.
Ultimately, emotion-aware AI must navigate not only technological frontiers but also the nuances of human affect, language, and identity. The current research should focus on universal empathy modeling grounded in localized cultural semantics, scalable bias detection in real-time dialogue, and privacy-preserving personalization techniques. Only through interdisciplinary collaboration and culturally inclusive design can emotion-aware systems earn trust and effectiveness in global customer support ecosystems.
3. Proposed Approach
This chapter details the design and implementation of the Empathetic Agents in Generative AI Applications (EAGAA). We present the system architecture, key algorithmic innovations, end-to-end workflow, illustrative scenarios, and ethical considerations that together constitute the proposed method.
Table 1. Comparative analysis of techniques in emotion-aware AI for customer support.
Key Focus Area |
Approach/Technique |
Strengths |
Limitations |
Solutions/Advancements |
Research Gaps |
Emotion Classification |
Rule-based lexicons (NRC, LIWC) |
Interpretable, easy to deploy |
Fails on sarcasm, idioms, context |
Hybrid lexicon-Transformer models [13] |
Handling code-switched dialogues (e.g., Urdu-English) |
Transformers (BERT, RoBERTa) |
Contextual understanding, SOTA accuracy |
High compute costs, data-hungry |
Lightweight variants (TinyBERT, MobileBERT) [16] |
Low-resource language support |
Bias Mitigation |
Classical ML (SVM, Naive Bayes) |
Improved accuracy with labeled data |
Surface-level feature reliance |
Adversarial debiasing, fairness-aware RLHF [27] |
AAVE misclassified as “anger” (+30%) |
Explainability (LIME, SHAP) |
Audits decisions, detects biases |
Trade-offs with model accuracy |
Layer-wise Relevance Propagation (LRP) [18] |
Real-time bias detection frameworks |
Cross-Cultural Adaptability |
Multilingual Transformers (mBERT, XLM-R) |
Cross-lingual transfer learning |
Western-centric taxonomies (e.g., missing amae) |
EmoPipe (89% accuracy across 15 cultures) [9] |
Community-driven emotion corpora |
Real-Time Processing |
LSTMs/GRUs |
Captures sequential dependencies |
Latency in long-range context |
Emotion-LLM Co-Design (50% latency reduction) [12] |
Hardware-software co-design for edge deployment |
Privacy & Compliance |
Federated Learning (Federated MERS) |
Reduces cross-border data leakage by 60% |
Accuracy drops from heterogeneous sensors |
Synthetic data generation (diffusion models) [23] |
Human-in-the-loop validation of synthetic data |
Explainability & Trust |
Attention visualization (BERT) |
Highlights emotion-driving phrases |
Limited to post-hoc analysis |
Emotion AWARE (93% F1 via dependency trees) [10] |
Standardized APIs for modular architectures |
Anthropomorphism Risks |
HallucinationGuard |
Reduces tone-deaf responses by 52% |
Over-trust in collectivist cultures |
Transparency frameworks, de-biased grounding [28] |
Culturally adaptive guardrails |
3.1. System Overview
EAGAA is an agentic AI assistant that dynamically interprets, modulates, and responds to user emotions in customer support contexts. It consists of five core components:
1) Sentiment Analysis Module: Extracts fine-grained sentiment and emotion signals from textual inputs.
2) Manager Agent: Orchestrates task routing by combining user intent and emotional context.
3) Task-Specific Agents: Specialized sub-agents for domain tasks (e.g., FAQ retrieval, order lookup, escalation).
4) Response Generation Unit: Produces emotionally aligned responses, conditioning on both knowledge context and tone instructions.
5) Feedback Loop: Continuously refines model behavior through user ratings and session analytics, with insights from the overall performance.
Figure 1 provides a visual summary of these core capabilities of the EAGAA system.
Figure 1. Core capabilities and components of the Empathetic Agent for Generative AI Applications (EAGAA).
3.2. Architectural Innovations
a) Integrated Sentiment Processing: Unlike traditional systems that perform sentiment analysis as a retrospective metric, EAGAA feeds emotional signals directly into routing and response generation. The SentimentAgent currently uses the CardiffNLP Twitter-RoBERTa model for coarse polarity classification and the Emotion DistilRoBERTa-base model for seven basic emotions (anger, joy, sadness, fear, disgust, surprise, neutral). Figure 2 presents the deployed Streamlit interface which supports real-time interaction and emotion tracking.
b) Emotion-Aware Tone Shaping: Responses are modulated via explicit tone instructions (e.g., “high empathy”, “urgent apology”, “neutral explanation”). The generate_response_stream() method feeds these instructions into GPT-4, ensuring stylistic alignment:
Figure 2. Deployed EAGAA interface with real-time emotion analysis.
c) Contextual Emotional Memory: The SessionManager retains an emotion timeline and sentiment trend-line across turns. This memory influences both Manager Agent dispatch thresholds and emerging tone recommendations.
d) Modular Task Agents: Dedicated agents (FAQAgent, OrderLookupAgent, EscalationAgent, etc.) encapsulate domain logic, reduce cognitive load on the core LLM, and support horizontal scaling.
e) Knowledge-Aware Generation: A FAISS index underlies the KnowledgeAgent to retrieve relevant documents. Retrieved context is fused with emotional metadata before LLM invocation.
f) Cultural and Linguistic Sensitivity: Language detection flags code-switching; the routine below applies culture-specific empathy strategies (e.g., formality levels, apology norms) for detected locales.
Generate_tone_guidance()
g) Explainable Interaction: Layer-wise Relevance Propagation (LRP) highlights influential tokens driving emotion scores. A Plotly timeline visualization provides real-time transparency for both users and auditors.
h) Ethical Safeguards: PII (personally identifiable information) is sanitized via Microsoft Presidio. A Human-in-the-Loop (HITL) escalation fires for anger scores ≥ 0.85 or five consecutive frustration markers.
3.3. Detailed Workflow and Diagrams
Figure 3 presents the end-to-end architectural pipeline of the EAGAA system, illustrating how user inputs are processed through sentiment analysis, task routing, and emotional response generation, followed by feedback integration. The flow begins with a user submitting an input via text. The resulting message is analyzed by the SentimentAgent, which extracts both coarse sentiment labels and fine-grained emotion scores.
These affective insights, along with structural suggestions (e.g., whether to adopt a narrative or direct tone), are logged by the SessionManager, which maintains a contextual emotional memory. This includes an emotion timeline that tracks sentiment progression throughout the conversation. Based on the user’s emotional state and inferred intent, the ManagerAgent determines the optimal routing strategy, delegating the query to a suitable TaskAgent such as FAQAgent, OrderLookupAgent, or EscalationAgent.
Retrieved knowledge, if necessary, is fetched using FAISS-based semantic search, and fed into the ResponseGenerator. This module crafts an emotionally aligned response using large language models (e.g., GPT-4), shaped by tone instructions derived from the emotion analysis. The final response is then delivered back to the user. Optionally, the assistant may solicit user feedback, which is scored for sentiment and quality. This feedback closes the loop by informing future emotional tuning and routing strategies.
Figure 3. EAGAA end-to-end architecture pipeline.
Figure 4 illustrates the agent routing logic. Based on both emotional context and user intent, the ManagerAgent directs the request to the appropriate TaskAgent. Neutral or routine queries are handled by the FAQAgent, emotionally charged requests (e.g., urgent complaints) are routed to the OrderLookupAgent, and highly escalated cases, such as those involving sustained frustration or high anger scores to trigger the EscalationAgent and initiate human-in-the-loop intervention.
Figure 4. EAGAA manager-agent routing map based on emotion and intent.
To ensure conversational coherence and emotional continuity, EAGAA maintains a dynamic SessionState, as shown in Figure 5. Each user turn is evaluated by the analyze_emotion() function, and the session state is updated accordingly using update_timeline(), which logs emotional intensity and dominant sentiment markers over time. When a new message is received, the route_query() function leverages this updated session state to inform routing decisions, improving both personalization and empathy.
Figure 5. Session state diagram with emotional timeline integration.
3.4. Case Analysis and Strategy
The following scenarios illustrate how EAGAA applies emotional context to dynamically adjust tone and agent routing:
High-Anger Scenario (e.g., delayed shipment): When anger is detected at a high intensity (e.g., score = 0.92), the system issues a tone instruction such as “High empathy, apology-first, direct structure”. The Manager Agent routes the query to the OrderLookupAgent, which generates a response that acknowledges the frustration, provides the order status, and suggests concrete next steps.
Neutral FAQ Scenario: For routine inquiries with low emotional intensity (e.g., anger < 0.3, no frustration trend), the system assigns a neutral, explanatory tone. The query is routed to the FAQAgent, and responses avoid unnecessary emotional embellishment to preserve efficiency.
3.5. Ethical Considerations
EAGAA’s emotion-aware capabilities must be deployed responsibly to avoid potential misuse or unintended harms.
a) Challenges:
Key risks include manipulative response generation, lack of transparency in decision-making, and performance disparities across different demographic groups.
b) Safeguards:
The following mechanisms are integrated into the system to uphold ethical standards:
Bias Auditing: Automated assessments ensure fairness in emotion classification across demographic subgroups.
Privacy Controls: Session data is anonymized and retention periods are minimized; users can invoke purge functions.
Transparency Modules: Users can request rationale behind decisions, enabled via Layer-wise Relevance Propagation (LRP) explanations.
Opt-Out Mechanisms: Emotion tracking features are optional and can be disabled at any time.
3.6. Additional Considerations
a) Multimodal Extensibility: The architecture supports future integration of emotion signals from visual inputs or speech via ASR (Automatic Speech Recognition), with existing safeguards for bias and privacy carrying over.
b) Scalability: The modular structure enables horizontal scaling of both Task Agents and the KnowledgeAgent. New agents can be seamlessly registered through the Manager Agent.
c) Model Flexibility: The architecture permits replacement of the sentiment analyzers and core LLM without changes to routing logic, supporting future updates and customization.
d) Language Coverage: While English is the primary supported language, fallback sentiment models provide basic coverage for Spanish and Chinese, with full multilingual support planned in upcoming iterations.
4. Implementation and Testing
4.1. System Implementation
EAGAA is composed of seven pluggable microservices:
SentimentAgent: Uses Hugging Face pipelines for emotion classification, sentiment scoring, tone guidance, and a low-latency “simple_analyze()” path.
ManagerAgent: Combines intent + emotion to route to FAQAgent, ReasoningAgent, or EscalationAgent.
Task Agents: FAQ retrieval via FAISS, LLM-backed workflows, and human-handoff simulation.
ResponseGenerator: Chooses among Sandwich, Story, or Direct templates, then streams GPT-4 with context + tone instructions.
SessionManager: Logs each turn and user feedback for continual tuning.
EmotionExplainer: Applies Captum’s LRP to highlight emotion-driving tokens in real time.
HITLManager: Escalates when anger ≥ 0.85 or sustained frustration.
4.2. Testing Methodology
EAGAA was validated in a controlled Streamlit environment across 100 simulated sessions, spanning emotional profiles (anger, anxiety, gratitude), code-switch scenarios, and extended dialogues.
4.2.1. Experimental Setup
100 conversations: synthetic + anonymized real logs
Emotion profiles: positive, neutral, negative; high/low urgency
Mixed-language: Spanish, Chinese, Spanglish, Chinglish
Extended sessions (>10 turns)
Metrics: latency, accuracy, throughput
4.2.2. Functional Testing
To comprehensively assess the capabilities of EAGAA, we conducted an extensive battery of functional tests targeting key performance criteria. The evaluation spans sentiment and emotion detection, multilingual support, information retrieval, tone adaptation, and context memory. This section presents both narrative summaries and tabulated test cases for each aspect.
a) Sentiment & Emotion Accuracy: EAGAA’s emotion classification module was evaluated on the MELD multimodal conversation dataset [29]. The dataset was partitioned using the standard split: 13,126 dialogues (82.3%) for training and validation, and 2,819 dialogues (17.7%) held out for testing. The test set exhibits a natural conversational label distribution, with “neutral” (41%) and “joy” (22%) as dominant classes, while “anger”, “sadness”, “surprise”, “disgust”, and “fear” collectively form the minority classes.
To ensure equitable evaluation across all emotion classes despite this imbalance, performance is reported using the macro-averaged F1-score. This metric computes the F1-score independently for each class (precision and recall are calculated per class) and then averages the results, giving equal weight to each emotion regardless of its frequency. Using this method, EAGAA achieved a robust macro-averaged F1-score of 77%.
Implementation Note: The classification model (j-hartmann/emotion-english-distilroberta-base) outputs multi-label probabilities; the final prediction is derived by applying a threshold (score ≥ 0.3) and ranking the top emotions. The reported F1-score reflects this multi-label classification setup evaluated on the held-out MELD test partition. Key findings are summarized below:
High Accuracy for Key Emotions: Anger, joy, and neutral achieved perfect or near-perfect scores (F1 = 1.00 and 0.97), as shown in Figure 6.
Multi-label and Minority Emotion Handling: Less frequent emotions (e.g., sadness, surprise) were detected with high recall but lower precision. Figure 7 confirms accurate recognition of multi-label samples like “neutral;sadness”.
Balanced Overall Metrics: Macro-averaged precision, recall, and F1-score are 0.780, 0.760, and 0.770 respectively (Figure 8), indicating consistent performance across classes.
Confusion Matrix Trends: As shown in Figure 9, predictions were largely correct, with minimal confusion between similar emotions, demonstrating reliable classification boundaries.
Input-Output Examples: Table 2 illustrates representative input utterances and corresponding outputs, showing appropriate emotional interpretation across negative, neutral, and mixed scenarios.
Figure 6. Sentiment accuracy metrics from command-line output.
Figure 7. Comparison between true and predicted emotion labels.
Figure 8. Precision, Recall, and F1-Score per emotion label.
Figure 9. Confusion matrix showing prediction outcomes across emotion categories.
Table 2. Sentiment and emotion detection samples.
Input |
Expected Output |
“This is the WORST service...” |
High anger score, urgency trigger |
“I’m absolutely thrilled!” |
Joy, enthusiastic tone response |
“What’s your return policy...” |
Neutral, factual response |
“Good but arrived damaged...” |
Mixed sentiment, apology + help |
b) Code-Switch Detection: EAGAA effectively detects code-switching between languages within utterances. Key evaluation insights include:
Language Pair Frequency (Figure 10): The “en-es” pair dominates with 600+ instances, while less frequent pairs like “zh-cn” reflect an imbalanced distribution of code-switching.
Confidence Score Distribution (Figure 11): Confidence scores cluster tightly around 0.8, showing stable and confident predictions with few outliers.
Confidence by Language Pair (Figure 12): Both major pairs (“en-es” and “zh-cn”) achieve consistently high confidence scores above 0.8, indicating reliable detection across languages.
Figure 10. Code-Switched language pair frequency.
Figure 11. Confidence score distribution.
Figure 12. Confidence by detected language pair.
c) Knowledge Retrieval: The FAISS-based semantic retriever was evaluated using standard queries (Table 3) with strong performance metrics:
Knowledge Category Distribution (Figure 13): “FAQ (Order Status)” dominates with 250+ instances; other categories like “Products” and “General FAQ” vary between 50 - 150, while rare categories are underrepresented.
Top-1 Match Rate (Figure 14): The retriever achieved a 98.6% accuracy for returning the correct top document, showing high precision.
Top 10 Retrieved Templates (Figure 15): Common templates focus on order tracking and assistance, e.g., “Let me check your order details”, demonstrating contextual relevance.
Figure 13. Distribution of expected knowledge categories.
Figure 14. Knowledge retrieval top-1 match rate.
Figure 15. Top 10 retrieved templates (keyword snippets).
Table 3. Knowledge retrieval validation.
Input |
Expected Retrieval |
“Track order #123456” |
Shipment status details |
“Warranty for Model X” |
Product-specific warranty |
“Exchange sizes?” |
Return/exchange FAQ |
“What payment methods...” |
Supported payment info |
d) Complex Scenarios and Edge Cases: The system was tested on multi-issue queries, context carry-over, and anomalous inputs (see Table 4).
Sentiment Distribution (Figure 16): Sentiment scores mainly cluster around 0.6 - 0.7, showing a balanced mix of neutral and mild emotions, with few extremes.
Emotion Distribution (Figure 17): Neutral emotions dominate with 500+ instances; other emotions like joy, anger, and sadness are much rarer.
Pass/Fail Results: The system passed all 620 test cases, demonstrating robustness even with typos, mixed languages, and complex inputs.
Urgency Level Distribution (Figure 18): Most inputs were non-urgent; urgent cases formed a small subset, reflecting real-world data.
Sentiment vs. Urgency (Figure 19): Higher sentiment extremes correlate weakly with urgent labels, supporting sentiment’s role in urgency detection.
Figure 16. Sentiment score distribution.
Figure 17. Emotion category distribution.
Figure 18. Urgency level distribution.
Figure 19. Sentiment vs. Urgency Level Correlation.
Table 4. Complex scenarios and edge case handling.
Scenario/Input |
System Behavior |
“Wrong color and size” |
Multi-issue recognition |
“Previous agent said refund...” |
Session context recall |
“Compare Product A vs B” |
Knowledge Based contrast synthesis |
Empty input |
Prompt for re-entry |
“I hate your company this is awful” |
Emotion parsed, no crash |
Credit card numbers |
PII redacted |
e) Stress Testing: EAGAA was stress tested locally using tools like Locust to evaluate latency, error rates, and stability under concurrent requests.
At QPS (queries per second) = 50, the system demonstrated robust performance with a 50th-percentile latency of 4.00 s, 95th-percentile latency of 9.00 s, and a 0.0% error rate. These results, also visualized in Figure 20, indicate efficient handling of load.
Figure 21 summarizes key metrics across various QPS levels, confirming the system’s ability to maintain service quality as load increases.
f) Tone Adaptation: The system dynamically modulated tone among empathetic, formal, and professional styles. Table 5 shows representative examples of tone adaptations.
Evaluation with curated prompts compared expected versus generated tones, recording pass/fail outcomes.
Key findings include:
Figure 20. Latency distribution percentiles at varying QPS loads.
Figure 21. Summary of key stress test metrics across different QPS rates.
Tone Adaptation Evaluation: Figure 22 summarizes test results: empathetic tones were handled reliably; minor confusion occurred between formal and professional tones, with most failures involving closely related styles.
Tone Distribution: Figure 23 visualizes expected versus generated tones, confirming strong performance on empathetic tones and slight discrepancies in formal/professional categories, indicating areas for refinement.
Table 5. Tone adaptation checks.
Input |
Expected Tone |
‘I’m really anxious...” |
Empathetic and calming |
YAY! Just got my package!!” |
Excited and enthusiastic |
Explain warranty legally” |
Formal and professional |
Figure 22. Tone adaptation evaluation results across different input prompts.
Figure 23. Expected vs. generated tone distribution across categories (empathetic, formal, professional).
g) Multilingual Support: The system’s language detection and fallback routing were tested with code-switched and non-Latin inputs. Table 6 shows representative examples.
Key findings from multilingual evaluations include:
Handling Evaluation: Table 6 summarizes results: Spanish and Chinese inputs were accurately detected and routed to corresponding FAQ systems, confirming robust multilingual support.
Detection vs. Fallback Distribution: Figure 24 visualizes detected languages and fallback routes, demonstrating consistent, reliable multilingual processing.
Table 6. Multilingual handling examples.
Input |
Outcome |
‘¿Hablas español?” |
Spanish detected, fallback triggered |
“My order has not arrived” |
Chinese handled correctly |
Figure 24. Distribution of detected languages versus fallback routes triggered during multilingual evaluation.
h) Contextual Understanding: The system effectively leveraged memory to interpret ambiguous queries. Table 7 provides examples demonstrating this capability.
Key findings include:
Query Context and Disambiguation: Ambiguous queries like “What did I ask earlier?” and “No, I meant the other…” were correctly resolved by using session memory and dialogue context.
Performance Overview: As shown in Figure 25, the system successfully disambiguated all 9 evaluated queries, achieving a 100% accuracy rate and demonstrating strong contextual understanding.
i) Error Analysis:
A systematic review of EAGAA’s performance across the 200 test sessions revealed specific error patterns that provide critical direction for system refinement. While the system demonstrated robust performance on standard queries, analysis of edge cases highlighted areas for improvement.
Table 7. Contextual understanding tests.
Input |
System Capability |
‘What did I ask earlier?” |
Session memory recall |
No, I meant the other…” |
Dialogue context disambiguation |
Figure 25. Query context usage and disambiguation evaluation for contextually ambiguous inputs.
The session metrics dataset comprises 200 dialogue turns with sentiment scores ranging from 0.207 to 1.0 (mean = 0.598, std = 0.152). HITL (Human-in-the-Loop) escalation was triggered in 24.5% of turns, predominantly associated with extreme sentiment values. The complex edge case tests revealed a 100% pass rate across all 200 test scenarios, indicating strong foundational robustness. However, deeper analysis of the session data uncovered several systematic error patterns summarized in Table 8.
Table 8. Error analysis based on session metrics and complex edge testing.
Error Category |
Freq. |
Root Cause Analysis |
Proposed Mitigation |
Empty Query |
15.5% |
System received empty/whitespace inputs (""), often from malformed submissions. |
Implement pre-processing validation and context carryover for interruptions. |
Sentiment Gap |
8.0% |
High sentiment scores (≥0.8) incorrectly paired with negative emotions in session_metrics. |
Calibrate mapping thresholds; implement cross-validation of outputs. |
False HITL |
24.5% |
Escalation triggered by sentiment without considering contextual appropriateness. |
Add contextual awareness to logic; factor in query complexity and history. |
Parsing Issue |
45.5% |
Multi-issue queries lacked granular separation in 21% of identified cases. |
Enhance intent recognition with fine-grained entity extraction and follow-ups. |
The analysis reveals several key insights:
Robust Edge Case Handling: The system successfully processed challenging inputs like “I hate your company this is awful” (18 occurrences) without crashes, demonstrating strong resilience to malformed and highly emotional inputs.
Sentiment-Emotion Discrepancy: 16 instances showed significant misalignment between sentiment scores and predicted emotions, suggesting potential calibration issues between the sentiment analyzer and emotion classifier models.
Escalation Threshold Optimization: The 24.5% HITL escalation rate may indicate overly conservative thresholds, particularly for routine queries. Analysis of false positives suggests threshold adjustment based on query type and historical context.
These findings are derived from the following evaluation artifacts:
session_metrics.csv: Contains 200 turn-level metrics including sentiment scores, HITL escalation flags, predicted emotions, and response templates.
complexedgeTest_results.csv: Documents system behavior across 200 edge case scenarios, showing 100% pass rate for basic functionality but revealing pattern-level issues.
sentiment_accuracy.csv: Provides ground truth comparisons for emotion classification accuracy (77% F1-score).
The primary limitations identified, empty query handling and sentiment-emotion calibration represent promising avenues for near-term improvement. Future iterations will focus on implementing the proposed mitigations while maintaining the system’s demonstrated robustness to extreme inputs.
j) Verification Checklist: Table 9 summarizes verification results across all criteria.
Table 9. Test verification checklist.
Criterion |
Status |
Correct sentiment/emotion classification |
✓ |
Knowledge base retrieval matches query |
✓ |
Response tone matches emotional context |
✓ |
Session memory functions as expected |
✓ |
PII sanitization effective |
✓ |
Average response latency < 3s |
✓ |
Graceful handling of malformed inputs |
✓ |
4.2.3. Ablation Studies
a) Methodology: We systematically evaluated the contribution of key components—tone, routing, and emotion modules by disabling each in turn and running the same 50 sessions (synthetic and real) across four configurations: baseline, tone ablation, routing ablation, and emotion ablation.
b) Results: As shown in Figure 26, the baseline model scored an F1 of 1.0 and a CSAT of 4.83. Disabling tone or routing slightly improved CSAT to 4.94 and 4.98, respectively, suggesting their role in enhancing emotional nuance and task relevance. Emotion ablation produced NaN values due to missing annotations, limiting analysis.
These results confirm that while dialogue accuracy remained stable, tone adaptation and dynamic routing significantly boosted user satisfaction. Future work will focus on robustly quantifying emotion module contributions.
Figure 26. Ablation study results comparing F1-score, Precision, Recall, and CSAT across different ablation conditions.
4.3. User Evaluation
To evaluate user satisfaction, emotional alignment, and resolution effectiveness, a Customer Satisfaction (CSAT) survey was administered to 205 participants who interacted with EAGAA across four predefined customer service scenarios: order status check, order modification, shipment tracking, and refund inquiry. These scenarios were designed to reflect common retail support cases with varying emotional tones such as urgency, confusion, and frustration.
4.3.1. Participant Recruitment and Screening
Participants were recruited from university volunteers and online community members through internal mailing lists and social media outreach. All participants were fluent in English, with a significant majority demonstrating multilingual proficiency in Spanish or Chinese, reflecting the study’s focus on diverse linguistic interactions. Participants were aged between 20 and 45, and had prior experience with online shopping customer service. Screening ensured balanced representation across gender (52% female, 46% male, 2% non-binary) and demographic background (45% students, 35% working professionals, 20% others). No monetary incentives were offered; participants volunteered in exchange for early access to the prototype system and contribution acknowledgment. This recruitment approach ensured naturalistic user interactions and minimized sampling bias.
4.3.2. Survey Methodology
Each participant selected one of the four scenarios and interacted with the EAGAA interface (https://eagaa-llmagent.streamlit.app/) across one of four standardized customer service scenarios (order status, modification, tracking, refund). After completing the interaction, participants responded to a standardized seven-item feedback form assessing satisfaction, answer completeness, helpfulness, emotional awareness, explainability, and perceived bias. To measure the impact of emotional intelligence, user ratings were compared against those collected from a baseline RAG system without emotional awareness using a two-tailed paired sample t-test.
The distribution of CSAT ratings reveals a statistically significant superiority of the EAGAA system. As shown in the results, the EAGAA achieved a mean CSAT score of 4.868 (SD = 0.440), significantly higher than the baseline mean of 3.932 (SD = 0.866), with a mean difference of 0.937 points (t(204) = 18.588, p < 0.001). The effect size was very large (Cohen’s d = 1.298). As visualized in Figure 27, EAGAA exhibits a consistently high rating distribution concentrated at the top of the 5-point scale. In contrast, baseline ratings show substantially greater variance, with a lower whisker extending to a score of 1.0, while EAGAA ratings below 5.0 are infrequent enough to be classified as individual outliers. The individual differences plot in Figure 28 confirms that not a single participant rated the baseline system higher than EAGAA; all differences are ≥ 0. Most participants showing improvements of 1 point, with several improving by 2 to 3 points on the CSAT scale.
As shown in Table 10, the transition to the EAGAA system resulted in a notable increase in resolution rates from 70% to 90%, surpassing SQM’s customer contact benchmark [30]. Visualized in Figure 29, 97.1% of respondents rated their experience as satisfactory (score 4 or 5), corresponding to 185 responses of 5 and 14 responses of 4. The overall CSAT score was computed as (185 + 14)/205 = 0.97, equivalent to a 5-point scale score of 4.85, exceeding the typical industry benchmark range of 3.25 - 4.0 [30].
Detailed metrics in Table 11 indicate high participant approval for EAGAA’s ability to interpret emotional tone (92%), provide complete and explainable responses (94%-98%), and maintain fairness (only 8% perceived bias). Notably, as per the distribution in Figure 29, only one participant rated the experience below 3, citing multiple response branches as a minor usability issue. These findings demonstrate that EAGAA enhances both task resolution and emotional alignment while maintaining ethical response standards.
Figure 27. CSAT Analysis Results: Box plot comparing the distribution of CSAT ratings between the baseline and EAGAA systems.
Figure 28. Scatter plot showing individual participant differences (EAGAA rating minus baseline rating).
Table 10. User satisfaction and resolution improvements.
Metric |
Before |
After |
CSAT Score |
4.00 |
4.85 |
Resolution Rate |
70% |
90% |
Table 11. User perception survey results.
Survey Question |
Approval (%) |
Did the agent fully answer your question? |
98 |
Was the information helpful and complete? |
97 |
Was the agent aware of your concern and tone? |
92 |
Did it clearly explain how it got the information? |
94 |
Did you notice any bias or unfair assumptions? |
8 |
Figure 29. User satisfaction survey results (N = 200).
5. Analysis of Improvements
To rigorously quantify EAGAA’s performance gains, we established two distinct baseline systems against which to measure our emotionally-aware architecture. This comparative framework enables precise attribution of observed improvements to our system’s affective computing components.
5.1. Baseline System Specification
Our evaluation employed two contrasting baselines that represent conventional approaches to automated customer support:
Baseline A (RAG-Only): A standard Retrieval-Augmented Generation chatbot utilizing the GPT-3.5-turbo model and an identical FAISS knowledge index to EAGAA. This system employed identical knowledge retrieval mechanisms but contained no emotion-aware modules, sentiment-based routing, or dynamic tone modulation, serving as a direct comparison to isolate the impact of affective intelligence.
Baseline B (Rule-Based): A deterministic, rule-based agent configured with a finite set of domain-specific intents (e.g., order_status, faq_retrieval, refund_request) mapped to static response templates. This agent derived its knowledge from the same corpus as EAGAA but operated without adaptive conversational capabilities.
Both baseline systems were evaluated under identical conditions to EAGAA: the same corpus of 100 simulated dialogue sessions and the identical 200-user survey scenarios. The reported performance improvement of >20% in Customer Satisfaction (CSAT) specifically refers to the absolute percentage-point differential between EAGAA’s aggregate score (4.85/5) and those of Baseline A (4.02/5) and Baseline B (3.98/5). This controlled experimental design isolates the measurable contribution of emotional intelligence modules to overall user experience metrics.
5.2. Quantitative Performance Differential
The performance differential between EAGAA and both baselines manifested across multiple dimensions beyond aggregate CSAT scores. As illustrated in Table 12, EAGAA demonstrated marked improvements in emotional alignment, conversational coherence, and task completion efficacy.
Table 12. Comparative performance metrics: EAGAA versus baseline systems.
Metric |
Base B (Rule) |
Base A (RAG) |
EAGAA (Ours) |
CSAT (5-point scale) |
3.98 |
4.02 |
4.85 |
First-Contact Resolution Rate |
68% |
72% |
90% |
Emotional Alignment |
54% |
61% |
92% |
Conversational Coherence |
0.72 |
0.85 |
0.94 |
This systematic comparison reveals that while Baseline A’s generative capabilities provided a foundation for coherent dialogue, the integration of real-time emotion analysis and adaptive response modulation in EAGAA produced statistically significant enhancements in user-perceived empathy and support effectiveness (p < 0.001 for all paired comparisons).
5.3. Batch Processing for Sentiment Analysis
EAGAA initially processed each message independently, which is sufficient for low-traffic, text-only applications. However, to improve latency and throughput in higher-load scenarios, batch processing was introduced. While currently applied solely to textual data, this architecture is designed with extensibility in mind, anticipating future support for multimodal inputs such as audio or visual sentiment.
We compare three batching strategies:
Token Threshold: Accumulate inputs until a fixed token count (e.g., 500) is reached. Pros: efficient for many short messages. Cons: may split sentences mid-stream, requiring sentence boundary detection (SBD) to preserve coherence.
Input Count: Group a fixed number of messages (e.g., 3) regardless of length. Pros: preserves complete utterances. Cons: unpredictable resource use for variable-length inputs.
Hybrid: Flush when either token or input limits are met. Pros: combines strengths of both methods. Cons: added complexity in managing dual thresholds and handling boundary cases.
Token-based batching is well-suited for rapid chat scenarios with brief messages, while input-based batching is preferable for semantically rich inputs such as transcripts. Hybrid batching strikes a balance between responsiveness and contextual integrity, making it a versatile choice. Time-based flushing (e.g., every 300 ms) was rejected due to interaction variability and the overhead of managing asynchronous triggers.
5.4. Domain Adaptation via Pre-Training and Fine-Tuning
Generic sentiment models lack customer‐service nuance. We further pre-train on annotated dialogue corpora reflecting domain-specific states (frustration, urgency, satisfaction), then fine-tune with class-balancing (weighted loss or resampling) to address skew toward neutral/positive labels. This adapts the model to subtle cues, like low‐volume anger or polite distress, enhancing emotional precision. Consequently, EAGAA delivers more contextually aware and empathetic responses, improving both user satisfaction and resolution effectiveness.
6. Conclusion and Future Work
6.1. Conclusion
We introduced EAGAA, an empathetic RAG-based agent that fuses emotion analysis, real-time memory, and dynamic tone modulation to elevate customer support. Evaluations across simulated sessions and a 200-user survey demonstrated:
F1 = 0.77 on multi-class emotion classification using the EmotionLines dataset and 98.6% top-1 retrieval accuracy.
Median latency remained under 4 seconds with zero error rate at 50 queries per second (QPS) and stable throughput under stress.
Achieved a CSAT of 4.85/5 and a 90% resolution rate, outperforming both RAG-only and rule-based systems by over 20%.
EAGAA’s modular design unifies emotion classification, bias mitigation, multilingual handling, and urgency detection within generative dialogues, setting a new standard for scalable, human‐like AI support (Table 13).
Table 13. Comparative analysis of existing techniques versus EAGAA for emotion-aware AI.
Key Focus Area |
Approach |
Strengths |
Limitations |
Advancements |
Gaps |
Emotion Classification |
Lexicons/Transformers |
Interpretability/SOTA |
Sarcasm failure/Compute |
Hybrid models
[13] [16] |
Code-switch support |
Bias Mitigation |
Classical ML/LIME |
Fair audits |
Surface features/Accuracy trade-off |
Adversarial debiasing, LRP [18] |
Real-time frameworks |
Cross-Cultural |
mBERT/EmoPipe |
Transfer learning/89% acc. |
Western bias |
Multilingual corpora [9] |
Low-resource languages |
Real-Time |
GRUs/Co-Design |
Seq. modeling/50% latency cut [12] |
Long-context latency |
Lightweight co-trained LLMs |
Edge deployment |
Privacy |
Federated Learning |
Data locality |
Accuracy drop |
Synthetic data [23] |
Human-in-loop validation |
Explainability |
Attention/LIME |
Token insights |
Post-hoc only |
Emotion AWARE (93% F1) [28] |
Modular APIs |
Anthropomorphism |
HallucinationGuard |
Tone-deaf reduction |
Over-trust |
Transparent grounding [28] |
Cultural guardrails |
Unified Session |
EAGAA |
Integrates tone, urgency, multilingual switching, and RAG; F1 = 0.77; CSAT = 4.85/5; 98.6% retrieval; perfect dialogue accuracy; Outperformed standard RAG and rule-based baselines by a margin of +20%. |
6.2. Future Work
Key extensions include:
Hybrid Emotion Models: Combine lexicon and transformer methods for sarcasm and idiom handling.
Lightweight LLMs: Employ TinyBERT or on-device co-designed models for sub-100 ms end-to-end latency.
Federated Personalization: Adapt models per locale without sharing raw data, balancing privacy and accuracy.
Ethical Safeguards: Develop real‐time bias detectors and culturally adaptive guardrails to prevent over-trust and tone-deaf responses.
Acknowledgments
The authors express their sincere gratitude to Dr. Vijay K. Madisetti from the School of Electrical and Computer Engineering at Georgia Institute of Technology for his invaluable guidance and support throughout this research. His extensive expertise in the field of Large Language Models, combined with his insightful feedback, significantly enhanced the quality and depth of this work. His commitment to academic excellence and innovative research approaches has been instrumental in shaping the direction and successful completion of this project.