Tri-modal chatbot for assessing patients' mental states through text, voice, and facial expression analysis, achieving 87% user satisfaction in clinical evaluations. Published at ACM ICAISS-2024.
This research project built a tri-modal chatbot system designed to assist in psychological counselling by analyzing patients' mental states through three simultaneous input channels: text conversations, voice tone analysis, and facial expression recognition. The system provides a more holistic assessment than text-only chatbots by capturing emotional signals across modalities.
The text modality uses NLP techniques to analyze sentiment, detect distress indicators, and identify cognitive patterns in patient responses. The voice modality processes speech characteristics including tone, pace, pitch variations, and hesitation patterns that correlate with emotional states. The facial expression modality uses computer vision to detect micro-expressions and emotional indicators during video sessions.
The three modalities are fused using a weighted ensemble approach that produces a composite mental state assessment. This multi-modal fusion allows the system to detect discrepancies (for example, when a patient says they're 'fine' but their voice trembles and facial expressions show distress), providing counsellors with deeper insight than any single modality alone.
The system achieved 87% user satisfaction in clinical evaluations and was published at ACM ICAISS-2024 (International Conference on Artificial Intelligence and Smart Systems).
class TriModalAssessor:
"""Fuses text, voice, and facial signals for mental state assessment."""
def __init__(self):
self.text_analyzer = SentimentAnalyzer()
self.voice_analyzer = VoiceEmotionDetector()
self.face_analyzer = FacialExpressionClassifier()
def assess(self, text: str, audio: np.ndarray, frame: np.ndarray):
text_score = self.text_analyzer.analyze(text)
voice_score = self.voice_analyzer.analyze(audio)
face_score = self.face_analyzer.classify(frame)
# Detect modality discrepancies (e.g., "I'm fine" + distressed voice)
discrepancy = abs(text_score.valence - voice_score.valence)
composite = MentalStateAssessment(
text=text_score,
voice=voice_score,
facial=face_score,
distress_level=weighted_fusion(text_score, voice_score, face_score),
modality_conflict=discrepancy > 0.4,
)
return composite