Sentimental LIAR: Multimodal Deception Detection

Updated 15 March 2026

Sentimental LIAR is a research paradigm for verifying emotional and factual claims using multimodal inputs such as speech, text, and facial cues.
It employs both unimodal (e.g., MFCC-based CNN-LSTM for speech) and multimodal architectures (e.g., miniXception CNN with SVM for text and face) to detect deception.
Empirical findings show high accuracy in controlled settings, while challenges remain with dataset diversity, sarcasm handling, and cross-modal fusion.

Sentimental LIAR systems constitute a research paradigm and set of technical implementations aiming to verify the truthfulness of emotional and factual claims in human–machine interaction, typically by leveraging multimodal signals such as text, voice, and facial cues. The unifying goal is to build automated models that can assess whether a speaker or writer's manifest statements align with inner beliefs or underlying affect, with use cases spanning fake news classification, conversational honesty detection, and emotion-consistency checking in both synchronous (e.g., live chat, phone calls) and asynchronous (e.g., social media) settings (Upadhayay et al., 2020, Patel et al., 2021, Thaler et al., 2021). The field integrates methods from speech processing, deep learning, sentiment and emotion analysis, and deception detection.

1. Multimodal and Unimodal Sentimental LIAR Architectures

Sentimental LIAR approaches diverge into unimodal and multimodal variants, each targeting different channels of deception or insincerity.

Unimodal (Acoustic) Systems: Thaler et al. (Thaler et al., 2021) introduce a paradigm where only the spectral properties of the audio signal are used. Speech is transformed into Mel-Frequency Cepstral Coefficients (MFCCs), with sequential post-processing (windowing, STFT, Mel-scale filtering, DCT), before being ingested by a hybrid CNN–LSTM architecture. This model eschews textual semantics, prosody-linguistics, or physiological signals.
Multimodal (Text + Face) Systems: Kulharni and Sanyal et al. (Patel et al., 2021) (“Lie-Sensor”) develop a two-stream architecture, where a miniXception CNN classifies facial emotions, and an SVM maps text to discrete emotion labels (Happiness, Sadness, Surprise, Hate). Consistency between modalities is used as a proxy for honesty; disagreement flags potential emotional deception.

A characteristic fusion strategy is simple logical comparison (e.g., $y_\text{face} = y_\text{text}$ for honesty), rather than probabilistic or late fusion.

2. Feature Extraction and Representation

Technical pipelines begin with targeted feature extraction, closely tied to the specific modality:

Speech Features: In (Thaler et al., 2021), MFCCs serve as primary features. The procedure involves:
1. Signal segmentation using empirically chosen hop sizes (e.g., 186 ms).
2. Short Time Fourier Transform (STFT) with Hamming window:
$X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$

Mel-scale filterbank mapping:

$\mathrm{Mel}(f) = 2595\log_{10}\left(1+\frac{f}{700}\right)$
Logarithm and discrete cosine transform for MFCC:

$c_{m,n} = \sum_{\ell=1}^K \log S_m(\ell)\cos\left[n(\ell-\tfrac12)\tfrac{\pi}{K}\right], \quad n=1\ldots M$

Both 13 and extended 40 MFCC sets are explored, the latter including delta and delta-delta coefficients.

Facial Features: (Patel et al., 2021) adopts $48\times48$ pixel grayscale facial images processed through a miniXception architecture, with a pipeline of initial $3\times3$ convolution, four or more residual depthwise-separable blocks, global average pooling, and 4-class softmax output.
Text Features: For textual claims (e.g., in fake news detection), Sentimental LIAR (Upadhayay et al., 2020) fuses BERT-Base contextual embeddings ( $h_\text{cls}$ , 768-dim) with precise sentiment (Google Cloud NLP) and emotion features (IBM Watson NLU: anger, disgust, fear, joy, sadness), plus speaker-credit history from Politifact. Feature vectors can reach 779 dimensions, concatenating all channels for downstream modeling.

3. Model Architectures and Training Protocols

Distinct neural and classical models are utilized, depending on use case and modality:

CNN-LSTM Hybrid for Speech: In (Thaler et al., 2021), MFCC sequences are processed with a time-distributed 1D convolutional layer (ReLU activation), then flattened and passed to an LSTM with tanh activation. Output is given by a dense layer with sigmoid, optimized via binary cross-entropy loss:

$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \ln \hat{y}_i + (1-y_i)\ln(1-\hat{y}_i) \right]$

Optimization uses ADAM ( $\beta_1=0.9$ , $\beta_2=0.999$ ); hyperparameters (filters, kernel size, LSTM units) are grid-searched with three-fold cross-validation.

Feed-Forward and CNN on BERT Embeddings: In (Upadhayay et al., 2020), BERT-Base outputs are passed through either a feed-forward network (multiple dense layers, ReLU activation, sigmoid output) or a 1D-CNN (for sequence-style modeling of embeddings plus metadata), with Adam optimizer and early stopping to avoid rapid overfitting.
miniXception CNN for Face + SVM for Text: (Patel et al., 2021) employs miniXception for FER-2013 face images, trained with categorical cross-entropy and Adam optimizer (learning rate $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 0), and an SVM for text (linear kernel, hand-tuned $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 1, feature extraction via TF-IDF or count vectors, lemmatization and de-duplication).

4. Experimental Data and Evaluation Metrics

Speech Conviction Data: Thaler et al. (Thaler et al., 2021) used 38 original debating-club recordings (20 “lying”, 18 “truthful”) from German speakers, expanded by augmentation to 304 samples. Evaluation splits were 70% train / 30% test, using accuracy, precision, recall, and $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 2-score.
Social-Text Fake Claim Data: (Upadhayay et al., 2020) extends the LIAR dataset with sentiment and emotion features; 12,836 statements labeled as “fake” or “genuine,” additional metadata (speaker, context, speaker’s historical veracity). Test accuracy and macro-F1 are primary metrics.
Facial and Chat Data: (Patel et al., 2021) uses FER-2013 for face classification (reduced to four emotion classes) and a Kaggle Twitter dataset for text (remapped to same four emotion labels). End-to-end performance on live chat is measured by accuracy, precision, and recall.

Typical evaluation metrics included:

$X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 3

5. Performance Benchmarks and Failure Modes

Key experimental findings:

(Thaler et al., 2021): The best 40-MFCC CNN-LSTM model attained 98.91% accuracy, 100% precision, 98.15% recall, and $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 4 of 99.05% in speaker-conviction detection. Using only 13 MFCCs, best accuracy dropped to 72.5%.
(Upadhayay et al., 2020): BERT-Base+CNN on Sentimental LIAR reached 70% accuracy and macro-F1 0.6370, outperforming previous benchmarks (e.g., 27.4% for CNN on 6-way LIAR).
(Patel et al., 2021): The live multimodal “Lie-Sensor” system achieved composite accuracy of ~73% in chat settings.

Across studies, inclusion of emotion features notably increased performance—adding emotion intensities boosted macro-F1 by $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 512 points in fake news detection (Upadhayay et al., 2020).

Key limitations and failure sources:

Small, homogeneous corpora (e.g., the speech-conviction dataset comprised only 38 German speakers).
Absence of multimodal or cross-lingual data for robust generalization.
Sensitivity to sarcasm and irony, especially in facial/text mismatches.
Overfitting, particularly evident in high-capacity speech models with extended MFCCs.
Potential for social desirability bias in self-reported “belief” labels.

6. Application Domains and Deployment Considerations

Practical deployments target:

Contact centers: Screening doubtful insurance claims or fraud by detecting conviction in remote speech (Thaler et al., 2021).
Recruitment and vetting: Real-time dishonesty assessment during interviews.
Social media: Automated fact-checking and fake-claim detection using sentiment–emotion-augmented BERT classifiers (Upadhayay et al., 2020).
Chat platforms: Real-time consistency checking between facial and textual expressions in messaging via browser plugins or microservices, with latency budget $X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}$ 660ms for end-to-end feedback (Patel et al., 2021).

Deployment architectures often feature browser-embedded inference (e.g., TensorFlow-Lite for face, scikit-learn/libsvm for text SVM), real-time face detection with OpenCV Haar cascades, and API-based licensing for control and privacy assurance.

7. Challenges, Open Questions, and Future Directions

Major proposed research directions include:

Scaling to larger, more culturally and linguistically diverse datasets to improve model robustness (Thaler et al., 2021, Upadhayay et al., 2020).
Incorporation of additional modalities (audio, facial, semantic, physiological) for true multimodal fusion (Patel et al., 2021).
Fine-tuning emotion/sentiment detection methods on discourse-specific corpora, rather than relying on generic APIs (Upadhayay et al., 2020).
Analytical studies to determine which subspace of MFCCs or which emotion signal dimensions most robustly encode deception or conviction.
Development of explicit sarcasm and irony detectors to mitigate degradation from incongruent affect (Patel et al., 2021).

There remains an open question regarding the minimal temporal or textual context needed for reliable live implementation, and the social/ethical implications of automated deception detection in sensitive human interactions.

References:

“Using NLP to analyze whether customer statements comply with their inner belief” (Thaler et al., 2021)
“Lie-Sensor: A Live Emotion Verifier or a Licensor for Chat Applications using Emotional Intelligence” (Patel et al., 2021)
“Sentimental LIAR: Extended Corpus and Deep Learning Models for Fake Claim Classification” (Upadhayay et al., 2020)