---
title: 'Sentimental LIAR: Multimodal Deception Detection'
url: https://www.emergentmind.com/topics/sentimental-liar
type: topic
---

# Sentimental LIAR: Multimodal Deception Detection

Sentimental LIAR systems constitute a research paradigm and set of technical implementations aiming to verify the truthfulness of emotional and factual claims in human–machine interaction, typically by leveraging multimodal signals such as text, voice, and facial cues. The unifying goal is to build automated models that can assess whether a speaker or writer's manifest statements align with inner beliefs or underlying affect, with use cases spanning fake news classification, conversational honesty detection, and emotion-consistency checking in both synchronous (e.g., live chat, phone calls) and asynchronous (e.g., social media) settings [2009.01047, 2102.11318, 2107.11175]. The field integrates methods from speech processing, deep learning, sentiment and emotion analysis, and deception detection.

## 1. Multimodal and Unimodal Sentimental LIAR Architectures

Sentimental LIAR approaches diverge into unimodal and multimodal variants, each targeting different channels of deception or insincerity.

- **Unimodal (Acoustic) Systems**: Thaler et al. [2107.11175] introduce a paradigm where only the spectral properties of the audio signal are used. Speech is transformed into Mel-Frequency Cepstral Coefficients (MFCCs), with sequential post-processing (windowing, STFT, Mel-scale filtering, DCT), before being ingested by a hybrid CNN–LSTM architecture. This model eschews textual semantics, prosody-linguistics, or physiological signals.
- **Multimodal (Text + Face) Systems**: Kulharni and Sanyal et al. [2102.11318] (“Lie-Sensor”) develop a two-stream architecture, where a miniXception CNN classifies facial emotions, and an SVM maps text to discrete emotion labels (Happiness, Sadness, Surprise, Hate). Consistency between modalities is used as a proxy for honesty; disagreement flags potential emotional deception.

A characteristic fusion strategy is simple logical comparison (e.g., $y_\text{face} = y_\text{text}$ for honesty), rather than probabilistic or late fusion.

## 2. Feature Extraction and Representation

Technical pipelines begin with targeted feature extraction, closely tied to the specific modality:

- **Speech Features**: In [2107.11175], MFCCs serve as primary features. The procedure involves:
  1. Signal segmentation using empirically chosen hop sizes (e.g., 186 ms).
  2. Short Time Fourier Transform (STFT) with Hamming window:
     $$
     X_m(k) = \sum_{n=0}^{N-1} x[n+mH]\,w[n]\,e^{-j2\pi kn/N}
     $$
  3. Mel-scale filterbank mapping:
     $$
     \mathrm{Mel}(f) = 2595\log_{10}\left(1+\frac{f}{700}\right)
     $$
  4. Logarithm and discrete cosine transform for MFCC:
     $$
     c_{m,n} = \sum_{\ell=1}^K \log S_m(\ell)\cos\left[n(\ell-\tfrac12)\tfrac{\pi}{K}\right], \quad n=1\ldots M
     $$
  Both 13 and extended 40 MFCC sets are explored, the latter including delta and delta-delta coefficients.

- **Facial Features**: [2102.11318] adopts $48\times48$ pixel grayscale facial images processed through a miniXception architecture, with a pipeline of initial $3\times3$ convolution, four or more residual depthwise-separable blocks, global average pooling, and 4-class softmax output.

- **Text Features**: For textual claims (e.g., in fake news detection), Sentimental LIAR [2009.01047] fuses BERT-Base contextual embeddings ($h_\text{cls}$, 768-dim) with precise sentiment (Google Cloud NLP) and emotion features (IBM Watson NLU: anger, disgust, fear, joy, sadness), plus speaker-credit history from Politifact. Feature vectors can reach 779 dimensions, concatenating all channels for downstream modeling.

## 3. Model Architectures and Training Protocols

Distinct neural and classical models are utilized, depending on use case and modality:

- **CNN-LSTM Hybrid for Speech**: In [2107.11175], MFCC sequences are processed with a time-distributed 1D convolutional layer (ReLU activation), then flattened and passed to an LSTM with tanh activation. Output is given by a dense layer with sigmoid, optimized via binary cross-entropy loss:
  $$
  \mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \ln \hat{y}_i + (1-y_i)\ln(1-\hat{y}_i) \right]
  $$
  Optimization uses ADAM ($\beta_1=0.9$, $\beta_2=0.999$); hyperparameters (filters, kernel size, LSTM units) are grid-searched with three-fold cross-validation.

- **Feed-Forward and CNN on BERT Embeddings**: In [2009.01047], BERT-Base outputs are passed through either a feed-forward network (multiple dense layers, ReLU activation, sigmoid output) or a 1D-CNN (for sequence-style modeling of embeddings plus metadata), with Adam optimizer and early stopping to avoid rapid overfitting.

- **miniXception CNN for Face + SVM for Text**: [2102.11318] employs miniXception for FER-2013 face images, trained with categorical cross-entropy and Adam optimizer (learning rate $10^{-3}$), and an SVM for text (linear kernel, hand-tuned $C\sim1.0$, feature extraction via TF-IDF or count vectors, lemmatization and de-duplication).

## 4. Experimental Data and Evaluation Metrics

- **Speech Conviction Data**: Thaler et al. [2107.11175] used 38 original debating-club recordings (20 “lying”, 18 “truthful”) from German speakers, expanded by augmentation to 304 samples. Evaluation splits were 70% train / 30% test, using accuracy, precision, recall, and $F_1$-score.

- **Social-Text Fake Claim Data**: [2009.01047] extends the LIAR dataset with sentiment and emotion features; 12,836 statements labeled as “fake” or “genuine,” additional metadata (speaker, context, speaker’s historical veracity). Test accuracy and macro-F1 are primary metrics.

- **Facial and Chat Data**: [2102.11318] uses FER-2013 for face classification (reduced to four emotion classes) and a Kaggle Twitter dataset for text (remapped to same four emotion labels). End-to-end performance on live chat is measured by accuracy, precision, and recall.

Typical evaluation metrics included:
$$
\begin{align*}
\text{Accuracy} &= \frac{TP+TN}{TP+TN+FP+FN} \\
\text{Precision} &= \frac{TP}{TP+FP} \\
\text{Recall} &= \frac{TP}{TP+FN} \\
F_1 &= 2\frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
\end{align*}
$$

## 5. Performance Benchmarks and Failure Modes

Key experimental findings:

- [2107.11175]: The best 40-MFCC CNN-LSTM model attained 98.91% accuracy, 100% precision, 98.15% recall, and $F_1$ of 99.05% in speaker-conviction detection. Using only 13 MFCCs, best accuracy dropped to 72.5%.
- [2009.01047]: BERT-Base+CNN on Sentimental LIAR reached 70% accuracy and macro-F1 0.6370, outperforming previous benchmarks (e.g., 27.4% for CNN on 6-way LIAR).
- [2102.11318]: The live multimodal “Lie-Sensor” system achieved composite accuracy of ~73% in chat settings.

Across studies, inclusion of emotion features notably increased performance—adding emotion intensities boosted macro-F1 by $\sim$12 points in fake news detection [2009.01047].

Key limitations and failure sources:
- Small, homogeneous corpora (e.g., the speech-conviction dataset comprised only 38 German speakers).
- Absence of multimodal or cross-lingual data for robust generalization.
- Sensitivity to sarcasm and irony, especially in facial/text mismatches.
- Overfitting, particularly evident in high-capacity speech models with extended MFCCs.
- Potential for social desirability bias in self-reported “belief” labels.

## 6. Application Domains and Deployment Considerations

Practical deployments target:
- **Contact centers**: Screening doubtful insurance claims or fraud by detecting conviction in remote speech [2107.11175].
- **Recruitment and vetting**: Real-time dishonesty assessment during interviews.
- **Social media**: Automated fact-checking and fake-claim detection using sentiment–emotion-augmented BERT classifiers [2009.01047].
- **Chat platforms**: Real-time consistency checking between facial and textual expressions in messaging via browser plugins or microservices, with latency budget $<$60ms for end-to-end feedback [2102.11318].

Deployment architectures often feature browser-embedded inference (e.g., TensorFlow-Lite for face, scikit-learn/libsvm for text SVM), real-time face detection with OpenCV Haar cascades, and API-based licensing for control and privacy assurance.

## 7. Challenges, Open Questions, and Future Directions

Major proposed research directions include:
- Scaling to larger, more culturally and linguistically diverse datasets to improve model robustness [2107.11175, 2009.01047].
- Incorporation of additional modalities (audio, facial, semantic, physiological) for true multimodal fusion [2102.11318].
- Fine-tuning emotion/sentiment detection methods on discourse-specific corpora, rather than relying on generic APIs [2009.01047].
- Analytical studies to determine which subspace of MFCCs or which emotion signal dimensions most robustly encode deception or conviction.
- Development of explicit sarcasm and irony detectors to mitigate degradation from incongruent affect [2102.11318].

There remains an open question regarding the minimal temporal or textual context needed for reliable live implementation, and the social/ethical implications of automated deception detection in sensitive human interactions.

---

**References**:  
- “Using NLP to analyze whether customer statements comply with their inner belief” [2107.11175]  
- “Lie-Sensor: A Live Emotion Verifier or a Licensor for Chat Applications using Emotional Intelligence” [2102.11318]  
- “Sentimental LIAR: Extended Corpus and Deep Learning Models for Fake Claim Classification” [2009.01047]

Source: https://www.emergentmind.com/topics/sentimental-liar