Classification of Audio Segments in Call Center Recordings using Convolutional Recurrent Neural Networks

Abstract: Detailed statistical analysis of call center recordings is critical in the customer relationship management point of view. With the recent advances in artificial intelligence, many tasks regarding the calculation of call statistics are now performed automatically. This work proposes a neural network framework where the aim is to correctly identify audio segments and classify them as either customer or agent sections. Accurately identifying these sections gives a fair metric for evaluating agents' performances. We inherited the convolutional recurrent neural network (CRNN) architecture commonly used for such problems as music genre classification. We also tested the same architecture's performance, where the previous class information and the gender information of speakers are also added to the training data labels. We saw that CRNN could generalize the training data and perform well on validation data for this problem with and without the gender information. Moreover, even the training was performed using Turkish speech samples; the trained network was proven to achieve high accuracy for call center recordings in other languages like German and English.