SignCol: Open-Source Software for Collecting Sign Language Gestures (1911.00071v1)

Abstract: Sign(ed) languages use gestures, such as hand or head movements, for communication. Sign language recognition is an assistive technology for individuals with hearing disability and its goal is to improve such individuals' life quality by facilitating their social involvement. Since sign languages are vastly varied in alphabets, as known as signs, a sign recognition software should be capable of handling eight different types of sign combinations, e.g. numbers, letters, words and sentences. Due to the intrinsic complexity and diversity of symbolic gestures, recognition algorithms need a comprehensive visual dataset to learn by. In this paper, we describe the design and implementation of a Microsoft Kinect-based open source software, called SignCol, for capturing and saving the gestures used in sign languages. Our work supports a multi-language database and reports the recorded items statistics. SignCol can capture and store colored(RGB) frames, depth frames, infrared frames, body index frames, coordinate mapped color-body frames, skeleton information of each frame and camera parameters simultaneously.

• The paper introduces SignCol which uses multi-modal capture with Microsoft Kinect to gather comprehensive sign language gesture data.
• It features real-time database management and statistics reporting for balanced and detailed dataset creation.
• The software employs a MVVM architecture with SQLite to support rigorous data handling, enhancing research in sign language recognition.

Overview of "SignCol: Open-Source Software for Collecting Sign Language Gestures"

The paper introduces SignCol, an open-source tool designed for capturing and storing sign language gestures. This tool leverages the capabilities of the Microsoft Kinect sensor to collect a comprehensive dataset that supports the development of robust sign language recognition systems. The authors focus on efficiently managing the complexities and variabilities found within different sign language systems.

Key Contributions

SignCol distinguishes itself through several notable features that enhance the collection and management of visual data for sign language gestures:

1. Multi-Modal Data Capture: The software utilizes Microsoft Kinect to collect various data types simultaneously, including RGB frames, depth frames, infrared frames, body index frames, and skeleton information. This extensive data capture supports multifaceted analysis and learning algorithms in gesture recognition.
2. Database Management and Statistics Reporting: SignCol not only captures data but also provides database management capabilities. It visualizes statistics for gesture categories in real-time, enabling users to identify and fill gaps in the dataset, ensuring a balanced representation of different signs and gestures.
3. Comprehensive Definition and Customization: The software allows users to define languages, sign items, performers, and classify these items into eight distinct categories, ranging from individual numeric and alphabetic signs to complex word and sentence gestures. This classification system reflects the diverse nature of sign languages and sets a framework for managing collected data efficiently.

Technical Implementation

Developed using C# and WPF, SignCol employs a Model-View-ViewModel (MVVM) architecture, incorporating SQLite for database management. The implementation supports rigorous data handling, catering to researchers requiring precise and comprehensive datasets for training sign recognition algorithms.

Implications for Research and Practice

The release of SignCol as an open-source software tool is poised to impact both theoretical and practical domains within sign language recognition:

• Theoretical Implications: By enabling the collection of diverse and rich datasets, SignCol supports the development of more accurate and adaptable recognition algorithms. It addresses the need for representing a wide range of gestures, which is pivotal in achieving high accuracy rates in automatic gesture recognition systems.
• Practical Applications: SignCol facilitates the creation of assistive technologies aimed at improving communication for individuals with hearing impairments. By lowering the barrier to acquiring datasets, more researchers and developers can engage in this field, potentially accelerating advances in real-world applications.

Future Directions

Future developments might focus on enhancing the multi-device and multi-view capabilities of SignCol. Incorporating additional sensors or improving the synchronization of data from multiple perspectives could further enhance the robustness of collected datasets. Such advancements could lead to improved sign recognition accuracy and broader applicability in different environmental and linguistic contexts.

In summary, the paper details the architecture and capabilities of SignCol, providing the research community with a valuable tool for advancing the field of sign language recognition. Its comprehensive, multi-modal data capture, coupled with robust database management, presents a formidable foundation for future research endeavors in this domain.