LitmusKt: Concurrency Stress Testing for Kotlin

Abstract: We present LitmusKt - the first tool for litmus testing concurrent programs in Kotlin. The tool's novelty also lies in the fact that Kotlin is a multiplatform language, i.e., it compiles into multiple platforms, which means that the concurrency has to be tested on several of them. Our tool allows writing litmus tests in a single custom DSL, and these tests are then run in Kotlin/Native and Kotlin/JVM, two main platforms for concurrent programming in Kotlin. Using LitmusKt, we discovered novel bugs in the Kotlin compiler, which we then fixed and they are no longer present. Moreover, LitmusKt was integrated into the CI pipeline for Kotlin. LitmusKt is available on GitHub: https://github.com/JetBrains-Research/litmuskt. The demo is available on YouTube: https://youtu.be/oWCZp_Huwss.

• The paper introduces LitmusKt, the first tool for concurrency litmus testing in multi-platform Kotlin, supporting both Kotlin/Native and Kotlin/JVM.
• LitmusKt employs techniques from tools like jcstress and a domain-specific language to amplify weak memory behaviors and simplify test creation.
• Empirical testing with LitmusKt revealed critical concurrency bugs in the Kotlin/Native compiler on Arm platforms, leading to its integration into the Kotlin development pipeline.

Analyzing LitmusKt: A Concurrency Stress Testing Tool for Kotlin

The paper "LitmusKt: Concurrency Stress Testing for Kotlin" presents LitmusKt as the pioneering tool for concurrency litmus testing focused on the Kotlin programming language. This tool addresses the need for robust testing of concurrent behavior in Kotlin programs, which are unique due to Kotlin's nature as a multi-platform language. Kotlin's ability to compile into multiple platforms—including Kotlin/Native and Kotlin/JVM—necessitates concurrency testing across these environments, given that each platform adheres to different underlying memory models.

Key Contributions and Methodologies

1. Multi-platform Support: LitmusKt is designed to handle litmus tests across Kotlin/Native and Kotlin/JVM, providing critical insights into concurrent behavior on different platforms. This capability is crucial as each platform possesses distinct execution models and optimizations, impacting the concurrency behavior observed.
2. Tool Architecture: The tool incorporates advanced methodologies from existing concurrency testing tools such as jcstress and herdtools/litmus7. It uses these techniques to magnify the occurrence of weak memory behaviors, thus facilitating the detection of uncommon concurrency issues during testing. Notably, for Kotlin/Native, a custom test runner was implemented due to the absence of existing comprehensive tools.
3. Syntax and Usability: LitmusKt introduces a domain-specific language (DSL) that allows developers to write concise and flexible tests. The DSL is structured to simplify the expression of litmus tests while maintaining the capability to specify complex concurrent behaviors.
4. Empirical Findings: The paper details significant findings from running standard litmus tests on diverse hardware architectures (x86 and Arm processors). Notably, the Unsafe Publication (UPUB) tests revealed critical bugs within the Kotlin/Native compiler, particularly unsafe memory behaviors on Arm platforms, which have since been acknowledged and rectified.
5. Integration and Utility: Consequent to the discoveries facilitated by LitmusKt, the Kotlin development team incorporated this tool into the continuous integration pipeline for Kotlin/Native, emphasizing its utility in ongoing development and error detection in concurrency implementations.

Implications and Future Directions

The successful application of LitmusKt underscores the necessity of concurrency testing tools tailored for multi-platform languages like Kotlin. It illustrates the potential discrepancies that can arise when concurrent programs interact with differing memory models. The tool's capability to highlight these differences aids in refining the underlying compilers and language specifications to ensure consistent and safe concurrency across platforms.

Further research directions proposed in the paper include exploring fuzzing techniques to trigger additional compiler optimizations potentially overlooked by current deterministic testing methods. Moreover, addressing limitations in preventing false sharing may enhance the tool's ability to discover subtler concurrency issues.

In conclusion, LitmusKt stands as a significant contribution to the field of concurrency testing in multi-platform languages, setting a precedent for similar testing needs in other languages. The tool's integration into the Kotlin development process underscores its value, and its insights pave the way for deeper exploration into optimizing and securing concurrent program behavior across varying computational environments.