SAINE: Scientific Annotation and Inference Engine of Scientific Research
Abstract: We present SAINE, an Scientific Annotation and Inference ENgine based on a set of standard open-source software, such as Label Studio and MLflow. We show that our annotation engine can benefit the further development of a more accurate classification. Based on our previous work on hierarchical discipline classifications, we demonstrate its application using SAINE in understanding the space for scholarly publications. The user study of our annotation results shows that user input collected with the help of our system can help us better understand the classification process. We believe that our work will help to foster greater transparency and better understand scientific research. Our annotation and inference engine can further support the downstream meta-science projects. We welcome collaboration and feedback from the scientific community on these projects. The demonstration video can be accessed from https://youtu.be/yToO-G9YQK4. A live demo website is available at https://app.heartex.com/user/signup/?token=e2435a2f97449fa1 upon free registration.
- Marco Campos. 2020. Yake: Yet another keyword extractor. https://github.com/LIAAD/yake.
- Sahil Chaddha. 2020. Python implementation of rapid automatic keyword extraction using nltk. https://csurfer.github.io/rake-nltk/_build/html/index.html.
- Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
- A review of annotation classification tools in the educational domain. Open Computer Science, 9(1):299–307.
- Maarten Grootendorst. 2020. Keybert: Minimal keyword extraction with bert.
- OpenAI. 2023. Gpt-4 technical report.
- Hierarchical classification of research fields in the "web of science" using deep learning. arXiv preprint arXiv:2302.00390.
- Keyword extraction in scientific documents. arXiv preprint arXiv:2207.01888.
- An overview of microsoft academic service (MAS) and applications. In Proceedings of the 24th international conference on world wide web, pages 243–246. ACM.
- Label Studio: Data labeling software. Open source software available from https://github.com/heartexlabs/label-studio.
- TogetherComputer. 2023. OpenChatKit: An Open Toolkit and Base Model for Dialogue-style Applications.
- Llama: Open and efficient foundation language models.
- Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396–413.
- A review of microsoft academic services for science of science studies. Frontiers in Big Data, 2.
- Judging llm-as-a-judge with mt-bench and chatbot arena.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
