Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 91 tok/s
Gemini 2.5 Pro 47 tok/s Pro
GPT-5 Medium 30 tok/s
GPT-5 High 33 tok/s Pro
GPT-4o 98 tok/s
GPT OSS 120B 483 tok/s Pro
Kimi K2 242 tok/s Pro
2000 character limit reached

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following (2403.03129v2)

Published 5 Mar 2024 in cs.CL

Abstract: With the advancement of LMs, their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. 2024. Minicpm: Unveiling the potential of end-side large language models. OpenBMB Blog.
  2. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Qwen technical report. arXiv preprint arXiv:2309.16609.
  4. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  6. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology.
  7. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701.
  8. A customized text sanitization mechanism with differential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5747–5758.
  9. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377.
  10. Challenges towards the next frontier in privacy. arXiv preprint arXiv:2304.06929.
  11. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Singapore. Association for Computational Linguistics.
  12. Giorgio Franceschelli and Mirco Musolesi. 2023. On the creativity of large language models. arXiv preprint arXiv:2304.00008.
  13. Specializing smaller language models towards multi-step reasoning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10421–10430. PMLR.
  14. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
  15. Specialized language models with cheap inference from limited domain data. arXiv preprint arXiv:2402.01093.
  16. Textbooks are all you need. arXiv preprint arXiv:2306.11644.
  17. Jennifer Haase and Paul HP Hanel. 2023. Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. arXiv preprint arXiv:2303.12003.
  18. Mistral 7b. arXiv preprint arXiv:2310.06825.
  19. Mixtral of experts. arXiv preprint arXiv:2401.04088.
  20. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.
  21. Protecting user privacy in remote conversational systems: A privacy-preserving framework based on text sanitization. arXiv preprint arXiv:2306.08223.
  22. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  23. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR.
  24. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097.
  25. Tuning language models by proxy. arXiv preprint arXiv:2401.08565.
  26. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374.
  27. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv:2310.12962.
  28. MLC team. 2023. MLC-LLM.
  29. Text embeddings reveal (almost) as much as text. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  30. Language model inversion. In The Twelfth International Conference on Learning Representations.
  31. Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117.
  32. CombLM: Adapting black-box language models through small fine-tuned models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2961–2974, Singapore. Association for Computational Linguistics.
  33. Learning interpretable style embeddings via prompting LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15270–15290, Singapore. Association for Computational Linguistics.
  34. Communicative agents for software development. arXiv preprint arXiv:2307.07924.
  35. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  36. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304.11406.
  37. Mitigating hallucinations and off-target machine translation with source-contrastive and language-contrastive decoding. arXiv preprint arXiv:2309.07098.
  38. H2o-danube-1.8 b technical report. arXiv preprint arXiv:2401.16818.
  39. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  40. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  41. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  42. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.
  43. Knowledge fusion of large language models.
  44. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.
  45. Automated evaluation of personalized text generation using large language models. arXiv preprint arXiv:2310.11593.
  46. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  47. Same author or just same topic? towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 249–268, Dublin, Ireland. Association for Computational Linguistics.
  48. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308.12067.
  49. Jason Weston and Sainbayar Sukhbaatar. 2023. System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829.
  50. Privacy-preserving in-context learning for large language models. arXiv e-prints, pages arXiv–2305.
  51. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851.
  52. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.04870.
  53. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  54. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196.
  55. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. arXiv preprint arXiv:2312.02003.
  56. A complete survey on generative ai (aigc): Is chatgpt from gpt-4 to gpt-5 all you need? arXiv preprint arXiv:2303.11717.
  57. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1–37.
  58. CRaSh: Clustering, removing, and sharing enhance fine-tuning without full large language model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9612–9637, Singapore. Association for Computational Linguistics.
  59. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.
  60. TextFusion: Privacy-preserving pre-trained model inference via token fusion. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8360–8371, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  61. TextObfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5459–5473, Toronto, Canada. Association for Computational Linguistics.
Citations (11)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube