Petals Run lets you run large language models like Llama 2, Stable Beluga 2, Guanaco-65B, or BLOOM-176B at home, collaboratively. You load a small part of the model, then join others serving the different parts to run inference or fine-tuning.
It is up to 10x faster than offloading, making it suitable for building chatbots and interactive apps. It also offers flexibility beyond classic language model APIs, allowing fine-tuning, custom paths, and viewing hidden states.