Run dolphin-2.2-yi-34b on IoT Devices (Also works as a Private OpenAI API Server)

The Yi-34B model is an impressive bilingual LLM with a large context length. It is then fine-tuned with the Dolphin dataset to “learn” reasoning and uncensor its biases. The result is an excellent LLM that can be further tuned for personality and alignment.
Using WasmEdge, I am running dolphin-2.2-yi-34b on a Jetson IoT device at 6 tps. Try it on your device and get uncensored LLM answers 👉 secondstate.io/articles/dolph...

[@snapo1750]

i dont consider 18GB vram useage for iot 🙂