I don't have much experience but I played around with a USB Coral TPU. I did not try it with an LLM, but with used some image recognition stuff. Seemed OK, but wasn't as intense as an LLM would be. You'd probably have better luck with the Jetson you mentioned, or with one of Coral's more powerful options.
I don't have much experience but I played around with a USB Coral TPU. I did not try it with an LLM, but with used some image recognition stuff. Seemed OK, but wasn't as intense as an LLM would be. You'd probably have better luck with the Jetson you mentioned, or with one of Coral's more powerful options.
Thanks!
Seen Flowise/Dify/Node-RED; specifically curious about compiled-to-binary patterns for on-device LLMs.