The model has 12 billion parameters and uses FP8, so 1 byte each. With some work...

		alecco on July 18, 2024 \| parent \| context \| favorite \| on: Mistral NeMo The model has 12 billion parameters and uses FP8, so 1 byte each. With some working memory I'd bet you can run it on 24GB. > Designed to fit on the memory of a single NVIDIA L40S, NVIDIA GeForce RTX 4090 or NVIDIA RTX 4500 GPU