I earn commissions when you sign up or buy through the affiliate links on this page, at no extra cost to you.
Homelab · Guide · By Mohammed Almuhanna · Updated
Local LLMs vs a Cloud GPU: When to Rent Instead of Buy
I run local models on a 10 GB RTX 3080 in my Proxmox box, Ollama with Open WebUI, usually a Qwen model. Let me be honest up front: a local model on consumer hardware does not come close to GPT-5 or Claude for real work. I keep it for small quick tasks, for privacy, and because I can hammer it all day for free. The wall you hit is VRAM. Once a model is too big for your card, you either buy a bigger GPU you will barely use, or rent one by the hour. This is how I decide.
The wall is always VRAM
A model has to fit its weights in the GPU's memory to run at any real speed. If it does not fit in VRAM, it spills into system RAM and crawls, or it just will not load. So forget how fast your GPU is. The only question that matters is whether the model fits. Most people run models at 4-bit, which is fine for nearly everything, and at that size the rough numbers are:
| Model size (4-bit) | VRAM needed | Runs comfortably on |
|---|---|---|
| 7B to 8B | ~5 to 6 GB | Almost any modern GPU, including my 10 GB 3080 |
| 13B to 14B | ~9 to 10 GB | A 12 GB card; tight on 10 GB |
| 30B to 34B | ~20 to 22 GB | A 24 GB card (RTX 3090, 4090) |
| 70B | ~40 to 48 GB | A 48 GB pro card, or a cloud GPU |
| 100B and up | 80 to 200+ GB | A cloud GPU (H100, H200, MI300X) |
My 3080 runs 7B and 8B models easily, and a 13B if I keep the context short. Past that it falls apart fast. A 70B will not run on consumer VRAM without slow offloading, and the big open models are way past even that. gpt-oss-120b at home on 10 GB? No. Not on a 4090 either. So once you want something bigger than your card holds, you have a choice to make.
When to keep it local
For small quick tasks, local wins. A 7B to 14B model on a card you already own handles a fast rewrite, a quick summary, or a throwaway question all day without you thinking about it. The power is cheap, a few cents an hour and even less where I am, there is no meter ticking, and your prompts never leave your network. So keep it local when the task is small, when the data is sensitive, or when you want to use it as much as you like for free. That is the real reason I run one. Just do not expect it to replace a frontier model on anything serious, because it will not.
When to rent a cloud GPU instead
Some self-hosters treat any use of the cloud as a cop-out. Ignore that. When your hardware cannot do the job, renting a GPU for a few hours is the right call. Rent for the work your card cannot do and that you do not do often enough to justify buying hardware for. Where it pays off:
- Running a model that does not fit. You want to try a 70B or a frontier-size model your VRAM cannot hold. Spin up an 80 GB or 141 GB GPU, run it, shut it down.
- Fine-tuning or training. Training needs far more memory than inference, and you usually do it in bursts, not constantly. Renting a big GPU for a few hours beats owning one that sits idle.
- Batch jobs. You have a one-off pile of documents to process or embeddings to generate. Rent a big GPU, run the batch, delete it.
- Try before you buy. Before spending real money on a bigger card, rent a cloud GPU for an evening, run the larger models you have been eyeing, and find out whether the upgrade is actually worth it to you.
Rent by the hour and delete it when you are done. An H100 runs a couple of dollars an hour, billed by the second, so a four-hour training run costs a few dollars instead of a few thousand. DigitalOcean's GPU Droplets work for this. Pick a GPU, start it, run the job, delete it, and the meter stops. You pay for the work, not for a card sitting in a case.
Two ways people waste money
Two ways people waste money here, and they are opposite mistakes.
Trap one, buying a big card for occasional use. If you would run a 70B model twice a month, do not drop thousands on a 48 GB GPU for it. At a few dollars an hour you can rent that twice a month for years and still spend less, with nothing sitting in a closet losing value. I have wasted real money on hardware that did not do what I assumed it would, so I take this one seriously.
Trap two, running a 24/7 service on a rented GPU. A cloud GPU left on around the clock is brutally expensive, easily one to two thousand dollars a month. If you want an always-on assistant, that is the job your own card is for. Rent for bursts, own for the constant load. Get those backwards and the bill is ugly either way.
How I do it
My daily driver is the local stack. Small and mid models on the 3080 through Ollama and Open WebUI, always on, private, nearly free, for the small stuff. When something does not fit, a model I want to test or a batch job that needs real memory, I start a GPU droplet, do the work, and kill it the same hour, with nothing expensive left sitting around after. The homelab covers the times I just need a quick answer now. The cloud covers the rare times I need a bigger model for a while. They are not competing for the same job.
Common questions
How much VRAM do I need to run a local LLM?
For a 4-bit quantized model, budget roughly 5 to 6 GB for a 7B to 8B model, 9 to 10 GB for a 13B, and around 20 GB for a 30B. A 70B model needs 40 GB or more, which is past consumer cards. Context length adds to that, so leave some headroom.
Is it cheaper to run AI locally or in the cloud?
For models that fit your card and that you use constantly, local is far cheaper. The hardware is a sunk cost and the electricity is pennies an hour. For big models you only need occasionally, renting by the hour beats buying a card that would sit idle. It comes down to how often you actually need the bigger GPU.
Can I run a 70B model on a consumer GPU?
Not comfortably. A 70B model at 4-bit needs around 40 to 48 GB of VRAM, well beyond a 24 GB 4090. You can offload layers to system RAM, but it gets slow fast. For 70B and larger, a rented cloud GPU with 80 GB or more is the practical route.
What is a cloud GPU good for if I already have a homelab?
The jobs your own card cannot do and that you do not do often: running a model too big for your VRAM, fine-tuning or training in bursts, and one-off batch work. You rent the GPU by the hour, run the job, and shut it down, instead of buying expensive hardware for occasional use.