Serious question: is fine-tuning open source models actually worth the hardware cost?

I was at a meetup last week and overheard two devs arguing about this... one guy said he spent $1,200 on GPUs just to fine-tune a 7B model for his startup's customer bot, and it still hallucinated half the time. The other dude claimed he got better results just using better prompts with GPT-4. Made me wonder if we're all chasing a trend that's overhyped for most use cases. What's your break-even point where fine-tuning makes sense?

2 comments

2 Comments

daniel_rivera5d ago

Nah, fine-tuning is a waste - better prompting and RAG beats it every time for most teams.

thea6925d ago

Wait, wait, wait - are you saying fine-tuning is totally useless? I mean, I get that RAG works great for pulling specific data from documents, but what about when you need a model to actually change its behavior for a specific task? Like, say you want it to always write in a certain tone for customer support. Prompting can only do so much before you hit a wall. RAG is amazing for knowledge retrieval, but fine-tuning teaches the model the right way to think. Has anyone actually tried fine-tuning a small model for a narrow task and seen it fail?