Fine tuning your own llama locally (ft. rem).

how you can give birth to your own friend and watch it learn from chat history, all on your gpu - explained like you are 5! (i mean, not 5 literally)

1. files
2. installing the sloth
3. making sure ur not gpu poor
4. cheating on your friend
5. download llama
6. finetuning
7. inference
8. merge and unload
9. convert to ollama model
10. run the model

!!! files 📂

follow the process and run the files as i say if ydk tf is happening

link to gist :>

link to the repo :>

from now on, we will use this file structure i made up using my last 3 brain cells

FRIENDBOT/
├─ models/
│  ├─ DeepSeek-R1-Distill-Llama-8B/
│  ├─ llama-2-7b-chat/
│  ├─ Mistral-7B-Instruct-v0.3/
│  └─ Qwen3-4B-Instruct-2507/
├─ datasets/
│  ├─ jsonl/
│  │  ├─ training_data_simple.jsonl
│  │  └─ training_data.jsonl
│  ├─ whatsapp_txt/
│  │  ├─ WhatsApp Chat with A.txt
│  │  ├─ WhatsApp Chat with B.txt
│  │  └─ WhatsApp Chat with C.txt
│  └─ scripts/
│     └─ whatsapp-to-alpaca/
│        ├─ simple_whatsapp_to_alpaca.py
│        └─ whatsapp_to_alpaca.py
├─ scripts/
│  ├─ llama.cpp/
│  ├─ finetuning.ipynb
│  └─ merge.ipynb
├─ venv/
└─ .gitignore

1. 🦥 install the ✨lovely✨ sloth with some protection

andd... mommy hopes you are using arch on wsl with zsh and python 3.11!!!! ☺️☺️

tldr; run the bash script and trust me babe

2. open a notebook and check the stuff 💫 (make sure ur not gpu poor)

name it finetuning.ipynb (its in the repo dw)

3. lets cheat on your friend 😈

then run

python datasets/scripts/whatsapp-to-alpaca/simple_whatsapp_to_alpaca.py

i was playing around with the original file so the simple one is suggested :P

4. lets download the 4 bit 7b prostitu- i mean... llama-2-7b-chat

now run this script to download the model, remember when you login, make the read access token and paste it in the terminal!!

huggingface-cli login && huggingface-cli download meta-llama/Llama-2-7b-chat --local-dir models/llama-2-7b-chat/llama-2-7b-chat --local-dir-use-symlinks False

if your review takes forver, you can replace llama with mistralai/Mistral-7B-Instruct-v0.3 and you are good to go.

this megatron is gonna take forever to download

5. til then, lets proceed to the next steps

so you seem to be interested?>

not so long... now we load the weights

importing the dataset now (>w<)

some prompting stuff idk wadat means

some prompting stuff x2

we are close! prepare our weapon!!

just set the num_train_epochs=1 when you wanna make it really learn from the dataset, we're just giving it a few lines for now

TRAIN THE DRAGONN 🐉🔥🔥🔥

close android studio 😭

6. lets make an inference on our own GGGGPPPUUUUUU and- its- its workinfg!!!!

7. m... m.... merge and unload these weights//

open merge.ipynb, these are cute imports and variables

now we load the base model and the training weights

these are complete weights, and dont forget loading its tokenizer too :3

we will then merge the trained weights to these

note: use the low_cpu_mem_usage if u got low vram (<30gb) else it crashes, also this offloads the weights and memory to disk so NEVER USE AN HDD 😭💀

rethink your life choices

ssd and mem go brrrr 💀

done

oh no my fresh new ssd 😢

confirm this folder structure, if u dont get this, ask me out <3

llama-2-7b-chat/
├─ checkpoints_finetuned/
├─ finetuned_weights/
│  ├─ adapter_config.json
│  ├─ adapter_model.safetensors
│  ├─ chat_template.jinja
│  ├─ README.md
│  ├─ special_tokens_map.json
│  ├─ tokenizer_config.json
│  ├─ tokenizer.json
│  └─ tokenizer.model
├─ llama-2-7b-chat/
└─ merged_weights/
   ├─ chat_template.jinja
   ├─ config.json
   ├─ generation_config.json
   ├─ model-00001-of-00003.safetensors
   ├─ model-00002-of-00003.safetensors
   ├─ model-00003-of-00003.safetensors
   ├─ model.safetensors.index.json
   ├─ special_tokens_map.json
   ├─ tokenizer_config.json
   ├─ tokenizer.json
   └─ tokenizer.model

8. convert ur child into an to ollama model

git clone https://github.com/ggerganov/llama.cpp scripts/llama.cpp

then convert to ollama's bff i mean, gguf format

python scripts/llama.cpp/convert_hf_to_gguf.py models/llama-2-7b-chat/merged_weights --outtype q8_0

output should be like this

grab the output gguf and move it to a new ollama_outputs folder (this helps)

now get the modelfile from the gist and save it to models/llama-2-7b-chat/ollama_outputs/Modelfile (if ur using mistral u need to change the file name in modelfile)

install ollama if you dont have it on wsl

curl -fsSL https://ollama.com/install.sh | sh

then create ollama model from the gguf and modelfile we just got ^_^

ollama create friendbot -f models/llama-2-7b-chat/ollama_outputs/Modelfile

9. run the ollama model we just made yayayayayay

though it seems just as a normal model right now, changing the num_train_epochs to 1 will get your shit done, atlas you fine tuned your first llama!!

congratss, mom will be so proud of you :D!!!

i am not an ml person and i accept that i may not have enough knowledge, but i tried my best to understand and to make you understand from what i learnt, im open to fixes and suggestions on this article and lets have a great day together <3

til then, see ya later!

original x article: https://x.com/forloopcodes/status/2012110089931690217