How to Run Llama 2 on a Linux Server

Meta released Llama 2 in July 2023 as the first genuinely capable open-source large language model. Running it on your own server keeps data private and eliminates API costs.

System Requirements

# Minimum requirements by model size
Llama 2 7B  (quantised Q4): 8GB  RAM, 4GB  VRAM (optional)
Llama 2 13B (quantised Q4): 16GB RAM, 8GB  VRAM (optional)
Llama 2 70B (quantised Q4): 48GB RAM, 24GB VRAM (optional)

Step 1 — Install Dependencies

sudo apt update
sudo apt install python3-pip python3-venv git -y
pip3 install torch --index-url https://download.pytorch.org/whl/cpu
pip3 install transformers accelerate sentencepiece --break-system-packages

Step 2 — Install llama.cpp (Fastest Method)

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j4

Step 3 — Download Llama 2 Model

# Download quantised model (smaller, faster)
wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf

Step 4 — Run the Model

# Interactive chat
./llama-cli -m llama-2-7b-chat.Q4_K_M.gguf -p "You are a helpful server admin assistant." -i

# API server mode
./llama-server -m llama-2-7b-chat.Q4_K_M.gguf --host 0.0.0.0 --port 8080

Step 5 — Test the API

curl http://localhost:8080/completion   -H "Content-Type: application/json"   -d '{"prompt": "Explain nginx rate limiting in simple terms", "n_predict": 200}'

Step 6 — Run as Systemd Service

sudo nano /etc/systemd/system/llama.service

[Unit]
Description=Llama 2 API Server
After=network.target

[Service]
ExecStart=/home/ubuntu/llama.cpp/llama-server -m /models/llama-2-7b-chat.Q4_K_M.gguf --host 127.0.0.1 --port 8080
Restart=always
User=ubuntu

[Install]
WantedBy=multi-user.target

sudo systemctl enable llama --now

Conclusion

Llama 2 on your own server provides private AI inference at zero ongoing cost. Our team configures AI server environments including GPU optimisation and model serving.