Introduction

llama.cpp is a wonderful project for running llms locally on your system. It is lightweight and provide state-of-the-art performance. It comes with GPU offloading support allowing you to use your GPU capabilities to run llms.

I am personally using it for running llms on my arch system and I found it better than ollama in terms of performance, while trying to set this it up, I found their documentation confusing and there were no guides specifically for arch linux, that's why I decided to write an article after figuring things out.
So, lets gets started with it.

Guide

1. lets start by cloning the repo and cd into that

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

1. After that we need to build this, it can be done by using make, but like me if you have nvidia GPU then for using it to for offloading you will need to build it with cuBLAS and for that we need cuda toolkit, You can download it from your Linux distro's package manager. (e.g. apt install nvidia-cuda-toolkit) here I am using aur helper, you can also install it manually.

paru -S cuda

now to build project with cuda run

make GGML_CUDA=1

Using CMake:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

It might take a while, after build is finished we can now finally run llms.

How to use

In llama.cpp folder you will find a executable named llama-server and llama-cli
which can be used to run the model
for cli mode just run

./llama-cli -m path-to-model -n no-of-tokens -ngl no-of-layers-to-offload-to-gpu

And for server run this

./llama-server -m path-to-model -n no-of-tokens -ngl no-of-layers-to-offload-to-gpu

You can now open your browser and in the URL section type http://localhost:8080/ and a web UI will appear.

Now you can have fun with your local llm.
Hope you will find this article helpful.