Running AI models locally with Ollama

Help for Current Versions of MX
When asking for help, use Quick System Info from MX Tools. It will be properly formatted using the following steps.
1. Click on Quick System Info in MX Tools
2. Right click in your post and paste.
Message
Author
User avatar
Adrian
Developer
Posts: 9164
Joined: Wed Jul 12, 2006 1:42 am

Re: Running AI models locally with Ollama

#21 Post by Adrian »

I agree with other posters, voice-to-text input seems to me like a different application, maybe even part of the Desktop Environment. It's also a bit bratty, you get an amazing tool for free and you complains it doesn't come with golden handles.

User avatar
Melber
Developer
Posts: 1375
Joined: Tue Mar 23, 2021 4:19 pm

Re: Running AI models locally with Ollama

#22 Post by Melber »

Handles? Who needs handles? They're so inefficient. All my doors are fitted with motion detectors...

Back on topic. Would be interested in trying this out, but don't think my aged laptop would be able to handle it. Perhaps also scared to confirm I wasted my time learning my very basic bash skills...

User avatar
MadMax
Posts: 483
Joined: Wed Aug 04, 2021 3:25 pm

Re: Running AI models locally with Ollama

#23 Post by MadMax »

Melber wrote: Sat Aug 03, 2024 12:24 pmPerhaps also scared to confirm I wasted my time learning my very basic bash skills...
You did not. I tried a little more sophisticated script yesterday: the task was to compare a number of random files between two directories to check a random sample for silent corruption - the actual prompt was more specific of course. After a few tries it gave me something what I'd call a "good template". Had to do some work to get it to run and expanded it further since then (added colors to the output, let it count the file size of the checked files implemented a second hash algorithm etc). It didn't work ootb like the simple one before.

BUT I have to admit the template was pretty useful as a starting point. The definition of the input variables and checking for correct inputs was done correctly.
If it ain't broke, don't fix it.
Main: MX 23 | Second: Mint 22 | HTPC: Linux Lite 7 | VM Machine: Debian 12 | Testrig: Arch/FreeBSD 14 | Work: RHEL 8

User avatar
Adrian
Developer
Posts: 9164
Joined: Wed Jul 12, 2006 1:42 am

Re: Running AI models locally with Ollama

#24 Post by Adrian »

llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"

User avatar
oops
Posts: 1998
Joined: Tue Apr 10, 2018 5:07 pm

Re: Running AI models locally with Ollama

#25 Post by oops »

Adrian wrote: Tue Oct 01, 2024 1:05 pm llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"
Right, llama3.2:3b (2.0 GB) is faster than llama3.1:8b (4.7 GB), and 8MB of RAM is enough.
( llama3.2:1b (1.3 GB) also exists for PC with only 2 or 4GB)
Thanks for the info:

https://ollama.com/library/llama3.2
Pour les nouveaux utilisateurs: Alt+F1 pour le manuel, ou FAQS, MX MANUEL, et Conseils MX Conseils Debian - Info. système “quick-system-info-mx” (QSI) ... Ici: System: MX-19-23_x64 & antiX23_x32 runit

User avatar
trawglodyte
Posts: 113
Joined: Tue Feb 13, 2024 7:35 am

Re: Running AI models locally with Ollama

#26 Post by trawglodyte »

It works fine on my 8 yr old machine with an i7-8700 and NVIDIA GTX 1060. The GPU hits 60-70% when the LLM is responding. I tried to add Open WebUI (as instructed in this article --> https://itsfoss.com/ollama-setup-linux/) but was unsuccessful. It does work fine from terminal though, and it's kind of fun that way.

User avatar
trawglodyte
Posts: 113
Joined: Tue Feb 13, 2024 7:35 am

Re: Running AI models locally with Ollama

#27 Post by trawglodyte »

Here is a simple chat GUI for ollama. It assumes you have ollama and at least one LLM installed already, and the ollama server running. You get a dropdown menu to choose between your installed LLMs and a basic chat window. And, yes, AI wrote most of this for me.

Code: Select all

import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, GLib
import requests
import threading
import subprocess
import json

# Ollama API endpoint
OLLAMA_API_URL = "http://localhost:11434/api/generate"

class ChatApp(Gtk.Window):
    def __init__(self):
        Gtk.Window.__init__(self, title="Ollama Chat")
        self.set_default_size(900, 600)

        # Main vertical box
        self.vbox = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
        self.add(self.vbox)

        # Dropdown for model selection
        self.model_store = Gtk.ListStore(str)  # Store model names
        self.model_combo = Gtk.ComboBox.new_with_model(self.model_store)
        renderer = Gtk.CellRendererText()
        self.model_combo.pack_start(renderer, True)
        self.model_combo.add_attribute(renderer, "text", 0)
        self.model_combo.set_active(0)  # Default to the first model
        self.model_combo.connect("changed", self.on_model_changed)
        self.vbox.pack_start(self.model_combo, False, True, 0)

        # Chat history (TextView inside ScrolledWindow)
        self.scrolled_window = Gtk.ScrolledWindow()
        self.scrolled_window.set_policy(Gtk.PolicyType.NEVER, Gtk.PolicyType.AUTOMATIC)  # Vertical scrollbar only
        self.vbox.pack_start(self.scrolled_window, True, True, 0)

        self.chat_history = Gtk.TextView()
        self.chat_history.set_editable(False)
        self.chat_history.set_wrap_mode(Gtk.WrapMode.WORD)
        self.scrolled_window.add(self.chat_history)  # Add TextView to ScrolledWindow

        # Input box (Entry)
        self.input_entry = Gtk.Entry()
        self.input_entry.set_placeholder_text("Type your message here...")
        self.input_entry.connect("activate", self.on_input_activate)
        self.vbox.pack_start(self.input_entry, False, True, 0)

        # Send button
        self.send_button = Gtk.Button(label="Send")
        self.send_button.connect("clicked", self.on_send_clicked)
        self.vbox.pack_start(self.send_button, False, True, 0)

        # Initialize model list and selected model
        self.models = self.get_installed_models()
        self.selected_model = self.models[0] if self.models else None
        self.populate_model_dropdown()

    def get_installed_models(self):
        """Run `ollama list` to get installed models."""
        try:
            result = subprocess.run(["ollama", "list"], capture_output=True, text=True, check=True)
            lines = result.stdout.splitlines()
            models = [line.split()[0] for line in lines[1:]]  # Skip header line
            return models
        except Exception as e:
            print(f"Error fetching models: {e}")
            return []

    def populate_model_dropdown(self):
        """Populate the dropdown with installed models."""
        for model in self.models:
            self.model_store.append([model])
        if self.models:
            self.model_combo.set_active(0)  # Select the first model by default

    def on_model_changed(self, combo):
        """Handle model selection change."""
        tree_iter = combo.get_active_iter()
        if tree_iter is not None:
            model = self.model_store[tree_iter][0]
            self.selected_model = model
            self.append_to_chat_history(f"Switched to model: {model}\n")

    def on_input_activate(self, entry):
        self.send_message()

    def on_send_clicked(self, button):
        self.send_message()

    def send_message(self):
        user_input = self.input_entry.get_text().strip()
        if not user_input:
            return

        # Append user input to chat history
        self.append_to_chat_history(f"You: {user_input}\n")
        self.input_entry.set_text("")  # Clear input box

        # Send request to Ollama in a separate thread
        threading.Thread(target=self.query_ollama, args=(user_input,)).start()

    def query_ollama(self, prompt):
        payload = {
            "model": self.selected_model,
            "prompt": prompt,
            "stream": False
        }

        try:
            response = requests.post(OLLAMA_API_URL, json=payload)
            if response.status_code == 200:
                response_data = response.json()
                response_text = response_data.get("response", "").strip()
                GLib.idle_add(self.append_to_chat_history, f"{self.selected_model}: {response_text}\n\n")
            else:
                GLib.idle_add(self.append_to_chat_history, f"Error: {response.status_code} - {response.text}\n\n")
        except Exception as e:
            GLib.idle_add(self.append_to_chat_history, f"Error: {str(e)}\n\n")

    def append_to_chat_history(self, text):
        buffer = self.chat_history.get_buffer()
        buffer.insert(buffer.get_end_iter(), text)
        # Scroll to the end of the chat history
        self.chat_history.scroll_to_iter(buffer.get_end_iter(), 0, False, 0, 0)

# Run the application
if __name__ == "__main__":
    app = ChatApp()
    app.connect("destroy", Gtk.main_quit)
    app.show_all()
    Gtk.main()

Bulat
Posts: 50
Joined: Thu Mar 04, 2021 1:27 am

Re: Running AI models locally with Ollama

#28 Post by Bulat »

Adrian wrote: Mon Jul 29, 2024 8:05 am If you want to play with AI tools you don't need to give up your privacy and use online tools you can use the latest Llama3 model with Ollama

1. Download and install Ollama https://ollama.com/download (it will crate a half of gig executable in /usr/local/bin/ollama

2. You might need to start ollama "ollama serve & disown" (& to put the service in the background, disown to make sure the service doesn't close when you close that terminal)

3. Download latest llama3 model with "ollama run llama3.1" (this will download about 4.3 GB worth of model to /usr/share/ollama/.ollama) and will crate a terminal prompt where you can ask questions. It's pretty good for programming/troubleshooting tasks, like "create a bash script that goes though each line of a file and compares to the entries in another file and display only the lines that don't match" and the like. Not only that it creates the bash script but it also explains pretty well how it works. Have fun!
Is it possible to download a model (say, llama3) separately, and then run ollama?

User avatar
operadude
Posts: 1017
Joined: Tue Nov 05, 2019 12:08 am

Re: Running AI models locally with Ollama

#29 Post by operadude »

Adrian wrote: Mon Jul 29, 2024 8:05 am If you want to play with AI tools you don't need to give up your privacy and use online tools you can use the latest Llama3 model with Ollama

1. Download and install Ollama https://ollama.com/download (it will crate a half of gig executable in /usr/local/bin/ollama

2. You might need to start ollama "ollama serve & disown" (& to put the service in the background, disown to make sure the service doesn't close when you close that terminal)

3. Download latest llama3 model with "ollama run llama3.1" (this will download about 4.3 GB worth of model to /usr/share/ollama/.ollama) and will crate a terminal prompt where you can ask questions. It's pretty good for programming/troubleshooting tasks, like "create a bash script that goes though each line of a file and compares to the entries in another file and display only the lines that don't match" and the like. Not only that it creates the bash script but it also explains pretty well how it works. Have fun!
:number1:

Thanks SO much for this :exclamation:

I have been planning to do this for some time, and am now finally ready to give it a spin on a new Testing Rig:

MX-23.6-KDE
Asus H81M-K Desktop (2009 ?):
i5-4690 @ 1.7GHz 64-bit ("Haswell")
8GB RAM (will expand to 16GB, when it arrives)
Nvidia GeForce GT 740 (2GB, v. 418.211.00)
===> tried to update to 418.226.00, but all text and images went to something like 500% (way off the screen), and had to roll-back with the built-in command:

Code: Select all

sudo ddm-mx -p nvidia

Thanks for the detailed instructions, emendations, and follow-up.

Will report back as to how it goes!

:crossfingers:

EDIT/ADDENDUM:

BTW, thanks for the "Monte Hall" problem:

CLASSIC :exclamation:

EDIT/ADDENDUM:

Booted and Updated my Testing Rig (see specs above).
Going through the install steps... :crossfingers:

Post Reply

Return to “MX Help”