Running AI models locally with Ollama

[Adrian]

I agree with other posters, voice-to-text input seems to me like a different application, maybe even part of the Desktop Environment. It's also a bit bratty, you get an amazing tool for free and you complains it doesn't come with golden handles.

[Melber]

Handles? Who needs handles? They're so inefficient. All my doors are fitted with motion detectors...

Back on topic. Would be interested in trying this out, but don't think my aged laptop would be able to handle it. Perhaps also scared to confirm I wasted my time learning my very basic bash skills...

[MadMax]

Perhaps also scared to confirm I wasted my time learning my very basic bash skills...

You did not. I tried a little more sophisticated script yesterday: the task was to compare a number of random files between two directories to check a random sample for silent corruption - the actual prompt was more specific of course. After a few tries it gave me something what I'd call a "good template". Had to do some work to get it to run and expanded it further since then (added colors to the output, let it count the file size of the checked files implemented a second hash algorithm etc). It didn't work ootb like the simple one before.

BUT I have to admit the template was pretty useful as a starting point. The definition of the input variables and checking for correct inputs was done correctly.

[Adrian]

llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"

[oops]

llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"

Right, llama3.2:3b (2.0 GB) is faster than llama3.1:8b (4.7 GB), and 8MB of RAM is enough.
( llama3.2:1b (1.3 GB) also exists for PC with only 2 or 4GB)
Thanks for the info:

https://ollama.com/library/llama3.2

[trawglodyte]

It works fine on my 8 yr old machine with an i7-8700 and NVIDIA GTX 1060. The GPU hits 60-70% when the LLM is responding. I tried to add Open WebUI (as instructed in this article --> https://itsfoss.com/ollama-setup-linux/) but was unsuccessful. It does work fine from terminal though, and it's kind of fun that way.

[trawglodyte]

Here is a simple chat GUI for ollama. It assumes you have ollama and at least one LLM installed already, and the ollama server running. You get a dropdown menu to choose between your installed LLMs and a basic chat window. And, yes, AI wrote most of this for me.

Code: Select all

import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, GLib
import requests
import threading
import subprocess
import json

# Ollama API endpoint
OLLAMA_API_URL = "http://localhost:11434/api/generate"

class ChatApp(Gtk.Window):
    def __init__(self):
        Gtk.Window.__init__(self, title="Ollama Chat")
        self.set_default_size(900, 600)

        # Main vertical box
        self.vbox = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
        self.add(self.vbox)

        # Dropdown for model selection
        self.model_store = Gtk.ListStore(str)  # Store model names
        self.model_combo = Gtk.ComboBox.new_with_model(self.model_store)
        renderer = Gtk.CellRendererText()
        self.model_combo.pack_start(renderer, True)
        self.model_combo.add_attribute(renderer, "text", 0)
        self.model_combo.set_active(0)  # Default to the first model
        self.model_combo.connect("changed", self.on_model_changed)
        self.vbox.pack_start(self.model_combo, False, True, 0)

        # Chat history (TextView inside ScrolledWindow)
        self.scrolled_window = Gtk.ScrolledWindow()
        self.scrolled_window.set_policy(Gtk.PolicyType.NEVER, Gtk.PolicyType.AUTOMATIC)  # Vertical scrollbar only
        self.vbox.pack_start(self.scrolled_window, True, True, 0)

        self.chat_history = Gtk.TextView()
        self.chat_history.set_editable(False)
        self.chat_history.set_wrap_mode(Gtk.WrapMode.WORD)
        self.scrolled_window.add(self.chat_history)  # Add TextView to ScrolledWindow

        # Input box (Entry)
        self.input_entry = Gtk.Entry()
        self.input_entry.set_placeholder_text("Type your message here...")
        self.input_entry.connect("activate", self.on_input_activate)
        self.vbox.pack_start(self.input_entry, False, True, 0)

        # Send button
        self.send_button = Gtk.Button(label="Send")
        self.send_button.connect("clicked", self.on_send_clicked)
        self.vbox.pack_start(self.send_button, False, True, 0)

        # Initialize model list and selected model
        self.models = self.get_installed_models()
        self.selected_model = self.models[0] if self.models else None
        self.populate_model_dropdown()

    def get_installed_models(self):
        """Run `ollama list` to get installed models."""
        try:
            result = subprocess.run(["ollama", "list"], capture_output=True, text=True, check=True)
            lines = result.stdout.splitlines()
            models = [line.split()[0] for line in lines[1:]]  # Skip header line
            return models
        except Exception as e:
            print(f"Error fetching models: {e}")
            return []

    def populate_model_dropdown(self):
        """Populate the dropdown with installed models."""
        for model in self.models:
            self.model_store.append([model])
        if self.models:
            self.model_combo.set_active(0)  # Select the first model by default

    def on_model_changed(self, combo):
        """Handle model selection change."""
        tree_iter = combo.get_active_iter()
        if tree_iter is not None:
            model = self.model_store[tree_iter][0]
            self.selected_model = model
            self.append_to_chat_history(f"Switched to model: {model}\n")

    def on_input_activate(self, entry):
        self.send_message()

    def on_send_clicked(self, button):
        self.send_message()

    def send_message(self):
        user_input = self.input_entry.get_text().strip()
        if not user_input:
            return

        # Append user input to chat history
        self.append_to_chat_history(f"You: {user_input}\n")
        self.input_entry.set_text("")  # Clear input box

        # Send request to Ollama in a separate thread
        threading.Thread(target=self.query_ollama, args=(user_input,)).start()

    def query_ollama(self, prompt):
        payload = {
            "model": self.selected_model,
            "prompt": prompt,
            "stream": False
        }

        try:
            response = requests.post(OLLAMA_API_URL, json=payload)
            if response.status_code == 200:
                response_data = response.json()
                response_text = response_data.get("response", "").strip()
                GLib.idle_add(self.append_to_chat_history, f"{self.selected_model}: {response_text}\n\n")
            else:
                GLib.idle_add(self.append_to_chat_history, f"Error: {response.status_code} - {response.text}\n\n")
        except Exception as e:
            GLib.idle_add(self.append_to_chat_history, f"Error: {str(e)}\n\n")

    def append_to_chat_history(self, text):
        buffer = self.chat_history.get_buffer()
        buffer.insert(buffer.get_end_iter(), text)
        # Scroll to the end of the chat history
        self.chat_history.scroll_to_iter(buffer.get_end_iter(), 0, False, 0, 0)

# Run the application
if __name__ == "__main__":
    app = ChatApp()
    app.connect("destroy", Gtk.main_quit)
    app.show_all()
    Gtk.main()