Running AI models locally with Ollama
Re: Running AI models locally with Ollama
I agree with other posters, voice-to-text input seems to me like a different application, maybe even part of the Desktop Environment. It's also a bit bratty, you get an amazing tool for free and you complains it doesn't come with golden handles.
Re: Running AI models locally with Ollama
Handles? Who needs handles? They're so inefficient. All my doors are fitted with motion detectors...
Back on topic. Would be interested in trying this out, but don't think my aged laptop would be able to handle it. Perhaps also scared to confirm I wasted my time learning my very basic bash skills...
Back on topic. Would be interested in trying this out, but don't think my aged laptop would be able to handle it. Perhaps also scared to confirm I wasted my time learning my very basic bash skills...
Re: Running AI models locally with Ollama
You did not. I tried a little more sophisticated script yesterday: the task was to compare a number of random files between two directories to check a random sample for silent corruption - the actual prompt was more specific of course. After a few tries it gave me something what I'd call a "good template". Had to do some work to get it to run and expanded it further since then (added colors to the output, let it count the file size of the checked files implemented a second hash algorithm etc). It didn't work ootb like the simple one before.Melber wrote: Sat Aug 03, 2024 12:24 pmPerhaps also scared to confirm I wasted my time learning my very basic bash skills...
BUT I have to admit the template was pretty useful as a starting point. The definition of the input variables and checking for correct inputs was done correctly.
If it ain't broke, don't fix it.
Main: MX 23 | Second: Mint 22 | HTPC: Linux Lite 7 | VM Machine: Debian 12 | Testrig: Arch/FreeBSD 14 | Work: RHEL 8
Main: MX 23 | Second: Mint 22 | HTPC: Linux Lite 7 | VM Machine: Debian 12 | Testrig: Arch/FreeBSD 14 | Work: RHEL 8
Re: Running AI models locally with Ollama
llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"
Re: Running AI models locally with Ollama
Right, llama3.2:3b (2.0 GB) is faster than llama3.1:8b (4.7 GB), and 8MB of RAM is enough.Adrian wrote: Tue Oct 01, 2024 1:05 pm llama3.2:latest is smaller 2GB and faster if you want to try: "ollama pull llama3.2:latest" and "ollama run llama3.2"
( llama3.2:1b (1.3 GB) also exists for PC with only 2 or 4GB)
Thanks for the info:
https://ollama.com/library/llama3.2
Pour les nouveaux utilisateurs: Alt+F1 pour le manuel, ou FAQS, MX MANUEL, et Conseils MX Conseils Debian - Info. système “quick-system-info-mx” (QSI) ... Ici: System: MX-19-23_x64 & antiX23_x32 runit
- trawglodyte
- Posts: 113
- Joined: Tue Feb 13, 2024 7:35 am
Re: Running AI models locally with Ollama
It works fine on my 8 yr old machine with an i7-8700 and NVIDIA GTX 1060. The GPU hits 60-70% when the LLM is responding. I tried to add Open WebUI (as instructed in this article --> https://itsfoss.com/ollama-setup-linux/) but was unsuccessful. It does work fine from terminal though, and it's kind of fun that way.
- trawglodyte
- Posts: 113
- Joined: Tue Feb 13, 2024 7:35 am
Re: Running AI models locally with Ollama
Here is a simple chat GUI for ollama. It assumes you have ollama and at least one LLM installed already, and the ollama server running. You get a dropdown menu to choose between your installed LLMs and a basic chat window. And, yes, AI wrote most of this for me.
Code: Select all
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, GLib
import requests
import threading
import subprocess
import json
# Ollama API endpoint
OLLAMA_API_URL = "http://localhost:11434/api/generate"
class ChatApp(Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self, title="Ollama Chat")
self.set_default_size(900, 600)
# Main vertical box
self.vbox = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
self.add(self.vbox)
# Dropdown for model selection
self.model_store = Gtk.ListStore(str) # Store model names
self.model_combo = Gtk.ComboBox.new_with_model(self.model_store)
renderer = Gtk.CellRendererText()
self.model_combo.pack_start(renderer, True)
self.model_combo.add_attribute(renderer, "text", 0)
self.model_combo.set_active(0) # Default to the first model
self.model_combo.connect("changed", self.on_model_changed)
self.vbox.pack_start(self.model_combo, False, True, 0)
# Chat history (TextView inside ScrolledWindow)
self.scrolled_window = Gtk.ScrolledWindow()
self.scrolled_window.set_policy(Gtk.PolicyType.NEVER, Gtk.PolicyType.AUTOMATIC) # Vertical scrollbar only
self.vbox.pack_start(self.scrolled_window, True, True, 0)
self.chat_history = Gtk.TextView()
self.chat_history.set_editable(False)
self.chat_history.set_wrap_mode(Gtk.WrapMode.WORD)
self.scrolled_window.add(self.chat_history) # Add TextView to ScrolledWindow
# Input box (Entry)
self.input_entry = Gtk.Entry()
self.input_entry.set_placeholder_text("Type your message here...")
self.input_entry.connect("activate", self.on_input_activate)
self.vbox.pack_start(self.input_entry, False, True, 0)
# Send button
self.send_button = Gtk.Button(label="Send")
self.send_button.connect("clicked", self.on_send_clicked)
self.vbox.pack_start(self.send_button, False, True, 0)
# Initialize model list and selected model
self.models = self.get_installed_models()
self.selected_model = self.models[0] if self.models else None
self.populate_model_dropdown()
def get_installed_models(self):
"""Run `ollama list` to get installed models."""
try:
result = subprocess.run(["ollama", "list"], capture_output=True, text=True, check=True)
lines = result.stdout.splitlines()
models = [line.split()[0] for line in lines[1:]] # Skip header line
return models
except Exception as e:
print(f"Error fetching models: {e}")
return []
def populate_model_dropdown(self):
"""Populate the dropdown with installed models."""
for model in self.models:
self.model_store.append([model])
if self.models:
self.model_combo.set_active(0) # Select the first model by default
def on_model_changed(self, combo):
"""Handle model selection change."""
tree_iter = combo.get_active_iter()
if tree_iter is not None:
model = self.model_store[tree_iter][0]
self.selected_model = model
self.append_to_chat_history(f"Switched to model: {model}\n")
def on_input_activate(self, entry):
self.send_message()
def on_send_clicked(self, button):
self.send_message()
def send_message(self):
user_input = self.input_entry.get_text().strip()
if not user_input:
return
# Append user input to chat history
self.append_to_chat_history(f"You: {user_input}\n")
self.input_entry.set_text("") # Clear input box
# Send request to Ollama in a separate thread
threading.Thread(target=self.query_ollama, args=(user_input,)).start()
def query_ollama(self, prompt):
payload = {
"model": self.selected_model,
"prompt": prompt,
"stream": False
}
try:
response = requests.post(OLLAMA_API_URL, json=payload)
if response.status_code == 200:
response_data = response.json()
response_text = response_data.get("response", "").strip()
GLib.idle_add(self.append_to_chat_history, f"{self.selected_model}: {response_text}\n\n")
else:
GLib.idle_add(self.append_to_chat_history, f"Error: {response.status_code} - {response.text}\n\n")
except Exception as e:
GLib.idle_add(self.append_to_chat_history, f"Error: {str(e)}\n\n")
def append_to_chat_history(self, text):
buffer = self.chat_history.get_buffer()
buffer.insert(buffer.get_end_iter(), text)
# Scroll to the end of the chat history
self.chat_history.scroll_to_iter(buffer.get_end_iter(), 0, False, 0, 0)
# Run the application
if __name__ == "__main__":
app = ChatApp()
app.connect("destroy", Gtk.main_quit)
app.show_all()
Gtk.main()
Re: Running AI models locally with Ollama
Is it possible to download a model (say, llama3) separately, and then run ollama?Adrian wrote: Mon Jul 29, 2024 8:05 am If you want to play with AI tools you don't need to give up your privacy and use online tools you can use the latest Llama3 model with Ollama
1. Download and install Ollama https://ollama.com/download (it will crate a half of gig executable in /usr/local/bin/ollama
2. You might need to start ollama "ollama serve & disown" (& to put the service in the background, disown to make sure the service doesn't close when you close that terminal)
3. Download latest llama3 model with "ollama run llama3.1" (this will download about 4.3 GB worth of model to /usr/share/ollama/.ollama) and will crate a terminal prompt where you can ask questions. It's pretty good for programming/troubleshooting tasks, like "create a bash script that goes though each line of a file and compares to the entries in another file and display only the lines that don't match" and the like. Not only that it creates the bash script but it also explains pretty well how it works. Have fun!
Re: Running AI models locally with Ollama
Adrian wrote: Mon Jul 29, 2024 8:05 am If you want to play with AI tools you don't need to give up your privacy and use online tools you can use the latest Llama3 model with Ollama
1. Download and install Ollama https://ollama.com/download (it will crate a half of gig executable in /usr/local/bin/ollama
2. You might need to start ollama "ollama serve & disown" (& to put the service in the background, disown to make sure the service doesn't close when you close that terminal)
3. Download latest llama3 model with "ollama run llama3.1" (this will download about 4.3 GB worth of model to /usr/share/ollama/.ollama) and will crate a terminal prompt where you can ask questions. It's pretty good for programming/troubleshooting tasks, like "create a bash script that goes though each line of a file and compares to the entries in another file and display only the lines that don't match" and the like. Not only that it creates the bash script but it also explains pretty well how it works. Have fun!

Thanks SO much for this

I have been planning to do this for some time, and am now finally ready to give it a spin on a new Testing Rig:
MX-23.6-KDE
Asus H81M-K Desktop (2009 ?):
i5-4690 @ 1.7GHz 64-bit ("Haswell")
8GB RAM (will expand to 16GB, when it arrives)
Nvidia GeForce GT 740 (2GB, v. 418.211.00)
===> tried to update to 418.226.00, but all text and images went to something like 500% (way off the screen), and had to roll-back with the built-in command:
Code: Select all
sudo ddm-mx -p nvidia
Thanks for the detailed instructions, emendations, and follow-up.
Will report back as to how it goes!

EDIT/ADDENDUM:
BTW, thanks for the "Monte Hall" problem:
CLASSIC

EDIT/ADDENDUM:
Booted and Updated my Testing Rig (see specs above).
Going through the install steps...
