Skip to Content

Build your own AI assistant: Ollama and Open WebUI

A private conversational assistant on your own hardware: the setup, the real cost, and the settings that decide whether your data leaves.

Running a language model on your own server has gone from a weekend project to an ordinary deployment. OpenAI's open-weight model, gpt-oss in its 20 billion parameter version, fits on a machine with as little as 16 GB of memory according to its installation page. And Ollama, the tool that makes all of this simple, announced in July 2026 that it serves 8.9 million developers.

We covered the theory in our article on local, private AI. Here is the actual setup, and more importantly the four or five things we check before recommending it to a client.

What does "local AI" actually mean?

A "local" AI is a language model that runs on a machine you control. Your questions, the documents you feed it, your drafts: everything stays inside your walls. No request crosses the internet to a provider, so there is no conversation sitting in someone else's account, no terms of service to re-read every six months, and no doubt about what is feeding the next model's training.

For an organization handling personal information or strategic files, this is often the only configuration that makes AI usable without transferring data to a third party. And it is the alternative our piece on Shadow AI recommended: your people are already using AI, the question is which tool they use.

The two pieces of the setup

Ollama is the engine. It downloads and runs open-weight models with a single command, and it is published under the MIT license. It handles memory, the graphics card and the model format. On its own it is a command-line tool: perfect for the technical team, unusable for everyone else.

Open WebUI is the front end. A polished web interface that gives Ollama the feel of an internal ChatGPT, and it is multi-user from the moment you install it. You get conversation history, folders and tags, plus three things that matter in an organization.

First, knowledge bases: you upload documents, and the model retrieves the relevant passages before answering. Second, access control, with roles, groups and per-model permissions, so that HR and sales are not looking at the same documents. And third, federated authentication over OIDC or LDAP, with automated account provisioning through SCIM 2.0. If you followed our article on migrating from Active Directory to Authentik, this is exactly the kind of service that belongs behind your directory instead of creating a second user list to maintain.

Which model, on which machine

This is the question that sets the budget, and it has a simple answer: the model's download size is roughly what you need in video memory, plus some headroom for the conversation in progress. The sizes published in Ollama's library give you the order of magnitude right away.

ModelDownload sizeWhat it takes
Qwen3 8 billion5.2 GBAn entry-level graphics card
Qwen3 14 billion9.3 GBA 12 GB card
gpt-oss 20 billion14 GB16 GB of memory, per the official page
Qwen3 32 billion20 GBA 24 GB card
gpt-oss 120 billion65 GBAn 80 GB card, datacentre class

For most SMBs, the interesting zone sits between 8 and 32 billion parameters, on a graphics card of 12 to 24 GB. It writes, summarizes, rephrases, classifies and answers properly against your documents. A plain processor works too, with no graphics card, but the answer then builds itself word by word in front of you instead of arriving all at once.

"Open weights" does not mean "open source"

Here is a distinction few people make, and it has real consequences. Ollama is MIT licensed, which is about as permissive as it gets. But the models it downloads each carry their own license, and those licenses are not alike.

Some are genuinely permissive: Qwen3, gpt-oss and the recent Mistral Small models ship under Apache 2.0, and DeepSeek-R1 under MIT. Do what you want with them.

Others are house licenses. Meta's Llama 4 license requires you to prominently display "Built with Llama" if you distribute a product containing it, and to prefix "Llama" to the name of any model you derive from it. It also states that past 700 million monthly active users, you must request a separate license from Meta.

Neither clause will ever bother a Quebec SMB, and that is not a reason to avoid Llama. But the distinction matters the day you want to resell a product built on top of it, or simply want to know what you are allowed to do. The model's license reads separately from the tool's.

The clause to read before you deploy

Open WebUI changed its license in April 2025, starting with version 0.6.6, and the update flew under a lot of radars. The foundation is still a permissive BSD-3-Clause: you can use, modify and redistribute the software, free of charge, with no cap on the number of users.

What was added is a branding protection clause. You may not remove, obscure or replace the "Open WebUI" branding in your deployment, except in three cases: your deployment has 50 users or fewer over a rolling 30-day period, you are a recognized project contributor with written permission, or you have secured an enterprise license. The code up to version 0.6.5 remains pure BSD-3, and it can be forked with none of these constraints.

What this changes in practice: for a team of 20 or 40 people, nothing at all, including if you swap the colours for your own. For a 200-seat deployment where management wanted its own logo in the corner, you either keep the branding visible or you pay. This is not a trap, it is a business model, and it is better known before the kickoff meeting than during it.

The settings that decide whether your data leaves

A local setup is not automatically airtight. It becomes airtight after a few decisions, and these are the ones we look at first.

The external connector is on by default. In Open WebUI, the ENABLE_OPENAI_API setting is True at install time, and the address points at api.openai.com. With no API key configured, nothing goes out. But the door is open, and all it takes is an administrator pasting a key in there on a Tuesday afternoon for traffic to leave your walls. We turn it off when the mandate is "nothing leaves".

Ollama now sells cloud capacity. Since 2025, the tool also offers models running on its own servers, with paid plans above a free tier. They appear in the same library as the local models, carrying a -cloud suffix. Ollama states that it does not retain the data passing through, and there is no reason to doubt that, but the gap between a model that stays with you and a model that leaves for a provider is a nine-character suffix. That needs documenting and framing.

Signup is open, activation is not. Anyone who reaches the address can create an account, but new accounts land in a "pending" state and an administrator has to activate them. That is a reasonable default. Still, maintaining a user list by hand is a chore that always ends up slipping, which is why wiring authentication to the existing directory is worth the effort.

The four settings to validate before opening access to the team:

  1. The connector to the external provider is disabled, or deliberate and documented.
  2. Models with the -cloud suffix are removed from the list offered to users.
  3. Authentication runs through your directory, with groups that match your actual teams.
  4. Knowledge bases are partitioned by group, not open to everyone by default.

What it does not replace

Four caveats, stated up front.

Raw capability, first. A properly sized local model is genuinely useful for writing, summarizing, rephrasing and classifying. The largest cloud models keep an edge on the genuinely complex tasks, and that gap is real even as it narrows.

Concurrency, next. A graphics card processes one request at a time. With three or four people querying the model simultaneously, a queue appears. That gets planned when you size the machine, not afterwards.

Accuracy, above all. A local model gets things wrong with exactly the same confidence as a cloud one. Hosting it yourself settles the confidentiality question, not the correctness question. Human review remains entirely necessary.

And cost, finally. There is no per-request billing, and Ollama specifies that running on your own hardware is unlimited. But a graphics card, the electricity it draws and someone's time to keep the whole thing current all carry a price. It is a trade-off between cost, speed and confidentiality, and it should be made with eyes open.

What we deploy

The typical setup is quickly summarized: Ollama and Open WebUI in containers on a server with a graphics card, a model chosen for the real use cases and the available hardware, authentication wired to the organization's directory, the external connector closed, and access published internally only. The team connects from a browser and installs nothing.

Local AI makes the most sense when confidentiality is not negotiable: professional practices, organizations handling sensitive files, businesses that would rather not feed a provider with their thinking. For those organizations, trading a bit of raw capability for complete control is a very good deal. We help size the hardware, pick the model, set the settings that matter, and put the result in the team's hands.

Not sure whether your current hardware would be enough? That is the first thing we look at, and we can run the numbers from your real use cases rather than from a spec sheet.

Curious what this would look like on your own servers? Drop us a line: we start with an inventory of what you already have.

Sources

Vaultwarden: your password vault, in house
The report listing every reused and compromised password on the team, passkeys that stay with the company when someone leaves, and the real math for twenty people.