Google Gemma 4: Run It Locally and Build Your First AI Agent

Google released Gemma 4 in early April: Apache 2.0, multimodal, several sizes from phone to workstation, and native function calling that is good enough to build a real agent on rather than a demo.

This is what I would tell someone who wants to run it on hardware they already own, and then have it call tools.

Re-checked on 21 August 2026. One thing changed since April, and it changed the recommendation, so read the sizes section even if you skimmed this in the spring.

What makes it worth the disk space

Gemma 4 comes out of the same research line as Google’s commercial Gemini models, and the jump from Gemma 3 on the reported benchmarks is large enough to be suspicious: AIME math from 20.8% to 89.2%, LiveCodeBench from 29.1% to 80.0%, GPQA from 42.4% to 84.3%. Those are Google’s numbers, not mine, and benchmark scores are the least interesting thing about a local model.

Three things matter more. The license is Apache 2.0, so there is no monthly-active-user cap and no regional carve-out to read twice, which is not true of Llama. The models are built for efficient inference on consumer hardware rather than being a server model that technically fits. And tool calling is trained in rather than bolted on, which you feel immediately the first time a model follows a schema without being nagged.

Context is 128K on the two edge models and 256K on everything above them. Long context matters more than usual in an agent loop, because every tool result you feed back is context you are spending.

The sizes, and which one you actually want

E2B, around 7GB on disk, 128K context. Runs on phones and edge devices. Impressive for its size and frustrating for anything past simple question answering unless you are shipping a mobile app.

E4B, 9.6GB, 128K context. The E stands for effective parameters. This was my recommended starting point in April and it is still a fine one: an 8GB GPU or a 16GB Mac handles it, and it is good enough for prototyping, code generation, and single-tool agent work. The gap to the bigger models shows up on multi-step reasoning.

12B, 7.6GB, 256K context. This one did not exist when I first wrote this. It landed in June and it is the change that matters: it is smaller on disk than E4B, it carries the full 256K context, and it is a dense 12B rather than an edge-tuned model. If you are starting today, start here rather than E4B unless you specifically need the edge build.

26B-A4B, 18GB, 256K context. The mixture-of-experts variant: 26 billion parameters total, roughly 4 billion active per token. You get most of the quality of the dense 31B for a fraction of the compute. On a 24GB card at Q4 it runs with the full context window, which is the single best quality-per-gigabyte deal in the family.

31B, 20GB quantized, 256K context. Dense, every parameter on every token, and the ceiling of what Gemma 4 does. At full precision you are looking at something like 80GB of memory, so in practice you run it quantized and give up the whole GPU to it. Worth it only if you have the hardware spare.

Short version: start at 12B. Move to 26B-A4B when you want production-quality output without buying a bigger card. Only bother with 31B if you already own the GPU.

Getting it running with Ollama

Still the shortest path.

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

Start the service if it is not already up:

ollama serve

Pull a model. The tags are the sizes:

ollama pull gemma4:12b     # 7.6GB, 256K context
ollama pull gemma4:26b     # 18GB, MoE
ollama pull gemma4:e4b     # 9.6GB, edge build, also gemma4:latest

Check it arrived, then talk to it:

ollama list
ollama run gemma4:12b

/bye exits. For everything else you want the local HTTP API on port 11434:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:12b",
  "messages": [
    {"role": "user", "content": "Explain quantum computing in two sentences."}
  ],
  "stream": false
}'

That is Ollama’s own API shape. There is also an OpenAI-compatible surface at /v1/chat/completions if you would rather point an existing OpenAI client at it. The examples below use the native one, because the tool-calling fields are clearer there.

Three failures cover most of what goes wrong. If it loads slowly or dies, you are out of VRAM: drop a size or accept heavier quantization (Ollama defaults to Q4, which is usually fine). If the output is nonsense, check the tag you pulled. If the connection is refused, ollama serve is not running.

Function calling, and a small agent

The flow has four steps. You describe the tools. The model returns a structured call instead of prose. You execute it and append the result. The model reads the result and answers.

The one detail people get wrong: when you send the result back, the message needs tool_name (or tool_call_id) alongside role and content. Leave it out and the model has no idea which call it is looking at.

import json
import requests

OLLAMA_URL = "http://localhost:11434/api/chat"
MODEL = "gemma4:12b"

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City name, e.g. 'Seoul' or 'San Francisco'",
                    }
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_calendar_events",
            "description": "Get calendar events for one date",
            "parameters": {
                "type": "object",
                "properties": {
                    "date": {
                        "type": "string",
                        "description": "Date in YYYY-MM-DD format",
                    }
                },
                "required": ["date"],
            },
        },
    },
]


def get_weather(location):
    # Replace with a real weather API call.
    return {"location": location, "temp": "18C", "condition": "Partly cloudy"}


def get_calendar_events(date):
    # Replace with a real calendar integration.
    return {"date": date, "events": ["Standup at 10:00", "Lunch at 12:00"]}


handlers = {
    "get_weather": get_weather,
    "get_calendar_events": get_calendar_events,
}


def chat(messages):
    response = requests.post(
        OLLAMA_URL,
        json={"model": MODEL, "messages": messages, "tools": tools, "stream": False},
        timeout=120,
    )
    response.raise_for_status()
    return response.json()["message"]


def run_agent(user_message, max_rounds=5):
    messages = [{"role": "user", "content": user_message}]

    for _ in range(max_rounds):
        message = chat(messages)

        if not message.get("tool_calls"):
            return message.get("content", "")

        messages.append(message)

        for call in message["tool_calls"]:
            name = call["function"]["name"]
            args = call["function"]["arguments"]
            handler = handlers.get(name)
            output = handler(**args) if handler else {"error": f"unknown tool {name}"}
            messages.append(
                {
                    "role": "tool",
                    "tool_name": name,  # required, or the model loses track
                    "content": json.dumps(output),
                }
            )

    return "Agent hit the round limit."


print(run_agent("What's the weather in Seoul and what's on my calendar today?"))

Run it and you get two tool calls and one synthesized answer. Every agent framework you have heard of is this loop plus retries, logging, and opinions.

Four things I would do differently from the naive version:

Keep tool descriptions specific. “Get weather for a city” beats “get weather information,” because vagueness comes back as invented arguments.

Validate arguments before you execute. Small models occasionally hand you a date in the wrong shape, and a schema check turns a confusing runtime error into a clear one.

Use the larger model when the agent has to choose between five tools in some order. The edge builds are fine at one tool and wobbly at orchestration.

Always cap the rounds. A confused model will happily loop until you kill it.

Against the other open models

Llama 4 went big: Maverick at 400B total with 128 experts, Scout at 109B with a 10M token context window. Impressive, and mostly not something you run at home. The license also carries a 700M monthly-active-user cap and regional restrictions, which for a commercial product is the whole conversation.

Mistral’s small models remain the other serious Apache 2.0 option and are strong on coding efficiency.

For local use the question is not which model wins a leaderboard, it is which one fits your card. Gemma 4’s 26B-A4B on a 24GB GPU is the best quality per gigabyte I know of right now, and the 12B is the best thing that runs on almost anything.

When local is the wrong answer

Go local when the data should not leave the machine, when you want interactive latency without a network hop, when you are prototyping and do not want to think about keys and rate limits, or when your volume makes the API bill worse than a GPU.

Stay on a cloud API when you need frontier reasoning, because Claude Opus 5, GPT-5.5, and the top Gemini models are still meaningfully better on hard problems and the gap, while shrinking, is real. Stay in the cloud when you need concurrency, because one local GPU does not serve hundreds of sessions. And stay in the cloud if you do not want to be the person debugging a driver at 2am.

What I actually do is split it. Local for high-volume, moderate-difficulty work: summarizing, classifying, simple tool use. Cloud for the hard 20%. The surprise was not the quality, which is good rather than magic. It was how quickly the routine tasks stopped going out over the network at all.

ollama pull gemma4:12b is about seven gigabytes and five minutes. It costs nothing to find out whether your split looks like mine.