Back to Blog
•7 min read

I Built an On-Device AI Agent in Chrome — No API Keys Needed

ChromeAIWeb DevelopmentJavaScriptLocal-First

Last spring I added a "chat with our docs" widget to a client site. It worked great — for about three weeks, until the invoice arrived. Every visitor question was being forwarded to an LLM API, and the bill for a moderately popular docs page was embarrassing enough that I yanked the feature and built a static FAQ instead.

That's the tradeoff nobody talks about when they demo an AI chatbot: someone pays for every question. Either you do, or your users do with their privacy.

So when Chrome started shipping its built-in AI — Gemini Nano running directly in the browser, no API keys, no server, no per-token billing — I was skeptical. It had to be a toy, right? Then I actually built a Q&A agent with it, and the thing is genuinely useful. Here's how, including the parts that will bite you.

What "Chrome built-in AI" actually is

Chrome ships a small language model called Gemini Nano that runs locally on your machine. No cloud call happens. Your text goes into the model on-device, and the answer comes back the same way. It started rolling out to stable desktop Chrome in mid-2025, and it's exposed through a set of JavaScript APIs under navigator.ai:

  • Prompt API (ai.languageModel) — the general-purpose one. Ask questions, get text back, maintain a conversation.
  • Summarizer (ai.summarizer) — condense long text.
  • Writer & Rewriter (ai.writer, ai.rewriter) — generate and restyle text.
  • Translator & Language Detector (ai.translator, ai.languageDetector) — translation and language identification.

For a Q&A agent, the Prompt API is the one you'll live in. It's a real model, not a demo gimmick — smaller than what you'd get from a paid API, but plenty for answering questions about a specific body of text.

Check what the user's machine can do

The first thing to understand: built-in AI isn't guaranteed to exist. Some users have it, some need a one-time model download, and some never will (older hardware, enterprise policies, some platforms). So before you do anything, feature-detect:

const capabilities = await navigator.ai.languageModel.capabilities()

if (capabilities.available === 'no') {
  // Show your fallback: a search box, a contact form, anything
  return
}

available returns one of three values:

  • 'readily' — model is installed and ready.
  • 'after-download' — the model will download on first use. That's a chunk of data, so warn users before triggering it.
  • 'no' — not supported. Build a graceful fallback; don't hide the entire feature.

I also check capabilities.defaultTemperature and maxTopK so my UI sliders match what the model actually supports, but for a Q&A agent you mostly don't need them.

The Prompt API in 60 seconds

Creating a session and asking a question is almost embarrassingly simple:

const session = await navigator.ai.languageModel.create({
  systemPrompt: 'You answer questions about the Acme docs. Be concise.',
  temperature: 0.3,
})

const answer = await session.prompt('What is the maximum upload size?')
console.log(answer)

// Always clean up when done — sessions hold memory
session.destroy()

For long answers, stream instead of blocking:

const stream = session.promptStreaming('Explain the pricing tiers')
for await (const chunk of stream) {
  render(chunk) // append to your UI as it arrives
}

One quirk: with promptStreaming, each chunk is the entire answer so far, not a delta. You replace the text, you don't append. Took me a confused afternoon to figure that one out.

Now the actual Q&A agent

A Q&A agent is really just retrieval plus a prompt. The model doesn't know your content — you have to hand it to the model in the prompt. Since there's no embeddings API in the built-in AI suite yet, I do retrieval the old-fashioned way:

  1. Split your content into chunks with a little overlap (I use ~400 words with 50 words of overlap).
  2. Score each chunk against the user's question with a keyword match.
  3. Take the top chunks, stuff them into the prompt, and tell the model to answer only from that context.
function scoreChunk(chunk, query) {
  const terms = query.toLowerCase().split(/\W+/).filter(Boolean)
  return terms.reduce((score, t) => score + (chunk.toLowerCase().includes(t) ? 1 : 0), 0)
}

async function answer(question, chunks) {
  const top = chunks
    .map(c => ({ text: c, score: scoreChunk(c, question) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 4)
    .map(c => c.text)
    .join('\n\n---\n\n')

  const session = await navigator.ai.languageModel.create({
    systemPrompt: 'Answer using only the context below. If the answer is not in the context, say so.',
  })

  try {
    return await session.prompt(`Context:\n${top}\n\nQuestion: ${question}`)
  } finally {
    session.destroy()
  }
}

Is keyword retrieval dumb compared to vector search? Yes, and it works anyway. For a docs site, a changelog, or a personal notes archive — content that uses consistent terminology — keyword overlap gets you 80% of the way. When the model sees the top 4 chunks plus an explicit instruction not to invent facts, the answers are surprisingly solid.

I tried a few failure modes: questions with synonyms the docs don't use, typos, multi-part questions. Synonym misses happen ("fee" vs "price"). Typos mostly survive because the model is forgiving. Multi-part questions degrade — chunk scoring splits them. If you need real semantic retrieval, you can compute embeddings with a small library like @xenova/transformers (WebGPU or WASM) and keep the built-in model for generation. That's a bigger post, but it's a clean upgrade path.

Make it feel fast, because first calls are slow

Honest numbers from my testing: the first prompt after a cold start took 2–6 seconds on my M-series Mac before tokens started flowing. Subsequent prompts were much quicker, but a cold start that slow will make users think it's broken.

Three tricks that helped:

  • Pre-warm on idle. After the page loads, wait for requestIdleCallback and create a throwaway session, then destroy it. That triggers the model load while the user is reading, not while they're waiting for an answer.
  • Reuse sessions. Creating a session is cheap-ish, but keep one around for follow-ups instead of creating and destroying per question. A follow-up question with conversation history is much better than re-answering from scratch anyway.
  • Stream everything. First token latency is what kills perceived performance. With promptStreaming, users see text start within a second or two of the model waking up.

And show a visible "model is loading" state the first time, especially when capabilities.available is 'after-download'. That download is a few gigabytes on some platforms — you do not want to surprise users with it silently mid-interaction.

The honest limitations

Built-in AI is great, and it is not a full replacement for a server-side model. Things I hit:

  • It's small. Gemini Nano is closer to a fast, capable mini model than a frontier model. Complex reasoning, long code generation, nuanced tone — it'll do okay, not great. Keep the agent's scope narrow.
  • Context is limited. You can't dump a whole book into one prompt. That's exactly why the chunk-and-retrieve step above is non-negotiable.
  • Users can be in "low-resource mode." On some devices Chrome downgrades the model behavior to protect battery and memory — your results get noticeably dumber. I detect it via capabilities.available === 'no' plus a model check where possible, and I cap output length so the UI never churns for 30 seconds.
  • The model can update under you. Chrome rolls out model versions independently of your code. Pin your system prompts tightly and write a few golden-question tests you can run manually after Chrome updates.
  • Not everywhere. Mobile Chrome support has lagged desktop. Ship the fallback, seriously.

Why I'm building with it anyway

The privacy story is the whole point. With a server-side chat, every user question becomes data you have to store, secure, and disclose. With on-device AI, the question and the answer never leave the machine. No API bill, no retention policy, no "we may share your queries with our LLM provider" paragraph in your privacy policy. For a docs widget, a personal knowledge base, or a local-first tool, that's the difference between a feature you can ship and a feature you have to defend in a security review.

It fits the same philosophy that makes local-first apps attractive: the user's data stays on the user's machine, and your infrastructure stays simple. I've started sketching an on-device Q&A over my markdown notes, and ZeroPad — the notes app I use daily — is the obvious home for it. A "ask my notes" button that runs entirely in the browser is a killer feature, and it costs me nothing to run.

Try it

Check navigator.ai.languageModel.capabilities() in your browser right now and see what you get. If it's ready, build the 30-line version above and ask it something about a page of your own docs. If it says 'after-download', let it download overnight and try again tomorrow.

The whole pattern — feature detection, pre-warm, chunk-and-retrieve, streaming — took me an evening to get right, and it's exactly the kind of thing I never want to rewrite from memory. It lives as a saved snippet in Snippet Ark alongside my other local-first boilerplate. What would you do with an LLM that runs for free, offline, and keeps everything on the user's device?

Back to Blog