Ollama

Integrate Ollama-hosted models with Portkey for local LLM deployment with full observability.

Portkey provides a robust gateway to facilitate the integration of your locally hosted models through Ollama.

Local Setup (npm or Docker)

First, install the Gateway locally:

npx

npx @portkey-ai/gateway

docker run -d -p 8787:8787 portkeyai/gateway:latest

Then, connect to your local Ollama instance:

Python

from portkey_ai import Portkey

portkey = Portkey(
    base_url="http://localhost:8787/v1",  # Your local Gateway
    provider="ollama",
    custom_host="http://localhost:11434"  # Your Ollama instance
)

response = portkey.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)

Integration Steps

1. Expose your Ollama API

   Expose your Ollama API using a tunneling service like ngrok or make it publicly accessible. Skip this if you’re self-hosting the Gateway.

   For using Ollama with ngrok, here’s a useful guide:

   ngrok http 11434 --host-header="localhost:11434"
   
2. Add to Model Catalog

   1. Go to Model Catalog → Add Provider
   2. Enable “Local/Privately hosted provider” toggle
   3. Select Ollama as the provider type
   4. Enter your Ollama URL in Custom Host: https://your-ollama.ngrok-free.app
   5. Name your provider (e.g., my-ollama)

   Complete Setup Guide

   See all setup options

3. Use in Your Application

   Python

   from portkey_ai import Portkey
   
   portkey = Portkey(
       api_key="PORTKEY_API_KEY",
       provider="@my-ollama"
   )
   
   response = portkey.chat.completions.create(
       model="llama3",
       messages=[{"role": "user", "content": "Hello!"}]
   )
   
   print(response.choices[0].message.content)
   

   import Portkey from 'portkey-ai';
   
   const portkey = new Portkey({
       apiKey: 'PORTKEY_API_KEY',
       provider: '@my-ollama'
   });
   
   const response = await portkey.chat.completions.create({
       model: 'llama3',
       messages: [{ role: 'user', content: 'Hello!' }]
   });
   
   console.log(response.choices[0].message.content);
   

   Or use custom host directly:

   Python

   from portkey_ai import Portkey
   
   portkey = Portkey(
       api_key="PORTKEY_API_KEY",
       provider="ollama",
       custom_host="https://your-ollama.ngrok-free.app"
   )
   

   import Portkey from 'portkey-ai';
   
   const portkey = new Portkey({
       apiKey: 'PORTKEY_API_KEY',
       provider: 'ollama',
       customHost: 'https://your-ollama.ngrok-free.app'
   });
   

Important: For Ollama integration, you only need to pass the base URL to customHost without the version identifier (such as /v1) - Portkey handles the rest!

---

Supported Models

Ollama supports a wide range of models including:

• Llama 3, Llama 3.1, Llama 3.2
• Mistral, Mixtral
• Gemma, Gemma 2
• Phi-3
• Qwen 2
• And many more!

Check Ollama’s model library for the complete list.

---

Next Steps

Gateway Configs

Add retries, timeouts, and fallbacks

Observability

Monitor your Ollama requests

Custom Host Guide

Learn more about custom host setup

BYOLLM Guide

Complete guide for private LLMs

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation