LlamaIndex in Production: RAG Pipeline with Cost Tracking in ₹

Where a RAG pipeline actually spends money

Most RAG cost discussions focus on the generation step — the final LLM call that produces an answer. In practice, a production LlamaIndex pipeline spends money in three separate places: embedding documents at ingestion time, embedding the query at retrieval time, and generating the final response. The first of these is often the largest cumulative cost if your document set is big or changes frequently, yet it's the one teams forget to budget for.

Setup with an OpenAI-compatible endpoint

LlamaIndex's OpenAILike classes accept a custom api_base, which is all you need to route through AICredits:

from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai_like import OpenAILikeEmbedding
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
 
Settings.llm = OpenAILike(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_base="https://api.aicredits.in/v1",
    api_key="sk-your-aicredits-key",
    is_chat_model=True,
)
 
Settings.embed_model = OpenAILikeEmbedding(
    model_name="text-embedding-3-small",
    api_base="https://api.aicredits.in/v1",
    api_key="sk-your-aicredits-key",
)
 
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

Querying and tracking cost per query

query_engine = index.as_query_engine()
response = query_engine.query("What is our refund policy for annual plans?")
print(response)
 
# LlamaIndex exposes token usage via callback handlers for cost tracking
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
import tiktoken
 
token_counter = TokenCountingHandler(tokenizer=tiktoken.encoding_for_model("gpt-4o").encode)
Settings.callback_manager = CallbackManager([token_counter])
 
# After running queries:
print(f"Embedding tokens: {token_counter.total_embedding_token_count}")
print(f"LLM prompt tokens: {token_counter.prompt_llm_token_count}")
print(f"LLM completion tokens: {token_counter.completion_llm_token_count}")

Cost breakdown by stage

For a knowledge base of 500 pages (~250,000 tokens) re-indexed monthly, plus 10,000 queries a month at 5 retrieved chunks (~800 tokens context) per query, answered with Claude 3.5 Sonnet:

| Stage | Volume | Model | Approx. monthly cost | |-------|--------|-------|------------------------| | Document embedding (re-index) | 250,000 tokens | text-embedding-3-small | ~₹0.46 | | Query embedding | 10,000 queries × ~15 tokens | text-embedding-3-small | negligible | | Retrieval + generation | 10,000 queries × (800 input + 150 output) | Claude 3.5 Sonnet | ~₹4,375 |

The pattern holds across most RAG deployments: embeddings are nearly free, and the generation step dominates. If cost is the bottleneck, the highest-leverage change is usually swapping the generation model, not the embedding model.

# Switching the generation model to cut cost ~10x on this workload
Settings.llm = OpenAILike(
    model="google/gemini-2.0-flash",
    api_base="https://api.aicredits.in/v1",
    api_key="sk-your-aicredits-key",
    is_chat_model=True,
)

Reducing retrieval cost without hurting quality

• Lower similarity_top_k if 5 chunks retrieve mostly redundant context — 3 chunks often perform just as well at 40% less input token cost.
• Chunk size matters twice: larger chunks cost more per embedding call and per retrieval, but too small loses context coherence. Start around 512 tokens and tune from actual query results, not intuition — the RAG chunker tool helps visualize this trade-off.
• Cache repeated system instructions with prompt caching if your query engine's prompt template is large and reused across every query — see the prompt caching guide.

Frequently Asked Questions

Which embedding model should I use for a production RAG pipeline?

text-embedding-3-small is the default choice for most applications — cheap and strong enough for typical retrieval tasks. Reach for text-embedding-3-large only if retrieval quality specifically improves in your evaluation, since it costs roughly 6x more per token.

Does LlamaIndex support streaming responses through AICredits?

Yes — query_engine.query(..., streaming=True) streams tokens as they're generated, using the same OpenAI-compatible streaming format AICredits implements.

How do I avoid re-embedding my entire document set on every deploy?

Use LlamaIndex's persistent storage (index.storage_context.persist()) or a vector store integration, and only re-embed documents that changed since the last index build — most production setups track document hashes to skip unchanged content.

Get started

• Create a free account and get an API key for both embeddings and generation.
• Estimate your own pipeline's cost with the calculator.
• Related: RAG Explained: Build an AI That Knows Your Own Data.