AI API Glossary: Terms Every Developer Should Know

A reference, not a tutorial

This is a lookup page. If you hit an unfamiliar term in a provider's docs or a blog post, it's probably here — with a plain-language definition and, where relevant, a link to a deeper guide.

Core concepts

Token — The basic unit LLMs process text in, roughly a word or word-fragment. Pricing, context windows, and rate limits are all measured in tokens, not characters or words. See What Are Tokens?.

Context window — The maximum number of tokens (input + output combined) a model can handle in a single request. Exceeding it causes the request to fail outright.

Input tokens / output tokens — Tokens in your prompt (input) vs. tokens the model generates in response (output). Output is almost always priced higher per token than input.

Temperature — A parameter controlling response randomness. Low temperature (near 0) gives consistent, deterministic-leaning output; high temperature gives more varied, creative output. See Temperature, top_p, top_k Explained.

System prompt — Instructions given to the model before the conversation starts, defining its role, tone, and constraints. Distinct from the user's actual message.

Techniques

Prompt engineering — Structuring instructions and examples in a prompt to get better, more consistent output, without changing the model itself.

Few-shot prompting — Including a small number of example input/output pairs in the prompt to demonstrate the desired pattern, as opposed to zero-shot (no examples).

Chain-of-thought (CoT) — Prompting a model to reason step by step before giving a final answer, which often improves accuracy on complex tasks.

RAG (Retrieval-Augmented Generation) — Fetching relevant documents at request time and injecting them into the prompt, so the model can answer using information it wasn't trained on. See RAG Explained.

Fine-tuning — Retraining a model's weights on your own examples to change its behavior consistently, as opposed to changing the prompt each time. See Fine-Tuning vs RAG vs Prompt Engineering.

Function calling / tool use — Letting a model request that your application execute a specific function (with structured arguments) rather than just generate text — the mechanism behind AI agents that can take actions.

Infrastructure and cost terms

Embedding — A numeric vector representation of text, used to measure semantic similarity between pieces of content. The foundation of RAG's retrieval step and semantic search.

Semantic cache — A cache that matches by meaning rather than exact text, so similar-but-not-identical queries can reuse a previously computed response.

Prompt caching — A provider-side feature that discounts the cost of reprocessing repeated context (like a large system prompt) across multiple requests. See Prompt Caching Explained.

Reasoning tokens — Hidden "thinking" tokens a reasoning model generates before its final answer, billed at the output token rate even though they're not shown to the user. See Reasoning Models.

Rate limit (RPM) — The maximum number of requests per minute an API key is allowed to make before receiving a 429 error.

Circuit breaker / fallback routing — A pattern where requests automatically redirect to an alternate model or provider if the primary one is failing or slow, to keep an application responsive.

Model and provider terms

LLM (Large Language Model) — A model trained on large amounts of text to predict and generate language; the underlying technology behind GPT, Claude, Gemini, and similar systems.

LLM gateway — Middleware that sits between your application and one or more LLM providers, offering a unified API, billing, and routing across them. See What Is an LLM API Gateway?.

Open-weight model — A model whose trained parameters are published and can be run independently, as opposed to a closed model only accessible via an API. See Open-Source vs Closed Models.

Multimodal — A model that accepts and/or generates more than just text — images, audio, or video alongside text.

Guardrails — Server-side controls (PII masking, keyword blocking, content filtering) applied to requests before they reach an LLM, to enforce safety or compliance policies. See Guardrails in Production.

Frequently Asked Questions

What's the difference between a token and a word?

A token is roughly 0.75 words for English text, but the exact mapping varies by tokenizer and language — see What Are Tokens? for the full explanation.

Is "LLM" the same thing as "AI"?

No — LLMs are one category of AI model, specifically trained on language. AI is the broader field including image models, recommendation systems, and many other approaches.

What does "OpenAI-compatible" mean?

It means an API implements the same request/response format as OpenAI's chat completions endpoint, so existing code written for the OpenAI SDK works against it with just a base URL change — this is how AICredits provides access to non-OpenAI models like Claude and Gemini through the same SDK.

Get started

• Create a free account and start experimenting with these concepts directly.
• Bookmark the docs for deeper reference material on any term above.