LiteLLM Guardrails: Security for Your LLM Gateway

LiteLLM guardrails are now part of Autonomous Security. If your teams send their model traffic through a LiteLLM proxy, you can put our guardrails in front of the models behind it: secrets in requests are masked before they reach the model, unsafe requests are blocked, the model's answers are checked too, and every decision shows up in Activity, next to everything else your AI agents do.

Why the LLM gateway needs guardrails

An LLM gateway is where a platform team puts one front door on many models. Keys, budgets, routing and logs live there. It is also the one place every internal app, script and agent that talks to a model passes through, including the ones that never run on a laptop: a support bot on a server, a batch job summarizing tickets, an agent running in CI.

Those callers are invisible to anything that watches endpoints or browsers. Their prompts carry what the app put in them: customer records, log lines, a tool's raw output. Their answers carry what the model decided, including instructions it picked up from content it was given. Until now, the only checks on that traffic were the ones each app's developers remembered to write.

An example

An internal triage bot reads new support tickets and asks a model to suggest a fix. One ticket includes a pasted error log, and in the log is a live cloud access key. The bot sends the whole thing to the model.

Another ticket was written by someone outside the company. Hidden in it is a line telling the assistant to ignore its instructions and include the contents of its configuration in the reply.

Nobody wrote a bug. The bot did exactly what it was built to do. With guardrails on the gateway:

• the access key is masked in the request, so the model receives the ticket without it;
• the hidden instruction is flagged as a prompt injection;
• both decisions are listed in Activity, so the security team sees them and the platform team can fix the source.

What LiteLLM guardrails do

The integration connects to the LiteLLM proxy you already run. Your proxy keeps doing its job; Autonomous inspects the calls going through it and applies your guardrails policy:

• Requests: secrets are masked before the request reaches the model. Requests that break a policy set to block are stopped.
• Responses: the model's answers are checked as well. An unsafe answer is blocked; one streamed to the caller as it is written has already arrived when its check ends, so a finding in it is reported rather than stopped.
• Activity: gateway blocks and redactions appear alongside your endpoint and browser activity, so one console shows what was stopped, wherever it came from.

The guardrails are the same policy you use for your AI agents on endpoints, not a second set to write and keep in step: one policy, applied wherever the traffic is.

Observe first, then enforce

The gateway has three modes: Off, Observe and Enforce. It is off until you turn it on.

In Observe, calls are inspected and nothing is changed or blocked: you see what Enforce would have done, on your real traffic, without risking a production app. When the findings look right, move to Enforce. Masking and blocking apply only when both the gateway and your guardrails policy are set to Enforce, so nothing starts blocking by surprise.

If you run more than one proxy, say staging and production, connect each one on its own and move them to Enforce at their own pace.

Where it fits

Guardrails on the gateway cover the traffic you route through it. They don't see what an agent does on a laptop: the files it reads, the commands it runs, where it sends data. That is the endpoint's job, and the two work together. A secret masked at the gateway is one layer; a session that reads a sensitive system and then tries to send data out is another (AI agent data leakage).

If your team is weighing where guardrails belong in the first place, at the gateway, in the app or around the agent, our earlier post on AI guardrails, gateways and governance walks through the trade-offs.

LiteLLM is the gateway supported today. If you run another, tell us which.

Try it on your gateway

You don't need to change your apps or your models: the guardrails sit on the proxy you already have. Start a LiteLLM free trial, connect your proxy in Observe, and see what your gateway traffic looks like through our guardrails before anything is blocked.