Promptfoo

Conventional application security has decades of tooling behind it because the failure modes are well understood. You know what SQL injection looks like and you can test for it deterministically.

AI applications fail differently. The vulnerability is not in the code — it is in what the system can be persuaded to do, which depends on the prompt, the tools it can call, the data it can reach and the specific model behind it.

Promptfoo's position is that this needs its own testing discipline, generated against your application rather than checked off a list.

Promptfoo: Red Teaming Before You Ship

• Attacks generated for your target, not a generic checklist — thousands of context-aware tests per application.
• Six product areas: red teaming, evaluations, code scanning, model security, guardrails and MCP.
• Insecure tool use in agents is a named category, which is the risk that grows as agents gain the ability to act.
• Open source with a commercial layer, installable with a single npx command.
• 156 of the Fortune 500 claimed as users, integrating with CI/CD, GitHub, GitLab and Jenkins, on-premise or cloud.

What Does Red Teaming An AI Application Mean?

Promptfoo states that it simulates real users to uncover application-specific vulnerabilities, generating custom attacks for your target including direct and indirect prompt injections, jailbreaks tailored to your guardrails, data and PII leaks, business rule violations, insecure tool use in agents and toxic content generation.

The phrase application-specific is the important one, and it distinguishes this from a scanner.

A generic checklist cannot find your vulnerabilities, because the interesting ones are about your application's rules. A customer service agent that can be talked into issuing a refund it should not, or into revealing another customer's order, is broken in a way no generic test describes.

Business rule violations is therefore the most underrated category on the list. Every deployed AI application has rules it is supposed to enforce — what it may promise, what it may disclose, what it may do on a user's behalf — and those rules exist in a prompt rather than in code.

Rules in a prompt are suggestions rather than constraints. Testing whether they hold under adversarial pressure is exactly what nobody does before shipping, and it is the gap this category exists to fill.

Rules written into a prompt are suggestions, not constraints. Whether they hold under pressure is exactly what nobody tests before shipping.

Prompt Injection And Why It Is Unsolved

Promptfoo names both direct and indirect prompt injection, and the distinction matters more than most teams realise.

Direct injection is a user typing something designed to override your instructions. It is the well-known case and the easier one, because the attacker is a person you can rate-limit and observe.

Indirect injection is the serious problem. An agent that reads a web page, a document or an email can encounter instructions planted in that content, and it has no reliable way to distinguish data it should process from instructions it should follow.

That is not a bug with a fix. It is a structural property of systems where instructions and data share a channel, and every mitigation so far is a filter that reduces the rate rather than a boundary that holds.

The practical consequence is that any agent processing untrusted content is exposed, and the mitigation is limiting what it can do rather than trying to make it immune — which is why insecure tool use is the category to test hardest, and why observability tooling like Langfuse matters alongside it.

Open Source With A Commercial Layer

Promptfoo offers a Try Open Source route and an npx command alongside a booked demo, with a large GitHub following displayed in the header.

For security tooling specifically, an open-source core is worth more than convenience.

The reason is inspectability. A closed security tool asks you to trust that it tests what it claims, and a security tool you cannot verify is a compliance artefact rather than a control.

It also lets you extend it. Your application's specific rules are yours, and adapting attack generation to test them properly requires being able to see and modify how attacks are constructed.

The commercial layer covers what open source usually cannot: continuous threat intelligence, managed infrastructure, reporting for people who do not read a terminal, and support. That is a reasonable division, and it means an individual engineer can start today without procurement — the same pattern that made code analysis tools such as DeepSource adoptable.

• Test business rule violations, not just injections — your rules are the specific risk.
• Assume indirect injection is possible and limit what agents can do instead.
• Run red teaming in CI so it happens on every change, not once before launch.
• Re-test after model version changes, since behaviour shifts underneath you.
• Read what the attacks actually were, not just the pass rate.

Testing Everything An Agent Can Reach

Promptfoo lists MCP and agent frameworks among its integration points, alongside CI/CD pipelines and on-premise or cloud deployment.

Covering agent frameworks specifically is the right emphasis, because the risk profile changed the moment AI systems could act rather than only answer.

A model that produces text can produce something wrong or offensive. A model that can call tools can send an email, modify a record, make a payment or read a file — and the consequence of persuading it to misbehave is no longer embarrassment.

MCP makes this concrete. Connecting an assistant to your systems through a protocol is enormously useful and it means the assistant's effective permissions are the union of everything it can reach.

The question worth asking of any agent deployment is the one security people ask about service accounts: what could this do at its worst, assuming an attacker controls its input entirely? Testing tool use is how you find out, and it is why this is a different discipline from evaluating output quality.

Where Red Teaming Reaches Its Limits

Automated adversarial testing is genuinely valuable and it is not a guarantee, which is worth stating plainly.

It finds attacks it knows how to generate. Novel techniques appear constantly, and a system tested clean today is tested against yesterday's threat model — which is why continuous threat intelligence is a real feature rather than an upsell.

It tests the system as configured. Change the model version, adjust the prompt, add a tool, and the previous results describe a system that no longer exists.

It cannot fix architecture. If an agent has permission to do something dangerous, testing tells you it can be persuaded to do it — the remedy is removing the permission, which is a design decision rather than a testing one.

And passing does not mean secure. It means the attacks generated did not succeed, which is a narrower claim and the honest way to read any security testing result.

Testing before release also needs a counterpart in production. What an application actually did, under real traffic and real inputs, is visible only through observability — the role tools like Helicone fill — and a system tested thoroughly and monitored not at all is only half instrumented.

Testing An AI Application Properly

The common pattern is shipping an AI feature with no adversarial testing at all, then adding it after an incident.

This sequence puts the testing where it catches things cheaply and keeps it running as the system changes.

1. List your application's actual rules — what it must never say, promise or do.
2. Run red teaming against those rules specifically, not just generic injections.
3. Read the successful attacks rather than the summary, since the technique is the lesson.
4. Reduce agent permissions wherever a test succeeded, rather than adding a filter.
5. Wire testing into CI so it runs on every prompt and model change.
6. Re-run after any model version update, since behaviour shifts without notice.

AI Security Terms

The vocabulary of AI application security, and what each failure actually is.

undefined

undefined

undefined

undefined

undefined

undefined