Langfuse

Debugging a language-model application is unlike debugging anything else, and the reason is that the failure is usually not an error.

The request succeeded, the response came back, the status code was 200, and the answer was wrong. Nothing in a conventional monitoring stack notices that, because nothing in a conventional monitoring stack reads the content.

Langfuse exists for that gap, and its most consequential decision is not technical. Every product feature is MIT licensed, which changes who can adopt it and what happens if the company's plans and yours diverge.

The Langfuse Summary

• All product features are MIT licensed — not a cut-down community edition, which is the unusual part.
• Hierarchical tracing captures LLM calls, tool invocations and retrieval steps as a tree, rather than logging completions in isolation.
• Evaluation three ways: LLM-as-a-judge, heuristic functions, and human review with collaborative annotation workflows.
• Self-host or use the cloud — Docker Compose, Kubernetes via Helm, or Terraform for AWS, GCP and Azure; managed cloud in US and EU regions.
• SOC 2 Type II and ISO 27001, GDPR compliance, and a HIPAA-ready region.

Why Does MIT Licensing Matter Here?

Langfuse states that all product features are MIT licensed and that you can fork, modify and contribute. That sentence is doing a great deal of work, and it is worth unpacking against how this category normally operates.

The standard arrangement is open core: a permissively licensed base with the features you actually need — SSO, audit logs, retention controls, sometimes the evaluation tooling — held behind a commercial licence. The open version exists to get you in, and the interesting part is not open at all.

MIT across the product means the escape hatch is real rather than theoretical. If the pricing changes, the roadmap moves away from you, or the company is acquired by someone whose plans you dislike, you can run what you already have indefinitely and modify it.

That matters more for observability than for most software, because observability is where your history lives. Migrating a year of traces, evaluations and prompt versions to a different platform is not a weekend, and a vendor knows it — which is precisely why lock-in in this category is so valuable to them and so expensive to you.

The honest counterweight is that a licence is not a support contract. Running Langfuse yourself means you own the upgrades, the storage growth and the 3am page, and for most teams the managed cloud is the correct choice regardless. The licence is insurance rather than a plan.

The alternative shape is a gateway. Helicone is also open source and sits in the request path instead, which is a different answer to the same worry about who holds your history.

MIT across the whole product means the escape hatch is real rather than theoretical. It is insurance, not a deployment plan.

What A Hierarchical Trace Captures That Logging Does Not

Langfuse's observability captures hierarchical traces covering LLM calls, tool invocations and retrieval steps, and the word hierarchical is the important one.

A flat log tells you a completion happened, what went in and what came out. That is enough for a single-call application and completely insufficient for an agent, because an agent's answer is the product of a sequence: retrieve, reason, call a tool, read the result, reason again, answer.

When the final answer is wrong, the question is which step went wrong — and a flat log cannot tell you, because it does not record that the steps were related. A trace tree does, and that is the difference between debugging and guessing.

The most common finding, once teams can actually see this, is that the model was fine and the retrieval was not. The prompt asked a reasonable question of documents that did not contain the answer, and the model did what models do with insufficient context.

That single insight changes where the fix goes. Without traces, teams spend weeks tuning prompts to fix a retrieval problem, which is an expensive way to make no progress.

Three Kinds Of Evaluation, And When Each One Lies

Langfuse offers LLM-as-a-judge, heuristic functions and human annotation, and using all three deliberately is better than picking one.

Heuristics are cheap, fast and honest about their limits. Does the output parse as JSON, contain the required field, stay under the length limit, avoid a forbidden term? These catch a surprising proportion of real failures and never disagree with themselves.

LLM-as-a-judge scales to the questions heuristics cannot express — is this answer relevant, is it grounded in the retrieved context, is the tone right. It is genuinely useful and it inherits every weakness of the model doing the judging, including a well-documented preference for longer, more confident answers over shorter correct ones.

Human annotation is the only ground truth, and it is expensive, which is why it should be spent where it settles disagreements rather than sprayed across everything. The right use is calibration: annotate a sample, check whether your judge agrees with your humans, and only then trust the judge on volume.

That calibration step is the one teams skip, and skipping it means your evaluation numbers measure the judge's preferences rather than your users' experience. Langfuse's human review workflows exist precisely so that step is possible.

This is also where framework-tied tooling such as LangSmith and evaluation-led platforms like Braintrust weight things differently, and the right choice depends on whether evaluation or tracing is your bottleneck.

• Use heuristics for anything expressible as a rule — they never drift.
• Use LLM-as-a-judge for relevance, grounding and tone, and expect a bias toward long confident answers.
• Spend human annotation on calibrating the judge, not on grading everything.
• Re-calibrate after any model change, since the judge's behaviour moves too.
• Treat a rising eval score with a flat user metric as evidence the eval is wrong.

Prompt Management Is Version Control For The Part That Changes Most

Langfuse offers prompt management with version control and one-click deployment, plus a playground that runs prompts against production inputs.

The problem it solves is unglamorous and universal. Prompts start in code, then someone needs to change one without a deploy, so they move to a config file, then to a database, and within a quarter nobody knows which version is live or who changed it last.

Versioning them properly makes regressions attributable. When quality drops on Tuesday, the useful question is what changed, and a system that can answer that in seconds is worth more than one that requires archaeology.

The playground running against production inputs is the detail that separates this from a text editor. Testing a prompt on examples you invented tells you it works on examples you invented; testing on the requests that actually failed tells you something.

Experiments close the loop by letting you define test cases and compare results across versions, which is what turns prompt changes from craft into something with evidence behind it.

None of this is visible to conventional monitoring. Sentry Seer will tell you the request threw; it has no view on a prompt change that quietly made every answer worse.

Instrumenting An Application So The Traces Are Worth Reading

The failure mode with any observability platform is not that it does not work — it is that the traces are too thin to answer the question you eventually ask.

Langfuse offers native Python and TypeScript SDKs and states 100+ integrations, including LangChain, the Vercel AI SDK, LiteLLM, Pydantic AI, CrewAI and OpenAI. An integration gets you started; it does not decide what you record.

1. Start with the framework integration if you use one, since it captures the call structure with almost no work.
2. Add session or user identifiers immediately — a trace you cannot tie to a conversation is far less useful than one you can.
3. Instrument retrieval explicitly, including what was retrieved, because that is where most quality problems actually live.
4. Record tool calls and their results, not just the model's decision to call them.
5. Add cost and latency dashboards before you need them, since the first billing surprise is always retrospective.
6. Write two or three heuristic evaluations on day one, then calibrate a judge against a human-annotated sample before trusting any score.