Kotlin AI Development: Cobrainer’s JVM Stack

The AI integration was still product software

The core platform already ran on Kotlin and Spring Boot. Domain logic, SQL access through Exposed, serialization through kotlinx.serialization, and async work through coroutines all lived in the same codebase. The LLM layer mostly connected to hosted model APIs. Swapping that layer to a different runtime wouldn't magically make the external models better - it would just add another deployment path and more separation between the AI layer and the domain model.

Those boundaries add cost. A second language can mean duplicated data types, extra serialization steps, separate build/release processes, and more places to look when a request fails. Sometimes that's worth it - if a team is doing model development, relies on Python-specific research tooling, or already has a mature Python platform. But it's not automatically justified just because you're adding an LLM API call.

Keeping the work on the JVM also made the contracts easier to review. Kotlin's null safety and type system make assumptions visible at compile time. Sealed types can model a defined set of outcomes, and exhaustive when checks force callers to handle each case. None of this guarantees correct model output. It does help contain uncertainty once the response is parsed into application-level data.

Optimize for the team's whole system

A monolith isn't automatically the right choice, but a small team should take the operational cost of additional services seriously. Every new deployable needs pipelines, monitoring, access controls, incident experience, and ongoing upgrades. If an AI service is worth splitting out - because you need independent scaling, a separate security boundary, or something similarly concrete - it can make sense. If it exists mainly because "AI means Python," it might just be added complexity without a real payoff.

Cobrainer's approach was to evaluate frameworks by building working implementations, not by comparing feature lists. The requirements were straightforward:

• Keep control flow easy to follow instead of burying it under large abstractions.
• Support multiple providers and handle differences in model capabilities.
• Make it work in a fast-changing model ecosystem without turning framework upgrades into a bottleneck.
• Carry typed contracts from the app code through serialization and back.

The point isn't that Kotlin is inherently better for AI. It's that production AI has to fit the system around it. For a JVM team using hosted models, staying in the existing stack reduces handoffs and preserves the tools the team already knows.

How the frameworks differed in this case

Spring AI gave the most complete Spring integration. Cobrainer still uses it for MCP endpoints, where the integration with REST infrastructure and the Spring Security filter chain is useful. In the agentic core, though, the team wanted a more direct way to express orchestration and iterate on behavior.

LangChain4j brought lots of features and familiar LangChain concepts to the JVM. The team could move quickly, but they found some failures inside a chain hard to trace back to the actual model call. Eventually, they replaced the remaining utilities that depended on it with a small amount of Kotlin code. That's a trade-off from their specific situation, not a blanket verdict on the library.

Koog was the better fit because its abstractions are built on Kotlin rather than forcing developers into a different programming model. The team could use serializable domain types as structured-output contracts, coroutines for async calls, and Kotlin DSL patterns to define agent graphs. The resulting orchestration stayed in the kind of code developers could read and inspect with familiar tooling.

Keep abstraction useful and observable

An AI framework should reduce repetitive integration work without hiding the parts engineers need to debug. Cobrainer wanted enough structure to reuse provider clients and orchestrate agents, but not so much that failures vanished inside an opaque chain.

Koog's provider and model capability abstractions mattered too. Not every model supports the same features. Depending on the target model, you may need to adjust things like temperature settings, token limits, or how structured output is handled. In Cobrainer's setup, the executor adapts the prompt for those differences while keeping the same domain contract and the same call site.

That design separates two concerns:

1. The application contract: what the domain expects to receive and work with.
2. The provider adaptation: how a specific model is asked to fulfill that contract.

The call site doesn't need to know which provider ends up answering. That makes fallback easier without scattering provider-specific branches through business logic. It also gives an engineer a clear path to trace from a domain type to its LLM contract and back to consumers.

Cobrainer moved its proof of concept into production for all customers in one day after cleaning it up. That wasn't just a framework feature. It came from being able to keep the existing domain model, serialization approach, concurrency model, tests, and deployment pipeline. When you evaluate a framework, the real question isn't only "Can it produce a demo quickly?" It's "How much of the production machinery can you keep using after the demo works?"

Make the contract explicit

Cobrainer represents possible changes with a Kotlin sealed hierarchy and kotlinx.serialization. Koog can use those serializable types to produce a JSON schema, including descriptions supplied through annotations. The application then receives a typed result instead of treating arbitrary text as trusted domain data.

A simplified example looks like this:

@Serializable
sealed interface ProposedChange {
@Serializable
@SerialName("rename")
data class Rename(
val itemId: String,
val newTitle: String,
) : ProposedChange
@Serializable
@SerialName("archive")
data class Archive(
val itemIds: List<String>,
) : ProposedChange
}

This reduces ambiguity, but you still need validation. Providers differ in how well they support native structured output, and weaker models can return malformed - or just "almost valid" - JSON. Cobrainer uses a structure-fixing parser with a lightweight model and a limited number of retries before the result reaches application code.

Once parsed, the sealed hierarchy helps consumers at compile time. An exhaustive when makes new change types visible to every consumer that has to handle them. If you add a required field, you'll also find call sites that need updates. That's safer than hoping everyone follows conventions embedded in prompts.

But the boundary still needs tests. A type-safe app can only be as reliable as the parsing and provider behavior that feed it. The type system enforces the shape after successful parsing; it doesn't prove the model's proposed action is correct or authorized.

Use agents for decisions, code for guardrails

Cobrainer's Job Architecture agent lets users request changes conversationally - for example, asking it to merge levels or draft missing descriptions. The design is a supervisor loop combined with deterministic sub-workflows: the model proposes what to do, application code routes consequential actions through controlled flows, and a human approves each change.

That separation matters. An agent can help interpret a request and prepare a proposed change, but it shouldn't become the authority that grants itself permission to act. Cobrainer exposes tools through central MCP servers and uses its existing authorization infrastructure. Permission scopes determine which tools the agent can see before a model call. The agent then operates with a short-lived delegated token, so it acts "as the user," not as an unrestricted service identity.

The agent is the main exception to Cobrainer's central monolith. It runs in a separate reactive Kotlin service built for long-running conversations, streaming, and structured concurrency. The services share types in one monorepo, while platform data stays behind the monolith in Postgres through Exposed. SurrealDB stores agent checkpoints, message history, and memory.

In other words: the useful lesson isn't "make agents a microservice." It's "separate the runtime when the workload has materially different needs" - here, long-running and resumable conversations. Keep the boundary narrow enough that shared contracts don't turn into a maintenance headache.

The product also uses graph events to show meaningful progress, like preparing changes for review, rather than assuming token streaming is always the best UX. For multi-step work, a clear progress signal can communicate more than a stream of words. The right feedback depends on what the user is waiting for and what state they need to understand.

Fallback chains should cross failure domains

Cobrainer's prompt executor adds retries with backoff, uses a configuration-driven provider fallback chain, and includes a circuit breaker for each model. Before each attempt, it adapts the prompt to the target model's capabilities. If a provider throttles requests or a model becomes unhealthy, the circuit breaker can remove that route from rotation while requests try alternatives.

A fallback chain built from multiple models within the same provider won't help much during a provider-wide capacity or availability problem. Crossing provider boundaries reduces that shared dependency, but it introduces trade-offs: differences in behavior, latency, cost, data handling, and structured-output support. Teams should test and monitor those differences instead of assuming a fallback model is interchangeable.

The earlier incident showed what the setup was trying to achieve. The customer workflow kept running, and the provider issue showed up in the LLM observability dashboard afterward. That's not an argument for hiding failures. It's the reason to make retries, fallback decisions, and model health observable, so the team can tell resilience from silent degradation.

Measure the whole request path

Cobrainer traces attempts and fallbacks through LLM observability, including latency, token usage, and errors. That makes practical questions answerable: did the provider fail, did parsing fail, or did application logic reject the result? Which fallback handled the request? Did latency or cost change after switching routes?

Contract tests complement monitoring. The team runs structured-output tests with real provider calls and parsing, alongside LLM contract tests in CI. Real calls reveal provider behavior that mocks won't reproduce, while CI checks catch changes before rollout. Neither removes model variability, but together they reduce the amount of guesswork required to ship.

The five-times gain came from coroutine cleanup

One of Cobrainer's biggest performance improvements didn't come from Koog or from any model change. Legacy blocking code inside coroutine paths starved dispatchers under AI-driven load. After the team found and removed the blocking behavior, throughput on the affected paths increased fivefold.

The operational warning is specific: avoid leaving runBlocking in production request paths. It can look harmless - especially in a blocking Spring Boot application - but under load it can tie up resources that coroutine work needs. In the failure mode described by Cobrainer, the app didn't necessarily crash; requests just stopped completing.

This is why AI performance investigations have to include the surrounding application. More model traffic can surface thread-pool starvation, blocking database calls, queue limits, and other bottlenecks that were rare before. If latency rises, swapping in a faster model might only treat the symptom while the real constraint stays in place.

Choose the boundary that fits your constraints

Cobrainer's results don't prove that every team should switch to Kotlin, Koog, or a monolith. What they show is how a team made the choices fit an existing system.

Python still makes sense for model research and training. Splitting out a separate service may also be justified by team skills, deployment needs, scaling requirements, or data isolation.

For a JVM team integrating hosted models, this case suggests a practical decision checklist:

• Does the AI work mostly call and orchestrate external APIs, rather than train models?
• Can your existing domain types express the outputs you need?
• Will one runtime reduce deployment and ownership overhead for your team?
• Can you observe provider attempts, validate outputs, and enforce permissions outside prompts?
• Does the framework keep failures and control flow inspectable?

If those answers point back to your existing JVM stack, Kotlin AI development is a credible production option - not a compromise made for novelty. If they don't, choose the extra runtime or tooling for a concrete operational reason.

For more detail, see Cobrainer's original engineering account, the Koog documentation, and the Kotlin documentation.