A former New York Times Covid reporter on the pandemic’s origins

Although it is one of the application’s most attractive features, the automatic routing service is also one of its pains. The routing service uses text similarity search in order to determine to which department a ticket should be routed. Most of the time it works well, but it suffers from a relatively high rate of classification errors. Therefore, you decide to replace it with a new routing service that uses an LLM in order to better understand the customer’s text in the ticket and improve the application routing feature. The new routing service is an example of an increasingly popular use case in which LLMs are used to replace a very specific and narrow function in a wider system. In this use case, the components that use an LLM are not required to have too much knowledge, memory, or deep reasoning capabilities. Rather, they are required to be able to understand and reason about a very specific type of question or problem (such as which department should handle a certain support ticket and why). Compared to the agentic AI vision that predominates the discourse these days, the idea of thin agents as LLM-powered microservices suggests a simpler and more pragmatic approach to the role of AI in distributed systems. It enables us to enhance specific functions in our application logic with relatively little overhead or “noise”, and focus on specific tasks rather than on our application infrastructure as a whole. It can be argued that even thin agents still involve the concerns implicated in maintaining AI infrastructure. One could obviously rely on the big SaaS LLMs in order to avoid managing this infrastructure, and most of the time this is probably the best way to go. But sometimes it is not. When scale and usage are high, then using the major LLM providers can get costly or create an unwanted dependency on a third party service. More importantly, when the relevant task requires domain specific knowledge and style (like understanding our support system and the relevant departments), then even the big LLMs might not be good enough or sufficiently stable. For these reasons, such use cases are usually pushed aside because they are perceived as complicated and resource-demanding. Even if we want to implement a thin LLM-powered service for a very specific and simple task, then we have to deal with fine-tuning, hosting, more expensive hardware, serving models at scale, etc.

A former New York Times Covid reporter on the pandemic’s origins

Новий список