Agents as Products

What is required to manage agents as products? First of all, like any other product in your organisation, you need to be able to manage the complete lifecycle of the product involving all stakeholders needed.

First you determine the agent's job-to-be-done. After you have identified the job of the agents, it is important to start thinking about the design aspects.

Design Phase

Define the capabilities of the agent:

• Model support: Which models should the agent have access to?
• Token Strategy: What is the token strategy? Prompt vs Generation Token Control.
• Token Usage: How to measure and analyze the token consumption?
• Toxicity Detection: Support of moderation models, on Prompt and/or Generation.
• Tooling: Access to tools, should the agent have the ability to add dynamic tools?
• Knowledge: What are the vector databases it connects to, one or multiple?
• Prompt Templates: Is the agent using templates to refine its guardrails?
• Prompt Decoration: Does the agent support decoration of prompts?
• Conversation Memory: Does it need / support conversation memory?
• Interaction modes: Synchronous, asynchronous. On which channels is it interacting?
• Ressources: How can the agent be leveraged?
• Data Types: What are the data types it can work with?
• Parameters: What are the parameters for this agent?

Prototype Phase

Once the design is completed, implement the prototype, including mocking. Implement the resources; how the agent is going to interact with endpoints, methods, data types and parameters. Also define the schemas for the prompt and response structures, token usage and control as well as Tooling capabilities. Document how a developer should interact with the composable agent.

Share Phase

At this stage involvement of stakeholders is important for collaboration and feedback. Understanding the benefits of composability and how to potentially reuse the agent in different systems and applications can improve the quality of the agent and improve the time to market for future initiatives. Essentially, you would need a place to publish the prototype so other collaborators can provide feedback. Publishing the assets to a dedicated repository or portal will encourage the collaboration between teams.

Build Phase

The build phase can start once all parties have agreed on the specification and prototype. During development, the agent capabilities, which were defined in the design phase, are implemented. During the implementation, the following key aspects must to be considered:

• What assets can be reused from previous implementation?
  • Any Knowledge Bases or RAG techniques
  • Any System / Process APIs
  • Any Voice capabilities, i.e. Speech-to-text and Text-to-speech
  • Any Prompt Templates
  • etc.
• What models need to be accessible?
  • Does the agent support multiple model strategy?
  • Should user bring their own APIKey?
• What data need to be logged for reporting purposes?
  • Token Usage
  • Request / Response
  • etc.
• Does a Toxicity detection need to be in place?
• Conversation History needed?

Test Phase

Testing agents is very important, also for any future changes of the agent. It is important to define the unit & integration tests for the agent implementation to verify what has been agreed at design time.

In best case, the test case have been created with the Agent-Test-First approach right after the design phase. These test cases can now validate the agent implementation.

Deploy Phase

The iterative deployment from test, staging to production. The decision needs to be made where the agent will be deployed, is it in the cloud, or on-premise. (more details to follow)

Secure Phase

Implementation of agent / LLM policies, which will secure the interaction between user and agent. (more details to follow)

Operate Phase

Operational activities for the agent: Analyzing logs, observing behaviour, and maintenance tasks. (more details to follow)

Monitor Phase

Once deployed in production, the monitoring of agents is very important.
You need to operate not only the technical architecture, but also make sure you can extract some business relevant data and insights on token governance, popular agents, etc.

Publish & Reuse

The agent prototype has already been published and shared with other teams, now the implementation of the agent can be shared and teams can start reusing the agent. (more details to follow)