Chat Operations
Chat | Answer Prompt
The Chat answer prompt
operation is a simple prompt request operation to the configured LLM. It uses a plain text prompt as input and responds with a plain text answer.
![Chat Answer Prompt](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fanswer-prompt.e9fdeda6.png&w=3840&q=75)
Input Configuration
Module Configuration
This refers to the MuleSoft Inference LLM Configuration set up in the Getting Started section.
General Operation Fields
- Prompt: Contains the prompt as plain text for the operation.
XML Configuration
Below is the XML configuration for this operation:
<mac-inference:chat-answer-prompt doc:name="Chat answer prompt" doc:id="f513c329-d277-41e6-932c-f5032b3365ac" config-ref="Github_Models" >
<mac-inference:prompt ><![CDATA[#[payload.prompt]]]></mac-inference:prompt>
</mac-inference:chat-answer-prompt>
Output Configuration
Response Payload
This operation responds with a json
payload containing the main LLM response. Additionally, token usage and other metadata are returned as attributes.
Example Response Payload
{
"response": "The capital of Switzerland is Bern. It's known for its well-preserved medieval old town, which is a UNESCO World Heritage site. Bern became the capital of Switzerland in 1848. The Swiss parliament, the Federal Assembly, is located in Bern."
}
Attributes
Along with the JSON payload, the operation also returns attributes, including information about token usage:
{
"tokenUsage": {
"outputCount": 9,
"totalCount": 18,
"inputCount": 9
},
"additionalAttributes": {}
}
- tokenUsage
- outputCount: The number of tokens used to generate the output.
- totalCount: The total number of tokens used for both input and output.
- inputCount: The number of tokens used to process the input.
Example Use Cases
This operation can be used in the following scenarios:
- Basic Chatbots: Answer simple user prompts.
- Customer Service Queries: Provide direct answers to frequently asked questions.
Chat | Completion
The Chat Completion
operation is useful when you want to retain conversation history for a multi-user chat operation. MuleSoft Object Store or any Database can hold previous user converation, which can be provided as messages into the Chat completion operation.
![Chat Answer Prompt with Memory](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fchat-completion.a856e208.png&w=3840&q=75)
Input Configuration
Module Configuration
This refers to the MuleSoft Inference LLM Configuration set up in the Getting Started section.
General Operation Fields
- Data: Contains the prompt for the operation.
- Memory Name: The name of the conversation. For multi-user support, enter the unique user ID.
- DB File Path: The path to the in-memory database for storing the conversation history. You can also use a DataWeave expression for this field, e.g.,
#["/Users/john.wick/Desktop/mac-demo/db/" ++ payload.memoryName]
. - Max Messages: The maximum number of messages to remember for the conversation defined in Memory Name.
XML Configuration
Below is the XML configuration for this operation:
<mac-inference:chat-completions doc:name="Chat completions" doc:id="b2c68037-6af9-4e2a-9297-c57749a38193" config-ref="Ollama" >
<mac-inference:messages ><![CDATA[#[payload.data]]]></mac-inference:messages>
</mac-inference:chat-completions>
Output Configuration
Response Payload
This operation responds with a json
payload containing the main LLM response, with additional metadata stored in attributes.
Example Response Payload
{
"response": "The capital of Switzerland is **Bern**. 🇨🇭 \n"
}
Attributes
Along with the JSON payload, the operation also returns attributes, including information about token usage:
{
"attributes": {
"tokenUsage": {
"outputCount": 15,
"totalCount": 37,
"inputCount": 22
},
"additionalAttributes": {
"finish_reason": "stop",
"model": "gemma2-9b-it",
"id": "chatcmpl-fc2425f6-0b70-4018-936f-f0df6c0b5da1"
}
}
}
- tokenUsage
- outputCount: The number of tokens used to generate the output.
- totalCount: The total number of tokens used for both input and output.
- inputCount: The number of tokens used to process the input.
- additionalAttributes
- finish_reason: The finish_reason for the LLM.
- model: The id of the model used.
- id: The id of the request.
Example Use Cases
This operation is particularly useful in scenarios where you want to retain a conversation history and provide context to the LLM:
- Customer Support Chats: Retaining the context of ongoing support conversations.
- Multi-user Chat Applications: Maintaining conversation history for different users.
- Personal Assistants: Keeping track of user interactions to provide more relevant responses.