Docs
MAC Inference
Supported Operations
Chat

Chat Operations

Chat | Answer Prompt

The Chat answer prompt operation is a simple prompt request operation to the configured LLM. It uses a plain text prompt as input and responds with a plain text answer.

Chat Answer Prompt

Input Configuration

Module Configuration

This refers to the MAC Inference LLM Configuration set up in the Getting Started section.

General Operation Fields

  • Prompt: Contains the prompt as plain text for the operation.

XML Configuration

Below is the XML configuration for this operation:

<mac-inference:chat-answer-prompt doc:name="Chat answer prompt" doc:id="f513c329-d277-41e6-932c-f5032b3365ac" config-ref="Github_Models" >
	<mac-inference:prompt ><![CDATA[#[payload.prompt]]]></mac-inference:prompt>
</mac-inference:chat-answer-prompt>

Output Configuration

Response Payload

This operation responds with a json payload containing the main LLM response. Additionally, token usage and other metadata are returned as attributes.

Example Response Payload

{
    "response": "The capital of Switzerland is Bern. It's known for its well-preserved medieval old town, which is a UNESCO World Heritage site. Bern became the capital of Switzerland in 1848. The Swiss parliament, the Federal Assembly, is located in Bern."
}

Attributes

Along with the JSON payload, the operation also returns attributes, including information about token usage:

{
  "tokenUsage": {
      "outputCount": 9,
      "totalCount": 18,
      "inputCount": 9
  },
  "additionalAttributes": {}
}
  • tokenUsage
    • outputCount: The number of tokens used to generate the output.
    • totalCount: The total number of tokens used for both input and output.
    • inputCount: The number of tokens used to process the input.

Example Use Cases

This operation can be used in the following scenarios:

  • Basic Chatbots: Answer simple user prompts.
  • Customer Service Queries: Provide direct answers to frequently asked questions.

Chat | Completion

The Chat Completion operation is useful when you want to retain conversation history for a multi-user chat operation. MuleSoft Object Store or any Database can hold previous user converation, which can be provided as messages into the Chat completion operation.

Chat Answer Prompt with Memory

Input Configuration

Module Configuration

This refers to the MAC Inference LLM Configuration set up in the Getting Started section.

General Operation Fields

  • Data: Contains the prompt for the operation.
  • Memory Name: The name of the conversation. For multi-user support, enter the unique user ID.
  • DB File Path: The path to the in-memory database for storing the conversation history. You can also use a DataWeave expression for this field, e.g., #["/Users/john.wick/Desktop/mac-demo/db/" ++ payload.memoryName].
  • Max Messages: The maximum number of messages to remember for the conversation defined in Memory Name.

XML Configuration

Below is the XML configuration for this operation:

<mac-inference:chat-completions doc:name="Chat completions" doc:id="b2c68037-6af9-4e2a-9297-c57749a38193" config-ref="Ollama" >
	<mac-inference:messages ><![CDATA[#[payload.data]]]></mac-inference:messages>
</mac-inference:chat-completions>

Output Configuration

Response Payload

This operation responds with a json payload containing the main LLM response, with additional metadata stored in attributes.

Example Response Payload

{
   "response": "The capital of Switzerland is **Bern**. 🇨🇭 \n"
}

Attributes

Along with the JSON payload, the operation also returns attributes, including information about token usage:

{
    "attributes": {
        "tokenUsage": {
            "outputCount": 15,
            "totalCount": 37,
            "inputCount": 22
        },
        "additionalAttributes": {
            "finish_reason": "stop",
            "model": "gemma2-9b-it",
            "id": "chatcmpl-fc2425f6-0b70-4018-936f-f0df6c0b5da1"
        }
    }
}
  • tokenUsage
    • outputCount: The number of tokens used to generate the output.
    • totalCount: The total number of tokens used for both input and output.
    • inputCount: The number of tokens used to process the input.
  • additionalAttributes
    • finish_reason: The finish_reason for the LLM.
    • model: The id of the model used.
    • id: The id of the request.

Example Use Cases

This operation is particularly useful in scenarios where you want to retain a conversation history and provide context to the LLM:

  • Customer Support Chats: Retaining the context of ongoing support conversations.
  • Multi-user Chat Applications: Maintaining conversation history for different users.
  • Personal Assistants: Keeping track of user interactions to provide more relevant responses.
AgentTools