Embedding Operations

Supported Model Providers

Azure OpenAI (opens in a new tab) (Microsoft): Managed access to OpenAI models through Azure's cloud platform.
Azure Vision AI (opens in a new tab) (Microsoft): Provides developers with access to advanced algorithms for processing images and returning information.
Einstein (opens in a new tab) (Salesforce): AI platform integrated into Salesforce for CRM automation and insights.
Google Vertex AI (opens in a new tab) (Google): Generative AI models hosted on Google's advanced, global infrastructure.
Hugging Face (opens in a new tab) (Hugging Face): Community-driven hub for machine learning models, datasets, and tools.
Mistral AI (opens in a new tab) (Mistral): Open-weight LLMs optimized for efficiency and customization in AI projects.
Nomic (opens in a new tab) (Nomic): Tools for understanding and visualizing large datasets with embeddings and AI.
Ollama (opens in a new tab) (Ollama): Platform offering tools and APIs for embedding-based search and AI-driven insights.
OpenAI (opens in a new tab) (OpenAI): Developer of advanced AI models like GPT and DALL·E for diverse applications.

Embedding | Generate from Text

The [Embedding] Generate from text operation optionally split the text into chunks of the provided size and creates numeric vectors for each text chunk.

How to Use

The [Embedding] Generate from text operation can be followed by either the [Store] Add or the [Store] Query operations. The output payload is ready to be used by both [Store] operations without any transformation.

Add Text to Store

When used in combination with [Store] Add operation the text along with the generated embeddings can be ingested into a vector store.

Generate Embeddings from Text - Add to Store

Query from Store

⚠️

When generating an embedding from text for query purposes do not provide any segmentation parameter. Leave blank Max Segment Size (Characters) and Max Overlap Size (Characters).

When used in combination with [Store] Query operation the provided text is first used to generated an embedding that is then used to perform a query against the vector store.

Generate Embeddings from Text - Query from Store

Input Fields

Module Configuration

This refers to the MuleSoft Vectors Embedding Configuration set up in the Getting Started section.

General

Input Texts (Array<String>): The text segments to generate embeddings for.

Embedding Model

Embedding Model Name: Indicates the embedding model to be used.

XML Configuration

Below is the XML configuration for this operation:

<ms-vectors:embedding-generate-from-text 
  doc:name="[Embedding] Generate from text" 
  doc:id="2835a3fe-28e1-4407-b4c0-ff681923d8ea" 
  config-ref="Einstein_Embedding_config" 
  embeddingModelName="sfdc_ai__DefaultOpenAITextEmbeddingAda_002"/>

Output Fields

Payload

This operation responds with a json payload.

Example

Here an example of the JSON output.

{
    "embeddings": [
      [-0.00683132, -0.0033572172, 0.02698761, -0.01291587, ...],
      [-0.0047172513, -0.03481483, 0.02046227, -0.037395656, ...],
      ...
    ]
    "text-segments": [
        {
            "metadata": {
                "index": "0"
            },
            "text": "In the modern world, technological advancements have become .",
        },
        {
            "metadata": {
                "index": "1"
            },
            "text": "E-commerce giants like Amazon and Alibaba have redefined ..",
        },
        ...
    ],
    "dimension": 1536
}

embeddings: The list of generated embeddings
- list-item (embedding)
text-segments: The list of segments.
- list-item (text-segment):
  - text: The text segment
  - metadata: The metadata key-value pairs.
    - index: The segment/chunk number for the uploaded data source.
dimension: The dimension of the selected embedding model.

Attributes

EmbeddingResponseAttributes:
- embeddingModelDimension: The dimension for the embedding model used.
- embeddingModelName: The embedding model name used.
- tokenUsage: The token usage for the embedding model used.
  - inputCount: The number of tokens used as input.
  - outputCount: The number of tokens used as output.
  - totalCount: The total number of tokens used.

Embedding | Generate from Media

The [Embedding] Generate from media operation generates numeric vectors from media content (images, audio, video, etc.).

How to Use

The [Embedding] Generate from media operation can be followed by either the [Store] Add or the [Store] Query operations. The output payload is ready to be used by both [Store] operations without any transformation.

Add Media to Store

When used in combination with [Store] Add operation the generated embeddings can be ingested into a vector store.

Generate Embeddings from Media - Add to Store

Query from Store

When used in combination with [Store] Query operation the provided binary is first used to generated an embedding that is then used to perform a query against the vector store.

Input Fields

Module Configuration

This refers to the MuleSoft Vectors Embedding Configuration set up in the Getting Started section.

General

Binary: The media binary to generate embeddings for.
Media Label: A short description/caption for the media.
Media Type: The type of the media. The default value is image.

Embedding Model

Embedding Model Name: Indicates the embedding model to be used.

XML Configuration

Below is the XML configuration for this operation:

<ms-vectors:embedding-generate-from-binary
  doc:name="[Embedding] Generate from binary"
  doc:id="3d4a32b7-dd47-465a-bd93-0c4cb0af84c2"
  config-ref="Embedding_Config_Vertex_AI"
  embeddingModelName="multimodalembedding" >
    <ms-vectors:binary ><![CDATA[#[payload.parts.image.content]]]></ms-vectors:binary>
    <ms-vectors:label ><![CDATA[#[payload.parts.question.content]]]></ms-vectors:label>
</ms-vectors:embedding-generate-from-binary>

Output Fields

Payload

This operation responds with a json payload.

Example

Here an example of the JSON output.

{
    "embeddings": [
      [-0.00683132, -0.0033572172, 0.02698761, -0.01291587, ...],
      [-0.0047172513, -0.03481483, 0.02046227, -0.037395656, ...],
      ...
    ]
    "text-segments": [
        {
            "metadata": {
                "index": 0
            },
            "text": "<The provided media label>"
        }
    ],
    "dimension": 1408
}

embeddings: The list of generated embeddings
- list-item (embedding)
text-segments: The list of segments.
- list-item (text-segment):
  - text: The media label.
  - metadata: The metadata key-value pairs.
    - index: The segment/chunk number for the uploaded data source.
dimension: The dimension of the selected embedding model.

Attributes

EmbeddingResponseAttributes:
- embeddingModelDimension: The dimension for the embedding model used.
- embeddingModelName: The embedding model name used.
- tokenUsage: The token usage for the embedding model used.
  - inputCount: The number of tokens used as input.
  - outputCount: The number of tokens used as output.
  - totalCount: The total number of tokens used.

Transform Store