Storage Operations
Supported Storage Options
- Local: Allows to load data from application local storage
- Azure Blob Storage (opens in a new tab): Allows to load data from Azure Blob Storage
- Amazon S3 (opens in a new tab): Allows to load data from Amazon S3 Buckets
- Google Cloud Storage (opens in a new tab): Allows to load data from Google Cloud Storage
Storage operations should be used to load a single file or a list of files. The file is at first parsed and then optionally split into chucks of the provided size.
These operations are usually followed by [Transform] Parse documentoperation in case of document file(s) or [Transform] Process media and [Embedding] generate embedding from media operation in case of media file(s).
Storage | Load file
The [Storage] Load file operation parse a document and optionally splits it into text chunks based on the provided size.
How to Use
Add Document to Store
The [Storage] Load file operation can be followed by [Transform] Parse document and/or [Transform] Chunk text.
Add Media to Store
The [Storage] Load file operation can be followed by [Transform] Process media and [Embedding] Generate from media.
Input Fields
Module Configuration
This refers to the MuleSoft Vectors Storage Configuration set up in the Getting Started section.
File Fields
- Context Path: Behaviour changes based on storage type.
- Local: Contains the path for the documents to be ingested into the embedding store. Ensure the file path is accessible. You can also use a DataWeave expression for this field, e.g.,
mule.home ++ "/apps/" ++ app.name ++ "/". - AZURE_BLOB: Contains container name and blob item name in the following format
<container-name>/<blob-item-name>(eg. ms-vectors-container/invoicesample.pdf, ms-vectors-container/folder/invoicesample.pdf, ...) - S3: Contains AWS S3 Bucket and AWS S3 Object Key in the following format
s3://<s3-bucket>/<s3-object-key>(eg. s3://ms-vectors-bucket/setup.adoc, s3://ms-vectors-bucket/folder/setup.adoc,...)
- Local: Contains the path for the documents to be ingested into the embedding store. Ensure the file path is accessible. You can also use a DataWeave expression for this field, e.g.,
XML Configuration
Below is the XML configuration for this operation:
<ms-vectors:storage-load-file
doc:name="[Storage] Load file"
doc:id="225cee0d-94ef-454f-a0d6-ee76db2ad77c"
config-ref="Local_Storage_config"
contextPath="#[attributes.queryParams.contextPath]"/>Output Fields
Payload
This operation responds with a Binary payload.
Attributes
- StorageResponseAttributes:
- fileName: The name of the file, where the text segment was found.
- metadata: The metadata key-value pairs.
- mimeType: The media mime type.
- otherAttributes: Additional attribute key-value pairs.
- path: The full path to the file.
Storage | Load file list
The [Storage] Load file list operation parse a list of documents and optionally splits them into text chunks based on the provided size.
How to Use
Add Document Folder to Store
The [Storage] Load file list operation can be followed by a Batch Job, For Each or For Each Parallel including
[Transform] Parse document and/or [Transform] Chunk text operations.
Add Media Folder to Store
The [Storage] Load file list operation can be followed by a Batch Job, For Each or For Each Parallel including
[Transform] Process media and [Embedding] Generate from media operations.
Input Fields
Module Configuration
This refers to the MuleSoft Vectors Storage Configuration set up in the Getting Started section.
File Fields
- Context Path: Behaviour changes based on storage type.
- Local: Contains the path for the documents to be ingested into the embedding store. Ensure the file path is accessible. You can also use a DataWeave expression for this field, e.g.,
mule.home ++ "/apps/" ++ app.name ++ "/". - AZURE_BLOB: Contains container name and blob item name in the following format
<container-name>/<blob-item-name>(eg. ms-vectors-container/invoicesample.pdf, ms-vectors-container/folder/invoicesample.pdf, ...) - S3: Contains AWS S3 Bucket and AWS S3 Object Key in the following format
s3://<s3-bucket>/<s3-object-key>(eg. s3://ms-vectors-bucket/setup.adoc, s3://ms-vectors-bucket/folder/setup.adoc,...)
- Local: Contains the path for the documents to be ingested into the embedding store. Ensure the file path is accessible. You can also use a DataWeave expression for this field, e.g.,
XML Configuration
Below is the XML configuration for this operation:
<ms-vectors:storage-load-file-list
doc:name="[Storage] Load file list"
doc:id="d4513319-57ed-48ba-9c66-abfd511f0517"
config-ref="Local_Storage_config"
contextPath="#[attributes.queryParams.contextPath]"/>Output Fields
Payload
This operation responds with a Binary payload.
Attributes
- StorageResponseAttributes:
- fileName: The name of the file, where the text segment was found.
- metadata: The metadata key-value pairs.
- mimeType: The media mime type.
- otherAttributes: Additional attribute key-value pairs.
- path: The full path to the file.