Generate Sitemap
This operation allows you to create a sitemap of a website to aid in SEO and site structure analysis. The results of this operation can be used to customize the way you want to crawl the website (eg specific pages)
Input Fields
Operation Fields
- Website URL: The website to generate a sitemap for.
- Maximum Depth : The generate sitemap's depth will be limited to this specified depth.
- Delay (millisecs) : To prevent websites from being overloaded, you can add a delay to your crawl. This delay is the time delay between crawling pages on a website. Specify 0 for no delay.
XML Configuration
Below is the XML configuration for this operation:
<mac-web-crawler:generate-sitemap doc:name="Generate sitemap"
doc:id="6410aae9-21d4-4005-8d86-ac9a136657b4"
url="#[payload.url]"
maxDepth="#[payload.maxDepth]"
delayMillis="#[payload.delay]"/>Output Field
This operation responds with a json payload.
Example Output
{
"url": "https://mac-project.ai/",
"children": [
{
"url": "https://www.mac-project.ai/",
"children": []
},
{
"url": "https://mac-project.ai/docs",
"children": []
},
{
"url": "https://mac-project.ai/",
"children": []
},
{
"url": "https://mac-project.ai/docs/mulechain-ai/getting-started",
"children": []
},
{
"url": "https://mac-project.ai/docs/contribute",
"children": []
},
{
"url": "https://mac-project.ai/docs/mulechain-ai/supported-operations",
"children": []
},
{
"url": "https://mac-project.ai/about",
"children": []
}
]
}