Very meta question, crawl ottomator?

I am going non-linear on crawling sites and documents to help me learn and code. I am going to submit my tool stack to the hackathon, and realized all the info and guides on preparing the submittal (i.e. api requirements, etc) is in the automator website. What I am now looking for is either a sitemap or git repo (Thanks Cole for showing us how to crawl both of those) to cleanly pull in that info for my coding assistant.

1 Like

Haha I love this! The Live Agent Studio doesn’t have a sitemap currently, but you can still use Crawl4AI to visit studio.ottomator.ai, grab all links from the home page, and then recursively scrape those.

See: Link & Media - Crawl4AI Documentation (v0.4.3bx)

Oddly enough it appears crawl4ai does not have one either, but I am checking. Really pushing here to submit my tool by the deadline. Beware scope creep!

Thanks!

1 Like

Their documentation page does have a sitemap!

https://docs.crawl4ai.com/sitemap.xml