I turned Archon to it can crawl any sitemap, this supports if i want to add more documents. We should name document, so we can check what document do we have.
A stop crawling button if crawling fail too many time.
Captured failed URL so we can recrawl those failed url.
Show document list + chunk size to let us know what we have
hope these should be in V7 something.
Hey, this a great idea! There is a current PR out there that might cover what you are looking for. PR-94 I would like to hear your thoughts if you are able to review.
Here is a screen shot of what it looks like:
I would like to add a more automatic documentation finder so you wouldn’t need the sitemap.xml for every source but I don’t know if I’ll have the time before Cole beats me to it.
I have some other plans in the immediate term for Archon so you will beat me to it! I would absolutely love it if you made a PR for your idea Mike!
Sure thing. I should have some time toward the end of the week to put something together; I’ll post an update here once it’s ready.
Thanks for sharing this, @cnguyen1411—it’s really helpful to see which features other folks are interested in.
