❓ [Help] "Task Not Found" Error with crawl4ai in n8n Automation

n8n
1

Hi everyone,

I’m setting up a web scraping automation using the crawl4ai Docker container, which runs locally. Most of it works fine, but occasionally I run into this error:

“The resource you are requesting could not be found – Task not found”

After some digging, I found a comment under one of Cole’s videos that explained the issue might be due to how crawl4ai handles task IDs:

“When you post a URL and receive a task ID as a response, the task ID disappears instantaneously without a database to store it. So when you make a GET request using that task ID, you’ll get an error because the record no longer exists.”

This makes sense, but I’m wondering:

  • Is there a way to prevent this from happening?
  • Has anyone implemented a workaround or persistent storage for the task data?
  • Is there any official documentation on this behavior? I checked the crawl4ai docs but couldn’t find anything.

Any help, pointers, or even unofficial docs would be hugely appreciated. Thanks in advance!

2

The task ID shouldn’t disappear instantly! You should be able to use it to get the full response in a subsequent request. Or am I understanding what you are saying wrong?

1 Like
3

Thanks for the reply Cole!

You’re right, the task ID doesn’t disappear immediately. After some testing, I noticed that the task data is accessible for about 2–3 minutes before it gets removed.

I was wondering:

  • Is there any built-in way in crawl4ai to persist the task ID and its associated data without having to re-crawl the URL?
  • If not, I’m thinking of using the Postgres node in n8n to store the task results in a custom table for future reference.

Before going that route, I just wanted to check if there’s a more native solution in crawl4ai. Would love to hear any suggestions!

4

I don’t think there is so you’ll have to store the task ID in Postgres like you are thinking! Problem is though I think the task ID is pretty useless after a while because the data you can retrieve with the task ID will also go away. Wouldn’t it make sense to use the task ID to retrieve the crawled data and then store that in a DB?