I cannot get the 5 failed out of 68 URLs and no data being piped to Supabase.com table. I am not sure what is occurring, I have been through the setup several times using both docker and Python via VS Code. Could use assistance if you are able. 4.4.25 - I ran again and got 1 failed URL.
Update: After debugging SQL script in supabase, I found the following will correct the run error, "-- Enable the pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;
â Create the site_pages table if it doesnât exist
CREATE TABLE IF NOT EXISTS site_pages (
id bigserial PRIMARY KEY,
url varchar NOT NULL,
chunk_number integer NOT NULL,
title varchar NOT NULL,
summary varchar NOT NULL,
content text NOT NULL,
metadata jsonb NOT NULL DEFAULT â{}â::jsonb,
embedding vector(768),
created_at timestamp WITH TIME ZONE DEFAULT timezone(âutcâ::text, now()) NOT NULL,
UNIQUE(url, chunk_number)
);
â Create an index for vector similarity search if it doesnât exist
CREATE INDEX IF NOT EXISTS site_pages_embedding_idx
ON site_pages USING ivfflat (embedding vector_cosine_ops);
â Create an index on metadata for faster filtering if it doesnât exist
CREATE INDEX IF NOT EXISTS idx_site_pages_metadata
ON site_pages USING gin (metadata);
â Create or replace the function to search for documentation chunks
CREATE OR REPLACE FUNCTION match_site_pages (
query_embedding vector(768),
match_count int DEFAULT 10,
filter jsonb DEFAULT â{}â::jsonb
) RETURNS TABLE (
id bigint,
url varchar,
chunk_number integer,
title varchar,
summary varchar,
content text,
metadata jsonb,
similarity float
)
LANGUAGE plpgsql
AS $$
#variable_conflict use_column
BEGIN
RETURN QUERY
SELECT
id,
url,
chunk_number,
title,
summary,
content,
metadata,
1 - (site_pages.embedding <=> query_embedding) AS similarity
FROM site_pages
WHERE metadata @> filter
ORDER BY site_pages.embedding <=> query_embedding
LIMIT match_count;
END;
$$;
â Enable RLS on the table (this is safe to run multiple times)
ALTER TABLE site_pages ENABLE ROW LEVEL SECURITY;
â Create a policy for public read access (replaces if it exists)
CREATE POLICY âAllow public read accessâ
ON site_pages
FOR SELECT
TO public
USING (true);"
This adds the following: âCREATE OR REPLACE FUNCTION match_site_pages (â. This is when the âsite_pagesâ error occurs.
After crawling each page and extracting content, it had the same 5 failed URLs and data not being saved to supabase.
Glad that is working now! The randomly failed chunks is either Crawl4AI having a small blip or an OpenAI rate limit issue. You could always scroll through the logs and see!
What did you end up doing to fix it completely?
To get where I am at (which is progress) I had to mirror the Environmental Variables I saw someone else put in. Unfortunately, using my own installed (locally ran) Ollama, I was not getting any âchunksâ into Supabase. I am still failing one URL each time I retry it. I do see rate limits being a limitation - good suggestion. This is the log that seems to repeat consistently: â[23:52:36] Error processing https://ai.pydantic.dev/contributing/: Error fetching https://ai.pydantic.dev/contributing/: we should not get here!â.
We should be able to integrate Grok. I am a Super-Groker and it is just wisdom to attach oneâs self to something that is truth-seeking.
The variance in different results leads me to believe it is a LLM error associated with rate limits or connection problems. Connection problems would not be an issue with me.
Attached are the logs I have based on 1 URL failure.
I am not fixed yet 
This is my Desktop Docker log:
Archon Crawling process.json (3.2 MB)
I looked up the error code - you were right. I exceeded the rate limit: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
2025-04-06 17:54:59 Error getting embedding: Error code: 429 - {âerrorâ: {âmessageâ: âYou exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.â, âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
2025-04-06 17:54:59 Error getting embedding: Error code: 429 - {âerrorâ: {âmessageâ: âYou exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.â, âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
2025-04-06 17:54:59 Error getting embedding: Error code: 429 - {âerrorâ: {âmessageâ: âYou exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.â, âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
2025-04-06 17:55:00 Error getting embedding: Error code: 429 - {âerrorâ: {âmessageâ: âYou exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.â, âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
2025-04-06 17:55:00 Error getting embedding: Error code: 429 - {âerrorâ: {âmessageâ: âYou exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.â, âtypeâ: âinsufficient_quotaâ, âparamâ: None, âcodeâ: âinsufficient_quotaâ}}
I think at this point, if there was a way to integrate an LLM like Grok where I donât have rate limits, it might work for me.
You can integrate any OpenAI compatible API so you could use Groq! Or use another provider like OpenRouter
Okay this is really good to know, thanks for pointing that out!
