Multi GPU support

Archon
1 
cmd = [
    "docker", "run", "-d",
    "--name", "archon-container",
    "--gpus", "all",  # Pass GPU access
    "-e NVIDIA_VISIBLE_DEVICES=all",
    "-e NVIDIA_DRIVER_CAPABILITIES=compute,utility",
    "-e NVIDIA_REQUIRE_CUDA=\"cuda>=12.8\"",  # Force CUDA 12.8 for ollama consistency
    "-p", "8501:8501",
    "-p", "8100:8100",
    "--add-host", "host.docker.internal:host-gateway"
]
2

Hi @johnmag2020,

can you add some more description to it, why this is needed/makes sense. I think not everyone knows why this is relevant, including me :slight_smile:

1 Like
3

This is only relevant if you self host ollama LLM’s and have multiple nvidia GPU’s.

4

So far so clear for me, but why I need this then set up for the archon container. Isnt it enough if the ollama container has the GPU configuration, cause the LLMs running in there?

5

The run_docker.py script creates the container and I was having conflicts for whatever reason with cuda versions after the initial setup…it kept trying to use 12.0 but ollama will not operate less than 12.8 to the best of my knowledge. Maybe relevant only to my setup but it did alter my cuda config, perhaps through requirements or a bug/mismatch. Here is my startup log snippet as well and it goes through now:

=== Starting Archon container ===
Running: docker run -d --name archon-container --network localai_default --gpus all -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPAB
ILITIES=compute,utility -e NVIDIA_REQUIRE_CUDA=“cuda>=12.8” -p 8501:8501 -p 8100:8100 --add-host host.docker.internal:host-gateway --env
-file /opt/localai/archon/.env archon:latest
291eb469a69c359449c2f1e39221d2c6bc03884a010526cd21444c979c21d62c

As a dev member maybe you can shed some light on the database connection as I am also experiencing a disconnect from the db as well…tables created/show up fine but on the Database tab I have a "Error checking table status: “Server disconnected without sending a response.” I verified a few things by throwing a random ip and key and if it’s not right Archon admirably responds with a no-start so I have to think it’s getting there with ip and security key matching. I see no unreasonable activity in the db log or Archon. This whole supabase-local has certainly been a challenge but it seems to be pretty stable now after I chanced onto Coles scripted-start setup which is super effective, though the analytics is still only partially operating, I use Portainer to take up the slack for logging and pgAdmin container which effectively did all the tables. Anyway if you have any insight about the database connection that would be awesome, my setup is not persistent yet as I pull the supabase-db address which changes with a restart but getting it working is the current objective…Cheers and again this is brilliant work!

6

Thanks for the explaining. I also just startet using Archon and try it out. I am just in the bolt.diy dev team at the moment, but this community is about both, so all bolt.diy moderators are also moderators for archon :slight_smile:

Just published a first video on how to install Archon and within some next steps I wanted to testout local supabase as well. Also had a quick talk already with cole, cause I missed this in the examples as well.

7

I have to say I have pursued the local supabase out of sheer belligerence because I just couldn’t get it to all connect and function and simply am not smart enough to quit and do it online…dog/bone syndrome. I would get a container debugged and another would start getting cranky and round and round I went for more evenings than I want to admit and that’s when I figured there had to be a better way and came across the updated local-ai-packaged which is what I had based my assault on this on and again it did not disappoint, in retrospect timing I think is critical and I never could figure what connected first, last and 23rd . My local-ai docker stack is up to 24 containers and is so helpful though. I think must have additions are Portainer, pgAdmin, Playwright and coqui is a fun one to play around with.
Thanks for the setup video and I will give it a thorough going over. Again thoughts on persistent db addressing would be great. Be well

8

ohhhh Nice…a task list!

1 Like
9

@johnmag2020 Feel free to make a PR for this if you want! Multi GPU support is definitely important. I’m thinking about adding it in soon myself too but I always love contributions from others like you :slight_smile:

10

I think many people like myself are looking at this type of stuff as a real time saver. At 60 and having 17 RE LLC’s for commerical and STR properties ALL with separate accounts, P&L’s, credit cards and etc…it’s a horror in certain respects and an absolute necessity in others but to the point I don’t want to have it anywhere but in MY office. With that in mind and having done this brain fart for a little over 2 months on and off (a lot of on), but investing significant time and a pair of 6000’s ($$ but I like gadgets and have Autodesk requirements), I would strongly advise you to explore the business case and model to actually create a complete Plug and Play stack ordinary business people can integrate and explore the possibilities of creative agent creation for their specific business models. I have a close friend with 120 or so employees in plumbing and he could buy my in-law doctor about 20 times over and get far greater benefits, but in-law has AI transcription and billing services and plumber has 4 computers from a decade ago. Plumber, like me has no interest in his data out in a “Cloud”, these days especially. Food for thought…he can walk up to his Star Trek terminal and say “Computer, I need an agent to tell me exactly how many millions I spent at FW Webb last month on copper fittings and keep track as tariffs cause the costs to sky rocket and I can make damn sure I am passing every nickle on to my customers and not absorbing it on my end -and- send me an email/text every time the cost of copper goes up on the merc market as a precursor”.
I am not a fan but have a sibling that for over 5 years sits about 10-15 feet from the Zuck when he’s in the office, his influence I fully understand as at his level of influence his prediction of billions of agents to me is more of a “think about, bring about” . Use case in my mind is agents in my back pocket like a cell phone connected to the internet. :+1:

1 Like