Hardware requirements:

• RAM: As much as the AI model requires. Most models have a variant that works well on 8 GB RAM
• GPU: GPU is recommended but not required. It also runs in CPU-only mode but will be slower on Linux, Windows, and Mac-Intel. On M1/M2/M3 Macs, the inference speed is really good.

(For some reason, my response to original comment isn’t showing up so reposting here)

How does it get it’s training data? Would this work offline?

Nicely done! What are options like for AMD GPUs? Any future plans to support?

Yes. This is the perfect excuse for me to buy a new GPU and put the old one in the server. 3080’s are old and crappy now anyway …right? (Yes that was sarcasm btw).

Does this make sense on a home server? The response time will not take several dozen seconds and the response quality will not be worse than chatgpt? I’m currently using openai api and it’s like lvl 0 for me. So is your project better or worse?

I can’t get it running with my GPU.

I get this error:

parsing /root/secure-ai-tools/docker-compose.yml: yaml: line 19: did not find expected key

This is my .yaml:

services:

web: image: public.ecr.aws/d8f2p0h3/secure-ai-tools:latest platform: linux/amd64 volumes: - ./web:/app/volume env_file: - .env environment: - INFERENCE_SERVER=http://inference:11434/ ports: - 28669:28669 command: sh -c “cd /app && sh tools/db-migrate-and-seed.sh ${DATABASE_FILE} && node server.js” depends_on: - inference

inference: image: ollama/ollama:latest volumes: - ./inference:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: ‘all’ capabilities: [gpu]

I know what iam doing this weekend. Try to run it on my rpi4.

What happens if you run something like this on a no GPU VPS, something like a racknerd or oracle free cloud box with 3vCPU and 3GB ram? Does it work? Is it so slow as to be useless?

This looks awesome! My little project was missing just that! Https://github.com/rogueghost93/fly-hi I’ll add it these days!

Any chance this will end up on Unraid community apps?

I’m going to be honest, I’m sick and tired of repackaged, industry standard software that is just an nginx reverse proxy and underpowered authentication system.

Self hosting is already easy. SSL is easy. LDAP, and SSO are easy. If people actually wanted to help they’d make tutorials instead of opinionated branded tools that aren’t as flexible.

Just my two cents

Hey Jay - so here’s a use case for you. I run a medium sized physical therapy clinic and my pie in the sky dream is to have a local AI that doesn’t connect to any outside services (for HIPAA reasons) but is capable of providing a conversational AI for my new staff so that they can ask it questions about policies and procedures in the clinic. Essentially an AI that can be a trainer when the trainer isn’t available; can answer questions like “how do I complete authorization for this patient who has this insurance” and it walks you through the steps, etc.

Do you think that’s doable with this tool? I’ve been trying to set up Danswer for this but it’s extremely resource heavy and I haven’t been able to budget a decent PC / server to run the Docker services for it.