Very cool to see a serious local first effort. Looking back at how far local models have come I definitely believe their usefulness combined with RAG or in domain specific contexts is soon to be (or already is) on par with general purpose gpt5-like massive parameter cloud models. The ability to generate quality responses without having to relinquish private data to the cloud used to be a pipedream. It's exciting to see a team dedicated to making this a reality.
- AI assistants for smaller practices without enterprise EHR. Epic at the moment integrates 3rd party AI assistants, but those are of course cloud services and are aimed at contracts with large hospital systems. They're a great step forward, but leave much to be desired by doctors in actual usefulness.
- Consumer/patient facing products to help people synthesize all of their health information and understand what their healthcare providers are doing. Think of a n on device assistant that can connect with something like https://www.fastenhealth.com/ to make local RAG of their health history.
Overall, users can feel more confident they know where their PHI is, and potentially easier for smaller companies/start-ups to get into the healthcare space without having to move/store people's PHI.
I am not sure if it is the future, but I am glad there is some movement to hinder centralization in this sector as much as possible ( yes, I recognize future risk, but for now it counts as hindering it ).
100%. We don't know what's going to happen in the future. Things are evolving so quickly. Hopefully pushing back on centralization now will keep the ecosystem healthier and give developers real options outside the big two/three cloud providers.
I am kind of militant about this. The ability to run great AI models locally is critical, not just for this sector, but for innovation overall. The bar of "build a few datacenters" is far too high for all but the largest countries and companies in the world.
Yep! We were working on an authentication startup (https://news.ycombinator.com/item?id=30615352) and built it to $1.5M in ARR, but then we saw even a bigger pain point; local AI is hard. When we tried building a corporate knowledge base with RAG and local models, we hit the same wall: a painful gap between prototype and production.
Production-ready enterprise AI requires solving model management, RAG pipelines, model fine-tuning, prompt engineering, failover, cost optimization, and deployment orchestration. You can’t just be good at one or two of these, you have to be great at all of them or your project won't succeed. And so Llamafarm was born!
Monetization-wise - We’re open source and free forever, with revenue coming from enterprise support, managed deployments, and compliance packages—basically, companies pay for confidence, not code.
So this sounds like an application layer approach, maybe just shy of a replit or base44, with the twist that you can own the pipeline. While there's something to that, I think there are some further questions around differentiation that need to be answered. I think the biggest challenge is going to be the beachead: what client demographic has the cash to want to own the pipeline and not use SaaS, but doesn't have the staff on hand to do it?
I think that enterprises and small businesses alike need stuff like this, regardless of whether they're software companies or some other vertical like healthcare or legal. I worked at IBM for over a decade and it was always preferable to start with an open source framework if it fit your problem space, especially for internal stuff. We shipped products with components built on Elastic, Drupal, Express, etc.
You could make the same argument for Kubernetes. If you have the cash and the team, why not build it yourself? Most don't have the expertise or the time to find/train the people who do.
People want AI that works out of the box on day one. Not day 100.
Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience.
Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normally have deep pockets).
Fair question. The core will always stay open source and free. We’ll monetize around it with things like managed hosting, enterprise support, and compliance options (HIPAA, SOC2, etc). Basically, we make money when teams want someone to stand behind it in production, not for using the software itself. But let us know if you have other ideas! We're still new to open source
I'm a contributor on this project and am very excited to hear your feedback. We really hope that this will become a helpful tool for building AI projects that you own and run yourself!
Yea of course. I was trying to click the docs link from the homepage on llamafarm.dev from two different networks on two different browsers: edge and brave. Neither worked. Phone didn’t either. It takes me to a supabase link that errors out. Hope that helps! Thanks for the link! (Btw I don’t see any errors in the browsers console)
Very cool. I jumped in here thinking it was gonna be something else though: a packaged service for distributing on-prem model running across multiple GPUs.
I'm basically imagining a vast.ai type deployment of an on-prem GPT; assuming that most infra is consumer GPUs on consumer devices, the idea of running the "company cluster" as combined compute of the company's machines
The hardest part, honestly, is the runtime. How do we make it super easy actually to deploy this. We are still working on that. Where do you see a few good places to focus at first? I was thinking AWS and Google, since both have good GPU pricing models, but I am probably missing a few good ones!
but wait, why should I do this for my first home grown orchestration instead of something else? Like, if I want to set up a local LLM running on my old laptop for some kind of RAG on all my hard drives why is this best? Or if I want agentic monitoring of alarms instead of paying for simplisafe or ring or whatever.
Right...there are lots of ways you could do that. Most of the ways we've seen enabling that sort of thing tend to be programmatic in nature. That's great for some people, but you have to deal with shifting dependencies, sorting out bugs, making sure everything connects properly, etc. Some people will want that for sure, because you do get control over every little piece.
LlamaFarm provides an abstraction over most (eventually all) of those pieces. Something that should work out of the box wherever you deploy it but with various knobs to customize as needed (we're working on an agent to help you with this as well).
In your example (alarm monitoring), I think right now you'd still need to write the agent, but you could use LlamaFarm to deploy an LLM that relied on increasingly accurate examples in RAG and very easily adjust your system prompt.
Good question — that’s actually the sweet spot for LlamaFarm.
You can wire things together yourself (LangChain, bash, Ollama, etc.), but LlamaFarm tries to make that repeatable and portable. It’s declarative orchestration for AI systems — you describe what you want (models, RAG, agents, vector DBs) in YAML, and it runs the same way anywhere: laptop, cloud, or fully air-gapped edge.
So instead of gluing frameworks and breaking them every update, you can do something like:
…and it just runs — same config, same behavior, whether you’re doing local RAG or home monitoring. The goal isn’t to replace the DIY route, just to make it composable and reproducible.
Hey thanks! I'm Rachel from LlamaFarm; we actually use LlamaIndex as one of our components. It's great for RAG, and we didn't want to reinvent what they've already done. LlamaFarm is about bundling the best of open source into a complete, production-ready AI project framework. Think of us like the integration and orchestration layer that makes LlamaIndex, plus model management, plus prompt engineering, plus deployment tools all work together seamlessly.
Where LlamaIndex gives you powerful RAG primitives, we give you the full production system - the model failover when OpenAI is down, the strategy system that adapts from development to production, the deployment configs for Kubernetes. We handle all the boring stuff that turns a RAG prototype into a system that actually runs in production. One YAML config, one CLI command, and you have everything from local development to cloud deployment. :)
This is super interesting! I'm the founder of Muna (https://docs.muna.ai) with much of the same underlying philosophy, but a different approach:
We're building a general purpose compiler for Python. Once compiled, developers can deploy across Android, iOS, Linux, macOS, Web (wasm), and Windows in as little as two lines of code.
Oh! Muna looks cool as well! I've just barely glanced at your docs page so far, but I'm definitely going to explore further. One of the biggest issues in the back of our minds is getting models running on a variety of hardware and platforms. Right now, we're just using Ollama with support for Lemonade coming soon. But both of these will likely require some manual setup before deploying LlamaFarm.
We should collab! We prefer to be the underlying infrastructure behind the scenes, and have a pretty holistic approach towards hardware coverage and performance optimization.
Very cool to see a serious local first effort. Looking back at how far local models have come I definitely believe their usefulness combined with RAG or in domain specific contexts is soon to be (or already is) on par with general purpose gpt5-like massive parameter cloud models. The ability to generate quality responses without having to relinquish private data to the cloud used to be a pipedream. It's exciting to see a team dedicated to making this a reality.
What are a few use-cases you want to see this used for?
Few ideas related to healthcare
- AI assistants for smaller practices without enterprise EHR. Epic at the moment integrates 3rd party AI assistants, but those are of course cloud services and are aimed at contracts with large hospital systems. They're a great step forward, but leave much to be desired by doctors in actual usefulness.
- Consumer/patient facing products to help people synthesize all of their health information and understand what their healthcare providers are doing. Think of a n on device assistant that can connect with something like https://www.fastenhealth.com/ to make local RAG of their health history.
Overall, users can feel more confident they know where their PHI is, and potentially easier for smaller companies/start-ups to get into the healthcare space without having to move/store people's PHI.
Cool ideas, thank you.
Thanks! It means a lot to hear you say that.
I am not sure if it is the future, but I am glad there is some movement to hinder centralization in this sector as much as possible ( yes, I recognize future risk, but for now it counts as hindering it ).
100%. We don't know what's going to happen in the future. Things are evolving so quickly. Hopefully pushing back on centralization now will keep the ecosystem healthier and give developers real options outside the big two/three cloud providers.
I am kind of militant about this. The ability to run great AI models locally is critical, not just for this sector, but for innovation overall. The bar of "build a few datacenters" is far too high for all but the largest countries and companies in the world.
Congrats on the launch. YC 2022? I'm assuming this was a pivot - what lead to it and how do you guys plan on making money long term?
Yep! We were working on an authentication startup (https://news.ycombinator.com/item?id=30615352) and built it to $1.5M in ARR, but then we saw even a bigger pain point; local AI is hard. When we tried building a corporate knowledge base with RAG and local models, we hit the same wall: a painful gap between prototype and production.
Production-ready enterprise AI requires solving model management, RAG pipelines, model fine-tuning, prompt engineering, failover, cost optimization, and deployment orchestration. You can’t just be good at one or two of these, you have to be great at all of them or your project won't succeed. And so Llamafarm was born!
Monetization-wise - We’re open source and free forever, with revenue coming from enterprise support, managed deployments, and compliance packages—basically, companies pay for confidence, not code.
So this sounds like an application layer approach, maybe just shy of a replit or base44, with the twist that you can own the pipeline. While there's something to that, I think there are some further questions around differentiation that need to be answered. I think the biggest challenge is going to be the beachead: what client demographic has the cash to want to own the pipeline and not use SaaS, but doesn't have the staff on hand to do it?
I think that enterprises and small businesses alike need stuff like this, regardless of whether they're software companies or some other vertical like healthcare or legal. I worked at IBM for over a decade and it was always preferable to start with an open source framework if it fit your problem space, especially for internal stuff. We shipped products with components built on Elastic, Drupal, Express, etc.
You could make the same argument for Kubernetes. If you have the cash and the team, why not build it yourself? Most don't have the expertise or the time to find/train the people who do.
People want AI that works out of the box on day one. Not day 100.
Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience.
Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normally have deep pockets).
What do you think is a good starting place?
Open source but backed by venture capital, so what is your monetization strategy?
Fair question. The core will always stay open source and free. We’ll monetize around it with things like managed hosting, enterprise support, and compliance options (HIPAA, SOC2, etc). Basically, we make money when teams want someone to stand behind it in production, not for using the software itself. But let us know if you have other ideas! We're still new to open source
I'm a contributor on this project and am very excited to hear your feedback. We really hope that this will become a helpful tool for building AI projects that you own and run yourself!
Likewise, it's been fun building in the open on this one. You can download the CLI with just: curl -fsSL https://raw.githubusercontent.com/llama-farm/llamafarm/main/... | bash
Then do LF init to get the project started!
I love the ethos of the project! I think your docs link might be broken, however. Looking forward to checking this out!
Hey thanks! Sorry about the broken link - here's a better docs link for now https://docs.llamafarm.dev/docs/intro mind sharing where it's broken?
Yea of course. I was trying to click the docs link from the homepage on llamafarm.dev from two different networks on two different browsers: edge and brave. Neither worked. Phone didn’t either. It takes me to a supabase link that errors out. Hope that helps! Thanks for the link! (Btw I don’t see any errors in the browsers console)
thank you! found and fixed 2 on the website - appreciate the comment and detailed testing
Just found and fixed a bad link on the bottom of the website - thanks again for pointing that out !
Very cool. I jumped in here thinking it was gonna be something else though: a packaged service for distributing on-prem model running across multiple GPUs.
I'm basically imagining a vast.ai type deployment of an on-prem GPT; assuming that most infra is consumer GPUs on consumer devices, the idea of running the "company cluster" as combined compute of the company's machines
Great point. I can see how you'd land there. Also a great idea! xD
Maybe a better descriptor is "self-sovereign AI?" "Self-hosted AI?"
We're building something closer to this at Muna: https://docs.muna.ai . Check us out and let me know what you think!
https://news.ycombinator.com/item?id=43119777
Hope you are right and we decentralize AI properly...
The hardest part, honestly, is the runtime. How do we make it super easy actually to deploy this. We are still working on that. Where do you see a few good places to focus at first? I was thinking AWS and Google, since both have good GPU pricing models, but I am probably missing a few good ones!
but wait, why should I do this for my first home grown orchestration instead of something else? Like, if I want to set up a local LLM running on my old laptop for some kind of RAG on all my hard drives why is this best? Or if I want agentic monitoring of alarms instead of paying for simplisafe or ring or whatever.
Right...there are lots of ways you could do that. Most of the ways we've seen enabling that sort of thing tend to be programmatic in nature. That's great for some people, but you have to deal with shifting dependencies, sorting out bugs, making sure everything connects properly, etc. Some people will want that for sure, because you do get control over every little piece.
LlamaFarm provides an abstraction over most (eventually all) of those pieces. Something that should work out of the box wherever you deploy it but with various knobs to customize as needed (we're working on an agent to help you with this as well).
In your example (alarm monitoring), I think right now you'd still need to write the agent, but you could use LlamaFarm to deploy an LLM that relied on increasingly accurate examples in RAG and very easily adjust your system prompt.
Good question — that’s actually the sweet spot for LlamaFarm.
You can wire things together yourself (LangChain, bash, Ollama, etc.), but LlamaFarm tries to make that repeatable and portable. It’s declarative orchestration for AI systems — you describe what you want (models, RAG, agents, vector DBs) in YAML, and it runs the same way anywhere: laptop, cloud, or fully air-gapped edge.
So instead of gluing frameworks and breaking them every update, you can do something like:
name: home_guarde runtimes: - detect_motion: {model: "phi-3", provider: "lemonade"} - alert: {model: "gpt-5", fallback: "llama3:8b"} rag: embedder: "nomic-embed-text" database: chromaDB
…and it just runs — same config, same behavior, whether you’re doing local RAG or home monitoring. The goal isn’t to replace the DIY route, just to make it composable and reproducible.
Looks great! Congrats on the launch. How is this different than llamaindex?
Hey thanks! I'm Rachel from LlamaFarm; we actually use LlamaIndex as one of our components. It's great for RAG, and we didn't want to reinvent what they've already done. LlamaFarm is about bundling the best of open source into a complete, production-ready AI project framework. Think of us like the integration and orchestration layer that makes LlamaIndex, plus model management, plus prompt engineering, plus deployment tools all work together seamlessly.
Where LlamaIndex gives you powerful RAG primitives, we give you the full production system - the model failover when OpenAI is down, the strategy system that adapts from development to production, the deployment configs for Kubernetes. We handle all the boring stuff that turns a RAG prototype into a system that actually runs in production. One YAML config, one CLI command, and you have everything from local development to cloud deployment. :)
This is super interesting! I'm the founder of Muna (https://docs.muna.ai) with much of the same underlying philosophy, but a different approach:
We're building a general purpose compiler for Python. Once compiled, developers can deploy across Android, iOS, Linux, macOS, Web (wasm), and Windows in as little as two lines of code.
Congrats on the launch!
Oh! Muna looks cool as well! I've just barely glanced at your docs page so far, but I'm definitely going to explore further. One of the biggest issues in the back of our minds is getting models running on a variety of hardware and platforms. Right now, we're just using Ollama with support for Lemonade coming soon. But both of these will likely require some manual setup before deploying LlamaFarm.
We should collab! We prefer to be the underlying infrastructure behind the scenes, and have a pretty holistic approach towards hardware coverage and performance optimization.
Read more:
- https://blog.codingconfessions.com/p/compiling-python-to-run... - https://docs.muna.ai/predictors/ai#inference-backends