Hey guys, I had a few esp32 boards around for a while and always thought - how do we go beyond just one device, would it be possible to apply cloud-native thinking to distributed edge computing?
Result is Kube32. I got the cluster and a full-stack web app with real-time sync running on esp32 s3 dev board and cardputer - https://7f0c-180-74-71-21.ngrok-free.app/ (may go down) (should support at least a few concurrent users - break it)
You won't see one cool thing though, which is the hud pod that uses Cardputer as a keyboard that syncs keystrokes to all clients in real-time.
Honestly, it opens so many possibilities for distributed edge computing and multi-device systems. Let me know what use-cases do you think would benefit from kube32 the most.
Thank you for working on this. The most impressive part for me is getting Kubernetes running on such low spec hardware. I can't think of a use case I have for Kube32 at the moment, but I do have use cases for Kubernetes running on a small amount of RAM. Is there any chance of porting this back to x86/x86-64 PCs or ARM SBCs? For example, being able to run this on a cluster of Raspberry Pis would be interesting as it would free up more RAM for applications.
I feel like a gradualist approach may be better for those applications. Start with "vanilla" k8s, and iteratively target subsystems to shrink down. Either by porting to Rust or simply optimizing the Go code for low RAM environments.
Hmm, have you tried k3s? It promises low ram, but it could still be too much, I'm not sure about your requirements. In the meanwhile I'll see if it can be pushed further, but I guess we both know the answer :)
Super cool! One thought that would make this way more useful to me (and probably a thousand times more complicated, sorry) is if it had a way to split out “GPIO” subroutines from the main app, and schedule them separately. So GPIO would become an addressable unit like pods or load balancers in k8s.
The idea is I could specify something like “I have this app that needs to pull data from a GPIO labeled as a ds18b20 and outdoors” and it takes care of scheduling my app, starting a ds18b20 reader, and networks them together. I never need to know if they’re on the same esp32, separate ones, or in a crazy alternate world the compute bit goes on a VM on a laptop with no GPIOs and the sensors go on a real esp32.
Having GPIOs be an addressable resource would be so cool. Like if nothing else, I could start a new esp32 project without wiring up a breadboard. Most of my stuff is some combination of the same 10 chips anyways.
You now should be able to just plug in whatever peripherals you want, label a k8s Service and your app will select them. Tested with a BME688, a Mini Scales and an RFID2 on a Cardputer.
Based on some measurements, compute in a pod is ~30-90x slower compared to native C. Wasm gas metering adds ~45% on top, sensor driver takes up to 25KB on a board without PSRAM.
I tried to push it as far as I could, there's an example app in the repo. Build a fullstack reactive web app, running as 3 services/4 pods - db, api, web and hud, with persistence. Handled several users in parallel, sub-200ms SSE and API response times, realtime keystrokes from Cardputer on all online clients. That was a hard one to fit, but I think it can be pushed even further. All of that was running on Cardputer and esp32-s3 devboard.
- trustless code execution on someone's hardware (kube32 supports gpio permissions, amongst other stuff)
- your standard Kubernetes cluster can now reach the physical world natively
- OTA updates at arbitrary scale
- fault-tolerance in the physical world, orchestration of workloads based on physical proximity or other sensor data
Exactly, and I discarded the idea of porting k8s, building k8s-compatible system instead. It still carries some overhead, yes, and I'm yet to measure it fully, but it's a good trade-off for the applications it unlocks. If we remove the overhead form equation, do you think the capabilities I mentioned are unlocked with k8s-like system, or do you have a way to achieve all of them at the same time using something else in mind?
Hey guys, I had a few esp32 boards around for a while and always thought - how do we go beyond just one device, would it be possible to apply cloud-native thinking to distributed edge computing?
Result is Kube32. I got the cluster and a full-stack web app with real-time sync running on esp32 s3 dev board and cardputer - https://7f0c-180-74-71-21.ngrok-free.app/ (may go down) (should support at least a few concurrent users - break it)
You won't see one cool thing though, which is the hud pod that uses Cardputer as a keyboard that syncs keystrokes to all clients in real-time.
Honestly, it opens so many possibilities for distributed edge computing and multi-device systems. Let me know what use-cases do you think would benefit from kube32 the most.
Thank you for working on this. The most impressive part for me is getting Kubernetes running on such low spec hardware. I can't think of a use case I have for Kube32 at the moment, but I do have use cases for Kubernetes running on a small amount of RAM. Is there any chance of porting this back to x86/x86-64 PCs or ARM SBCs? For example, being able to run this on a cluster of Raspberry Pis would be interesting as it would free up more RAM for applications.
I feel like a gradualist approach may be better for those applications. Start with "vanilla" k8s, and iteratively target subsystems to shrink down. Either by porting to Rust or simply optimizing the Go code for low RAM environments.
True, for RPI it makes more sense. For microcontrollers it's a different story :D
I think it'd be great to put some effort into this, as a separate project perhaps.
Hmm, have you tried k3s? It promises low ram, but it could still be too much, I'm not sure about your requirements. In the meanwhile I'll see if it can be pushed further, but I guess we both know the answer :)
k3s does well on low ram but I've found it still uses a chunk of CPU.
I've got some k3s measurements here, it's really heavy for Rpi, damn
https://arterialist.github.io/kube32/benchmarks/#on-a-comput...
[dead]
Super cool! One thought that would make this way more useful to me (and probably a thousand times more complicated, sorry) is if it had a way to split out “GPIO” subroutines from the main app, and schedule them separately. So GPIO would become an addressable unit like pods or load balancers in k8s.
The idea is I could specify something like “I have this app that needs to pull data from a GPIO labeled as a ds18b20 and outdoors” and it takes care of scheduling my app, starting a ds18b20 reader, and networks them together. I never need to know if they’re on the same esp32, separate ones, or in a crazy alternate world the compute bit goes on a VM on a laptop with no GPIOs and the sensors go on a real esp32.
Having GPIOs be an addressable resource would be so cool. Like if nothing else, I could start a new esp32 project without wiring up a breadboard. Most of my stuff is some combination of the same 10 chips anyways.
Shipped, wired parts are now Kubernetes Devices (https://kubernetes.io/docs/concepts/extend-kubernetes/comput...)
You now should be able to just plug in whatever peripherals you want, label a k8s Service and your app will select them. Tested with a BME688, a Mini Scales and an RFID2 on a Cardputer.
More info at https://arterialist.github.io/kube32/devices/
Yes! This is the philosophy. Omw to ship!
What kind of overhead does the Wasm approach add in terms of memory and cpu time? I also have way too many ESP32 boards lying around
Based on some measurements, compute in a pod is ~30-90x slower compared to native C. Wasm gas metering adds ~45% on top, sensor driver takes up to 25KB on a board without PSRAM.
More on the benchmarks here - https://arterialist.github.io/kube32/benchmarks/
I'll measure and let you know later, would appreciate if can do measurements on your hardware too as I only have a few options to test on
cloud native emedded devices are an oxymoron
[dead]
Did you test its limits?
I tried to push it as far as I could, there's an example app in the repo. Build a fullstack reactive web app, running as 3 services/4 pods - db, api, web and hud, with persistence. Handled several users in parallel, sub-200ms SSE and API response times, realtime keystrokes from Cardputer on all online clients. That was a hard one to fit, but I think it can be pushed even further. All of that was running on Cardputer and esp32-s3 devboard.
What's the benifit?
there are few imo:
- trustless code execution on someone's hardware (kube32 supports gpio permissions, amongst other stuff) - your standard Kubernetes cluster can now reach the physical world natively - OTA updates at arbitrary scale - fault-tolerance in the physical world, orchestration of workloads based on physical proximity or other sensor data
those are just a few that come to mind
None of this needs the overhead of k8s
Exactly, and I discarded the idea of porting k8s, building k8s-compatible system instead. It still carries some overhead, yes, and I'm yet to measure it fully, but it's a good trade-off for the applications it unlocks. If we remove the overhead form equation, do you think the capabilities I mentioned are unlocked with k8s-like system, or do you have a way to achieve all of them at the same time using something else in mind?
Slop
[flagged]
[dead]
[dead]