It's a bit surprising that they dismissed qcow2, because it does exactly what they want. It's also pretty easy to implement in a vm. The file format is a 2 level page table per snapshot, with pointers to blocks. I suspect they didn't really look at it enough.
Here's the implementation I did for openbsd; it's around 700 lines, including the gunk to interface with the hypervisor.
It's not a good choice for computing diffs, but you can run your VM directly off a read-only base qcow2, with all deltas going into a separate file. That file can either be shipped around or discarded. And multiple VMs can share the same read only base.
So, it probably would have been better to write the code for the hypervisor, and end up with something far more efficient overall.
Not entirely sure of their reasons but their main stumbling block was that they wanted snapshots in-VM, that seemed a bit like a counter-intuitive considering that part of their reason for a VM to begin with was to ensure security.
When working with microVM's, which I'm assuming Cognition is doing, you don't have the option of doing qcow2. Raw disks as either block devices or files is all you've got.
Qcow2 is great for use with vms but I think is actually underrated otherwise.
I use it to back up external disks, usb sticks etc. Because the resulting qcow2 images are sparse and compressed they use less storage which is great for backups.
I wonder why EC2 is so slow on snapshots. We use CEPH internally and if we wanted we could export the diffs [0] (we use proxmox backup instead). Snapshots always felt blazing fast (I still often forget to do them, need a solution to trigger them from the guest)
IIUC, on EC2, the disk (EBS)is one service, and snapshots are in another service (S3). Taking a snapshot involves copying the entire disk to S3, while restoring a snapshot pages blocks in from S3 as the VM accesses the disk.
Ideally, I would like to use something like this without being forced to use a specific file system. This is essentially what qcow2 does, and it's a shame that it's not supported by all hypervisors. But then your implementation would need to be much more complex, and implement what CoW filesystems give you for free, so I appreciate that this is possible in 600 LOCs.
Also, your repo doesn't have a license, which technically makes it unusable.
Created an issue of this on their github as I wanted this issue to also go to their github, let's hope they add a permissive license like MIT or apache
Thanks for writing the blog post; it was a fascinating read!
I was curious about a couple of things:
* Have you considered future extensions where you can start the VM before you completed the FS copy?
* You picked XFS over ZFS and BTRFS. Any reason why XFS in particular?
* You casually mention that you wrote 'otterlink', your own hypervisor. Isn't that by itself a complicated effort worthy of a blog post? Or is it just mixing and matching existing libraries from the Rust ecosystem?
I've been poking around with both, and FIEMAP has some nice benefits when working with VM disks. For example, if you have the same shared based across multiple VMs, you can use FIEMAP for dirty tracking of blocks, by detecting which extents are no longer shared with the common base. With LSEEK you'll lose that and have to snapshot the entire backing drive.
I love this, and the post made a complex topic easy to follow for a mere mortal like me. Including images of tables with no alt text is a bit of a barrier though, especially when the company is called Cognition.
I wonder didn't they used VDO thin provisioning with LVM2.
Also, a few years ago I've implemented VM management tool called 'vmess', in which the concept is to maintain a tree of QCOW2 files, which R/W snapshots are at the leafs and R/O snapshots are the nodes of the tree. The connection up to the root is made via QCOW2 backing-file store mechanism, so a newly created leaf starts a 0 space. I did this because libvirt+qemu impose various annoying limitations surrounding snapshots-with-in-qcow2, and I liked the idea of file-per-snapshot.
Can this be used in a public cloud provider , to speed up VM provisioning in CI/CD pipelines? Im looking for ways to speed up app provisioning for e2e tests.
The blog's title can be misleading here, "we" in this context refers to the Cognition team. I don't work at Cognition, just thought this was interesting.
I don’t know how to express to you how stupid, inadequate, and envious I feel of this level of competence.
For me this article has the density of slaps of a plutonium ingot.
It’s moving to read (and “maybe” understand, given how well it’s written).
Wow, maximum respect, truly.
It's a bit surprising that they dismissed qcow2, because it does exactly what they want. It's also pretty easy to implement in a vm. The file format is a 2 level page table per snapshot, with pointers to blocks. I suspect they didn't really look at it enough.
Here's the implementation I did for openbsd; it's around 700 lines, including the gunk to interface with the hypervisor.
https://github.com/openbsd/src/blob/master/usr.sbin/vmd/vioq...
It's not a good choice for computing diffs, but you can run your VM directly off a read-only base qcow2, with all deltas going into a separate file. That file can either be shipped around or discarded. And multiple VMs can share the same read only base.
So, it probably would have been better to write the code for the hypervisor, and end up with something far more efficient overall.
Not entirely sure of their reasons but their main stumbling block was that they wanted snapshots in-VM, that seemed a bit like a counter-intuitive considering that part of their reason for a VM to begin with was to ensure security.
When working with microVM's, which I'm assuming Cognition is doing, you don't have the option of doing qcow2. Raw disks as either block devices or files is all you've got.
That's nonsense. Qcow2 is a file, with a pretty easy to implement format.
Qcow2 is great for use with vms but I think is actually underrated otherwise.
I use it to back up external disks, usb sticks etc. Because the resulting qcow2 images are sparse and compressed they use less storage which is great for backups.
Also VDI/VHD was not mentioned which is interesting.
I wonder why EC2 is so slow on snapshots. We use CEPH internally and if we wanted we could export the diffs [0] (we use proxmox backup instead). Snapshots always felt blazing fast (I still often forget to do them, need a solution to trigger them from the guest)
[0] https://ceph.io/en/news/blog/2013/incremental-snapshots-with...
IIUC, on EC2, the disk (EBS)is one service, and snapshots are in another service (S3). Taking a snapshot involves copying the entire disk to S3, while restoring a snapshot pages blocks in from S3 as the VM accesses the disk.
This is interesting. Is it hypervisor-agnostic?
Ideally, I would like to use something like this without being forced to use a specific file system. This is essentially what qcow2 does, and it's a shame that it's not supported by all hypervisors. But then your implementation would need to be much more complex, and implement what CoW filesystems give you for free, so I appreciate that this is possible in 600 LOCs.
Also, your repo doesn't have a license, which technically makes it unusable.
https://github.com/CognitionAI/blockdiff/issues/3
Created an issue of this on their github as I wanted this issue to also go to their github, let's hope they add a permissive license like MIT or apache
Thanks for writing the blog post; it was a fascinating read!
I was curious about a couple of things:
* Have you considered future extensions where you can start the VM before you completed the FS copy?
* You picked XFS over ZFS and BTRFS. Any reason why XFS in particular?
* You casually mention that you wrote 'otterlink', your own hypervisor. Isn't that by itself a complicated effort worthy of a blog post? Or is it just mixing and matching existing libraries from the Rust ecosystem?
Not the author, but:
> Any reason why XFS in particular?
XFS is still the default filesystem of choice for many enterprise systems. For instance, Red Hat states in their manual [1]:
There are good reasons to choose other file systems, but if you just want good performance on simple storage, XFS is a pretty good default.[1] https://docs.redhat.com/en/documentation/red_hat_enterprise_...
Also: did you consider using LVM snapshots?
I see you use flags to determine if a file needs syncing. When we used fiemap within GNU cp we required FIEMAP_FLAG_SYNC to get robust operation.
(We have since removed the fiemap code from cp, and replaced it with LSEEK_DATA, LSEEK_HOLE)
I've been poking around with both, and FIEMAP has some nice benefits when working with VM disks. For example, if you have the same shared based across multiple VMs, you can use FIEMAP for dirty tracking of blocks, by detecting which extents are no longer shared with the common base. With LSEEK you'll lose that and have to snapshot the entire backing drive.
Have you considered https://github.com/containerd/overlaybd? It seems to offer very similar features to blockdiff.
I love this, and the post made a complex topic easy to follow for a mere mortal like me. Including images of tables with no alt text is a bit of a barrier though, especially when the company is called Cognition.
I wonder didn't they used VDO thin provisioning with LVM2.
Also, a few years ago I've implemented VM management tool called 'vmess', in which the concept is to maintain a tree of QCOW2 files, which R/W snapshots are at the leafs and R/O snapshots are the nodes of the tree. The connection up to the root is made via QCOW2 backing-file store mechanism, so a newly created leaf starts a 0 space. I did this because libvirt+qemu impose various annoying limitations surrounding snapshots-with-in-qcow2, and I liked the idea of file-per-snapshot.
VDO: https://docs.kernel.org/admin-guide/device-mapper/vdo.html (original project URL: https://github.com/dm-vdo/kvdo )
vmess: https://github.com/da-x/vmess
Can this be used in a public cloud provider , to speed up VM provisioning in CI/CD pipelines? Im looking for ways to speed up app provisioning for e2e tests.
Interesting tool. Something like btrfs send/receive but on a file level and fs agnostic.
qemu-img convert supports copy_file_range(2) too. Was the `--copy-range-offloading` option used in the benchmark?
It would be helpful to share the command-line and details of how benchmarks were run.
The blog's title can be misleading here, "we" in this context refers to the Cognition team. I don't work at Cognition, just thought this was interesting.
I don’t know how to express to you how stupid, inadequate, and envious I feel of this level of competence. For me this article has the density of slaps of a plutonium ingot. It’s moving to read (and “maybe” understand, given how well it’s written). Wow, maximum respect, truly.