Personally I wouldn't teach OpenGL ES 3.0 or any OpenGL because its state machine (https://webgl2fundamentals.org/webgl/lessons/resources/webgl...) and horrible error system means tracking down bugs is hard. Instead I'd teach WebGPU as it's portable and has great error messaging.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
I mostly agree regarding GL's programming model, but tbf some problems have been fixed by WebGL2 (much better validation and error reporting - e.g. very helpful error messages right on the browser console - and it's probably the least buggy and most portable 3D API implementation of them all, including the current state of WebGPU.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
WebGPU will however always suffer from being an API designed for the Web, and targeting 2015 thereabouts hardware.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
I agree with Adam Sawicki that it's not that relevant any more but also I do believe it's valuable lesson to tech students where the modernity comes from.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
> carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need to distribute shaders as either source code or a portable intermediate representation, in both cases compiled at runtime for the user's actual GPU. There are also many fixed function parts of the graphics pipeline with wildly different behavior on different hardware which you need to abstract over. And it all runs in parallel with the CPU, often with a separate pool of RAM, so you need implicit or explicit synchronization and data transfer which adds a lot of complexity especially in error cases.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
I think most of it comes down to 'historical accidents' tbh.
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
The thing that makes it "hard" is that it's all indirect. You compile a program for the GPU (a shader). Then you have to setup data (textures and buffers) for it. Then, you need to spend a bunch of time telling the GPU how to use that data, offsets where each piece of data is, what formats the data is in,, how to advance to the next piece of data, etc.. etc..
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
instead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
actually yes you could design an alternative API and even implement it....
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree.
Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
TL;DR because WebGL 2 is basically equivalent to OpenGL ES 3.0, making it the most portable GPU API in the world by far, running on everything from laptops and phones to smart fridges and car head units.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.
Personally I wouldn't teach OpenGL ES 3.0 or any OpenGL because its state machine (https://webgl2fundamentals.org/webgl/lessons/resources/webgl...) and horrible error system means tracking down bugs is hard. Instead I'd teach WebGPU as it's portable and has great error messaging.
I'm sure people will objected but I'd argue WebGPU is easier to understand, has better errors, has no state machine, works C++, Rust, JS/TS and other languages, and closer to modern GPU programming if you want to move to Metal or D3D12 or Vulkan later.
Further, OpenGL (and ES) are a nightmare of bugs and incompatibilities students might run into that WebGPU fixes.
That said, the article mentions Apple platforms where you can use ANGLE if you happened to have students stuck there.
I mostly agree regarding GL's programming model, but tbf some problems have been fixed by WebGL2 (much better validation and error reporting - e.g. very helpful error messages right on the browser console - and it's probably the least buggy and most portable 3D API implementation of them all, including the current state of WebGPU.
For teaching I would generally use the web browser via WebGL2 or WebGPU, just because those APIs require much less boilerplate in the browser environment to get to your first triangle and the edit-test-loop is pretty much instant.
WebGPU will however always suffer from being an API designed for the Web, and targeting 2015 thereabouts hardware.
Better than OpenGL, but don't expect that much improvements coming into their way, other than what the Chrome team might feel like pushing, as I doubt Firefox or Safari teams have much left to say about where it goes beyond version 1.0.
I agree with Adam Sawicki that it's not that relevant any more but also I do believe it's valuable lesson to tech students where the modernity comes from.
I just don't see personally sense to focus on it longer than just 1mo of intro, tutorial, get through simple example, show limitations, carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles.
> carry on to Vulkan/WGPU with background story that there is also DX12 and Metal that follow similar principles
Except that's a minefield all of its own. DirectX 12 is its own special beast that is unlike the others. Vulkan is already mired in 3 revisions of architecture changes. WGPU is based on the (outdated) first cut of Vulkan, not the current model. Of all of these, only Metal has a reasonable on-ramp for newcomers to 3D graphics.
Nice breakdown. I've been wondering why it's so complicated to program on the GPU compared to the CPU. Is it a language paradigm issue or maybe a hardware design limitation? I always think of this when looking at the graphic APIs, espeically Vulkan.
It's mostly hardware design limitations. GPU hardware is way more diverse than CPU hardware, so you need to distribute shaders as either source code or a portable intermediate representation, in both cases compiled at runtime for the user's actual GPU. There are also many fixed function parts of the graphics pipeline with wildly different behavior on different hardware which you need to abstract over. And it all runs in parallel with the CPU, often with a separate pool of RAM, so you need implicit or explicit synchronization and data transfer which adds a lot of complexity especially in error cases.
If you really want to see why all the cruft is there, and how much you could remove if you dropped support for older hardware, look no further than Sebastian Aaltonen's "No Graphics API" blog post (which he actually implemented too): https://www.sebastianaaltonen.com/blog/no-graphics-api/
I think most of it comes down to 'historical accidents' tbh.
E.g. if CPUs would have closed ISAs and heavily guarded IPs, you'd also need a 'driver' and an 'OpenCPU API' to make any use of them (of course the public ISA of a CPU is essentially the driver interface baked into the hardware).
The other big difference is focus on decent single-thread performance for CPUs versus performing thousands of small tasks in parallel on GPUs, but each one slowly. This difference causes a different programming model (e.g. running an entire application in a single 'main' function from start to finish, versus spawning small specialized tasks which are running across hundreds of threads).
And the specific problem with Vulkan is that it tried to solve two conflicting requirements: on one hand being an explicit low-level API without 'magic abstractions', on the other hand supporting very different GPU architectures (especially the mobile GPU zoo). The resulting mess was quite predictable to be honest...
One thing that helped me understand it was to reframe it as a network application, where you have a client running on the CPU and a server running on the GPU (the shaders).
A bunch of the complexity comes from having to write both client and server, and the communication protocol between them, then some more comes from the client being responsible for spinning up the server. As with every network service, client/server traffic is the slowest part of the architecture, so a bunch of complexity then comes from having to minimise the amount of network calls (or draw calls, in this case).
Then you hit the part where all of this is performance critical (soft-realtime requirements), so the whole thing is geared towards “writing your own userspace IP stack in C++” rather than “writing a flask application in Python”.
part of the reason is historical cruft & history path dependency. GPUs didn't start as general computing device for parallelism but a device for the specific purpose of accelerating graphics with very specific fixed functions for that job and only that. with time that fixed pipeline became more complex and more configurable, not fixed/more flexible and hacks were in parallel used to make use of it for tasks outside graphics those hacks were later made into something more formal (CUDA/OpenCL).
CUDA being the winner (closed source, not really usable on non-nvidia GPUs) between those 2 and nvidia not wanting to open their competitive advantage to the competition for the niche of non gamings uses / scientific computing (and later AI, which became a huge win for nvidia).
vulkan came around after as this open thing (based on a previous attempt by AMD called mantle) that's trying to break down the leaky abstractions and the divide between language for graphics & video games and separate language/frameworks for gpu non-graphic computing with the promise of giving more control and more room for squeezing performance from the hardware at the cost of more complexity with the idea that you'll probably build your own abstractions on top of it for your subset of applications.
If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
don't forget that the drivers for GPUs would until recently get updates to patch specific video games for performance issues (not sure if they still do that ?)
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (and if the competition between the duopoly didn't sabotage things)
Sony tried that with the PS3's Cell processor, just a wise parallel vector processor instead of dedicated hardware.
If you knew how to use it, you could get great results, which is how Naughty Dog's games became so spectacular. But if you just tried to port an existing game, things went downhill quickly, which is why GabeN talked on stage how much he hated the PS3.
Oh and this complexity led to lots of games just switching to one of the existing engines that supported it natively, and started the modern dominance of Unreal Engine. But it's also why engines introduced more parallel architectures, which a few years later became very useful when quad-core processors in PCs became mainstream.
Good summary, thanks. Sounds like it's all still pretty early, even for graphics but much more for parallel compute, which seems the direction the hardware is going in. I'm sure there is a CUDA equivalent for CPU but I would guess it's all the way back in the 70s or earlier. Hopefully open standards gets some traction.
> If GPU were from the get go designed as a parallelism tool first and video game tools second the hardware architecture and the language would probably be different than today and simpler (...)
The market demand was for video game tools, though.
People didn't wanted faster compute, they wanted their computer games to work smoothly on the hardware they already own. Therefore, they preferred to spend 100-200€ on a point upgrade, much like RAM, than scrapping their 2k€ PC.
To give people a reason to buy these upgrades, games had to use them. Therefore video cards were designed specially for the usecases and development flow that game developers already followed.
If it wasn't for the video game market, NVidia wouldn't exist as we know it today.
The thing that makes it "hard" is that it's all indirect. You compile a program for the GPU (a shader). Then you have to setup data (textures and buffers) for it. Then, you need to spend a bunch of time telling the GPU how to use that data, offsets where each piece of data is, what formats the data is in,, how to advance to the next piece of data, etc.. etc..
There are a few libraries that try to abstract over all this indirection but all the graphics APIs themselves are lower-level so you're left to do all of that yourself.
https://docs.swmansion.com/TypeGPU/
https://github.com/google/toucan
It's not more complicated, you have things like CUDA which is basically C/C++ that compiles to GPU.
But GPUs have thousands of cores, which means if you want any benefit, the smallest data type you must use is something like array[n * 1024], or your GPU is wasted doing nothing. So you must shape your programs into that memory model or you're just wasting your time (and GPU).
That's >50% of the problem.
Right, that makes sense. There's a ton of overhead in the setup though, I'm assuming that's because everything runs on the CPU by default. Is this then an OS design choice? Thanks for your input.
There can be overhead, though GPU memory + drivers are some of the most optimized software in the world.
Some OS/hardware have separate GPU memory (Nvidia), some have unified memory (Macos, modern consoles), but moving memory is usually not the problem.
Things don't "run on CPU by default", it's up to the programmer where to run it. But the CPU and GPU paradigm of "running a program" are very different, since on GPU you're running SIMD (i.e. running your code on arrays) or else there is no point to use a GPU.
For me, if-branching is the hardest to wrap my head around. "If"s are so natural to CPU programming that it's second nature to any developer but on GPU, it's expensive.
instead of iterating over an array and if-branching to one of two paths for each item, conceive of your conditionals as logic operations which test for set membership (i.e. "is this array item a member of the set i care about?") and with this array mask of "do the thing or don't do the thing" flags, you can mutate your data 0 or 1 times.
the key for me is thinking in terms of sets—much of your conditional logic is probably about sorting things into sets anyway.
It's partially historical; I'm sure if we redesigned GPGPU today from the ground up, we'd have an entirely different pipeline. And partially due to hardware vendor lock-in; you can't really design an alternative to Vulkan/d3d12 even if you had the skill and drive to do so.
But also partly in the fact that the GPU is specialized for certain operations over a CPU. It wants and need to obtain data in a different way from a CPU to maximize throughput. So language constructs are designed around making sure developers stay on this fast path. The complexity can be seen as some intentional gatekeeping to signal that this kind of programming is only for specific kinds of problems.
actually yes you could design an alternative API and even implement it....
there's HW references + the linux drivers for AMD (both UMD and KMD), there's refs for NV and their open linux KMD (no UMD though), intel also has HW refs and open drivers for linux
I think it would be possible to put together as long as you don't mind only working for certain hardware, like a prototype you know
I might give it a shot to make a proof-of-concept "high level low level" graphics API ngl if no one else feels like so...
I think there's space for a "two-level" design like have the driver consist of the HW-specific parts, expose things in terms of GPU VAs, explicit sync, explicit memory management, command buffers etc. then have a high-level wrapper over it to provide vaguely GL-ish semantics, and you could intermix the two APIs, obviously you wouldn't be able to get the VA for a high-level object because that'd break the autotracking, if you use low-level you're responsible for syncing, cycling etc.
GLES isn't a bad way to teach graphics but I wish there was a middle ground between it and VK.
The worst part of GL is the mysterious global state you will never understand without studying the spec. The world could use an API that's basically GL but without global hidden singleton (per thread!) BS behind the scenes.
> but I wish there was a middle ground between it and VK
There's WebGPU (probably the best option for teaching), D3D11 and Metal.
And (shameless plug): sokol-gfx tries to find a balance between simplicity and modern 3D APIs:
https://github.com/floooh/sokol
Demos with links to C source and shader code:
WebGL2 backend: https://floooh.github.io/sokol-html5/
WebGPU backend (has a couple more demos which use features not available on GLES3.0/WebGL2, like compute shaders): https://floooh.github.io/sokol-webgpu/
That middle ground is often called DirectX 11 but sadly it's stuck to a specific platform.
I've learned more graphics writing software rasterizers and ray tracers than using any APIs. If the goal is to teach GPU rendering specifically, then OpenGL is indeed more amenable to teaching than the others. But if the goal is to teach graphics and rendering concepts, then OpenGL et al are just miserable and counter-productive.
Disagree. Trying to express all of graphics and rendering through the lens of OpenGL will of course fail and doing 3D rendering itself via raytracing is simply not possible in rasterization focused OpenGL without dirty workarounds. That is obvious.
A fundamental shift in thinking that got me into high performance graphics programming and parallelized image processing, the "Aha" Moment, is being forced to express how to calculate the final pixel independently, without having access to neighboring pixels of the same output buffer, confined to the straight jacket of the fragment shader. OpenGL with it's combination of hand-holding and relative simplicity as compared to Vulkan and limits of a predefined shader pipeline as compared to a pure software implementation is exactly the thing that I needed to get into graphics programming. It's an excellent jump in point I think.
The way I’m interpreting their post is, you could far more easily teach things like introductory projective geometry without a GPU.
The linked webpage refuses keyboard navigation e.g. cursors, pageup/down and home/end seem to do nothing.
Awful.
I think it's because of the webgl demo at top stealing focus
Specifically, it registers a keydown handler on the <html> that swallows everything.
Sounds like a great way to make sure students understand fundamental concepts.
TL;DR because WebGL 2 is basically equivalent to OpenGL ES 3.0, making it the most portable GPU API in the world by far, running on everything from laptops and phones to smart fridges and car head units.
Web browsers have also done a lot of behind-the-scenes work on compatibility. The behavior of WebGL 2 is much, much more consistent across platforms than native OpenGL [ES] ever was.