> It's an important comparison of the mechanisms, even in 2023, you can still find binaries on modern Linux distributions with executable stacks due to the fail-open design, 20 years later.
Unfortunately, for C code using GCC’s nested functions extension (or for languages that want to be ABI-compatible with C and support nested functions, like that paragon of advanced features called Pascal /s ), there’s no other compilation strategy in current ABIs. The patches to switch C (and not just Ada) to function descriptors[1] with an ABI break have been sitting on the GCC mailing list since approximately forever[2], but it doesn’t seem like there’s been any progress.
[1] The strategy is basically to compile (*fp)() not as
thus essentially inlining the (currently stack-allocated) closure calling thunk at all indirect call sites. It is ABI-compatible on x86 and x86-64 with all code that does not involve nested functions, place functions at odd addresses, or tag function pointers itself (and I think with all arm64 and riscv code, although arm32’s usage of the low pointer bit for Thumb interworking is bound to make this trickier).
Now all software must pay the price and miss out on important mitigations, for all eternity, just because of some largely unused feature in one compiler?
IBT is already further along here. The hypothetical solution for executable stacks is to recompile all of your nested-function-using or -calling code with -ftrampolines (except that won’t work without the patch above—silently, really GCC?..). The already real and working solution for IBT is to recompile all of your indirect-branch-using code with -fcf-protection=branch. So, ignoring the fact that nested functions are in practice much rarer, if you accept the former as valid you’ll need to accept the latter as well, as far as logic as concerned.
I wouldn’t characterize this as a “largely unused feature in one compiler” screwing things up, but rather as the ABI on most Linux and -adjacent platforms (except SysV Itanium and FDPIC IIRC) being incapable of supporting closures (without executable stacks). That these are missing from standard C, and only present in languages that are either niche (Pascal, Ada) or don’t care about following the platform ABI (Rust, Go, C++’s lambdas), is a defect of C (and that’s at least a somewhat popular opinion among ISO C committee members[1]).
Of course, OpenBSD essentially does not have a stable ABI, so it’s much freer to experiment here.
Unfortunately, for C code using GCC’s nested functions extension (or for languages that want to be ABI-compatible with C and support nested functions, like that paragon of advanced features called Pascal /s ), there’s no other compilation strategy in current ABIs. The patches to switch C (and not just Ada) to function descriptors[1] with an ABI break have been sitting on the GCC mailing list since approximately forever[2], but it doesn’t seem like there’s been any progress.
[1] The strategy is basically to compile (*fp)() not as
but as (untested) thus essentially inlining the (currently stack-allocated) closure calling thunk at all indirect call sites. It is ABI-compatible on x86 and x86-64 with all code that does not involve nested functions, place functions at odd addresses, or tag function pointers itself (and I think with all arm64 and riscv code, although arm32’s usage of the low pointer bit for Thumb interworking is bound to make this trickier).[2] https://gcc.gnu.org/legacy-ml/gcc-patches/2019-01/msg00735.h...