So perhaps the answer is to allow Homebrew to bootstrap itself with a basic toolchain if one isn't available. It could even deprecate its toolchain if you later installed Xcode.
There's no technical reason why this isn't possible -- the Apple gcc toolchain is, by necessity, available as Open Source.
I use Xcode rather a lot, so this isn't a big deal for me, but it seems like a sane answer for "the rest". Sure, you'd still need Homebrew, but that plus a toolchain is a much smaller burden.
This sounds absolutely ideal. Homebrew bootstrapping itself using packaged builds of the xcode toolchain which, as someone else mentioned, apple has to provide as it's open source. Homebrew itself is pure ruby, which is already part of the OS.
It could be done as part of the homebrew install process. This would be a step forward for everyone. Most OS X devs use homebrew anyway, and for those that don't use xcode this would simplify things a lot.
I hate downloading that massive DMG. I can't even use wget easily because I have to be authenticated. Who cares about what apple decides to do or not do with the tools the community developed for itself? That's what the licenses are for, people. So we can look after ourselves.