This is not entirely a rustc issue; LLVM is a huge part of this as well. Sometimes, some steps need more than four gigs of RAM. loc says that rustc has 1.4 million lines of Rust, and LLVM has 5 million of C++. It's just a huge project.
You would only need cross compilation of your project is as big as rustc, but if it were you would likely want the fastest available machine at your disposal and a lot of effort is poured into making cross compilation as painless as possible.
Making rustc compilable in a 32bits host might be a good idea, but it is far from a priority. If that is a deal breaker, as it is for OpenBSD for example, Rust is certainly a language that you should look at in the short term.