Were you on passenger for apache or passenger for nginx, and were you on passenger 3? Those make quite a difference. Memory usage and performance have been greatly improved in 3, with important parts optimized in C. The dynamic multiprocessing can save you more memory during idle time.
We were using passenger with apache if i remember correctly. It's a few years ago and i was more on the developer side rather than sysadmin side of things back then.
Its probably worth taking another look? For my ruby applications at the moment i use nginx+unicorn, which works really great.
