How much can we learn about the residents and the program during one day interviews though? I mean, if I just went with my first impressions all the time, I would've missed out on some of my best friends. People present themselves differently on different days, and our feelings of programs differ depending on our mood.
Co-resident personalities and personal fit are important to me, but I wouldn't necessarily rule out a place if I didn't fall in love with everybody during my visit -- especially if other things about the program fall nicely into place. I think there is some value to considering some of the more objective factors and reputation for fellowship match, especially if training at top places with some of the more renowned people is important to you.
Fit is important, but my initial gut feelings have frequently been wrong. You could always go for second looks, reach out to former graduates from your medical school, and talk to your advisors for more exposure and different opinions.