Speaking with respect and actually respecting the other are two different things altogether
. I think it is difficult to respect someone you don't know, no matter how old he is, as you don't know how he lived his life. But I believe that one should speak with respect to an older person, out of respect for his age, and not the person himself. In general, an older man will be wiser than a younger one, and will have done much more things in his life.
Speaking with respect is not only Japan-specific. For example in my country (Greece, Europe), it is the norm to speak with respect to people older than you. Our language has different words for you (single person) and you (plural, as in: you two), so we generally use plural forms when talking to older people.