[nijiro]

I'm in a very similar situation as you. Luckily for me a nearby community college (Montgomery CC) has Japanese as a foreign language. Being close to Philadelphia I considered Temple briefly but as you said they only have Japanese as a minor (the next closest major is Asian Studies).
My plan after community college is to go to Florida State, which has a good Japanese program. While they don't have a flashy Tokyo campus like Temple, and aren't as close as Hawaii, they have great professors and a lot of opportunities for studying abroad. My sister went there (she took some Japanese courses but majored in Interior Design) and also briefly worked in their Distance Learning department so that is how I know. It isn't "renowned" as far as I know but renown doesn't always equal proficiency.
Also Tallahassee (the city it's located in) is a great place to live. I lived there for two years. :3
My recommendation is to go to collegeboard.com and use their college search function. You can look for schools with the major you want, and any other qualities you like in a university.
Or you can go to FSU with me!
Good luck and let me know how it turns out!