The taxi drivers speak pretty much no english. They are also generally older, so unless your Japanese accent is pretty good they won't understand you even if you speak Japanese. The one exception to that is if you say Hiroshima Station, they all seem to understand that one. However, there is a really good street car system in Hiroshima. It will allow you to get almost anywhere in the city for 150 yen.
Hiroden Street Car Map If you get on the #2 or #6 street car, you can get off at Genbaku Dome-mae and you will be right next to the Peace Memorial Park. This stop is announced in English so that tourists will know where to get off.
For other ideas of things to do, check out
GetHiroshima.com It has a list of events that are happening and also bars, restraunts, and shops in Hiroshima and nearby.