For me, まさか is something I would use (or expect to be used) when the thing said is truly unbelievable.
マジ is more when it`s surprising, but not quite as far fetched.
Another thing to note is that まさか is something you can use when the subject is yourself. It can be just an expression of complete dismissal - as the topic would never happen, don`t be silly.
ie:
「ともちゃんのお兄さんと付き合ってるの？」
「まさか～」
「ともちゃんが留年だって」
「え、マジで?」
本当に is fine. You`ll also probably hear 本気(で).
正気か is pretty common among (my) friends.
It falls into the same kind of meaning as "Are you insane?"