Physical attraction? Well, if you count how much my boyfriend adores Taylor Swift in this particular area, then I guess I'm already in this situation.
Nothing wrong with being attracted to others, it's human nature. No amount of love can stop someone from being unattracted to other people.
Your partner wanting to be with someone else? Been in that situation too. Let them go, cry a few days, get over it. If only one person wants it to work out, it won't.