My Boss was sleeping with my istant and I quit after months of dealing with it.?

I'm interviewing for new positions..what do I tell my prospective employer why I left? I held a high position in the company but the affair among other unethical actions undermined my position with everyone in the company as they buddied up with my istant because they knew she was intimate with the boss. I think one prospective employer called him and he did not give me a good recommendation but I can't prove it. What do I say why I left? It's a closed knit industry and if I say what really happened, it could sound like sour gs...if I say I'd rather not discuss it, it leaves the reason to their imagination. And now I don't want employers to call for a reference because I fear he will submarine me. My istant, though unqualified, was promoted to my position when I left. Has anyone ever faced this dilema?