Kinsey on Haredim

So I went to see the pre-screening of Kinsey last night at Doc. It is an important topic and I could imagine screening spots in Bnei Akiva to get reactions and whatnot (but obviously not the whole thing). It was a lame movie in my mind, oh well. Still worth the free ticket I paid and two hours I guess.
My conundrum comes like this: In the movie Kinsey is a advocate of doing what is normal, "hey if the birds and bees do it..." argument for why people should be sexually liberal. Great. The movie also puts stress on wife swapping though. The argument put forward is, sex should/need not be exclusive to love. It is a good argument as far as I am concerned, except for one point: no one gets married for purely a-sexual motives. But what if there were people who do not marry due to sexual pressures?
It appears to me that in Haredi communities marriages are arranged based (almost) purely on people's behavior. "He is such a talmid chachum and she is such a balas chessed". Thus when two people marry, were one spouse to "cheat" on the other and claim, "...but you are the only one I want to share my life with." I would buy it. Marriage is established to gain a life partner and not for mere sexual fulfillment. While sex might come with marriage, other things do as well. One would never claim that their spouse were cheating if he or she ate food prepared by someone else.
I am not making a halachik argument. Obviously it is assur. I am also not well versed on the rules of keddushin, so there might exist technical stipulations built into the marriage contract which would prohibit this on absract legal grounds. My question is: is it immoral? If sexuality is not a precondition for marriage (and seeing the kallah once before marriage is not sufficient to claim that there is sexual selection to marriage, a priori) I cannot see how it is immoral to sleep with another partner on a limited basis.