On one level, many people have been erroneously raised under the concept of cultural relativism: every culture or belief is equally OK. The dogma of infallibility contradicts that, and states emphatically that there is a SINGLE set of right teachings. These people, these "I'm OK, you're OK" people, are offended at this incredibly politically incorrect view of the world. To them, such thinking is backward and outdated.
Related to the above, it is truly hard to believe that good people who believe in Christ are somehow wrong. I personally don't struggle with this: to me its clear: Christ established the Catholic Church and guaranteed its total fidelity (not the total fidelity of Her members, but Her as a whole) to Him and His Word; any one who disagrees with the Church disagrees with Christ and is therefore wrong. That being said, there are many who have loved ones who don't belong to the same denomination or Church they do. Perhaps their children apostacized from the Catholic faith, or maybe their siblings all belong to different denominations. Do you really, really, want to say they are in error? I mean is it nice to believe that well-meaning, Bible-believing, sincere people are somehow wrong? When the idea that Chirst founded ONE, Holy, CATHOLIC and Apostolic Church and not several related churches that are ok, is tossed around, do we want to think that our loved ones are in error and not part of the Church that Christ founded? God is too loving to let that happen, right?
Alternately, as a Methodist growing up, I was proud to call myself Methodist, and would defend my denomination against any claim that it was wrong, because it was mine and I belonged to it!!! There was an emotional attachment to it. It is a form of ethno-centrism, which is a view that one's own culture/belief system is the reference point from which they view the world, yet perfectly normal. Infallibility disrupts that emotional attachment a non-Catholic has with the denomination of their childhood (or of their choosing). That denomination is part of who you are, and to admit it is wrong is a horrendous blow to your idea of who you are. This blow is even harder when you choose that denomination (often at great personal cost). To think that someone else is calling your church wrong upsets you.
The two above are hardly separated from each other. It is only logical that a person believe that their denomination is right (which is why they are a part of it to begin with, partly). Also, I don't know a single person who only has friends or relatives that belong to their denomination. Thus, while you believe your denomination is right, those of your family or friends can't be wrong. You can't have both, so the idea of infallibility or discernable religious truth becomes fantasy: it sounds nice but can't be real (because if it is, then either you or your loved one-or both-are wrong).
Even more personally, however, infallibility is an affront to people's pride and their own arrogance. The United States is inherently a Protestant nation. The entire ethic of "rugged individualism" is based on the Protestant idea of individual interpretation of Scripture. We relish that, we thrive off that, we LOVE that. It gives us POWER, puts US in control. For some this individual interpretation allows them to justify their vices. For others, it fuels their ego to create new and better theologies. It feeds their importance. Infallibility creates an uncomfortable alternative: perhaps you are wrong! For those trapped in vices they cherish (like masturbation, sex before marriage or homosexuality), infallibility would require them to give up something they love. For those who thrive on creating new and better theologies, it undermines their ego and checks their importance. For both of these groups, infallibility is too condemning.
_______________________________________________________
I really want to say that there are those who honestly, not for the selfish or deluded reasons above, cannot accept the idea that a Church has the gift of infallibility. But when I think of the people I know, they all fit into the molds of the above (to greater or lesser extent). In each and every case, admittance of infalliblity would undermine or confront their lifestyle, or possibly convict their consciences.
Thomas Aquinas wrote that those who truly seek the Truth will be granted the Grace to find Jesus and His Church. I honestly believe that. Those who continue to reject the concept of infallibility are not really searching whole-heartedly for the truth. If they were, wouldn't they want a Church that speaks authoritatively, and can, without doubt, guide them to the Truth?