Several ahadith approve Ali's (rta) khilafat. Abu Bakar's (rta) khilafat was either an accident or deliberation. And in the light of the above; was he a true khalifah al-Rasool? Was Muawiya's decision to appoint Yazid as his successor, legitimate especially in the presence of Hussayn?
Khilafah of Abu Bakr Siddique (rta) was acknowledged by the entire community of the Companions of the Prophet (sws). The Companions as a community, could not have joined hands on deviating from the command of the Prophet (sws). The Holy Qur'an affirms their truthfulness and puts on them the huge responsibility of communicating the religion of God to the world. Since their veracity is confirmed by the Book of God we cannot claim they disregarded the command of the Prophet (sws) regarding imamat of Ali (rta). Who would have known more than them that the Prophet (sws) intended them to take Ali (rta) as a caliph if that was the case? The whole dispute of khilafat, it needs to be appreciated, originates in the fact the Prophet (sws) did not appoint anybody.
I am sorry I cannot comment on the decision of the Caliph Mu'awiyah. This is because I do not and cannot know the ground realities in that time. We believe that Islam does not bestow any special privilege on religious basis to the members of the family of the Prophet (sws).