Verse 9:28 from the Qur'an reads:
O you who believe (in Allah's Oneness and in His Messenger (Muhammad SAW)! Verily, the Mushrikun (polytheists, pagans, idolaters, disbelievers in the Oneness of Allah, and in the Message of Muhammad SAW) are Najasun (impure). So let them not come near Al-Masjid-al-Haram (at Makkah) after this year, and if you fear poverty, Allah will enrich you if He will, out of His Bounty. Surely, Allah is All-Knowing, All-Wise.
According to the idea of "Itmaam e Hujjah", it is not for man to declare another as 'kaafir'. How then can the disallowance of "non-Muslims" from entering the haram be justified today? And are we not denying them an experience of witnessing devotion and submission, to the One, at its best?
You have rightly observed that the referred verses pertain to the addressees of the Messenger of Allah. However, the directive seems to be general in its implication. This generalization is because of the statement of the reason behind the directive in the referred verse.
It is said that it is their being polluted with the crime of polytheism that the polytheists should not even come near the Holy place. It should be noted that in addition to the inherent information of the creator coming down from the first man and perpetual interaction of the Almighty with his prophets and messengers He has performed Itmam-e-Hujjah regarding Shirk through both of the branches of the progeny of Abraham i.e., the Israelites and Ishmaelites. This establishment of Itmam-e-Hujjah has become a historical fact and has bearing over all generations for all times to come. God has ordained that the Ka`bah is the house of worshiping only the one God, therefore all those who are polluted with the crime of associating partners with the Master of the House should not be allowed to enter the place.
However, one thing that needs to be appreciated is that the word 'Mushrik' (polytheist), as it is used in the Qur'an, does not include all non-Muslims. 'Mushrik' is an adjective, which signifies consciously adhering to polytheism as one's religion. Thus, many non-Muslims, like the People of the Book - Jews and Christians - are not included in the scope of this word. Due to this reason, we see that the Jews and the Christians were not subjected to the final punishment of execution after the supremacy of Islam in Arabia. Therefore, the prohibition from entering the Ka`bah given in the cited verse does not hinder us from allowing the People of the Book - the Jews and the Christians - to enter the Ka`bah.