What is the punishment in Islam, if a unmarried brother had sex with unmarried sister, who born to one parents?
Islam does not allow Incest (http://www.understanding-islam.com/related/text.asp?type=question&qid=2496) or marrying relations like mothers, sisters.
The Qur'an says:
Forbidden unto you are your mothers, and your daughters, and your sisters, and your father's sisters, and your mother's sisters, and your brother's daughters and your sister's daughters,- - - - - - (4:23)
Once it is established that you cannot marry these relations because of the sanctity of this relationship then violating this sanctity through a most condemned of an act i.e. adultery would incur the offenders the sin of equal weight.
However, Islam does not introduce any special punishment for such criminals. They would receive the prescribed punishment for the crime in the Qur'an. For the punishment of adultery and fornication please refer to: The Punishment For Zina (Fornication) (http://www.understanding-islam.com/related/text.asp?type=question&qid=387)
It should also be noted that this punishment is only to be given once the matter has been established through the verdict of a court. If the case has not been reported, Islam wants that this sin should be hidden. The two involved should sincerely repent before the Almighty and hope for His forgiveness.