You have to remember, rights don't come in groups we shouldn't have 'gay rights'; rights come as individuals, and we wouldn't have this major debate going on. It would be behavior that would count, not what person belongs to what group.
The most important element of a free society, where individual rights are held in the highest esteem, is the rejection of the initiation of violence.
All initiation of force is a violation of someone else's rights, whether initiated by an individual or the state, for the benefit of an individual or group of individuals, even if it's supposed to be for the benefit of another individual or group of individuals.
I would like to restore your right to drink raw milk anytime you like.