Why think about society at all? Who can stop you? Who can stop you, unless you are prepared to be stopped? And why are we prepared to be stopped? Because we have entered into a surreptitious kind of contract with the society.
You are not afraid of the society.
You are afraid of letting go of your privileges, your emoluments that you get from the society.