Why does my neighbor want me to pay more taxes?

Last year I paid over $50,000 in federal taxes. My income was over $200,000 and I am not married and don't have kids. My neighbor makes $50,000 a year, is married and has 2 kids. Last year he paid zero federal taxes. All the money he had withheld from his paycheck during the year was returned to him in his refund. Yet he tells me I am part of the "wealthy" and that I am not paying my "fair share"........hmmmm, my share is $50,000 and his share is zero. How can he really think that I am the one not paying their "fair share" ???