Why does my cat hate me?

Some animals just like to be let alone for awhile... Maybe something happened to it in the past? I know this stray cat we've been feeding every night for 3 years, he came to our door one day and we started feeding him. every morning and every night he would meow and scratch at the door, when we would open it up he would step back some and wait for us to put food down for him, he wouldn't let us even pet him for a whole year after feeding him..after the second year he would get to where he would come in the house and then try to run back to the door to see if we would let him out. We would let him out as soon as he went for the door, everytime someone would start talking above whispering he would get really scared and try to get outside... and now he got used to us and he comes in and lets everyone pet him and he even sits in our laps and watches TV with us.. he's still a bit jumpy when other people visit though.. he's not used to everyone, only us..