George Lopez -- Defends Using White Caddies ... Not Racist, It's a Fact (VIDEO)

George Lopez says he's NOT racist for only using white caddies when he golfs -- telling TMZ Sports, "How is it racist if it's a fact?"

Just last week, Lopez -- a huge golf fanatic -- told us when it comes to selecting caddies, "I use the whitest guy I can find."

Most people found it funny, some people called it racist.

So, when we got Lopez out at LAX Thursday -- he basically told his critics to stop being so damn sensitive.

Then he took a shot at Donald Trump and Trump's ancestors.