X Factor to TMZ -- 'Play Nice'

Simon Cowell and Britney Spears want TMZ to know ... despite our stories about Britney having a tough time judging "X Factor" ... they couldn't be happier. Let's be real -- Britney says she's talking to Harvey in the video, but she's really talkin' to you.

As TMZ first reported, Britney walked off during shooting on Thursday ... leaving four contestants to sing to an empty chair. Yesterday Britney took one short break during taping, but was otherwise much more outgoing than on Day 1.

We starting poking around late yesterday because we heard rumblings that Britney might be having second thoughts, but she and Simon want to make it clear -- Britney is going nowhere.

We gotta ask ...