Speaking at the premiere of One Direction: This Is Us, O'Leary said: "If Simon [Cowell] comes back, the dynamic would be very different. Gary wouldn't be there - he'll be leaving at the end of this year."
Dermot also said he would like to join the panel. "I'd love to judge," he said. "I'd be so strict. I'd be nice and strict."
There has been no official word from Gary Barlow on his future plans concerning The X Factor.
