It probably said it was easier to have normal conversations when Stark wasn't tying himself into knots trying to make sure he didn't say something wrong.

i promise not to look better than you

He meant all of it. That was the problem. Or part of it, at least.