"You would have been invited. I don't know about you meaning the world and the stars to me, but you're my closest friend that I've made while here." Leo's being honest. He's got no reason to be glib about this. Or sarcastic. Or ironic. Or any of the other things Dirk thinks he's being.