Oh, conversation is inevitable. For the most part, only a prickly response or a clearly stated, 'I am busy; I must be on my way' (or something to that effect) would prompt him to leave so soon. "...Nice t'meetchya, Mr.Julian." He extends his hand after tugging off his snow-crusted glove.
Action