Yeah, doesn't look like it can be done...unless perhaps clearing recents in the Phone app, the FaceTime app, and/or deleting all conversations in the Meaaages app might have some effect on it.
I wonder how recents work at all there. In my case the few contacts that showed up there seemed to be mainly from a few message threads I have, although some aren't that rexent not all that active or have much in them. Then suddenly just now another contact appeared out of nowhere without me doing anything related to contacts or phone or messaging and it's for someone I almost never interact with but did have a couple of calls about a couple of weeks ago. Seems like this might not be all that hashed out yet. Hopefully there will be a way to control or clear it.