As an App developer, can they write an app that will access your contacts? As a partner who unlocks your phone? Or someone who’ll logs onto your iCloud account using a browser?
Any app that wants to access your contact list has to ask for your permission (once) to do so. If you decline, then it can't access it. A typical legitimate use would be a 3rd party email app.