Post #3 above leads me to guess that the OP is using iTunes (probably installed via
Retroactive) on a Mac which doesn't officially support iTunes. The supported method of syncing iPods and iPhones on macOS Catalina and newer is via Finder, and the mechanism behind Finder's support of devices can lead to conflicts with iTunes. When this happens, one thing to try is to open a terminal window and kill
AMPDevicesAgent:
If you're going to use iTunes exclusively, a more permanent solution is to kill AMPDeviceDiscoveryAgent at login time,
as described here.
My guess as to the the OP's situation could be wrong. On Macs which actually shipped with iTunes, there have been reports of the "open in another window" message and the suggested workaround is to click the Store tab in iTunes, and then click back to the Library tab. I've never experienced the problem on a supported OS, so I can't say if that works or not, but it's easy enough to try.