Quite normal for people to experience anxiety from coffee (or anything with caffeine in it). It's part of many peoples' genetic makeup ( CYP1A2 gene variations) and it's likely you're a slow metaboliser from the description of your side effects.
Again, not unusual at all. Many people have the same variant. You can even see the variations of that particular gene in the advanced sections of those DNA tests you see online.
Great suggestion earlier was to switch to decaf, though even decafs can have a bit of caffeine. If you've been relying on caffeine for years for its stimulant effects and want to stop that then there's good and bad news. Bad news is the heavier you've been relying on it, the worse quitting will likely feel. Most people have a window of around 7 to 9 days until the worst cold turkey effects wear off, and the first 4 days can be absolute hell for some people too.
Even after that period it can take weeks until your energy levels normalise and you can feel perky and alert without caffeine. I've seen a colleague who took months until he felt great energy wise after quitting.
I don't just say this from theory, my partner and I have both also been heavy caffeine drinkers and quit partially for the sake of experimenting, and partially as I'd have liked to quit before having a baby (going cold turkey while pregnant is stressful on your body and could possibly have some knock on effect). Our experiences really mimicked colleagues and bloggers, Youtubers etc that we checked out. It was hell. Genuinely rough. You hear about the headaches and eye strain, but the incredible lethargy that can hit it overwhelming. Not dissimilar to clinical depression.
Partner was worried being a cyclist, and an upper Cat one fitness wise, that it'd screw with his upper end power or TT'ing etc. He'd been using caffeine for performance for years, as pretty much any cyclist does. Interestingly it had no effect at all, though his evaluation of that and experience there is a huge tangent on top of this already sizeable tangent that wouldn't interest anyone except other cyclists. It does highlight though our reliance on caffeine. Was guilty of needing it for energy, alertness, for getting work done. And I had my excuses, I was doing 10 to 14 hour days at an investment bank for nearly 5 years. It was only after relocating for work that gave me a gap to try this. No way I'd have been able to go cold turkey while working, though tapering down may have been more apt instead, yet less of an experiment and less of a learning experience.
Anyhow, I still occasionally have a decaf when out, maybe once a fortnight. I drank coffee for nearly 20 years and love the smell and feeling of comfort. No need to give that part of it up. But in terms of energy and mental acuity it's a crutch I no longer need, but as I said earlier, it looks me weeks to shake off a lingering fatigue.