I have always found BTT very quirky and sometimes it works as I expect/hope and sometimes not. Never sure whether that is me or BTT! But I do keep using it. When I can, I prefer to use Keyboard Maestro (there is some overlap between them) because it is powerful and works as expected (when I can find the right actions!).
I tried your Quit action and all it does for me is to display the context menu, but not take the Quit action. So I can reproduce what you get. I tried other actions too. I suspect there is something about a Dock context menu which is different to an application context menu.
-------
But, you got me intrigued. The Trigger Context Menu Item action does show the Context Menu when I hover over an item in the Dock. Quit is always at the bottom, so add an uparrow and a return keystroke. Like this:
That does the quit when I hover over an app in the Dock.
I am so pleased with myself for working that out - thank you for the idea!
As a bonus, if you do the same trigger for an app which is closed (but permanently in the dock) it opens the app (because Open is at the bottom of the context menu).