Some apps I don't bother killing, but some apps never re-initialize correctly unless you kill them. So I kill the apps with re-initialization issues. I also usually kill apps I use infrequently.
To be honest the only app that I absolutely want to kill every time is not used is Safari, the one and only memory hog on this iOS ....
Btw so far iOS 7.1 is really really better in memory management.