Video encoding is a finicky process. Especially with h.264 and its myriad of options. The full h.264 feature set is not supported by any appliance device. A quick google search shows a number of people having issues with PQ DVD and the Apple TV. It sounds like it just sometimes creates files that the ATV doesn't like (there are differences in the decoding for the iPod and ATV). That doesn't mean it's a bad app, and is more of a problem with h.264 (this is why I wish DivX was support by Apple. It just works). You should contact the vendor with your problem.
For the time being, there is a Windows version of Handbrake that you may have better luck using. You can use DVD43(free) or AnyDVD(retail) for dealing with the disc copy protection in real time.
This will work for ripping DVDs, but I'm not sure what to recommend for transcoding videos in windows... maybe ffmpeg.