Speaking from some experience in developing a small tool myself, I've used YUI's rich control set with Prototype for DOM manipulation in the past and experienced no issues. Admittedly, this was a small tool that didn't use a wide array of the controls.
Even so, I'm always hesitant to use multiple frameworks on my web projects; however, if you're only using jQuery's DOM functionality and YUI's control functionality, then I think you're fine - there's not really a conflict of interest there. Plus, with jQuery's noConflict() mode and YUI's namespacing, the two frameworks really shouldn't trump one another.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…