While the Delta is external, it isn't really designed with ultimate sound in mind. They have to pack a lot of inputs and outputs in there for the money. The PSU could probably be improved as well.
It just won't compare to a high-end DAC. I don't even find my computer via an Edirol FA-101 optical out compares to my low-end Cambridge Audio transport with AES/EBU output, both into the same DAC.
Still, YMMV.
What are you comparing it to? It should hold up well to budget-end players up to £600 or so.
Edited to add, have you tried it against something from the likes of Edirol? I found Edirol had slightly better sound. More refined. Not really a lot in it though.