In general, you should have people assigned to One Project for large blocks of time, you should group your bugs by category so a bunch can be fixed at once without a lot of re-setup time, and you shouldn't constantly change your mind about which task has higher priority over which other task.
But we do this all the time; "We need combat fixes to show on the next rev!" "We need those memory optimizations for the next burn!" "We need, we need, we need..."
So is it really as bad as all that?
Joel Spolsky puts forth a situation where a guy is rotating between Task A, Task B, and Task C. What usually happens in the real world is actually a stack: guy works on Task A, the higher priority Task B supersedes that, and the higher priority Task C supersedes that. Task C ends up done ASAP, Task B less so, Task A less-less-so. You might get some wanker asking "Joe Coder started Task A two months ago and it's still not finished!" at which point you point out it was that or Task C and they shut up.
So, in short: no, it's not really always as bad as all that.
Still, there's something to be said for not switching to a higher priority task if your current task is close to being finished. In the long run that'll pay off.