The biggest single issue facing internet TV is spikes.
A single server at a decent hosting facility can handle an extraordinary amount of traffic - around 200 simultaneous connections at 1Mbps. Say that the viewer watches for half an hour a day, that's 48 x 200, or almost 10,000 viewers a day. Fifty servers and you can handle half a million viewers a day, and only the top 1% of anything on the web achieves this kind of popularity.
But, the problem is that traffic isn't even. Most things that drive traffic to a website also cause spikes. For example, announcing a URL on a traditional broadcast channel will cause a huge immediate spike (showing how many people surf and view..).
There are a handful of companies - Akamai, Limelight, Mirror Image and Internap amongst them, who have made a virtue from this. P2P doesn't help much since the requests are unexpected and often relate to new content that hasn't been widely distributed.
Multicasting - the internet equivalent to broadcasting, where everyone sees the same stream is an option which may become more prevalent, but for the time being the only answer is over-capacity, which is expensive. Today, for example, I have struggled with Adobe and Radiotimes' websites, to the point where I gave up.
For internet TV services the problem is even more acute and the number of services I visit that simply don't work in a reliable way is becoming very annoying.
The valuations given to Limelight and Akamai (but not to mixed model providers such as Narrowstep) shows the value in this part of the market.
The net neutrality debate is likely to impact this in due course, but for the time being the clever money is with those that can handle the heat in spike city.