Seems that the internet is about to crash, if you believe MS and Intel. Some of the woes stem from the very nature of HTTP request/response model, and is aggravated by the rise of RSS/Atom news aggregators. So, is a push model the solution?
Or should we just queue the requests. But if queuing, where does the queue lie, on the producer or consumer side?
The short answer for me would be to queue on the consumer, and if over HTTP, this can be accomplished with a 503 HTTP status code and an optional Retry-After header.
Consider the fact that what is making the request is a news reader/aggregator, and it is highly probable this tool just made a fetch only an hour or so ago. So asking it to come back later isn't nearly the faux pas as it would be to tell a web browser to try later. The human driving the browser will just forget to return, after cursing your inability to provide instant gratification.
Of course for this to work, it should be knowable that the cost of creating a TCP/HTTP connection must be significantly less than the cost of creating that connection and serving the RSS/Atom content.
Given the trend (as employed by MS) is to reduce the richness of the content, thus reducing its cost. This trend negatively impacts the usefulness of the RSS/Atom feed. And if the RSS content is just abridged/summarized, the reader will visit the site eventually and retrieve it anyways, if they can discern the usefulness of the content from the summary/excerpt.
Considering that, you are really queuing on the consumer side, but reducing the overall impact your site delivers by putting up a barrier. For example, I rarely read a blog that doesn't include the whole article in the feed. The extra hassle is rarely worth the message.
But as you reduce the cost of the RSS content by reducing it's richness, you approach the unavoidable cost of creating the TCP connection. Which this queue solution won't solve as you still need the connection to say 'come back later', assuming a time-out doesn't deliver the same message.
Regardless, the value of the 503 queue would be to spread out requests, that are naturally bursty, for content that is rich and useful.
And I would expect most aggregators are robust enough to not cause the user grief if it can't reach the server periodically. So I don't believe any code would be required in the client to provide a basic level of support.
Anyways, I do believe adding a push semantic to the web would be great and look forward to playing with it.
[update: learned to spell]
Leave a comment