The problem is that now everyone who wants to build a decent crawler/scraper has to do it as well. Used to be any kid in his parents' basement could put wget and sed together and explore the internet. Now Timmy has to bolt a whole damn browser runtime onto his script to make sure he's getting the content text from blog posts and twitter feeds.

Asking Timmy to "step up his game" is heartless, and it'd be unnecessary if we were better citizens.

Maybe Timmy should just have a look at the Google specification! And he would see that he could continue to crawl the web with his favorite tool! He has just to replace #! with an _escaped_fragment_ query parameter, that's not so heartless ;)