Over the last six months we\’ve been working on a rewrite of our blog crawling infrastructure from the ground up. Sometimes it seems like blogs are like fingerprints, no two are alike. With a new architectural approach, we\’ve added a lot better coverage of many new blogging conventions, better visibility and tracking of what is happening in the internals of the crawler, and how to do a better job of recognizing all the great content bloggers are discussing and linking to.
The time has come to retire the original Technorati crawler. Most blogs from the larger platforms have been using the new crawler for a few months and we\’re now ready to convert the rest of the blogs in our index.
We\’ve been moving large portions of blogs during the past two weeks and still have a lot to go. The large scale shift has put some strain on our crawl queues and data management systems during this wave of updates. The big push has resulted in slower times to index than usual. We\’re working hard to keep the new posts coming in and expect the majority of the backlog to be sorted out during this next week. We appreciate your patience. I think you\’ll like the results when everything is converted.
Keep on blogging. The Crawler Development Team.
