Personal Bytes

While AJAX strives through the Web, Service Oriented Architectures (SOA) pave their way onto the Enterprise. One of the best things SOA has brought is interoperability -- no more proprietary protocols and freakish RPC (this comes with one disadvantage though -- the cost of parsing XML).

The key point I'd like to introduce is that we are entering an era of cooperation rather than competition, at least to what concerns to software development. How?

blogaqui.com has a crawler to fetch all the registered blog's feeds and add them to a local database. This allows several things on blogaqui.com being the most popular one the planets. The crawler was initially built in Ruby. Here's a very brief overview.

It served its purpose but it didn't prime for efficiency. A quick profiling revealed that 80% of the CPU power was being taken by the XML parsing. That XML parsing was handled by REXML, a Ruby XML library. libxml addresses this issue being way faster than REXML. Unfortunately I was using a feed parsing library (do not reinvent the wheel) so there were two options:

• Change all the feed parsing library to make use of libxml instead of REXML or build an Adapter to convert REXML calls to libxml;
• Use a faster feed parsing library

Although there are some other options for parsing feeds, none were as complete as the one I was using (FeedTools by the way) and the majority also used REXML to handle the parsing. Then I stumbled upon myself thinking "Why should I insist in doing the crawler in Ruby? It's an offline operation anyways which does not interfere with the main Rails application". Light shined and then there was Python.

Python has a great XML parsing library and an also great feed parser library. It's slick as Ruby and.. well, it has everything I could praise for. The script took like 10 minutes to write and 20 for fine-tuning it (wasted figuring out which would be the most common exceptions and handle them).

Let the era of the cooperation and interoperability commence!

If you're now wondering why I talked about SOA (this had nothing to do with SOA maybe except for the fact of also parsing XML) it was simply for introducing the cooperation concept.