I have worked with a web-crawling company for more than a year, we used our own custom solution for that which was written entirely on ruby. I have experience writing custom crawlers for clients to fetch a lot of data (thousands-millions of records) from various sources.