I have did many website scrapping work, I have worked on Google Scholar, Supar Pages,Yelp and many ... Web scrapping an art to get public data in our system , I user php curl and python to do it. for this curl is mainly used to develop scrapping scripts. Its time consuming process but i do with cron jobs that makes system easier to get data.