Web Scraping using Python and BeautifulSoup

[mokrunka]

Hello folks. This topic doesn't fit perfectly anywhere, so I dumped it here.

Just curious. Have any of you guys ever scraped data from the web for something? I'm learning how to use BeautifulSoup (a python library that makes it easier to extract web data from html), and I'm trying to brainstorm fun ideas to try.

What have you guys done?

 

[seco]

But beautifulSoup is not capable of scraping JavaScript.
Selenium can do the job. This article shows some examples of how to scrape Ajax web pages
Check this link
Happy scraping

 

[Gimli]

Diffbot and phantomjs on github are also other option selenium however does seem to be the go to.

 

[seco]

Absolutely!