Download as PDF, PPTX
![Xpath selectors
Expression Meaning
name matches all nodes on the current level with
the specified name
name[n] matches the nth element on the current level
with the specified name
/ Do selection from the root
// Do selection from current node
* matches all nodes on the current level
. Or .. Select current / parent node
@name the attribute with the specified name
[@key='value'] all elements with an attribute that matches
the specified key/value pair
name[@key='value'] all elements with the specified name and an
attribute that matches the specified key/value
pair
[text()='value'] all elements with the specified text
name[text()='value'] all elements with the specified name and text](https://image.slidesharecdn.com/pydata-160409224139/85/Pydata-Python-tools-for-webscraping-20-320.jpg)
![Webscraping
pip install webscraping
#Download instance
D = download.Download()
#get page
html =
D.get('http://pydata.org/madrid2016/schedule/')
#get element where is located information
xpath.search(html, '//td[@class="slot slot-talk"]')](https://image.slidesharecdn.com/pydata-160409224139/85/Pydata-Python-tools-for-webscraping-26-320.jpg)
![Scrapy Shell
scrapy shell <url>
from scrapy.select import Selector
hxs = Selector(response)
Info = hxs.select(‘//div[@class=“slot-inner”]’)](https://image.slidesharecdn.com/pydata-160409224139/85/Pydata-Python-tools-for-webscraping-36-320.jpg)
![Scrapy Cloud /scrapy.cfg
# Project: demo
[deploy]
url =https://dash.scrapinghub.com/api/scrapyd/
#API_KEY
username = ec6334d7375845fdb876c1d10b2b1622
password =
#project identifier
project = 25767](https://image.slidesharecdn.com/pydata-160409224139/85/Pydata-Python-tools-for-webscraping-51-320.jpg)
![Xpath selectors
Expression Meaning
name matches all nodes on the current level with
the specified name
name[n] matches the nth element on the current level
with the specified name
/ Do selection from the root
// Do selection from current node
* matches all nodes on the current level
. Or .. Select current / parent node
@name the attribute with the specified name
[@key='value'] all elements with an attribute that matches
the specified key/value pair
name[@key='value'] all elements with the specified name and an
attribute that matches the specified key/value
pair
[text()='value'] all elements with the specified text
name[text()='value'] all elements with the specified name and text](https://image.slidesharecdn.com/pydata-160409224139/75/Pydata-Python-tools-for-webscraping-20-2048.jpg)
![Webscraping
pip install webscraping
#Download instance
D = download.Download()
#get page
html =
D.get('http://pydata.org/madrid2016/schedule/')
#get element where is located information
xpath.search(html, '//td[@class="slot slot-talk"]')](https://image.slidesharecdn.com/pydata-160409224139/75/Pydata-Python-tools-for-webscraping-26-2048.jpg)
![Scrapy Shell
scrapy shell <url>
from scrapy.select import Selector
hxs = Selector(response)
Info = hxs.select(‘//div[@class=“slot-inner”]’)](https://image.slidesharecdn.com/pydata-160409224139/75/Pydata-Python-tools-for-webscraping-36-2048.jpg)
![Scrapy Cloud /scrapy.cfg
# Project: demo
[deploy]
url =https://dash.scrapinghub.com/api/scrapyd/
#API_KEY
username = ec6334d7375845fdb876c1d10b2b1622
password =
#project identifier
project = 25767](https://image.slidesharecdn.com/pydata-160409224139/75/Pydata-Python-tools-for-webscraping-51-2048.jpg)
Uploaded byJose Manuel Ortega Candel
Pydata-Python tools for webscraping
Python tools for webscraping provides an overview of scraping techniques like screen scraping, report mining, and web scraping using spiders and crawlers. It then demonstrates various Python libraries for web scraping including Selenium, Requests, Beautiful Soup, PyQuery, Scrapy, and Scrapy Cloud. The document shows how to scrape data from websites using these tools and techniques.
More Related Content
Similar to Pydata-Python tools for webscraping
![[DSC Europe 24] Domagoj Maric - Modern Web Data Extraction: Techniques, Tools...](https://cdn.slidesharecdn.com/ss_thumbnails/domagojmaric-modernwebdataextractionfinal-250218225444-c7bcad20-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Pydata-Python tools for webscraping
- 1. Python tools for webscraping JoséManuel Ortega @jmortegac
- 2.
- 3.
- 4. Agenda Scraping techniques Introduction towebscraping Python tools for webscraping Scrapy project
- 5. Scraping techniques Screenscraping Report mining Web scraping Spiders /Crawlers
- 6. Screen scraping Selenium Mechanize Robobrowser
- 7. Selenium Open Sourceframework for automating browsers Python-Module http://pypi.python.org/pypi/selenium pip install selenium Firefox-Driver
- 8. Selenium find_element_ by_link_text(‘text’): findthe link by text by_css_selector: just like with lxml css by_tag_name: ‘a’ for the first link or all links by_xpath: practice xpath regex by_class_name: CSS related, but this finds all different types that have the same class
- 9.
- 10.
- 11.
- 12.
- 13. Python tools Requests Beautiful Soup 4 Pyquery Webscraping Scrapy
- 14. Spiders /crawlers AWeb crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing. A Web crawler may also be called a Web spider. https://en.wikipedia.org/wiki/Web_crawler
- 15.
- 16.
- 17.
- 18.
- 19. Web scraping withPython 1. Download webpage with requests 2. Parse the page with BeautifulSoup/lxml 3. Select elements with Regular expressions,XPath or css selectors
- 20. Xpath selectors Expression Meaning namematches all nodes on the current level with the specified name name[n] matches the nth element on the current level with the specified name / Do selection from the root // Do selection from current node * matches all nodes on the current level . Or .. Select current / parent node @name the attribute with the specified name [@key='value'] all elements with an attribute that matches the specified key/value pair name[@key='value'] all elements with the specified name and an attribute that matches the specified key/value pair [text()='value'] all elements with the specified text name[text()='value'] all elements with the specified name and text
- 21. BeautifulSoup Parsers supportlxml,html5lib Installation pip install lxml pip install html5lib pip install beautifulsoup4 http://www.crummy.com/software/BeautifulSoup
- 22. BeautifulSoup soup =BeautifulSoup(html_doc,’lxml’) Print all: print(soup.prettify()) Print text: print(soup.get_text()) from bs4 import BeautifulSoup
- 23. BeautifulSoup functions find_all(‘a’)Returnsall links find(‘title’)Returns the first element <title> get(‘href’)Returns the attribute href value (element).text Returns the text inside an element for link in soup.find_all('a'): print(link.get('href'))
- 24.
- 25.
- 26. Webscraping pip install webscraping #Downloadinstance D = download.Download() #get page html = D.get('http://pydata.org/madrid2016/schedule/') #get element where is located information xpath.search(html, '//td[@class="slot slot-talk"]')
- 27. Pydata agenda codestructure
- 28. Extract data frompydata agenda
- 29.
- 32.
- 33. Scrapy Uses a mechanismbased on XPath expressions called Xpath Selectors. Uses Parser LXML to find elements Twisted for asyncronous operations
- 34. Scrapy advantages Fasterthan mechanize because it uses asynchronous operations (Twisted). Scrapy has better support for html parsing. Scrapy has better support for unicode characters, redirections, gzipped responses, encodings. You can export the extracted data directly to JSON,XML and CSV.
- 35.
- 36. Scrapy Shell scrapy shell<url> from scrapy.select import Selector hxs = Selector(response) Info = hxs.select(‘//div[@class=“slot-inner”]’)
- 37. Scrapy Shell scrapy shellhttp://scrapy.org
- 38. Scrapy project $ scrapystartproject <project_name> scrapy.cfg: the project configuration file. tutorial/:the project’s python module. items.py: the project’s items file. pipelines.py : the project’s pipelines file. setting.py : the project’s setting file. spiders/ : spiders directory.
- 39.
- 40. Spider generating $ scrapygenspider -t basic <SPIDER_NAME> <DOMAIN> $ scrapy list Spiders list
- 41.
- 42.
- 43. Pipelines ITEM_PIPELINES = {'pydataSchedule.pipelines.PyDataSQLitePipeline':100, 'pydataSchedule.pipelines.PyDataJSONPipeline':200,} pipelines.py
- 44.
- 45. Execution $ scrapy crawl<spider_name> $ scrapy crawl <spider_name> -o items.json -t json $ scrapy crawl <spider_name> -o items.csv -t csv $ scrapy crawl <spider_name> -o items.xml -t xml
- 46.
- 47.
- 48.
- 49. Launch spiders withoutscrapy command
- 50. Scrapy Cloud http://doc.scrapinghub.com/scrapy-cloud.html https://dash.scrapinghub.com >>pip installshub >>shub login >>Insert your ScrapingHub API Key:
- 51. Scrapy Cloud /scrapy.cfg #Project: demo [deploy] url =https://dash.scrapinghub.com/api/scrapyd/ #API_KEY username = ec6334d7375845fdb876c1d10b2b1622 password = #project identifier project = 25767
- 52. Scrapy Cloud $ shubdeploy
- 53.
- 54.
- 55.
- 56. Scrapy Cloud Scheduling curl-u APIKEY: https://dash.scrapinghub.com/api/schedule.json -d project=PROJECT -d spider=SPIDER
- 57. References http://www.crummy.com/software/BeautifulSoup http://scrapy.org https://pypi.python.org/pypi/mechanize http://docs.webscraping.com http://docs.python-requests.org/en/latest http://selenium-python.readthedocs.org/index.html https://github.com/REMitchell/python-scraping
- 58.
- 59.