Download as PDF, PPTX
![Python
• Entry
if __name__ == '__main__':
# do something
• Method
def main ():
# do something
• Package
import [package]
import [package] as [alias]
• Format
'%s' % ([parameters ...])
4
demo01.py
demo02.py
demo03.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-4-320.jpg)
![Python
• If … Else …
if [condition]:
# do something
else:
# do something
• For Loop
for item in list:
# do something
• Array Slice
array[start:end]
5](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-5-320.jpg)
![Python
• Array Creation From For Loop
– Object
[item.attr for item in array]
– Dictionary
[item[key] for item in array]
6
demo04.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-6-320.jpg)
![Built-in
• Regular Expression
– Expressions
1. Range
[Start-End]
[0-9], [a-z], [A-Z], [a-zA-Z], [0-9a-zA-Z], ...
20
demo12.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-20-320.jpg)
![Built-in
• Regular Expression
– Expressions
1. Numbers
d = [0-9]
2. Words
w = [a-zA-Z0-9] (ANSI)
w = [a-zA-Z0-9] + Non-ANSI Characters (UTF-8)
3. Spaces, Tabs, …
s
22
demo12.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-22-320.jpg)
![Packages
• Requests
– Response
1. Status Code
response.status_code
2. Headers
response.headers[<str:name>]
3. Cookies
response.cookies[<str:name>]
35
demo14.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-35-320.jpg)
![Packages
• PyQuery
– Find
p = d(<str:expression>)
– Element To HTML
p.html()
– Extract Text From Element
p.text()
– Get Value From Element Attribute
val = p.attr[<str:name>]
38
demo15.py](https://image.slidesharecdn.com/crawler-180810205620/85/Python-Crawler-38-320.jpg)
![Python
• Entry
if __name__ == '__main__':
# do something
• Method
def main ():
# do something
• Package
import [package]
import [package] as [alias]
• Format
'%s' % ([parameters ...])
4
demo01.py
demo02.py
demo03.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-4-2048.jpg)
![Python
• If … Else …
if [condition]:
# do something
else:
# do something
• For Loop
for item in list:
# do something
• Array Slice
array[start:end]
5](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-5-2048.jpg)
![Python
• Array Creation From For Loop
– Object
[item.attr for item in array]
– Dictionary
[item[key] for item in array]
6
demo04.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-6-2048.jpg)
![Built-in
• Regular Expression
– Expressions
1. Range
[Start-End]
[0-9], [a-z], [A-Z], [a-zA-Z], [0-9a-zA-Z], ...
20
demo12.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-20-2048.jpg)
![Built-in
• Regular Expression
– Expressions
1. Numbers
d = [0-9]
2. Words
w = [a-zA-Z0-9] (ANSI)
w = [a-zA-Z0-9] + Non-ANSI Characters (UTF-8)
3. Spaces, Tabs, …
s
22
demo12.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-22-2048.jpg)
![Packages
• Requests
– Response
1. Status Code
response.status_code
2. Headers
response.headers[<str:name>]
3. Cookies
response.cookies[<str:name>]
35
demo14.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-35-2048.jpg)
![Packages
• PyQuery
– Find
p = d(<str:expression>)
– Element To HTML
p.html()
– Extract Text From Element
p.text()
– Get Value From Element Attribute
val = p.attr[<str:name>]
38
demo15.py](https://image.slidesharecdn.com/crawler-180810205620/75/Python-Crawler-38-2048.jpg)
Uploaded byCheng-Yi Yu
Python Crawler
This document provides an overview of Python web crawling from beginner to intermediate level. It introduces Python basics like conditional statements, loops, arrays and dictionaries. It also covers built-in modules for handling JSON, XML, URLs, regular expressions. Popular web crawling packages like Requests, PyQuery, Beautiful Soup are demonstrated. Selenium for controlling browsers is explained. The document contains code examples and suggests hands-on workshops on scraping news sites and social media pages.
More Related Content
Python Crawler
- 1. PYTHON CRAWLER From Beginner ToIntermediate 1
- 2. Self Introduction Cheng-Yi, Yu erinus.startup@gmail.com •LifePlus Inc. Technical Manager • Paganini Plus Inc. Senior Software Developer • Freelancer ~ 10 Years 2
- 3.
- 4. Python • Entry if __name__== '__main__': # do something • Method def main (): # do something • Package import [package] import [package] as [alias] • Format '%s' % ([parameters ...]) 4 demo01.py demo02.py demo03.py
- 5. Python • If …Else … if [condition]: # do something else: # do something • For Loop for item in list: # do something • Array Slice array[start:end] 5
- 6. Python • Array CreationFrom For Loop – Object [item.attr for item in array] – Dictionary [item[key] for item in array] 6 demo04.py
- 7. Python • In – String if<str> in <str>: – Array if item in list: – Dictionary if <str:key> in <dict>: 7
- 8. Installation • Ubuntu Fonts https://design.ubuntu.com/font/ •Source Han Sans Fonts https://github.com/adobe-fonts/source-han-sans • Visual Studio Code And Extensions https://code.visualstudio.com/ 8
- 9. Installation • Cmder http://cmder.net/ • Python3.6 https://www.python.org/ • Python Packages pip install requests pip install pyquery pip install beautifulsoup4 pip install js2py pip install selenium 9
- 10.
- 11. Visual Studio Code •Install Python Extensions 11
- 12. Visual Studio Code •Open Integrated Terminal 12
- 13. Visual Studio Code •Open Integrated Terminal 13
- 14. Built-in • Json import json –json.loads json.loads(<str>) – json.dumps json.dumps(<dict>) 14 demo05.py demo06.py
- 15. Built-in • Xml import xml.etree.ElementTreeas ET – Load From File tree = ET.ElementTree(file=<str:filepath>) tree = ET.parse(<str:filepath>) root = tree.getroot() – Load From String root = ET.fromstring(<str>) 15 demo07.py demo08.py
- 16. Built-in • Xml – ChildNodes Only One Level for node in root: # do something – XPath Multiple Levels nodes = root.findall(<str:expression>) for node in nodes: # do something 16 demo07.py demo08.py
- 17. Built-in • Url import urllib.parseas UP – urlparse result = UP.urlparse(<str:url>) – urlunparse url = UP.urlunparse(<ParseResult>) – quote str = UP.quote(<str:unquoted>) – unquote str = UP.unquote(<str:quoted>) 17 demo09.py demo10.py
- 18. Built-in • Regular Expression importre – re.search Find First Match match = re.search(<str:pattern>, <str:text>) match.group(<int:index>) – re.findall Find All Matches finds = re.findall(<str:pattern>, <str:text>) for find in finds: # do something 18 demo11.py
- 19. Built-in • Regular Expression –re.split Split By Pattern re.split(<str:pattern>, <str:text>) – re.sub Replace By Pattern re.sub(<str:pattern>, <str:replace>, <str:text>) 19 demo11.py
- 20. Built-in • Regular Expression –Expressions 1. Range [Start-End] [0-9], [a-z], [A-Z], [a-zA-Z], [0-9a-zA-Z], ... 20 demo12.py
- 21. Built-in • Regular Expression –Expressions 1. Zero Or More Times * 2. One Or More Times + 3. Zero Or One Time ? 21 demo12.py
- 22. Built-in • Regular Expression –Expressions 1. Numbers d = [0-9] 2. Words w = [a-zA-Z0-9] (ANSI) w = [a-zA-Z0-9] + Non-ANSI Characters (UTF-8) 3. Spaces, Tabs, … s 22 demo12.py
- 23. Built-in • Regular Expression –Expressions 1. Start With ^ 2. End With $ 23 demo12.py
- 24. Built-in • Regular Expression –Expressions 1. Named Group (?P<name>expr) (?P<country>+d+)-(?P<phone>d+) 24 demo13.py
- 25. AnalySIS • Chrome DeveloperTools – Elements See Elements In DOM Id, Class, Attribute, ... – Network See Requests, Responses Urls, Methods, Headers, Cookies, Bodies, ... 25
- 26. Elements • Find Elementby Mouse Pointer 26
- 27. Elements • Find Elementby HTML Tag 27
- 28. Networks • See AllRequests And Responses 28
- 29. Networks • See DetailsOf Request And Response 29
- 30. Networks • See ResponseContent 30
- 31. Networks • See CookiesSent And Set 31
- 32. Documents • Requests http://docs.python-requests.org/ • PyQuery https://pythonhosted.org/pyquery/ •Beautiful Soup 4 https://www.crummy.com/software/BeautifulSoup/bs4/doc/ • Js2Py https://github.com/PiotrDabkowski/Js2Py 32
- 33. Packages • Requests import requests –Request 1. Method GET, POST, ... response = requests.get(<str:url>) response = requests.post(<str:url>, data=<str:body>) response = requests.post(<str:url>, data=<dict:body>) 2. Session session = requests.Session() response = session.get(<str:url>) 33 demo14.py
- 34. Packages • Requests import requests –Request • Headers response = requests.get(<str:url>, headers=<dict>) • Cookies response = requests.get(<str:url>, cookies=<dict>) 34
- 35. Packages • Requests – Response 1.Status Code response.status_code 2. Headers response.headers[<str:name>] 3. Cookies response.cookies[<str:name>] 35 demo14.py
- 36. Packages • Requests – Response 1.Binary Content response.content 2. Text Content response.text 3. Json Content response.json() 36 demo14.py
- 37. Packages • PyQuery import pyquery –Load From String d = pyquery.PyQuery(<str:html>) – Load From Url d = pyquery.PyQuery(url=<str:url>) – Load From File d = pyquery.PyQuery(filename=<str:filepath>) 37 demo15.py
- 38. Packages • PyQuery – Find p= d(<str:expression>) – Element To HTML p.html() – Extract Text From Element p.text() – Get Value From Element Attribute val = p.attr[<str:name>] 38 demo15.py
- 39. Packages • Beautiful Soup4 import bs4 – Load From String d = bs4.BeautifulSoup(<str:html>, 'html.parser') 39 demo16.py
- 40. Packages • Beautiful Soup4 – Find p = d.find_all(<str:tag>, <attr-key>=<attr-val>, ...) p = d.find_all(<regex>, <attr-key>=<attr-val>, ...) p = d.find_all(<array>, <attr-key>=<attr-val>, ...) p = d.find(<str:tag>, <attr-key>=<attr-val>, ...) p = d.find(<regex>, <attr-key>=<attr-val>, ...) p = d.find(<array>, <attr-key>=<attr-val>, ...) p = d.select(<str:expression>) p = d.select_one(<str:expression>) 40 demo16.py demo17.py
- 41. Packages • Beautiful Soup4 – Extract Text From Element p.get_text() – Get Value From Element Attribute p.get(<str:name>) 41 demo16.py demo17.py
- 42. Packages • Js2Py import js2py –Eval js2py.eval_js(<str:code>) res = js2py.eval_js('var o = <str:js>; o') 42 demo18.py
- 43.
- 44. WORKSHOP • Apple Daily https://tw.appledaily.com/ –Realtime News https://tw.appledaily.com/new/realtime 44
- 45. WORKSHOP • Facebook Page –Cookies – Feed 45
- 46.
- 47.
- 48. SELENIUM • Import import selenium.webdriver •Initialize option = elenium.webdriver.ChromeOptions() • Start driver = selenium.webdriver.Chrome(chrome_options=option) • Browse driver.get(<str:url>) • Close driver.quit() 48
- 49.
- 50. SELENIUM • Find One –find_element_by_id – find_element_by_name – find_element_by_tag_name – find_element_by_class_name – find_element_by_css_selector 50
- 51. SELENIUM • Find Multiple –find_elements_by_name – find_elements_by_tag_name – find_elements_by_class_name – find_elements_by_css_selector 51
- 52. SELENIUM • Actions – send_keys –click 52
- 53. WORKSHOP • Facebook Page –Login – Feed 53
- 54.