Downloaded 67 times
![Referer [sic]](https://image.slidesharecdn.com/webscraping-090512152244-phpapp02/85/Web-Scraping-with-PHP-23-320.jpg)
![SimpleXML Extension $sxe = new SimpleXMLElement( $markupString ); $sxe = new SimpleXMLElement( $filePath , null, true); echo $sxe ->body->ul->li[0], PHP_EOL; $children = $sxe ->body->ul->li; $children = $sxe ->body->ul->children(); foreach ( $children as $li ) { echo $li , PHP_EOL; } echo $sxe ->body->ul[ 'id' ]; $attributes = $sxe ->body->ul->attributes(); foreach ( $attributes as $name => $value ) { echo $name , '=' , $value , PHP_EOL; }](https://image.slidesharecdn.com/webscraping-090512152244-phpapp02/85/Web-Scraping-with-PHP-38-320.jpg)
![Referer [sic]](https://image.slidesharecdn.com/webscraping-090512152244-phpapp02/75/Web-Scraping-with-PHP-23-2048.jpg)
![SimpleXML Extension $sxe = new SimpleXMLElement( $markupString ); $sxe = new SimpleXMLElement( $filePath , null, true); echo $sxe ->body->ul->li[0], PHP_EOL; $children = $sxe ->body->ul->li; $children = $sxe ->body->ul->children(); foreach ( $children as $li ) { echo $li , PHP_EOL; } echo $sxe ->body->ul[ 'id' ]; $attributes = $sxe ->body->ul->attributes(); foreach ( $attributes as $name => $value ) { echo $name , '=' , $value , PHP_EOL; }](https://image.slidesharecdn.com/webscraping-090512152244-phpapp02/75/Web-Scraping-with-PHP-38-2048.jpg)
Uploaded byMatthew Turland
Web Scraping with PHP
The document discusses web scraping, including: 1. The basics of web scraping including how it differs from normal web browsing and why it is useful for tasks like data integration. 2. The techniques involved in web scraping including making HTTP requests, parsing responses, and analyzing retrieved data. 3. The libraries and tools available in PHP for performing web scraping tasks like DOM parsing, CSS selection, and regular expressions. 4. Best practices for web scraping such as approximating human behavior, batching requests, and thoroughly testing scrapers.
More Related Content
Web Scraping with PHP
- 1. Web Scraping with Matthew Turland Acadiana Open Source Group April 30, 2009
- 2.
- 3.
- 4. Difference #1: ImmediateAudience
- 5.
- 6. Why Is ItUseful?
- 7. Data Without WebServices
- 8.
- 9.
- 10. With plain text,we give ourselves the ability to manipulate knowledge, both manually and programmatically, using virtually every tool at our disposal. 3.14 The Power of Plain Text, The Pragmatic Programmer
- 11.
- 12. Potential Lack ofStability
- 13.
- 14.
- 15. No Nice NeatData Package
- 16.
- 17.
- 18. The Web WeWeave GET / HTTP/1.1 User-Agent: ... HTTP/1.1 200 OK Content-Type: ...
- 19. GET /index.php?foo=bar HTTP/1.1 <a href= "/index.php?foo=bar" > Index </a> <form method= "post" action= "/index.php" > <input name= "foo" value= "bar" /> </form> POST /index.php HTTP/1.1 foo = bar Browsing -> Requests
- 20. HTTP/1.1 200 OKContent-Type : image/gif Content-Length: 8558 Responses -> Rendered Elements <img src= "/intl/en_ALL/images/logo.gif" /> GET /intl/en_ALL/images/logo.gif HTTP/1.1 Host: google.com
- 21. Not As EasyAs It Looks
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29. PHP: Glue forthe Web
- 30. HTTP Client LibrariesPEAR::HTTP_Client pecl_http Zend_Http_Client Streams , cURL
- 31. Simple Streams Example$uri = 'http://www.example.com/some/resource' ; $get = file_get_contents( $uri ); $context = stream_context_create( array ( 'http' => array ( 'method' => 'POST' , 'header' => 'Content-Type: ' . 'application/x-www-form-urlencoded' , 'content' => http_build_query( array ( 'var1' => 'value1' , 'var2' => 'value2' )) ) ) ); $post = file_get_contents( $uri , false, $context );
- 32. pecl_http Example $http = new HttpRequest( $uri ); $http ->enableCookies(); $http ->setMethod(HTTP_METH_POST); $http ->addPostFields( array ( 'var1' => 'value1' )); $http ->setOptions( 'useragent' => 'PHP ' . phpversion (), 'referer' => 'http://example.com/some/referer' )); $response = $http -> send (); $headers = $response ->getHeaders(); $body = $response ->getBody();
- 33. pecl_http Request Pooling$pool = new HttpRequestPool; foreach ( $urls as $url ) { $request = new HttpRequest( $url , HTTP_METH_GET); $pool ->attach( $request ); } $pool -> send (); foreach ( $pool as $request ) { echo $request ->getUrl(), PHP_EOL; echo $request ->getResponseBody(), PHP_EOL; }
- 34. HTTP Resources RFC2616 HyperText Transfer Protocol RFC 3986 Uniform Resource Identifiers "HTTP: The Definitive Guide" (ISBN 1565925092) "HTTP Pocket Reference: HyperText Transfer Protocol" (ISBN 1565928628) "HTTP Developer's Handbook" (ISBN 0672324547) by Chris Shiflett Ben Ramsey's blog series on HTTP
- 35.
- 36. Tidy Extension $config = array ( 'output-xhtml' => true); $tidy = tidy_parse_string( $markupString , $config ); $tidy = tidy_parse_file( $markupFilePath , $config ); $output = tidy_get_output( $tidy );
- 37. DOM Extension $doc = new DOMDocument; $doc ->loadHTML( $htmlString ); $doc ->loadHTMLFile( $htmlFilePath ); $listItems = $doc ->getElementsByTagName( 'li' ); $xpath = new DOMXPath( $doc ); $listItems = $xpath ->query( '//ul/li' ); foreach ( $listItems as $listItem ) { echo $listItem ->nodeValue, PHP_EOL; }
- 38. SimpleXML Extension $sxe = new SimpleXMLElement( $markupString ); $sxe = new SimpleXMLElement( $filePath , null, true); echo $sxe ->body->ul->li[0], PHP_EOL; $children = $sxe ->body->ul->li; $children = $sxe ->body->ul->children(); foreach ( $children as $li ) { echo $li , PHP_EOL; } echo $sxe ->body->ul[ 'id' ]; $attributes = $sxe ->body->ul->attributes(); foreach ( $attributes as $name => $value ) { echo $name , '=' , $value , PHP_EOL; }
- 39. XMLReader Extension $doc = XMLReader::xml( $xmlString ); $doc = XMLReader::open( $filePath ); while ( $doc -> read ()) { if ( $doc ->nodeType == XMLReader::ELEMENT) { var_dump ( $doc ->localName); var_dump ( $doc ->hasValue); var_dump ( $doc ->value); var_dump ( $doc ->hasAttributes); var_dump ( $doc ->getAttribute( 'id' )); } }
- 40. CSS Selector LibrariesphpQuery Simple HTML DOM Parser Zend_Dom_Query $doc1 = phpQuery::newDocumentFile( $markupFilePath ); $doc2 = phpQuery::newDocument( $markupString ); $listItems = pq( 'ul > li' ); // uses $doc2 $listItems = pq( 'ul > li' , $doc1 );
- 41.
- 42.
- 43.
- 44.
- 45. Batch Jobs, Non-PeakHours
- 46.
- 47.
- 48.
- 49.
- 50.