Scrapy - Revision history

Admin at 22:38, 3 August 2015

2015-08-03T22:38:11Z

2012-05-23T17:24:24Z

← Older revision		Revision as of 17:24, 23 May 2012
Line 294:		Line 294:
	tutorial/spiders/dmoz_spider.py		tutorial/spiders/dmoz_spider.py
	</pre>		</pre>



			[[Category:Programs]]
			[[Category:Python]]

2012-05-21T01:59:28Z

Simple Example: Radio Buttons

← Older revision		Revision as of 01:59, 21 May 2012
Line 133:		Line 133:
	'''The Form:'''		'''The Form:'''

	<~~source lang="html"~~>		<pre>

	<html>		<html>
Line 146:		Line 146:
	</html>		</html>

	</~~source~~>		</pre>

	And when this form is submitted, it results in the following page:		And when this form is submitted, it results in the following page:
Line 152:		Line 152:
	'''The Result Page:'''		'''The Result Page:'''

	<~~source lang="html"~~>		<pre>

	<p>Submit is set.</p>		<p>Submit is set.</p>
Line 158:		Line 158:
	<div id="scrape">100</div>		<div id="scrape">100</div>

	</~~source~~>		</pre>

	===Start the request===		===Start the request===

2012-05-21T01:57:19Z

Dealing with Forms

@@ Line 113: / Line 113: @@
 =Dealing with Forms=
-Scrapy has the capability to deal with HTML forms built into it, via the FormRequest object type.
+This section will illustrate the ability to use Scrapy spiders to deal with HTML forms.
-The spider can simulate someone submitting an HTML POST form, specifying key-value fields, by making the spider return a FormRequest object.
+The scenario will suppose that you want to set the variables in some form, submit the form, and receive a results page, which is then processed using Scrapy.
-A FormRequest object
+==Use FormRequest==
-* http://www.pixelmender.com/2010/10/12/scraping-data-using-scrapy-framework/
+The FormRequest object type will allow a Scrapy spider to deal with forms. It is available by adding this to the beginning of a spider's definition:
 <source lang="python">
 from scrapy.http import FormRequest
-from scrapy.http import Request
+</source>
-from scrapy.selector import HtmlXPathSelector
-from waliquor.items import BrandCategoryItem
+==Simple Example: Radio Buttons==
 </source>
-First, create a request:
+===Start the request===
 <source lang="python">
-        brandCategoriesRequest = Request("http://liq.wa.gov/homepageServices/brandsearch.asp",
+    def start_requests(self):
 </source>
 =Traversing Content=

2012-05-16T03:27:39Z

Dealing with Forms

← Older revision		Revision as of 03:27, 16 May 2012
Line 120:		Line 120:

	* http://www.pixelmender.com/2010/10/12/scraping-data-using-scrapy-framework/		* http://www.pixelmender.com/2010/10/12/scraping-data-using-scrapy-framework/

			<source lang="python">
			from scrapy.spider import BaseSpider
			from scrapy.http import FormRequest
			from scrapy.http import Request
			from scrapy.selector import HtmlXPathSelector
			from waliquor.items import BrandCategoryItem
			</source>

			First, create a request:

			<source lang="python">
			brandCategoriesRequest = Request("http://liq.wa.gov/homepageServices/brandsearch.asp",
			callback=self.parseBrandCategories)
			</source>

	=Traversing Content=		=Traversing Content=

2012-05-14T16:02:56Z

Dealing with Forms

← Older revision		Revision as of 16:02, 14 May 2012
Line 119:		Line 119:
	A FormRequest object		A FormRequest object

			* http://www.pixelmender.com/2010/10/12/scraping-data-using-scrapy-framework/

			=Traversing Content=

	=Errors=		=Errors=

2012-05-13T03:34:10Z

← Older revision		Revision as of 03:34, 13 May 2012
Line 109:		Line 109:

	==More Complex Parsing==		==More Complex Parsing==


			=Dealing with Forms=

			Scrapy has the capability to deal with HTML forms built into it, via the FormRequest object type.

			The spider can simulate someone submitting an HTML POST form, specifying key-value fields, by making the spider return a FormRequest object.

			A FormRequest object


	=Errors=		=Errors=

2012-05-12T23:28:31Z

Run Spider

2012-05-12T22:48:51Z

Create Spider

← Older revision		Revision as of 22:48, 12 May 2012
Line 50:		Line 50:
	A spider grabs the HTML content of a web page (using an HTTP request, same as a web browser), but instead of displaying that HTML as a web page (as a browser might), the spider processes the contents to extract information.		A spider grabs the HTML content of a web page (using an HTTP request, same as a web browser), but instead of displaying that HTML as a web page (as a browser might), the spider processes the contents to extract information.

	The spider ~~should~~ define:		The spider must define three things:
	* ~~a URL or~~ list of ~~URLS~~ to ~~crawl~~		* the spider's name (<code>name</code>)
	* ~~how~~ to parse ~~the contents of URLs~~		* list of URLs to scrape (<code>start_urls</code>)
			* what to do with the scraped material (<code>parse()</code>)

	The spider's behavior is defined in <code>tutorial/spiders/dmoz_spider.py</code>		The spider's behavior is defined in <code>tutorial/spiders/dmoz_spider.py</code>

2012-05-12T22:45:52Z

Create Project

← Older revision		Revision as of 22:45, 12 May 2012
Line 17:		Line 17:
	</source>		</source>

	=~~Create~~ Project=		=Tutorial Project=

	Following the instructions [http://doc.scrapy.org/en/latest/intro/tutorial.html#intro-tutorial here], create a new project:		Following the instructions [http://doc.scrapy.org/en/latest/intro/tutorial.html#intro-tutorial here], create a new project: