selenium - Defer parts of scrape in scrapy -


i have parse method given below, uses selenium first load page, visits pages cannot accessed scraping directly spider, collects individual urls parse method extracts items pages. problem is, parse method blocks other parsing untill pages visited. chokes system. tried adding sleep, stops engine alltogether, , not parse method.

any pointers how optimize this, or atleast make sleep work doesnt stop engine?

def parse(self, response): '''parse first page , extract page links''' item_link_xpath = "/html/body/form/div[@class='wrapper']//a[@title='view & apply']" pagination_xpath = "//div[@class='pagination']/input" page_xpath = pagination_xpath + "[@value=%d]" display = display(visible=0, size=(800, 600)) display.start() browser = webdriver.firefox() browser.get(response.url) log.msg('loaded search results', level=log.debug) page_no = 1 while true: log.msg('scraping page: %d'%page_no, level=log.debug) link in [item_link.get_attribute('href') item_link in browser.find_elements_by_xpath(item_link_xpath)]: yield request(link, callback=self.parse_item_page) page_no += 1 log.msg('using xpath: %s'%(page_xpath%page_no), level=log.debug) page_element = browser.find_element_by_xpath(page_xpath%page_no) if not page_element or page_no > settings['pagination_pages']: break page_element.click() if settings['pagination_sleep_interval']: seconds = int(settings['pagination_sleep_interval']) log.msg('sleeping %d'%seconds, level=log.debug) time.sleep(seconds) log.msg('scraped listing pages, closing browser.', level=log.debug) browser.close() display.stop()

this may help:

# delayspider.py scrapy.spider import basespider twisted.internet import reactor, defer scrapy.http import request delay = 5 # seconds class myspider(basespider): name = 'wikipedia' max_concurrent_requests = 1 start_urls = ['http://www.wikipedia.org'] def parse(self, response): nextreq = request('http://en.wikipedia.org') dfd = defer.deferred() reactor.calllater(delay, dfd.callback, nextreq) return dfd

output:

$ scrapy runspider delayspider.py 2012-05-24 11:01:54-0300 [scrapy] info: scrapy 0.15.1 started (bot: scrapybot) 2012-05-24 11:01:54-0300 [scrapy] debug: enabled extensions: logstats, telnetconsole, closespider, webservice, corestats, spiderstate 2012-05-24 11:01:54-0300 [scrapy] debug: enabled downloader middlewares: httpauthmiddleware, downloadtimeoutmiddleware, useragentmiddleware, retrymiddleware, defaultheadersmiddleware, redirectmiddleware, cookiesmiddleware, httpcompressionmiddleware, chunkedtransfermiddleware, downloaderstats 2012-05-24 11:01:54-0300 [scrapy] debug: enabled spider middlewares: httperrormiddleware, offsitemiddleware, referermiddleware, urllengthmiddleware, depthmiddleware 2012-05-24 11:01:54-0300 [scrapy] debug: enabled item pipelines: 2012-05-24 11:01:54-0300 [wikipedia] info: spider opened 2012-05-24 11:01:54-0300 [wikipedia] info: crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2012-05-24 11:01:54-0300 [scrapy] debug: telnet console listening on 0.0.0.0:6023 2012-05-24 11:01:54-0300 [scrapy] debug: web service listening on 0.0.0.0:6080 2012-05-24 11:01:56-0300 [wikipedia] debug: crawled (200) <get http://www.wikipedia.org> (referer: none) 2012-05-24 11:02:04-0300 [wikipedia] debug: redirecting (301) <get http://en.wikipedia.org/wiki/main_page> <get http://en.wikipedia.org> 2012-05-24 11:02:06-0300 [wikipedia] debug: crawled (200) <get http://en.wikipedia.org/wiki/main_page> (referer: http://www.wikipedia.org) 2012-05-24 11:02:11-0300 [wikipedia] info: closing spider (finished) 2012-05-24 11:02:11-0300 [wikipedia] info: dumping spider stats: {'downloader/request_bytes': 745, 'downloader/request_count': 3, 'downloader/request_method_count/get': 3, 'downloader/response_bytes': 29304, 'downloader/response_count': 3, 'downloader/response_status_count/200': 2, 'downloader/response_status_count/301': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2012, 5, 24, 14, 2, 11, 447498), 'request_depth_max': 2, 'scheduler/memory_enqueued': 3, 'start_time': datetime.datetime(2012, 5, 24, 14, 1, 54, 408882)} 2012-05-24 11:02:11-0300 [wikipedia] info: spider closed (finished) 2012-05-24 11:02:11-0300 [scrapy] info: dumping global stats: {}

it uses twisted's calllater sleep.


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more