python - Scrapy retrieves text encoding incorrectly, hebrew as \u0d5 etc -


first time working stuff. checked out other sof questions internalization / text encoding.

i'm doing scrapy tutorial, when got stuck @ part: extracting data, when extract data, text instead of hebrew displayed series of \uxxxx.

it's possible check out scraping this page example;

scrapy shell http://israblog.nana10.co.il/blogread.asp?blog=167524&blogcode=13348970 hxs.select('//h2[@class="title"]/text()').extract()[0]

this retrieve

u'\u05de\u05d9 \u05d0\u05e0\u05e1 \u05e4\u05d5\u05d8\u05e0\u05e6\u05d9\u05d0\u05dc\u05d9?'

(unrelated:) if try print in console, get: traceback (most recent call last): file "<stdin>", line 1, in <module> file "c:\python27\lib\encodings\cp437.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_map) unicodeencodeerror: 'charmap' codec can't encode characters in position 0-1: cha racter maps <undefined>

tried setting encoding through settings, tried converting manually, feel tried everything.

(i've gone 5 pomodoros trying fix this!)

what can hebrew text should there: "מי אנס פוטנציאלי?"

(disclaimer: went first blog , post noticed on http://israblog.co.il, i'm in no way related blog or blog owner, used example)

what can hebrew text should there: "מי אנס פוטנציאלי?"

test.py:

# coding: utf-8 = u'\u05de\u05d9 \u05d0\u05e0\u05e1 \u05e4\u05d5\u05d8\u05e0\u05e6\u05d9\u05d0\u05dc\u05d9?' b = 'מי אנס פוטנציאלי?' print print b

result:

vic@wic:~/projects/snippets$ python test.py מי אנס פוטנציאלי? מי אנס פוטנציאלי? vic@wic:~/projects/snippets$

as see same. it's different representation of same unicode string. don't worry it's not scraped correctly.

if want save file:

python 2.7.3 (default, apr 20 2012, 22:39:59) [gcc 4.6.3] on linux2 >>> = u'\u05de\u05d9 \u05d0\u05e0\u05e1 \u05e4\u05d5\u05d8\u05e0\u05e6\u05d9\u05d0\u05dc\u05d9' >>> u'\u05de\u05d9 \u05d0\u05e0\u05e1 \u05e4\u05d5\u05d8\u05e0\u05e6\u05d9\u05d0\u05dc\u05d9' >>> f = open('test.txt', 'w') >>> f.write(a) traceback (most recent call last): file "<stdin>", line 1, in <module> unicodeencodeerror: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) >>> f.write(a.encode('utf-8')) >>> f.close()

Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more