ruby - String#encode not fixing "invalid byte sequence in UTF-8" error -


i know there multiple similar questions error, , i've tried many of them without luck. problem i'm having involves byte \xa1 , throwing

argumenterror: invalid byte sequence in utf-8

i've tried following no success:

"\xa1".encode('utf-8', :undef => :replace, :invalid => :replace, :replace => "").sub('', '') "\xa1".encode('utf-8', :undef => :replace, :invalid => :replace, :replace => "").force_encoding('utf-8').sub('', '') "\xa1".encode('utf-8', :undef => :replace, :invalid => :replace, :replace => "").encode('utf-8').sub('', '')

each line throws error me. doing wrong?

update:

the above lines fail in irb. however, modified application encode lines of cvs file using same string#encode method , arguments, , same error when reading line file (note: works if perform operations on same string w/o using io).

bad_line = "col1\tcol2\tbad\xa1" bad_line.sub('', '') # not fail puts bad_line # => col1 col2 bad? tmp = tempfile.new 'foo' # write line file emulate real problem tmp.puts bad_line tmp.close tmp2 = tempfile.new 'bar' begin io.foreach tmp.path |line| line.encode!('utf-8', :undef => :replace, :invalid => :replace, :replace => "") line.sub('', '') # fail: invalid byte sequence in utf-8 tmp2.puts line end tmp2.close # fail if above error didn't halt execution csv.foreach(tmp2.path) |row| puts row.inspect # fail: invalid byte sequence in utf-8 end ensure tmp.unlink tmp2.close tmp2.unlink end

it seem ruby thinks string encoding utf8, when do

line.encode!('utf-8', :undef => :replace, :invalid => :replace, :replace => "")

it doesn't because destination encoding same current encoding (at least that's interpretation of code in transcode.c)

the real question here whether starting data valid in encoding isn't utf-8 or whether data supposed utf-8 has few warts in want discard.

in first case, correct thing tell ruby encoding is. can when open file

file.open('somefile', 'r:iso-8859-1')

will open file, interpreting contents iso-8859-1

you can ruby transcode you

file.open('somefile', 'r:iso-8859-1:utf-8')

will open file iso-8859-1, when read data bytes converted utf-8 you.

you can call force_encoding tell ruby string's encoding (this doesn't modify bytes @ all, tells ruby how interpret them).

in second case, want dump whatever nasty stuff has got utf-8, can't call encode! have because that's no-op. in ruby 2.1 , higher, can use string#scrub, in previous versions can this

line.encode!('utf-16', :undef => :replace, :invalid => :replace, :replace => "") line.encode!('utf-8')

we first convert utf-16. since different encoding, ruby replace our invalid sequences. can convert utf-8. won't lose data because utf-8 , utf-16 2 different ways of encoding same underlying character set.


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more