language agnostic - Methods for automated synonym detection -


i working on neural network based approach short document classification, , since corpuses working around ten words, standard statistical document classification methods of limited use. due fact attempting implement form of automated synonym detection matches provided in training. question more resolving situation follows:

say have classifications of "involving food", , 1 of "involving spheres" , data set follows:

"eating apples"(food);"eating marbles"(spheres); "eating oranges"(food, spheres); "throwing baseballs(spheres)";"throwing apples(food)";"throwing balls(spheres)"; "spinning apples"(food);"spinning baseballs";

i looking incremental method move towards following linkages:

eating --> food apples --> food marbles --> spheres oranges --> food, spheres throwing --> spheres baseballs --> spheres balls --> spheres spinning --> neutral involving --> neutral

i realize in specific case these might suspect matches, illustrates problems having. general thoughts if incremented word appearing opposite words in category, in case end incidentally linking word "involving", thought decrement word appearing in conjunction multiple synonyms, or non-synonyms, lose link between "eating" , "food". have clue how put algorithm move me in directions indicated above?

there unsupervized boot-strapping approach explained me this.

there different ways of applying approach, , variants, here's simplified version.

concept:

start assuming if 2 words synonyms, in corpus appear in similar settings. (eating grapes, eating sandwich, etc.)

(in variant use co-occurence setting).

boot-strapping algorithm:

we have 2 lists,

  • one list contain words co-occur food items
  • one list contain words food items

supervized part

start seeding 1 of lists, instance might write word apple on food items list.

now let computer take over.

unsupervized parts

it first find words in corpus appear before apple, , sort them in order of occuring.

take top 2 (or many want) , add them co-occur food items list. example, perhaps "eating" , "delicious" top two.

now use list find next 2 top food words ranking words appear right of each word in list.

continue process expanding each list until happy results.

once that's done

(you may need manually remove things lists go wrong.)

variants

this procedure can made quite effective if take account grammatical setting of keywords. 

subj ate nounphrase nounphrase are/is moldy workers harvested apples. subj verb apples might imply harvested important verb distinguishing foods. other occurrences of subj harvested nounphrase

you can expand process move words categories, instead of single category @ each step.

my source

this approach used in system developed @ university of utah few years successful @ compiling decent list of weapon words, victim words, , place words looking @ news articles.

an interesting approach, , had results.

not neural network approach, intriguing methodology.

edit:

the system @ university of utah called autoslog-ts, , short slide can seen here towards end of presentation. , link paper here


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more