multithreading - Application design for parallel collection processing -


i'm experimenting system.collections.concurrent namespace have problem implementing design.

  1. my input queue (concurrentqueue) getting populated fine thread doing i/o @ startup read , parse.
  2. next kick off parallel.foreach() on input queue. i'm doing i/o bound work on each item.
  3. a log item created each item processed in foreach() , dropped result queue.

what kick off logging start reading input because may not able fit of log items in memory. best way wait items land in result queue? there design patterns or examples should looking at?

i think pattern you're looking producer/consumer pattern. more specifically, can have producer/consumer implementation built around tpl , blockingcollection.

the main concepts want read are:

  1. task,
  2. blockingcollection,
  3. taskfactory.continuewhenall(will allow perform action when set of tasks/threads finished running).
  4. bounding , blocking in blockingcollection. allows set maximum size output collection (for memory reasons) , producer thread(s) wait consumers pick elements in case maximum size specify reached.
  5. blockingcollection.completeadding , blockingcollection.iscompleted can used synchronize producers , consumers (producer can when it's finished, consumer can check , keep running until producer(s) finised).

a more complete sample in second article linked.

in case think want consumer pick things result queue , dispose of them possible (write them logging store, or similar).

so final collection, dump log items should blockingcollection, not concurrentqueue.


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more