parallel processing - Does Clojure (or JCE, or JVM, or...?) introduce parallelism automatically? -


i running cpu-intensive clojure code within intellij idea (i don't think that's important - seems spawn process). according both htop , top, using 4 cores (well, 2 + hyperthreading) on laptop. despite me not having explicit parallelism in code.

a little more detail: top shows single process ~380% cpu use, while htop shows "parent" process , 4 "children", each 1/4 time , ~100% cpu.

is normal? or mean have got wrong somewhere? code involves many lazy sequences, @ core modifies mutable data structure (a mutable - not clojure data structure - hash accumulates results). i not using explicit parallelism.

a significant amount of time (i haven't profiled) spent in jca/jce (crypto lib) - using multiple aes ciphers in ctr mode, each stream of secure random bytes (code here), implemented lazy seqs. perhaps parallelized?

more random ideas: related io? i'm running on encrypted ssd , program processing data disk, lot of reading. htop shows system time red, , these green.

sorry such vague question. can post more info if required. clojure 1.4 on 64bit linux (jdk 1.7.0_05). code being executed here it's pretty messy (more apologies) , spread across various files (most cpu time spent in nearest-in-dump in code there). note - please don't waste time trying run code reproduce, expects pre-existing data-dump on disk (which isn't in git).

debugger running in debugger (thanks, a-m) shows 4 threads (if understand debugger correctly), 1 executing program. labelled finalizer, main (the program), reference handler, , signal dispatcher. finalizer + ref handler in wait state; signal dispatcher has no frames available. tentatively think means parallelism @ lower level, perhaps in crypto implementation?

aha think it's parallel gc (java has concurrent collector). @ start, cpu use jumps way when actual process pauses (it prints out regular tick). , since it's churning through lots of data it's generating lot of short-lived objects (confirmed using -xx:+useserialgc reduces cpu use 100%)

ok, feel bit dumb posting looks pretty obvious, seems parallel gc. processing lot of data (sucking in ssd) , generating lots of short-lived objects. , appears jvm has parallel gc. see http://blog.ragozin.info/2011/12/garbage-collection-in-hotspot-jvm.html

it may sign of problem - what going on java gc? permgen space filling up? - investigate tomorrow (i didn't mention - although in retrospect should have - borderline running out of memory).

update: running -xx:+useserialgc reduces total cpu use 100% (ie 1 core). didn't mean 2 explanations above exclusive, better configuration and/or code reduce amount of gc.


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more