Scipy optimization algorithms? (for minimizing neural network cost function) - python -


i wrote neural network object in python has cost function , determines gradients via back-propogation. see bunch of optimization functions here, have no idea how implement them. i'm having hard time finding example code learn from.

clearly need somehow tell parameters i'm trying change, cost function i'm trying minimize, , gradient calculated backprop. how tell, say, fmin_cg what's what?

bonus question: can learn differences in uses of various algorithms?

===== ok, update =====

this have atm:

def train(self, x, y_vals, iters = 400): t0 = concatenate((self.hid_t.reshape(-1), self.out_t.reshape(-1)), 1) self.forward_prop(x, t0) c = lambda v: self.cost(x, y_vals, v) g = lambda v: self.back_prop(y_vals, v) t_best = fmin_cg(c, t0, g, disp=true, maxiter=iters) self.hid_t = reshape(t_best[:,:(hid_n * (in_n+1))], (hid_n, in_n+1)) self.out_t = reshape(t_best[:,(hid_n * (in_n+1)):], (out_n, hid_n+1))

and, error it's throwing:

traceback (most recent call last): file "<stdin>", line 1, in <module> file "netset.py", line 27, in <module> net.train(x,y) file "neuralnet.py", line 60, in train t_best = fmin_cg(c, t0, g, disp=true, maxiter=iters) file "/usr/local/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 952, in fmin_cg res = _minimize_cg(f, x0, args, fprime, callback=callback, **opts) file "/usr/local/lib/python2.7/dist-packages/scipy/optimize/optimize.py", line 1017, in _minimize_cg deltak = numpy.dot(gfk, gfk) valueerror: matrices not aligned

...halp!

i never used fmin_cg. guess v weight vector. did not find error in code, when read documentation. searched error , found this: matrices not aligned error: python scipy fmin_bfgs

in addition, think not garantueed g(v) calculated after c(v). thus, backpropagation function should forward propagate x once again:

c = lambda v: self.cost(x, y_vals, v) g = lambda v: self.cost(x, y_vals, v); self.back_prop(y_vals, v) # ------------------------

or can pass 1 function returns cost function , gradient tuple avoid 2 forward propagations issam laradji mentioned.

good articles optimization algorithms artificial neural networks are:

i can recommend levenberg-marquardt. algorithms works well. unfortunately every iteration step has cubic complexity o(n^3).


Comments

Post a Comment

Popular posts from this blog

javascript - backbone.js Collection.add() doesn't `construct` (`initialize`) an object -

<!doctype html public "-//w3c//dtd xhtml 1.0 strict//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-strict.dtd"> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script> <script src="http://documentcloud.github.com/underscore/underscore-min.js"></script> <script src="http://documentcloud.github.com/backbone/backbone-min.js"></script> </head> <body> <button id="cmd_create_event" name="cmd_create_event" type="button">create new `event`</button> <script type="text/javascript"> var eventmodel = backbone.model.extend({ initialize : function() { console.log("`event` initialized. id: " + this.cid); this.bind("change:status", function() { console.log(this.get("status") + " value `status`"); }); this.bind("error", function(model, e...
Read more

php - Get uncommon values from two or more arrays -

is there function in php give array of uncommon values 2 or more arrays? for example: $array1 = array( "green", "red", "blue"); $array2 = array( "green", "yellow", "red"); .... $result = function_needed($array1, $array2,...); print_r($result); should give output: array("blue", "yellow", ...); use array_diff , array_merge : $result = array_merge(array_diff($array1, $array2), array_diff($array2, $array1)); here's demo. for multiple arrays, combine callback , array_reduce : function unique(&$a, $b) { return $a ? array_merge(array_diff($a, $b), array_diff($b, $a)) : $b; } $arrays = array( array('green', 'red', 'blue'), array('green', 'yellow', 'red') ); $result = array_reduce($arrays, 'unique'); and here's demo of that.
Read more

Adding duplicate array rows in Php -

i create function in php adds rows of array shared column value. so input. $test = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); and output. $testduplicatefree = array( array("c", 7, 9), array("test", 5, 6) ) ); i'm thinking function combine_duplicates($array,$col){ ... ... return $duplicatefreearray; } where $col duplicate free array. so, in case, combine_duplicates($test,0); would me desired output. this. <?php function combine_duplicates($array,$col) { $index = array(); foreach ($array $row) { $key = $row[$col]; if (!isset($index[$key])) { $index[$key] = $row; } else { ($i = 0; $i < count($row); ++$i) { if ($i != $col) { $index[$key][$i] += $row[$i]; } } } } return array_values($index); } $array = array( array("c", 5, 6), array("c", 2, 3), array("test", 5, 6) ); print_r(combine_duplicates($array, 0)); try here: http://codepad.org/mdhesqhi
Read more