Mantis - ALGLIB
Viewing Issue Advanced Details
525 Data analysis feature have not tried 2013-06-20 12:24 2014-06-03 16:31
SergeyB  
SergeyB  
normal  
assigned  
open  
none    
none  
Unspecified
0000525: Neural network improvements
Neural structure:
* layered
* complex interactions between layers
* several activation functions: tanh(), tanh()+linear, fast sigmoid

SGD:
* algorithm without learning rates: http://yann.lecun.com/exdb/publis/pdf/schaul-icml-13.pdf
* page 72 of http://learning.stat.purdue.edu/mlss/_media/mlss/bottou.pdf - important trick
* important info on BP acceleration http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
* ADADELTA seems to be best method - http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf
* ADAGRAD???
* "On the importance of initialization and momentum in deep learning", http://jmlr.org/proceedings/papers/v28/sutskever13.pdf

Improvements:
* shortcut layer, see "Deep Learning Made Easier by Linear Transformations in Perceptrons". Maybe - pre-training linear layer separately.

Decide on:
* minibatch training
* bagging for ensembles

* parallel errors for ensembles
* sparse errors for ensembles
* subset errors for ensembles


* decay in ensemble training?
* investigate ensemble tendency to overfit on GLASS dataset

* mini-batch LBFGS training
* approximate Hessian preconditioning
* FindBestDecay
* FindBestNetwork

* "ensemble selection", better way of constructing ensemble

* model compression
* sparse autoencoders?
* stacked autoencoders/autodecoders?

Convolutional Neural Networks (weight sharing = constraints and projections):
* http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf
Issue History
2013-06-20 12:24 SergeyB New Issue
2013-06-20 12:24 SergeyB Status new => assigned
2013-06-20 12:24 SergeyB Assigned To => SergeyB
2013-06-20 12:24 SergeyB Programming language => Unspecified
2013-06-20 13:59 SergeyB Description Updated
2013-06-21 10:42 SergeyB Description Updated
2013-06-22 14:40 SergeyB Description Updated
2013-06-22 14:44 SergeyB Description Updated
2013-06-22 14:56 SergeyB Description Updated
2013-06-22 16:23 SergeyB Description Updated
2013-06-22 17:09 SergeyB Description Updated
2013-06-23 23:44 SergeyB Description Updated
2013-06-26 14:46 SergeyB Description Updated
2013-06-26 15:12 SergeyB Description Updated
2013-06-27 14:19 SergeyB Description Updated
2013-06-30 13:42 SergeyB Issue Monitored: SergeyB
2013-06-30 13:42 SergeyB Issue End Monitor: SergeyB
2013-06-30 13:42 SergeyB Description Updated
2013-07-02 16:42 SergeyB Description Updated
2013-07-04 15:20 SergeyB Description Updated
2013-07-08 12:50 SergeyB Description Updated
2013-10-05 15:04 SergeyB Target Version Next release => Next 'Data mining' release
2014-03-21 09:46 SergeyB Description Updated
2014-03-30 11:06 SergeyB Description Updated
2014-05-10 11:25 SergeyB Description Updated
2014-06-03 12:23 SergeyB Description Updated
2014-06-03 16:24 SergeyB Description Updated
2014-06-03 16:31 SergeyB Description Updated

There are no notes attached to this issue.