Mantis Bugtracker
Main | My View | View Issues | Change Log | Roadmap | Docs
  

Viewing Issue Advanced Details [ Jump to Notes ] [ View Simple ] [ Issue History ] [ Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0000525 [ALGLIB] Data analysis feature have not tried 2013-06-20 12:24 2014-06-03 16:31
Reporter SergeyB View Status public  
Assigned To SergeyB
Priority normal Resolution open Platform
Status assigned   OS
Projection none   OS Version
ETA none Fixed in Version Product Version
  Target Version Next 'Data mining' release Product Build
Summary 0000525: Neural network improvements
Description Neural structure:
* layered
* complex interactions between layers
* several activation functions: tanh(), tanh()+linear, fast sigmoid

SGD:
* algorithm without learning rates: http://yann.lecun.com/exdb/publis/pdf/schaul-icml-13.pdf
* page 72 of http://learning.stat.purdue.edu/mlss/_media/mlss/bottou.pdf - important trick
* important info on BP acceleration http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
* ADADELTA seems to be best method - http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf
* ADAGRAD???
* "On the importance of initialization and momentum in deep learning", http://jmlr.org/proceedings/papers/v28/sutskever13.pdf

Improvements:
* shortcut layer, see "Deep Learning Made Easier by Linear Transformations in Perceptrons". Maybe - pre-training linear layer separately.

Decide on:
* minibatch training
* bagging for ensembles

* parallel errors for ensembles
* sparse errors for ensembles
* subset errors for ensembles


* decay in ensemble training?
* investigate ensemble tendency to overfit on GLASS dataset

* mini-batch LBFGS training
* approximate Hessian preconditioning
* FindBestDecay
* FindBestNetwork

* "ensemble selection", better way of constructing ensemble

* model compression
* sparse autoencoders?
* stacked autoencoders/autodecoders?

Convolutional Neural Networks (weight sharing = constraints and projections):
* http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf
Steps To Reproduce
Additional Information
Programming language Unspecified
Attached Files

- Relationships

There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
2013-06-20 12:24 SergeyB New Issue
2013-06-20 12:24 SergeyB Status new => assigned
2013-06-20 12:24 SergeyB Assigned To => SergeyB
2013-06-20 12:24 SergeyB Programming language => Unspecified
2013-06-20 13:59 SergeyB Description Updated
2013-06-21 10:42 SergeyB Description Updated
2013-06-22 14:40 SergeyB Description Updated
2013-06-22 14:44 SergeyB Description Updated
2013-06-22 14:56 SergeyB Description Updated
2013-06-22 16:23 SergeyB Description Updated
2013-06-22 17:09 SergeyB Description Updated
2013-06-23 23:44 SergeyB Description Updated
2013-06-26 14:46 SergeyB Description Updated
2013-06-26 15:12 SergeyB Description Updated
2013-06-27 14:19 SergeyB Description Updated
2013-06-30 13:42 SergeyB Issue Monitored: SergeyB
2013-06-30 13:42 SergeyB Issue End Monitor: SergeyB
2013-06-30 13:42 SergeyB Description Updated
2013-07-02 16:42 SergeyB Description Updated
2013-07-04 15:20 SergeyB Description Updated
2013-07-08 12:50 SergeyB Description Updated
2013-10-05 15:04 SergeyB Target Version Next release => Next 'Data mining' release
2014-03-21 09:46 SergeyB Description Updated
2014-03-30 11:06 SergeyB Description Updated
2014-05-10 11:25 SergeyB Description Updated
2014-06-03 12:23 SergeyB Description Updated
2014-06-03 16:24 SergeyB Description Updated
2014-06-03 16:31 SergeyB Description Updated

Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker