0000525: Neural network improvements - Mantis

Mantis Bugtracker

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

ID

Category

Severity

Reproducibility

Date Submitted

Last Update

0000525

[ALGLIB] Data analysis

feature

have not tried

2013-06-20 12:24

2014-06-03 16:31

Reporter

SergeyB

View Status

public

Assigned To

SergeyB

Priority

normal

Resolution

open

Status

assigned

Product Version

Summary

0000525: Neural network improvements

Description

Neural structure:
* layered
* complex interactions between layers
* several activation functions: tanh(), tanh()+linear, fast sigmoid

SGD:
* algorithm without learning rates: http://yann.lecun.com/exdb/publis/pdf/schaul-icml-13.pdf
* page 72 of http://learning.stat.purdue.edu/mlss/_media/mlss/bottou.pdf - important trick
* important info on BP acceleration http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
* ADADELTA seems to be best method - http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf
* ADAGRAD???
* "On the importance of initialization and momentum in deep learning", http://jmlr.org/proceedings/papers/v28/sutskever13.pdf

Improvements:
* shortcut layer, see "Deep Learning Made Easier by Linear Transformations in Perceptrons". Maybe - pre-training linear layer separately.

Decide on:
* minibatch training
* bagging for ensembles

* parallel errors for ensembles
* sparse errors for ensembles
* subset errors for ensembles

* decay in ensemble training?
* investigate ensemble tendency to overfit on GLASS dataset

* mini-batch LBFGS training
* approximate Hessian preconditioning
* FindBestDecay
* FindBestNetwork

* "ensemble selection", better way of constructing ensemble

* model compression
* sparse autoencoders?
* stacked autoencoders/autodecoders?

Convolutional Neural Networks (weight sharing = constraints and projections):
* http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

Additional Information

Programming language

Unspecified

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
2013-06-20 12:24	SergeyB	New Issue
2013-06-20 12:24	SergeyB	Status	new => assigned
2013-06-20 12:24	SergeyB	Assigned To	=> SergeyB
2013-06-20 12:24	SergeyB	Programming language	=> Unspecified
2013-06-20 13:59	SergeyB	Description Updated
2013-06-21 10:42	SergeyB	Description Updated
2013-06-22 14:40	SergeyB	Description Updated
2013-06-22 14:44	SergeyB	Description Updated
2013-06-22 14:56	SergeyB	Description Updated
2013-06-22 16:23	SergeyB	Description Updated
2013-06-22 17:09	SergeyB	Description Updated
2013-06-23 23:44	SergeyB	Description Updated
2013-06-26 14:46	SergeyB	Description Updated
2013-06-26 15:12	SergeyB	Description Updated
2013-06-27 14:19	SergeyB	Description Updated
2013-06-30 13:42	SergeyB	Issue Monitored: SergeyB
2013-06-30 13:42	SergeyB	Issue End Monitor: SergeyB
2013-06-30 13:42	SergeyB	Description Updated
2013-07-02 16:42	SergeyB	Description Updated
2013-07-04 15:20	SergeyB	Description Updated
2013-07-08 12:50	SergeyB	Description Updated
2013-10-05 15:04	SergeyB	Target Version	Next release => Next 'Data mining' release
2014-03-21 09:46	SergeyB	Description Updated
2014-03-30 11:06	SergeyB	Description Updated
2014-05-10 11:25	SergeyB	Description Updated
2014-06-03 12:23	SergeyB	Description Updated
2014-06-03 16:24	SergeyB	Description Updated
2014-06-03 16:31	SergeyB	Description Updated

Mantis 1.1.6[^]

Copyright © 2000 - 2008 Mantis Group