Graduation Project

My Master's Thesis consists of two parts. The first part concentrates on comparing and combining algorithms based on Back-Propagation to train Multi-layered Feed-forward Networks. I looked at the following algorithms:

Back-Propagation (BP)
Quick-Propagation (QP)
Epsilon-Back-Propagation (EBP)
Randomized Back-Propagation (RBP) - My own version of Back-Propagation using Random Generators.

I compared the algorithms using the Parity-12 Task using three criteria

Reliability (smallest classification error after 1000 epochs)
Time (time to train the network for 1000 epochs)
Speed (how fast the network learn in terms of epochs)

Each criteria had its own winner: Randomized Back-Propagation won the reliability contest, Epsilon-Back-Propagation took the least time, and Quick-Propagation was the fastest. Finally I combined algorithms to try and combine the best of each. The only succesful attempyt was combining Quick-Propagation and Back-Propagation into the QPBP algorithm. Not only did it learn better than one of the algorithms mentioned above, it was also the fastest making Quick-Propagation look like a tortoise (or was it turtle). The second part of my thesis looked at the BP-SOM architecture. This architecture was developed by A.J.M.M. (Ton) Weijters of the Computer Science Department of the University of Maastricht . A Technical Report is available for more information about the BP-SOM architecture and learning rule. The BP-SOM architecture combines a Multi-Layered Feed-forward Network with one or more Self-Organizing Maps trained with Kohonen Feature Mapping. Although the BP-SOM architecture is originally trained with the BP-SOM learning rule, I also combined it with the learning algorithms above resulting in

BP-SOM
QP-SOM ( Quick-Propagation learning rule + BP-SOM architecture)
EBP-SOM (Epsilon-Back-Propagation learning rule + BP-SOM architecture)
RBP-SOM (Randomized Back-Propagation learning rule + BP-SOM architecture)
QPBP-SOM ( a combination of QP-SOM and BP-SOM)

Once again we compared the algorithms using the three criteria reliability, time and speed. Like QPBP outperformed the BP variants in terms of reliability and speed, QPBP-SOM was the best when comparing BP-SOM variants. Epsilon-Back-Propagation also showed its strength by haveing a much lower computation time than any of the other algorithms. Using the unique architecture of BP-SOM I also designed two rule-extraction methods called REBA-1 and REBA-2 which use the both the Self-Organizing Mapand the Feed-forward Network for extracting rules.

The final version of my Master's Thesis is available in postschript or dvi versions. I all goes well It can also be found on the ftp-server or on the page of Ida Sprinkhuizen Kuyper .