On the Dynamic Behaviour of Back Propagation Networks

Abstract:
As neural networks are often used to solve problems which are not completely understood or which are hard to solve with the more traditional AI techniques, it is important to know how well a neural network can learn to solve such a problem. One category of problems that can be solved with a neural network, is the category of association problems; each input of the problem space has to be associated with a correct output. These association problems are often solved with so-called back propagation (BP) networks. A BP network will be trained with a set of inputs of the problem space and the correct outputs that are corresponding to these inputs. During the training session, the weights of the network will converge to a point in the network's weight space, in which the problem examples will be known to the network; when the network has reached this point in its state space, it can give the correct output for each example input. However, due to the nature of the network, it can not always learn the problem exactly; the output produced by the network when presented with an input problem, will sometimes only be an approximation of the exact output. Therefore it is usually hard to tell when the network has learned the problem. To understand more of the problem of determining when the BP network has learned the association problem, the dynamic behaviour of the neural network during its training should be studied. To investigate this dynamic behaviour of a BP network, a tool has been developed under OSF/Motif that can trace the internal state of a BP network during a training session. This report gives the results of the investigations done with this tool. This report has the following structure. The following section gives a description of BP networks and which variant has been used in the tool. Section 3 will describe the properties of BP networks and the problems which can arise with them. In section 4 will be explained which parameters can be set to investigate the network and in section 5 the various ways to train the network with the tool will be given. Section 6 will describe which properties of the network can be viewed with the tool and some traces made with the tool will be analysed. In section 7 some conclusions will be given and some topics for further research can be found in section 8. The report will end with an appendix showing the exact solution for the ID1 function learned by a one layer network.