Temporal pattern identification in a self-organizing neural network with an application to data compression

Most of the research in neural networks to date has dealt with spatial pattern recognition. Recently, more work in processing temporal patterns is performed with a recurrent network that is trained by a back propagation algorithm to predict the next term in the input sequence. The network requires a long time to train, and the training algorithm is subject to being caught in local minima.

A new neural network based on a tree structure capable of learning symbol strings very easily and avoiding the problem of local minima is presented. Given that the network can learn strings easily, the main problem addressed is finding the most important temporal strings in a long sequence. A measure of string importance called the information rate is derived from theoretic principles and is related to the tree structure.

An approach to finding these strings is an optimization network based on minimizing an energy function that includes the information rate. However, this approach is shown to be inadequate. Another approach is a three tiered network that incorporates a growing tree structure and determines the information rate on this tree. Simulation results indicate that this tiered network is capable of finding many of the important strings.

Since the information rate is the minimum bound on the data compression given the set of strings in the tree structure, the three tier network is modified slightly in two ways to compress realistic text, data, and binary files. The compression ratios found by these modified networks are compared with standard techniques for data compression and are found to be poorer. A problem is the means of obtaining the best encoding for the strings to cover the input sequence due to accumulating changing statistics over short segments. The networks may not find the significant strings that are best used for encoding. More memory and some pretraining are expected to help improve performance.

Various neural implementations are discussed for the derived network and the standard techniques. A new application of a technique for digital optical computing called symbolic substitution is string matching and data compression and is introduced here.