Tài liệu Nnet

Neural Network Toolbox For Use with MATLAB ® Howard Demuth Mark Beale Computation Visualization Programming User’s Guide Version 4 How to Contact The MathWorks: www.mathworks.com comp.soft-sys.matlab Web Newsgroup [email protected] Technical support Product enhancement suggestions Bug reports Documentation error reports Order status, license renewals, passcodes Sales, pricing, and general information 508-647-7000 Phone 508-647-7001 Fax The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 Mail [email protected] [email protected] [email protected] [email protected] [email protected] For contact information about worldwide offices, see the MathWorks Web site. Neural Network Toolbox User’s Guide  COPYRIGHT 1992 - 2002 by The MathWorks, Inc. The software described in this document is furnished under a license agreement. The software may be used or copied only under the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written consent from The MathWorks, Inc. FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by or for the federal government of the United States. By accepting delivery of the Program, the government hereby agrees that this software qualifies as "commercial" computer software within the meaning of FAR Part 12.212, DFARS Part 227.7202-1, DFARS Part 227.7202-3, DFARS Part 252.227-7013, and DFARS Part 252.227-7014. The terms and conditions of The MathWorks, Inc. Software License Agreement shall pertain to the government’s use and disclosure of the Program and Documentation, and shall supersede any conflicting contractual terms or conditions. If this license fails to meet the government’s minimum needs or is inconsistent in any respect with federal procurement law, the government agrees to return the Program and Documentation, unused, to MathWorks. MATLAB, Simulink, Stateflow, Handle Graphics, and Real-Time Workshop are registered trademarks, and TargetBox is a trademark of The MathWorks, Inc. Other product or brand names are trademarks or registered trademarks of their respective holders. Printing History: June 1992 April 1993 January 1997 July 1997 January 1998 September 2000 June 2001 July 2002 First printing Second printing Third printing Fourth printing Fifth printing Sixth printing Seventh printing Online only Revised for Version 3 (Release 11) Revised for Version 4 (Release 12) Minor revisions (Release 12.1) Minor revisions (Release 13) Contents Preface Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Basic Chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Mathematical Notation for Equations and Figures . . . . . . . xxi Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Weight Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Layer Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Figure and Equation Examples . . . . . . . . . . . . . . . . . . . . . . . . xxii Mathematics and Code Equivalents . . . . . . . . . . . . . . . . . . . xxiii Neural Network Design Book . . . . . . . . . . . . . . . . . . . . . . . . . xxiv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv Introduction 1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 Basic Chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 Help and Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 What’s New in Version 4.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control System Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . New Training Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Design of General Linear Networks . . . . . . . . . . . . . . . . . . . . . . Improved Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3 1-3 1-3 1-3 1-4 1-4 iii Generalization and Speed Benchmarks . . . . . . . . . . . . . . . . . . . 1-4 Demonstration of a Sample Training Session . . . . . . . . . . . . . . 1-4 Neural Network Applications . . . . . . . . . . . . . . . . . . . . . . . . . . Applications in this Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . Business Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aerospace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Automotive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Banking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Credit Card Activity Checking . . . . . . . . . . . . . . . . . . . . . . . . . . Defense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Electronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Entertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Financial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Industrial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Insurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Manufacturing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Medical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oil and Gas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Securities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Telecommunications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5 1-5 1-5 1-5 1-5 1-5 1-5 1-6 1-6 1-6 1-6 1-6 1-6 1-6 1-7 1-7 1-7 1-7 1-7 1-7 1-7 1-7 Neuron Model and Network Architectures 2 Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simple Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transfer Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Neuron with Vector Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2 2-2 2-3 2-5 Network Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8 A Layer of Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8 Multiple Layers of Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11 iv Contents Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulation With Concurrent Inputs in a Static Network . . . . Simulation With Sequential Inputs in a Dynamic Network . . Simulation With Concurrent Inputs in a Dynamic Network . 2-13 2-13 2-14 2-16 Training Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-18 Incremental Training (of Adaptive and Other Networks) . . . . 2-18 Batch Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-20 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24 Figures and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-25 Perceptrons 3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2 Important Perceptron Functions . . . . . . . . . . . . . . . . . . . . . . . . . 3-3 Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4 Perceptron Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6 Creating a Perceptron (newp) . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7 Simulation (sim) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8 Initialization (init) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-9 Learning Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12 Perceptron Learning Rule (learnp) . . . . . . . . . . . . . . . . . . . . 3-13 Training (train) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16 Limitations and Cautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 Outliers and the Normalized Perceptron Rule . . . . . . . . . . . . . 3-21 v Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction to the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create a Perceptron Network (nntool) . . . . . . . . . . . . . . . . . . . Train the Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Export Perceptron Results to Workspace . . . . . . . . . . . . . . . . . Clear Network/Data Window . . . . . . . . . . . . . . . . . . . . . . . . . . Importing from the Command Line . . . . . . . . . . . . . . . . . . . . . Save a Variable to a File and Load It Later . . . . . . . . . . . . . . . 3-23 3-23 3-23 3-27 3-29 3-30 3-30 3-31 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33 Figures and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33 New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36 Linear Filters 4 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4 Creating a Linear Neuron (newlin) . . . . . . . . . . . . . . . . . . . . . . . 4-4 Mean Square Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8 Linear System Design (newlind) . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Linear Networks with Delays . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 Tapped Delay Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 Linear Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 LMS Algorithm (learnwh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13 Linear Classification (train) . . . . . . . . . . . . . . . . . . . . . . . . . . 4-15 vi Contents Limitations and Cautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overdetermined Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Underdetermined Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linearly Dependent Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . Too Large a Learning Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18 4-18 4-18 4-18 4-19 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20 Figures and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-21 New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-25 Backpropagation 5 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulation (sim) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4 5-4 5-8 5-8 Faster Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Variable Learning Rate (traingda, traingdx) . . . . . . . . . . . . . . Resilient Backpropagation (trainrp) . . . . . . . . . . . . . . . . . . . . . Conjugate Gradient Algorithms . . . . . . . . . . . . . . . . . . . . . . . . Line Search Routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Quasi-Newton Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Levenberg-Marquardt (trainlm) . . . . . . . . . . . . . . . . . . . . . . . . Reduced Memory Levenberg-Marquardt (trainlm) . . . . . . . . . 5-14 5-14 5-16 5-17 5-23 5-26 5-28 5-30 Speed and Memory Comparison . . . . . . . . . . . . . . . . . . . . . . . 5-32 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-49 Improving Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-51 5-52 5-55 5-57 vii Preprocessing and Postprocessing . . . . . . . . . . . . . . . . . . . . . Min and Max (premnmx, postmnmx, tramnmx) . . . . . . . . . . . Mean and Stand. Dev. (prestd, poststd, trastd) . . . . . . . . . . . . Principal Component Analysis (prepca, trapca) . . . . . . . . . . . . Post-Training Analysis (postreg) . . . . . . . . . . . . . . . . . . . . . . . . 5-61 5-61 5-62 5-63 5-64 Sample Training Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-66 Limitations and Cautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-71 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-73 Control Systems 6 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2 NN Predictive Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Predictive Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using the NN Predictive Controller Block . . . . . . . . . . . . . . . . . 6-4 6-4 6-5 6-6 NARMA-L2 (Feedback Linearization) Control . . . . . . . . . . Identification of the NARMA-L2 Model . . . . . . . . . . . . . . . . . . NARMA-L2 Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using the NARMA-L2 Controller Block . . . . . . . . . . . . . . . . . . 6-14 6-14 6-16 6-18 Model Reference Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-23 Using the Model Reference Controller Block . . . . . . . . . . . . . . 6-25 Importing and Exporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-31 Importing and Exporting Networks . . . . . . . . . . . . . . . . . . . . . 6-31 Importing and Exporting Training Data . . . . . . . . . . . . . . . . . 6-35 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-38 viii Contents Radial Basis Networks 7 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-2 Important Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . 7-2 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exact Design (newrbe) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . More Efficient Design (newrb) . . . . . . . . . . . . . . . . . . . . . . . . . . Demonstrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-3 7-3 7-4 7-5 7-7 7-8 Generalized Regression Networks . . . . . . . . . . . . . . . . . . . . . . 7-9 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-9 Design (newgrnn) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-10 Probabilistic Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . 7-12 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-12 Design (newpnn) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-15 Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-16 New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7-18 8 Self-Organizing and Learn. Vector Quant. Nets Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2 Important Self-Organizing and LVQ Functions . . . . . . . . . . . . . 8-2 Competitive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating a Competitive Neural Network (newc) . . . . . . . . . . . . Kohonen Learning Rule (learnk) . . . . . . . . . . . . . . . . . . . . . . . . . Bias Learning Rule (learncon) . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3 8-3 8-4 8-5 8-5 ix Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-6 Graphical Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-7 Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-9 Topologies (gridtop, hextop, randtop) . . . . . . . . . . . . . . . . . . . . 8-10 Distance Funct. (dist, linkdist, mandist, boxdist) . . . . . . . . . . 8-14 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-17 Creating a Self Organizing MAP Neural Network (newsom) . 8-18 Training (learnsom) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-19 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-23 Learning Vector Quantization Networks . . . . . . . . . . . . . . . Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating an LVQ Network (newlvq) . . . . . . . . . . . . . . . . . . . . . LVQ1 Learning Rule(learnlv1) . . . . . . . . . . . . . . . . . . . . . . . . . Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Supplemental LVQ2.1 Learning Rule (learnlv2) . . . . . . . . . . . 8-31 8-31 8-32 8-35 8-36 8-38 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Vector Quantizaton Networks . . . . . . . . . . . . . . . . . . Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-40 8-40 8-40 8-41 8-42 Recurrent Networks 9 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-2 Important Recurrent Network Functions . . . . . . . . . . . . . . . . . . 9-2 Elman Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Creating an Elman Network (newelm) . . . . . . . . . . . . . . . . . . . . Training an Elman Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . x Contents 9-3 9-3 9-4 9-5 Hopfield Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-8 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-8 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-8 Design (newhop) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-15 Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-16 New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9-17 10 Adaptive Filters and Adaptive Training Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-2 Important Adaptive Functions . . . . . . . . . . . . . . . . . . . . . . . . . 10-2 Linear Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-3 Adaptive Linear Network Architecture . . . . . . . . . . . . . . . . 10-4 Single ADALINE (newlin) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-4 Mean Square Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-7 LMS Algorithm (learnwh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-8 Adaptive Filtering (adapt) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9 Tapped Delay Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9 Adaptive Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-9 Adaptive Filter Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-10 Prediction Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-13 Noise Cancellation Example . . . . . . . . . . . . . . . . . . . . . . . . . . 10-14 Multiple Neuron Adaptive Filters . . . . . . . . . . . . . . . . . . . . . . 10-16 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-18 Figures and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-18 New Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10-26 xi Applications 11 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2 Application Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-2 Applin1: Linear Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thoughts and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-3 11-3 11-4 11-4 11-6 Applin2: Adaptive Prediction . . . . . . . . . . . . . . . . . . . . . . . . . 11-7 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-7 Network Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8 Network Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8 Network Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-8 Thoughts and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-10 xii Contents Appelm1: Amplitude Detection . . . . . . . . . . . . . . . . . . . . . . . Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Improving Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-11 11-11 11-11 11-12 11-13 11-13 11-15 Appcr1: Character Recognition . . . . . . . . . . . . . . . . . . . . . . . Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . System Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11-16 11-16 11-17 11-20 11-22 Advanced Topics 12 Custom Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-2 Custom Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-3 Network Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-4 Network Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-12 Additional Toolbox Functions . . . . . . . . . . . . . . . . . . . . . . . . Initialization Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transfer Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12-16 12-16 12-16 12-17 Custom Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Initialization Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Self-Organizing Map Functions . . . . . . . . . . . . . . . . . . . . . . . 12-18 12-18 12-24 12-27 12-36 Network Object Reference 13 Network Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-2 Subobject Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-6 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-9 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-12 Weight and Bias Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-14 Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-16 Subobject Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Outputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Biases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Input Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Layer Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13-17 13-17 13-18 13-25 13-25 13-26 13-28 13-32 xiii Reference 14 Functions — By Category . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-2 Functions by Network Type . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-2 Functions by Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-2 Transfer Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-13 Transfer Function Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-14 Functions — Alphabetical List . . . . . . . . . . . . . . . . . . . . . . . 14-19 Reference Headings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14-19 Glossary A Bibliography B C Demonstrations and Applications Tables of Demonstrations and Applications . . . . . . . . . . . . . Chapter 2: Neuron Model and Network Architectures . . . . . . . Chapter 3: Perceptrons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 4: Linear Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 5: Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 7: Radial Basis Networks . . . . . . . . . . . . . . . . . . . . . . . Chapter 8: Self-Organizing and Learn. Vector Quant. Nets . . . Chapter 9: Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 10: Adaptive Networks . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 11: Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv Contents C-2 C-2 C-2 C-3 C-3 C-4 C-4 C-4 C-5 C-5 Simulink D Block Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transfer Function Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Net Input Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Weight Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control Systems Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2 D-2 D-3 D-3 D-4 Block Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-7 Code Notes E Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-2 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-3 Utility Function Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-4 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-7 Code Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-8 Argument Checking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-9 Index xv xvi Contents Preface Neural Networks . . . . . . . . . . . . . . . . . . xviii Basic Chapters . . . . . . . . . . . . . . . . . . . . xx Mathematical Notation for Equations and Figures Basic Concepts . . . . . . . . . . . . . . . . . Language . . . . . . . . . . . . . . . . . . . . Weight Matrices . . . . . . . . . . . . . . . . . Layer Notation . . . . . . . . . . . . . . . . . Figure and Equation Examples . . . . . . . . . . . Mathematics and Code Equivalents . . . . . . . . . . . . xxi xxi xxi xxi xxi xxii . . . . . . . . . xxiii Neural Network Design Book . . . . . . . . . . . . xxiv Acknowledgments . . . . . . . . . . . . . . . . . xxv Preface Neural Networks Neural networks are composed of simple elements operating in parallel. These elements are inspired by biological nervous systems. As in nature, the network function is determined largely by the connections between elements. We can train a neural network to perform a particular function by adjusting the values of the connections (weights) between elements. Commonly neural networks are adjusted, or trained, so that a particular input leads to a specific target output. Such a situation is shown below. There, the network is adjusted, based on a comparison of the output and the target, until the network output matches the target. Typically many such input/target pairs are used, in this supervised learning, to train a network. Target Input Neural Network including connections (called weights) between neurons Compare Output Adjust weights Batch training of a network proceeds by making weight and bias changes based on an entire set (batch) of input vectors. Incremental training changes the weights and biases of a network as needed after presentation of each individual input vector. Incremental training is sometimes referred to as “on line” or “adaptive” training. Neural networks have been trained to perform complex functions in various fields of application including pattern recognition, identification, classification, speech, vision and control systems. A list of applications is given in Chapter 1. Today neural networks can be trained to solve problems that are difficult for conventional computers or human beings. Throughout the toolbox emphasis is placed on neural network paradigms that build up to or are themselves used in engineering, financial and other practical applications. xviii Neural Networks The supervised training methods are commonly used, but other networks can be obtained from unsupervised training techniques or from direct design methods. Unsupervised networks can be used, for instance, to identify groups of data. Certain kinds of linear networks and Hopfield networks are designed directly. In summary, there are a variety of kinds of design and learning techniques that enrich the choices that a user can make. The field of neural networks has a history of some five decades but has found solid application only in the past fifteen years, and the field is still developing rapidly. Thus, it is distinctly different from the fields of control systems or optimization where the terminology, basic mathematics, and design procedures have been firmly established and applied for many years. We do not view the Neural Network Toolbox as simply a summary of established procedures that are known to work well. Rather, we hope that it will be a useful tool for industry, education and research, a tool that will help users find what works and what doesn’t, and a tool that will help develop and extend the field of neural networks. Because the field and the material are so new, this toolbox will explain the procedures, tell how to apply them, and illustrate their successes and failures with examples. We believe that an understanding of the paradigms and their application is essential to the satisfactory and successful use of this toolbox, and that without such understanding user complaints and inquiries would bury us. So please be patient if we include a lot of explanatory material. We hope that such material will be helpful to you. xix Preface Basic Chapters The Neural Network Toolbox is written so that if you read Chapter 2, Chapter 3 and Chapter 4 you can proceed to a later chapter, read it and use its functions without difficulty. To make this possible, Chapter 2 presents the fundamentals of the neuron model, the architectures of neural networks. It also will discuss notation used in the architectures. All of this is basic material. It is to your advantage to understand this Chapter 2 material thoroughly. The neuron model and the architecture of a neural network describe how a network transforms its input into an output. This transformation can be viewed as a computation. The model and the architecture each place limitations on what a particular neural network can compute. The way a network computes its output must be understood before training methods for the network can be explained. xx