Tài liệu Báo cáo thực tập-neural networks

NEURAL NETWORKS Lecturer: Primož Potočnik University of Ljubljana Faculty of Mechanical Engineering Laboratory of Synergetics www.neural.si [email protected] +386-1-4771-167 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #1 TABLE OF CONTENTS 0. 1. 2. 3. 4. 5. 6. 7. 8. Organization of the Study Introduction to Neural Networks Neuron Model – Network Architectures – Learning Perceptrons and linear filters Backpropagation Dynamic Networks Radial Basis Function Networks Self-Organizing Maps Practical Considerations © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #2 1 0. Organization of the Study 0.1 Objectives of the study 0.2 Teaching methods 0.3 Assessment 0.4 Lecture plan 0.5 Books 0.6 SLO books 0.7 E-Books 0.8 Online resources 0.9 Simulations 0.10 Homeworks © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #3 1. Objectives of the study • Objectives – Introduce the principles and methods of neural networks (NN) – Present the principal NN models – Demonstrate the process of applying NN • Learning outcomes – Understand the concept of nonparametric modelling by NN – Explain the most common NN architectures • • • • Feedforward networks Dynamic networks Radial Basis Function Networks Self-organized networks – Develop the ability to construct NN for solving real-world problems • Design proper NN architecture • Achieve good training and generalization performance • Implement neural network solution © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #4 2 2. Teaching methods • Teaching methods: 1. Lectures 4 hours weekly, clasical & practical (MATLAB) • Tuesday 9:15 - 10:45 • Friday 9:15 - 10:45 2. Homeworks home projects 3. Consultations with the lecturer • Organization of the study – – – Nov – Dec: Jan: Jan: lectures homework presentations exam • Location – Institute for Sustainable Innovative Technologies, (Pot za Brdom 104, Ljubljana) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #5 3. Assessment • ECTS credits: – EURHEO (II): 6 ECTS • Final mark: – Homework – Written exam 50% final mark 50% final mark • Important dates – Homework presentations: – Written exam: © 2012 Primož Potočnik Tue, 8 Jan 2013 Fri, 11 Jan 2013 Fri, 18 Jan 2013 NEURAL NETWORKS (0) Organization of the Study #6 3 4. Lecture plan (1/5) 1. Introduction to Neural Networks 1.1 1.2 1.3 1.4 1.5 1.6 1.7 What is a neural network? Biological neural networks Human nervous system Artificial neural networks Benefits of neural networks Brief history of neural networks Applications of neural networks 2. Neuron Model, Network Architectures and Learning 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 Neuron model Activation functions Network architectures Learning algorithms Learning paradigms Learning tasks Knowledge representation Neural networks vs. statistical methods © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #7 4. Lecture plan (2/5) 3. Perceptrons and Linear Filters 3.1 3.2 3.3 3.4 3.5 3.6 Perceptron neuron Perceptron learning rule Adaline LMS learning rule Adaptive filtering XOR problem 4. Backpropagation 4.1 4.2 4.3 4.4 4.5 Multilayer feedforward networks Backpropagation algorithm Working with backpropagation Advanced algorithms Performance of multilayer perceptrons © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #8 4 4. Lecture plan (3/5) 5. Dynamic Networks 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 Historical dynamic networks Focused time-delay neural network Distributed time-delay neural network NARX network Layer recurrent network Computational power of dynamic networks Learning algorithms System identification Model reference adaptive control © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #9 4. Lecture plan (4/5) 6. Radial Basis Function Networks 6.1 RBFN structure 6.2 Exact interpolation 6.3 Commonly used radial basis functions 6.4 Radial Basis Function Networks 6.5 RBFN training 6.6 RBFN for pattern recognition 6.7 Comparison with multilayer perceptron 6.8 RBFN in Matlab notation 6.9 Probabilistic networks 6.10 Generalized regression networks © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #10 5 4. Lecture plan (5/5) 7. Self-Organizing Maps 7.1 7.2 7.3 7.4 7.5 Self-organization Self-organizing maps SOM algorithm Properties of the feature map Learning vector quantization 8. Practical considerations 8.1 8.2 8.3 8.4 8.5 8.6 8.7 Designing the training data Preparing data Selection of inputs Data encoding Principal component analysis Invariances and prior knowledge Generalization © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #11 5. Books 1. Neural Networks and Learning Machines, 3/E Simon Haykin (Pearson Education, 2009) 2. Neural Networks: A Comprehensive Foundation, 2/E Simon Haykin (Pearson Education, 1999) 3. Neural Networks for Pattern Recognition Chris M. Bishop (Oxford University Press, 1995) 4. Practical Neural Network Recipes in C++ Timothy Masters (Academic Press, 1993) 5. Advanced Algorithms for Neural Networks Timothy Masters (John Wiley and Sons, 1995) 6. Signal and Image Processing with Neural Networks Timothy Masters (John Wiley and Sons, 1994) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #12 6 6. SLO Books 1. Nevronske mreže Andrej Dobnikar, (Didakta 1990) 2. Modeliranje dinamičnih sistemov z umetnimi nevronskimi mrežami in sorodnimi metodami Juš Kocijan, (Založba Univerze v Novi Gorici, 2007) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #13 7. E-Books (1/2) List of links at www.neural.si – An Introduction to Neural Networks Ben Krose & Patrick van der Smagt, 1996 Recommended as an easy introduction – Neural Networks - Methodology and Applications Gerard Dreyfus, 2005 – Metaheuristic Procedures for Training Neural Networks Enrique Alba & Rafael Marti (Eds.), 2006 – FPGA Implementations of Neural Networks Amos R. Omondi & Mmondi J.C. Rajapakse (Eds.), 2006 – Trends in Neural Computation Ke Chen & Lipo Wang (Eds.), 2007 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #14 7 7. E-Books (2/2) – Neural Preprocessing and Control of Reactive Walking Machines Poramate Manoonpong, 2007 – Artificial Neural Networks for the Modelling and Fault Diagnosis of Technical Processes Krzysztof Patan, 2008 – Speech, Audio, Image and Biomedical Signal Processing using Neural Networks [only two chapters], Bhanu Prasad & S.R. Mahadeva Prasanna (Eds.), 2008 – MATLAB Neural Networks Toolbox 7 User's Guide, 2010 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #15 8. Online resources List of links at www.neural.si • Neural FAQ – by Warren Sarle, 2002 • How to measure importance of inputs – by Warren Sarle, 2000 • MATLAB Neural Networks Toolbox (User's Guide) – latest version • Artificial Neural Networks on Wikipedia.org • Neural Networks – online book by StatSoft • Radial Basis Function Networks – by Mark Orr • Principal components analysis on Wikipedia.org • libsvm – Support Vector Machines library © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #16 8 9. Simulations • Recommended computing platform – MATLAB R2010b (or later) & Neural Network Toolbox 7 http://www.mathworks.com/products/neuralnet/ Acceptable older MATLAB release: – MATLAB 7.5 & Neural Network Toolbox 5.1 (Release 2007b) • Introduction to Matlab – Get familiar with MATLAB M-file programming – Online documentation: Getting Started with MATLAB • Freeware computing platform – Stuttgart Neural Network Simulator http://www.ra.cs.uni-tuebingen.de/SNNS/ © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #17 10. Homeworks • EURHEO students (II) 1. Practical oriented projects 2. Based on UC Irvine Machine Learning Repository data http://archive.ics.uci.edu/ml/ 3. Select data set and discuss with lecturer 4. Formulate problem 5. Develop your solution (concept & Matlab code) 6. Describe solution in a short report 7. Submit results (report & Matlab source code) 8. Present results and demonstrate solution • Presentation (~10 min) • Demonstration (~20 min) © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #18 9 Video links • Robots with Biological Brains: Issues and Consequences Kevin Warwick, University of Reading http://videolectures.net/icannga2011_warwick_rbbi/ • Computational Neurogenetic Modelling: Methods, Systems, Applications Nikola Kasabov, University of Auckland http://videolectures.net/icannga2011_kasabov_cnm/ © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #19 © 2012 Primož Potočnik NEURAL NETWORKS (0) Organization of the Study #20 10 1. Introduction to Neural Networks 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 © 2012 Primož Potočnik What is a neural network? Biological neural networks Human nervous system Artificial neural networks Benefits of neural networks Brief history of neural networks Applications of neural networks List of symbols NEURAL NETWORKS (1) Introduction to Neural Networks #21 1.1 What is a neural network? (1/2) • Neural network – Network of biological neurons – Biological neural networks are made up of real biological neurons that are connected or functionally-related in the peripheral nervous system or the central nervous system • Artificial neurons – Simple mathematical approximations of biological neurons © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #22 11 What is a neural network? (2/2) • Artificial neural networks – – – – – Networks of artificial neurons Very crude approximations of small parts of biological brain Implemented as software or hardware By “Neural Networks” we usually mean Artificial Neural Networks Neurocomputers, Connectionist networks, Parallel distributted processors, ... © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #23 Neural network definitions • Haykin (1999) – A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: – Knowledge is acquired by the network through a learning process. – Interneuron connection strengths known as synaptic weights are used to store the knowledge. • Zurada (1992) – Artificial neural systems, or neural networks, are physical cellular systems which can acquire, store, and utilize experiential knowledge. • Pinkus (1999) – The question 'What is a neural network?' is ill-posed. © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #24 12 1.2 Biological neural networks Cortical neurons (nerve cells) growing in culture Neurons have a large cell body with several long processes extending from it, usually one thick axon and several thinner dendrites Dendrites receive information from other neurons Axon carries nerve impulses away from the neuron. Its branching ends make contacts with other neurons and with muscles or glands 0.1 mm © 2012 Primož Potočnik This complex network forms the nervous system, which relays information through the body NEURAL NETWORKS (1) Introduction to Neural Networks #25 Biological neuron © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #26 13 Interaction of neurons © 2012 Primož Potočnik • Action potentials arriving at the synapses stimulate currents in its dendrites • These currents depolarize the membrane at its axon, provoking an action potential • Action potential propagates down the axon to its synaptic knobs, releasing neurotransmitter and stimulating the post-synaptic neuron (lower left) NEURAL NETWORKS (1) Introduction to Neural Networks #27 Synapses • Elementary structural and functional units that mediate the interaction between neurons • Chemical synapse: pre-synaptic electric signal  chemical neurotransmitter  post-synaptic electrical signal © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #28 14 Action potential • Spikes or action potential – Neurons encode their outputs as a series of voltage pulses – Axon is very long, high resistance & high capacity – Frequency modulation  Improved signal/noise ratio © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #29 1.3 Human nervous system • Human nervous system can be represented by three stages: Stimulus • Receptors Neural net (Brain) Effectors Response Receptors – collect information from environment (photons on retina, tactile info, ...) • Effectors – generate interactions with the environment (muscle activation, ...) • Flow of information – feedforward & feedback © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #30 15 Human brain Human activity is regulated by a nervous system: • Central nervous system – – • Brain Spinal cord Peripheral nervous system ≈ 1010 neurons in the brain ≈ 104 synapses per neuron ≈ 1 ms processing speed of a neuron  Slow rate of operation  Extrem number of processing units & interconnections  Massive parallelism © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #31 Structural organization of brain Molecules & Ions ................ transmitters Synapses ............................ fundamental organization level Neural microcircuits .......... assembly of synapses organized into patterns of connectivity to produce desired functions Dendritic trees .................... subunits of individual neurons Neurons ............................... basic processing unit, size: 100 μm Local circuits ....................... localized regions in the brain, size: 1 mm Interregional circuits .......... pathways, topographic maps Central nervous system ..... final level of complexity © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #32 16 1.4 Artificial neural networks • Neuron model • Network of neurons © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #33 What NN can do? • In principle – NN can compute any computable function (everything a normal digital computer can do) • In practice – NN are especially useful for classification and function approximation problems which are tolerant of some imprecision – Almost any finite-dimensional vector function on a compact set can be approximated to arbitrary precision by feedforward NN – Need a lot of training data – Difficulties to apply hard rules (such as used in an expert system) • Problems difficult for NN – – – – Predicting random or pseudo-random numbers Factoring large integers Determining whether a large integer is prime or composite Decrypting anything encrypted by a good algorithm © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #34 17 1.5 Benefits of neural networks (1/3) 1. Ability to learn from examples • • • • Train neural network on training data Neural network will generalize on new data Noise tolerant Many learning paradigms • • • Supervised (with a teacher) Unsupervised (no teacher, self-organized) Reinforcement learning 2. Adaptivity • • • Neural networks have natural capability to adapt to the changing environment Train neural network, then retrain Continuous adaptation in nonstationary environment © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #35 Benefits of neural networks (2/3) 3. Nonlinearity • • • Artificial neuron can be linear or nonlinear Network of nonlinear neurons has nonlinearity distributed throughout the network Important for modelling inherently nonlinear signals 4. Fault tolerance • • Capable of robust computation Graceful degradation rather then catastrophic failure © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #36 18 Benefits of neural networks (3/3) 5. Massively parallel distributed structure • • Well suited for VLSI implementation Very fast hardware operation 6. Neurobiological analogy • • • NN design is motivated by analogy with brain NN are research tool for neurobiologists Neurobiology inspires further development of artificial NN 7. Uniformity of analysis & design • • Neurons represent building blocks of all neural networks Similar NN architecture for various tasks: pattern recognition, regression, time series forecasting, control applications, ... © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #37 www.stanford.edu/group/brainsinsilicon/ © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #38 19 1.6 Brief history of neural networks (1/2) -1940 von Hemholtz, Mach, Pavlov, etc. – General theories of learning, vision, conditioning – No specific mathematical models of neuron operation 1943 McCulloch and Pitts – Proposed the neuron model 1949 Hebb – Published his book The Organization of Behavior – Introduced Hebbian learning rule 1958 Rosenblatt, Widrow and Hoff – Perceptron, ADALINE – First practical networks and learning rules 1969 Minsky and Papert – Published book Perceptrons, generalised the limitations of single layer perceptrons to multilayered systems – Neural Network field went into hibernation © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #39 Brief history of neural networks (2/2) 1974 Werbos – Developed back-propagation learning method in his PhD thesis – Several years passed before this approach was popularized 1982 Hopfield – Published a series of papers on Hopfield networks 1982 Kohonen – Developed the Self-Organising Maps 1980s Rumelhart and McClelland – Backpropagation rediscovered, re-emergence of neural networks field – Books, conferences, courses, funding in USA, Europe, Japan 1990s Radial Basis Function Networks were developed 2000s The power of Ensembles of Neural Networks and Support Vector Machines becomes apparent © 2012 Primož Potočnik NEURAL NETWORKS (1) Introduction to Neural Networks #40 20