Search Results (1 - 4 of 4 Results)

Sort By  
Sort Dir
 
Results per page  

Shao, YuanlongLearning Sparse Recurrent Neural Networks in Language Modeling
Master of Science, The Ohio State University, 2014, Computer Science and Engineering
In the context of statistical language modeling, we explored the task of learning an Elman network with sparse weight matrices, as a pilot study towards learning a sparsely con-nected fully recurrent neural network, which would be potentially useful in many cases. We also explored how efficient and scalable it can be in practice. In particular, we explored these tasks: (1) We adapted the Iterative Hard Thresholding (IHT) algorithm into the BackPropagation Through Time (BPTT) learning. (2) To accel-erate convergence of the IHT algorithm, we designed a scheme for expanding the net-work by replicating the existing hidden neurons. Thus we can start training from a small and dense network which is already learned. (3) We implemented this algorithm in GPU. Under small minibatch sizes and large network sizes (e.g., 2000 hidden neurons) it achieves 160 times speedup compared to the RNNLM toolkit in CPU. With larger mini-batch sizes there could be another 10 times speedup, though the convergence rate be-comes an issue in such cases and further effort is needed to address this problem. (4) Without theoretical convergence guarantee of the IHT algorithm in our problem setting, we did an empirical study showing that learning a sparse network does give competitive perplexity in language modeling. In particular, we showed that a sparse network learned in this way can outperform a dense network when the number of effective parameters is kept the same. (5) We gathered performance metric comparing the computational effi-ciency of the matrix operations of interest in both sparse and dense settings. The results suggest that for network sizes which we can train in reasonable time at this moment, it’s hard for sparse matrices to run faster, unless we are allowed to have very sparse networks. Thus for research purposes we may want to focus on using dense matrices, while for en-gineering purposes a more flexible matrix design leveraging the power of dense and sparse matrices might be necessary.

Committee:

Eric Fosler-Lussier, Dr. (Advisor); Mikhail Belkin, Dr. (Committee Member)

Subjects:

Artificial Intelligence; Computer Science

Keywords:

language modeling; recurrent neural networks; sparse recurrent neural networks

Mehta, Manish P.Prediction of manufacturing operations sequence using recurrent neural networks
Master of Science (MS), Ohio University, 1997, Industrial and Manufacturing Systems Engineering (Engineering)
Prediction of manufacturing operations sequence using recurrent neural networks

Committee:

Luis Rabelo (Advisor)

Subjects:

Engineering, Industrial

Keywords:

Recurrent Neural Networks; Computer-Aided Process Planning; Automation and Computer Integrated Manufacturing

Putchala, Manoj KumarDeep Learning Approach for Intrusion Detection System (IDS) in the Internet of Things (IoT) Network using Gated Recurrent Neural Networks (GRU)
Master of Science (MS), Wright State University, 2017, Computer Science
The Internet of Things (IoT) is a complex paradigm where billions of devices are connected to a network. These connected devices form an intelligent system of systems that share the data without human-to-computer or human-to-human interaction. These systems extract meaningful data that can transform human lives, businesses, and the world in significant ways. However, the reality of IoT is prone to countless cyber-attacks in the extremely hostile environment like the internet. The recent hack of 2014 Jeep Cherokee, iStan pacemaker, and a German steel plant are a few notable security breaches. To secure an IoT system, the traditional high-end security solutions are not suitable, as IoT devices are of low storage capacity and less processing power. Moreover, the IoT devices are connected for longer time periods without human intervention. This raises a need to develop smart security solutions which are light-weight, distributed and have a high longevity of service. Rather than per-device security for numerous IoT devices, it is more feasible to implement security solutions for network data. The artificial intelligence theories like Machine Learning and Deep Learning have already proven their significance when dealing with heterogeneous data of various sizes. To substantiate this, in this research, we have applied concepts of Deep Learning and Transmission Control Protocol/Internet Protocol (TCP/IP) to build a light-weight distributed security solution with high durability for IoT network security. First, we have examined the ways of improving IoT architecture and proposed a light-weight and multi-layered design for an IoT network. Second, we have analyzed the existingapplications of Machine Learning and Deep Learning to the IoT and Cyber-Security. Third, we have evaluated deep learning’s Gated Recurrent Neural Networks (LSTM and GRU) on the DARPA/KDD Cup '99 intrusion detection data set for each layer in the designed architecture. Finally, from the evaluated metrics, we have proposed the best neural network design suitable for the IoT Intrusion Detection System. With an accuracy of 98.91% and False Alarm Rate of 0.76 %, this unique research outperformed the performance results of existing methods over the KDD Cup ’99 dataset. For this first time in the IoT research, the concepts of Gated Recurrent Neural Networks are applied for the IoT security.

Committee:

Michelle Cheatham, Ph.D. (Advisor); Adam Bryant, Ph.D. (Committee Member); Mateen Rizki, Ph.D. (Committee Member)

Subjects:

Computer Science

Keywords:

Deep Learning; Internet of Things; Machine Learning; Gated Recurrent Unit; Recurrent Neural Networks

Perumal, SubramoniamStability and Switchability in Recurrent Neural Networks
MS, University of Cincinnati, 2008, Engineering : Computer Science

Artificial Neural Networks (ANNs) are being extensively researched for their wide range of applications. Among the most important is the ability of a type of ANNs—recurrent attractor networks—to work as associative memories. The most common type of ANN used for associative memory is the Hopfield network, which is a fully connected network with symmetric connections. There have been numerous attempts to improve the capacity and recall quality of recurrent networks, with the focus primarily on the stability of the stored attractors, and the network's convergence properties. However, the ability of a recurrent attractor network to switch between attractors is also an interesting property, if it can be harnessed for use. Such switching can be useful as a model of switching between context-dependent functional networks thought to underlie cognitive processing.

In this thesis, we design and develop a stable-yet-switchable (SyS) network model which provides an interesting combination of stability and switchability. The network is stable under random perturbations, but highly sensitive to specific targeted perturbations which cause it to switch attractors. Such functionality has previously been reported in networks with scale-free (SF) connectivity. We introduce networks with two regions: A densely connected core region, and a sparsely connected and larger periphery. We show that these core-periphery (CP) networks are better for providing a combination of stability and targeted switching than scale-free networks. We develop and validate a specific approach to switching between attractors in a targeted way. The CP and SF models are also compared with each other and with randomly connected homogeneous networks.

Committee:

Dr. Ali Minai (Advisor); Dr. Raj Bhatnagar (Committee Member); Dr. Anca Ralescu (Committee Member)

Subjects:

Computer Science; Engineering

Keywords:

recurrent neural networks; core-periphery networks; switchability; switching between attractors; stability and switchability