Sparse and efficient deep neural network optimisation for embedded intelligence

Add to your list(s) Download to your calendar using vCal

• Jia Bi (Southampton)
• Monday 02 November 2020, 16:00-17:00
• Teams.

If you have a question about this talk, please contact George A Constantinides.

Many emerging applications are driving the development of Artificial Intelligence (AI) for embedded systems that require AI models to operate in resource constrained environments. Desirable characteristics of these models are reduced memory, computation and power requirements, that still deliver powerful performance. Deep learning has evolved as the state-of-the-art machine learning paradigm becoming more widespread due to its power in exploiting large datasets for inference. However, deep learning techniques are computationally and memory intensive, which may prevent them from being deployed effectively on embedded platforms with limited resources and power budgets. To address this problem, I focus on improving the efficiency of these algorithms. I show that improved compression and optimization algorithms can be applied to the deep learning framework from training through inference to meet this goal.

This presentation introduces a new compression method that significantly reduces the number of parameters requirements of deep learning models by first-order optimization and sparsity-inducing regularization. This compression method can reduce model size by up to 300X without sacrificing prediction accuracy. To improve the performance of deep learning models, optimization techniques become more important, especially in large-scale applications. As a result, I proposed a new first-order optimization algorithm that improve over existing methods by controlling the variance of the gradients, determining optimal batch sizes, scheduling adaptive learning rates, and balancing biased/unbiased estimations of the gradients, which can reduce the up to 75% training time.

https://teams.microsoft.com/l/meetup-join/19%3aa0c52689bbb349f78ca7c2c5b714c910%40thread.tacv2/1600333873217?context=%7b%22Tid%22%3a%222b897507-ee8c-4575-830b-4f8267c3d307%22%2c%22Oid%22%3a%22d306b47e-ec3d-4bc2-8098-e7a639b10a4c%22%7d

This talk is part of the CAS Talks series.

This talk is included in these lists:

• Andrea Picciau's list
• CAS Talks
• Circuits and Systems Group: Internal Seminars
• Teams

Note that ex-directory lists are not shown.