Advanced Methods of
Quantization, Compression and Learning in Artificial Intelligence
Com-in-AI

About


Artificial intelligence (AI) algorithms are typically used to solve the most challenging problems, achieving state-of-the-art performances and providing a substantial impact on all aspects of human life. However, because of their computational complexity and high number of parallel processing operations, AI algorithms require significant computational resources (processing power and memory), they consume a considerable amount of energy and require powerful hardware (such as GPUs, servers, clouds). All of these constraints, significantly limit their applicability. In recent years, there has been an increasing need for implementation of AI algorithms on devices with limited resources (memory space, processing power and available energy). Therefore, the problem of implementing complex AI algorithms on devices with limited resources is one of the most topical and challenging research problems in the field of AI. This project deals with solving the mentioned problem, where the overall objective is to propose and implement new approaches to reduce the complexity of AI algorithms using advanced quantization and compression methods, based on the strong expertise of the project team in the field. An important research question we will answer is how to make state-of-the-art DNN, with numerous parameters, more compact and efficient. We will exploit the fact that reduced precision of the DNN parameters, i.e. going from 32-bit floating point to a low-bit fixed point, can be achieved by thoughtful application of quantization and compression, which will result in a worthy compression ratio and, accordingly in the reduced storage and energy cost, and computational requirements. Majority of the research directions has been set toward achieving the highest possible compression ratio of DNN parameters without significant accuracy loss. Since this field of research is still in an early stage, significant improvements are possible. This project offers new approaches to advanced quantization and compression of DNN parameters in order to outperform state-of-the-art solutions. Besides their fundamental scientific importance in the field of AI, all theoretical results will also be fully applicable in practice. Hence, an important part of the project concept is to validate all the proposed theoretical results and to examine their applicability. This will be done by implementing all theoretical solutions on the developed DNN, as a part of the target use case. Eventually, all theoretical results will be implemented and validated and their performance will be evaluated within a real industrial application using data from a real industrial environment.

Abstract


Very topical issues present in generally very powerful AI algorithms related to decreasing computational complexity and memory resources that are of particular importance in portable and edge computing devices with limited memory and processing power are the driving force behind this project proposal. Specifically, this project contributes to a new and fastly growing research in the worldwide AI science solving these particular issues by prudent and deliberate application of quantization theory. Due to the topicality of these issues and the fact that related research is still in the early stage and deserves further investigation, our project team will explore and propose innovative methods of compression and quantization of DNN parameters (weights, biases, activations) and deep features. Also, the goal of the project is to develop a state-of-the-art deep neural network model with a high performance not only on the hardware usually used for AI applications but also on devices with limited computational resources and thus enabling them to support energy demanding and memory constraint applications. In order to achieve these goals, the project proposes an integrated approach to quantization and compression of DNN parameters, based on statistical modeling of the data per layers, as well as of the data subsets within layers and adaptation of the quantizers itself on the statistical characteristics of the input data. Moreover, from the exploration of the compression and quantization effects in DNNs vis-à-vis its accuracy, the benchmark of our methodology will be defined. The researchers’ interdisciplinary competence ensures successful development of innovative quantization, compression and learning methodologies that will enable reducing the complexity of AI algorithms and its much wider usage. The results obtained within this project will find a wide range of applications in both academia and industry, particularly in numerous latency-critical services.

Objectives


  • To achieve significant compression ratio regarding the full precision representation with the negligible or tolerant accuracy loss of DNN by novel quantizers and compression techniques for quantization and compression of DNN parameters;
  • To significantly decrease the data amount of DNN’s deep features with novel optimal quantization and compression methods at the same time to preserve high performance (accuracy) of DNN;
  • To increase the efficiency of the clustering and classification algorithms, by designing quantizers for Gaussian mixture model (GMM);
  • To verify developed quantization and compression methods through application based on DNN and data from the real industrial environment (to implement and evaluate all obtained scientific results);
  • To achieve a high level of visibility and knowledge sharing of the project results through a comprehensive set of Dissemination, Exploitation and Communication activities for wide-scale project promotion among wide target groups (scientific community, economy, etc).

Impacts


  • Future research in the field of AI because of offering innovative research approaches, ideas and directions, as well as significant scientific results;
  • Serbian science due to launching new research topics, increasing visibility, reputation and networking capacity of Serbian scientists;
  • Industry 4.0 due to providing more efficient, intelligent and automated industrial systems, by reducing the complexity of AI algorithms and therefore allowing their implementation on edge devices and much wider usage in industrial environments;
  • ELFAK curriculums, which will be enhanced and will therefore provide high-quality workforce to both, the academy and industry.

TEAM


The project team is fully compatible, since all team members have basic knowledge of quantization and coding theory, as well as of neural networks and AI algorithms, which will greatly facilitate their collaboration and communication and will improve work efficiency. Moreover, the project team has a great complementarity since each team member has a unique deep knowledge and strong expertise required for the project realization. Team members have been working successfully at the University of Niš, Faculty of Electronic Engineering, on joint scientific projects and papers for years and therefore know each other’s strengths and possibilities very well, which will give the optimal outputs of this project.
Professor Zoran Perić as a PI has a stunning academic and scientific career, as well as necessary management and administrative skills to lead this project. With years of experience and significant scientific and applicable results, a Full Professor Dejan Ćirić will give great contribution in the activities within this project related to AI (DNN design, application and evaluation), audio signal processing and sound acquisition. The special expertise of Associate Professor Aleksandra Jovanović is design of vector quantizers and also design of power efficient signal constellations. Therefore, it is expected that she will predominantly contribute to the part of the project related to data clustering and classification where the multidimensional space partition is an important issue. Assistant Professor Milan Dinčić has strong expertise in the design of VLC (variable-length coding) compression algorithms as well as in the statistical modeling of input data for DNNs and other AI algorithms. His contribution mostly will be in the domain of compression. Associate Professor Jelena Nikolić will contribute to this project with her expertise in speech compression and its application in machine learning and AI. In particular, she will give great contribution in the project activities related to quantization and compression of DNN parameters, where her expertise can be exploited greatly. Ph.D. student Nikola Vučić as a young researcher brings fresh energy to the project and provides support to the part related to quantization and compression. Ph.D. student Bojan Denić will contribute to the part of the project related to data classification, as it is closely related to his research areas. Professor Vladimir Despotović is an expert in machine learning, natural language processing and fractional calculus. His knowledge and extensive international experience considerably increase the capacity of the team to develop advanced methods of learning in AI. All project objectives will be realized through an interdisciplinary approach and knowledge synergy of all team members. Eventually, team members are carefully selected so that everyone has a precisely defined role in the project.

Full Professor Zoran Perić has an academic and scientific career longer than 30 years. He served as a vice-dean of the Faculty of Electronic Engineering in Niš for 11 years. Right now, professor Perić is a member of the council of the University of Niš. He has been the supervisor of 12 Ph.D. thesis and over 80 master and bachelor thesis, and an author of 310 papers, among them 135 are published in the journals from the SCI/SCIe list (with IF). His excellent research results were awarded by Telenor foundation in 2017, with the prize “Professor Ilija Stojanović” for the contribution in the field of telecommunications in the category of scientific papers published over the previous two years in renowned international journals. Professor Zoran Perić was a reviewer for numerous reputable IEEE and Elsevier journals, as well as a member of the Editorial Board of the journals Elektronika ir Elektrotechnika, Information and Facta Universitatis Series: Electronics and Energetics. He served as a Lead Guest Editor in Information journal (special issue: Signal Processing and Machine Learning) and in the journal Computational Intelligence and Neuroscience (special issue: Advanced Signal Processing and Adaptive Learning Methods) and also as an Editor-in-Chief in the journal Facta Universitatis Series: Electronics and Energetics. During his career, Dr Perić has participated in 8 national projects funded by the Ministry of Science and Technological Development of the Republic of Serbia, 2 bilateral projects between Serbia and Slovakia and in numerous international projects. He was also a PI of many sub-projects realized within the Faculty of Electronic Engineering. Moreover, he has established an international collaboration with numerous colleagues from the USA, Germany, Slovakia and other countries. Professor Perić with his valuable experience definitely has professional and personal abilities to lead this project. During these years, the area of AI has become very familiar to him, which has been proven by the impressive number of scientific papers. He will also bring the project management and administrative skills acquired during plenty of projects and positions.

Full Professor Dejan Ćirić brings into this project research and scientific experience of more than 21 years in the fields of Acoustics, Audio signals, Audio Analytics and AI. He is an author/co-author of more than 130 research papers (18 articles in refereed scientific journals). Professor Ćirić has participated in more than 25 national, international and projects financed by external companies both as a project leader and participant. Several of them are closely connected to the subject of this proposal (e.g., AI-based sound event detection, audio analytics of industrial sound, machine learning approach for sound source radiation control, etc.). He has international experience working as a guest researcher at IRCAM, Paris, France and as a research assistant and Ph.D. student at Acoustics, Department of Electronic Systems, Faculty of Engineering and Science, Aalborg University, Aalborg, Denmark. His strong points relevant for this project also include experience with transfer of scientific research results into commercial products in the AI applications in sound.

Associate Professor Aleksandra Jovanović has strong expertise in signal compression algorithms, signal constellation design and signal detection problems. As a participant of several projects she developed different algorithms and models for signal compression and modulation. She verified the achieved results by publishing over 80 papers (27 of which are published in journals with IF) and one monograph. She was awarded by Telenor foundation in 2017. She received the best paper award Professor Ilija Stojanović for contribution in the field of Telecommunications in the category of scientific papers published over the past two years in renowned international journals. She was a reviewer for reputable journals, conferences and bilateral projects. The knowledge and experience she gained in quantization, compression and detection can be applied in AI to solve the topical problems in classification and clustering in data mining and pattern recognition.

Assistant Professor Milan Dinčić, received 2 Ph.D. degrees in two different areas (Telecommunications and Metrology). He finished his master study with the highest average grade of 10 and was awarded as the best graduate student of the University of Niš. Besides the title in higher education: Assistant Professor, he also received the scientific title: Scientific Associate. He is an author of 26 papers in the journals from the SCI/SCIe list. Dr. Dinčić achieved The Seal of Excellence for a H2020 project. He collaborates with colleagues from the academic community (IHP - Frankfurt (Oder) Germany, Universities of Luxembourg, Maribor, Skopje Macedonia), as well as with people from the industry. He has participated in 4 scientific projects and in one bilateral project between Serbia and Slovenia. Also, he was the project leader of the project SENSORS within The Program for Higher Education Development 2019. He was a reviewer for international scientific journals. For this project, particularly important is his strong expertise of quantization, coding and compression theory, and of acquisition, processing and modeling of sensors’ data (as input data for AI systems), as well as his extensive experience in working with measurement and laboratory equipment.

Associate Professor Jelena Nikolić, contributes to this project with her expertise in the field of quantization and speech compression and with her ability to integrate and apply the broad theoretical knowledge acquired in these fields in cutting edge research in the field of AI. She is an author of 113 papers, 46 in the journals from the SCI/SCIe list. Because of her excellent scientific publications, she was awarded by Telenor foundation in 2017. She received the best paper award Professor Ilija Stojanović for contribution in the field of Telecommunications in the category of scientific papers published over the past two years in renowned international journals. Dr. Jelena Nikolić was a reviewer for numerous reputable journals and distinguished textbooks. She is a member of the Editorial Board of the Journal of Advanced Computer Science & Technology and of the journal Mathematical Problems in Engineering. By participating in 9 projects, she established international scientific collaboration. As she follows the latest trends in science, has a large number of publications, she would significantly contribute to publications of the results planned for this project in reputable journals and conferences.

Nikola Vučić graduated at the Faculty of Electronic Engineering in Niš as B.Sc. and M.Sc. in 2013 and 2014, respectively, with the best marks (average grade 10/10). He is a Ph.D. student mentored by PI. From 2018 he has been working as a Junior Research Assistant at the Faculty of Electronic Engineering in Niš. He has published 14 papers and 4 among them are in journals with IF. During his education, he has been awarded many times with prestigious prizes. Nikola Vučić has been involved in Electrical Engineering Students’ European Association- Local committee Niš (EESTEC LC Niš) for almost 10 years, whereby he got an experience in various tasks and positions (chairperson, contact person, finances, bookkeeping, legal issues, grant projects, public relations, international cooperation, logistics etc.). He is used to the team work in domestic and international projects. He has taken part in many workshops, trainings, conferences and other events both as a participant and organizer. Although a young researcher, he has a wealthy experience from various fields that could be used in this project.

Bojan Denić currently works as a Junior Research Assistant at the Faculty of Electronic Engineering, University of Niš. His main research areas include scalar quantization, signal processing and fractional calculus. He is an author/coauthor of 19 scientific papers, 9 of them are published in international scientific journals with IF. He is engaged as a researcher to one national project funded by the Ministry of Education, Science and Technological Development of the Republic of Serbia, and one bilateral project. He was a reviewer for international scientific journals. Bojan Denić brings into this project the valuable experience achieved in the field of quantization and signal processing that can be applied to the problems of quantization and classification in AI.

Dr Vladimir Despotović currently works as a Postdoctoral Researcher at the University of Luxembourg. Previously he was engaged as Associate Professor at the University of Belgrade and Postdoctoral Researcher at the Paderborn University, Germany. His main research interests include speech/audio/image signal processing, machine learning, natural language processing and fractional calculus. He is an author of 92 papers, out of which 25 in the peer-reviewed journals with IF, one book, two book chapters and one patent. He was a PI of two international projects, member of the management committee of one COST project and engaged as a researcher on three national projects (one as a project coordinator). Vladimir Despotovic was granted an Erasmus Mundus Action 2 postdoctoral fellowship at the Paderborn University, Germany, for research on unsupervised spoken language understanding using machine learning techniques in 2014. During 2009 he received an OeAD scholarship at the Vienna University of Technology, Austria, for research on nonlinear prediction of speech signal. During 2008 and 2007 he was granted scholarships for research on identification of fractional order systems at the University of Kosice, Slovakia, from SAIA and the government of the Slovak Republic, respectively. He brings to this project extensive international experience on the use of machine learning and AI in various signal processing applications, as well as project management skills.

Previous projects related to the project


Members of our project team have participated in several projects closely related to the topic of this project. PI, P2 and P4 have been involved in the project Human-Machine Speech Communication, related to a detailed analysis of all problems in speech to machine communication in both directions with the aim to give machines intelligence and human-like capability to speak and understand speech. Also, PI, P2 and P4 have been involved in the project The Development of Dialogue Systems for Serbian and Other South Slavic Languages considered the development of dialogue systems using artificial neural networks, which is highly correlated with this project. PI, P5 and P6 have taken part in the bilateral project, Fractional calculus approach to machine learning, between Serbia and Slovakia. The project developed a novel approach to machine learning by introducing fractional-order calculus to optimization methods used in machine learning algorithms. P1 was a project leader of recently finished project SONO360 - Smart audio interface financed by the Innovation Fund of the Republic of Serbia. The project dealt with the development of a prototype of smart audio device capable of acquiring sound from 3D space applying direction of arrival (DoA) and beamforming technique using spherical microphone array, as well as detecting and recognizing domestic sound events based on a specific design of a DNN. There is a close relation to this project with regards to DNN design and implementation in the field of sound. The experience and knowledge gained working on those projects are very valuable for the accomplishment of the tasks and goals we have defined in this project.

News


The first project presentation

October 9th 2020

The project Com-in-AI was presented on October 9th 2020 at Kalemegdan (Belgrade) within the Science Fund exposition dedicated to the promotion of the Program for the development of projects in the domain of AI. More details at link.

PROMOTION


1. The first project presentation
2. Procured promotional material

The dissemination, exploitation and communication activities will be carried out continuously throughout the lifetime of the project, but also after the project completion, to ensure long-term effects of the project. The dissemination activities will be focused on promoting the scientific results, the potential usage and also the project in general among a wide scientific community as well as among the target groups relevant for exploitation of the achieved results. Workshops to be organized will target the scientific and the interested non-scientific stakeholders (industry, AI companies, Public authorities at national and local levels, relevant decision makers) to exchange ideas on the scientific, economic and social aspects as well as applicability of the project results and to discuss possible collaboration.

RESULTS


Publications


  1. Zoran Perić, Bojan Denić, Milan Savić and Vladimir Despotović, "Design and analysis of binary scalar quantizer of Laplacian source with applications," Information, vol. 11, no. 11, 501, 18 pages, 2020. https://doi.org/10.3390/info11110501
  2. Zoran Perić, Aleksandar Marković, Nataša Kontrec, Stefan Panić and Petar Spalević, "Novel composite approximation for the Gaussian Q-function," Elektronika Ir Elektrotechnika, vol. 26, no. 5, pp. 33-38, 2020. https://doi.org/10.5755/j01.eie.26.5.26012
  3. Slobodan A. Vlajkov, Aleksandra Ž. Jovanović and Zoran H. Perić, "Improvement of energy efficiency of PAM constellation by applying optimal companding quantization in constellation design," in Proc. of 55th International Scientific Conference on Information, Communication and Energy Systems and Technologies, ICEST 2020, pp. 155-158, Niš, Serbia, September 10-12, 2020.
  4. Slobodan A. Vlajkov, Aleksandra Ž. Jovanović and Zoran H. Perić, "The influence of compression parameter μ on the energy efficiency of PAM constellation based on μ-law companding quantization," in Proc. 55th International Scientific Conference on Information, Communication and Energy Systems and Technologies, ICEST 2020, pp. 159-162, Niš, Serbia, September 10-12, 2020.
  5. Zoran Perić, Jelena Nikolić, Danijela Aleksić and Anastasija Perić, "Symmetric quantile quantizer parameterization for the Laplacian source: qualification for contemporary quantization solutions," Mathematical Problems in Engineering, vol. 2021, Article ID 6647135, 12 pages, 2021. https://doi.org/10.5755/j01.eie.26.5.26012
  6. Zoran Perić, Goran Petković, Bojan Denić, Aleksandar Stanimirović, Vladimir Despotović and Leonid Stoimenov, "Gaussian source coding using a simple switched quantization algorithm and variable length codewords," Advances in Electrical and Computer Engineering, vol. 20, no. 4, pp. 11-18, 2020. https://doi.org/10.4316/AECE.2020.04002
  7. Đorđe Damnjanović, Dejan Ćirić, Miljan Miletić and Dejan Vučić, "Usage of different wavelet families in DC motor sounds feature analysis," in Proc. of 7th International Conference on Electrical, Electronic and Computing Engineering, IcETRAN 2020, pp. 43-48, ISBN: 978-86-7466-852-8, Belgrade, Čačak, Niš, Novi Sad, Serbia, September 28-29, 2020.
  8. Marko Milivojčević and Dejan Ćirić, "Izdvajanje značajnih akustičkih karakteristika motora sa unutrašnjim sagorevanjem," u Zborniku radova 64. konferencije za elektroniku, telekomunikacije, računarstvo, automatiku i nuklearnu tehniku, ETRAN 2020, str. 49-53, ISBN: 978-86-7466-852-8, Beograd, Čačak, Niš, Novi Sad, Srbija, Septembar 28-29, 2020.
  9. Zoran Perić, Nikola Vučić, Milan Dinčić, Dejan Ćirić, Bojan Denić and Anastasija Perić, "Design of uniform scalar quantizer for discrete input signals," in Proc. of 28th Telecommunications Forum (TELFOR), pp. 181-184, ISBN: 978-0-7381-4242-5, Belgrade, Serbia, November, 24-25, 2020.

Knowledge database


Here you can find a list of sorted literature relevant to the project topic:
>>> Link to the list of relevant literature <<<


  • Adress
  • University of Niš
    Faculty of Electronic Engineering
    Aleksandra Medvedeva 14,
    18000 Niš, Serbia
  • Tel
  • +381 (0)18 529367


The Science Fund of the Republic of Serbia

This research was supported by the Science Fund of the Republic of Serbia, 6527104, AI- Com-in-AI.
This website is made with the financial support of the Science Fund of the Republic of Serbia.
For the website's content is only responsible University of Niš, Faculty of Electronic Engineering and the content doesn't represent the attitudes of the Science Fund of the Republic of Serbia.