information+processing
-
KAIST Proposes a New Way to Circumvent a Long-time Frustration in Neural Computing
The human brain begins learning through spontaneous random activities even before it receives sensory information from the external world. The technology developed by the KAIST research team enables much faster and more accurate learning when exposed to actual data by pre-learning random information in a brain-mimicking artificial neural network, and is expected to be a breakthrough in the development of brain-based artificial intelligence and neuromorphic computing technology in the future.
KAIST (President Kwang-Hyung Lee) announced on the 23rd of October that Professor Se-Bum Paik 's research team in the Department of Brain Cognitive Sciences solved the weight transport problem*, a long-standing challenge in neural network learning, and through this, explained the principles that enable resource-efficient learning in biological brain neural networks.
*Weight transport problem: This is the biggest obstacle to the development of artificial intelligence that mimics the biological brain. It is the fundamental reason why large-scale memory and computational work are required in the learning of general artificial neural networks, unlike biological brains.
Over the past several decades, the development of artificial intelligence has been based on error backpropagation learning proposed by Geoffery Hinton, who won the Nobel Prize in Physics this year. However, error backpropagation learning was thought to be impossible in biological brains because it requires the unrealistic assumption that individual neurons must know all the connected information across multiple layers in order to calculate the error signal for learning.
< Figure 1. Illustration depicting the method of random noise training and its effects >
This difficult problem, called the weight transport problem, was raised by Francis Crick, who won the Nobel Prize in Physiology or Medicine for the discovery of the structure of DNA, after the error backpropagation learning was proposed by Hinton in 1986. Since then, it has been considered the reason why the operating principles of natural neural networks and artificial neural networks will forever be fundamentally different.
At the borderline of artificial intelligence and neuroscience, researchers including Hinton have continued to attempt to create biologically plausible models that can implement the learning principles of the brain by solving the weight transport problem.
In 2016, a joint research team from Oxford University and DeepMind in the UK first proposed the concept of error backpropagation learning being possible without weight transport, drawing attention from the academic world. However, biologically plausible error backpropagation learning without weight transport was inefficient, with slow learning speeds and low accuracy, making it difficult to apply in reality.
KAIST research team noted that the biological brain begins learning through internal spontaneous random neural activity even before experiencing external sensory experiences. To mimic this, the research team pre-trained a biologically plausible neural network without weight transport with meaningless random information (random noise).
As a result, they showed that the symmetry of the forward and backward neural cell connections of the neural network, which is an essential condition for error backpropagation learning, can be created. In other words, learning without weight transport is possible through random pre-training.
< Figure 2. Illustration depicting the meta-learning effect of random noise training >
The research team revealed that learning random information before learning actual data has the property of meta-learning, which is ‘learning how to learn.’ It was shown that neural networks that pre-learned random noise perform much faster and more accurate learning when exposed to actual data, and can achieve high learning efficiency without weight transport.
< Figure 3. Illustration depicting research on understanding the brain's operating principles through artificial neural networks >
Professor Se-Bum Paik said, “It breaks the conventional understanding of existing machine learning that only data learning is important, and provides a new perspective that focuses on the neuroscience principles of creating appropriate conditions before learning,” and added, “It is significant in that it solves important problems in artificial neural network learning through clues from developmental neuroscience, and at the same time provides insight into the brain’s learning principles through artificial neural network models.”
This study, in which Jeonghwan Cheon, a Master’s candidate of KAIST Department of Brain and Cognitive Sciences participated as the first author and Professor Sang Wan Lee of the same department as a co-author, will be presented at the 38th Neural Information Processing Systems (NeurIPS), the world's top artificial intelligence conference, to be held in Vancouver, Canada from December 10 to 15, 2024. (Paper title: Pretraining with random noise for fast and robust learning without weight transport)
This study was conducted with the support of the National Research Foundation of Korea's Basic Research Program in Science and Engineering, the Information and Communications Technology Planning and Evaluation Institute's Talent Development Program, and the KAIST Singularity Professor Program.
2024.10.23 View 979 -
Before Eyes Open, They Get Ready to See
- Spontaneous retinal waves can generate long-range horizontal connectivity in visual cortex. -
A KAIST research team’s computational simulations demonstrated that the waves of spontaneous neural activity in the retinas of still-closed eyes in mammals develop long-range horizontal connections in the visual cortex during early developmental stages.
This new finding featured in the August 19 edition of Journal of Neuroscience as a cover article has resolved a long-standing puzzle for understanding visual neuroscience regarding the early organization of functional architectures in the mammalian visual cortex before eye-opening, especially the long-range horizontal connectivity known as “feature-specific” circuitry.
To prepare the animal to see when its eyes open, neural circuits in the brain’s visual system must begin developing earlier. However, the proper development of many brain regions involved in vision generally requires sensory input through the eyes.
In the primary visual cortex of the higher mammalian taxa, cortical neurons of similar functional tuning to a visual feature are linked together by long-range horizontal circuits that play a crucial role in visual information processing.
Surprisingly, these long-range horizontal connections in the primary visual cortex of higher mammals emerge before the onset of sensory experience, and the mechanism underlying this phenomenon has remained elusive.
To investigate this mechanism, a group of researchers led by Professor Se-Bum Paik from the Department of Bio and Brain Engineering at KAIST implemented computational simulations of early visual pathways using data obtained from the retinal circuits in young animals before eye-opening, including cats, monkeys, and mice.
From these simulations, the researchers found that spontaneous waves propagating in ON and OFF retinal mosaics can initialize the wiring of long-range horizontal connections by selectively co-activating cortical neurons of similar functional tuning, whereas equivalent random activities cannot induce such organizations.
The simulations also showed that emerged long-range horizontal connections can induce the patterned cortical activities, matching the topography of underlying functional maps even in salt-and-pepper type organizations observed in rodents. This result implies that the model developed by Professor Paik and his group can provide a universal principle for the developmental mechanism of long-range horizontal connections in both higher mammals as well as rodents.
Professor Paik said, “Our model provides a deeper understanding of how the functional architectures in the visual cortex can originate from the spatial organization of the periphery, without sensory experience during early developmental periods.”
He continued, “We believe that our findings will be of great interest to scientists working in a wide range of fields such as neuroscience, vision science, and developmental biology.”
This work was supported by the National Research Foundation of Korea (NRF). Undergraduate student Jinwoo Kim participated in this research project and presented the findings as the lead author as part of the Undergraduate Research Participation (URP) Program at KAIST.
Figures and image credit: Professor Se-Bum Paik, KAIST
Image usage restrictions: News organizations may use or redistribute these figures and image, with proper attribution, as part of news coverage of this paper only.
Publication:
Jinwoo Kim, Min Song, and Se-Bum Paik. (2020). Spontaneous retinal waves generate long-range horizontal connectivity in visual cortex. Journal of Neuroscience, Available online athttps://www.jneurosci.org/content/early/2020/07/17/JNEUROSCI.0649-20.2020
Profile: Se-Bum Paik
Assistant Professor
sbpaik@kaist.ac.kr
http://vs.kaist.ac.kr/
VSNN Laboratory
Department of Bio and Brain Engineering
Program of Brain and Cognitive Engineering
http://kaist.ac.kr
Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea
Profile: Jinwoo Kim
Undergraduate Student
bugkjw@kaist.ac.kr
Department of Bio and Brain Engineering, KAIST
Profile: Min Song
Ph.D. Candidate
night@kaist.ac.kr
Program of Brain and Cognitive Engineering, KAIST
(END)
2020.08.25 View 10343 -
Scientists Observe the Elusive Kondo Screening Cloud
Scientists ended a 50-year quest by directly observing a quantum phenomenon
An international research group of Professor Heung-Sun Sim has ended a 50-year quest by directly observing a quantum phenomenon known as a Kondo screening cloud. This research, published in Nature on March 11, opens a novel way to engineer spin screening and entanglement. According to the research, the cloud can mediate interactions between distant spins confined in quantum dots, which is a necessary protocol for semiconductor spin-based quantum information processing. This spin-spin interaction mediated by the Kondo cloud is unique since both its strength and sign (two spins favor either parallel or anti-parallel configuration) are electrically tunable, while conventional schemes cannot reverse the sign.
This phenomenon, which is important for many physical phenomena such as dilute magnetic impurities and spin glasses, is essentially a cloud that masks magnetic impurities in a material. It was known to exist but its spatial extension had never been observed, creating controversy over whether such an extension actually existed.
Magnetism arises from a property of electrons known as spin, meaning that they have angular momentum aligned in one of either two directions, conventionally known as up and down. However, due to a phenomenon known as the Kondo effect, the spins of conduction electrons—the electrons that flow freely in a material—become entangled with a localized magnetic impurity, and effectively screen it. The strength of this spin coupling, calibrated as a temperature, is known as the Kondo temperature.
The size of the cloud is another important parameter for a material containing multiple magnetic impurities because the spins in the cloud couple with one another and mediate the coupling between magnetic impurities when the clouds overlap. This happens in various materials such as Kondo lattices, spin glasses, and high temperature superconductors.
Although the Kondo effect for a single magnetic impurity is now a text-book subject in many-body physics, detection of its key object, the Kondo cloud and its length, has remained elusive despite many attempts during the past five decades. Experiments using nuclear magnetic resonance or scanning tunneling microscopy, two common methods for understanding the structure of matter, have either shown no signature of the cloud, or demonstrated a signature only at a very short distance, less than 1 nanometer, so much shorter than the predicted cloud size, which was in the micron range.
In the present study, the authors observed a Kondo screening cloud formed by an impurity defined as a localized electron spin in a quantum dot—a type of “artificial atom”—coupled to quasi-one-dimensional conduction electrons, and then used an interferometer to measure changes in the Kondo temperature, allowing them to investigate the presence of a cloud at the interferometer end.
Essentially, they slightly perturbed the conduction electrons at a location away from the quantum dot using an electrostatic gate. The wave of conducting electrons scattered by this perturbation returned back to the quantum dot and interfered with itself. This is similar to how a wave on a water surface being scattered by a wall forms a stripe pattern. The Kondo cloud is a quantum mechanical object which acts to preserve the wave nature of electrons inside the cloud.
Even though there is no direct electrostatic influence of the perturbation on the quantum dot, this interference modifies the Kondo signature measured by electron conductance through the quantum dot if the perturbation is present inside the cloud. In the study, the researchers found that the length as well as the shape of the cloud is universally scaled by the inverse of the Kondo temperature, and that the cloud’s size and shape were in good agreement with theoretical calculations.
Professor Sim at the Department of Physics proposed the method for detecting the Kondo cloud in the co-research with the RIKEN Center for Emergent Matter Science, the City University of Hong Kong, the University of Tokyo, and Ruhr University Bochum in Germany.
Professor Sim said, “The observed spin cloud is a micrometer-size object that has quantum mechanical wave nature and entanglement. This is why the spin cloud has not been observed despite a long search. It is remarkable in a fundamental and technical point of view that such a large quantum object can now be created, controlled, and detected.
Dr. Michihisa Yamamoto of the RIKEN Center for Emergent Matter Science also said, “It is very satisfying to have been able to obtain real space image of the Kondo cloud, as it is a real breakthrough for understanding various systems containing multiple magnetic impurities. The size of the Kondo cloud in semiconductors was found to be much larger than the typical size of semiconductor devices.”
Publication:
Borzenets et al. (2020) Observation of the Kondo screening cloud. Nature, 579. pp.210-213. Available online at https://doi.org/10.1038/s41586-020-2058-6
Profile:
Heung-Sun Sim, PhD
Professor
hssim@kaist.ac.kr
https://qet.kaist.ac.kr/
Quantum Electron Correlation & Transport Theory Group (QECT Lab)
https://qc.kaist.ac.kr/index.php/group1/
Center for Quantum Coherence In COndensed Matter
Department of Physics
https://www.kaist.ac.kr
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, Republic of Korea
2020.03.13 View 13261 -
Prof. Jae-Kyu Lee Campaigns on "Bright Internet" Worldwide
Professor Jae-Kyu Lee (pictured on the right) from the College of Business at KAIST is one step closer to fulfilling his dream of achieving the “Bright Internet,” a campaign that he first proposed as he became the president of the Association for Information Systems (AIS) in June 2015.
On December 12, 2015, Professor Lee signed a memorandum of understanding (MOU) at a convention center in Fort Worth, Texas, between the AIS and the International Telecommunication Union (ITU)—a specialized agency of the United Nations that is responsible for issues related to information and communication technologies—on a collaborative research and development program to make the Internet safer for everyone.
The MOU pursues building a trusted international information and communication technology (ICT) infrastructure by proposing telecommunication policies, developing international standards, and organizing interdisciplinary conferences.
The Bright Internet is an initiative to protect online users from cyber terrors, privacy breaches, and cyber-crimes. Further, it involves putting accountability to those who initiate or deliver cyber threats, thereby eliminating the possible source of Internet related crimes.
Following the MOU agreement, Prof. Lee delivered a speech on his campaign at the 2015 International Conference on Information Systems and received positive responses from the audience.
The Bright Internet campaign has been selected as visions of various ICT organizations worldwide including the Korea Society of Management Information System and the International Federation of Information Processing. KAIST and Tsinghua University in China adopted it as an academic topic for research and teaching.
Prof. Lee claimed that the Internet should be used in a manner based on the values of trust, ethics, and decorum. He further noted that it is important to build Internet environments that not only protect individuals from cyber threats or attacks, but also hold those who commit online crimes accountable for their actions.
2015.12.28 View 8026 -
Professor Key-Sun Choi Receives the Order of Service Merit Green Stripes from the Korean Government
The award recognizes Professor Choi’s life-long research effort to make Korean language digitally available, both nationally and internationally.
Professor Key-Sun Choi of the School of Computing at KAIST received the Order of Service Merit Green Stripes from the Korean government at the 569th Korean Language Day, held annually to commemorate the invention of the Korean language, Hangeul. The ceremony took place on October 9, 2015, at the Sejong Center in Seoul.
Professor Choi has distinguished himself in the field of natural language processing (NLP), including Korean language. He developed a Korean NLP parser that enabled information processing and data analysis of Korean language, as well as a digital Korean dictionary, contributing to the advancement of Korean language-based information technology.
Professor Choi also led the way to widespread use of Korean natural language in computing by developing and commercializing open source software to process the Korean language.
He has served leading roles in many of the international academic societies and standardization organizations, among others, as the vice president of Infoterm (the International Information Center for Terminology), president of the Asia Federation of Natural Language Processing, vice chair of ISO/TC 37, a technical committee in the International Organization for Standardization (ISO), and a council member for the International Association of Machine Translation.
2015.10.08 View 7576 -
The 9th International Conference on Entertainment Computing Held, Sep 8-11, 2010
The cyber world is no longer an unrealistic place for a contemporary man who spends most of his time in front of a computer nowadays. The entertainment contents industry, which materializes the cyber world, leads the new knowledge economy and is emerging as a new growth engine for high value-added industry.
Professionals in entertainment computing gathered to discuss how to make the cyber space more elaborate and entertaining. The 9th 2010 International Conference on Entertainment Computing (ICEC) was held from September 8 to September 11 at Seoul COEX by KAIST and International Federation for Information Processing (IFIP).
This year’s theme is “Creative and Innovative Science, Computing and Design for Digital and Entertainment Contents in 21C”, with fifteen global leaders of industry-university-institute collaboration speakers including George Joblove (Executive VP of Sony Pictures Technologies), Massimiliano Gasparri (VP of Warner Bros. Advanced Digital Services), Don Marinelli (Executive Producer of Entertainment Technology Center at University of Carnegie Mellon), Keith Devlin (Founding Executive Director of Stanford Media-X and Executive Director of Stanford H-STAR), Roy Ascott (President of Planetary Collegium).
Speeches, paper sessions, workshops, exhibitions on the high-tech digital entertainment industry including computer graphics, cyber reality, telepresence, 3D/4D, mobile games, animation, special effects, robot design, content production and distribution, media art were held at the conference this year.
This event was sponsored by IEEE, ACM, IPS, ADADA, Elsevier, ETRI, SK Telecom, KIISE, KMMS, HCI Korea, KCGS and KCGS.
2010.09.17 View 12598