![]() Right now Eshraghian has only released the code for this smallest model, and he is still training the two larger ones. The first is the smallest scale, at 45 million parameters, close in size to the largest-ever SNN that had been developed up to this point. “It was taking the best of both worlds – the low complexity of sequential models and the performance of self-attention.” In a preprint paper, Eshraghian describes three versions of SpikeGPT. “By coming up with a way to break down that backbone of language models into sequences, we completely squashed down that computational complexity without compromising on the ability of the model to generate language,” Eshraghian said. When trying to combine self-attention with SNNs, there was a similar complexity problem, until Eshraghian and his incoming graduate student Ruijie Zhu developed a technique to feed each data point in the sequence into the SNN model one by one, eliminating the need to do matrix-matrix multiplication. The mathematics behind this requires matrix-matrix multiplication, a complexity which is computationally expensive. Large language models, such as ChatGPT, use a technique called self-attention, taking a sequence of data, such as a string of words, and applying a function to determine how closely each data point is related to each other. But Eshraghian has pioneered methods to circumvent these problems and apply the optimization techniques developed for traditional deep learning for the training of SNNs. Many of the optimization strategies that have been developed for regular neural networks and modern deep learning, such as backpropagation and gradient descent, cannot be easily applied to the training of SNNs because the information inputted into the system is not compatible with the training techniques. Spiking neural networks, however, face their own challenges in the training of the models. This introduces a temporal dimension into the equation, because the functions are concerned with how the neurons behave over time. ![]() Instead of constantly transmitting information throughout the network, as non-spiking networks behave, the neurons in SNNs stay in a quiet state unless they are activated, and therefore spike. Using spikes is a much more efficient way to represent information.” Neural networks in general are based on emulating how the brain processes information, and spiking neural networks are a variation that try to make the networks more efficient. We’re taking an informed approach to borrowing principles from the brain, copying this idea that neurons are usually quiet and not transmitting anything. “Large scale language models rely on ridiculous amounts of compute power, and that’s pretty damn expensive. “Brains are way more efficient than AI algorithms,” Eshraghian said. ![]() Using SNNs for language generation can have huge implications for accessibility, data security, and green computing and energy efficiency within this field. He and two students have recently released the open-sourced code for the largest language-generating SNN ever, named SpikeGPT, which uses 22 times less energy than a similar model using typical deep learning. Language models typically use modern deep learning methods called neural networks, but Eshraghian is powering a language model with an alternative algorithm called a spiking neural network (SNN). But UC Santa Cruz Assistant Professor of Electrical and Computer Engineering Jason Eshraghian created a new model for language generation that can address both of these issues. However, these algorithms are both computationally expensive to run and depend on maintenance from just a few companies to avoid outages. With teacher training, getting-started guides, and lesson plans, this solution is a great way to help build students’ confidence, strengthen their STEAM foundation, and set them up for success.Language generators such as ChatGPT are gaining attention for their ability to reshape how we use search engines and change the way we interact with artificial intelligence.From easy entry lessons to the limitless creative designs, SPIKE Prime engages students regardless of their learning level in thinking critically, analyzing data and solving complex problems with real-world relevance. ![]()
0 Comments
Leave a Reply. |