In 2024, the tech industry is diversifying, with edge intelligence, RISC-V and small chips emerging as the three main directions to watch.
One direction in the development of edge intelligence technology is indicated in a recently released research paper from Apple, which explores the problem of efficiently running large language models (LLMs) on internally-constrained devices by storing the model parameters in flash memory and loading them into dynamic random access memory (DRAM) on demand. In this paper, Apple's research team presents their approach: constructing an inference cost model that matches the behavior of flash memory in order to optimize in two key areas: first, reducing the amount of data transferred from flash; and second, reading data in larger and contiguous blocks. To this end, they introduce two techniques: one is called "windowing", which strategically reduces data transfers by reusing previously activated neurons, and the other is "row and column packing", which is used to increase the size of data blocks read from flash memory.
The application of these techniques not only improves the efficiency of running large language models on edge devices, but also demonstrates the flexibility and potential of edge intelligence technologies to adapt to different hardware constraints. With the deepening of this type of research, we can foresee that edge intelligence will play an increasingly important role in enhancing device intelligence and optimizing user experience.
The paper mentions, "Taken together, these approaches are able to run much larger scale models with only half the DRAM capacity, and improve inference speed by 4 to 5 times and 20 to 25 times compared to direct loading methods employed on CPUs and GPUs, respectively. Together, our knowledge of sparsity, context-adaptive loading strategies, and design optimized for hardware open the way for efficient inference of large language models on memory-constrained devices."
This means that the overall trend in the industry is to deploy more machine learning models and inference tasks in edge computing environments. In this way, faster and more efficient data processing and analytics can be performed closer to the source of the data, thus improving overall intelligent processing power and responsiveness.
In August 2023, several companies announced that they would be joining forces to form a new company, one that aims to accelerate the rollout of commercialization of RISC-V hardware on a global scale. In recent weeks, the new company has been officially named Quintauris, and Alexander Kocher has been announced as CEO, with headquarters in Munich, Germany, and investors behind the company including such well-known names as Bosch, Infineon, Nordic Semiconductor, NXP Semiconductors, and Qualcomm Technologies.
Although Quintauris' official website is not yet rich in content, it already clearly states the company's direction and goals: "Quintauris will be a one-stop platform to promote RISC-V based product compatibility, provide reference architectures, and help to establish solutions for widespread use in the industry. The company will initially focus on applications in the automotive industry, but in the future plans to gradually expand into other areas such as mobile devices and the Internet of Things (IoT)."
Meanwhile, interest has been expressed in the relationship between Quintauris and the RISC-V International Association, which is headquartered in Zurich, Switzerland, and maintains and manages the specification of the RISC-V instruction set architecture. Quintauris, on the other hand, is likely to be an important resource and partner for developers who need off-the-shelf reference boards and systems for customized development. The collaboration between the two will undoubtedly have a positive impact on the development and adoption of RISC-V technology.
Chiplets have become a hot trend in the industry over the last year, offering an alternative to avoiding the need for a single, large chip using state-of-the-art process technology in the face of the enormous computing power demanded by today's machine learning (ML) centric products. As Chuck Sobey, chairman of last year's Die Summit in California, noted, "Die can play a huge role in improving the scalability, modularity and flexibility of chips. But the successful implementation of this concept assumes that product developers can integrate them quickly and cost-effectively. Effective integration platforms require the support of numerous tools. Vendors in all areas need to provide such platforms and support the building of an ecosystem and open source projects to fill the gaps in interfaces and software."
This quote highlights mandrels as an innovative solution to current technological challenges and emphasizes that in order to realize the full potential of mandrels, a concerted effort by the entire industry is required. This includes providing the necessary tools and support to facilitate effective integration between different components to ensure that the development and application of the technology can proceed smoothly.
With the rise of artificial intelligence, devices that were once considered simple in their functionality are now becoming intelligent, a change that has led to an unprecedented number of computing devices that we need to process. To be able to process data from these smart devices in real time, high-performance computing technologies are being widely used. This trend has led to a significant increase in the demand for computing power in hyperlocal areas, compared to the relatively small increase in demand in traditional centralized areas. These diverse and complex computing environments require different solutions and more specialized vendors will emerge.
