Reinventing language models for tiny devices
Infineon's team tackled a challenge at the frontier of edge AI: developing a functional language model that operates within the strict constraints of devices with less than 32 MB of memory. While typical language models span billions of parameters and require substantial computational resources, the team set out to create a model family capable of natural language understanding and task execution entirely on-device. Without cloud connectivity or external processing. Their target platform, Infineon's PSOC™ Edge, represents a new class of ultra-low-power microcontrollers designed for wearables, IoT devices, and embedded systems where energy efficiency and privacy are paramount.
“The conventional paradigm in deep learning emphasizes scale - larger datasets, bigger models, more compute,” said Niall Lyons, Senior Staff Deep Learning Researcher at Infineon. “This project required us to fundamentally rethink that approach. We had to distill language modeling down to its essential components and reimagine what's possible within extreme resource constraints.”
Access to sufficient compute was a major bottleneck. Their internal GPU cluster had limited availability and accessibility. The typical alternative would have been to purchase an additional cluster, potentially costing them hundreds of thousands of dollars, just to get started.
Iterating rapidly on limited hardware, saving a year of development time
Over 4–5 months, the team designed a foundational model family from scratch, carefully optimizing data preprocessing, training, and fine-tuning under strict memory and compute constraints. The family includes three variants: an 8-million-parameter model for always-listening applications with an ultra-small footprint, a 13-million-parameter model offering a balanced footprint with lower energy and memory requirements, and a 25-million-parameter model delivering a full assistant experience. All optimized for MCU-level power consumption. They started with short sequences and gradually scaled to longer ones, then applied RLHF methods to ensure the models produced accurate and relevant answers through continuous human feedback loops.
“When Lightning came in, we were able to get up and running the very first day we got access to it, and we haven’t looked back ever since” shared Niall. “Getting access to 8 GPUs, for example, in under 10 minutes, was brilliant,” Niall added.
On-demand GPU access enabled the team to overcome the resource constraints of their internal cluster. Checkpointing provided the flexibility to pause and resume experiments seamlessly, while comprehensive experiment tracking facilitated comparison across hundreds of runs to identify optimal approaches. Scalable multi-GPU support allowed them to expand experiments effortlessly without infrastructure reconfiguration, and the cloud-hosted environment eliminated the overhead of local software installation and maintenance. Together, these capabilities saved the team months of engineering time, allowing them to concentrate entirely on model design and rapid iteration.
“Having the data prep rooms there where you could run the whole dataset pre-processing pipeline and be able to test it on 4 CPUs, make sure it works, then switch to 8 CPUs. Seeing how that works and then pushing it all the way to that 96 CPU range, and then after to GPUs, helped streamline our process,” shared Niall.
Maximizing GPU efficiency without extra code
One of the unexpected benefits the team discovered was comprehensive visibility into GPU performance metrics. “The platform provided real-time monitoring of all hardware resources - GPU utilization, temperature, memory allocation, and VRAM consumption - without requiring any instrumentation code,” Niall explained. “I could immediately identify whether a GPU was operating at 99% utilization with near-maximum VRAM capacity, or if significant headroom remained. This telemetry enabled us to optimize resource utilization by safely increasing batch sizes or scaling model complexity, eliminating the need for conservative estimates or trial-and-error approaches.”
This real-time insight into GPU usage removed a layer of overhead the team normally would have spent coding for, freeing them to focus on experiments and optimization. The ability to monitor GPU metrics across runs meant that each experiment was more efficient, and resources were fully leveraged, something the team had never experienced before on other platforms.
Delivering a model that fits in 32 MB
The result is a fully functional language model family that operates on Infineon's PSOC™ Edge with under 32 MB of memory. Development that might have required 1.5–2 years on traditional infrastructure was completed in just 4–5 months. The models deliver lightweight inference with real-time on-device responses while maintaining exceptional energy efficiency.
“The platform fundamentally accelerated our development timeline,” Niall reflected. “Access to scalable compute resources and rapid iteration capabilities enabled us to compress what would have been 1.5–2 years of development into 4–5 months. Without this infrastructure, achieving our ambitious goals for on-device AI within this timeframe simply wouldn't have been feasible.”
Expanding access to edge AI
This breakthrough unlocks new possibilities for AI in energy-efficient, offline devices, empowering small teams, researchers, and innovators to experiment with edge AI on accessible hardware. The PSOC™ Edge platform demonstrates that sophisticated language models can run on resource-constrained devices, bringing AI capabilities to applications previously limited by power, cost, or connectivity constraints. What once required massive compute infrastructure is now achievable on a single edge device.
“This is about democratizing AI for devices that couldn't support it before,” the team explained. “The PSOC™ Edge enables practical, human-centered AI experiences in everyday products. From smart home devices to wearables, without compromising on privacy or energy efficiency. That's what makes this work meaningful.”
Read more case studies and blogs:
• Slalom Accelerates Model Development And Cuts Setup Time Using Lightning AI
• How NNAISENSE Built an AI-Powered Hedge-Fund And Cut Infrastructure Costs By 35%
• View all