This paper explores the rapid growth of on-device AI and the crucial role of heterogeneous computing in addressing the increasing performance demands and power challenges of edge computing.