Beyond Programming: Working with Physical AI Robots and Humanoids

In his keynote at CES 2025 last month, Nvidia CEO Jensen Huang discussed the potential for physical AI and humanoids as he introduced Cosmos, world foundation models for boosting development of physical AI systems.

“It’s really about teaching the AI, not about generating creative content, but teaching the AI to understand the physical world,” Huang said.

Humanoid robots are a type of physical AI that interact with humans and their environment. They’re already used in fields such as healthcare, education and entertainment.

“Physical AI enables autonomous machines like robots and self-driving cars to perceive, understand and perform complex actions in the real world,” Deepu Talla, vice president of robotics and edge AI at Nvidia, told Dice. “This interdisciplinary field combines artificial intelligence, robotics, and advanced engineering. It provides technical professionals from various backgrounds an opportunity to work together and transform the world’s heavy industries and robotics.” 

Early use cases for physical AI have been in manufacturing and logistics, said Pras Velagapudi, CTO of Agility Robotics, which manufactures the robot Digit. Safety issues have been addressed more in industrial and manufacturing than in retail or healthcare.

Although industrial robots can perform tasks like picking up items in a warehouse, they still have limitations. “The dexterity for robots just isn’t there yet,” said UN AI Adviser Neil Sahota. “So that movement limitation does constrain what we’re able to do. That’s one of the reasons why we don’t have a full Rosie the robot yet.”

Although physical AI is in the early stages, Talla sees adoption increasing significantly in 2025 and beyond. Nvidia’s humanoid robotics partners are adopting physical AI to power their humanoid robots to perform a multitude of tasks in industrial environments.

Right now, robots in industrial environments perform tasks such as carrying things around. However, Cosmos and Nvidia’s Project Groot, a foundational model for humanoid robots, could expand the capabilities of robots, according to George Chowdhury, an analyst at ABI Research. However, Groot is still a proof of concept, he said.

“The ambition through physical AI and Groot in particular and Cosmos, which was announced at CES, is to have these robots doing complex, dexterous tasks, the kind of things that only humans can do, not just carry things around,” Chowdhury added. “It is my belief that the only way they will demonstrate real value is if they begin doing things that previously only humans could do in terms of complexity and dexterity.”

More complex tasks for physical AI robots include assembling parts (like in automotive manufacturing) and “pick and place,” including picking up small pieces, Chowdhury said: “But I haven’t seen that commercially mature or commercially deployed yet.”

What Tech Professionals Should Learn for Physical AI

Chowdhury describes the role of robotics engineers as the “Swiss Army knife of software engineers or computer science.” They need a range of technologies including embedded C, C++, Python, hex and binary as well as the communication protocols for these components.

Although programming is a key part of upskilling for tech professionals, for robotics and physical AI, it’s all about training the robots, according to Sahota: “We don’t program AI; we teach it… And what we’ve learned is we need to have more people that are stellar and understand good training strategies.”

The idea with physical AI is learning the type of data you need to collect and how to tune the model to the desired behavior in the robot, rather than just programming a system to do what the programmer wants.

“They have to know all of these technologies, coding languages and frameworks that any individual, a software engineer, might base their entire career around, but the robotics engineer must know how to use all of them,” Chowdhury said.

Chowdhury added that tech professionals should learn User Datagram Protocol (UDP, Transmission Control Protocol (TCP) and Checksum, a way to check that messages received through UDP are valid. Also, learn simulation software such as ROS Gazebo.

Tech professionals should become familiar with simultaneous localization and mapping (SLAM). “[SLAM] is how robots use their sensors, the IMUs and the gyroscopes aboard the robot and the images that it’s seeing through cameras or LIDAR to build a map of its environment and navigate that environment,” Chowdhury said.

Ultimately, algorithms drive this new world. “It is independent robots that need to be able to localize, machine vision, drones, simulation, C++ and Python,” Chowdhury explained. “The old world is ladder code, programming PLCs. It’s programming robot controllers, and it’s working in the proprietary programming languages and simulation software that established vendors would sell.”

Those ‘established vendors’ include General Electric, Rockwell Automation, Omron and Schneider Electric.

Chowdhury recommends learning the Robot Operating System (ROS), a set of software libraries and tools for building robot applications.

To learn physical AI, developers can check out the Nvidia Deep Learning Institute (DLI), which offers training in tech that powers physical AI, including accelerated computing, deep learning graphics and simulation, and generative AI.

Sahota recommends that tech professionals understand the convergence of cognitive science, AI and the metaverse. That means combining IoT sensors and AI systems, and involves “bringing these different emerging technologies and sciences together to create that exponential growth factor,” he said.

As you progress on your robotics journey, study how physical AI tools work by experimenting with open tools like Google DeepMind’s MuJoCo, an open-source physics engine for research and development in robotics. Learn what type of data these systems entail and how to go about collecting it, Velagapudi advised.

Tech professionals could master similar models in both physical AI and generative AI, according to Velagapudi: “They’re basically overlapping in the sense that generative AI describes a style of AI model that is sometimes part of what’s used in physical AI… In a lot of cases, the new models that are being used in physical AI are coming from the model constructions of generative AI. They are the same model types.”

Earlier generative AI models were trained on text, while physical AI were trained on robot execution data and images from a robot’s sensors. “Now you have a generative AI model that has a bunch of robot-centric data helping to inform what that output should be,” Velagapudi added.

Skills to Learn in Physical AI

Nvidia’s Talla suggests the following skills to learn in physical AI:

• AI and machine learning (ML): These skills include acquiring an understanding of accelerated computing, AI and deep learning. It also involves grasping generative AI and multimodal foundation models. Learn how to “fine-tune models with domain specific data” and “understand what it takes to collect, curate large-scale real and synthetic data for training foundation models,” Talla said.

• Programming: Learn programming language and frameworks for AI, data analysis, embedded systems and robotics. These languages include Python and C++.

• Robotics: Study robotic design, reinforcement and imitation learning and simulation with autonomous mobile robots (AMRs), manipulator arms and humanoids, Talla advised. Robotics training also entails “proficiency in building low-latency, high-throughput hardware that powers these AI models and allows robots to perceive, navigate and perform gross as well as fine motor skills.”

• Simulation and modeling: Tech pros should gain “proficiency in using simulation environments developing physically accurate digital twins ranging from factories to large cities,” Talla said. Also study how to use software-in-loop (SIL) and hardware-in loop (HIL) methods to test and optimize these autonomous systems.

• Orchestration and Management: Construct large automation tools for managing a robot fleet that ranges from building software to managing route optimization to over-the-air (OTA) software updates. Also develop “orchestration tools for scaling, moving robotics workloads across a hybrid infrastructure (e.g. robotics DevOps),” Talla advised.