Eureka! NVIDIA Research Breakthrough Puts New Spin on Robot Learning
- Thread starter bnew
- Start date
1/11
@watneyrobotics
FOLD! FOLD! FOLD!
An overnight timelapse of one of our live deployments teleoperated from over 7000 miles away. Our robots fold 24/7/365 with no human intervention, handling long-tail edge cases with no downtime.
https://video.twimg.com/ext_tw_video/1861138021095219204/pu/vid/avc1/1280x720/sLlF9tfyreiiETtr.mp4
2/11
@watneyrobotics
Teleoperation is hard. Teleoperation with high packet loss, variable latency, and constrained bandwidth is almost impossible. We built our system from the ground up (in Rust) to perform dynamic tasks with unprecedented dexterity in real world environments.
3/11
@watneyrobotics
We are extending our teleoperation infrastructure to enable remote inference for cloud-hosted robot foundation models. Robot policies that had to be invoked on edge can now be run from the cloud with no performance loss.
4/11
@watneyrobotics
We've also collected the world's largest dataset of cloth manipulation in a real world environment... Stay tuned!
5/11
@pranaysuyash
then what does tele operated mean if not human intervention? both of which in the post together don't make sense
6/11
@ziademarcus
which third world country do you use for slave labor to “teleoperate” these “without human intervention”
rust, btw
7/11
@Bunagayafrost
i think you're saying, it's being monitored by humans for safety and edge cases, but it handled folding autonomously without any intervention, right?
8/11
@MikeBravoCharly
How do they fold without a chin.
9/11
@MaxRovensky
> teleoperated
> with no human intervention
bruh
10/11
@NaturallyDragon
"teleoperated" "with no human intervention"
11/11
@sweepwhale
it's coming
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@watneyrobotics
FOLD! FOLD! FOLD!
An overnight timelapse of one of our live deployments teleoperated from over 7000 miles away. Our robots fold 24/7/365 with no human intervention, handling long-tail edge cases with no downtime.
https://video.twimg.com/ext_tw_video/1861138021095219204/pu/vid/avc1/1280x720/sLlF9tfyreiiETtr.mp4
2/11
@watneyrobotics
Teleoperation is hard. Teleoperation with high packet loss, variable latency, and constrained bandwidth is almost impossible. We built our system from the ground up (in Rust) to perform dynamic tasks with unprecedented dexterity in real world environments.
3/11
@watneyrobotics
We are extending our teleoperation infrastructure to enable remote inference for cloud-hosted robot foundation models. Robot policies that had to be invoked on edge can now be run from the cloud with no performance loss.
4/11
@watneyrobotics
We've also collected the world's largest dataset of cloth manipulation in a real world environment... Stay tuned!
5/11
@pranaysuyash
then what does tele operated mean if not human intervention? both of which in the post together don't make sense
6/11
@ziademarcus
which third world country do you use for slave labor to “teleoperate” these “without human intervention”
rust, btw
7/11
@Bunagayafrost
i think you're saying, it's being monitored by humans for safety and edge cases, but it handled folding autonomously without any intervention, right?
8/11
@MikeBravoCharly
How do they fold without a chin.
9/11
@MaxRovensky
> teleoperated
> with no human intervention
bruh
10/11
@NaturallyDragon
"teleoperated" "with no human intervention"
11/11
@sweepwhale
it's coming
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
1/11
@UnitreeRobotics
Unitree B2-W Talent Awakening!
One year after mass production kicked off, Unitree’s B2-W Industrial Wheel has been upgraded with more exciting capabilities.
Please always use robots safely and friendly.
/search?q=#Unitree /search?q=#Quadruped /search?q=#Robotdog /search?q=#Parkour /search?q=#EmbodiedAI /search?q=#IndustrialRobot /search?q=#InspectionRobot /search?q=#IntelligentRobot /search?q=#FoundationModels /search?q=#LeggedRobot /search?q=#WheeledLegs
https://video.twimg.com/ext_tw_video/1871092619549548544/pu/vid/avc1/1920x1080/V7GIVOa5H6spKq5_.mp4
2/11
@_CryptoPete_
3/11
@xsaltwedgex
Uhhhhh… is this real or CGI?!?
4/11
@AEInfinity_
What are potential use cases?
5/11
@JohnFStifter
@sierracatalina1 run
6/11
@dexteryy
Build it tank-sized with an added soul (AGI), and the Tachikoma becomes real.
7/11
@ViralMuzik1989
Black mirror robo dog episode coming IRL next door
8/11
@Soulnebulize
9/11
@7n39_igolnik
Want a big one and drive this to work.
10/11
@pinchthaddeus
how is this possible overnight when there was no evidence of this step change in dexterity before this
11/11
@moral_minority
What are its capabilities beyond driving around?
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@UnitreeRobotics
Unitree B2-W Talent Awakening!
One year after mass production kicked off, Unitree’s B2-W Industrial Wheel has been upgraded with more exciting capabilities.
Please always use robots safely and friendly.
/search?q=#Unitree /search?q=#Quadruped /search?q=#Robotdog /search?q=#Parkour /search?q=#EmbodiedAI /search?q=#IndustrialRobot /search?q=#InspectionRobot /search?q=#IntelligentRobot /search?q=#FoundationModels /search?q=#LeggedRobot /search?q=#WheeledLegs
https://video.twimg.com/ext_tw_video/1871092619549548544/pu/vid/avc1/1920x1080/V7GIVOa5H6spKq5_.mp4
2/11
@_CryptoPete_
3/11
@xsaltwedgex
Uhhhhh… is this real or CGI?!?
4/11
@AEInfinity_
What are potential use cases?
5/11
@JohnFStifter
@sierracatalina1
run6/11
@dexteryy
Build it tank-sized with an added soul (AGI), and the Tachikoma becomes real.
7/11
@ViralMuzik1989
Black mirror robo dog episode coming IRL next door
8/11
@Soulnebulize
9/11
@7n39_igolnik
Want a big one and drive this to work.
10/11
@pinchthaddeus
how is this possible overnight when there was no evidence of this step change in dexterity before this
11/11
@moral_minority
What are its capabilities beyond driving around?
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
Nvidia unveils Isaac GR00T blueprint to accelerate humanoid robotics
Nvidia announced the Isaac GR00T blueprint to accelerate humanoid robotics development using large synthetic datasets.
venturebeat.com
Nvidia unveils Isaac GR00T blueprint to accelerate humanoid robotics
Dean Takahashi@deantak
January 6, 2025 8:08 PM
Nvidia Isaac GR00T makes it easier to design humanoid robots.
Image Credit: Nvidia
Nvidia has announced an Isaac GR00T blueprint to accelerate humanoid robotics development.
At Nvidia CEO Jensen Huang’s CES 2025 keynote, Huang said that Isaac GR00T workflows for synthetic data and Nvidia Cosmos world foundation models will supercharge development of general humanoid robots.
Robots on the march
Over the next two decades, the market for humanoid robots is expected to reach $38 billion. To address this significant demand, particularly in the industrial and manufacturing sectors, Nvidia is releasing a collection of robot foundation models, data pipelines and simulation frameworks to accelerate next-generation humanoid robot development.
The Nvidia Isaac GR00T blueprint for synthetic motion generation helps developers generate exponentially large synthetic motion data to train their humanoids using imitation learning.
Imitation learning — a subset of robot learning — enables humanoids to acquire new skills by observing and mimicking expert human demonstrations. Collecting these extensive, high-quality datasets in the real world is tedious, time-consuming and often prohibitively expensive.
Implementing the Isaac GR00T blueprint for synthetic motion generation allows developers to easily generate exponentially large synthetic datasets from just a small number of human demonstrations.
Starting with the GR00T-Teleop workflow, users can tap into Apple Vision Pro to capture human actions in a digital twin. These human actions are mimicked by a robot in simulation and recorded for use as ground truth.
The GR00T-Mimic workflow then multiplies the captured human demonstration into a larger synthetic motion dataset. Finally, the GR00T-Gen workflow, built on the Nvidia Omniverse and Nvidia Cosmos platforms, exponentially expands this dataset through domain randomization and 3D upscaling.
The dataset can then be used as an input to the robot policy, which teaches robots how to move and interact with their environment effectively and safely in Nvidia Isaac Lab, an open-source and modular framework for robot learning.
World foundation models narrow the sim-to-real gap
Also at CES, Nvidia announced Cosmos, a platform featuring a family of open, pretrained world foundation models purpose-built for generating physics-aware videos and world states for physical AI development. It includes autoregressive and diffusion models in a variety of sizes and input-data formats. The models were trained on 18 quadrillion tokens, including 2 million hours of autonomous driving, robotics, drone footage and synthetic data.
In addition to helping generate large datasets, Cosmos can reduce the simulation-to-real gap by upscaling images from 3D to real. Combining Omniverse — a developer platform of application programming interfaces and microservices for building 3D applications and services — with Cosmos is critical, because it helps minimize potential hallucinations commonly associated with world models by providing crucial safeguards through its highly controllable, physically accurate simulations.
An expanding ecosystem
Collectively, Nvidia Isaac GR00T, Omniverse and Cosmos are helping physical AI and humanoid innovation take a giant leap forward. Major robotics companies including Boston Dynamics and Figure have started adopting and demonstrating results with Isaac GR00T.
Humanoid software, hardware and robot manufacturers can apply for early access to Nvidia’s humanoid robot developer program.
1/11
@TheHumanoidHub
Jensen, at the CES 2025 stage with 14 humanoid robots standing in the background, announced NVIDIA Isaac GR00T Blueprint.
It's a simulation workflow for synthetic motion generation, enabling developers to create large datasets for training humanoids using imitation learning.
https://video.twimg.com/ext_tw_video/1876489889535180800/pu/vid/avc1/1280x720/Vjmd5uZd6tgoGvx_.mp4
2/11
@victor_explore
bet those humanoids still can't fold laundry though
3/11
@BriscoeCrainIV
It’s happening so fast!
4/11
@HunterReveur
Impressive. I’m now wondering if we’ll see a public release by summer’25.
5/11
@kianerfaan
iron man 2 vibes
6/11
@Mnewbis
Straight up thought this was William shatner for a second
7/11
@LawStud0842619
Why no @Tesla_Optimus ?
8/11
@EERandomness
Interesting that Optimus was not in the lineup?
9/11
@BLUECOW009
How do I get this developer kit?
10/11
@argoexp
ARBE anyone? Nvidia just invested in them yesterday
11/11
@__plotnikova
Really impressive how fast this is happening
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@TheHumanoidHub
Jensen, at the CES 2025 stage with 14 humanoid robots standing in the background, announced NVIDIA Isaac GR00T Blueprint.
It's a simulation workflow for synthetic motion generation, enabling developers to create large datasets for training humanoids using imitation learning.
https://video.twimg.com/ext_tw_video/1876489889535180800/pu/vid/avc1/1280x720/Vjmd5uZd6tgoGvx_.mp4
2/11
@victor_explore
bet those humanoids still can't fold laundry though
3/11
@BriscoeCrainIV
It’s happening so fast!
4/11
@HunterReveur
Impressive. I’m now wondering if we’ll see a public release by summer’25.
5/11
@kianerfaan
iron man 2 vibes
6/11
@Mnewbis
Straight up thought this was William shatner for a second
7/11
@LawStud0842619
Why no @Tesla_Optimus ?
8/11
@EERandomness
Interesting that Optimus was not in the lineup?
9/11
@BLUECOW009
How do I get this developer kit?
10/11
@argoexp
ARBE anyone? Nvidia just invested in them yesterday
11/11
@__plotnikova
Really impressive how fast this is happening
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
1/11
@TheHumanoidHub
NVIDIA just introduced Cosmos, a platform for world foundation models designed for robotics.
⦿ It features advanced tokenizers, an AI-accelerated data pipeline, and integration with NVIDIA Omniverse. Humanoid makers 1X, Figure, and Agility are among the first to adopt Cosmos.
⦿ Cosmos generates synthetic, physics-based data, accelerating model training and customization.
⦿ It also features a CUDA-accelerated data processing pipeline that enables developers to process, curate, and label 20 million hours of videos in 14 days using the NVIDIA Blackwell platform.
https://video.twimg.com/ext_tw_video/1876485245312339968/pu/vid/avc1/1280x720/Res7shCLkFbL1Vsm.mp4
2/11
@TheHumanoidHub
Technical Blog:
NVIDIA Launches Cosmos World Foundation Model Platform to Accelerate Physical AI Development
3/11
@Brenten55
This Blew Me Away !!!
Entire video:
https://invidious.poast.org/live/k82RwXqZHY8?si=XBzhsKBcFFgyfCJK
4/11
@lwasinam
Speaking robot: Our new AI model translates vision and language into robotic actions
5/11
@steve_ike_
NVIDIA is cooking! Wonder if this helps some of the automakers who have fallen behind in self driving tech catch up?
6/11
@leo_grundstrom
Can this be used for video generation?
7/11
@robot_machines
This is the foreseeable future of robotics.
8/11
@AppyPieInc
Game-changer for robotics! Cosmos is setting a new standard for scalable training and deployment. Excited to see how pioneers like 1X and Agility leverage this!
9/11
@BriscoeCrainIV
I’m buying more NVIDIA in the morning!
10/11
@lateboomer88
This sounds like the pre version of terminator Skynet.
11/11
@saumil_chandira
Enabling both offline and online agents in one keynote
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@TheHumanoidHub
NVIDIA just introduced Cosmos, a platform for world foundation models designed for robotics.
⦿ It features advanced tokenizers, an AI-accelerated data pipeline, and integration with NVIDIA Omniverse. Humanoid makers 1X, Figure, and Agility are among the first to adopt Cosmos.
⦿ Cosmos generates synthetic, physics-based data, accelerating model training and customization.
⦿ It also features a CUDA-accelerated data processing pipeline that enables developers to process, curate, and label 20 million hours of videos in 14 days using the NVIDIA Blackwell platform.
https://video.twimg.com/ext_tw_video/1876485245312339968/pu/vid/avc1/1280x720/Res7shCLkFbL1Vsm.mp4
2/11
@TheHumanoidHub
Technical Blog:
NVIDIA Launches Cosmos World Foundation Model Platform to Accelerate Physical AI Development
3/11
@Brenten55
This Blew Me Away !!!
Entire video:
https://invidious.poast.org/live/k82RwXqZHY8?si=XBzhsKBcFFgyfCJK
4/11
@lwasinam
Speaking robot: Our new AI model translates vision and language into robotic actions
5/11
@steve_ike_
NVIDIA is cooking! Wonder if this helps some of the automakers who have fallen behind in self driving tech catch up?
6/11
@leo_grundstrom
Can this be used for video generation?
7/11
@robot_machines
This is the foreseeable future of robotics.
8/11
@AppyPieInc
Game-changer for robotics! Cosmos is setting a new standard for scalable training and deployment. Excited to see how pioneers like 1X and Agility leverage this!
9/11
@BriscoeCrainIV
I’m buying more NVIDIA in the morning!
10/11
@lateboomer88
This sounds like the pre version of terminator Skynet.
11/11
@saumil_chandira
Enabling both offline and online agents in one keynote
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
1/11
@TheHumanoidHub
EngineAI posted a new video on LinkedIn:
SE01 robot walking around among people at the company's office campus in Shenzhen.
[Quoted tweet]
Finally, a humanoid robot with a natural, human-like walking gait.
Chinese company EngineAI just unveiled their life-size general-purpose humanoid SE01.
https://video.twimg.com/ext_tw_video/1877059859025375236/pu/vid/avc1/720x720/Iu_smDs6rckRXLUc.mp4
https://video.twimg.com/ext_tw_video/1849351674361757696/pu/vid/avc1/1280x720/D429gHgPIpumy_IL.mp4
2/11
@anatomyumea
3/11
@TheHumanoidHub
Optional Beyonce walk upgrade, but the knees warranty voided.
4/11
@noxlonga
terminator walks freely - mistake number one
5/11
@Dieter75
That thing makes me want to reach for my RPG.
6/11
@joehansenxx
Starting to look scary actually!
7/11
@stlgotmynikeson
Oh boy
8/11
@BlackApple
Impressive
9/11
@LarryPanozzo
“Return to your homes.
A curfew is in effect.
Return to your homes.
A curfew is in effect.”
10/11
@fmh_iii
Is it possible to give that robot some shoes? It would soften the impact and reduce noise. Would SE01 still be able to stabilize properly with shoes on?
11/11
@J65516006
@quality_scents
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@TheHumanoidHub
EngineAI posted a new video on LinkedIn:
SE01 robot walking around among people at the company's office campus in Shenzhen.
[Quoted tweet]
Finally, a humanoid robot with a natural, human-like walking gait.
Chinese company EngineAI just unveiled their life-size general-purpose humanoid SE01.
https://video.twimg.com/ext_tw_video/1877059859025375236/pu/vid/avc1/720x720/Iu_smDs6rckRXLUc.mp4
https://video.twimg.com/ext_tw_video/1849351674361757696/pu/vid/avc1/1280x720/D429gHgPIpumy_IL.mp4
2/11
@anatomyumea
3/11
@TheHumanoidHub
Optional Beyonce walk upgrade, but the knees warranty voided.
4/11
@noxlonga
terminator walks freely - mistake number one
5/11
@Dieter75
That thing makes me want to reach for my RPG.
6/11
@joehansenxx
Starting to look scary actually!
7/11
@stlgotmynikeson
Oh boy
8/11
@BlackApple
Impressive
9/11
@LarryPanozzo
“Return to your homes.
A curfew is in effect.
Return to your homes.
A curfew is in effect.”
10/11
@fmh_iii
Is it possible to give that robot some shoes? It would soften the impact and reduce noise. Would SE01 still be able to stabilize properly with shoes on?
11/11
@J65516006
@quality_scents
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
1/7
@TheAIVeteran
You've seen it walking through the street today, here is another look at the SE 01 robot by Engine AI.
You can have Gemini 2.0 reverse engineer those actuators in AI studio for fun.
"A product with human-like bionic design, offering exceptional flexibility and powerful joint dynamics. With 32 degrees of freedom throughout its body and a maximum joint torque of up to 330 N・m, it easily handles even the most complex and challenging movements.
Natural Gait, Human-like Walking
Powerful and Sustained Performance
Enhanced Training, Accelerated Evolution
ENGINEAI——Dedicated to developing world-leading general-purpose humanoid robots while continuously accelerating innovation in the embodied intelligence revolution."
-Engine AI
https://video.twimg.com/ext_tw_video/1877081630478839808/pu/vid/avc1/640x360/1ME3CoktGtRaSgFT.mp4
2/7
@TheAIVeteran
One of my favorite robot videos lately. Great music selection. Here's the SE 01 humanoid robot by Engine AI.
When the engineer hits the enter key and the plastic wrapped robot starts walking, he legit has a "wtf" moment. It was their third walking test at speed.
"The SE01 is dedicated to being the most human-like general-purpose humanoid AI Hardware. It features a set of self-developed harmonic joint module and combines reinforcement learning with imitation learning, achieving unprecedented excellence in natural gait through end-to-end neural network models.
The design of the SE01 narrows the gap between humanoid robotics and humans, making the robot more agile. With smooth, swift, and graceful walking, the SE01 leaves behind the stereotypical unnatural gait of past robots. Our relentless pursuit of technical excellence at EngineAI is set to establish new industry standards."
-Engine AI
https://video.twimg.com/ext_tw_video/1877081668823154688/pu/vid/avc1/640x360/JP6PoC8l1gekR80h.mp4
3/7
@TheAIVeteran
Bonus look at Engine AI's PM01 robot, the smaller one.
"EngineAI proudly unveils the PM01, our next-gen lightweight, high-dynamic, open-source humanoid robotic platform. With its interactive display, agile motion, and robust support for secondary development, PM01 is designed to be the most versatile tool for developers worldwide.
PM01 is now available for purchase! We invite developers, researchers, and businesses to explore the future of robotics with PM01. Let’s push the boundaries of what robotics can achieve across different industries and use cases.
ENGINEAI——Dedicated to developing world-leading general-purpose humanoid robots while continuously accelerating innovation in the embodied intelligence revolution."
-Engine AI
https://video.twimg.com/ext_tw_video/1877084513253445632/pu/vid/avc1/1280x720/4iJV1Vi-a3QkcWCC.mp4
4/7
@IOI101IOI101I
I like to think of a future were kids will be in their garage building robots the way we use to build like soapbox cars.
5/7
@TheAIVeteran
Really, if we build robust software now, then robots could end up essentially plug-and-play with their hardware. There are already modular robots that can have different attachments added, such as the Borg robot. So, may not be too far off. There's already lots of robotics toys out there, though.
6/7
@RayZrDev
I wish they'd make another season west world.
7/7
@TheAIVeteran
Soon, we can just make Westworld for real. Maybe without the shooting, though.
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@TheAIVeteran
You've seen it walking through the street today, here is another look at the SE 01 robot by Engine AI.
You can have Gemini 2.0 reverse engineer those actuators in AI studio for fun.
"A product with human-like bionic design, offering exceptional flexibility and powerful joint dynamics. With 32 degrees of freedom throughout its body and a maximum joint torque of up to 330 N・m, it easily handles even the most complex and challenging movements.
Natural Gait, Human-like Walking
Powerful and Sustained Performance
Enhanced Training, Accelerated Evolution
ENGINEAI——Dedicated to developing world-leading general-purpose humanoid robots while continuously accelerating innovation in the embodied intelligence revolution."
-Engine AI
https://video.twimg.com/ext_tw_video/1877081630478839808/pu/vid/avc1/640x360/1ME3CoktGtRaSgFT.mp4
2/7
@TheAIVeteran
One of my favorite robot videos lately. Great music selection. Here's the SE 01 humanoid robot by Engine AI.
When the engineer hits the enter key and the plastic wrapped robot starts walking, he legit has a "wtf" moment. It was their third walking test at speed.
"The SE01 is dedicated to being the most human-like general-purpose humanoid AI Hardware. It features a set of self-developed harmonic joint module and combines reinforcement learning with imitation learning, achieving unprecedented excellence in natural gait through end-to-end neural network models.
The design of the SE01 narrows the gap between humanoid robotics and humans, making the robot more agile. With smooth, swift, and graceful walking, the SE01 leaves behind the stereotypical unnatural gait of past robots. Our relentless pursuit of technical excellence at EngineAI is set to establish new industry standards."
-Engine AI
https://video.twimg.com/ext_tw_video/1877081668823154688/pu/vid/avc1/640x360/JP6PoC8l1gekR80h.mp4
3/7
@TheAIVeteran
Bonus look at Engine AI's PM01 robot, the smaller one.
"EngineAI proudly unveils the PM01, our next-gen lightweight, high-dynamic, open-source humanoid robotic platform. With its interactive display, agile motion, and robust support for secondary development, PM01 is designed to be the most versatile tool for developers worldwide.
PM01 is now available for purchase! We invite developers, researchers, and businesses to explore the future of robotics with PM01. Let’s push the boundaries of what robotics can achieve across different industries and use cases.
ENGINEAI——Dedicated to developing world-leading general-purpose humanoid robots while continuously accelerating innovation in the embodied intelligence revolution."
-Engine AI
https://video.twimg.com/ext_tw_video/1877084513253445632/pu/vid/avc1/1280x720/4iJV1Vi-a3QkcWCC.mp4
4/7
@IOI101IOI101I
I like to think of a future were kids will be in their garage building robots the way we use to build like soapbox cars.
5/7
@TheAIVeteran
Really, if we build robust software now, then robots could end up essentially plug-and-play with their hardware. There are already modular robots that can have different attachments added, such as the Borg robot. So, may not be too far off. There's already lots of robotics toys out there, though.
6/7
@RayZrDev
I wish they'd make another season west world.
7/7
@TheAIVeteran
Soon, we can just make Westworld for real. Maybe without the shooting, though.
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
Robots Are Learning to Conduct Surgery on Their Own by Watching Videos
Just don't tell them about porn.
gizmodo.com
Robots Are Learning to Conduct Surgery on Their Own by Watching Videos
Just don't tell them about porn.
By Thomas Maxwell Published December 30, 2024 | Comments (28)
The artificial intelligence boom is already starting to creep into the medical field through the form of AI-based visit summaries and analysis of patient conditions. Now, new research demonstrates how AI training techniques similar to those used for ChatGPT could be used to train surgical robots to operate on their own.
Researchers from John Hopkins University and Stanford University built a training model using video recordings of human-controlled robotic arms performing surgical tasks. By learning to imitate actions on a video, the researchers believe they can reduce the need to program each individual movement required for a procedure. From the Washington Post:
To be sure, robotics have been used in the surgery room for years now—back in 2018, the “surgery on a grape” meme highlighted how robotic arms can assist with surgeries by providing a heightened level of precision. Approximately 876,000 robot-assisted surgeries were conducted in 2020. Robotic instruments can reach places and perform tasks in the body where a surgeon’s hand will never fit, and they do not suffer from tremors. Slim, precise instruments can spare nerve damage. But robotics are typically guided manually by a surgeon with a controller. The surgeon is always in charge.
The concern by skeptics of more autonomous robots is that AI models like ChatGPT are not “intelligent,” but rather simply mimic what they have already seen before, and do not understand the underlying concepts they are dealing with. The infinite variety of pathologies in an incalculable variety of human hosts poses a challenge, then—what if the AI model has not seen a specific scenario before? Something can go wrong during surgery in a split second, and what if the AI has not been trained to respond?
At the very least, autonomous robots used in surgeries would need to be approved by the Food and Drug Administration. In other cases where doctors are using AI to summarize their patient visits and make recommendations, FDA approval is not required because the doctor is technically supposed to review and endorse any information they produce. That is concerning because there is already evidence that AI bots will make bad recommendations, or hallucinate and include information in meeting transcripts that was never uttered. How often will a tired, overworked doctor rubber-stamp whatever an AI produces without scrutinizing it closely?
It feels reminiscent of recent reports regarding how soldiers in Israel are relying on AI to identify attack targets without scrutinizing the information very closely. “Soldiers who were poorly trained in using the technology attacked human targets without corroborating [the AI] predictions at all,” a Washington Post story reads. “At certain times the only corroboration required was that the target was a male.” Things can go awry when humans become complacent and are not sufficiently in the loop.
Healthcare is another field with high stakes—certainly higher than the consumer market. If Gmail summarizes an email incorrectly, it is not the end of the world. AI systems incorrectly diagnosing a health problem, or making a mistake during surgery, is a much more serious problem. Who in that case is liable? The Post interviewed the director of robotic surgery at the University of Miami, and this is what he had to say:
The idea that AI will ever be infallible is hard to take seriously when no technology is ever perfect. Certainly, this autonomous technology is interesting from a research perspective, but the blowback from a botched surgery conducted by an autonomous robot would be monumental. Who do you punish when something goes wrong, who has their medical license revoked? Humans are not infallible either, but at least patients have the peace of mind of knowing they have gone through years of training and can be held accountable if something goes wrong. AI models are crude simulacrums of humans, behave sometimes unpredictably, and have no moral compass.
Another concern is whether relying too much on autonomous robots to conduct surgeries could end up resulting in doctors having their own abilities and knowledge atrophy; similar to how facilitating dating through apps results in relevant social skills becoming rusty.
If doctors are tired and overworked—a reason researchers suggested why this technology could be valuable— perhaps the systemic problems causing a shortage should be addressed instead. It has been widely reported that the U.S. is experiencing an extreme shortage of doctors due to the increasing inaccessibility of the field. The country is on track to experience a shortage of 10,000 to 20,000 surgeons by 2036, according to the American Association of Medical Colleges.
1/31
@TheHumanoidHub
CMU researchers, in collaboration with NVIDIA, present ASAP, a two-stage framework for humanoid robot agility.
It pre-trains motion policies on human data, then refines them with real-world corrections using a delta action model, which adjusts for simulation mismatches.
https://video.twimg.com/ext_tw_video/1886678917207220225/pu/vid/avc1/1080x1080/BtA1xXQFtdlZc0fI.mp4
2/31
@TheHumanoidHub
Paper:
https://agile.human2humanoid.com/static/asap.pdf
3/31
@DaveySwift
Robotics need to replicate the human foot to get the balance right
4/31
@AlexCho20159172
One day..the robot will question itself if their memory is real or its someone else as it felt so real.
5/31
@squarecapo
we need the hand skills, object manipulation
6/31
@hvo_e_acc
Humans have effectively created an alien species, another form of life coexisting with us that even surpasses our own intelligence and physical capabilities. It’s not sci-fi. It’s real. And they’re happening at this moment, reshaping civilization in ways we can’t even begin to imagine.
7/31
@karma_x30
NEXT LABEL?
8/31
@Bwallace_esq
@Octavian45678
9/31
@alby13
i did not think that that robot model could pull off the smoothness and complexity of those human motions while staying in-line with the balancing of the individual that made the motions.
10/31
@RealUncleTesla
Meet your new basketball coach
11/31
@jconorgrogan
Who are lakers gonna trade for this bro
12/31
@MMBeckerman
Very soon, within just a few years, you will have a Robot NBA, Robot NFL, Robot MLS, Robot NHL and Robot MLB. Then, you will also have Robot Olympics as well. It's just a matter of time now.
13/31
@oliphant428
Looks better than anything @EAMaddenNFL can do digitally.
14/31
@DevoFlyer
I wonder if Optimus is doing this?
15/31
@xInannaa
Deep reinforcement learning
16/31
@Mikeoftheforest
Nice, do they have a demo of it doing something useful?
17/31
@TeslaSpacexfan
18/31
@RiyanMendonsa
Supercool!
Love how fluid the motions are...
Curious of the resolution of these systems and can they compare with human ones
19/31
@GeneralDegener8
Gentlemen, we can rebuild him. We have the technology.
20/31
@BlakeC
Why LeBron drunk tho?
21/31
@hvo_e_acc
They’re going to make them compete with humans first. Then, gradually, the physical capabilities start surpassing humans, much like what they’re doing now with cognition/intelligence. Ultimately, robots will have their own sports leagues.
22/31
@BriscoeCrainIV
those are some seriously good moves!!
23/31
@jeff_schvey
It’s good they keep the safety noose on the robot incase it decides to go rogue
24/31
@gcharles10x
25/31
@leopardsnow
Human like movements
26/31
@mustbesimo
I want to play with this NOW
27/31
@solvempr0bler
The suii was pretty neat
28/31
@HunterReveur
ALWAYS STRIVING
N
D
PROSPERING!!!!!
29/31
@moneyrobel
I need robot nba action like nascar and fencing combined
30/31
@KusanagiMo67070
It would be interesting to create a street basketball match where real interactions are taken into account the sneaky plays, the loss of balance, the pressure to win in front of others, THE ego, and more ++
31/31
@WWong2022
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@TheHumanoidHub
CMU researchers, in collaboration with NVIDIA, present ASAP, a two-stage framework for humanoid robot agility.
It pre-trains motion policies on human data, then refines them with real-world corrections using a delta action model, which adjusts for simulation mismatches.
https://video.twimg.com/ext_tw_video/1886678917207220225/pu/vid/avc1/1080x1080/BtA1xXQFtdlZc0fI.mp4
2/31
@TheHumanoidHub
Paper:
https://agile.human2humanoid.com/static/asap.pdf
3/31
@DaveySwift
Robotics need to replicate the human foot to get the balance right
4/31
@AlexCho20159172
One day..the robot will question itself if their memory is real or its someone else as it felt so real.
5/31
@squarecapo
we need the hand skills, object manipulation
6/31
@hvo_e_acc
Humans have effectively created an alien species, another form of life coexisting with us that even surpasses our own intelligence and physical capabilities. It’s not sci-fi. It’s real. And they’re happening at this moment, reshaping civilization in ways we can’t even begin to imagine.
7/31
@karma_x30
NEXT LABEL?
8/31
@Bwallace_esq
@Octavian45678
9/31
@alby13
i did not think that that robot model could pull off the smoothness and complexity of those human motions while staying in-line with the balancing of the individual that made the motions.
10/31
@RealUncleTesla
Meet your new basketball coach
11/31
@jconorgrogan
Who are lakers gonna trade for this bro
12/31
@MMBeckerman
Very soon, within just a few years, you will have a Robot NBA, Robot NFL, Robot MLS, Robot NHL and Robot MLB. Then, you will also have Robot Olympics as well. It's just a matter of time now.
13/31
@oliphant428
Looks better than anything @EAMaddenNFL can do digitally.
14/31
@DevoFlyer
I wonder if Optimus is doing this?
15/31
@xInannaa
Deep reinforcement learning
16/31
@Mikeoftheforest
Nice, do they have a demo of it doing something useful?
17/31
@TeslaSpacexfan
18/31
@RiyanMendonsa
Supercool!
Love how fluid the motions are...
Curious of the resolution of these systems and can they compare with human ones
19/31
@GeneralDegener8
Gentlemen, we can rebuild him. We have the technology.
20/31
@BlakeC
Why LeBron drunk tho?
21/31
@hvo_e_acc
They’re going to make them compete with humans first. Then, gradually, the physical capabilities start surpassing humans, much like what they’re doing now with cognition/intelligence. Ultimately, robots will have their own sports leagues.
22/31
@BriscoeCrainIV
those are some seriously good moves!!23/31
@jeff_schvey
It’s good they keep the safety noose on the robot incase it decides to go rogue
24/31
@gcharles10x
25/31
@leopardsnow
Human like movements
26/31
@mustbesimo
I want to play with this NOW
27/31
@solvempr0bler
The suii was pretty neat
28/31
@HunterReveur
ALWAYS STRIVING
N
D
PROSPERING!!!!!
29/31
@moneyrobel
I need robot nba action like nascar and fencing combined
30/31
@KusanagiMo67070
It would be interesting to create a street basketball match where real interactions are taken into account the sneaky plays, the loss of balance, the pressure to win in front of others, THE ego, and more ++
31/31
@WWong2022
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
1/11
@borgrobotic
Our first mission at Borg is to help restore America's industrial and manufacturing strength with the help of our Ai integrated robots.
In this video, you'll see our autonomous pallet jack and our modular humanoid robot Borg 01 in action. Borg 01 modular design enables autonomous switching between a bipedal and wheeled base, as well as interchangeability of end effectors.
We're gearing up to launch three robots in late Q2 including the pallet jack, the humanoid Borg 01, and one more yet to be announced. Stay tuned. Built in Detroit.
https://video.twimg.com/amplify_video/1885814332409126912/vid/avc1/1080x608/yKcfa_1CmJ8DHZO8.mp4
2/11
@R2rule1
They look very efficient.
I hope your manufacturing industry can see that walking robot are only for shows, not for working areas.
3/11
@borgrobotic
Agree. We will have a bipedal version but when a wheeled base can be used use one. Efficiency and speed!
4/11
@TheAIVeteran
I've been waiting for this video. It is some of the most efficient factory robot footage I've seen. Well done!
5/11
@borgrobotic
Thanks, yes when you can use a wheeled base use one.
6/11
@Phd1455
Awesome tech. We’ll all know AI has reached its pinnacle when the humanoid is seen ‘flipping off’ the pallet jack for not placing closer to the belt
7/11
@borgrobotic
8/11
@dannyngwsh
Is the base the smallest it can get?
9/11
@borgrobotic
That shown is our wheeled base X, we also have a much smaller wheeled base S.
10/11
@osmaanshah
Whats the expected payload capacity?
11/11
@borgrobotic
Payload for the autonomous pallet jack 2000kg
Payload for humanoid Borg 01 15kg.
These are version 1s next couple versions payload will increase.
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196
@borgrobotic
Our first mission at Borg is to help restore America's industrial and manufacturing strength with the help of our Ai integrated robots.
In this video, you'll see our autonomous pallet jack and our modular humanoid robot Borg 01 in action. Borg 01 modular design enables autonomous switching between a bipedal and wheeled base, as well as interchangeability of end effectors.
We're gearing up to launch three robots in late Q2 including the pallet jack, the humanoid Borg 01, and one more yet to be announced. Stay tuned. Built in Detroit.
https://video.twimg.com/amplify_video/1885814332409126912/vid/avc1/1080x608/yKcfa_1CmJ8DHZO8.mp4
2/11
@R2rule1
They look very efficient.
I hope your manufacturing industry can see that walking robot are only for shows, not for working areas.
3/11
@borgrobotic
Agree. We will have a bipedal version but when a wheeled base can be used use one. Efficiency and speed!
4/11
@TheAIVeteran
I've been waiting for this video. It is some of the most efficient factory robot footage I've seen. Well done!
5/11
@borgrobotic
Thanks, yes when you can use a wheeled base use one.
6/11
@Phd1455
Awesome tech. We’ll all know AI has reached its pinnacle when the humanoid is seen ‘flipping off’ the pallet jack for not placing closer to the belt
7/11
@borgrobotic
8/11
@dannyngwsh
Is the base the smallest it can get?
9/11
@borgrobotic
That shown is our wheeled base X, we also have a much smaller wheeled base S.
10/11
@osmaanshah
Whats the expected payload capacity?
11/11
@borgrobotic
Payload for the autonomous pallet jack 2000kg
Payload for humanoid Borg 01 15kg.
These are version 1s next couple versions payload will increase.
To post tweets in this format, more info here: https://www.thecoli.com/threads/tips-and-tricks-for-posting-the-coli-megathread.984734/post-52211196