How AI Can Save Lives: A Journey Through Human Action Recognition

Imagine a world where emergencies are detected and addressed in real-time, where lives are saved by systems that never tire or miss a moment. Traditional CCTV monitoring often relies on human operators, who may struggle to maintain focus over extended periods. In such cases, critical moments can slip by unnoticed. But what if AI could take the reins?

Advancements in machine learning are making it possible for AI systems to monitor, detect, and respond to emergencies faster and more accurately than human operators. One of the most promising areas in this field is Human Action Recognition (HAR)—training AI to understand human movements and actions in video streams.


The Challenge of Monitoring 

Monitoring CCTV networks is no small task. As the number of cameras increases, the ability to track and analyze video feeds in real-time diminishes. Often, these systems are used only to review incidents after they occur, providing little opportunity for proactive intervention. AI, however, can transform these systems into lifesaving tools by detecting human actions that indicate emergencies as they happen. But understanding human actions is complex. It requires AI to interpret movements over time and in context—something humans do intuitively but machines must be taught.


The Complexity of Human Action Recognition

Human actions are dynamic, involving intricate movements that unfold over time. Recognizing these actions requires analyzing not just individual video frames but their sequence, along with the context in which the actions occur. Training AI models for this purpose demands a significant amount of labeled video data, which is often limited or biased toward certain actions or contexts.

To overcome these challenges, researchers are developing innovative solutions for more robust training and inference.


Scaling Human Action Recognition: WHAM and Action2Motion

To build AI models capable of real-time action recognition, we need to overcome the limitations of existing data. This is where WHAM and Action2Motion come in, offering powerful techniques for creating high-quality training data.


WHAM: Reconstructing World-Grounded Humans with Accurate 3D Motion 

WHAM, a cutting-edge model from CVPR 2024, converts 2D videos into precise 3D motion data. It starts with a sequence of 2D keypoints detected in video frames and encodes them into a motion feature. By integrating image features from a pretrained image encoder, WHAM refines these motion features to estimate 3D human motion, foot-ground contact probability, and global trajectory.

The process involves:

  • Local Motion Decoding: Predicts 3D motion in the camera coordinate system.
  • Trajectory Decoding and Refinement: Estimates and refines global orientation and velocity using camera dynamics and foot-ground contact.

The result? Pixel-aligned, world-grounded 3D human motion that accurately represents how humans move in global space.


Action2Motion: Generating Synthetic Action Data

Once 3D motion data is available, Action2Motion models can generate synthetic sequences of the same action, expanding the dataset. By creating realistic 3D human motion patterns, these models help address the problem of limited labeled data, enabling the AI to learn from diverse examples of the same action.


Data Augmentation: Projecting 3D Back to 2D

To make AI models robust and generalised, the synthetic 3D motion data is projected back into 2D from different angles. This augmentation simulates various camera perspectives, ensuring the AI can recognise actions regardless of viewpoint.


Action Recognition with VideoMAE V2

Once the dataset is enriched with diverse 2D and 3D representations of human actions, the training moves to VideoMAE V2—a state-of-the-art model for video-based action recognition.

VideoMAE V2 excels at capturing temporal and spatial dynamics in video data. By using dual masking techniques, it focuses on the most critical regions of each frame, efficiently learning from 2D videos while interpreting the
m as 3D representations. This mimics human perception, enabling the model to recognize actions with remarkable accuracy.


A Safer Future with AI

By integrating technologies like WHAM, Action2Motion, and VideoMAE V2, we can build AI systems capable of detecting emergencies in real-time. These systems can alert authorities, prevent crises, and ultimately save lives. The potential applications are vast:

  • Smarter surveillance in public spaces.
  • Faster response to medical emergencies.
  • Enhanced safety in workplaces, schools, and homes.

The journey to achieving real-time action recognition isn’t without challenges, but the progress is undeniable. With each breakthrough, we edge closer to a future where AI isn’t just a tool but a lifesaving ally.


The future of emergency response is here, and AI is leading the way.

 

- Presented by Mohammad Khaled Moselmany in the I2SC weekly meeting.

References

 

Popular posts from this blog

Blog: Talking about flood, shit, democracy and trust in the Netherlands

Image
Jianlong stands in front of his poster and explains the experimental setup. In April 2024, I (Jianlong) embarked on my first work trip to the Netherlands. I spent two days at the CWI in Amsterdam 1 presenting a poster and attending workshops on AI, media and democracy , and another day in Utrecht for the annual conference of the Digital Society . The CWI workshops attracted participants working in various sectors, from education and startups to city councils and media organizations. Among the many consensus shared by the stakeholders, one that most strongly resonated with me as a researcher is the scope of responsibilities of academics. “Doing science” should no longer be equated to writing papers and presenting papers; it is just as important to interact with 1) regulators and lawmakers to enable them to make informed decisions about AI and 2) the public to educate them about the limitations of AI. There had also been a lot of chatters about how advances in AI (i.e. language and imag
Read more

Blog: I2SC goes to the UN!

Image
  After working as an intern in the United Nations Information Integrity Unit (previously called the Addressing Mis/Disinformation Unit) for five months, I (Brahmani) finally had a chance to visit the United Nations headquarters located in New York in April 2024.  As I approached the end of my internship, it was great to meet the team I only interacted with virtually until that point. For someone who comes from a purely technical background, it was a new experience to work in a truly multidisciplinary team.   As part of my internship, I worked on developing network analysis techniques to analyse the behaviour of different actors in the information environment, along with testing LLMs for narrative analysis. I've also contributed to a retrospective report on Mali, which looks at the shift in information space during the 7 months when the mission withdrew from Mali . Apart from this, I've worked on integrating new data streams and developing new training modules for in-mission tr
Read more