Challenges in Data Management in Robotics Workshop

October 01, 2024, by Reik Leiterer, data innovation alliance

On the 29th of August, the Workshop on Challenges in Data Management in Robotics took place in Biel at the premises of the Swiss Cobotics Competence Center (S3C) to bring together robotics enthusiasts, researchers, and industry professionals to discuss and address the pressing issues surrounding data management in robotics applications.

The workshops was organized by the Innovation Booster Robotics, S3C and the Innovation Booster Artificial Intelligence. S3C represents the national hub for industrial and academic partners to jointly develop and test the next generation of industrial cobotic solutions or their components, and to learn how to design and integrate cobotic solutions under human-centric aspects. The Innovation Booster (IB) are Innosuisse supported programs. IB Robotics is dedicated to fostering radical innovation in the Swiss robotics ecosystem. IB Artificial Intelligence is an open innovation initiative to foster radical innovation using socially- and economically viable AI.

The aim of the full-day workshop was to use keynotes, case studies and subsequent breakout sessions to provide participants with insights into data-related challenges in robotics and to jointly develop possible solutions for overcoming them.

Four complex subject areas with their respective challenges were defined:

Challenge 1: Co-Working and Learning

Understanding human practices within the context of human-robot interaction is crucial for robots to adjust their behaviours appropriately. Not only do technical tasks and ethical considerations come into play, but also the fact that different individuals interact in unique ways, making the development of universally applicable algorithms extremely difficult. Furthermore, ensuring precise communication and task allocation in the human-robot workflow is essential, especially considering the proximity between humans and robots.

Challenge 2: Data Management

Most robots work with both filtered and uncensored data, not only for robot training and learning tasks but also directly acquired in-process by physical sensors. Nowadays, tasks such as training may require a large amount of data. This can be challenging in terms of data overload, costs of data analytics integration, and robust data management practices.

Challenge 3: Responsible Collaboration

Both humans and machines occasionally fail at specific tasks, making it essential to consider failure scenarios in developing robust interaction frameworks. By understanding and leveraging failure data, we aim to improve the design, development, control, and robustness robots, ensuring more resilient and adaptive human-robot interactions. There is a need to explore methods for identifying and capturing failure events, analysing their causes, and utilizing this data to improve the reliability and performance of robotic systems.

Challenge 4: Self-improvement of Robots

One aspect of robustness involves the capacity for a robot to enhance its knowledge and behaviour autonomously. This means it should possess a degree of flexibility to adapt and apply abilities to new situations as needed. Achieving this context-driven, adaptive autonomy, which relies on common-sense knowledge and practical manipulation tasks, demands extensive programming and often involves on-platform data management & analytics.

The workshop day began with a welcome by the organizers and presentations about the opportunities the Innovation Booster’ and the S3C offer for interested organizations from industry and research. Afterwards, the first two challenges were introduced with inspiring keynotes followed by a moderated panel discussion with all the presenters.

The topic “Coworking and Learning” was opened by Baptiste Busch – CEO and Cofounder of AICA. He pointed out the demand for advanced and flexible robotic solutions in the reindustrialization process and the challenges around the sensing of and co-working with robots. Opportunities in this regard were seen around sensing-based control, behavioural programming increasing self-adaptation capabilities, and the use of foundation models. On the way towards cageless cohabitation, changes in the collaboration between robot manufacturers, integrators and end-users would be necessary while regulatory changes will bring additional challenges (e.g. ISO 10218-1:2011 and EU AI Act).

Maryam Rezayati from ZHAW presented as the next speaker possible applications of human-robot interaction (HRI) and outlined how context understanding, contact perception and possible interpretation of human intention by robots could be the way to go. The keynote was closed with project insights of the SmartSenseAI project, which aims for developing robust, extremely low-cost, customizable, flexible multi-purpose skin for cobots, enhanced by cognitive information processing.

Björn Jensen from HSLU and Yves Albers-Schoenberg of Roboto AI addressed the challenges around “Data Management” – from data retrieval and data analysis to data augmentation and the generalizability of data models. It was discussed, how the combination of data, models, and simulations is the base towards data-centric robotics – and with reference to the Moravec’s paradox, that assembling the right data and deploying the right models for sensory processing and perception skills require enormous computational resources.

After the panel discussion, the participants were assigned to the two challenges and worked together on possible solutions and what would be the requirement for starting projects on the identified solutions.

The outcomes of the presentations and discussions around the “Co-Working and Learning” challenge were:

Collaborative robots are still not widely accepted in the industry. There is a need for robots to recognize human actions and adjust their movements accordingly for safety. However, integrating these capabilities is complex, and there’s a general reluctance among manufacturers and integrators due to liability concerns. Third-party entities could play a crucial role in advancing human-robot collaboration by bridging these gaps.

The industry is pushing for more flexible and collaborative production environments where robots can take on repetitive tasks. Yet, existing robots are limited by predefined motions and struggle with understanding context and interpreting human interactions. The challenge lies in transitioning these technologies from research to practical industrial applications.

Human-robot interaction is constantly evolving, and collaboration requires robots to manage the variability in human behaviour, which is not easily predictable. Current approaches rely heavily on data, but there is also a need to focus more on understanding human behaviour beyond scripted interactions. Empowering collaboration requires better data retrieval, analysis, and generalization techniques.
Reinforcement learning in simulated environments helps robots become robust, but translating this learning to real-world environments remains difficult. Robots often struggle to adapt outside of simulations, indicating a need for them to learn more effectively from human behaviours.

Based on the identified challenges, the following ideas were developed and possible collaborations between the participants discussed:

Develop a more effective system for categorizing the skills required based on specific tasks and types of collaboration. And create a unified platform to hare categories.
Establish industry-wide safety standards for human-robot collaboration that are universally applicable, including the creation of adaptable standards, norms, and guidelines for various environments.
Promote partnerships between industry and academic institutions to utilize common datasets on skills, bridging the gap between research and practical applications. This approach could help accelerate the transition of innovative solutions from the lab to the factory floor.

The outcomes of the presentations and discussions around the “Data Management” challenge were:

Managing large, complex datasets is a significant challenge in robotics. It takes considerable effort to curate and process this data for practical use. There is a pressing need for large-scale, public datasets to support advancements in human-robot collaboration.

Data management challenges are central to improving human-robot collaboration. Key issues include the retrieval, analysis, and augmentation of data to handle the variability in human interactions. Effective data management is essential for robots to generalize from specific data to broader applications.

Foundation models hold the potential to transform human-robot collaboration, but they require extensive amounts of data to function effectively. The complexity of managing this data, coupled with increased regulatory liabilities, adds to the challenges.

The following ideas were developed:

Consider ways to handle the variability in human behaviour. These could be trained on diverse datasets to improve robots’ adaptability in real-world situations, allowing for more seamless and effective collaboration.
Create a centralized platform for managing large-scale robotics data. This platform could provide standardized tools for data collection, curation, and processing, making it easier to develop and refine foundation models.
Establish open-source repositories for robotics datasets, curated with respect of the real world.
Develop new data compression methods to handle the vast amounts of data generated by robots. This would enable more efficient storage and faster processing, making it easier to utilize data for training and collaboration.

After the intensive and inspiring morning, there was a networking lunch with the opportunity to have a look on the various services and test options offered by the S3C, including i) realistic, industrial-like test environments with confidential testing conditions, ii) adaptable, modular test benches and setups, and iii) scalable options from quick tests to extensive, long-term testing.

After lunch, the afternoon session starts with the topic “Responsible Collaboration”. Andreas Hufschmid from PILZ explained why a holistic safety concept is required in the field of industrial robot systems and while dealing with HRI. This considers both machinery safety and industrial security and is mainly addressed in the ISO 10218 standard, created in recognition of the particular hazards that are presented by industrial robots and industrial robot systems. For this, risk assessments should be conducted to determine what the protective measures should be.

Andrei Cramariuc from the Robotic System Lab (ETHZ) presented the challenges in cases, were robots need to respond to human interactions or should perform complex, everyday tasks in dynamic environments. For this, approaches such as kinaesthetic guidance or teleoperation are currently explored.

Sina Mirrazavi from the AI Institute and Sylvain Calinon from the IDIAP research institute talked about the state of the art of “Self-improvement of Robots”. Even if robots can act highly robust in simulated environments they still often struggle if exposed to real-world applications. This could be solved, if robots are enabled to adapt und improve their performance using their own real-world experiences (e.g. via tensor networks) and/or learning form human behaviours. But here, too, there is no one-size-fits-all solution.

After the break-out sessions again, the outcomes around the “Responsible Collaboration” challengewere presented:

Safety is a critical concern in human-robot collaboration, with collision forces being a primary issue. Current safety standards are essential but require ongoing verification and validation to ensure effectiveness as regulations evolve.
Robots must be able to respond to human interactions and perform complex, everyday tasks in dynamic environments, i.e. dealing also with uncertainties and adapt to changes. Different approaches (e.g. kinaesthetic guidance and teleoperation) are being explored to improve how robots learn and interact in real-time scenarios.
Balancing safety with innovation is a major challenge, especially in public spaces and assistive robotics. There is a growing recognition that safety by design must be integrated into the development process, with reinforcement learning playing a potential role in achieving this balance.

Related to the “Self-Improvement of Robots” the following challenges were identified:

Robots often struggle when transitioning to real-world applications. The challenge is to enable robots to adapt and improve their performance based on real-world experiences, learning from human behaviour.
Research is focused on enhancing robots’ capabilities beyond basic tasks, such as using their entire body for interactions. Efforts are underway to develop more sophisticated data structures and learning strategies, like self-learning with tensor networks, to improve robots’ adaptability and performance.
A key area of focus is training robots to specialize in specific tasks, such as assembly, by refining foundation models. The ability to standardize data and share solutions across platforms could significantly advance the self-improvement of robots.

Based on these challenges, the following ideas were discussed and shaped:

Facilitate the sharing of models and simulations across different platforms, including sharing of successful learning strategies among different robots and platforms (potentially through a centralized knowledge base).
Extend the application of reinforcement learning beyond simulations by enabling robots to learn directly from real-world interactions, possibly within controlled environments where they can safely experiment and refine their skills, i.e. gradually expanding their ability to adapt to real-world norms as they become more capable. In addition, implementing real-time feedback loops could be of interest to allow robots to instantly adjust their actions based on human responses. This approach could also encourage mutual learning between humans and robots.
Develop specialized foundation models tailored to specific tasks, such as assembly or inspection, to enhance efficiency and effectiveness in improving robot performance within these domains.

Ideas discussed during the workshop can now be submitted to the IB Robotics (next DL 13th of September) or to the IB Artificial Intelligence for further funding in order to develop them further in terms of feasibility, viability and desirability.

back