The IKUN project, “Development of Large Multimodal Models for quality assurance and support for operators in Smart Industry,” seeks to adapt advances in Artificial Intelligence to the industrial environment.
Artificial intelligence (AI) is transforming strategic sectors thanks to advances in language models, computer vision, and content generation such as GPT-4, LLaMA, DALL-E, and CLIP. However, its integration into industrial environments still presents significant challenges, from the need for large volumes of labeled data to interfaces poorly adapted to the realities of production plants.
In this context, the IKUN project was born with the objective of adapting Large Multimodal Models (LMMs) to the industrial context, opening the door to the development and implementation of smarter, more accessible, and efficient solutions.
To this end, the project addresses several key technological challenges:
- Creation of multimodal industrial datasets, essential for adapting models to the production environment;
- Development of MLLM adaptation techniques aimed at ensuring robust, explainable, and reliable systems;
- Generation of synthetic data (images and time series) for training advanced quality assurance systems that reduce the need for real data;
- Design of conversational interfaces that allow operating personnel to interact naturally with the systems, whether through text, voice, or images.
With a practical and progressive approach, IKUN addresses everything from defining pilots and collecting real data to validating models in industrial environments. Specific solutions are being developed for different types of data, improving the interaction between people and systems and ensuring their usefulness and transfer under real-world operating conditions.
In addition to participating in the definition of multimodal data for the various industrial pilots running within the project, as part of the IKUN consortium, AZTERLAN plays a relevant role in research related to the industrial standardization of multimodal information storage. As explained by Javier Nieves, head of the Intelligent Manufacturing Technologies research line at AZTERLAN, “in the industrial manufacturing ecosystem, there is a great heterogeneity of data and information (images, sensors, equipment, management systems, production protocols, etc.) that must be processed, and each data-typoloy requires ad-hoc storage methods and specific case studies. The ultimate goal is to enable this data to be used in such a way that the different systems with access to it can achieve a standard definition for its exploitation. As an illustrative example, we would have, for instance, the development of systems capable of describing and interpreting video sequences monitoring industrial production. Although these tools have been widely developed in other application areas, there is a significant gap in valuable descriptive labeling that allows for the interpretation of industrial events. In this case, at the level at which we aim to exploit the data, it is not enough to identify that a certain image corresponds to a certain industrial process; the description must provide information on whether what is happening in it is correct or not with respect to the standardized process.”
Product quality assurance and improved interaction between people and systems
Expected results include the creation of new industrial datasets and the generation of high-quality synthetic images and time series for quality assurance. “Among other practical applications, we are working on the development of an advanced anomaly (defect) detection system and synthetic image generation, using the latest techniques in generative models and deep learning. Visual inspection remains a crucial process for quality control and industrial safety assurance, so automation has a high impact, which can lead to greater efficiency, accuracy, and cost savings, generating a competitive advantage in the industrial sector.”
The research team is also working on the design of intelligent conversational assistants to help plant personnel on the production line or when consulting technical documentation, and the validation of prototypes ready for transfer to the industrial ecosystem. “At AZTERLAN, we have been working for some time on intelligent systems aimed at facilitating decision-making and the implementation of actions by plant production personnel, so far primarily through advanced control systems and predictive technologies. However, facilitating the consumption and understanding of useful and relevant information through user-friendly conversational systems is the most efficient way to transfer metallurgical and process knowledge to the industry and is a priority area of technological development for our team.”
The IKUN consortium is made up of the technology centers VICOMTECH (project leader), TECNALIA, IKERLAN, TEKNIKER, and AZTERLAN; the University of the Basque Country (EHU); the business R&D unit IKOR TECHNOLOGY CENTER; and the intermediary agent IMH Campus. The project is funded by the Basque Government’s ELKARTEK 2024 program.
