This article is a summary of our publications and presentations, describing how we implemented artificial intelligence solutions in modern automation teams. The solution we present here is developed and tested by KBA AUTOMATIC Sp. z o.o., which is a milestone towards Industry 4.0. In this solution, we combine Industrial Automation with the latest developments in computer science and artificial intelligence.

Our solution includes:

• An innovative solution for palletization of heterogeneous physical objects using an industrial robot and a neural network model, using deep learning mechanisms.
• An intelligent vision system for detecting and classifying heterogeneous objects on a conveyor belt.

1. Palletizing Heterogeneous Physical Objects Using Deep Learning and an Industrial Robot

1.1     Introduction

Our work has led to the development of both software and hardware for implementing an intelligent method of palletizing heterogeneous physical objects. The interdisciplinary approach used in creating our solution allowed us to combine elements from computer science, artificial intelligence, industrial automation, and robotics. The verification studies we conducted confirmed the effectiveness of the developed system. Depending on the size of the objects used, the average efficiency ranged from 40% to 85% pallet fill.

1.2 Neural Network Training Process

In our application, we used three learning methods:

• Reinforcement learning
• Supervised learning
• Multi-model learning

1.3    Developed Software

We developed a custom application implementing the trained neural network based on the Unity graphic engine (v2020.3.4f1). Our solution allows for the generation of advanced simulations of physical processes and real-time simulations.

For most of the application scripts, we used C# language using the Visual Studio. To avoid conflicts between different software versions, we created virtual Python environments (Venv), making it easy to change the configuration of the libraries used, and as a result two separate virtual environments were created.

The following libraries were used in the application:

• For the host with Unity Engine installed:
  • Python (v3.7.8)
  • Tensorflow (v2.6.0),
  • Keras (v2.6.0),
  • OpenCV-Python (v4.1.0.25),
  • tqdm (v4.31.1),
  • Unity Engine (v2020.3.4f1),
  • Unity ML Agents (v1.0.8)

• For the Python virtual environment (venv):
  • ML Agents,
  • Pillow
  • Numpy
  • Tqdm
  • Pytorch

Implementing the system on a physical object aligns with the “digital twin” model paradigm.

1.4     Verification Studies

After development phase, verification tests were conducted to ensure the proper functionality of the system. The tests were carried out in two ways:

1. A neural network trained on packages of random sizes, with verification of system performance using packages of random dimensions

2. A neural network trained on packages of the same size, with verification of system performance using packages of the same size

2. Intelligent Vision System

2.1  Introduction

In order to fully leverage the potential of artificial intelligence in our system, we developed an intelligent vision system for detecting and classifying objects moving on a conveyor belt.

This system is an integral part of our Intelligent Robotic Cell 4.0, providing a leap forward in automation technology.

Our technology is characterized by:

• High accuracy,
• Adaptability,
• Cost-efficiency,
• Effectiveness.

2.3 Data Acquisition and Visualization

Our approach uses a depth camera that captures images of objects in three dimensions. The background is then extracted from the obtained image, which is crucial for further feature extraction (dimensions, shapes, and texture). It is worth noting that the efficiency of this solution is not dependent on lighting, making it highly adaptable for various environmental conditions.

The extracted features are input into our neural network, designed to classify objects on the conveyor belt. The system includes Python-based software and a GUI, streamlining data acquisition, neural network training, and object classification.

Our approach uses a depth camera to capture images of objects in three dimensions. It should be noted that the depth camera has a time difference between capturing the bitmap image and the depth map. For moving objects, such as those on a conveyor belt, this time difference causes a phase shift between the objects seen in the bitmap image and the depth map. As a result, the bitmap map is not used for depth descriptor extraction, and it is only shown for visualization purposes.

2. 3 System description

In this solution, we distinguish two layers. The first layer is responsible for data acquisition and analysis from the depth camera images, and the second layer is our neural network used for object classification.

To better understand our technology, we include the analysis of the depth map as a block diagram and the system’s visualization.

Extracted Descriptors

After determining the object to be analyzed, we extract 29 descriptors. The first ten are responsible for the statistical parameters of the depth map containing only the object, and the remaining 19 describe the shape of the analyzed object.

Classification Model Description

After extracting the descriptors, we normalize them to enhance the model’s learning capabilities and improve prediction accuracy. This allows us to input the values into the model, which creates a three-layer artificial neural network. The dimensions of the layers are determined by the number of features and classes. For the input layer, the size is equal to the number of features, and for the hidden layer, the size is determined by the sum of features and classes.

Anomaly Detection Algorithm Description

In addition to the aforementioned algorithms, we also developed an anomaly detection algorithm. The use of this algorithm increases classification efficiency. The anomaly detection algorithm identifies features that do not belong to the class of the object being checked, without disrupting the primary classification process.

The classification result and the anomaly detection algorithm are returned as a percentage probability.

Classified Objects

Our vision system can classify various objects based on their characteristics. We do not impose any fixed number of classes, however, our application is aimed at sorting diverse objects. Therefore, the objects typically classified are cardboard and bagged packages.

2.4 System Testing in Physical Implementation

To verify the correctness of the developed system, we implemented and tested it physically — initially statically, and later dynamically (with a package speed of 0.95 m/s). The training set consisted of 200 images, and the test set included 300 images.

The GUI designed for our system allowed efficient testing of our model and verification of the approach in the environment.

2.5 Strength and limitations of Our Solution

The strengths of our solution include:

• Simple structure allowing for fast classification, requiring few training images, and is easy to interpret.
• Clear feature extraction increases interpretability, making it easier to understand the decisions made by the model during classification.
• The solution reduces costs as it does not require any external light source. Moreover, tests have shown high accuracy and adaptability to varying environmental conditions.
• The system is intuitive to operate, requiring only a photo of an empty conveyor belt for reference.
• Supports any number of classes due to the flexible neural network structure that adapts to these variations.

The limitations include:

• The feature extraction process (critical to the system’s operation) is slower than classification speed, which can lead to a bottleneck effect when processing faster-moving packages that require quick data processing.
• The lack of a comparative analysis using colour images limits the evaluation of the effectiveness of a depth-based system compared to more conventional vision systems.

3. Conclusion

The solutions we have developed undoubtedly create a symbiosis that represents a higher level of automation. The combination of modern solutions — intelligent vision systems and automated palletizing using an industrial robot — emphasizes quality, efficiency, precision, and innovation. Our system is universal, easily adaptable, and capable of working in various environmental conditions. These features make it compliant with modern standards and enable cost reduction in case of system modifications.

In the next section, we provided a collection of conferences, where we presented our solution.

4. Places where we presented our solution

• Majewski, Ł. Klar and K. Bochenek,: “Interdyscyplinarność pomiędzy zagadnieniami Informatyki, Sztucznej Inteligencji i Automatyki Przemysłowej – studium przypadku dla zrobotyzowanej celi” (Prezentacja on-line na konferencji: “XVI Interdyscyplinarna Konferencja Naukowa TYGIEL 2024- “Interdyscyplinarność kluczem do rozwoju”” dnia 21 marca 2024r.)
• Majewski, Ł. Klar and K. Bochenek,: “Inteligentny system wizyjny do detekcji i klasyfikacji heterogenicznych obiektów na przenośniku taśmowym – badania porównawcze na obiekcie rzeczywistym” (prezentacja on-line na konferencji “IV Ogólnopolska Konferencja Naukowa „Rozwiązania i technologie XXI wieku”” dnia 9 maja 2024).
• Majewski, A. Łysiak, Ł. Klar and K. Bochenek, “An intelligent vision system for the detection and classification of heterogeneous objects moving on a belt conveyor,” 2024 28th International Conference on Methods and Models in Automation and Robotics (MMAR), Poland, 2024, pp. 264-269, doi: 10.1109/MMAR62187.2024.10680789. (Prezentacja na międzynarodowej konferencji “INTERNATIONAL CONFERENCE ON METHODS AND MODELS IN AUTOMATION AND ROBOTICS” “MMAR 2024” dnia 28 sierpnia 2024).