Simulations & Virtual Reality, Digital Twins and Augmented Reality Supporting Automation and Production Solutions (pt 3)
AAE (Grauel) pushes technical boundaries. We provide high tech printing & assembly solutions. We support smart printing and manufacturing equipment with Industry 4.0 technologies and solutions. This series of articles provide background information on how these developments are supported.
In the previous articles we looked at simulations and digital twins and how they support industry 4.0. Another interesting technique that is employed supporting industry 4.0 is augmented reality (AR).
Using augmented reality, the communication in product and machine design as well as production development improves. It helps to identify and avoid design errors in early stages of the development process.
Looking at augmented reality more generally, it can be used to support increasing production by enhancing the user experience (during the development, manufacturing, maintenance, etc.).
In this article we will look at the history and definition of augmented reality (AR) in the first paragraph. After that we take a “deeper look” at the technical aspects of Augmented Reality. The last two paragraphs deal with the benefits for both the automation supplier and the manufacturing company when using Augmented Reality.
Augmented Reality (AR), history and definition
Augmented Reality has a new feel to it, but it actually dates back to 1957; depending a bit on the definition that is used.
At that time Augmented Reality was first achieved, by a cinematographer called Morton Heilig. He developed the Sensorama which included visuals, vibrations sounds and smell to the viewer. This did not include the use of a computer (or AR glasses) but it was the first example and attempt at adding additional (enhanced) information to an experience.
Then a more modern approach appeared in 1968 where Ivan Sutherland (American computer scientist and early Internet influence) invented the head-mounted display. The display offered a sort of (viewing) window into a virtual world.
But it is a little hard to provide a clear date for the birth of Augmented Reality as back in those days there was not yet a clear discrimination between “virtual reality” and “augmented reality”.
This discrimination, between the two (2) phrases, appeared around 1990 where Virtual Reality was invented by Jaron Lainer of VPL Research in 1989 and by Thomas P. Caudell of Boeing in 1990. Boeing is still a leading company in the use of Augmented Reality.
An overview of the history of Augmented Reality is shown below where Augmented Reality is assumed to have started in 1968.
This brings us to the definition of Augmented Reality where Virtual Reality and Augmented Reality have the same basis (two sides of a coin). Simply put, Augmented Reality is Virtual Reality but then with one foot in the real world.
Augmented Reality places artificial objects (or data) in a real environment, where Virtual Reality creates an artificial environment.
We can formulate the definition of Augmented Reality as:
A technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.
A system can be projected, using a tablet in this case, to present an FMS (Flexible Manufacturing System) onto an existing environment. The image we presented earlier is shown below*1 where a system is shown on a production floor (see the paragraph “Model Presentation, first steps” below for the actual app).
*1: development for a Microsoft Hololens is similar
Augmented Reality, a deeper look
There are various software packages available to support the creation of augmented reality. Schematically the “components” needed can be presented as follows:
The environment is scanned using a device (phone in this case) and the environment (trigger) is recognized. Based on the recognition, information is pulled from a database and projected on top of the scanned image as captured by the camera.
The first we need is something that triggers the information. This is often called a “trigger point” or “marker.”
A trigger point will execute the superimposed information and show it on the screen. Once the trigger is recognized the model and / or information is retrieved from the database and shown. The basic trigger point includes:
- Image recognition (information appears when an (flat) image is recognized)
- Model recognition (information appears when a model is recognized)
- Positioning recognition (information appears when a certain location is reached, ‘Pokémon Go’ is a well-known example).
Augmented reality that always superimposes the information on the screen is also present; these are also referred to as “markerless” trigger points. This is typically used when you want to “try” furniture at a specific location in your house. By pressing a button, the image is superimposed on the screen (and thus projected onto the world, e.g., showing a new tv on the wall).
Model Presentation, the first steps
Once we have a trigger point, we want to show useful information. In our case we want to show a model of a FMS (with robot and handling units).
Unfortunately, a CAD model cannot be used directly and needs to be modified (through tessellation, see also previous article). Fortunately, developments in this field are progressing fast, more software becomes available allowing for the transition of CAD models to AR models.
Note: implementation of this digital transition requires a new approach for engineering departments. But once implemented, the communication with customers and other departments becomes much more effective.
An interesting article on this engineering topic is “Augmented Reality system for aiding Engineering Design Process of Machinery Systems” by Marcin Januszka and Wojciech Moczulski for those interested in this subject.
We receive the models from the engineering department which are then tessellated*2 (optimized for use in an AR app). When completed, the models can be imported and used for the development of the Augmented Reality *3 (we use a “markerless” trigger point in this case).
*2: tessellation becomes more complex when e.g., animations are to be included with accurate movements.
*3 the example below shows the model on an Android tablet; the app can be generated for multiple devices.
The model that is shown in the image above is pulled from a database. This can be stored locally or in the cloud. Cloud storage is preferred also looking at revision control and display options for multiple users.
Adding More Functionality
With the model included in the app, we get a good understanding of how the model fits in the environment. Additional information can be added for an even more immersive experience. This may include movements, technical data or even a complete software menu. For additional technical support or presentations, parts of the machine can be removed or made transparent to see and understand the operation.
In the Augmented Reality app, we can zoom in on the handling units (by clicking on them), an additional view is generated which shows more details.
When the Augmented Reality app is used after the machine is installed, spare parts and technical drawings can be retrieved to present them on top of the actual machine. Also, the assembly of dedicated parts can be shown. A commercial approach is shown below where images are pulled from the Internet (cloud) and shown when the robot is selected.
Note: implementation of the augmented reality strategy requires planning and good communication between the different departments. This is a new approach for many companies but results in great benefits for all parties involved.
Augmented Reality (AR) Supporting Automation Decision
We looked at an example in the previous paragraph, but AR offers more benefits for automation suppliers. We will take a look at the benefits in general.
AR supports communication during the machine designing process. The design process can be streamlined by the collaboration between the parties involved.
Developing a product or machine (from concept through development) is a long and resource-intensive process. It requires frequent communication between different parties and involves numerous revisions of the initial concept. This all takes place before the product or machine reaches production and manufacturing.
By using augmented reality all parties involved can see the product or machine in development in real-time (supported by actual machine operation when required).
Once developed, AR supports pre-installation requirements where AR can be used to project the machine onto the work floor allowing to check the installation requirements.
In case of operation and maintenance, the experience and knowledge of the machine builder may be required. This normally involves travelling to the site where the machine is operation. With various machines installed, this involves a lot of travelling. Using AR, the expert(s) can look through the eyes of the technician who is physically doing the maintenance (a HoloLens is a typical example to support this).
The expert can leave notes or annotate on the field of view of the technician, so they can point out particular features of interest in what the technician is seeing.
In addition to maintenance support, the spare parts to be installed or replaced can be projected onto the machine to ensure correct installation or to provide assembly instructions.
Augmented Reality (AR) Supporting Manufacturing Decision
Also, for manufacturing companies, AR offers great advantages. Modern manufacturing involves complex automation solutions. A wide range of components are put together using a wide range of processes.
AR is used to display relevant process information, technical drawings and even videos onto the AR app. This can be used for a wide range of purposes including training, quality or even validation.
An example is included below showing a product with the proposed areas of interest (e.g., bar code printing location and colors and pre-treatment area). The product and print are projected onto the real world (this example is created using the Hololens).
When actual production data is overlayed onto the AR app, operators can verify the current operation with the defined parameters. For example, looking at a robot, operators can see information such as temperature, current load profile, speed of the various axis at a glance.
An example is included below where data from the robots (and FMS) is projected on top of the real world (this example projects the information on a booth at a trade fair).
For operators using an AR headset, this provides an additional benefit as they can keep their hands free while performing operational or maintenance tasks. Using AR this way it enables manufacturers streamline predictive maintenance programs. It helps workers to see the equipment’s potential failure points, letting them quickly see if there is a problem and, crucially, identify which parts are at fault. Replacement parts can then be ordered from a supplier without the need for costly unplanned downtime.
For quality inspectors, AR can be used to provide a virtual overlay of an assembled product. This overlay can be used to verify if ha products has been assembled correctly and meets quality demands. Especially for complex products this provides a useful tool.
Stay tuned for more, coming up next month! In the meantime, please consider following Grauel, a brand of AAE, on LinkedIn for weekly updates and extra content.
‘Simulations & Virtual Reality, Ditigal Twins and Augmented Reality Supporting Automation and Production Solutions’, by Ivo Brouwer – Business Developer Production Automation at AAE b.v.
De Pace, F., Manuri, F., Sanna, A. (2018). Augmented Reality in Industry 4.0, Available at: https://www.researchgate.net/publication/324565731_Augmented_Reality_in_Industry_40 [Accessed October 15, 2020].
Liu, X., Sohn, YH., Park, DW (2018), Application Development with Augmented Reality Technique using Unity 3D and Vuforia, Available at: https://www.ripublication.com/ijaer18/ijaerv13n21_33.pdf [Accessed October 10, 2020].
Sääski, J., Salonen, T., Liinasuo, M., Pakkanen., J., Vanhatalo, M., Riitahuhta, A. Augmented reality efficiency manufacturing industry: A case study. NordDesign 2008, pp 1 –11.
Januszka, M., Moczulski, W., (2011). Augmented reality system for aiding engineering design process of machinery systems, Available at: https://www.researchgate.net/publication/225901566_Augmented_reality_system_for_aiding_engineering_design_process_of_machinery_systems [Acessed October 10, 2020].
Mourtzis, D., Doukas, M., Riitahuhta, Bernidaki, D (2014), Simulation in Manufacturing: Review and Challenges, Available at https://www.sciencedirect.com/science/article/pii/S2212827114010634 [Accessed October 10, 2020].
https://prespective-software.com/white-paper/ (2020), Digital Twin Tech and your mechatronical system [online]. Available at https://prespective-software.com/white-paper/ [ Accessed October 12, 2020].
Eyre, J., Prof. Dodd, T., Freeman, C.,Lanyon-Hogg, R., Dr. Lockwood, A., Prof. Scott, R. Demonstration of an Industrial Framework for an Implementation of a Process Digital Twin. ASME 2018 International Mechanical Engineering Congress and Exposition, 2018. Page 1 – 8.
The future Factory. Industry 4.0 and Digital Twins: Key lessons from NASA [Online]. Available at: https://www.thefuturefactory.com/blog/24 [Accessed September 20, 2020[.
Exorint. What is the Difference Between a Simulation and a Digital Twin? [Online]. Available at: https://www.exorint.com/en/blog/what-is-the-difference-between-a-simulation-and-a-digital-twin [Accessed September 25, 2020].
Simul8. What is Industry 4.0 and could simulation help unlock its potential? [Online]. Available at: https://blog.simul8.com/what-is-industry-4-0-and-could-simulation-help-unlock-its-potential/ [Accessed September 21, 2020].
Behrtech. Digital Twins for Industry 4.0: Applications, Benefits, and Considerations [Online]. Available at: https://behrtech.com/blog/digital-twins-for-industry-4-0/ [Accessed September 21, 2020].
Onirix. Augmented Reality for Industry 4.0 [Online]. Available at: https://www.onirix.com/learn-about-ar/augmented-reality-for-industry-4-0/ [Accessed October 15, 2020].
Advantech. Enabling Industry 4.0 with Integrated Automation & Cloud Innovations [Online]. Available at: https://www.advantech.com/industrial-automation/industry4.0/emo#my_cen. [Accessed October 05, 2020].