- The Port of Livorno has become a testbed not only for how future ports may look, but also how any future industrial setting may look.
- Here, a digital twin allows workers to operate at the intersection of cyber and physical worlds.
- In our 2020 technology trends, we call this the ‘collaborative and automated physical world’.
A recent article published in Ericsson, written by Marzio Puleri, Teresa Pepe, Giulio Botari and Roberto Sabella deals with the digitalization of port operations.
Find out how it works below.
Port of Livorno: 5G, AI and a digital twin
One of those ports is the Livorno port terminal which now serves as an experimental area covered by a live 5G network.
Through 5G connectivity, we have enabled the capture of massive data from smart sensors, 3D LIDARs, and Wide Dynamic Range (WDR) cameras in the area.
Operation control system
Having these data available in real-time, an operation control system based on (AI), developed and deployed by Ericsson for the project, determines the sequence of logistics tasks and activities, by correlating a real-time information flow captured from smart sensors, from cameras, and from vehicles operating in the area. Tasks include the movement, tracking and positioning of forklifts and freights, as well as the inventories of goods, orders of loading and unloading of ships. This could also mean identifying specific activities, like using cranes and moving larger loads, as with a “high risk” flagging. The detailed view of everything happening at the port allows the AI engine to provide feedback on processes and share live updates with port supervisors.
On-demand identification
The operation control system is interfaced with the Port Authority control platform which provides on-demand identification of any available vehicles in the port area to accomplish a specific task. In addition, a high-precision positioning system based on multi-stereo vision techniques ensures an accurate positioning estimation even when freights are far and angled with respect to the cameras.
The data collected through 5G feeds a digital twinengine, developed and deployed by Ericsson, which elaborates a virtual replica of the port area in real time.
A look inside the port’s digital twin
The first element that has been modeled is the outdoor port area where the general cargo freights are stored. The following figure reports an image of the digital twin of the area and, in the inset, a satellite view of the real area inside the port terminal.
A second important element for the digital twin representation is the modeling of freight such as wooden boxes. If we take this in the context of a real example, the following figure shows a set of boxes in the digital twin scenario and, in the inset, the real boxes stored on the dock.
The area where goods are stocked is monitored by a set of cameras. Cameras are also modeled in the digital twin virtual scenario. The role of cameras is particularly important in positioning and tracking tasks.
Accurate goods localization
Accurate goods localization is fundamental to optimize the storage space and allow for a fast retrieval of the proper object at the right time for an efficient loading of the ship. Cameras stream video, through 5G connectivity, to a software application in charge of identifying objects and their position.
Here’s how it works in a real-life example in the port environment:
- A box (freight) is picked from a truck, the box is identified and registered by the control system in a database.
- An available forklift then transfers the freight to the storage area. As soon as the nearest camera detects the forklift arriving at the storage area, the tracking of the forklift and the freight can begin.
- The freight is tracked by the camera until its destination, where it is temporarily stored.
- Using AR, the forklift driver is then directed where to place the box in the storage by the control system information.
- When the freight is positioned, the cameras are used to crosscheck the final position of the freight. The new position is then registered in the database.
- Any following repositioning is also tracked by the cameras. When the same box is moved again, the cameras recognize that an action is undertaken on the object and tracks its movement. The new position is then registered.
- During the loading phase of a ship, forklifts are sent to the storage area to pick the goods to be loaded. When the forklift enters the storage area the control system provides AR information to the driver about the position of the freight to pick. The operation is automatically monitored and controlled using the information provided by the cameras.
Wide Dynamic Range (WDR) cameras
The use case is deployed using Wide Dynamic Range (WDR) cameras. These special devices can handle bright and dark conditions at the same time, resulting in a perfect operation even in an outdoor scenario where the light intensity and direction changes continuously due to the weather conditions and time of the day.
Having the port area and the general cargo freights cloned in the digital twin makes it possible to provide operators of the terminal with smart VR and AR applications.
Virtual reality (VR) and the digital twin
The VR application, which is experienced using Oculus headsets, allows operators to virtually navigate inside the digital twin model.
The right-hand controller of the Oculus is used to direct a green virtual laser pointer to the desired freight, for example a box placed on the storage area. Immediately, all the information related to that box is identified in the database such as ID, content, dimensions, weight, etc. These data appear in a virtual clipboard that can be manipulated in the VR environment through the left-hand controller. It is a very realistic way of “visiting” the storage area while remaining in the back-office with a significant increase of safety for terminal operators.
Virtual laser
By directing a “virtual laser” on a camera pole in the virtual environment, it’s possible to view on the clipboard what the camera is currently capturing from the top of the pole. With this technique it’s possible to have an overview of the storage area directly inside the VR application.
Special algorithm
In addition to the software dedicated to the virtual representation of the area, we have also deployed a specific algorithm which makes it possible to achieve an automated and optimal sorting of freights in the area and enable a quality check by inspecting the storage area and yard virtually from a remote site. The algorithm is fully integrated with the VR application so that the result of the optimal sorting is visible in the digital twin before being executed in field.
Augmented reality (AR) in port operations
The port pilot includes a specific AR application for the forklift drivers. When driving these vehicles, they are equipped with a rugged “see-through” smart headset built for industrial needs where protective helmets are mandatory. Through the headset, forklift drivers can see graphical signs and instructions of the freight they should move next. These directions appear laid on top of the operator’s vision of the real world.
This augmented video content is “assembled” by exploiting video streaming captured by the WDR cameras that track all objects in the storage area. The heterogenous data flow is blended by a software running on a cloud on the premises. This software processes images, recognizes objects, determines instructions for the drivers (i.e. direction) and delivers real-time direction laid on top of reality inside the AR headset.
This specific AR application is particularly challenging in terms of connectivity needs required to the 5G network. Here, the exchange of consistent video flows demands strict low-latency performance so as to correlate the data flows from the various sources. This correlation must be fast enough to deliver the resulting AR indications in seamless synchronization with the real world.
The port’s 5G network architecture
The 5G NR network installed in the port area is based on 3GPP R15 Option 3.x architecture based on Ericsson AIR 6488 operating at 3.7 GHz. The gNodeB is an Ericsson Baseband 6630 installed in the same cabinet where the cloud platform is also located.
The 5G NR for user plane coverage is supported by LTE in an NSA (non-standalone) architecture. This is the best choice for an initial 5G deployment where an operator already has 4G coverage in the area. The infrastructure runs all Virtual Network Functions (VNF) instances supporting the virtual Core to provide user plane and control plane functions, in line with the Distributed Cloud framework (equivalent 3GPP declination of the MEC paradigm).
The user plane function is interconnected with the local processing infrastructure through an application server with the computing capabilities necessary for the use cases deployed in the pilot. The choice of a local 5G vEPC co-located with the radio equipment and the 5G radio interface is technically related to ensure low latency and high throughput requirements. More specifically, the local termination of the user plane allows the application to be as close as possible to the user applications. In addition, the 5G NR radio interface is designed to reduce the radio access latency contribution in the end-to-end delay budget. High throughput levels in the radio access section are assured using wider spectrum portions dedicated to NR coverage in accordance to available LTE radio channelization. In a possible evolution, the implementation of part or all functions governing the network control plane could be centralized in the core network located in a remote operator’s central office, leaving locally just the functions dedicated to user plane handling.
The continuous and frequent exchange of massive amounts of data among sensors (including cameras), the control system, and the VR/AR applications poses tight requirements on the 5G network. The end-to-end latency target is posed at 10 milliseconds because the time budget imposed provides smooth feedback in AR to the forklift driver at 50 milliseconds but the AR software processing time absorbs 80 percent of this time budget (i.e. 40 milliseconds). The WDR cameras used for positioning, tracking, and AR transmit a continuous data flow of about 10 Mbps in uplink. Scaling the dimension of the use case to a larger area would require many cameras with a cumulative rate of hundreds of megabits per second in uplink.
Benefits of digital twins and other technologies in port operations
The overall port pilot is expected to bring significant benefits to the port. This includes:
- Reduction of the registration phase from 3 minutes to 2 minutes
- Reduction in the average execution time of moving an object from A to B with a forklift from 8 minutes to 7 minutes
- Reduction in the space occupied in the storage phase by 10 percent, for example from 5000sqm to 4500sqm
- Reduction in the unloading/loading times of a general cargo ship from an average of 18 hours to 16 hours, meaning the relevant idle time at berth is reduced from 36 hours to 34 hours
For the latter, the time saving can be immediately translated into significant cost saving for the ship’s owners. In addition, more ships could be served through the course of the year, scaling up the port’s capacity without the need to enlarge the existing terminals.
All these efficiency gains also result in a reduction in movements during cargo handling. This can significantly optimize the overall process, lowering fuel consumption as well as associated CO2 emissions by 8.2 percent as explained in the recently published Port of the Future report.
Conclusions
Ports are gateways to the world that are of great importance in an interconnected and globalized scenario. One of the main challenges that ports face today is how they can evolve to become more efficient, competitive and sustainable. With its low latency, high capacity and greater flexibility, 5G is an unprecedented innovation platform to increase operational efficiency. Artificial intelligence, running in a control module, as well as the use of advanced AR/VR-based services enables faster and better management of the general cargo and optimizes the intra-terminal operations.
For the results achieved, Ericsson has received the prestigious “Industrial Energy Efficiency Award” during the Hannover Messe Digital Days on July 2020. The 5G trial underway in Livorno is also part of the European project H2020 Corealis.
Did you subscribe to our daily newsletter?
It’s Free! Click here to Subscribe!
Source: Ericsson
