“Interfaces” by Alessio Grancini, Milan Design Week 2019 from the Morphosis Installation “Interfaces” (Photo Credits Morphosis)

As 2021 starts, there is growing evidence that technologies connecting us together are scaling up the game. The COVID-19 lockdown has introduced a wider community to more new emerging technologies, like VR and AR, and we find ourselves integrating more smart devices into our daily lives. As the digital environment grows and the meaning of space starts shifting, the field of design and architecture is preparing to explore new media and expand its domain. In this miniseries, we are taking a closer look at a number of emerging technologies and software applications that are building the foundation of the future understanding of space. We are doing it, with the expertise of 4 guests that deal with the future of technologies in their everyday life in a series of four installments.

Our first conversation is between Paul Petrunia and Alessio Grancini, Prototyper Engineer at Magic Leap and ex-Morphosis XR developer. 

With a hybrid background between design and technology, Alessio Grancini is a prototype engineer at Magic Leap. He is interested in AR and VR meeting human interaction. He taught Unity workshops internationally, exhibited work in art galleries, and published multiple articles about AR/VR.

Tell us about your journey and what led you to focus entirely on augmented reality

Augmented reality impacted my education as a student to the extent of making it my greatest passion and eventually, my profession. I started to work with AR because I felt that it was an additional method of representation for explaining my projects. The design reviews at SCI-Arc became fully centered on the use of the media only, so I started to look into it more closely, investing time in the process and not necessarily the outcome of it. I discovered that AR is not at all something new. The first use of this technology, and the concept of it,  emerged in the 70s. At this point in history, however, the technology is finally taking off, making it such an exciting time to be working with it. As a student, and generally as a person, I thought that my strongest contribution could have been only if I explored this technology at all its extent and I started my career with this goal in mind. 

In your last exhibit at “Manifattura Tabacchi” for Cinema Nervi in Florence, your video art piece was centered on the concept of in Internet Of Things (IoT). Can you discuss what IoT means for architectural practice?

The so-called Internet Of Things refers to a network of data exchange between physical objects that act as inputs for collecting data and making them available on an internet connection or output of the data collected on the network. Between the most common IoT devices–Amazon Echo, Alexa, Smart Light switch, etc.–the bi-directional way that the objects interact with us is clear. Alexa can become “smarter” thanks to the detection of our speech, and it is up to date with the last available news, etc.

The concept of IoT, however, is not limited to these interior high-end “smart-devices.” The “real game” is way wider than that. The scale of IoT applications is quite variable and it's affecting the field of design and urbanism. To think about this shift in other terms, today, we can see how hard it became to make any choice without having the support of the internet pivoting our answers. This model of thinking, making us “accurate”,  is dynamic and continuously up to date. In this way, the same model applied to a profession as one of the designers, could lead to new ways of working.

The scale of IoT applications is quite variable and it's affecting the field of design and urbanism

Specific to the field of design, new emerging technologies adopting the IoT platform model are rising, challenging the fundamentals of the profession. All of a sudden many steps of the process need to be re-evaluated, and at this stage, design might benefit from a quick look into these new avenues and understand what are the pros and cons of it. For example, digital twins are probably the most interesting case for design management. 

Digital twin is a term I'm seeing used in the context of architecture. What are digital twins and how is it applied to our industry?

The switch toward the management of physical assets through a digital proxy entered the profession of architecture through the BIM technology. But, an important distinction we should make is the one between BIM and digital twins. “BIM” is a process that defines the generation and management of digital representations. A “digital twin” refers to the most accurate digital replica of any physical assets. 

A “digital twin” refers to the most accurate digital replica of any physical assets

As BIM refers to a temporary technologically-specific integration, the concept of digital twin seems to be opening the door for a new phase of “unlimited” maintenance of a project looping information back and forth. It almost seems to be referring to a “second life” of the building as we consider “second life,” our digital identity on social media. Digital Twin is not a concept bonded to design, it is an application of digital management to design as it could be an application of it for many other fields. Chris O’Connor from IBM describes the digital twin as “the ability to virtually visualize elements and dynamics of how an IoT device operates.” He also says that if done correctly a digital twin can influence design choices, becoming the driving element of a design process.

Meanwhile, this shouldn’t be interpreted as a replacement of a creative process, it should be more taken into consideration as a transformation in the way we think about design and how much of this digital transformation we would like to take into consideration during the process. 

Architecture is a profession that is always exploring new approaches to work. What role does augmented reality play in the evolution of the field?

I faced the concept of digital twin for the first time as a student, joining a hackathon at USC where Autodesk was one of the main sponsors of the event. With a group of friends and students I met at the event, I built an application that would link a Revit model on an Augmented Reality wearable. The application would showcase data as materials and measurements in real-time between a web portal named “Forge,” an Autodesk product, and the AR wearables we used named “Mira.” It was just a prototype, but it made me understand the potential of the application. Most importantly, however, it let me question the boundaries of the profession. If the dynamicity of these models is unlimited, when does a designer stop delivering? What platform would regulate the future flux of data? Who’s defining the structure that is displaying these data? Are designers involved? What data significant in the process will be staying? Does the scale of the project affect the authority on the data? What sort of agreement is regulating this process? And more.

Today, Autodesk offers a set of tools for accessing the subject of digital twin and subdivides the aspects integrated into digital twin as Visualization, Live, Analytics, Simulation, Predictions, and Automation. I would like to stress the concept of prediction. As the word says, we are not always collecting data from physical devices but we are also thinking about the future and projecting results according to a set of data that might be partially synthetic. Autodesk defines the digital twin as a bi-directional entity that collects data and sends data to its physical twin to execute them.

The output of this system and its repercussions are endless and make us wonder what’s next, however, not being too general, I believe a good topic to be focusing on is the one of Location Based Augmented Reality and its relation with a digital twin model.  

How does Augmented Reality differ from Location Based Augmented Reality?

If you are familiar with the mobile game Pokemon Go, you will also be indirectly familiar with the concept of Location Based Augmented Reality or markerless AR. Location Based AR uses data from a positioning system to identify a device location and reconstruct a digital scenery in the real world. As with the concept of digital twin, one general definition, doesn't really express the full potential and capability of this software application. There are a lot of options for experiencing markerless AR and the accuracy of the location highly depends on how the application has been built. Markerless AR is strongly connected with the topic of digital twin since the experience is largely based on the reconstruction of a digital model in the real world. Augmented Reality may also be a way of visualizing a digital twin.

The application takes advantage of networking between devices–it’s built on an IoT platform through IoT devices–smartphones, in this case, the quality of the application is determined by the quality of the internet signal. Apple developed a faster way to determine the location of the content through computer vision and extracting local points from Apple Maps. This system is a mix between a positioning system and camera detection. In a dark lighting condition, this could prevent a smooth AR experience, for now.

Markerless AR faces a lot of challenges, and there is no real established method for how it is functioning; improvements keep coming out on a daily basis. For targeting urban spaces, however, there is no doubt that Markerless AR has the greatest potential for a large-scale application of Augmented Reality.

How does architecture play a role in markerless AR? 

Markerless AR applications are based on GPS and their accuracy is based on computer vision. This category of applications opens up a lot of potential collaboration between designers and developers. Let’s assume that these applications are gathering data continuously in urban environments. Until now there is no clear collaboration between physical location and these categories of devices. For example, considering large-scale projects as conference centers, stations, airports where boundaries between exterior and interior start fading, the relations between distances and the location of future IoT receiver devices, could start impacting the design process of a building in the future.

Markerless AR applications are based on GPS and their accuracy is based on computer vision. This category of applications opens up a lot of potential collaboration between designers and developers

Furthermore, as the light gets darker, AR applications start performing more poorly. Another application might be integrating devices and spatial areas where this data is provided through a formula that does not need real-time lightning detection. In both the previously listed cases, the collaboration between space and application seems to be the key factor for future developments of design and IoT based apps.

The last Facebook Connect, the annual FB conference, introduced the first time a tech company openly spoke about city layering and urban planning. What are your thoughts on the presentation regarding their plans for the development of Augmented Reality experiences?

Facebook recently showed its plans to build the future of Augmented Reality. “Live Maps” is the core infrastructure of FRL’s research and thanks to machine vision aims to separate the real world in digital layers and construct experiences through a live dynamic index. The overall digital model superimposed to the real physical work is named “Real World Index.” 

This research takes into consideration not only urban spaces, but also interiors. All of the components of the worldwide digital twin will be live-updated thanks to our devices and AR wearables. The ambition of the company is unprecedented, and given its weight in our daily lives, it is hard to believe that this plan could fail. Facebook's research takes all of the previously analyzed topics and merges them together, giving them a proper structure and hierarchy. It is the biggest effort in coordinating physical and digital together of our times. Andrew Bosworth, head of Facebook Reality Labs says that this research will change the way we interact with each other forever. During Facebook connect, Mark Zuckerberg also introduces the new smart wearable device “Aria,” differentiating itself from most other AR wearables, has a live-connection with a centralized system. “The device is a device of research”, he says and the device, with its collected data, will help build the maps and experiences of the future.

Stay tuned for the next part in this series, releasing tomorrow, a conversation between Alessio Grancini and Matthew Hallberg, Tech YouTuber and Software Engineer at Third Aurora.

Interested in learning more about the future of architecture and the shift to smart technologies across the world?  Make sure to check out Smart Buildings/ Smart Cities: Integrated Equity & Resilience, February 9 & 10 (8:30am - 12:30pm Pacific Standard Time). Innovators and experts from London, Silicon Valley, New Orleans, and more will present new advances along with Alessio who will discuss his work in "Planning the Digital Through Augmented Reality: Empowering designers in the new city : converging professions, approaches, environments."