In this second installment of our 4-part series Designers in Tech, Alessio Grancini, Prototyper Engineer at Magic Leap and ex-Morphosis XR developer, speaks with Matthew Hallberg about his work and research into Extended Reality (XR) and its potential in architecture and urban design.
With 50k+ subscribers, Third Aurora Software Developer and YouTuber Matthew Hallberg is well known for speculative cutting edge prototypes and demos that feature the most recent technologies available in the field of XR.
Your popular YouTube channel features augmented reality experiments and more. The accessibility of the tutorials makes your audience, flowing from any field, also the one of design. Could you talk a little about the mission of your XR company Third Aurora?
My YouTube channel features quite a lot of my experiments and interests. Meanwhile, at Third Aurora, we are looking for building purposeful collaborations, considering also the field of design. We are making a few different AR platforms that allow companies to author their own AR content in a web-based portal. Our main product, Winerytale, allows wineries to augment their wine labels. Currently, we are working on one for beer cans. Users can point their phone at a beer can and see images and video in AR as well as some other relevant information.
So when we say “more accessible” we mean that first, we are doing the market research to find exactly what types of information and media users would care to see in AR. Then we design multiple layouts or experiences to display that information. A new craft beer brand comes along and wants AR on their can, now all they have to do is create an account on our website, upload their label, a few images and videos, and now they have a fully branded AR experience without doing any of the leg work.
On a more relevant note, we did make a platform for showing display homes. Companies could upload a 3D model of a house and users could place it down on a plot of land to visualize it and physically walk through it. This is generally a very common model of visualization at this stage, and even if I am not directly involved in this design practice I can see how things are changing.
Companies could upload a 3D model of a house and users could place it down on a plot of land to visualize it and physically walk through it
Generally, we think more in terms of solutions but our work with designers so far has been one of our biggest challenges. Usually, the design process for AR includes a lot of different steps and each one of those includes a wide variety of software that the designer should be comfortable with. Our solutions require both 2D and 3D design proficiency which I am realizing is an uncommon skill set to find. We want the brand’s content to be exclusively 2D so they can use existing marketing materials (like images and video), but we want to present this in a 3D environment.
In regards to augmented reality and my YouTube videos, I do lean more heavily to the technical side but my content does arouse interest with designers. Through this, I have met a lot of talented individuals and I’m grateful I can help them build out solutions in some way.
How do you think Markerless AR will be infrastructuring our future as urban citizens?
Being a little speculative, I believe it’s going to add another dimension or layer to our reality. The digital layer will be superimposed on the physical, and will show ads, location-aware social media posts, and maybe additional information about businesses and other points of interest around you. I have heard of a few companies that are currently selling this virtual real estate tied to physical locations. I have been hearing about a startup called “Earth2” a lot lately, and they are doing just that. This “metaverse” is going to need a central body and infrastructure to dictate what virtual content can go where but ultimately whatever firm wins the race to get these devices in consumers' hands (or over their eyes) is going to have the final say. The concept of the metaverse is a core vision for most of the companies dealing with XR like Facebook, Microsoft, Magic Leap, Snap, TikTok etc. We can think of it as a collective visual shared space that lets us bond virtual and physical content seamlessly.
In one of your videos, you did some research into synthetic databases. Synthetic databases are empowering AI algorithms with “fake data.” You faced the challenge of making a database on your own to see how that process works. What are the issues you found? And how can this work affect the future of Augmented Reality for urban spaces?
I need to preface this by saying that machine learning is not my main focus, but one major problem with training custom machine learning models to recognize various items is gathering the data. In order to train a model to recognize where an object is located inside an image, like a dog, for example, you need a couple of hundred images of dogs. This could be easily obtained from a google image search. If you wanted to track something more non-traditional you may have to gather and label the data yourself. That means taking a few hundred pictures yourself (or a few thousand if you are training from scratch) and then labeling each image on the computer by drawing a box around the desired item. To make things more difficult, each image needs to be in a different environment with different lighting in order to train a robust detector.
This is a blocking factor for a lot of people and use cases. Synthetic data comes in handy since you can use a computer to generate and label this training data automatically. My project in particular used the unity video game engine to achieve this. In this experiment, I decided to track a few different Pokemon plush toys because these would not be included in any publicly available data set. So I 3D scanned them with an iPhone to obtain a 3D model of each one. I put those 3D models into Unity and made an application that would take pictures and label all the data automatically. Instead of taking days to create a new dataset for training, this application could create training data that was ready for training in TensorFlow in a matter of minutes.
In the context of urban space, this tool could be powerful for detecting architectural elements and showing some type of experience at each one. You could train that model with synthetic data since a lot of urban scenarios are very similar to each other.
How do you think AR + Machine vision by Apple or Snap could influence how we experience the city?
I think there has been a lot of influence already. Google has live language translation. This is incredibly useful for tourists if provided in the right context. As these types of vision capabilities get better and possibly tie into location-aware applications, I can totally see people having the ability to click or select an object in the real world to get more information about it. Augmented reality ads are imminent. Burger King did something a year or two ago where you would scan a Mcdonald's ad, it would catch fire in AR, and then you would be presented with a Burger King coupon.
AI is always embedded in Augmented Reality, customized inputs are needed to enhance the experiences and AI offers a great solution to that. What are the best tools to use if someone would like to get started?
Heartbeat.fritz.ai has a dataset generator for mobile machine learning which I have never used personally but it sounds like a great way for individuals to create their own data for custom machine learning applications that are optimized for mobile. Another one I have not yet tried is Unity’s perception tools. They basically came out with an official way to create synthetic data in their video game engine (so you don’t need my hacked-together solution). Another huge problem when you start trying to work in this sphere is that running machine learning models in an optimized fashion for mobile is very difficult. Apple has coreML and Google has MLkit for this purpose. A really attractive looking solution for me lately has been SnapML. This allows you to run ML models inside Snap studio in the .ONNX format. This is similar to Unity’s Barracuda engine but they have done all the optimization work for you. Effectively you can quickly test your models on mobile through Snap studio after you get it into the right format.
Stay tuned for the next part in this series, releasing tomorrow, a conversation between Alessio Grancini and Lucia Tahan, Product Designer for AR/VR at Facebook Reality Labs.
Interested in learning more about the future of architecture and the shift to smart technologies across the world? Make sure to check out Smart Buildings/ Smart Cities: Integrated Equity & Resilience, February 9 & 10 (8:30am - 12:30pm Pacific Standard Time). Innovators and experts from London, Silicon Valley, New Orleans, and more will present new advances along with Alessio who will discuss his work in "Planning the Digital Through Augmented Reality: Empowering designers in the new city : converging professions, approaches, environments."
With a hybrid background between design and technology, Alessio Grancini is a prototype engineer at Magic Leap. He is interested in AR and VR meeting human interaction. He taught Unity workshops internationally, exhibited work in art galleries, and published multiple articles about AR/VR.
No Comments
Block this user
Are you sure you want to block this user and hide all related comments throughout the site?
Archinect
This is your first comment on Archinect. Your comment will be visible once approved.