Introduction
Microsoft Dynamics 365 Layout is a mixed reality application that enables you to take your physical designs from concept to completion with confidence. Layout helps you make informed decisions before you build. Visualize room layouts with holograms in the physical world or in virtual reality. Easily import floorplans from Visio and 3D models from other apps to experience designs at scale and in context.
Feature Set
| Place | Modify | Copy | Group |
| Measure | Nudge | Collision | APR |
Issues
Layout was released in Oct 2018 with a focus on space planning and design of manufacturing spaces. Since then, the product had seen low to limited success with our customers. It was later pivoted to an even narrower use case of Fit Testing - used only to test a machine's fit in a given space and saw even fewer MAU. It also had failed to bring in strategic alignment with Microsoft D365 suite.
There was a strong need to pivot away from the fit testing scenario and to find newer genuine use cases for Layout that bring in tighter alignment with D365 and other mixed reality offerings from Microsoft.
Secondary Research
Layout needed to find the right industry need keeping the core IP of Layout and
leveraging the D365 portfolio. We wanted to leverage on the Layout IP of spatial understanding and its design tools and create an offering that is relevant to the industry of tomorrow.
Our secondary research involved finding industry needs, competitive analysis, generating broad hypothesis and gathering needs through customer interviews.
Industry Needs
If we look at the history of the manufacturing industry, it has been through three radical shifts, starting from steam power through assembly line processes and the push of automation, the tools used also changed with the need of the industry, we had the dwaring boards, then came CAD, and we now have BIM 3D models.
With the increasing adoption of IoT, the next wave the Industry 4.0 will be about not only gathering data these devices and use artificial intelligence and machine learning to generate insights on machine parts to inventory management and product life cycles.
Digital Twin
In the previous section we understood the evolution of the industry, the next step was to look at the emerging technologies that would aid this new era of manufacturing that lead us to a technology that is so anticipated that its become a buzz word in the manufacturing industry - Digital Twin
What is a digial twin?
A digital twin is a digital replica of a physical object or system. It includes small products to large items such as buildings, factories and even cities. The idea first arose at NASA: used on the ground to mirror and diagnose problems in orbit (Apollo 13). A digital twin can be used to mimic the physical object/environment and run low cost time effective simulations to gain critical insights about your processes before you commit.
Customer Needs
We wanted to understand first hand from the industry if they felt the need to develop digital twins in their factories, what it meant to them, how far have they come in their journey if any, and what would be their core needs that must be met for them to adopt this technology into their processes. Within our weekly calls with s400 customers like ZF, Toyota, Chevron, we began our enquires to unearth needs and infer opportunity areas.
Almost all of them were eager to implement their version of digital twin solutions, and some of them were already on their journey of doing so but understood that they would fall short of implementing it on a large scale on their own.
Customer research insight document :
| ↓ | DT Interviews | PDF |
|---|
Top Three Requirements
01
02
03
Near term and long term goals for digital twins
The near term comprise of having a connected digital twin which is an accurate representation of the factory while long term goals include running simulations and using machine learning to optimize the production process.
Need for scalable solutions
A diverse range of equipment with varying
endpoints for data collection which makes it difficult to
create a one-size-fits-all solution.
Customers expressed a need for a DT solution that can scale to incorporate all the endpoints.
Accurate and up to date representations
Manufacturing customers face challenges with maintaining
accurate 3D models of their facilities that are a representative of the physical world and expressed a need
for easy solutions for keeping these layouts up to date.
Strategy Hypothesis
To actualise the digital twin we divided our strategy into three stages.
Step 1: Constructing the Digital Twin
Constructing the digital replica of the factory floor using Scene Understanding (SU) and Scene Recognition as a Service (SRaaS) capabilities of the HoloLens to build planar representation of the factory and Object MR to recogonise the machines in the factory and overlay them with digital replicas.
Step 2: Making the Digital Twin data aware
Microsoft's portfolio contains diverse products, D365 ERP encompasses every business operation, from Finance & Operations to Field Services. PowerBI is a strong data viz. tool and Microsoft Flow automates IFTTT operations. We can leverage Microsoft's existing products to create an intelligent, data aware digital twin.
Step 3: Deriving insights from the Digital Twin
The advantage of a digital twin is not only to gain insights and take actions in context at the shop floor but also from anywhere in the world remotely. A remote miniature version of the factory floor can be used to co create, group decision making, remote expert help and to execute physical commands remotely.
Build
Connect
Operate
Storyboard
Prototype
We went with a broad range of prototypes that represent the power of digital twins + mixed reality. The idea was to use these prototypes as a conversation starter to understand the gaps and opportunities that they might spark when experienced first hand. Below are some of the main prototypes that were presented.
Primary Research
We had the opportunity to visit our customers (ZF, SC and Chevron, LA) and present our prototypes as conversation starters to gather needs from their leadership, management and most importantly the front line workers. Our methologies involved : Demos, foucs group studies, 1:1 interview and shadowing. The team was made up of the triad (PM, Engg and UX) + a design researcher.
Strategy Based Inights
Spatial twins can underpin and connect needs across operations, engineering, planning, and more.
A well-understood spatial model (vs schematics) as a central reference has huge potential for wide variety of data streams pertaining to the operation of a facility, all of which cross at the point of physical elements in a space.
3D visualization holds huge benefits.
3D visualization allows more holistic understanding, by allowing engineers to view designs prior to installation, and
by providing a quick method for collision testing with real-world elements.
Maintaining as-built models allows faster, more correct engineering and operations actions.
Due to
disparities between archived "as-designed" models and the true "as-built" state of facilities, engineers are wary of errors. Maintaining a single
"source of truth" in 3D model would allow greater accuracy of actions.
Artificial Intelligence and Machine Learning to derive insights
Providing real time insights for faster turn around time, better root cause analysis and also using AI for predictive maintenance was seen as a potential use case.
Proposal Ideation V2.0
After an insightful primary research, we got back to the
drawing board with the opportunities we collected, we had four months to come up with more detailed solution oriented prototypes for the next round of user testing.
Our digital twin strategy was a plug and play model over the scanned spatial view of the factory floor,
Layout was a means to connect disconnected data systems to form contextual and meaning full data insights. For this ideation our scope was to
plug in 1P apps like Power BI for data viz., D365 Field Service for work order information, D365 Remote Assist and D365 Guides as tools for learning and communication.
DISCLAIMER : The ideation process was not in the sequence presented below, It was more haphazard and has been restructured for better storytelling and understanding.
Download the full ideation documentation here:
| ↓ | Ideation file | PDF |
|---|
Design Principles
|
|
|
|---|---|---|
Building Blocks
Overview
The goal of the overview is to provide a quick glanceable view of the day i.e, the number of scheduled work orders, their priority, a brief description and their current status. Each work order in turn contains its own checklists, data, media and documentation recommendations that are explored further in detail.
Behaviour
We had some great learnings from the usage and customer feedback of the existing layout menu - which was limited in its interaction pattern to GGV (Gaze, Gesture, Voice). The video below explains the primary way of interacting in Layout.
|
|
|
|---|---|---|
| World Locked | Resizes by Distance | GGV Interaction |
| Menu blocks Holograms, and vice versa |
Hologram clutter during design and presentations |
Fatigue from arm above heartline |
HoloLens 2 provides articulated hand tracking and near field manipulation that allow natural interaction patterns that can be leveraged to create intuitive contextual interfaces, is easily invoke-able, hands free, and is intuitive to use.
DISCLAIMER: Prototypes were recorded at home, during the COVID 19 pandemic using Win VR + Leap Motion using MRTK.
Spatial View & Navigation
One of the core advantages of a digital twin is its real time spatial awareness. We wanted to reimagine the dollhouse to serve the maintenance engineer. Our primary research revealed the need for engineers to quickly orient themselves to their space, get to work orders faster and identify any points of interest.
With the data overlays we can not only accurately point to all the work orders spatially, we can recommend the best paths, areas to avoid, provide turn by turn navigation using anchored holographic signages.
3D Overview : An as is 3D miniature representation of the factory floor. Quickly gets cluttered with information overlay.
2D Overview - A schematic representation of the factory floor with a 3D view of all points of interest.
Root Cause & Fix
On a factory floor time is money, any downtime can cause loss of hundreds if not millions of dollars. Therefore, it is extremely important for the maintenance engineer to not only quick fix the issue but to get to the bottom of the root cause quickly and deploy a solution with confidence. In this section we'll take you through some explorations that provide the right tools to the engineer to fix the root cause with confidence and ease.
Checklist
Once the maintenance engineer reaches the affected machine, the first order of business is to do a status check and record any findings, a checklist is often used to follow a step by step standard routine so no stone is left unturned, it also acts like an audit report for future use. Currently this is done mostly on a piece of paper, something that the engineer has to carry along, and submit at the end of the day as a part of their work order summary process.
A quick audit of a few company checklists gave us the top types of input that is required from the engineer - step complete, boolean input, multi choice input and value enter. For this prototype we refrained from subjective inputs as they are tedious to enter using holographic keyboards and voice based inputs might not work in a noisy factory floor environment.
Data & more
A core value prop of our digital twin strategy is to enable frontline workers with real time on demand in context data, which enables them to take more informed decisions. In context documents and media add to information on the go promise. 1P apps like Remote Assist and Guides provide communication with remote experts and step by step guidance respectively.
The engineer is provided recommended data, documents and media based on previous usage, fixes performed on similar work order types. The recommendations improve over time providing the engineers precisely what they need when they need it. So they can focus on deriving a solution rather than hunting down the right information.
Window Interaction / Data : Window placed at a distance of 2m for optimum gaze comfort, Billboards. Follow me can be activated to carry the window along.
Window Interaction / Media & Docs : Window placed at a distance of 2m for optimum gaze comfort, Billboards. Contextual recommendations.
Open Windows : A safety/convenience feature to access all open windows from reachable distance instead of reaching out to them in an unsafe environment.
Rich Documentation
Documentations is an overhead for maintenance engineer - used more often than not for audit purposes, the longer it takes, longer is the delay of the engineer performing the actual fix. The images and videos captured using a digicam are not linked to the work order and are usually close up of parts of a machine that are hard to decipher later by both the engineer and other stakeholders.
Our goal was to eliminate the overhead of documentation by automation, rich capture using HL2 capabilities and additionally re-route these it into the data pool to help other engineers watch and get to the fix faster.
Image Capture Mode : Air-Tap to quickly capture a still. Mode allows for batch capturing.
Video Capture Mode : Capture motion for audit / tutorials with co-ordinate information.
Auto-Generated Anchors
Each physical machine has an underlying 3D object. When the engineer captures a fix, HoloLens can track the gaze and spatial co-ordinates of focus on the machine, creating a heat map of focus and then using it to enrich the capture with spatial information and anchor itself to various parts when used by others. Orienting viewers spatially to the part the capture is referring to.
Anchors : Associate the current data in window with part/area in space. Adding a spatial context to the information.
Spatial Scrubber : A non-linear, anchor based scrubber allows to spatially jump to point of interest.
Summary
Finally, summary provides the engineer with a quick overview of all the steps performed on the work order, captures - audio/video are attached with a auto generated captions for better recall later. The engineer can choose to view older work order summaries to understand what issues have taken place earlier and what were the fixes performed.
With the help of layout and its tools, Emily is able to complete her first work order of the day faster and with confidence. But her day isn't over, she's off to solve the next issue in her list...
"One of our visions for HoloLens is we want to be able to walk out to any piece of equipment in any facility anywhere in the world, and be able to see all data, basically holographically floating above, in the right context for the work, the actual work process that person is providing. That's the vision."
- Factory Floor Manager
Release Work
As a part of regular release cycles I was involved in weekly customer connects to gather requirements/feedback and owned the experiences of end to end features. Below, I summarize a few key feature releases I was involved in.
Import Tool
Microsoft D365 Import tool for Layout is a 3D asset management & conversion Windows (UWP) app that optimizes and prepares your models & layouts for mixed reality using the Simplygon cloud service. Optimization involves polygon reduction, remeshing, texture optimization & conversion to Layout consumable formats for it to be viewed & interacted with in various endpoints without any performance concerns.
Features & Fixes
1. Model re-optimization workflow - Depending on the number of estimated 3D models in the layout, the primary target device, the user can optimize and re optimize the model to get the desired output.
2. Asset concierge - An end to end workflow for users to submit their 3D models securely to Microsoft and have us optimize their model for them and download it directly in app when ready.
3. UX refresh - The V1 of the app had multiple usability problems that we wanted to tackle as the part of the release. This also included the UI modernization effort - Fluent-tising and dark mode.
4. Accessibility - Correcting the color contrasts, defining narrator talk back, keyboard tab order and localization.
To know more about the design process :
| ↓ | Import Tool Doc | PPT |
|---|
Locked Layouts
The ask for locking an entire layout and/or locking a particular model within a layout came as a constant ask from customers - during presentations the viewer might accidentally move the holograms around leading to reworking on the design. The constraint was to use existing Layout UX patterns and produce a low cost layout locking mechanism to unblock users.
To know more about the design process :
| ↓ | Locked Layout Doc | PDF |
|---|
Layout on HoloLens 2
Layout was set to be an inbox app on the HoloLens 2. This meant we had potential customers trying out the application across the world. From a UX standpoint HL2 not only introduced new interaction models but also came with a bigger FOV. I was the POC for a bunch of work streams for this release.
GGV to Far Field Interaction
We wanted to move away from GGV to newer ways of interacting with holograms. We did an audit of all UX surfaces in the app and assessed the changes that needed to be made to successfully port to FF. A user test was done that gave us great feedback, we performed a few iterations before getting ready for the release.
| ↗ |
|---|
Simplified Room Scan
The steady feedback from our customers was to simplify the process of scanning a room and getting started, especially in the context of a factory floor where selecting the walls and the ceiling was a tedious if not an impossible task. We reduced to step to only scanning the floor to get started.
| ↗ |
|---|
OOBE & Demo Layout UX
As an inbox app, we wanted the users to try the app without signing in. This experience was broken into two parts an auto playing 3D OOBE experience and a demo zone play ground to try out various features of the app.
I was UX consultant on this feature. Please reach out to @ThommenLukose who lead this effort with his art team for more.
| ↗ |
|---|
