About three decades after the term “metaverse” was coined, Facebook rebranded as Meta as they increased their focus on building virtual and augmented reality platforms. Today, Agustya Mehta helps lead Meta’s development of the next generation of XR products in his role as their Director of System Platforms for Reality Labs Hardware. 

An exploded view of the Meta Quest 3 VR/XR headset. Image used courtesy of Meta

 

In this episode of the Moore’s Lobby podcast, Agustya and host Daniel Bogdanoff discuss building tomorrow’s AR/VR systems using today’s technology. Agustya also shared lessons learned along the way from working at several of the world’s leading tech companies including Apple and Microsoft.

 

A Big Thank You to Our Sponsor! 

 

 

Meet Agustya Mehta

As Director of System Platforms within Meta's Reality Labs hardware organization, Agustya Mehta leads a cross-discipline engineering team responsible for early development platforms for augmented reality. Prior to Meta, Agustya led the hardware team at Nauto (driver safety / autonomous vehicle startup), led electrical engineering teams at Microsoft (HoloLens), and held electrical engineering design roles at Apple (iPhone 4S, 5, 5S, 6) and Microsoft (Xbox 360 S, Kinect). 

 

 

Agustya holds a number of patents related to his work on consumer electronics. He received his BS in Physics, BS in Electrical Engineering, and M.Eng. in Electrical Engineering and Computer Science from MIT.

 

Transcript

The following transcript has been lightly edited for clarity.

 

All About Circuits’ Daniel Bogdanoff: From EETech Media and All About Circuits, this is Moore's Lobby. I'm your host, Daniel Bogdanoff. Today, to kick off season seven, we're joined by Agustya Mehta, the Director of System Platforms for Reality Labs Hardware at Meta.

We're going to explore the engineering behind cutting-edge virtual reality hardware and systems that are ushering in the metaverse. We'll chat displays, SoCs, batteries, and a lot more. As a side note, this episode is borrowed from an Industry Tech Days keynote, but we enjoyed it so much we had to bring it to you here on the podcast as well. So let's get going. Augustya, thanks for being here. I'm really excited for this.

Meta’s Agustya Mehta: I'm super excited too. I think spending some time with the engineering community and just talking about war stories is something I'm really excited about. So looking forward to it.

 

Entry Into Technology

DB: I'm curious, what first got you excited about technology and the world of engineering and tech?

Mehta: I've always liked figuring out how things work. This is probably a fairly common story among engineers, but I loved taking things apart when I was a kid; sticking my hand in the VCR and realizing that if I whittled the right wheel, it would change the time indicator on the display, stuff like that. Sometimes I'd be able to put stuff back together.

I think that kind of kicked it off. Definitely have the knack of that in my family. My dad's a physicist by training, a lot of my relatives are. So that core of what is going on behind the scenes and how do things work has always been a curiosity for me.

DB: Do you have any stories of things you took apart that couldn't get back together?

Mehta: I forgot a lot of them. My parents would have even more. I think probably one of the first ones was an old Nintendo that I had as a kid. So one of my big things growing up was playing the NES, and at one point my cousin's NES stopped working. And so we were going to take it apart to try to figure out whether something was wrong with it. I was pretty little at this point, so I'm not sure what I would have done with it once I took it apart. And then there was a little bit of that panicky horror feeling when trying to put the thing back together.

Now with the stuff I'm working on doing that sort of is small potatoes, but at the time was a little bit daunting and intimidating, but also inspiring. I kind of wanted to figure out, well, what are all these components and how does this thing work?

 

Academic Career

DB: So is that why you ultimately ended up going into electrical engineering for school?

Mehta: Partially, yes. I think part of it's that, and part of it is of the engineering disciplines. So you learn physics in undergrad, and usually it starts with kinematics and mechanics. There are a lot of really big innovations that have happened in the past like the invention of calculus, but fundamentally, like throwing balls and seeing them hit each other, there's some intuition around that, right? Like, physical things are what we've evolved to deal with.

Electricity and magnetism have always fascinated me because it's sort of the closest analog that we have to magic. If you were to go back in time and flip a light switch and turn the light on, explaining that to people who didn't already have some understanding of what's going on… it's kind of cool. So I think that's what's always attracted me, no pun intended, to the EM side of things.

DB: Even now, it's a little bit of magic still. And that's part of the fun of the space.

Mehta: Yeah, I like things that kind of go beyond intuition.

DB: So I was looking. You have a bachelor's in EE and physics, and then you also did, like, master's in computer science. Can you tell me about your cross-discipline training and how that's informed formed your world?

Mehta: The way I've been taught from a young age about how to think about problems is to have a strong, fundamental core. If you have a really strong foundation, you can kind of build up anything on top of it. If you spend your time just trying to memorize answers, cram for tests, that kind of thing, it tends to be a little bit of a weaker foundation. And even if you get a good grade, you probably won't remember this stuff five years down the road.

So to me, that's what physics is all about—the fundamentals of how things work. As an undergrad, I started with a focus on physics courses, partially because I was interested in it, and partially because I have a lot of people in my family that went down that path. That felt comfortable and familiar.

And then I started taking on more of the EE stuff just because it was fun and there was sort of like this tangible thing that you could build and see immediate results from. It still tied into the physics stuff. In retrospect, did I have to do both? I probably would have had more fun as an undergrad and maybe spent more time on some of those formative networking experiences people talk about had I not.

But I'm pretty happy about it. And there's a decent amount of overlap. Right? Like a lot of the solid-state materials and optical electronic stuff that I did as an undergrad kind of spanned both, so I'm glad I did it. Although, was it strictly necessary for my career? Probably not.

DB: Well, I don't think you're supposed to have fun at MIT while you're going to engineering school. I thought that was the rule.

Mehta: Actually, it's a little bit different. It's pretty bimodal. There's not much in the middle. There's a lot of partying and quite a lot of fun that happens there. And then there's a lot of insane stress and imposter syndrome you build because you're around all these people who are way smarter than you. Not much even-keeled in the middle.

Looking back, I tend to have the rose-colored glasses and think fondly of how much fun I had, because there was a lot of fun. But at the time, there was a lot of stress, too. MIT has this informal slogan, IFTFP, which stands for I hate this place. Yes, but there are a lot of acronyms around it. The institute has fabulous professors and stuff like that. Extreme work hard, play hard kind of place.

 

Hacking the Game Genie

DB: Touching back on the NES, I heard you figured out a way to hack back Game Genie a little bit. Can you tell me a little bit about the early hacking days for you?

Mehta: Sure. I mean, I don't want to overstate either. I think probably a lot of folks in the audience played with the Game Genie and other kinds of tools like that.

I think, for me, the thing that was fascinating was noticing patterns. The premise of this thing was basically there were four hex inputs that would map to an address in memory and four that would map to the value to put in that address. That's how the thing works. It's not so straightforward. They don't use one through F. They use, like, random English characters, and it's not one-to-one. Stuff scrambles around a little bit.

But still, if you take the codebook that ships with the thing, you can kind of figure out that, hey, if I tweak some of the things in that last section of four hex characters, I can change the value, and you start noticing patterns. And to me, that was super fascinating, like kind of understanding maybe how things are stored in the game, right?

Text in a lot of these old games wasn't logical text. It was addressing something in a sprite table. A lot of times, people would reuse sprites for things like Super Mario Bros 3, the little bushes with the smiley face, the clouds, a smiley face, those are actually like the same sprite, just different colors. And so you can kind of get pieces and glimpses of how people thought about putting those games together, even from playing around, trying to make up your own codes with Game Genie.

I remember a lot of fond memories. My brother and I, as kids, liked trying to map out our own codes by figuring out patterns from the ones we had. And we had a whole handwritten notebook of codes we'd figured out over time, which was great.

 

Working At Microsoft and the Xbox 360

DB: That's awesome. Yeah, they do some fun things with the music there, too, back in the early days, but that's probably out of scope for today. But touching on game systems, your first job was to go work with the Xbox 360. Can you tell me how that came to be and what you were working on?

Mehta: Yeah, given that was something I was super into when I was growing up. Obviously, I was more of like Team Nintendo as a kid, but when I was finishing my master's, there were a few different options. Some were sort of like traditional software engineering jobs at Big Tech companies.

When Microsoft came through with the offer on Xbox, I just kind of went for it because it was a product I used. It was something I was super passionate about. I actually didn't spend too much time thinking about the details of the day-to-day of the job so much as like, this product is awesome, I'm going to just go for it. That was the big attractor for me was the product.

DB: So what was your job? And did anything about that surprise you as you started up?

Mehta: Yeah, I learned a ton. I'm going to disclaim here that I don't want to disparage anyone or any discipline. I think that the whole gamut of hardware engineering, from conception to making early designs and architecture, to executing on first prototypes, to building the thing, to validation, verification, and support is super important.

That being said, I didn't have a good sense of how large companies worked at that point. The team I joined…there was a rotation program, so I got to spend some time on validation, and some time on design. A lot of fascinating stuff to both. The team I joined was focused primarily on verification.

So after my rotation period ended, I was focused on voltage regulator validation for the Xbox 360s, which is like their slimmed-down version, and the Kinect, the first depth camera that we shipped with the Xbox. I had a ton of fun. I learned a lot.

I also kind of learned that in large organizations, sometimes there can be silos where you focus only on one thing. And I found that I really love the entire system and not just verification and validation. So I'm glad I took the role.

I learned a ton. If I were to go back in time and kind of nudge myself in a direction, I probably would have pushed myself to design earlier on. That being said, I think the lesson I learned was on building my own teams. I try not to create hard lines between a validation team and a design team. I like the concept of engineers that are building designs, taking ownership of the thing from beginning to end, and figuring out how not only how a circuit would be implemented, but how it would be validated and making sure that it is actually a reasonable design and not just a theoretically good one.

I think if you have too much focus on having a set of folks focused only on validation, you can risk making them not get to experience all the fun of creating the design and working it on the front end. So that was kind of my takeaway from that experience. I had a lot of fun. I learned a lot. But I like to blur the lines between those two disciplines.

DB: The cross-functionality seems super important. Even just having the context of this has to go through verification and here's what that looks like can influence your design work as well, which I think is really important.

Mehta: Yep. I think this is a non-technical thing, but I also learned a lot about how people work and how human structures and human organizations work. Most companies have some level of a pecking order, whether it's attributed to their culture. In Apple's case, design and sort of the look and feel of the product kind of takes precedent over everything. Whether it's fair or not, a lot of times in engineering teams, you will notice a little bit of a pecking order as well. And if I'm going to be very explicit, it can sometimes come down to design maybe taking precedent over validation. I don't think that's right. And I've tried to basically make that not happen in teams I've built further on down the road. But it was something I definitely observed going into it and a valuable learning that I would not have gotten from school.

 

Apple and the iPhone

DB: And you mentioned Apple. You spent some time there working on the iPhone. Can you tell me a little bit about your time at Apple?

Mehta: We had this saying from undergrad about how MIT education is like drinking from a fire hose. I definitely felt that way about Apple! I joined in 2011. Day two I needed to push a schematic into perforce for the iPhone 4s. It was one of those things where they were trying to get me to learn the tools, but this actually was an update that went into the design for that product.

And then beginning of week two, my manager, who I still stay in touch with, a really great guy, he pinged me and was like, hey, do you still have a China visa from your last job? I need you to fly to Shenzhen, ideally next week for the build. Can you do that?

So that was definitely a really fun just jump into. Let's see how things go kind of know. People talk about moving fast being a part of many different companies' cultural mantras, and I definitely felt it there at Apple, even though they don't use those words there.

DB: Yeah, that was a big era, too, for optics and camera systems. I know we talked to Mark Papermaster as part of the Moore's Lobby podcast, who was helping lead that charge, and he talked a lot about cameras. How did that movement? Or were you involved at all in that camera optics movement as part of that 4s phone?

Mehta: Very tangentially, I was on electrical system integration. So I got to touch and play with every subsystem in the design. The areas I focused on more were on sensors like Touch ID. So I worked with the camera team a bunch, did a lot of work with trying to mitigate desense or basically jamming the antennas based on noise generated by the camera modules.

So to that extent, I had some interfacing in. But I would say where I really dove in deep was just understanding the system as a whole and how to basically cram a bunch of things that don't really want to coexist into a small volume. I would say that is very applicable across the mobile space and AR/VR, and basically any modern consumer electronic design, especially those.

DB: Are there any other key takeaways from your time at Apple that you've taken with you throughout your career?

Mehta: Yeah. I worked a lot on FPCs or flex circuits, rigid-flexes, as well as just pure flexes. And that line of thinking where you don't just purely implement an electrical schematic into a 2D rigid board. Not trivializing the rest of the work that we've all done, but you also have to spend a ton of time with mechanical engineering teams and DFM teams to understand how the stuff's going to fit and fold and coexist.

There are constraints—you might have PCIe or MIPI on one of these things that has to be a well ground referenced, but it's got to bend and fold in ways that don't enable you to have a solid ground plane. So how do you deal with cross-hatching ground and having things twist and bend within that? You might have flexes that dynamically bend and flex. They're connected to buttons or things like that, and over time you'll find that copper traces crack unless you think about how they're routed through the bends and what type of material you use. These are all things that I learned a lot about through the process of implementing designs and working with people outside of the electrical discipline.

It made me a lot more comfortable working on cross-discipline designs and not just staying within my EE bubble. And I've been able to take those learnings and apply them to pretty much everything else I've done. So that was great.

I think the relationships I built there were awesome too. One of my mentors from Apple is the person who pulled me out from Apple to come back to Microsoft to go on HoloLens. So this is probably a common thread among career conversations, but that networking and interpersonal interaction you have, those things carry with you your whole career. They're super important.

 

Microsoft Take Two: Working on the HoloLens

DB: Yeah, it's interesting to think MIPI, especially like CFI-DFI, it's all about getting camera’s displays back and forth. There's a lot of routing and flexing and movement going on there. Seems very relevant to everything you're doing these days. You mentioned your second stint at Microsoft. You went over to work on the HoloLens. Can you tell me about the early days of that and what that program looked like?

Mehta: You know, it's funny, a piece of feedback I often give people I interview when they ask about the company I'm working at. Now, we were told this is like a startup inside a big company, and I often roll my eyes at that because startups are startups and big companies are big companies. They're both cool. There are good things about working at both, but they're very different.

Microsoft was no different. But I will say that working on a fledgling team that is just being built even within a large company is special and really fun. And that early time in HoloLens, those first years there were awesome. We had a really tiny team. We didn't know what the heck we were doing. We were building stuff that no one had really built before and trying to take huge benchtop prototypes that were non-portable and consolidate them into an untethered headset that was basically a full-on computer with a display no one had ever invented before. So yeah, I look very fondly on those times. I learned a ton and built a lot of great relationships. It was a blast.

When you look at it where it was then versus where it is now, I know there's a lot of changes and a lot of evolution that's happened. But if you had to pick one or two, what do you think are the most significant advances that have been made?

Mehta: Specific to AR/VR, or consumer electronics in general?

DB: From HoloLens to what exists today on the market?

Mehta: I think the really big advances I've seen are advances in silicon development. As a systems guy, it's very tempting to think of the silicon design as a black box, and I try not to do that. But advances in performance, especially when compared against thermal output or power draw, have been huge. A modern mobile SoC today is capable of a kind of order of magnitude more operations per second than even at that time. And seeing that advancement has been great.

I think there's also been a push within the silicon design world on application-specific subsystems. Within HoloLens, we actually had a custom silicon design drawn up to do a lot of the graphics processing and head position detection. A lot of that stuff has now been implemented in mobile SoCs that are on the market today, which is really exciting.

In addition, there are a lot of DNN blocks that have been implemented in modern SoCs to essentially allow compute of some very basic ML work on board the device without having to upload stuff and send it back to the cloud, which has been really cool. So I think that's been really big.

The other big bucket has been on the display and optic side, and a lot of the magical stuff that I'm working on now, unfortunately, I can't talk about. But there have been some really large advancements in microdisplays, in optical combiners, and the kind of technology that will enable us to one day implement a really full-fledged AR system in this form factor, which I think is great.

Tons of other things going on, too. Battery technology is huge, materials technology on pulling heat away from parts of a design, antenna design, and all sorts of stuff. So I am not meaning to trivialize any of those other things, but I think probably silicon tech and display tech are the two big ones that come to mind.

DB: To me, it's kind of interesting, because if I had to guess, I would say it would go the opposite way, where you started with off-the-shelf type components and then moved into full custom silicon, but it's almost, you had custom silicon, and now those features are getting implemented into SoCs that exist.

Mehta: No, you're not wrong. I mean, I think more of what I'm saying is a lot of these things that were kind of off the wall are now becoming mainstream.

DB: That makes sense.

Mehta: Again, without leading the way on stuff I can't talk about. There is definitely work on custom designs going on within all the major companies working on this stuff, and I think that will continue to happen. But I think without HoloLens you probably wouldn't have the interest you're seeing among the big silicon vendors on making AR/VR a thing they care about in their roadmap.

DB: In the world of AR and VR devices and wearables. What's the situation for EMI and EMC? Are there any unusual things happening there?

Mehta: This probably ties in a lot to my work at Apple, although it was pretty relevant to HoloLens as well. And this is kind of that stuff I was mentioning earlier around thinking beyond your immediate discipline and thinking at a whole system level.

For example, the naive approach to designing something that is going to radiate noise is to make sure that you stop problems at the source. So shield your design really well. Space something that's noisy, far away from something that's sensitive to noise. Isolate the power going to your noisy stuff from the power rails going to your sensitive stuff.

Those are all good design practices, but you can't always do that in a tight form factor. So how do you still ship your design? You can get creative. If it's a harmonic of your clock frequency that's coupling for one of your power supplies, that's coupling onto one of your radios…maybe when you're using a given cellular band or wireless band…you could shift your clock frequency of the aggressor so that that spike doesn't go into the band you care about.

Obviously, you still have to meet compliance requirements, so there's no hand-waving around that. But at least in terms of your own systems performance, you can do tricks like this to make a product that fundamentally can't be large enough to isolate everything the way you want to is still to work.

The other thing that is important to think about is planning ahead in your design. So early on, thinking about what types of components you might need for isolation…high-frequency caps, common mode chokes, and ferrite beads. There's a fine line between sandbagging to make your job easier later, which I'm not advocating for, and thinking ahead to make sure you don't put yourself in a box.

When you discover that maybe a sensor module pollutes the area around it with noise, then you've got a couple of tools in your back pocket to isolate it out by sticking some components in before shipping. So like near board-to-board connectors, I like to camp out a little bit of space for high-frequency caps and ferrite beads and those sorts of things. Even if you end up stuffing those with jumpers or in your final design, no stuffing and just having a rail go, a copper go under it.

It's generally good practice to do that kind of stuff if you have the space and the ability. And yeah, you'll get pushback from the product design team because they want to make the thing as thin and light as possible. And that's where the negotiation comes in.

DB: Do you have any moments that you can share where you're like, no, we absolutely need to build in this headroom space and you either got not overruled, but this decision was ultimately, “we hope this works with the amount of space that we have.”

Mehta: Yeah, I can give a vindicating one, and credit for this goes to my manager more so than me. This was one of those first builds I was mentioning where I was kind of being taken under their wing and learning the ropes.

There was a set of basically zero ohm resistors that were left in the design around one of the audio subsystems for one of the products that were released. It turns out that the headphone jack designs in the US and in China are a little bit different.

If you have a headphone jack that's got a mic, there's tip, ring, sleeve, and then another sleeve. So the tip is left, and the ring is right. In the US, that next ring is ground and the bottom ring is mic. In China the next ring is mic and the bottom base ring is ground.

So the codec chip we were using was designed to have different variants for the China market and the US market. But there was this pop-click noise that would occur for the Chinese variant of the phone that wasn't caught until we were basically pretty close to production.

Because we had these zero ohm resistors stuffed, we were able to basically just do a last-minute stuffing option to isolate the problem pin from the codec on the Chinese variant without having to respin the design. So we felt really vindicated that we did that, even though in some sense having that extra component there, maybe it wasn't purely necessary. It actually reduced a lot of risk for putting in a new codec chip into the product and ultimately let us ship without having to respin the design at the last minute, which was great.

 

From the Biggest Tech Companies to a Startup

DB: And then you took a pretty big shift from Microsoft and Apple to a small startup, NAUTO. Can you tell me what was the culture like? You've alluded to big cultures and small cultures being different already. Can you tell me what that shift was like?

Mehta: Oh yeah, huge difference! So part of my personal motivation for doing the startup is that I'd done the big company thing basically since I finished my master's. I had advised and consulted for some friend’s startups, and I was still close with my advisor from MIT who does a bunch of startups on the side.

And there was this part of me that kind of felt a little bit of FOMO, maybe on that startup lifestyle. Just like your destiny is within your control. If you fail, it's kind of your own fault. If you succeed, you are taking ownership of your best success.

By the way, big disclaimer. I'm not saying that's not true in large organizations! It definitely is, but your stake in the game is much greater in a small organization. Touching on my analogy from before, even my early time on HoloLens felt risky and fun. In that sense, we didn't know if this product team was going to be a thing, but we were still within Microsoft.

So in some sense, there was know you have health insurance, your comp is pretty good. If things really go to a bad place, there are other products that you could probably shift over to without the stress of interviews.

Startup is a little bit different from that. So that was my motivation. I really wanted to do one. I'd gotten to a point where I was worried that I was just full-on down the big company path. I'd never do it, if I didn't do it then.

Much bigger differences in how problems were approached. In a large organization, there's a lot of infrastructure that exists so you don't have to reinvent the wheel, like revision control for designs, what design tools are you going to use, or even basic things like how do you collaborate with other people.

At a startup, all that's being figured out. So your design file might be sitting in someone's Dropbox somewhere, or someone's local machine. You're getting the file to the vendor by emailing it to them. Maybe these things aren't great. They're not like a good way of operating. There's lots of room for mistakes, and there's lots of room for getting a good process in place. But a lot of things you maybe would take for granted at a large organization aren't the case.

I think the other aspect that was really eye-opening at the startup was there wasn't anyone else there to kind of be the adult in the room and say, okay, this is okay or this is not okay. We think this is a good idea, let's just go and get it done. I think you can move a lot faster with a smaller team. You can also maybe make bigger mistakes with a smaller team. So there is risk there.

But the way that a small team of folks who are dedicated to one problem and are not mired in inter-organizational politics or worrying about what some VP somewhere thinks can operate…It's amazing how fast you can move! So yeah, I really enjoyed the time.

DB: Of the startup and you got to build a team there as well. What was it like approaching that as a new challenge for you?

Mehta: It was big. I had managed at Microsoft on HoloLens. I built up an electrical engineering team and that was a big leap. It was sort of a natural one. Even at MIT I really loved TAing and tutoring folks and that kind of thing, that really interpersonal, bringing other people up really gets me energized and motivated.

So it was sort of a natural path to go into management, but within EE since that was my core discipline. Even when you transition to management and maybe you're not in all the technical problem-solving, you know what's right and what's BS even when you hire for disciplines outside of your own core.

I would hire power EEs, analog EEs, and folks focused on RF. Those weren't things I had deep expertise in, but still, the core fundamental stuff is in domain. At NAUTO, I had to build up a hardware team that included firmware, mechanical engineering, TPMs, and flexing that muscle of basically hiring people that worked on stuff that I couldn't do myself or couldn't learn how to do quickly necessarily was a little bit scary, but a lot of fun.

I think that's the core thing that I think one needs to develop when you transition into a leadership role— having that trust in others and basically building enough spidey sense and pattern recognition around hiring people who are really good at something that you are not good at and then having enough trust to let them do their thing without needing to be there to go and tell them how to do it. That was kind of the big learning moment for me, I would say.

DB: Do you have any core philosophies or even tactics that you find especially helpful when you're trying to hire someone who's not a direct match for your skill set?

Mehta: Leaning on the network you've already built is a really great start. No person is an island. Having done a bunch of design work in large organizations and having a pretty good network of people I went to school with, you get a sense of people outside of discipline who have a pretty good track record for doing successful things or making designs that work well.

You can lean on them for advice, both for referrals as well as getting a sniff test. Hey, this person worked on product X. You worked on a similar product or the same product…are they legit?

All those things are really helpful things to lean on. So lean on your network I think would be a big one.

The other one is just raw experience. Right after you have a few shipped products under your belt, you start to get a sense of what works and what doesn't. And you can gauge patterns even outside your discipline on who is BSing and who really knows their stuff. And I think that's a really important skill to learn. I've met people who are able to really nail that down quickly, far more quickly than I am.

Some of my friends and people from my cohort and undergrad and whatnot went straight into the startup world or straight into doing VC stuff. And the cynical part of me was always like, well, how are you going to go and buy stuff? You've never really done anything. But I have met people who just have that knack and are able to get it really quickly.

I think that the ability to figure stuff out outside of your domain really quickly is a really valuable one. You can hone it though by building things and shipping them and starting to build that muscle.

 

Meta and AR/VR Hardware

DB: About five years ago, you jumped back into the world of big tech, into Meta. Can you talk about what that transition looked like and what your role is now?

Mehta: Sure. I'll be super blunt, too. I love NAUTO, and I still am in touch with that team. They're still going strong, and I'm incentivized to want them to be successful.

For me, there are two things that caused me to want to transition to a new role. First, just being really personal—I have three boys at home. At the time, my second son had just been born and that lifestyle of going hard and going fast from a career standpoint is really invigorating. But it takes a personal toll, right? Like working 80-plus hour weeks, flying to China on a moment's notice to go to a build. And by the way, at a startup, that's not a business-class ticket.

That stuff flies pretty well when you're making work kind of like the end all, be all, main center point of your life as you're trying to juggle more things. It doesn't mean work has to take a second seat, but some level of balance and time bounding of things and basically being more protective of what time is your time and what time is work. Time is important.

I found that large organizations tend to fit better within that mold, because in a small company, if you're not there to pick up the phone, the company might die. In a large organization, they have the benefit of resourcing things with ways to reduce risk. So that was sort of the personal motivation to switch back to big company land.

Outside of that, NAUTO, at its core, is a data company. The vision there was to collect data about how humans behave behind the wheel, make them safer drivers and train autonomous algorithms to function better by basically learning from how humans behave in weird situations. Highway driving, where everyone's going the speed limit, is not a super hard problem in good weather. But city driving, where people are jumping out in front of you...a lot of that stuff we were hoping to get from the data we collected at NAUTO.

Once that core hardware was built to get a working prototype out the door, I think the long-term path for that company is to integrate with the sensors that already exist within modern vehicles to keep things moving. In other words, I felt like I'd accomplish what I had to do with hardware at that company. And I had a lot of friends in the AR/VR space that were trying to lure me back into that.

And I think, for better or worse, AR/VR is very capital-intensive. A lot of these innovations I'm mentioning around silicon and around display tech require a lot of money and are not necessarily conducive to a startup. And so that's kind of what pulled me back.

Personal motivation for AR/VR, which we can touch on more as we chat, is also around making tech to help people with disabilities. That’s a topic that's really near and dear to my heart is helping folks with visual impairment. I've got folks in my family with that. You can probably tell from these glasses, my eyes are not the best in the world. And so I've seen how things like voiceover on the iPhone have really just changed the world for people who can't see in a really positive way.

My belief is that glasses that have the capability to not just use optics to make your vision more clear, but can use cameras and sensors and compute to describe the world around you or enhance your vision or enhance your hearing, this stuff is just around the corner. I think it's going to improve a ton of people's lives. So that's my personal motivation for it.

D.B.: Thinking through that, as you're saying this, it's helpful for me. I wear glasses, but my eyes are okay. But for someone who has severe vision issues, how much more could AR and VR help those folks? It's a very noble cause.

Mehta: Well, there's this concept called curb cuts. When we talk about accessible tech, the curb cut is like when you're at a shopping mall and you see that the curb kind of dimples and slopes down to get to the road. The reason that is a thing in the US and a lot of the Western world is due to legislation to protect those with disabilities, those in wheelchairs. But the end result is it's really good for everyone, right? Using a stroller or a shopping cart is now a doable thing. I'm a strong believer in curb cuts and basically universal design.

Yes, you're right, someone with very severe disabilities could benefit from this stuff. But I know a ton of people who use assistive features on their mobile devices who are probably not traditionally considered disabled people. For me, I use closed captioning when I watch TV, a lot. I don't have hearing issues, but I can watch TV without the volume cranked up.

I think back to the glasses analogy, there's a use case in a dimly lit restaurant where you're trying to read the menu. Maybe you're not officially visually impaired but you're having trouble with it, like having the text just pop out or read to you, that, I think, would be a really helpful thing for everyone.

DB: Can you talk me through, just on a very fundamental level, the world you envision for AR and VR, how they're different and how they're the same.

Mehta: It's interesting. There's a common mantra that people focus on in the industry saying that, hey, VR is really cool and has made a big splash, but it's sort of limited to gaming. AR has a much bigger market share.

My personal take on it (and my disclaimer is these are my own views and not necessarily the views of the company) is that I think there's a blurry line between them. As you've seen with products like the Quest Pro, which we released about a year ago, we have mixed reality mode where you can see the real world and virtual content on a VR headset. You're also seeing announcements from our competitors that they are trying to follow suit and do similar things. I think that's awesome.

I think that there's a world where a glasses form factor is the right call. I think there's going to be a set of things that you can do on a larger, beefier headset with more sensors that you probably won't be able to do on anytime soon. So I think the space has room for both. Yes, gaming and immersive, entertaining content lend themselves very well to VR, but I do think there's a place in the workplace for VR headsets as well. I think AR glasses and VR will coexist. That’s my opinion.

DB: I would love to get rid of my monitors and just have a comfortable VR headset where if I'm editing a video or something, I just have a timeline that goes all the way across the room. Sounds wonderful.

For a long time, the perception was that these are just big, clunky things, and I have to slap my phone in or be hooked up to an in-room system or a big, powerful desktop computer. Can you tell me about some recent advancements in AR and VR that you're really excited about and maybe have contributed to?

Mehta: Sure. I tend to shy away from overstating my own contributions because a lot of these things are big team efforts. It's not like back in the day where you can say, hey, this one physicist invented this one thing. These are big team things. Some things that I've been thankful for and lucky enough to be a part of, though, are advancements that both Microsoft and Meta have taken around inside-out tracking.

With early headsets, you'd have to set up either sensors inside the room or lighthouses inside the room that would be picked up by sensors on the device to understand where you were spatially. And there are still these super accurate spatial technologies that we use for doing early prototypes and research.

But advancements in computer vision and SLAM (Simultaneous Localization and Mapping) have enabled us to basically build devices that have inside-out facing cameras that understand where the device is in the real world and are very accurate at understanding the position of the device. And then if you augment that with data coming in from IMU and other stuff like that, tou can very quickly update information going to the display based on quick head movement.

There is latency involved in taking the camera data and understanding where your head position is. If you're purely depending upon that, you might end up with swim and stuff like that. You can then augment that with IMU data to quickly reproject the display.

There's some really amazing technology that Microsoft innovated on HoloLens in that realm, and similar technology that Meta has innovated on with the Quest in that realm. And I think those advancements are big reasons why:

• One, we're able to build devices that don't need to be a whole day-long ordeal of setting up a special room to use.

• Two, we're able to start building devices in the industry that can mitigate issues like vestibular discomfort or motion sickness. Basically, the thing that makes people nauseous when they use some of these really early VR headsets is when your inner ear, which tells it's basically your body's IMU, disagrees with what your eye is seeing. You can also get this on a boat or on a roller coaster or other situations like that. And so the better job you can do of locking the actual position and movement of the device with what it's displaying, the less you have those problems.

And I think we're quickly getting to a point where that early feedback people gave on really early HMDs where, oh, this thing makes me feel discomfort or sickness, I think that's quickly going to be not a thing. For me already it is not a thing. And I think for pretty much the entire population, we're getting to a point where people will be comfortable using these.

DB: When when it comes to AR and VR, maybe XR in the middle. How are the architectures the same, and how are these core architectures different?

Mehta: At the core, the architecture for a modern mobile compute device, whether it be a tablet, or a phone, or AR/VR system, there are a lot of similarities. There's going to be similarities in compute. Similar classes of SoCs are used for these products, similar types of sensors with similar interfaces. Cameras connected via MIPI to an SoC, radios and modems connected via a PCIe to an SoC. Those are all things that you're going to see in all cases.

I think the constraints that an AR/VR system has that may be a little bit different than that of a phone or a tablet are size and weight. Obviously, people want smaller phones and don't want to take up tons of space in their pocket. But if you have a lot of weight on your head, it becomes uncomfortable really quickly. And so dialing that down by reducing unnecessary stuff from the HMD is really critical.

That also means dialing down power, because the less power that you're consuming, the smaller your batteries have to be and the less large your thermal solution needs to be, which tends to be a big component of the mass of the device. So I think all of those things, while they are constraints in any mobile device, tend to be a big constraint in AR/VR.

You also run into requirements around mechanical rigidity and basically mechanical interfacing between components that may not exist on a mobile device. Like, yes, the IMU on your phone probably should be pretty rigidly coupled to the chassis of the phone, so it's giving you an accurate IMU reading. But on HMD, it's really important that the device knows where it is in space, and so the display and the sensing subsystems need to be coupled really well together.

I think that AR/VR is also pushing the envelope a little bit in a way that the mobile phone space was a decade ago. On phones, we saw cameras going…this is more than a decade ago, kind of dating myself here…but cameras became kind of novel. They were initially a novelty of like, oh, we put cameras in phones too. That's like a table stakes thing to the point where now people carry phones that are capable of taking pictures that rival the output from large SLRs that people used to spend thousands of dollars on in the past.

I think sensing technology, like fingerprint sensing and facial scanning, that's all been stuff that's been pushed heavily in the mobile space. I think we're going to see more of that in the AR/VR space around sensing technology to understand what you're doing and what emotions you are trying to convey and relaying those emotions to somebody on the other side.

Meta has kind of shown in some pretty cool videos that you can find online around mapping people's expressions and emotions onto an avatar. That stuff, I think is going to be pretty awesome and intense when it comes out.

DB: Yeah, it will be nice when you've been meeting with someone remotely for years and you finally meet them in person. Today that feels like a moment when you're like, oh, it's finally nice to finally meet you in person, but I can definitely see a future where it's like, oh, yeah, we've been in the room the whole time. That would be really nice.

Mehta: That's the dream. Yeah.

 

AR/VR Hardware Design Challenges

DB: Is there anything for engineers who are coming from mobile design or maybe a more conventional design space, when they come to this AR/VR head-mounted device space, is there anything that especially surprises them or they have to kind of change their thinking about?

Mehta: Yeah, I think whenever you work on a different category, there's going to be surprises. I think that's a natural thing. When I'm looking at strong fundamentals for a candidate, I tend not to over-index too much on whether they've had experience working on AR/VR products in the past. It's a pretty new category, so the set of people who have had experience is fairly small.

I think that the ability to solve problems that you haven't faced before, be able to regale challenges that were curveballs, fire alarm situations, whether it was on a mobile product or something else, and learn and adapt, I think that is sort of a key competency that I’d lock for. I think the first thing is, I wouldn't stress out if you haven't worked in the product category that you want to work in. That's a very natural, normal thing.

I think in terms of stuff that might surprise somebody who hasn't worked on an AR/VR headset, that weight and comfort component I mentioned is a big one. A lot of these system constraints that exist on every consumer electronic product are all the more important. On a laptop, it's important the thing is light and thin, but fundamentally it's not going to be an unusable product if it's just slightly heavier. It might not be as good of a product, but it's not going to be unusable.

Something that you put on your head, though. Different story, right? Like there are hard constraints around how big that thing can be—how much of a moment arm it can have out from your neck. That quickly takes something away from being an, oh, this is a really cool experience to I never want to put this thing on again experience. So I think that's huge.

I think coexistence challenges are a big deal in modern mobile phones. I would say definitely as much of a big deal in AR/VR systems, if not more because you're going to be constrained in ways that are really insane on a glasses form factor. You're going to have to take shortcuts and break rules around high-speed rails and how well they're referenced. You're going to have to break rules around shielding components, all sorts of things. Things may get a little bit more empirical because you can't necessarily just design on the best design principles and still fit within the product surface.

DB: I imagine simulation becomes pretty important in that case.

Mehta: Yep, simulation is really important. I think my personal take on this is that simulation and empirical data work hand-in-hand together. EMI sims and running your design through basically the best practices around routing high-speed stuff. That's all table stakes.

But at a certain point, you're probably going to get into a situation where you're kind of breaking the rules. You're not really fitting into the eye diagram, but the thing still needs to work. And that's where I think getting data and getting enough data to make you feel confident that you can ship the thing are important.

DB: There's also, on the empirical side, there's a does it work or not test. Very kind of like old-school HDMI. It didn't have clear guidance on frame drops and that sort of thing. It was like, does it look okay? And that was the test for whether it worked or not. It's got to be some of that.

Mehta: A lot of these things are kind of being invented, right? Like bit error rate standards for wireless data transmission is another analogy similar to what you're kind of alluding to. I want no desense is like a great statement to make, but in reality, that's probably not going to be a thing. What error rate are you okay with before you can no longer ship this product?

And for the companies that were the first folks in the space of making these devices, the standards they kind of came up with ended up becoming or influencing the products or the industry. And I think we're going to see a lot of that in AR/VR as well.

 

A Vision for AR/VR Applications

DB: You mentioned earlier helping people with vision impairments and that sort of thing. What gets you excited about this technology in the space for people who really stand the most to benefit from it?

Mehta: I think a couple of things. Some of this stuff doesn't seem like a huge deal but ends up being a game changer.

First is that these products are wearable. There are already some products on the market that have applied OCR and scene description to help folks who can't see. There's an app that Microsoft developed while I was there. I helped on it a little bit called Seeing AI. And it's basically an iOS app that you can put currency in front of it. It'll tell you what the money is.

You can put it in front of a scene. It'll describe that. Hey, “there's a man wearing headphones with some electronics equipment in the background” is probably what would say if I held it up to you right now.

It can read text and intelligently parse things like restaurant menus and paragraphs in a book. So it's really cool. But the one problem is you have to have your hand holding your phone. And so that's a problem for a couple of reasons.

The first reason is if you can't see your hands are probably a pretty big component in how you're understanding the space around you, whether it's using a cane, using a guide dog if you're on the go, or just feeling stuff that's around you. So if you occupy one of those hands with the phone, you've already kind of taken some of that convenience or way you operate away.

The second is you have to aim the phone at the thing you want to aim it at. You can do a little bit of, like, hot-cold, Marco Polo kind of stuff in your design to make that easier. But even as a sighted person, it's not always apparent what I'm aiming my phone at unless I'm looking at the viewfinder. So having the thing wearable, I think that's a huge deal because there's a lot of human context cues around where you're pointing your face, that map to the thing you're paying attention to, and you can use the thing without your hands touching it, which I think offers a lot.

The other aspect that I think is really compelling is there's a large spectrum of ability around sensory issues folks have. I think it's often easy to assume that, okay, somebody can't hear, they're deaf, or somebody who can't see, they are blind. And there's a large spectrum of people's abilities. Somebody may have low vision, and they might have issues like walking down steps because they can't see the contrast from one step to the next, but they can otherwise get around day-to-day very well.

Being able to highlight those steps with higher contrast could be a really great way to improve their life without necessarily giving them some device that's like a one-off assistive thing.

Same for hearing. I can imagine a world where being able to focus on the conversation that you're having in a loud environment could be really useful to somebody who otherwise may use a hearing aid or something like that, but they also want to be able to use the technology that is sold to the mass market.

I also think there's something really compelling about making products that are used by everybody, usable by everybody. What I mean by that is there are things on the market that exist to help folks with disabilities but because the naive business-centric view is that the market is small, the result is that the cost of those devices is prohibitively high.

There are some products on the market that, for example, are glasses that have a camera on them and do some stuff, but they are thousands of dollars. They look like a thing that an engineer like you or I designed. They don't look like an industrially designed product that is designed for the mass market. And so they call attention to the user. All things that might not be what you want.

One of the things that I found was really magical about how Apple approached the design of the iPhone was that they integrated voiceover and all these other accessibility features into a device out of the box. If I buy an iPhone, I turn the thing on. If I don't interact with it long enough, after I take it out of the box, it'll start talking to me and saying, hey, you might not be able to see the screen. This is how voiceover works. That's amazing1

Before voiceover, I would say the industry standard for tech to help folks who couldn't see a screen was third-party applications that were thousands of dollars and had a very steep learning curve. So I really love the direction the industry has gone. I'm confident and optimistic that AR/VR is going to stay on that track.

DB: Yeah, that’s really exciting. I try to tell other people, I think, how much AR could help people and just change my life and their lives. How much do you think the public is ready to accept that? I know we've had things like the Google Glass and stuff like that, but that was more of a novelty than a real thing they would consider.

Mehta: Not to plug my own company too hard, although I will. I think the approach we took with Ray-Ban Stories was a really smart one. Right. We went with a design-forward approach where people in the mass market like buying Ray-Ban Wayfarers. It's a cool, iconic design. It's not a tech thing on your face. It's a Ray-Ban product, and yet it has some really amazing, cool features. I think that's a really good approach to take with wearable stuff in general.

I think Apple has managed to somewhat do that with the watch in the sense that, yes, it's tech, but it's also a desirable thing people want to have. I think that's a wise direction to go with AR devices.

DB: So instead of designing from a desktop computer server towards an end consumer device, is it more looking at the consumer device and figuring out how to pull that closer to the tech side?

Mehta: Yep. I mean, I think you need both to be clear, right? You can't get one without the other. And that's why I'm excited about some of the advancements that are existing in the bulkier, larger form factor things on the market.

But I do think having that eye on what consumers actually want and the understanding that when you put a piece of technology on yourself, that's a little bit more of an intimate thing than buying a laptop or using a tablet. And so we have to be cognizant that we're building things that people actually want to wear. I think that's a wise approach to solving kind of the problem you're alluding to.

 

Insights on Team Building

DB: As you're looking forward at what might be coming and products that you're working on that haven't been released yet, what to you, makes just a killer R&D team to build a killer modern product.

Mehta: I think when you're building a team, working on a new thing, I think it's really critical to hire people that don't get hung up on the way things are supposed to be or too hung up on the right process. That doesn't mean throw process out the window. I think it's really critical to do things the right way and not cut corners on good design practices. But I do think it means it's important to find people that aren't going to get so hung up on, oh, this has never been done before, or I feel awkward and confused because I've never built something like this before.

Rather, you want to bring people onto the team who are willing to say, “Hey, let's just go for it and see what happens.” I think getting more of that style really helps move fast, not get the team wound up on getting frustrated by churn and those sorts of things.

I think it's also really important to bring people on that are they have shown a track record of success in building devices in a very constrained form factor and dealing with constraints that maybe bend or break the best practices and rules on electronics design.

For example, there are a lot of rules of thumb around designing flex circuits and rigid-flexes around what type of components can be placed on those devices. If you were able to bring somebody on the team who has experience outside of just pure electrical design, but on Design for Manufacture, and basically on working with vendors on bending the rules and inventing new processes, you can then be the one pushing theHeading 4 envelope instead of adhering to the standards the industry has set. So I think those are big ones?

DB: Are there any core sets of disciplines or any key folks like a UX designer or something like that, that might surprise people who, if they were starting to build a team, wouldn't necessarily include that role as part of their list of people.

Mehta: It's amazing you mentioned that because I just hired somebody onto my team whose background is in design and not on product design, but art. They are able to get things done within Meta in a way I've never seen before among most of the rest of the folks inside the wider organization, simply because they understand how the company works. They understand how to get things done, and they have deep relationships with the vendors that we work with. So they're able to kind of move more quickly than other people I've seen.

So I think the takeaway there is maybe not getting so hung up on somebody's background or somebody's title and being a little bit more focused on what they've accomplished and what they're able to do. If I think about the star players on my team, they are people that are very hard to put into a bucket.

There's the person I just mentioned. I have another really senior, super high-level IC on my team that had a background in chemical engineering and worked at a defense contractor for a while. And they're the sort of person that can just throw at a project and they can go deep on optics, they can go deep on EE, they can write firmware.

Finding these people that basically are hackers and can just get stuff done and don't put themselves in a box too much, I think that is the sort of person that can really make or break a team. That being said, if you're building a complex design, yes, you need a robust electrical team and a mechanical team and people who can just do things the right way. I think that's super important, so you don't make stuff that's too flaky. But I think if you're asking about what are the game-changing people that I've hired in my organization, they tend to be people that are really hard to put into a box and they're really hard to categorize.

DB: Yeah, I love that. That's exciting for someone who's had a hard time finding a box for themselves. Happy to hear that.

 

The Future of AR/VR

DB: As you look at the existing market of what's announced in public and launched today, where do you see gaps and where do you think the world of AR and VR could go future looking? Five years, ten years? Maybe even 50 or 100 years.

Mehta: I think that humans have a natural reaction to both underestimate and overestimate change. I know that sounds like a very vague statement, but I'll dig into it a little bit. There are a lot of articles in the mainstream press now about how the metaverse hype is overblown, this and that.

It reminds me very much of reading articles in the 90s about how there's a very famous news article…I don't know if you've read it…about how Internet shopping will never be a thing. I think the authors even commented more recently on it. They look fondly on that article because of how wrong it was.

I think that we're very quickly going to see AR/VR become more of a mainstream thing and become applied in ways that help everyday people and aren't associated with a niche application like gaming is sometimes, I think, unfairly described.

Realtors already have these 3D views, for example, for open houses, and you click through it awkwardly on your computer screen, like putting it through an HMD. That's like a no-brainer that I know folks are already working on.

One of the big applications for HoloLens when we were there was working with the aerospace industry on helping a technician get trained up more quickly. You're working on a really complex part inside of an engine. You look at it and you can immediately see info about what the part is supposed to do or how to service it.

I think those are nuggets we're already starting to see. I think that fully immersive entertainment is another big one that people underestimate. The first time I put on a VR headset was a very “oh, wow!”
moment for me. I think it's hard to describe that for people.

I remember when I was very small, the first time I ever used a mouse. That's sort of the closest thing I can kind of make the analogy to. I think we're in this very exciting time frame where if anyone tells you how exactly things are going to look in the next five years, they're probably making some stuff up. I don't know what things are going to look like, but I do have this inkling that we're on the cusp of something great. And that's why I'm so excited to be working in this field.

DB: There's something special to being as wrong as possible, like in that Newsweek article, and it'll be interesting to see how it develops that's for sure. Thank you so much for this conversation. This was fantastic.

Mehta: Yeah, it was fantastic on both ends. Thanks for the time, and I really appreciate the opportunity to chat with you today.

DB: What gets me excited about the world of VR and AR is that there's just so much room for growth, and we're so early in this tech, I think a lot like cell phones we'll look back at this time and wonder how we got stuff done without it. But that's all for today's podcast. If you like it, please share this episode with someone else you think will enjoy it, and also share it on social media. I'm Daniel Bogdanoff. Thanks for joining me today. I’ll see you right back here in two weeks when where I'll be joined by Henrik Mannesson of Texas Instruments.