We continue our conversation with Mike Haley, Autodesk’s head of R&D, about how Autodesk plans to get AI into design. Part 1 of our interview is here.
RT: Tell me more about where multimodal learning falls into the AI scheme.
MH: Multimodal is the state of the art. I think it’s still emerging. You see what Google’s DeepMind has done over the years. That’s agent-based learning, which is reinforcement learning. You’ve seen what AlphaFold has done with protein folding. Google launched AlphaMissense in November and it is discovering new chemistry and new materials. That is similar to generative design. It’s exploratory with the use of AI. It’s moving past what people have done. It’s able to reason and find strategies for the discovery of new ideas. It’s emerging. The next 5 to 10 years are going to be very interesting,
Yes, very interesting. But how is AI going to impact the world of design and engineering? Can AI learn what I’m starting to draw and autocomplete? Can it infer from what I’m doing, know the context of what I want, where I am, what I’ve made before? Can AI go into a point cloud and discern a flange, a pipe? I’m going to call that design assistance. That’s not ‘Hey ChatGPT, make me a car.’” I don’t think AI is ever going to make me a car or a house.” Nor do I want it to. But I want some design assistance. I want AI to finish a part, electrify a product….
Take electric cars. I want to electrify a car that was previously burning gas. Auto companies have electrified lots of cars by now. There’s a lot of data and knowledge to draw from. Why can’t AI use that wealth of knowledge, which may already be in-house, use it to do the next electrification? Tell me if any work is being done towards that.
If you go to our research website, you’ll see how much work has been done. Our AI Lab has published about 60 papers in the last 5 years on things related to AI in design, like how you would use AI with n geometry and that sort of thing. There’s an enormous amount of patterns in everything we do—human beings are constantly surprised by this—but so much of what we do is repetitious. The way you write articles, the way we write code, the way we do designs … there are patterns and repetition that are occurring. This leads to a couple of things. The first is: when do you interact with the software? If you want to specify a house, a car, a simple component … you go into Fusion 360 or whatever CAD you’re using. It’s a calculator. It’s as dumb as it was the day you first bought it. You’re going to have to show it everything. You’re going to have to create every feature. You have to put in every dimension. What’s more, you’re going to have to do that with a keyboard and a mouse—input devices that have only recently appeared in our existence. Do we interact with anything else that way? If we had to talk to each other that way, we would be annoyed and we would probably just stop interacting if we had to pull out a mouse and a keyboard. But we can talk images, we can use language, we can do all sorts of things to express ourselves. That’s the natural way. So, the first thing that’s going to happen—and pretty quickly—is that AI is going to transform the user interface. It’s going to transform the human/computer interaction between designers and software.
Funny, we started on this years ago when I was still relatively new at Autodesk. I was super impressed by what we had done. Back then, I was speaking to an architect who told me how much he hated using our software. “I suck at it,” he said. “I’m a creative person. I have these ideas in my head. I love sketching. I love sitting with my pad and sketching stuff. That makes me happy.”
I’ve heard this conversation, too. Let’s say I’m an architect. I want to have a grand design for a space. I don’t want to get mired in drawing a door or a window or a doorknob. I get stuck on that stuff and I lose my vision.
This is true in animation, too. It’s true with all areas of design. What’s going to happen now is AI can say “explain your idea, sketch something. Show me an example of what you’re talking about. Point to something you’ve done before.” All very natural things. This is going to unlock doors to our software, make it more accessible. If you are a creative person and your software has been a speed bump, it is about to become super creative.
Would you agree, then, that all CAD programs have been very difficult to learn? Architects did not go to school to learn software. Engineers—same. They don’t want to get bogged down in the software. They want to solve problems.
Yes. When we launch a new feature in our software and we track when people will start using it (in aggregate), it sometimes takes years—sometimes 5 years—for people to start using the new feature. It’s because they’re already bogged down in so many other features. They can’t get to the new features. I think we’re going to see fundamental changes in software in the engineering and design space.
Are we talking about a natural language interface to design and a higher order of creation? Instead of drawing a line, will we have the software to select/make a door, for example?
It will be a combination. Natural language is part of it. Sketching is part of it. Referencing material is part of it. It will depend on where you are in the lifecycle of design. That will determine what is most appropriate. Natural language interface is going to happen. For example, you open Revit and you’re about to start designing a building and the first thing you do is write a paragraph about what you’re thinking of designing. Say, a six-story skyscraper on a waterfront with a mezzanine on the bottom … some more language of that sort. That will be the input. Then a week later, when you’re designing the mezzanine floor in more detail. It’s helping you predict something else. It’s still using the original prediction, but it keeps adding to it. This notion of different modalities, inputs is used throughout the lifecycle of design. It’s the idea of capturing design intent. That has been the golden chalice everybody’s been after. There’s no software today that really truly captures design intent. We’re seeing the first glimmers of that now. Just being able to input a phrase about what the building is for—I’ve expressed some intent.
The next thing, which you already alluded to, is the actual design. The design breaks into two categories. First, there is repetitious/tedious work. That’s 70 to 80 percent of all design work. That is a lot of unfun stuff to do. That’s producing the final drawings, getting all the dimensions right, making sure your systems are fully constrained, finding the door.… It’s not creative. It’s just stuff that you have to do. Those workflows have definitive patterns. AI is able to learn these patterns, even the early AIs. They can focus on the patterns. Like “Hang on. I know what kind of doors these buildings need. I’m just going to select them for you. You can go and correct it if you think I’m wrong, but 99.9 percent probability that is what you need.”
So, if I’m making a wall, it understands where I’m making it, understands the studs have to be so far apart, right? As an architect, I don’t want to be bothered with the framing.
Exactly. How many times have you told it to make studs 16 inches apart? We’ll just do it. That’s the first thing we are taking away from you and what we refer to as automation. We believe that there’s going to be a bunch of workflows that are just going to be automated away. They’re the kind of work nobody really wants to be doing. They’re time-consuming. They’re also places where people make a lot of errors. Take sketch constraints. People often get confused when setting up sketch constraints. They get them wrong, then something else goes wrong later. It’s not creative work. It is just tedious and difficult. We’ll take that work away. That gives you more time to actually spend on creative work. Cuisines. Which brings us to the second category of work.
That brings us to generative design. In the early days, generative design gave designers other ways to look at the world. It gave other solutions, solutions no designer in the world could have thought of. Generative design has never been “we’re going to give you the perfect answer to your problem.” It was to give you awareness of some other ways to solve that problem that you might never have thought of because we were able to search the entire solution space for you. That’s what we’re going to do now with the second category. We call it augmentation. Before, we had automation. Now we’re going to augment the designer. The important thing is we’re not replacing any designers. I don’t want to live in a building that was entirely 100 percent designed by an AI. It’s going to lack that human sensibility, the creativity, and the enjoyment that we have in creating it. If you’re an architect or an engineer and you’re designing, you’d have this tool with you that will be able to help you reason, to show you things that may have been difficult to see before.
Generative design wasn’t the easiest technology in the world to use—let’s face it. After we launched it, the adoption rate was reasonable, but not 100 percent. It didn’t transform the world. One reason was that it was difficult to use. When you use generative design, you have a human-computer interface problem.
You were doing a finite element analysis.
Yes. Simulation engineers were the best at using generative design because they were used to doing load cases or restraints…. Now imagine if we take generative AI and add the ease of interaction, the awareness of the context, awareness of the intent and we couple them all together. Now things really begin to change. Instead of specifying every little point, every little interface location, we let AI figure out what you’re designing. It will know the range of load cases. It will stay plugged in and run scenarios for you. Then you are really designing because you have an assistant that frees you from having to spend days specifying everything. You have a system that is predictive, that is watching you work, that understands the context and doesn’t need to have everything set up for it. This is the sort of thing that will inspire you and take you in new directions and help you get there.
That is where the value of AI lies.