217. When NOT to use AI in business and warfare

If we know that ChatGPT makes things up, when should we avoid Large Language Models? 

Is generative AI really safe to use when it matters?

Listen to this interview with Dr Heidy Khlaaf to find out.

Dr Khlaaf is the Principal Research Scientist at the AI Now Institute focusing on the assessment and safety of AI within autonomous weapons systems.

She previously worked at OpenAI and Microsoft, amongst others. 

Timestamps

00:00 Introduction

06:32 The Problem-First Approach to AI

14:20 Limitations of Large Language Models

20:49 Augmenting Human Knowledge with AI

 23:37 AI Systems Gone Wrong

28:22 AI in Safety Critical Systems

33:47 Questioning Technological Determinism

38:19 AI in Defense

For more career & tech lessons, subscribe to Tech for Non-Techies on:

• Apple
• Spotify
• YouTube
• Amazon Podcasts
• Stitcher
• Pandora

Growth Through Innovation

If your organisation wants to drive revenue through innovation, book a call with us here.

Our workshops and innovation strategies have helped Constellation Brands, the Royal Bank of Canada and Oxford University.

Transcript

Sophia Matveeva (00:01.395)
So Dr. Heidi Khlaaf, when people say to you that AI is changing the world, what is your answer? Actually, you know, I'll start again. Am I saying Khlaaf correctly? Okay. Dr. Heidi Khlaaf, when you hear that people are saying the word, Dr. Heidi Khlaaf, when you hear AI, shit, you laugh. That was your fault. Dr. Heidi Khlaaf.

When you hear people say that AI is changing the world, what's your response?

Heidy Khlaaf (00:34.143)
Is it changing the world for the better? That's really the main question that comes to

Sophia Matveeva (00:39.6)
And so what do you think?

Heidy Khlaaf (00:41.686)
Not necessarily. I think there was a period where AI really was heading towards that direction. We're seeing, you know, protein folding advances that came out of Google DeepMind. To me, that seemed like, wow, what a really promising use of AI. And then large language models came out, which are sort of these non -specific AI models that are meant to be general purpose, whatever that may be.

And I think that definitely made it seem like, okay, let's put AI in everything without understanding its purpose, without understanding its use cases and its risks. And it became in some sense, a hype term without many people even knowing what AI is, its history, its other uses and whatnot. So I think there's definitely positive use cases for AI, but I don't believe the way that large language models are being sort of deployed and put out is really that.

Sophia Matveeva (01:34.931)
So what do you think of large language models? Do you think that essentially it's a tool that we're still trying to find a use case for and that basically the Silicon Valley companies, have such expensive PR department that, you know, chief executives are being pushed to use them even if they don't really understand what they do and haven't really found a use case for them.

Heidy Khlaaf (02:00.0)
Absolutely. mean, I'll give an example of where large language models are useful, right? To not make it seem like the whole thing is a disaster. You know, there's apps like Grammarly, right? There's people who are non -English speakers or people who write quite informally that might want to write, you know, a bit more formally. These are kind of uses for large language models that are absolutely valid and I can see how it helps some people and it definitely should be something put out in the world. But most other use cases...

are not applicable. A lot of people think that large language models are fax machines or database machines, but they're not. They're actually just predictive models. They just look at one word and predict the next token based on the context of the sentence or the question or the text that it was given to it. So it's really not factual. And a lot of people are trying to use it in that way. Hey, let me use this large language models to generate historical facts or to write a letter to a lawyer. That's going to be filled.

with tons of errors that is going to cause more harm than good. And I do definitely think that there is this hype machine and there a lot of marketing that is trying to figure out what is this AI for and can we put it in everything? And unfortunately, most people who are using large language models have been sold this message. It should be in everything and everywhere when really there's only some specific use cases that it's good for. And...

As a result of that, people are putting it in our information system and it's causing a lot of disinformation. It's causing a lot of people to confuse facts and fiction and to really not understand that this is not a search engine. This is not a database. This is not a Wikipedia, but that is kind of the use case that we're seeing for it right now, unfortunately.

Sophia Matveeva (03:43.751)
So are you saying that people should really educate themselves about what it is and what it is not to basically manage our expectations because you know I often bring up the case of that lawyer, know, very well -paid lawyer who basically just used chat GPT to prepare for court and you know arguably he should have known that this is this is not what you should be doing this is not going to be enough.

Heidy Khlaaf (04:13.122)
It's sort of like a, I think there's multiple people who need to sort of change their perception towards AI. One, I don't think that AI labs or companies should be putting out these models and making these false claims without any repercussions because we are seeing that. We're seeing advertisements, we're seeing newspaper article without questioning of the claims that they are making about these systems. And again, I have not seen

except until recently, I think with the FTC, people saying you shouldn't be allowed to make these claims without being able to back them up. So that's one side of it. The other side of it is that whatever technology you're going to adapt, that sort of it's on you to prove that it is a right sort of use case for your product or your business model or for your personal work. And most people just watch these like advert like relentless bombardment of advertisements.

And they say, well, that must be true without really understanding that AI has a lot of downsides that need to be understood before you can integrate it well. so really where I don't want to put the entire blame on the user here because it's, you're seeing it every single day breakthroughs, AGI is coming, these, you know, super intellectual machines are here and people take that as fact. But at the same time,

If you work in something that requires facts as a basis of the foundation of your work, like being a lawyer or a doctor, or even a computer scientist, you have to understand that these are not factual machines and start sort of your work from there. You can then work backwards. Okay, well, what can I use it for? But most people have just accepted it as this is a machine that's smarter than me and it is factual because it is a computer.

And that to me is very much sci -fi rather than based on any sort of scientific foundation of what they actually do.

Sophia Matveeva (06:05.693)
before we get to the downside, I really want to, well, first of all, express some sympathy for these chief executives who I think are under so much pressure from equity analysts who are often, you know, equally uneducated about AI because there is all of this hype and, when they get asked, what's your AI strategy, right now, no chief executive can say we don't have an answer.

That's, we don't have a strategy. That is not...

possible in the world that we live in today. also, know, recruiters are asking this. Everybody is kind of expected to have a view and everybody is expected to understand how to use it. And so it's very difficult for those people to stand up in the face of hype. But I do think that we forget that actually there are very well capitalized players who are funding this and they have very well capitalized PR

but equally, I don't want to say that AI is useless because I use it in my company, but I use it for very specific use cases. So for example,

Heidy Khlaaf (07:15.864)
Exactly.

Sophia Matveeva (07:18.277)
We now have a tool that helps us make clips. So there'll be some clips of this interview that will go on social media and AI does that. And generally the clips are pretty good. I mean, we do my assistant needs to do some editing, but generally they're pretty good. And that saves us a lot of time because previously he had to do it manually.

So that's definitely a use case, but it's a very, very narrow use case. It's not a specific, it's not kind of a general LLM. And I find that when you do give tasks to a general LLM, you have then to do quite a lot of work to basically make it usable. And that still stays signed. So often I open up chat GPT when I'm sitting there in front of a blank piece of paper and I need to write something. I don't know, I need to write a podcast episode and I just need to get

started with my thinking, but it's definitely not something that I would ever pass off as my own because well then nobody would listen. And so what would you say to those chief executives who are under pressure to say, you know, when they get asked that question, what's your AI strategy? What are they supposed to say?

Heidy Khlaaf (08:32.376)
think really the best thing for them to do is to have a strategy where they understand the risks and use cases for AI. That's really a good starting point. I agree. You can't just say, well, we don't have a strategy. And actually that's not really acceptable. Not only because you're dismissing the people who are interested in and invest in your company, but also because you should be understanding the up and coming technology and always keeping an eye out if this is going to be useful for your business or not. So your AI strategy should be we're having a look.

at AI, we're having a look at the use cases and risks and then applying it across the business and seeing if there's an applicable use case for it while understanding any sort of failures that could occur and what that could mean for our money, our employees, and so on. I think that is a good strategy to have about any up and coming technology. And it shows that you're being sort of, you're educating yourself, right? You're looking where it can be applicable, but

Typically what we've seen is across all businesses, the AI sort of strategy is put it in everything, right? And I think people believe that's the same thing as understanding the risks, understanding the use cases, and it's not. And so if you're sort of reluctant about...

Sophia Matveeva (09:37.532)
Mm

Heidy Khlaaf (09:47.15)
putting it in your business, think that is a good place to start and to sort of show the investors or to show the C -suite that you are thinking about it and taking it seriously because you're right, there could be a use case where you're finding that it automates something that takes a lot of manual labor that doesn't really require factual accuracy, for example. The video is a really good example of that. I'll give a good example of where we use it or where we think it could be useful to have AI and safety critical systems. know, safety critical systems require

a lot of reliability and accuracy.

Sophia Matveeva (10:18.36)
Could you first tell us what's a safety critical system?

Heidy Khlaaf (10:21.366)
Sure, a safety critical system is basically any system where its failure can lead to deaths or environmental harm and destruction and even monetary loss. So you're looking at airplanes, nuclear power plants, cars, really any of the main infrastructure that you're probably taking for granted, you know, your electricity and your, your, the power grid is a safety critical system and they're developed in quite a different way from typical software systems because their failure could mean death, unfortunately, you know, like a power outage.

can take out hospitals and that would be like an incredibly, you know, unsavory situation and nuclear power plant failing equally could lead to, you know, environmental disasters as well. And so you would think, okay, we don't need AI in this at all, right? But there is a really interesting use case that I always refer to when people are like, so are you anti -AI? And I'm like, no, not at all. That my job for the past 10 years has been to integrate, find...

and understand how AI can be integrated into these systems. So there's a group in Manchester in the UK that work on a project called RAIN, and they've been looking at robotics, right? They're not looking at large language models, right? They're looking at a very specific system with a specific use case that can essentially help nuclear operators. So for example, if you have a leak in a water tank, typically you would have a human diver go investigated and they could be sort of exposed to incredibly high levels of radiation.

That scary, right? That is a scary situation. But what if instead you could have a robot dive in there instead? Or have some sort of mechanical automated detection?

of a nuclear leak, right? And in that case, it's actually enhancing the safety of the system, right? It's not replacing a human operator. It is not replacing an entire system that has, you know, has been accurately studied and built for the past 50 years. It is instead enhancing what we already do. And I think that's really the best way to view AI. Can you augment human knowledge and human expertise instead of trying to replace it? And I think that is a mistake a lot of people are making about AI. They're like,

Heidy Khlaaf (12:30.432)
like how do we replace humans? You will never be able to replace humans. Computers and humans have very different pros and cons. Like a lot of people are like, but it's super intelligent. And it's like, well, calculators have existed for a long time.

And we can't match like calculators. I'm sorry, can be a, you know, a genius in mathematics, but you will never be able to compute as fast as a calculator. That's why we love computers. That's why we use them to automate things. So if you could bring that perspective, right? Like, how do I use a calculator to augment my tasks? Right? How do I use an AI to augment and audit help automate the things that really don't matter, but you need to think.

I'm not going to be replacing human expertise. You should be looking at AI and how it helps people. If it slows them down, like we're seeing a lot of studies doing that, like you were saying, hey, if I'm using a text thing, I actually have to undo a lot of the work because it was wrong about this thing and it was wrong about that thing. And it turns out.

actually, if I had just written it myself to begin with, it would have been easier, right? That is a case where large language models are sowing down human productivity rather than enhancing them. So you should be thinking about how do I use these AI models or large language models to enhance what I do rather than let's replace operators and humans altogether.

Sophia Matveeva (13:50.259)
It's interesting because it seems that this whole AI revolution is very much technology first as opposed to problem first. And, you know, it is a technological achievement and it has been invented by some very clever computer scientists. So, of course, they want to show off their invention because, I mean, that's normal when you've been working really hard on something and then it works. You want to tell everybody. But that doesn't necessarily mean that we should all then be bought into it, whereas when I

talk to entrepreneurs, always say, start with the problem first, investigate the problems, and then create a solution. So are you saying that in a lot of cases, AI is a solution looking for a problem as opposed to the other way around?

Heidy Khlaaf (14:36.63)
Absolutely, I would absolutely agree with that same and I would also say that novelty and technology doesn't translate to progress. We're seeing this in a lot of aspects of our life today.

like touchscreens, right? Touchscreens, sure, shiny, new, novel, but for them to replace the knobs that we use in cars is actually really problematic and has led to a lot of car accidents. And so I think this lesson can also be applied to a lot of other things. And again, this is not a comment specific to AI, right? We shouldn't look at any...

sort of new technologies coming out, whether it's touch screens or crypto or AI and think that is clearly going to be the solution to all my problems. We should be looking at the problem we're trying to solve. And if there is a technological solution to it that can help, that's fantastic. But that doesn't mean that every problem requires a technological solution in general, whether it's AI or not.

Sophia Matveeva (15:39.814)
But when we last spoke, we talked about technological determinism. And could you tell the audience what that is and why it matters or why it's annoying?

Heidy Khlaaf (15:50.326)
Yeah, there's a lot of people in Silicon Valley who subscribe to this idea called technological determinism, which really translates to if there's any problem, the solution has to be technological. And that's simply not true. You know, unfortunately we live with the human condition and we are not.

computers, we are not technological beings and we have a lot of societal and political solutions that we need to look to before looking to technological solutions. The way that I see technology is can it help us automate or help us in some sense speed up some of our very manual processes. But there will never be a technological solution to the human problem. You will never be able to find a computer that helps you understand democratic elections, right? Or help you deal with grief. But that is what we're seeing.

We're seeing large language models being deployed as therapists, despite the fact that we know that these things are not therapists. They are not qualified therapists. cannot help you. Maybe they can help some people, but they are not really a solution to just dealing with everyday problems of the human life. And we're now seeing this philosophy, which is...

Sophia Matveeva (16:38.418)
Mm

Heidy Khlaaf (17:00.148)
actually progress or solution equals technology and that has never been true of any part of like human history. So I say that this is a view that a lot of people in Silicon Valley take and when you really think about it, why do they take that view? Well, it feeds into their bottom line. If you can sell a solution or a technology to every problem that a politician has or a human has, you will make a lot more money. And so I think when people subscribe to these philosophies or read about them or read about what a CEO of a big tech company

many things, they should really be thinking in the back of their mind at the end of the day, they want their bottom line to be successful, right? They want, you know,

as many people as possible to use this technology. So of course they're going to market themselves that way. And you should always have a critical lens, right? These people are not your friends. These people run a business and they want their business to be successful. Just like a lot of the people who watch this podcast, right? So it is definitely a philosophy that's taken by many. And I think it doesn't mean that we have to accept it. I think most people would find that, you know, throughout human history, our,

Problems are really complex. They require social technical solutions and it's not just going to be AI that's going to suddenly solve that.

Sophia Matveeva (18:16.412)
A socio -technical solution is that a combination of humans and technology.

Heidy Khlaaf (18:24.206)
Absolutely. mean, I love computers at the end of the day, that's why I decided to be a computer scientist. And I think that computers have an ability to help us in ways that we haven't really thought about before. At the end of the day, they have to augment.

That's the thing they have to help us rather than replace us. And I think that also what we're seeing with this technological determinism is that there being a lot of CEOs are being told you can now replace your workers, you can replace your artists, right? You can replace your programmers. But what we're now seeing is that AI being deployed, making a lot of mistakes, right? Because it is not a human, it is not a human expert. And then you're hiring people to clean up the mess. So really.

did you solve the problem? Instead, you now have a process where instead of building something correct to begin with or writing something constructive to begin with, you're now having sort of some gibberish being output and someone cleaning up that process, which I would argue is a lot more wasteful, not as productive, not really effective. So what we should be looking at instead is how do we solve this problem? Here are our goals to it. Here's the way that we can approach it. Can AI.

fit into any of these boxes? Can it help us automate the specific tasks rather than the solution end to end? That's sort of a good way to really think about it rather than let's replace humans altogether.

Sophia Matveeva (19:53.533)
Before I ask you about what AI has messed up and how humans have had to clean it up, because I think everybody wants to know that, I want to share a terrible story about AI grads. And I was actually a witness to this, which really shows technological determinism. So I was in conversation with a couple of AI graduates from Stanford. So very intelligent.

you know, two young men, were slightly younger than me and we're in the US. And one of them was saying that, you know, he really would like to have a girlfriend or meet some girls. And he was struggling with it, which, you know, kind of like, well, you know, you are AI grad, that's kind of normal. They also dressed appropriately and had terrible haircuts. And I was like, well, you know, there are some very simple things that we could do here. But then...

Heidy Khlaaf (20:39.576)
you

Heidy Khlaaf (20:45.112)
you

Sophia Matveeva (20:49.949)
His former classmates said, well, actually, you know, I've got an idea. I've been thinking about this too. And there is a way that we can actually game the algorithm on Bumble. And then you can basically get a much better selection and then all your romantic problems are going to be solved.

And if that's not an example of technological determinism, I don't know what is, because, you know, there was one young man saying, I am lonely, I would like a partner, or maybe I would like some fun, I would like some human company. And the solution was, let's hack the algorithm.

Heidy Khlaaf (21:27.15)
Absolutely. And I think the end result really is not going to be constructive because at the end of the day, they will have to meet another human being on the end of the system. you know, whatever, whatever, even if you were able to hack it, that person has to meet you and you have to have a connection. And I think what we're seeing a lot of the times is like, instead of people investing in community and investing in themselves, they're thinking, how do I sort of

Sophia Matveeva (21:32.165)
No.

Sophia Matveeva (21:44.794)
Imagine that.

Heidy Khlaaf (21:54.082)
beat the algorithm, right? How do I get it to benefit me? And I think that gamification actually comes from a lot of our use in social media. So if you think about it, I see a lot of the times when people ask me, what are the big problems in AI? I say, I think a lot of them kind of started in social media, you know, where we have Instagram kind of favoring specific videos or photos over others. And then people want to gamify the system because they want, you know, more eyes on their posts or a lot of people make money out of that kind of system. So there you want more engagement.

engagements to you know have higher profits in their business and It ends up people thinking about the world this way rather than how do I have a connection with another human being? how do I have community with others and Technology really can't I mean, yes, it would be great if there was a technological solution to the human condition But unfortunately there isn't right at the end of the day you can game the system But you will have to meet this person face to face right that AI is not gonna help you there

Sophia Matveeva (22:53.777)
I just think, and I've also seen in the marketing realm that there are actually now more people talking about brand and building a brand and that building a brand is more important in the age of AI than it was before because a brand is essentially what you stand for, whether it's a personal brand or a company brand, it's well, who are you? Why are you different from your competitors?

What do you like? What's it like working with you? Essentially, what are your qualities as opposed to, how do I gamify our stuff? So it's seen by other people. But anyway, I promised the audience that we were going to have some examples of when AI systems went wrong and people had to clean it up.

Heidy Khlaaf (23:37.24)
think this is a day -to -day thing, right? Like this isn't one event that is occurring. There's a lot of conversation back and forth, especially with AI in sort of like law and computer science and fields that require a of factual information. I'm seeing examples of it every single day. You talked about one case where a lawyer wrote, you know,

this defense and they cite, this is multiple situations actually, and they cite lawsuits that didn't exist, right? Or they make citations to paper, we've seen this in scientific papers over and over again, where they basically make citations to paper that do not exist. One.

really harmful example that we're seeing, because my background is safety and security, so this is a bit of a technical example, is that you have it recommend some code for you, right? And there's pros and cons to using something like Codex or having AI generate code for developers. And a really interesting use case is that...

You know, they call it hallucination where an AI model makes something up. I don't like to use that term because in some sense it makes it seem like it's human, but it's not. It's just not factually correct. It's just guessing. It's like, maybe this sounds right, right? Like that's kind of how they operate. And so it's making recommendations of libraries that don't exist. So a lot of times when we're coding, we like import specific libraries. So someone that wrote code that already exists somewhere. So we're not having to repeat it.

And libraries have specific names and usually if you just import it, you know, it will, it will go and like grab that code essentially. It turns out that they're making a lot of, made up names about libraries that don't exist. Right. And people, instead of verifying whether or not that exists, kind of accept it. well, it must be in there, right? It's guessing it. So it's pretty, must be accurate. And what a lot of hackers are doing is actually.

Heidy Khlaaf (25:32.118)
looking at what those guest library names are that don't exist and then going and implementing harmful libraries for them. So that way it can essentially take over, know, if you, if you have a piece of code that's been deployed and it used a library that doesn't exist and someone then managed to create that library and implement it on GitHub so that it's sort of imported it instead, you're now having compromised code in your code base and it can be easily hacked by, by someone. They basically put in what we call a backdoor.

Right. And so now you have code that you didn't realize was malicious, right? Because the large language models like hallucinated or really made up something that didn't exist. And we're seeing a lot of use cases of this daily. Like I'm constantly on my social media feeds, seeing people making wrong citations that didn't exist or worse. We've seen some impacts on people. Like we have seen, like a large language model make recommendations about.

people or misquoting journalists, right? And so people are getting in trouble for something that they didn't say. And the journalists are now having to go and trying to figure out who's accountable for this. I didn't say that and now I'm getting heat for it. Where did this rumor start? To only find that someone was using a large language model to like write an article of some sort and it made up a quote for someone. And we're not just seeing this for journalists, we're seeing it for like regular people.

being misquoted on day to day. So it certainly makes you also question, it's not even just about technical facts, it's about what is reality and what isn't reality. And I think that is one of the biggest problems that we're seeing. And I have an even larger concern in that we're seeing the Department of Homeland Security, for example, in the US trying to use large language models for our infrastructure, so our safety critical systems. And they're even talking about putting in power grid. And so

That hasn't happened yet, but they're working towards that. So what does it mean when a mistake happens there? How, like will lives, what lives will be impacted? And so we're now getting into the territory where sure, there's like a lot of issues with the disinformation that's like occurring. There's people who are being targeted for things that they didn't say. There's people whose code is being hacked for not understanding really, you know, how these backdoor attacks are being implemented. And now we have.

Heidy Khlaaf (27:51.146)
it going into our safety critical infrastructure. When it fails, if it's already failing in all these smaller use cases, what does it mean if it's being put into something like our power grid? Right. So that's a really big concern that I also see not just, you know, about current use cases that occurred, but looking at the future as well.

Sophia Matveeva (28:09.514)
And so what would be your point of view on safety critical systems? Is it let's not use these things at all or is it let's use them but always with conjunction of experts, of human experts?

Heidy Khlaaf (28:22.808)
For safety critical systems, you cannot use them at all because they don't even come close to meeting the expectation and specifications about accuracy and performance and safety integrity levels. It's like a whole thing how we measure these systems. And large language models aren't even like.

close, you your mobile phone has a better job of passing these safety standards than like large language models. So, but what we are seeing is something that you brought up early in the conversation is that there's a lot of regulatory capture where a lot of these tech CEOs are able to go and talk to politicians and sell them something that doesn't exist, right? That doesn't, it's not accurate what they're marketing, but unfortunately politicians and you know, what

Whoever is in charge of implementing some of these technologies do not seem to understand again We're back to the point of people not understanding the fit what we call the failure modes and the risks that come out of using these systems and we're seeing it every day, right? We're seeing like people don't know this but AI can't do math like your calculator does a better job of doing one plus one equals two AI can't do that and every day there's examples of hey I try to get it to do some basic math and it confidently answers with the wrong

Answer it just tells you no I am confident this is this and people try to show Even if you try to reason and correct with it. It doesn't it doesn't budge because again, this is not a factual You know technology it is a probabilistic technology it guesses what the answer is right? And so you can't have people guessing accurate numbers. You can't have people Guessing. I think a lawsuit exists. Maybe let's make it up, right? That's kind of the the impact that we're seeing here

Sophia Matveeva (30:07.666)
So I guess when we are using these things, then what we want to do is we want to think about, what's the risk? If I use this thing and it gets the facts wrong or it gets something wrong, what's the risk? So with my social media example, I mean, the risk is it's going to cut a less cohesive bit of the interview. So basically it's not going to be very interesting. It's not going to make sense and nobody's going to watch it. I mean, it's a shame.

But worst things happen in my life every day. So it doesn't really matter. But also what we do is we don't publish the AI clips straight away. My assistant views them. He gets rid of the nonsense because there is always nonsense. He gets rid of the nonsense. He keeps the good stuff. And then the system works. And so first of all, the risk.

Heidy Khlaaf (30:37.518)
Absolutely. Absolutely.

Sophia Matveeva (31:04.036)
if the risk is minimum. And also there is a human check. And would you say that kind of that system of, okay, evaluate the risks and have human checking, is that a good system in general, apart from safety critical systems?

Heidy Khlaaf (31:20.78)
Yeah, not necessarily because I think it depends on what you're automating. It might turn out that you're doing more work on doing and checking the work of AI than doing it yourself. There's also a lot of studies in what we call human factors integration. This is actually a field where you study sort of how humans in the loop work, how accurate is it, right? And there are studies that have shown that checking the work is more error prone.

than just doing it yourself. That depends on the task, right? So this is very specific. For a video where you're having to go and cut and edit clips and so on, that actually, I can see how that's really useful to use AI for that, right? But there are other cases like you're writing a scientific paper. You're going in and now checking every single fact when you could have just written that yourself.

I think that you then need to be more mindful and it might be just better off for you to use AI maybe, this sentence doesn't sound right. Can you rewrite it for me? Right. Instead of asking it to produce, you know, citations or to produce facts or to summarize things, because that's another thing we're seeing, especially in the medical field where we're having a lot of, we're seeing a huge marketing push to place doctor notes with AI, but the AI summarizes is summaries are not.

based on facts, again, again, people go back to thinking that it is like, but we gave it context. No, it guesses what the next word is based on its entire training data. It doesn't mean that it's going to summarize it for you correctly. And so in that case, I would say, no, don't use AI to summarize notes, right? Especially if it's something that is going to affect a patient's life, but.

Yeah, if you're looking to AI to cut up some clips preliminary for you that just makes it easier and you just go back and modify bits of it, that's a fantastic use case. So it's not always a case of adding a human in the loop. That's, you know, what we call it is the right solution. because it's then you're, you know, you're more error prone. You might be doing more work instead. So you have to look at it at a case by case, you know, basis, essentially.

Sophia Matveeva (33:23.868)
So essentially the world is more complicated and this whole debate about AI is more nuanced than I guess what Silicon Valley would have us believe. Well, that's no surprise. And the last topic I wanted to cover with you is that you've had quite a career pivot in recent years. And so I would love for the audience to understand what happened and why.

Heidy Khlaaf (33:47.566)
Absolutely. So for the past decade, I really focused a lot on safety critical systems and safety of AI. That was really what I did. So I worked on either building vulnerabilities to show people why AI is not safe or building what we call safety cases to try to at least figure out where AI can fit in some sort of, you know, product's life cycle and defining the safety metrics of what that means. That's really what I largely have worked on for a decade. And then we have had

multiple wars like the Ukraine war and the Gaza war with where AI is now being deployed in things like drone swarms, right? And that has been a really big cause of concern because those are all safety critical systems. AI armaments are safety critical systems. In fact, the entire safety field was really born from World War II, right?

Defense is actually kind of the leading research in safety or was really until until very recently and What ends up going through your mind when you see that civilians are dying at the hands of systems that you know how inaccurate they are is right

AI is already in safety critical systems and that's in military. And historically, anything that the military has done from a safety critical perspective has sort of trickled down into our civilian infrastructure. Because like I said, they're kind of two sides of the same coin.

Safety research is a lot of times done in defense sector. And a lot of the times what's determined by their standards ends up being adopted by civilian infrastructure. So if we're seeing something as serious as weapons, having AI implemented in them, it means that stakeholders have already accepted the failures for those systems. And we're going to see that trickle down. And also, I do not want to see civilian lives, you know,

Heidy Khlaaf (35:41.44)
essentially gone because I know those systems are inaccurate. I feel like that's something we can do about that, right? This isn't an inevitable outcome. I essentially, my whole life I've avoided working for defense, even though a lot of the work that I've done has ended up being used in defense. Again, safety is kind of the other side of it, is work in defense. And I decided let's...

dive right in, let's go into defense, let's think about autonomous weapons systems, let's think about AI in any military use cases and to try to bring attention to the DOD, anyone who has sort of a stake in this, that they have failures that they have not really thought through the consequences of, right? Because this is going to be catastrophic if we continue going in this direction.

And so I've started really diving into more why these AI systems are inaccurate from a safety critical perspective and to try to show people, to really build safety cases and make them understand all of these failure modes that we were talking about today in the context of military use cases and also civilian infrastructure and impact to civilian lives as a whole. And this push, think a lot of people might think, well, you know.

doesn't have anything to do with me. I'm luckily not in a place where there's a war. We are already seeing them deployed in, in civilian use cases. Like, you know, we're seeing drones with AI capacities being deployed in America, right? In America, you're having drones are flying around, being tested for policing purposes. And so if you're having an, you have inaccurate, vision system or an inaccurate algorithm,

really that detects someone as being an adversary when they're not right. And even who defines what an adversary is, is a big question. You're now looking at these automated weapons are going to go around and, you know, harm people or even kill them. Even, even like there's a really big concern that these algorithms are going to make predeterminations about who's guilty and who is not, despite how

Heidy Khlaaf (37:54.03)
much we know how like they're really inaccurate, right? We're looking, we're not looking at like, they're 99 % accurate. We're looking at they're 50 % accurate. It's a flip of a coin, whether you live or die. So I've really moved my career towards, right? I've been fighting, going into defense. I've been walking this fine line my whole life. It's now is the time that we're seeing it deployed. I think that means, okay, the flood gates have opened.

We really need to make people understand kind of the repercussions to our society and to future wars because of this implementation.

Sophia Matveeva (38:25.808)
And do you find that your message is being well received?

Heidy Khlaaf (38:32.174)
would say from a civilian perspective, yes. I would say from the perspective of those who develop weapons, no, because they are not really concerned, again, with civilian lives. They're concerned with their bottom line. And if they can sell another technology in which an entire military can depend on, they will do that.

they will do that because it benefits their bottom line. That is, I see the world in a very, like, kind of some sense, cold way. Those are people who producing those weapons, want money. And if they can convince stakeholders in the military that this is the right device without ever informing them of the consequences, they will.

Sophia Matveeva (39:13.938)
So we are back to what we talked about before, which is, okay, if you're a decision maker and you are being sold at all, which is essentially what this is, you whether you're a decision maker in government, buying a weapon system or whether you are a decision maker in a civilian corporate, you know, buying an LLM for your industry, then essentially you need to understand what are the risks, like what does this thing really do? And also be really clear about the fact that the person selling it to you has a

incentive to basically not really tell you what the risks are.

Heidy Khlaaf (39:48.43)
Absolutely, they have an incentive to make money.

Sophia Matveeva (39:52.147)
Awesome. Well, what a cheerful note to end on. Thank you. Thank you very much. I've really I've learned so much from this conversation. And if people want to learn more from you, I know that you've written a lot. And so where can people learn more about you and get more information from you?

Heidy Khlaaf (39:55.522)
You

Heidy Khlaaf (40:10.752)
Right now I'm working for the AI Now Institute and I'm writing a lot of, I'm doing a lot of research and writing a lot of white papers to sort of reflect this kind of work in defense. In terms of my previous work, there's my website, is heidk .com. I think you'll have to spell that because my name is spelled differently. Really I've...

Sophia Matveeva (40:30.544)
It'll be in the show notes.

Heidy Khlaaf (40:32.146)
Great, yeah, there's a media section there where I've spoken to people in the press about these specific issues. So if you're interested, certainly go there. But also I'll give a shout out to also where I work right now because that is where I'm producing my work at AI Now.

Sophia Matveeva (40:36.529)
you

Sophia Matveeva (40:47.762)
Awesome. Well, thank you very much for this very informative conversation. think everybody who's listened is going to have a lot to think about. Thank you, Heidi.

Heidy Khlaaf (40:56.6)
Thanks for having me, Sophia.