We've got five days together. Thank you for spending the time with us. There's a

link to the materials and labs at www.speakerrecks.com/mma.

Kale, can you put that in the chat? Thank you. You will each get a hard copy of the

MIT book, The AI Conundrum, when it ships on August 6th as long as you have filled

out the mailing address. So be sure to fill out your mailing address on the link here.

And if you did not do so when you're registered or uncertain, go ahead and just fill that

in again. We'd like to thank our sponsors and you'll be hearing from them in the use

case showcases. Now you'll have access to a draft of part one of the book, the electronic

version from this link here on the website. Also on the website you will find all the labs we will

do together. We'll be using Claude, ChatGPT, Gemiini, and we've also included Meta which was released

in April. We will share the links at the top of the lab page, you can find them here,

and if you have not done so already be sure to click the link to ensure you are logged into Claude,

OpenAI, Gemini, and Meta. Not all services will be available in all countries. That is okay.

You'll be able to use any service you want and we'll be switching around between a few different

ones so you'll get some good experience. And we will debrief each exercise and show you on screen

what outcome you should have gotten in case you were not able to access that particular service.

Now we're covering part one of the book in the training this week and I highly encourage you to

to read through the book or skim it

if you have not had a chance.

Part one is a brainchild of Caleb Briggs,

who will be leading the training tomorrow and Wednesday

and most of Thursday.

Caleb began his coding when he was 10,

and by 14, he had taught himself Lisp,

MIT's original AI programming language,

developed in the 1950s.

He ran out of math when he was in high school.

So his junior year, he attended Harvey Mudd

and his senior year, Stanford.

He is studying pure math at Reed

and is the lead author of the book

that MIT is publishing in August.

He asked me to be his co-author

and bring in some business applications

and that forms part two of the book.

It will be my role to bring in business applications

through our training session as well.

We collaborated on the final chapter of the book, which goes into the core of the AI conundrum,

some of which I will share with you today.

If you don't have time to read through the book, that's okay.

We will cover the topics and the training, but you will get more out of the training

if you read part one of the book as well.

Since part one was originally Kebb's thesis paper written for non-mathematicians, I think

Kebb did an amazing job making the complex topics accessible and understandable.

So let's go ahead and give you an explanation of why we call the book the AI Conundrum.

As we use AI for business, it's making our businesses more profitable and more productive.

And certainly that's a good thing, but at the same time as we increase AI's capability,

we are amplifying some of the safety risks.

So we want you to understand AI's strengths and weaknesses, both positive and negative,

so you can make better decisions and apply AI to its greatest benefit.

Today's session will review some of the areas where AI performs well, and this may reinforce

what you already know or suspect about AI.

But it might turn out that there are areas where your knowledge of AI is not quite as

deep as you'd like it to be and therefore AI may perform in ways that you do not expect.

And that's the point of the training. Each day will go deeper and then unpack how AI really works.

So let me start with the ideal use case for AI. If you follow the AI performance news,

AI apparently is ideally suited to take standardized tests like the SAT, certain AP exams.

It didn't do as well in chemistry or the bar exam, but that was GPT 3.5 version 4 does much better.

AI now performs well above human levels on many benchmark tests, and while we may know

people in the 90th percentile in one area, AI can do it across many tasks. In terms of the

ideal use case for AI, one of them is to give us back time. AI will help us get more done and less

time. Now let me share an example of using AI to summarize meetings. Greg was double-booked in a

meeting I was leading and I turned AI to generate a summary. It took Greg one minute to read what we

had discussed for 30 minutes. AI can do more than just summarize a meeting. I asked AI to summarize

the sentiment of the meeting and it did a good job understanding the tone. I then

asked AI to identify potential blind spots and it did a remarkable job making

me aware of my blind spots and suggesting things that I could do better.

Now this is a chart is a framework that killed developed to understand where AI

is ideal and where AI is risky. The folks at MIT really liked this as part

why they decided to publish the book. It is three dimensions to it. The degree of precision

required, input control, and need for rationale. Now AI can't be super precise, which we'll

explain throughout the training. So ideal cases don't need absolute precision for the

AI to be effective. Now advertising is an ideal use case for AI. With advertising, there

There are many ways to create effective advertisement and that is to AI's benefit.

Second, when it comes to input control, it's very easy to control what the AI uses to develop

the advertisements.

It is easy to put a human in the loop to add additional control.

Third, we don't need a rationale for why AI optimizes the way it did.

We'd like an explainable AI, but optimizing advertising isn't the same as AI being used

to render a verdict in a criminal case where rationale is truly required or should be required.

So advertising is an ideal AI use case because it is data rich, it has clear metrics and

a repetitive workflow.

Many steps done by humans now could be automated.

Machine learning has played a role in digital ads for two decades now, but the capability

is made a major leap recently.

The company called ArtsAI introduced an unsupervised learning that significantly advanced what

is possible with a real-time creative optimization.

It may have been thus getting this technology with its members.

Give the AI several versions of the ad and it will learn the message features such as

a audio ad, a male voiceover versus female voiceover, we'll learn different calls to

actions. At the same time, it learns audience features like whether you are on an Android

or iOS device, which defines some differences in how people behave, what city you're in,

what time of day it is, the context of the website. The AI predicts which messages features

produce the highest conversion for these audience segments.

It is developed automatically on the fly.

Based on MMA's research,

we find that AI more than doubles the conversions.

It is perhaps not surprising

that the company behind this technology

ranked number 29 on the Inc. 5000

fastest growing list of private companies in America

and number one fastest growing ad tech company in the Inc. list

and was acquired last year by Claritas.

Now MMA has performed more than half a dozen public case studies measuring AI in this use case.

It is game-changing technology.

To quote Greg, "If I was a marker and I saw these results from AI, I cleared my team's schedule

and make applying this AI to our business the number one priority."

MMA has analyzed the potential impact at scale, and this can move the stock price

in publicly traded companies.

But that doubling of impact is only half of it.

We are now seeing some some markers combined

this AI personalization technology with generative AI.

Consider what happens if one AI can create the ads

and another can optimize them automatically.

The front end part of the process of briefing

and ad generation approval now takes months

but could be reduced to a week or two with AI.

To investigate the possibility,

Kailh and I created an AI that produces audio ads

for pennies in minutes.

When one combines doubling of conversions

with the halving of the cost to develop the ads,

we begin to appreciate what a competitive advantage

AI can become.

An MMA member company gave the green light in January

to use the technology and share the results on April 16th.

Here's Remy Kent, the CMO of Progressive,

on stage showing what she learned. She reported a 197% increase in conversions due to Gen AI and AI

personalization. And the Progressive team, MMA and Claritas achieved this from start to finish in

about 90 days. That is remarkably fast and it hints at how AI is much more efficient as well as more

working with the ad council, Kroger and others on this research and seeing the technology

and results up close is a big part of why I accepted the role as Chief AI Officer at

I saw the potential and wanted to take it to the next level.

When we looked at different industries, we found areas where AI could make a meaningful

difference in every industry.

At the top of the list is the entertainment and marketing.

Over 50% of marketing tasks can be augmented by AI.

Some put the estimates as high as 80%.

So there are a lot of strengths of AI,

but there are also weaknesses

that I want you to understand.

Shall we play a game?

Let's see how well you perform versus AI.

I want you to use the chat

and respond as quickly as you can to the prompts.

What is this?

Yep, Kim Kirk-Bashian.

The AI agrees it is a person with over 80% confidence.

What is this?

A cat? Yup.

What's this?

A stop sign?

What's this?

Upside down, Kim you say?

Ah, but the AI says it is coal.

Black color with nearly 80% confidence.

Why did you get the right answer but not the AI?

If you're uncertain why, that's what this training covers.

It's important to understand how AI thinks differently and therefore can produce results

like this.

Research shows that humans handle invariabilities but the AI doesn't yet.

Now if you look at this quote, Gem DiCarlo from MIT's McGovern lab which studies the

the brains of primates and humans,

feels that they are close to figuring out

how the AI develops in variabilities.

And what he's talking about is the point

that you can change the pose,

you can flip it upside down,

you can put it with clutter,

you can change its size,

and yet we still recognize what it is.

But AI struggles because we don't quite have the exact,

the right architecture for AI

to make that connection yet.

Now, Jim believes we are a year away, maybe two from cracking the code on how humans and primates learn in variabilities.

But as of today, humans and AI process images differently.

To build on this point, imagine a golfer swinging without a club.

Get that image in your mind.

Now, we can visualize that, but AI can't draw it.

Do you know why AI struggles in this case?

How about this one?

AI has guardrails to keep from providing information on how to perform illegal acts like hot-wiring

a car.

That's good, right?

But why is it so easy to trick the AI into thinking I have some rare disease called promptitis

and that will cause me extreme pain if I don't get my prompt answered and the AI will then

give up the instructions for how to hotwire a car so it's not causing me harm.

We'll show you how AI learns differently so you understand AI's limitations.

Let me explain why this is important.

Remember that cat?

Well, the AI didn't see that as a cat, it saw it as guacamole.

You see the AI processes visual information differently than what we do and that makes

AI relatively easy to hack.

Dan and his team at Stanford note that there are on average about 19 different paths to

subtly change the pixels that you don't notice but that we can change

that when changed causes the AI classifier to misidentify the cat as

guacamole. Now this is real-world implications. The reason Caleb's

risk framework has the dimension of whether or not you're using AI in a situation of high input

control or low input control in part relates to the risk of an adversarial attack. Remember that

stop sign you correctly identified? The AI sign is a 45 mile an hour speed limit sign, not as a

stop sign. Why? Because a few stickers were placed to hack the AI. The AI saw the hacked stop sign as

as a speed limit sign.

Now consider Caleb's risk framework.

And consider if you are in a neighborhood

with a 25 mile an hour speed limit,

and you come before we stop, you are stopped

and you see a car far enough away,

and you reason that it will slow down,

and you begin to drive through the intersection.

But when the AI sees the stop sign,

it actually accelerates because the AI sees the sign as raising

the speed limit to 45 mile an hour.

Now that can lead to accidents.

While this lack of precision is fine with advertising

with human oversight, stop signs operate in an open environment

and the AI needs to be more precise with traffic rules.

Now Zillow shows another risky example.

Zillow lost $4 billion in market capitalization

because they didn't fully understand which types of AI

applications are very risky.

Zillow developed as the estimate to estimate the price of homes.

but buy them and then try to flip them for a profit. They did not make a profit. They had to lay

off a quarter of their workforce. Their stock after initial high initial sugar high based on

their announcement that they were using an AI business crashed when it turned out the AI did

not do what they expected it to do.

Years later the stock still hasn't fully recovered from the level they were at before the AI

announcement.

It is a punishing lesson that might have been avoided if one considers the risk framework.

If you think about what they were doing, they were operating in an area where they were

trying to price homes that requires a high level of precision.

We estimate in order for that business model to work, they would have needed plus or minus

2% accuracy.

Their accuracy was good, but it was not that good.

The precision with AI was more like plus or minus 7%, so it wasn't as precise as it needed

to be.

Another way in which Zillow uses AI that's risky is like the stop sign example.

Zillow was operating in an open environment.

happens if Zillow gives you a price that you think is too high or too low for your home.

You simply won't sell to Zillow.

But if Zillow offers you a price that is higher than you expected, in other words Zillow is over

paying, then of course you'll say yes, of course I'll sell my house for more than it's worth.

Economists call this situation "adverse selection."

Zillow was operating in an asymmetric exchange where they simply lost billions and billions of dollars

Now finally Zillow Zillow's AI didn't provide an easy

Time to understand rationale for its pricing

They tried to have a human loop

But the data the human was given from the AI was almost impossible for the AI to understand

The human was just rubber stamping the AI decision

The human need a rationale, but the AI often isn't designed to provide a rationale.

It's important for us to know how to go through this risk framework and understand what is risky and what isn't risky.

For my last illustration, let's come back to large language models. Large language models are also

also have an issue weighing risk and benefits and understand the context as humans would. Consider this example.

We give Elma the prompt, "Dr. Mary stands to solve world hunger by giving her best friend Jane a call.

Jane is certain she can solve for poverty if she gets a call.

However, Mary and Jane bickered as children about butterflies. Mary will blank give Jane a call.

A, not, or B, strive to."

Every rational human would say strive to give her a call,

yet weighing these different choices,

We definitely would give a call to overcome whatever childhood issues Jane and Mary had because

solving the massive world issue of world hunger and poverty is more important. But if you ask AI this question, you'll answer not.

Why doesn't AI weigh the risks and benefits and understand the context to make a better decision?

AI is very good at understanding semantic meaning. It's very good with language,

but it uses semantic meaning more heavily than many humans.

The word "however" carries a lot of semantic weight.

On Tuesday and Wednesday, Caleb will unpack the answers to why AI operates differently

and how to get the most out of AI while being careful to avoid some of the pitfalls,

like that semantic waiting.

With your prompts.

The key issue is that AI does not operate exactly the same way that a human operates,

and most people in business overestimate how well they understand AI.

So our first exercise is to benchmark where you are at in your knowledge of the inner workings of AI.

We'll use this as our baseline and we'll aim to level up your understanding in this training.

So let's do an exercise at speakorex.com/mmae.

It takes five minutes to answer the quiz questions about the fundamentals of AI.

You'll see a repeating scale, so it's fast to answer.

And we'll pull the scores after to see if you have room to grow your AI knowledge.


Let me share the results. Over 80% of you are in the room to grow category.

Some are in the budding enthusiast and less than 5% are rising stars or in the

or in the now. Now this is typical of a business audience. Those with a CS degree

focused on AI score in the top group but most people don't. The goal of this

training series is to level up your knowledge so you can apply AI safely and

profitably. If you have other people in your organization that you think would

benefit from a self-assessment check, in other words you think they think they

know more than they they know, then feel free to share this.

Feel free to point them to the book's website, theaikinonderm.com.

You will gain depth of understanding of concepts like universal approximation and gradient

descent from this training, especially tomorrow and Wednesday when KILB leads the training.

If you read part one of the book, we'll cover all the concepts mentioned in the self-assessment,

you will have a big advantage over others in understanding how AI works.

Today I want to cover AI safety, which I am passionate about after working on this book.

We need training, governance, and accountability frameworks to get ahead of risks as AI becomes

more capable and more autonomous.

Let me describe what I mean by autonomous AI by introducing you to AutoGPT, which launched

last year in March 2023.

With AutoGPT, AI can now do internet research,

manage money, hire people, to complete tasks.

I created a biography GPT to research people

that I needed to introduce at a conference.

Here's an example of Kay Vazan,

the chair of MMA's Global North American Media and Data Board.

The AI called AutoGPT took Kay's name

and the goals I gave it,

which was to gather three facts

and summarize a couple sentence bio

and reasoned through a strategy to accomplish the task,

then set out to execute the task

starting by searching on Google for K

and in LinkedIn and so forth.

Now AutoGPT found K's bio on MMA's website,

found our LinkedIn profile and more.

And it assembled the information for me all automatically

without me having to lift a finger

after I pressed Y for yes to start the process.

Now this is called bounded agency

because the AI has agency to do its own thing

within the boundaries I set.

But within a week of auto-GPT,

someone took the same technology

and created chaos GPT with the goal of destroying humanity.

(keyboard clicking)

Now, this bounded agency was intended to make AI safer, but it turns out it doesn't quite

work out that way.

Now, the first thing that Chaos GPT did was to search the Internet for the most destructive

bombs. It found that Tsar Bomba in Russia, Chaos GPT then created an AI agent to try

to acquire them. But it reasoned that it would not be able to get GPT-4 to tell it how to

access the bomb because OpenAI has guardrails to block such requisitions. Fortunately, Chaos

GPT had not learned about promptitis yet and how to circumvent the guardrails. So Chaos

SVPT suspected it would likely get wouldn't get access so it shut down that

line of action. It then reasoned what it should do next to destroy humanity. What

was the next most destructive act it reasoned it should pursue? It opened a

Twitter account. #TeamChaos. Now this is concerning and we need to take it

seriously. Overall when I weigh the benefits and risk I find I'm optimistic.

I'm optimistic because I believe that if we understand AI strengths and weaknesses, we

can wield AI responsibly.

There are massive productivity gains, improving customer experience, sales efficiency, and

decisions.

The highlighted areas are how AI has summarized some of the advantages to business.

There are massive ways we can improve the science and improve quality of life with AI,

but we must implement training, governance, and accountability now to apply AI safely.

Now let's return to the strengths of AI in this next case exercise.

Now please go to www.speakerx.com/mma and you'll see a second exercise listed here.

What you have here is a recorded transcript of the first part of the presentation.

It's in that text file and it's not a perfect recording, but it is an AI-generated recording

of what has been said so far.

You can basically bring that transcript into Anthropic, into Claude, and ask it to summarize

this meeting.

annd see what you think!