Hey everybody, my name is Dan Johnson

and I'm on the protocol security team

here at Coinbase. I'm joined here today

with Mitchell Amodore, the CEO of

Immunifi, a web 3 security company

that's been dedicated to protecting the

onchain ecosystem since 2020. Today

we're serving up our 11th episode of our

ongoing Coinbase security series where

every two weeks our team gets together

and we connect with the great builders

in the web 3 community and discuss a

topic of blockchain security and how to

build safely in the space. As is the

case with each of these sessions, please

note the disclaimer that's at the bottom

of the screen. Uh, and with that, we've

got a great session for you today. So,

I'll let Mitchell take it away.

>> Okay. Okay. Exciting stuff. We're going

to talk about something terrifying

today. No pressure. Doesn't mean you

should run away. It means it's going to

be even more exciting, but just heads up

on that. It is. First things, hear me

loud and clear. We're ready to go, Dan.

>> Yes, sir. Sounds great. And uh you can

go ahead and test uh sharing your screen

and getting the presentation up and

running as well.

>> Favorite part the the difficult

>> difficulties right right here.

>> I'm scared already. But you're going to

you're going to make the uh you're going

to make this topic less scary for

everyone by by educating us all.

>> Only a little bit. I suppose it's all

contextual at the end of the day.

>> So, [laughter]

you know, we're going to make some parts

less scary. We're going to give you

agency. We're going to give you some

amount of control, but we're also going

to show you the horrors of reality and

what can go wrong. And I think that's

just as important here. Now, uh to kick

off, this is going to be an important

presentation. If you are a builder on

chain, this could be the presentation,

the content that you consume that saves

your project from total destruction. And

if you're a builder, you probably have

all of your network plugged into that.

This could be the thing that saves

everything. I'm going to show you how to

survive the hack experience. It's not

going to be easy. It's not going to be

quick, but I'm going to give you the

tools that have been tested over

hundreds of incidents in the industry,

many of which I have seen and

participated in and help save firsthand

to stay safe when you are targeted

successfully for exploitation by

blackout hackers, no matter where

they're from. A little bit about me for

context. As Dan said, I'm the founder

and CEO of Munifi. We are the leading

security services platform in crypto. We

protect the vast majority of the

industry around uh 180 or 200 billion

worth of assets and contracts and beyond

today. Uh we have one goal which is to

keep the industry safe. That's why we're

made in the first place and we're still

doing that to this very day. We made

incredible progress in that time and a

lot of that progress via on the you know

the bug bounty side the technical side

security operations opsec infrastructure

a lot of those experiences right that we

those lessons that we were able to learn

come from surviving and seeing how bad

things can get firsthand okay these war

rooms are the key to that but just

because something scary is going to

happen doesn't mean it has to be a fatal

experience and we're going to show you

how So, for starters, the first thing

that I want all of you to understand is

that war rooms are inevitable. If you're

building something big, you're building

something exciting, you're building

something meaningful on chain,

contributing to this whole new world

that we have going for us, then you're

going to hit some problems and they're

going to be very, very scary and

eventually you're going to make a

mistake. You're going to get pulled into

some incident or another. If you're

lucky, the damage will be small and

contained. If you're unlucky, the damage

can be total and fatal. We've seen many

such cases. A little example of what

we're looking at here, right? On the one

side of small and contained, maybe you

have, you know, certain backend

infrastructure gets compromised allowing

users to spoof or get spoofed or or

man-in-the-middle attacked in various

ways. And if you can just find that

piece of infrastructure that only

affects a certain amount of your users,

you stem the bleeding. Great. You're

safe. You made it through. Okay? Or you

could have your front end hijack. Great.

If you move fast enough, you know, most

of your users may not be affected by

that. On the other hand, you have

incidents like compromising of treasury.

Okay, so a good example of that is

something like the buy bet hack. This

has occurred on so many exchanges with

hot wallet compromise. You could lose

all your assets overnight. And of course

in in DeFi, the most exciting expression

of the onchain economy, this can become

even more total because we typically

pool funds in various ways. Not always.

There are there are a variety of

infrastructures, but you may be pooling

your funds in whatever architecture of

the contracts that you've made. And so

if someone can compromise that pool,

they might be able to steal all of the

user funds. And we've seen a number of

such cases. We've defended against a

number of such cases. But the important

thing to keep in mind again is that

these war rooms, they can't be avoided.

You're no matter how good you are,

you're going to make mistakes. Everybody

does. The attackers have infinite time

and resources to attack and you only

have so much time to defend. So the

solution is not to avoid war rooms. The

solution is to be prepared and to know

how to carry it out when it inevitably

comes and do your best to prevent it in

the interm. Now when you enter a war

room, you have two simple objectives. I

make this real clear for you guys. Your

first goal is to stop further losses.

Wherever think of yourself like a

doctor. You have a patient. He comes

into the ER, what do you do first? You

stem the bleeding, right? You need to

stop the thing that will directly

contribute to loss of life or problems.

You need to stop further losses. You

need to stop the bleeding of users. It

is very common for attacks to be carried

out over multiple blocks or to be

subject to a variety of particular

conditions and not be totally executable

in a single block or in a single

transaction. In such cases, you can have

a chance to prevent the majority of

damage. In most of the major hacks that

we've seen, this has been the case to

greater or lesser degrees. Okay, this is

especially so the case for for hacks

that are not strictly onchain hacks. So,

you need to find the cause and you need

to apply that tourniquet. You need to

arrest it. Your second goal is to

preserve user trust. And this is also

critical because even if you

successfully stop the bleeding and save

the users's funds. If you fumble if you

fumble the post hack communication,

people may trust you less and you may

end up destroyed anyways.

Congratulations. You'll have saved the

users, but you will have been burned.

You will have been scapegoed because

your your inability to maintain

legitimacy despite your good actions.

And so we need to make sure that we

maintain user trust. And this is a

matter of communication, messaging, and

of course, the conduct that we take in

the war. I'm going to show you how to do

that, too. But starting from the top,

how do we stem the bleeding? What do we

do? Step one, we confirm the nature of

the vulnerability. We confirm the nature

of the attack. How did this happen the

way it did? We identify the root cause

of it. How did this happen? Can it

happen again? Very often, uh, an exploit

is the tip of the iceberg for how it can

be exploited again and again and again.

So a good example of that is when you

have projects with multiple pools uh

which is a great many DeFi protocols for

example if they can exploit one pool

they may be able to exploit many other

pools and it's not a given that the

attacker will have done the research to

identify all of those other vulnerable

tool uh pools. In fact most attacks

historically have been different

versions of driveby attacks. The the

attackers cannot trivially check all

these things. So you need to take action

to find that exploit and see what you

can do to to stop the bleeding from

there. And that gets us to the next

step, right? Which is to pause all

affected contracts. And very often if

you can't tell what that means, like

which contracts are affected, if you're

just not sure, you're having struggles

reverse engineering the exploit and

seeing what it really is, then you

should be pausing all contracts. You

don't need to take the risk. Pause them

all. Save the users. Most people won't

even notice to be quite frank. It's not

like they're watching you all the time,

but they will be thankful after the fact

and you will buy yourself precious hours

to identify what really went wrong. The

third step that you need to take is you

need to lock down all permissions, all

admin access whatsoever. need to know

exactly how the state can change from

here and anything

you need to eliminate all possible you

know gray zones or gaps that you might

have anything that that could have been

the surprise and make sure all your

other infrastructure is taken care of as

well because you don't necessarily know

how far the breach goes. Historically,

most onchain hanachics were discrete

events in the sense that they happen um

just onchain. But what we've begun to

see over the last few years with the

rise of more state sponsored hacking

groups, most famously the Lazarus group

with North Korea, is that they are

compromising a variety of pieces of

infrastructure on their road to ultimate

theft. Okay. And this is more typical of

traditional uh advanced persistent

threat behavior in web 2 and the broader

internet. So you got to lock everything

down. Don't take any chances. You don't

know who you're dealing with. They could

be very scary people. They could be

wellresourced people. They probably are.

Be maximally paranoid. Lock it all down.

With this, your situation will be

paused. Hopefully, you will have frozen

the state of your contracts, of your

protocols as much as you possibly can,

and you will be ready for the next step.

Hopefully, this is you stemming the

bleeding when it comes down to actual

thefts. [snorts]

And if you can take nothing away from

this presentation, then take these two

things, first objectives, stop further

losses and preserve user trust. Second,

these three basic steps. Find the

exploit, pause everything, lock down all

your infrastructure. This gives you time

to live another day. Now, all of that is

easier said than done. So, now we're

going to talk a little bit about how

that works. We're going to talk about

how you should be constructing that war

room to get all that done. And at the

highest level, there are three basic

roles in my view that every war room

should consist of. These roles, these

functions

which can be performed by one or the

same people but are best performed each

by individuals is the operations lead,

the security analyst and the

communications lead. Let's touch a

little bit about what each one of them

does. First is the operations lead. Your

job is to command and to coordinate

functions. When I'm in war rooms, this

is typically the job I get to do. It's

also one of the most painful and most

difficult jobs and you get the

unpleasant responsibility of making some

very difficult very hard choices very

often. In this function, your goal is to

relieve all unnecessary thought and work

from your compatriots so that they can

do what they do best while you organize

overall sequencing and managing of the

remediation process. Okay.

Your job here is to set up all the

workflows, set up tracking,

coordination communication decisions

and make the best calls that you can,

often as a group at every single point.

The ops lead may be a founder or CEO,

very often is not, but is someone who

will defer ultimately the the the most

difficult decisions to uh a more formal

business leader who will make the final

call. But the ops lead is responsible

for organizing the work in the leadup to

that. Set in another way, your job is to

keep the entire context of the hack and

the war room in your mind at all times.

And what this takes form in practice is

you end up creating the documentation.

You end up writing and keeping track of

what's going on. You keep track of all

the participants in the war room. You're

identifying and directing them to all

the tasks that they should be doing.

You're setting the sync cycles, aka how

much time you should allow to pass

before you come and check in, which I

would typically recommend about 15 to 30

minutes. Every 15 to 30 minutes, you

want to check back in with your teams

and see how they're doing on their

different problems. And you get the

unfortunate job of time boxing the

decision, the decisions, and being like,

"Okay, well, how long do we have to make

this call? And what are the trade-offs

here?" Sometimes you will be deferring

that final decision to the formal

business leader. Sometimes you're going

to have to make the call yourself. But

again, your job is to have the full

context, the command and coordination

function of the entire war room. And

it's really critical that you get this

right. Okay? Because in a war room, what

happens to everybody is they freeze.

They get stuck. They get worried. They

panic. They say, "Can I really do this?

What am I supposed to do? My network, my

net worth is going up in smoke. Am I

going to go to jail? Did I make a

mistake? Is this my fault? Am I guilty?"

And as an obsolete, you have to put an

end to all that. There's no room for

such sentimentality. You need to go and

fix the problem. You have to provide

leadership, structure, and clarity to

the team members and inspire them to do

what needs to be done. In the vast

majority of war rooms, the offset may be

the only person who is experienced, who

has done it before. Most people

obviously have never done this before.

And you need to provide them the

certainty and the conviction that they

can go forward and succeed and that

there's going to be a future on the

other side of this. So that's the

obsolete function. This role can be

folded in to the others but ideally

stands alone. The second function is the

security analyst. The security analyst

has the critical and difficult job of

identifying exactly what the attack

vector was and what the exploit was and

how big the scope of damage and risk is

so that they can prepare a mitigation

path and ultimately apply that and block

any other exploitations. This is the

person who is doing the hard technical

work to fix the situation, right? If if

the ops lead is like um a doctor

coordinating, you know, a hospital

situation, then the security analyst is

like a surgeon who's got the difficult

job of actually doing the cutting,

making sure he's found exactly the right

problem and addressing the wounds and

and suturing them up all properly. Now,

you might think, oh, you know, security

analyst, isn't that the most important

part? Why don't we just fold the the

offset role into that? No, we don't do

that because the security analyst role

is extremely cognitively intense and

they are in a race against time to

identify the nature of the attack vector

and to mitigate it before it can be

exploited further. And no matter what

kind of hack you're in, you never know

if there is more damage in the

beginning. You have to suspect that

you're at the beginning of a process,

not at the end, that things can get

worse. So you need to take your security

analysts that could ideally it's uh one

or or or two people working as a team

should be a small group and you need to

strip everything that's not critical

away from their mind. Anything they

don't have to think about they shouldn't

be thinking about. Anything they don't

have to do, they shouldn't be doing.

They got one job. Find the source of the

bleeding and stem it as rapidly as they

can. That's their one job. Now,

typically how this works in practice is

after you come together in a say a chat

room of some kind with your ops lead and

your comms lead and your security

analyst and there might be one or

multiple of each of those roles if

you're lucky. The security analyst will

go off on their own into a private chat

to go and do this work and trade ideas

directly. You want to allow them to go

and do their own thing because focus is

going to enhance the quality of the work

and the speed at which they're going to

get it done. If at all possible, you

don't want to remain in the same group

chat where you can hear everybody else

speaking. Instead, the security analyst

should be regrouping with the overall

war room every 15 to 30 minutes. 30

minutes is a great cadence in my view.

Uh to bring back their findings and

their recommendations on what to do

next. Obviously, if they have some type

of breakthrough that it makes sense to

do that even sooner, like you do

whatever makes sense. But you want to

give space to focus, right?

Communication in a war room is

expensive. It costs time and attention

and stress and you only want to do as

much as you absolutely need to get to

the right outcome. It is fine to have

multiple security analysts. In fact,

generally uh I think a a small team of

two to three is is is good. Two is

probably best providing people know the

protocol well.

one is good, but getting extra eyes on

that is better. It's not a case of, you

know, more is better. You keep adding

more people, you have more people to

coordinate, things get more complex,

people disagree about things like you

don't want an infinitely large team. Two

to three is probably uh where it's ideal

to sit. But this is the security analyst

role. If [snorts] you're unsure of who

to bring on, then you, you know, I

recommend if you don't have a security

analyst in a pinch, you go and you call

your security providers and see if they

might help. This is how we have done so

many wars at Immunifi. We have done I've

personally done dozens and dozens and

doz like maybe somewhere at least the

high dozens of war rooms at this point

and that's because our customers are

calling us in. They have a problem. They

don't know how to deal with it. Nobody

else they know knows how to deal with

it. They trust us. We come in there. We

manage the situation with them to come

and deal with it as best we can. You can

also call Seal 911. They have a great uh

free service that you can use that they

also get involved with war rooms with

experts from around the industry. You

can also call an auditor or similar such

player or you can just pick the best dev

on your team. Like you do you. You're

going to have a variety of options

available to you. Uh I suggest that you

take all of them that you have on

access. If you have access to the

resource, leverage it because now is the

moment to cash in on that favor. The

third function is trust management and

that means your comms lead, your

communications leader who is responsible

for coordinating all the messaging that

needs to go out. And believe me, war

rooms have tons of messaging. This is

rapid fire PR that you have to get done

and avoid making any mistakes because

one wrong mistake when a user is in a

state of fear and panic is a reputation

forever lost. and TVL that will never be

returning to your contracts. So you need

to focus very very finely, very very

specifically on sending the right

message and inspiring confidence and

trust in your users that they can trust

you to handle these situations. Okay,

war rooms are inevitable because hacks

are inevitable and everybody on chain

knows that. So your users need to

believe that you can handle these

situations and by the end of this

presentation you will know how to. Now,

the comm's lead is best suited to be

someone else. They can be technical, but

typically writing is a specialized skill

in my experience, but they do need to be

focused. They need to be quick on their

feet. They need to understand the

project as well as the as well as

possible. And their job here is to

communicate exactly what the community

needs to know at every step. In

practice, that typically leads to calm,

clear factual updates, where you avoid

hyperbole, but also where you avoid uh

how do I say making things rosier than

they look. You don't want to speculate.

You don't want to project positivity

prematurely. Like you can save that

stuff for the end in the moment while

these worms are happening. And worms can

be wild. I mean it can be a few it can

be you know 15 or 20 minutes for

exceptionally short cases but it can

also be 24 or 48 hours you know I I've

done done some war rooms that end up

being 3 or 4 days because we couldn't

really manage the issues over the time

frame which is very painful uh I've done

several such cases like it can it can

get long you don't want to say anything

that would cause users to lose trust in

you later and so intuitively the best

approach the one that inspires the most

confidence is to treat them like an

adult. Clarity,

calmness, but at the same time, you

know, confidence, conviction, and don't

say what you don't need to say. People

will have the temptation to try and make

people feel better by making claims that

they should not. Don't do it. It's a

trap. Also, cadence. Very important. You

should be communicating at regular

intervals. Other than the hack itself,

one of the most terrifying things is

when a project goes silent on you after

all these events. You don't know is that

because they freeze. It's a hack. You

also have to suspect was it an insider

attack? Are they rugging you? Right?

There's lots of reasons to be worried

when a project goes silent. So don't do

it. Consley needs to come in there,

communicate clearly, communicate

regularly and inspire confidence that

they can handle the situation.

Typically, if they need to do their

work, they will also go into a separate

chat to go and coordinate that. There

could be one or two people. You might

also bring in a PR firm if you have um

access to partners there. Um you will

often work with the ops lead and the

security analyst to really make sure

that the framing and the messaging is

correct from a technical perspective and

from an operational perspective.

But otherwise you're working somewhat

similar to the analyst. Unlike the

analyst whose job is really to to

investigate how an exploit occurred, a

comms lead role is much more

forwardlooking. So as comms lead you

should be preparing for multiple

different timelines not just one

situation but multiple paths forward for

example we stemmed the bleeding

everybody's safe right all users are

fine for example we couldn't stem the

bleeding everybody is unsafe here's what

you have to do next for example it was

never a hack at all we found the cause

of it was a user error or something

great that's dealt with as comms lead

you need to prepare for everything in

order of the severity and the danger uh

of the scenario. So you end up with

pages and pages of documentation in that

process. In parallel, there's another

thing the comms lead should do. The

comms lead should work with the ops lead

to document what is happening in the war

room as the events proceed. Creating a

living timeline of events because the

more the comms lead can take over the

documentation duty of the war room which

is going to be needed. All this

information about what happened, who was

in the war room, what exactly they did,

why they made the choices they did, all

that's going to come out in the

postmortem, which can be an internal

doc, but it's best also an external

documentation that is shared with the

the broader community so that everybody

can learn how to stay safe. All this

information should be recorded by the

comm's lead while the events are

ongoing. Yes, you can have the

operations lead do it, too, but if you

do that, you're going to be distracting

them from the most important choices.

So, wherever possible, the comm's lead

should take point on that. Again, in

general, every role, every function

should be done by a single person and

not merged if you have the option.

That's the setup of a war room. Those

are the three main roles you need. Those

are the three things that they're going

to be doing. That's how they're going to

be driving the process. And we heard a

little bit about the sequencing of

events. Now, the next thing now that we

have this context, the important thing

for you to remember to keep in mind is

to remain calm. Remain very calm. Like I

said in the beginning, people mess up,

people freeze, people become afraid.

Your ability to remain calm lets you

make good decisions about the matter.

But more than that, your calmness is

contagious. Your reason

transmits. The calmer you are, the more

centered, the more focused, the more

centered, focused, calmness you're going

to bring to everybody else in the war

room and the better they're going to

perform. So whatever you do, remain

calm. Very important. Now, in terms of

how this goes, we walked through a bit

of the functions. All these things are

actually happening in parallel. Now, you

see why we have an operations lead in

the first place because we're not doing

these things one at a time. No, no, no,

no, no. We're doing all of these things

at the same time. And we're in a race

against time while we do it. Even more

difficult, even more stressful. We're in

a race against time to identify the

source of the breach of the source of

the attack to block it and then to

preserve the trust of our users and the

community at large. So while you the ops

lead or whoever you may be is preparing

the decision flow, mapping out how the

the sequence of events could go and what

are the various pathways and how you're

going to respond at those pathways to

make the calls. The security analyst

needs to be doing the forensic work,

understanding exactly how a fix and a

mitigation can be implemented and

preparing everything needed to execute

that fix. And the comm's lead for their

part, they need to be preparing the

messaging in parallel for the various

scenarios. You don't know how much time

you're going to have to execute this.

You don't know how much time, you know,

emotional contagion uh can take from you

and how quickly it can spread. And

again, remember, it's not just the hack

that can kill you. It's also the loss of

user trust and legitimacy that can do

the job. So you need to be ready for it.

So everything needs to be done in

parallel as fast as possible. And the

ops fleet has the venerable duty of

coordinating all this and keeping every

decision the entire state of the war

room in their head at all possible times

so that they can maximize the speed of

execution. Having said that, once you're

done all that, you can now proceed with

postmortem. Okay. Postmortems are

typically done 24 to 48 hours after this

events are are wrapped up and the fix is

applied. And the reason for that is

because these war rooms are exhausting.

It is not usually feasible to do the

postmortem right away afterwards. People

are too tired. People are too stressed.

You know, it's just too much to deal

with. But the next day, you want to get

this postmortem out as soon as you

possibly can. The users are waiting for

them. They want to see that you handled

the situation correctly. They want to

know that they should feel safe. They

want to know that their money has not

all been stolen from them and your

postmortem is the way you do that. Four

key components. Root cause of the

exploit, the attack and postmortem

timeline of what exactly happened, the

fix that was discovered and applied, and

what actions will be taken further into

the future to ensure that this type of

attack never happens again. Very simple,

little bit stressful to write, I'll

admit. But this is the outline for every

postmortem that you should do. If you do

this, you're not just going to be

helping your community members, keeping

them engaged, keeping them loyal,

keeping your your own project legitimate

in their eyes and worthy of trust.

You're also going to be supporting the

entire future security community on base

and in the broader blockchain ecosystem

as we all learn the lessons of that

difficult experience you had to go

through. But sometimes you're not so

lucky. Sometimes the black hats win and

they steal the money. Well, luckily for

you, that's not necessarily the end of

the situation. You have options. There

are things you can do to help with the

situation. You can try and negotiate

with the black hats. And make no

mistake, you should always, always,

always try. Although you should also

understand that it will very rarely

work. Okay? [snorts] And the reason why

we want to do that is because it costs

you very little, no matter what the

cost, to try and negotiate, to try and

get the money back. Whatever amount a

black hat may request of it as payment

is probably going to be a good deal for

you if you should take it. And you don't

give up any of your legal rights by

doing so. The black hat may have won

that battle, but if there's ever a

chance to get the money back, you should

take it. Don't get your hopes up about

it. I'm not saying this is going to

work. It usually doesn't work at all,

but you should try. In terms of how you

do that, there are three things that you

should keep in mind. Number one, you

need to open comm's channels. Luckily

for you, the blackhead is almost

certainly watching what you do next.

Typically, they enjoy getting the jump

on you. So, you need to open up a way to

communicate. You can start just with

your public X or or whatever community

infrastructure you're using. Chances are

the black hat will read it. Offer a safe

anonymous communication channel that

they can use without compromising

themselves. If you offer something that

is unsafe, they won't even bother. The

classical example is an onchain message

in a transaction. But you know some

people set up proton mail accounts or or

or other types of you know discrete

email services. You know these things

can also work. The second thing that you

should do is offer a safe return path.

Make it really really easy. I always

recommend people adopt Immunifi safe

harbor which provides a recovery address

at all times. Safe harbor is a new

standard created in conjunction with

seal to make it safer for white hats to

protect projects. But we can also use it

as a as a clear vetted place for black

hats to return money. Why not? So, if

you have Safe Harbor already set up,

this is a really great solution for

offering that and saying, "Hey, look, we

already have a safe place for you to

send the money." And finally, you want

to offer them structured incentives.

What that basically means is you want to

cut them a deal. They keep some amount

of money as long as they send the rest

back to you. And guess what? No matter

what deal they agree to, if they agree

to any deal, you should take it. The

reason that you should take it is

because you still have every legal right

to take action against them if you so

wish.

Okay?

If you think that's really the right

call, you have that right. But if you

can get the money back, it's always a

good deal.

Unfortunately, this same dynamic, the

fact that we as defensive parties cannot

offer realistic legal safe harbor, which

is really the thing that black hats need

in order to negotiate safely. Uh this

thing makes it very very difficult for

these negotiations to proceed because at

the end of the day, governments around

the world can always decide. They don't

care what deals we've made uh with the

onchain community, even if we're hostile

to one another. They'll just override

it, call it a crime, and prosecute them.

That does not actually help the

situation here, right? Law enforcement,

while a necessary part of these types of

experiences, is not typically going to

be all that helpful in dealing with

these situations. Certainly, that's the

case if we look at historical DeFi

hacks. But that doesn't stop us from

trying. We should still try and come to

an arrangement even if the blackheads

have succeeded. Now, when things have

failed, right, when negotiations failed,

you still have even more options. And

some of these you should do right away.

Okay,

high level. First, forensic firms. In my

experience, having had to track down

black hat hackers a number of times now,

the single most useful thing that you

can do is post hack forensics because

hackers typically leave fingerprints on

the crime scene. Most people who can

carry out onchain onchain attacks do not

have the sophistication or the resources

to effectively cover their tracks. And

so if you can find them, you can now

effectively require you can incentivize

them to disclose right what they have

done and return the money. If they do

not, you simply report them to law

enforcement. This is typically how most

of the forensic firms work in the space.

And in my view, when if everything else

has failed and you've lost the money,

you know, forensic firms are probably

your best bet. Number two, exchanges.

You can reach out to the exchanges to

try and freeze all the related

addresses. In my view, you should be

doing this as soon as possible

regardless. Unfortunately though,

getting in touch with the network of

exchanges is really not easy. So, this

takes a hell of a lot of work, but I

recommend it regardless. try and send it

to the appropriate contact at the

various exchanges and see what can be

done to freeze illicit funds. Number

three, legal escalation. You can

threaten legal action against them.

Usually, this is also done as part of

public warnings. And I do not recommend

jumping to this conclusion if the

attacker attackers are smart. Like these

are very intelligent people. They'll

know that you can prosecute them civily

at the very least and very probably

criminally. Okay, they understand this.

You should not jump to threats

if you can afford to. You should not

jump to uh engage

law enforcement if you can afford to

because these people cannot necessarily

solve your problems. And this is a very

counterintuitive thing. It's very

non-obvious. You know, if you let me

I'll give a good example. If you

contact, you know, the the Mexican

federal police, I'm sure there's

brilliant investigators there, and you

say, "Hey, look, a Russian hacker has

stolen my money, they have precisely

zero influence over the situation." But

if you antagonize them in public, you

might lose the only chance that you have

at getting to some type of deal. So, you

should always wait. You should leave

public uh threats of legal action with

law enforcement until the final step.

But it doesn't mean that you can't

engage law enforcement. You can. You can

do that right away if you want. Just

don't tell everybody about it because

it's not going to help your situation.

It's going to make you impossible to

negotiate with. Okay? Leave that to the

last step. At the end of the day, we

live in a big world where increasingly

the the law enforcement and legal

agreement networks are breaking down as

part of geopolitical competition. And we

cannot rely on the traditional legal

systems to solve our problems for us

here. We have to do our best on our own

first and then bring in whatever support

we can. I don't like that state of

affairs. I'm trying to change that state

of affairs every day and make the

industry a little bit safer so that we

don't have to deal with this type of

situation. But that is how it is for

now. And as the operator of a project

that just had to go through a war room,

you need to be prioritizing what's best

for your users and what's best for the

broader community. you need to be

prioritizing keeping people safe. So be

intelligent about it. Okay, very

important. As a good example of this,

I'll go through one little case and uh

then we can move towards Q&A and see

what would be most helpful for you guys.

Uh we have a famous case called

Primitive Finance uh many years ago and

uh this was a great team building a very

novel derivatives product. fascinating

product and the key things that proved

critical there. Well, simply put, you

had a vulnerability that was impossible

to fix. There was no upgradability on

these contracts. There was nothing that

we could do. So, what were we supposed

to do? Well, we ended up finding three

solutions to that issue right here. We

we risked both the theft of all money in

all the derivatives pools. The team in

question was very talented, but these

were very difficult uh types of of of

exploits to deal with. and they risked

losing all their legitimacy. Even though

they built some really wonderful

technology that was meaningful, even

though they were a great team that had a

lot to offer the community, they

couldn't deal easily with the situation

at hand. And we ended up doing three

things that solved that problem. Number

one, they invited trusted experts from

the security community that had worked

with them to help solve this issue.

Specifically, that was us at Immunifi,

yours truly, and our friends over at

DOP, who had done some audits with them.

Together, we were able to provide the

support that they needed to find the

exploits, arrange a fix, and execute it.

Second, that solution ended up needing a

white hat rescue where we had to hack

the funds ourselves from each and every

vulnerable pool. Pretty scary stuff, but

sometimes that's the right call. We did

that. We saved users millions upon

millions upon millions of dollars. Okay,

it was a great day, but that's what we

had to do at the end. That's an option

that you have. every situation will be

unique in particular, but you need to be

prioritizing what's best for users. And

finally, at the end of that whole

process, we didn't hide it. We didn't

pretend that it didn't happen. We could

have we could have lied about it. Like

the situation was such that most people

would not have known that there were

other compromised pools, but we didn't.

Instead, we went forward with

transparency and that ended up

preserving the trust and legitimacy of

primitive for another day. And they

ended up doing great things with that.

This is one example about how these

simple actions that you can take on the

comm side, on the operation side, on the

security side can end up making the big

difference. We executed this entire war

room playbook that I just described to

you here and it worked phenomenally. And

we've all done that now. Both me and

many other many other security experts

in the space have done so dozens and

dozens and dozens of times. So, this

really works you guys. This can really

get to you to the other side. Having

said that, I spend every day trying to

protect the space as much as I can. We

prevented tens of billions of dollars in

hacks. so far, but there's a lot more to

protect and I know there are more

questions on how to do this than what

I've covered here. This is very narrow.

So, thank you very much. And uh why

don't we bring it back and talk a little

bit about what else people need to know

about this and I'll try to answer the

questions as best I can.

>> All right. Yeah. Awesome. Awesome stuff,

Mitchell. Um I know you're a busy guy

being a CEO and all, but feel like you

could also double up as everyone's

favorite professor as well. Uh that was

that was great stuff. Seriously. Um,

yeah, I'll scan through our chat here

and make sure uh we we capture all of

the the questions and feedback. Um, I

see we have some from through our

channels

um and some internal as well. Uh, you

know, why don't we kick things off with

uh this is a good question. you ever,

you know, with all of this with with all

of these war rooms and um, you know,

exploits that are going on, do you ever

consult with teams just to run dry

tabletop exercises involving uh, threat

or exploit response? Um, and and at what

cadence or how often should teams be

conducting these rehearsals internally?

>> Sure. Absolutely. Absolutely. We do so.

Now, unfortunately, it's not super

exciting to do what this is what we

would call a war game, right? or a

tabletop exercise where we go and

simulate it. These things are expensive.

You're going to spend a day or two just

doing this for a situation that hasn't

happened yet. So, you have to be very

proactive and forward thinking to invest

in it, but we do do it. There's

specifically a firm that we love to work

with on this called Shield 3 that has

done incredible war games across the

industry that we would really recommend

to other people. Uh, and this is this is

an approach we very much believe in, but

you have to have the bandwidth for it.

If you do have the bandwidth for it, I

would recommend you do it at least once

a year, if not twice a year.

>> Kind of cool.

>> Yeah.

>> But, you know, generally only the

largest projects get to that stage. When

you have like when you're at a smaller

scale and you have one security lead

who's responsible for every single

problem, you know, one does not easily

get permission to go and do a war game,

>> right? Yeah. And and I mean it might be

expensive. Um, but I mean even I would

say leveraging other builders in the

space um that you that you trust. Um,

and you don't have to give out sensitive

information or anything, but just kind

of doing a sort of a dry run with them

and bouncing ideas off of each other.

Even if you're a solar solo builder in

the space, you know, you you can still

take time to plan out the steps that

would be involved um with actually

triaging a situation like this. and

it'll just save that precious time in

those earlier critical hectic chaotic

moments of a of an exploit. You know,

out of curiosity, because I think you

know the data a lot better than I do,

let's let's envision maybe the top 10%

of onchain protocols that are out there

right now. Like what percentage would

you say are adequately prepared to

smoothly exe execute in line with the

playbook that you've kind of just laid

out here?

>> Sure. And you know your your last

comment on how wise it is even for

solabilities. This is totally true,

right? This is the definition of an

ounce of prevention is worth more than a

pound of cure kind of situation. But you

know, everybody's just like, I just

don't want vitamins. I want a pill.

[laughter]

And you'll see this like when you look

at the numbers, I I would bet that less

than 10% of projects are well prepared

for this. Okay.

>> Interesting. Yeah.

>> And I think 10% is a glorious number

compared to where we used to be. When I

was starting this stuff, I had to deal

with a war room every single week. And

they almost always happen on the

weekends for reasons I'm sure you can

guess if you think about it, right?

Think about it. And you like, "Oh, okay.

So that's who the attackers are for one

thing and why they're doing their

attacks."

>> Yeah.

>> It almost always happened on the

weekends. But now these things have

gotten more rare. We've made a lot more

progress. So 10% is way better things

used to be. I can imagine a world over

the next few years if we work really

hard with you know educational stuff

like this and people really see maybe we

can even get to like 20 or 25 that would

be crazy but everything is possible

right

>> I think so yeah

so um you know with uh with sort of in

line with this theme of lean teams and

you know we're not talking the top 10%

protocols that do have uh engineers that

are internal uh like a a massive team to

bounce ideas off of let's say or either

solo builders or maybe a founder and a

co-founder. Um, and that's pretty much

all running all of the wearing all of

the hats. Um, are there any like

templates or channels that they can like

in terms of communication templates um

things that they resources that are

available anywhere online for

communicating with impacted users,

partners, some cases media um things

like that?

>> Sure. two places to look and we have to

caveat this with the fact that every

hack is unique and every protocol is

unique. So there aren't good cookie

cutter templates for approaching this

because they won't match yours. But we

do have some pretty good things in the

form of postmortems themselves

>> and guides on how to deal with worms. So

for example, this presentation here

that'll be made available to people is a

great uh resource that they should check

out for how they think about it. I

really recommend reading postmortems. We

put out a lot of postmortems at immunifi

which we call bug fix reviews for kind

of almost hacks that we manage to

prevent and many projects do put out

postmortems these days. I recommend you

read that to understand how things are

framed after you read you know five or

10 of those. You kind of understand how

it goes and what wiggle room you have.

And then I really recommend you you you

do some crisis training if you're lucky

enough to have a PR firm or or somewhere

similar who can provide media training

to you. do a little bit of crisis

training uh to understand what it's like

and and how you're going to be squeezed

and and understand how they respond to

things because these

are you know hacks onchain hacks are

unique to us and our situation but

crises are not okay we know how to

communicate and manage these difficult

situations your media experts have

already gone through this I mean but I I

was prepared for this because I I had

already managed dozens of crises before

I started running you know defi war

rooms like it was all old hat to me at

the end of the day. And you already know

people who are experts in this. You just

have to ask them for advice and for

training and they can come and help you

with that and they can come and show you

how to frame that messaging. And maybe

if you're lucky and you bully them

enough, they can even come into the war

room and do that messaging for you. And

that is a much better solution than

relying on a template. And speaking of

which, right, I'm being prepared. Like

the best preparation is just to know who

you're going to call and have that set

up in advance. Yeah. Yeah. That even in

those moments of panic, your brain's not

not thinking. It's thinking about a

thousand different things and even

anything that can simplify the immediate

response to that is gives you great

momentum that is very important uh to

handle things uh the right way. So, all

right. Uh and then I think we'll have

time for one last multiplepart question.

This one I admit I I I wrote this one.

So, uh, this one's for me. I'm just

curious your take, uh, and what your

answer is. Again, kind of databased. Um,

but what's your best guess and the in

terms of the percentage breakdown of,

uh, you know, white hats, black hats, or

gray hat actors in the space? Um, and do

you think it's common for uh, you know,

a hacker to change the color of their

hat? Then I have one followup as well.

>> That's a scary question, Dan.

You know, we were trying to derisk it,

make people feel a little bit more

comfortable with the situation. Now,

we're going to blow it back up and spook

them all. That's how it's going to go.

Look, I think that crypto has an

incredible community of white hats.

Like, really, the best people in crypto

are the White Hat security community in

my view. I've been lucky enough to spend

years with them. They're by far my

favorite people in crypto. I don't know,

like I think what we do in the industry

is incredible, but really these people

are are are kind of shining lights in so

many ways. And unfortunately on the

downside, while we do have these literal

Batman's among us, I think they probably

make up 10 to 20% of the hacker

community.

>> Okay,

>> sounds about right.

>> Right now, I don't think that the rest

of it is black hats. I think, you know,

black hats are probably sub 10%.

>> An even smaller.

>> Makes me feel better.

>> Right. But the challenge is almost

everybody in between is what we call a

gray hat, which means that, you know,

they're going to have their own motives

and they're going to do what is in

typically the primary financial

self-interest. And the thing with these,

you know, hacker types in my experience,

and I'm sure, you know, Dan, you've kind

of seen the same is these are very smart

people. They will all have kind of their

own codes of honor or ethics, right? But

the challenge is whatever that code is,

it may not be the same as yours and it

can be very very gray. And a lot of

these guys are very very gray, right? So

do I see them changing the color of

their hat? Yeah, I've seen it many times

and that's the scary part, right? That's

why, you know, atmunifi we did so much

work on bug bounties in in years ago to

make those multi-million. It's to deal

with this part. Okay. Let's always make

sure that the incentives make people go

white by default and not black.

>> Yeah. Yeah. I guess that you I mean you

kind of answered my my final followup

there which is uh you know are there

situations that you've seen where you

know someone probably had black hat

intentions at the start but then had

that remorse afterwards or the fear of

consequences that

>> after having done so. Yeah.

>> Many times usually after the fear of

consequences is you know nudged in their

way often via email to their personal

email. That's usually when they're like,

"Oh, suddenly I'm very remorseful about

my life choices,

>> right? Found me.

>> Yeah. Yeah. Watch your back. Look over

your shoulder. Build your own paranoia

because Yeah. I mean, it's

>> it it's it's not going to be worth the

the hassle like you said throughout this

presentation. Uh it's and it's feels a

lot a lot more rewarding, I would

imagine, to to be be on the good side of

things. So, um, and feel like you're

actually doing the community, you know,

a greater good. Uh, so yeah, I mean,

this was this was an awesome chat and

and I really appreciate you taking your

time out of your busy day to educate the

rest of the builders in the space um on

this super important topic. Um, you

know, we have another session coming up

in two weeks, everybody. Um, we're going

to be talking about setting up bug

bounty programs and we'll have our

special guest Cantina join us for that

session. And so make sure you tune in in

two weeks from now on the 19th. Um but

thank you so much Mitchell and the

entire Immunifi team uh for for joining

us on today's session and educating the

community.

>> U later

>> be well.

>> All right. Thanks guys.