Loading video player...
Hey everybody, my name is Dan Johnson
and I'm on the protocol security team
here at Coinbase. I'm joined here today
with Mitchell Amodore, the CEO of
Immunifi, a web 3 security company
that's been dedicated to protecting the
onchain ecosystem since 2020. Today
we're serving up our 11th episode of our
ongoing Coinbase security series where
every two weeks our team gets together
and we connect with the great builders
in the web 3 community and discuss a
topic of blockchain security and how to
build safely in the space. As is the
case with each of these sessions, please
note the disclaimer that's at the bottom
of the screen. Uh, and with that, we've
got a great session for you today. So,
I'll let Mitchell take it away.
>> Okay. Okay. Exciting stuff. We're going
to talk about something terrifying
today. No pressure. Doesn't mean you
should run away. It means it's going to
be even more exciting, but just heads up
on that. It is. First things, hear me
loud and clear. We're ready to go, Dan.
>> Yes, sir. Sounds great. And uh you can
go ahead and test uh sharing your screen
and getting the presentation up and
running as well.
>> Favorite part the the difficult
>> difficulties right right here.
>> I'm scared already. But you're going to
you're going to make the uh you're going
to make this topic less scary for
everyone by by educating us all.
>> Only a little bit. I suppose it's all
contextual at the end of the day.
>> So, [laughter]
you know, we're going to make some parts
less scary. We're going to give you
agency. We're going to give you some
amount of control, but we're also going
to show you the horrors of reality and
what can go wrong. And I think that's
just as important here. Now, uh to kick
off, this is going to be an important
presentation. If you are a builder on
chain, this could be the presentation,
the content that you consume that saves
your project from total destruction. And
if you're a builder, you probably have
all of your network plugged into that.
This could be the thing that saves
everything. I'm going to show you how to
survive the hack experience. It's not
going to be easy. It's not going to be
quick, but I'm going to give you the
tools that have been tested over
hundreds of incidents in the industry,
many of which I have seen and
participated in and help save firsthand
to stay safe when you are targeted
successfully for exploitation by
blackout hackers, no matter where
they're from. A little bit about me for
context. As Dan said, I'm the founder
and CEO of Munifi. We are the leading
security services platform in crypto. We
protect the vast majority of the
industry around uh 180 or 200 billion
worth of assets and contracts and beyond
today. Uh we have one goal which is to
keep the industry safe. That's why we're
made in the first place and we're still
doing that to this very day. We made
incredible progress in that time and a
lot of that progress via on the you know
the bug bounty side the technical side
security operations opsec infrastructure
a lot of those experiences right that we
those lessons that we were able to learn
come from surviving and seeing how bad
things can get firsthand okay these war
rooms are the key to that but just
because something scary is going to
happen doesn't mean it has to be a fatal
experience and we're going to show you
how So, for starters, the first thing
that I want all of you to understand is
that war rooms are inevitable. If you're
building something big, you're building
something exciting, you're building
something meaningful on chain,
contributing to this whole new world
that we have going for us, then you're
going to hit some problems and they're
going to be very, very scary and
eventually you're going to make a
mistake. You're going to get pulled into
some incident or another. If you're
lucky, the damage will be small and
contained. If you're unlucky, the damage
can be total and fatal. We've seen many
such cases. A little example of what
we're looking at here, right? On the one
side of small and contained, maybe you
have, you know, certain backend
infrastructure gets compromised allowing
users to spoof or get spoofed or or
man-in-the-middle attacked in various
ways. And if you can just find that
piece of infrastructure that only
affects a certain amount of your users,
you stem the bleeding. Great. You're
safe. You made it through. Okay? Or you
could have your front end hijack. Great.
If you move fast enough, you know, most
of your users may not be affected by
that. On the other hand, you have
incidents like compromising of treasury.
Okay, so a good example of that is
something like the buy bet hack. This
has occurred on so many exchanges with
hot wallet compromise. You could lose
all your assets overnight. And of course
in in DeFi, the most exciting expression
of the onchain economy, this can become
even more total because we typically
pool funds in various ways. Not always.
There are there are a variety of
infrastructures, but you may be pooling
your funds in whatever architecture of
the contracts that you've made. And so
if someone can compromise that pool,
they might be able to steal all of the
user funds. And we've seen a number of
such cases. We've defended against a
number of such cases. But the important
thing to keep in mind again is that
these war rooms, they can't be avoided.
You're no matter how good you are,
you're going to make mistakes. Everybody
does. The attackers have infinite time
and resources to attack and you only
have so much time to defend. So the
solution is not to avoid war rooms. The
solution is to be prepared and to know
how to carry it out when it inevitably
comes and do your best to prevent it in
the interm. Now when you enter a war
room, you have two simple objectives. I
make this real clear for you guys. Your
first goal is to stop further losses.
Wherever think of yourself like a
doctor. You have a patient. He comes
into the ER, what do you do first? You
stem the bleeding, right? You need to
stop the thing that will directly
contribute to loss of life or problems.
You need to stop further losses. You
need to stop the bleeding of users. It
is very common for attacks to be carried
out over multiple blocks or to be
subject to a variety of particular
conditions and not be totally executable
in a single block or in a single
transaction. In such cases, you can have
a chance to prevent the majority of
damage. In most of the major hacks that
we've seen, this has been the case to
greater or lesser degrees. Okay, this is
especially so the case for for hacks
that are not strictly onchain hacks. So,
you need to find the cause and you need
to apply that tourniquet. You need to
arrest it. Your second goal is to
preserve user trust. And this is also
critical because even if you
successfully stop the bleeding and save
the users's funds. If you fumble if you
fumble the post hack communication,
people may trust you less and you may
end up destroyed anyways.
Congratulations. You'll have saved the
users, but you will have been burned.
You will have been scapegoed because
your your inability to maintain
legitimacy despite your good actions.
And so we need to make sure that we
maintain user trust. And this is a
matter of communication, messaging, and
of course, the conduct that we take in
the war. I'm going to show you how to do
that, too. But starting from the top,
how do we stem the bleeding? What do we
do? Step one, we confirm the nature of
the vulnerability. We confirm the nature
of the attack. How did this happen the
way it did? We identify the root cause
of it. How did this happen? Can it
happen again? Very often, uh, an exploit
is the tip of the iceberg for how it can
be exploited again and again and again.
So a good example of that is when you
have projects with multiple pools uh
which is a great many DeFi protocols for
example if they can exploit one pool
they may be able to exploit many other
pools and it's not a given that the
attacker will have done the research to
identify all of those other vulnerable
tool uh pools. In fact most attacks
historically have been different
versions of driveby attacks. The the
attackers cannot trivially check all
these things. So you need to take action
to find that exploit and see what you
can do to to stop the bleeding from
there. And that gets us to the next
step, right? Which is to pause all
affected contracts. And very often if
you can't tell what that means, like
which contracts are affected, if you're
just not sure, you're having struggles
reverse engineering the exploit and
seeing what it really is, then you
should be pausing all contracts. You
don't need to take the risk. Pause them
all. Save the users. Most people won't
even notice to be quite frank. It's not
like they're watching you all the time,
but they will be thankful after the fact
and you will buy yourself precious hours
to identify what really went wrong. The
third step that you need to take is you
need to lock down all permissions, all
admin access whatsoever. need to know
exactly how the state can change from
here and anything
you need to eliminate all possible you
know gray zones or gaps that you might
have anything that that could have been
the surprise and make sure all your
other infrastructure is taken care of as
well because you don't necessarily know
how far the breach goes. Historically,
most onchain hanachics were discrete
events in the sense that they happen um
just onchain. But what we've begun to
see over the last few years with the
rise of more state sponsored hacking
groups, most famously the Lazarus group
with North Korea, is that they are
compromising a variety of pieces of
infrastructure on their road to ultimate
theft. Okay. And this is more typical of
traditional uh advanced persistent
threat behavior in web 2 and the broader
internet. So you got to lock everything
down. Don't take any chances. You don't
know who you're dealing with. They could
be very scary people. They could be
wellresourced people. They probably are.
Be maximally paranoid. Lock it all down.
With this, your situation will be
paused. Hopefully, you will have frozen
the state of your contracts, of your
protocols as much as you possibly can,
and you will be ready for the next step.
Hopefully, this is you stemming the
bleeding when it comes down to actual
thefts. [snorts]
And if you can take nothing away from
this presentation, then take these two
things, first objectives, stop further
losses and preserve user trust. Second,
these three basic steps. Find the
exploit, pause everything, lock down all
your infrastructure. This gives you time
to live another day. Now, all of that is
easier said than done. So, now we're
going to talk a little bit about how
that works. We're going to talk about
how you should be constructing that war
room to get all that done. And at the
highest level, there are three basic
roles in my view that every war room
should consist of. These roles, these
functions
which can be performed by one or the
same people but are best performed each
by individuals is the operations lead,
the security analyst and the
communications lead. Let's touch a
little bit about what each one of them
does. First is the operations lead. Your
job is to command and to coordinate
functions. When I'm in war rooms, this
is typically the job I get to do. It's
also one of the most painful and most
difficult jobs and you get the
unpleasant responsibility of making some
very difficult very hard choices very
often. In this function, your goal is to
relieve all unnecessary thought and work
from your compatriots so that they can
do what they do best while you organize
overall sequencing and managing of the
remediation process. Okay.
Your job here is to set up all the
workflows, set up tracking,
coordination communication decisions
and make the best calls that you can,
often as a group at every single point.
The ops lead may be a founder or CEO,
very often is not, but is someone who
will defer ultimately the the the most
difficult decisions to uh a more formal
business leader who will make the final
call. But the ops lead is responsible
for organizing the work in the leadup to
that. Set in another way, your job is to
keep the entire context of the hack and
the war room in your mind at all times.
And what this takes form in practice is
you end up creating the documentation.
You end up writing and keeping track of
what's going on. You keep track of all
the participants in the war room. You're
identifying and directing them to all
the tasks that they should be doing.
You're setting the sync cycles, aka how
much time you should allow to pass
before you come and check in, which I
would typically recommend about 15 to 30
minutes. Every 15 to 30 minutes, you
want to check back in with your teams
and see how they're doing on their
different problems. And you get the
unfortunate job of time boxing the
decision, the decisions, and being like,
"Okay, well, how long do we have to make
this call? And what are the trade-offs
here?" Sometimes you will be deferring
that final decision to the formal
business leader. Sometimes you're going
to have to make the call yourself. But
again, your job is to have the full
context, the command and coordination
function of the entire war room. And
it's really critical that you get this
right. Okay? Because in a war room, what
happens to everybody is they freeze.
They get stuck. They get worried. They
panic. They say, "Can I really do this?
What am I supposed to do? My network, my
net worth is going up in smoke. Am I
going to go to jail? Did I make a
mistake? Is this my fault? Am I guilty?"
And as an obsolete, you have to put an
end to all that. There's no room for
such sentimentality. You need to go and
fix the problem. You have to provide
leadership, structure, and clarity to
the team members and inspire them to do
what needs to be done. In the vast
majority of war rooms, the offset may be
the only person who is experienced, who
has done it before. Most people
obviously have never done this before.
And you need to provide them the
certainty and the conviction that they
can go forward and succeed and that
there's going to be a future on the
other side of this. So that's the
obsolete function. This role can be
folded in to the others but ideally
stands alone. The second function is the
security analyst. The security analyst
has the critical and difficult job of
identifying exactly what the attack
vector was and what the exploit was and
how big the scope of damage and risk is
so that they can prepare a mitigation
path and ultimately apply that and block
any other exploitations. This is the
person who is doing the hard technical
work to fix the situation, right? If if
the ops lead is like um a doctor
coordinating, you know, a hospital
situation, then the security analyst is
like a surgeon who's got the difficult
job of actually doing the cutting,
making sure he's found exactly the right
problem and addressing the wounds and
and suturing them up all properly. Now,
you might think, oh, you know, security
analyst, isn't that the most important
part? Why don't we just fold the the
offset role into that? No, we don't do
that because the security analyst role
is extremely cognitively intense and
they are in a race against time to
identify the nature of the attack vector
and to mitigate it before it can be
exploited further. And no matter what
kind of hack you're in, you never know
if there is more damage in the
beginning. You have to suspect that
you're at the beginning of a process,
not at the end, that things can get
worse. So you need to take your security
analysts that could ideally it's uh one
or or or two people working as a team
should be a small group and you need to
strip everything that's not critical
away from their mind. Anything they
don't have to think about they shouldn't
be thinking about. Anything they don't
have to do, they shouldn't be doing.
They got one job. Find the source of the
bleeding and stem it as rapidly as they
can. That's their one job. Now,
typically how this works in practice is
after you come together in a say a chat
room of some kind with your ops lead and
your comms lead and your security
analyst and there might be one or
multiple of each of those roles if
you're lucky. The security analyst will
go off on their own into a private chat
to go and do this work and trade ideas
directly. You want to allow them to go
and do their own thing because focus is
going to enhance the quality of the work
and the speed at which they're going to
get it done. If at all possible, you
don't want to remain in the same group
chat where you can hear everybody else
speaking. Instead, the security analyst
should be regrouping with the overall
war room every 15 to 30 minutes. 30
minutes is a great cadence in my view.
Uh to bring back their findings and
their recommendations on what to do
next. Obviously, if they have some type
of breakthrough that it makes sense to
do that even sooner, like you do
whatever makes sense. But you want to
give space to focus, right?
Communication in a war room is
expensive. It costs time and attention
and stress and you only want to do as
much as you absolutely need to get to
the right outcome. It is fine to have
multiple security analysts. In fact,
generally uh I think a a small team of
two to three is is is good. Two is
probably best providing people know the
protocol well.
one is good, but getting extra eyes on
that is better. It's not a case of, you
know, more is better. You keep adding
more people, you have more people to
coordinate, things get more complex,
people disagree about things like you
don't want an infinitely large team. Two
to three is probably uh where it's ideal
to sit. But this is the security analyst
role. If [snorts] you're unsure of who
to bring on, then you, you know, I
recommend if you don't have a security
analyst in a pinch, you go and you call
your security providers and see if they
might help. This is how we have done so
many wars at Immunifi. We have done I've
personally done dozens and dozens and
doz like maybe somewhere at least the
high dozens of war rooms at this point
and that's because our customers are
calling us in. They have a problem. They
don't know how to deal with it. Nobody
else they know knows how to deal with
it. They trust us. We come in there. We
manage the situation with them to come
and deal with it as best we can. You can
also call Seal 911. They have a great uh
free service that you can use that they
also get involved with war rooms with
experts from around the industry. You
can also call an auditor or similar such
player or you can just pick the best dev
on your team. Like you do you. You're
going to have a variety of options
available to you. Uh I suggest that you
take all of them that you have on
access. If you have access to the
resource, leverage it because now is the
moment to cash in on that favor. The
third function is trust management and
that means your comms lead, your
communications leader who is responsible
for coordinating all the messaging that
needs to go out. And believe me, war
rooms have tons of messaging. This is
rapid fire PR that you have to get done
and avoid making any mistakes because
one wrong mistake when a user is in a
state of fear and panic is a reputation
forever lost. and TVL that will never be
returning to your contracts. So you need
to focus very very finely, very very
specifically on sending the right
message and inspiring confidence and
trust in your users that they can trust
you to handle these situations. Okay,
war rooms are inevitable because hacks
are inevitable and everybody on chain
knows that. So your users need to
believe that you can handle these
situations and by the end of this
presentation you will know how to. Now,
the comm's lead is best suited to be
someone else. They can be technical, but
typically writing is a specialized skill
in my experience, but they do need to be
focused. They need to be quick on their
feet. They need to understand the
project as well as the as well as
possible. And their job here is to
communicate exactly what the community
needs to know at every step. In
practice, that typically leads to calm,
clear factual updates, where you avoid
hyperbole, but also where you avoid uh
how do I say making things rosier than
they look. You don't want to speculate.
You don't want to project positivity
prematurely. Like you can save that
stuff for the end in the moment while
these worms are happening. And worms can
be wild. I mean it can be a few it can
be you know 15 or 20 minutes for
exceptionally short cases but it can
also be 24 or 48 hours you know I I've
done done some war rooms that end up
being 3 or 4 days because we couldn't
really manage the issues over the time
frame which is very painful uh I've done
several such cases like it can it can
get long you don't want to say anything
that would cause users to lose trust in
you later and so intuitively the best
approach the one that inspires the most
confidence is to treat them like an
adult. Clarity,
calmness, but at the same time, you
know, confidence, conviction, and don't
say what you don't need to say. People
will have the temptation to try and make
people feel better by making claims that
they should not. Don't do it. It's a
trap. Also, cadence. Very important. You
should be communicating at regular
intervals. Other than the hack itself,
one of the most terrifying things is
when a project goes silent on you after
all these events. You don't know is that
because they freeze. It's a hack. You
also have to suspect was it an insider
attack? Are they rugging you? Right?
There's lots of reasons to be worried
when a project goes silent. So don't do
it. Consley needs to come in there,
communicate clearly, communicate
regularly and inspire confidence that
they can handle the situation.
Typically, if they need to do their
work, they will also go into a separate
chat to go and coordinate that. There
could be one or two people. You might
also bring in a PR firm if you have um
access to partners there. Um you will
often work with the ops lead and the
security analyst to really make sure
that the framing and the messaging is
correct from a technical perspective and
from an operational perspective.
But otherwise you're working somewhat
similar to the analyst. Unlike the
analyst whose job is really to to
investigate how an exploit occurred, a
comms lead role is much more
forwardlooking. So as comms lead you
should be preparing for multiple
different timelines not just one
situation but multiple paths forward for
example we stemmed the bleeding
everybody's safe right all users are
fine for example we couldn't stem the
bleeding everybody is unsafe here's what
you have to do next for example it was
never a hack at all we found the cause
of it was a user error or something
great that's dealt with as comms lead
you need to prepare for everything in
order of the severity and the danger uh
of the scenario. So you end up with
pages and pages of documentation in that
process. In parallel, there's another
thing the comms lead should do. The
comms lead should work with the ops lead
to document what is happening in the war
room as the events proceed. Creating a
living timeline of events because the
more the comms lead can take over the
documentation duty of the war room which
is going to be needed. All this
information about what happened, who was
in the war room, what exactly they did,
why they made the choices they did, all
that's going to come out in the
postmortem, which can be an internal
doc, but it's best also an external
documentation that is shared with the
the broader community so that everybody
can learn how to stay safe. All this
information should be recorded by the
comm's lead while the events are
ongoing. Yes, you can have the
operations lead do it, too, but if you
do that, you're going to be distracting
them from the most important choices.
So, wherever possible, the comm's lead
should take point on that. Again, in
general, every role, every function
should be done by a single person and
not merged if you have the option.
That's the setup of a war room. Those
are the three main roles you need. Those
are the three things that they're going
to be doing. That's how they're going to
be driving the process. And we heard a
little bit about the sequencing of
events. Now, the next thing now that we
have this context, the important thing
for you to remember to keep in mind is
to remain calm. Remain very calm. Like I
said in the beginning, people mess up,
people freeze, people become afraid.
Your ability to remain calm lets you
make good decisions about the matter.
But more than that, your calmness is
contagious. Your reason
transmits. The calmer you are, the more
centered, the more focused, the more
centered, focused, calmness you're going
to bring to everybody else in the war
room and the better they're going to
perform. So whatever you do, remain
calm. Very important. Now, in terms of
how this goes, we walked through a bit
of the functions. All these things are
actually happening in parallel. Now, you
see why we have an operations lead in
the first place because we're not doing
these things one at a time. No, no, no,
no, no. We're doing all of these things
at the same time. And we're in a race
against time while we do it. Even more
difficult, even more stressful. We're in
a race against time to identify the
source of the breach of the source of
the attack to block it and then to
preserve the trust of our users and the
community at large. So while you the ops
lead or whoever you may be is preparing
the decision flow, mapping out how the
the sequence of events could go and what
are the various pathways and how you're
going to respond at those pathways to
make the calls. The security analyst
needs to be doing the forensic work,
understanding exactly how a fix and a
mitigation can be implemented and
preparing everything needed to execute
that fix. And the comm's lead for their
part, they need to be preparing the
messaging in parallel for the various
scenarios. You don't know how much time
you're going to have to execute this.
You don't know how much time, you know,
emotional contagion uh can take from you
and how quickly it can spread. And
again, remember, it's not just the hack
that can kill you. It's also the loss of
user trust and legitimacy that can do
the job. So you need to be ready for it.
So everything needs to be done in
parallel as fast as possible. And the
ops fleet has the venerable duty of
coordinating all this and keeping every
decision the entire state of the war
room in their head at all possible times
so that they can maximize the speed of
execution. Having said that, once you're
done all that, you can now proceed with
postmortem. Okay. Postmortems are
typically done 24 to 48 hours after this
events are are wrapped up and the fix is
applied. And the reason for that is
because these war rooms are exhausting.
It is not usually feasible to do the
postmortem right away afterwards. People
are too tired. People are too stressed.
You know, it's just too much to deal
with. But the next day, you want to get
this postmortem out as soon as you
possibly can. The users are waiting for
them. They want to see that you handled
the situation correctly. They want to
know that they should feel safe. They
want to know that their money has not
all been stolen from them and your
postmortem is the way you do that. Four
key components. Root cause of the
exploit, the attack and postmortem
timeline of what exactly happened, the
fix that was discovered and applied, and
what actions will be taken further into
the future to ensure that this type of
attack never happens again. Very simple,
little bit stressful to write, I'll
admit. But this is the outline for every
postmortem that you should do. If you do
this, you're not just going to be
helping your community members, keeping
them engaged, keeping them loyal,
keeping your your own project legitimate
in their eyes and worthy of trust.
You're also going to be supporting the
entire future security community on base
and in the broader blockchain ecosystem
as we all learn the lessons of that
difficult experience you had to go
through. But sometimes you're not so
lucky. Sometimes the black hats win and
they steal the money. Well, luckily for
you, that's not necessarily the end of
the situation. You have options. There
are things you can do to help with the
situation. You can try and negotiate
with the black hats. And make no
mistake, you should always, always,
always try. Although you should also
understand that it will very rarely
work. Okay? [snorts] And the reason why
we want to do that is because it costs
you very little, no matter what the
cost, to try and negotiate, to try and
get the money back. Whatever amount a
black hat may request of it as payment
is probably going to be a good deal for
you if you should take it. And you don't
give up any of your legal rights by
doing so. The black hat may have won
that battle, but if there's ever a
chance to get the money back, you should
take it. Don't get your hopes up about
it. I'm not saying this is going to
work. It usually doesn't work at all,
but you should try. In terms of how you
do that, there are three things that you
should keep in mind. Number one, you
need to open comm's channels. Luckily
for you, the blackhead is almost
certainly watching what you do next.
Typically, they enjoy getting the jump
on you. So, you need to open up a way to
communicate. You can start just with
your public X or or whatever community
infrastructure you're using. Chances are
the black hat will read it. Offer a safe
anonymous communication channel that
they can use without compromising
themselves. If you offer something that
is unsafe, they won't even bother. The
classical example is an onchain message
in a transaction. But you know some
people set up proton mail accounts or or
or other types of you know discrete
email services. You know these things
can also work. The second thing that you
should do is offer a safe return path.
Make it really really easy. I always
recommend people adopt Immunifi safe
harbor which provides a recovery address
at all times. Safe harbor is a new
standard created in conjunction with
seal to make it safer for white hats to
protect projects. But we can also use it
as a as a clear vetted place for black
hats to return money. Why not? So, if
you have Safe Harbor already set up,
this is a really great solution for
offering that and saying, "Hey, look, we
already have a safe place for you to
send the money." And finally, you want
to offer them structured incentives.
What that basically means is you want to
cut them a deal. They keep some amount
of money as long as they send the rest
back to you. And guess what? No matter
what deal they agree to, if they agree
to any deal, you should take it. The
reason that you should take it is
because you still have every legal right
to take action against them if you so
wish.
Okay?
If you think that's really the right
call, you have that right. But if you
can get the money back, it's always a
good deal.
Unfortunately, this same dynamic, the
fact that we as defensive parties cannot
offer realistic legal safe harbor, which
is really the thing that black hats need
in order to negotiate safely. Uh this
thing makes it very very difficult for
these negotiations to proceed because at
the end of the day, governments around
the world can always decide. They don't
care what deals we've made uh with the
onchain community, even if we're hostile
to one another. They'll just override
it, call it a crime, and prosecute them.
That does not actually help the
situation here, right? Law enforcement,
while a necessary part of these types of
experiences, is not typically going to
be all that helpful in dealing with
these situations. Certainly, that's the
case if we look at historical DeFi
hacks. But that doesn't stop us from
trying. We should still try and come to
an arrangement even if the blackheads
have succeeded. Now, when things have
failed, right, when negotiations failed,
you still have even more options. And
some of these you should do right away.
Okay,
high level. First, forensic firms. In my
experience, having had to track down
black hat hackers a number of times now,
the single most useful thing that you
can do is post hack forensics because
hackers typically leave fingerprints on
the crime scene. Most people who can
carry out onchain onchain attacks do not
have the sophistication or the resources
to effectively cover their tracks. And
so if you can find them, you can now
effectively require you can incentivize
them to disclose right what they have
done and return the money. If they do
not, you simply report them to law
enforcement. This is typically how most
of the forensic firms work in the space.
And in my view, when if everything else
has failed and you've lost the money,
you know, forensic firms are probably
your best bet. Number two, exchanges.
You can reach out to the exchanges to
try and freeze all the related
addresses. In my view, you should be
doing this as soon as possible
regardless. Unfortunately though,
getting in touch with the network of
exchanges is really not easy. So, this
takes a hell of a lot of work, but I
recommend it regardless. try and send it
to the appropriate contact at the
various exchanges and see what can be
done to freeze illicit funds. Number
three, legal escalation. You can
threaten legal action against them.
Usually, this is also done as part of
public warnings. And I do not recommend
jumping to this conclusion if the
attacker attackers are smart. Like these
are very intelligent people. They'll
know that you can prosecute them civily
at the very least and very probably
criminally. Okay, they understand this.
You should not jump to threats
if you can afford to. You should not
jump to uh engage
law enforcement if you can afford to
because these people cannot necessarily
solve your problems. And this is a very
counterintuitive thing. It's very
non-obvious. You know, if you let me
I'll give a good example. If you
contact, you know, the the Mexican
federal police, I'm sure there's
brilliant investigators there, and you
say, "Hey, look, a Russian hacker has
stolen my money, they have precisely
zero influence over the situation." But
if you antagonize them in public, you
might lose the only chance that you have
at getting to some type of deal. So, you
should always wait. You should leave
public uh threats of legal action with
law enforcement until the final step.
But it doesn't mean that you can't
engage law enforcement. You can. You can
do that right away if you want. Just
don't tell everybody about it because
it's not going to help your situation.
It's going to make you impossible to
negotiate with. Okay? Leave that to the
last step. At the end of the day, we
live in a big world where increasingly
the the law enforcement and legal
agreement networks are breaking down as
part of geopolitical competition. And we
cannot rely on the traditional legal
systems to solve our problems for us
here. We have to do our best on our own
first and then bring in whatever support
we can. I don't like that state of
affairs. I'm trying to change that state
of affairs every day and make the
industry a little bit safer so that we
don't have to deal with this type of
situation. But that is how it is for
now. And as the operator of a project
that just had to go through a war room,
you need to be prioritizing what's best
for your users and what's best for the
broader community. you need to be
prioritizing keeping people safe. So be
intelligent about it. Okay, very
important. As a good example of this,
I'll go through one little case and uh
then we can move towards Q&A and see
what would be most helpful for you guys.
Uh we have a famous case called
Primitive Finance uh many years ago and
uh this was a great team building a very
novel derivatives product. fascinating
product and the key things that proved
critical there. Well, simply put, you
had a vulnerability that was impossible
to fix. There was no upgradability on
these contracts. There was nothing that
we could do. So, what were we supposed
to do? Well, we ended up finding three
solutions to that issue right here. We
we risked both the theft of all money in
all the derivatives pools. The team in
question was very talented, but these
were very difficult uh types of of of
exploits to deal with. and they risked
losing all their legitimacy. Even though
they built some really wonderful
technology that was meaningful, even
though they were a great team that had a
lot to offer the community, they
couldn't deal easily with the situation
at hand. And we ended up doing three
things that solved that problem. Number
one, they invited trusted experts from
the security community that had worked
with them to help solve this issue.
Specifically, that was us at Immunifi,
yours truly, and our friends over at
DOP, who had done some audits with them.
Together, we were able to provide the
support that they needed to find the
exploits, arrange a fix, and execute it.
Second, that solution ended up needing a
white hat rescue where we had to hack
the funds ourselves from each and every
vulnerable pool. Pretty scary stuff, but
sometimes that's the right call. We did
that. We saved users millions upon
millions upon millions of dollars. Okay,
it was a great day, but that's what we
had to do at the end. That's an option
that you have. every situation will be
unique in particular, but you need to be
prioritizing what's best for users. And
finally, at the end of that whole
process, we didn't hide it. We didn't
pretend that it didn't happen. We could
have we could have lied about it. Like
the situation was such that most people
would not have known that there were
other compromised pools, but we didn't.
Instead, we went forward with
transparency and that ended up
preserving the trust and legitimacy of
primitive for another day. And they
ended up doing great things with that.
This is one example about how these
simple actions that you can take on the
comm side, on the operation side, on the
security side can end up making the big
difference. We executed this entire war
room playbook that I just described to
you here and it worked phenomenally. And
we've all done that now. Both me and
many other many other security experts
in the space have done so dozens and
dozens and dozens of times. So, this
really works you guys. This can really
get to you to the other side. Having
said that, I spend every day trying to
protect the space as much as I can. We
prevented tens of billions of dollars in
hacks. so far, but there's a lot more to
protect and I know there are more
questions on how to do this than what
I've covered here. This is very narrow.
So, thank you very much. And uh why
don't we bring it back and talk a little
bit about what else people need to know
about this and I'll try to answer the
questions as best I can.
>> All right. Yeah. Awesome. Awesome stuff,
Mitchell. Um I know you're a busy guy
being a CEO and all, but feel like you
could also double up as everyone's
favorite professor as well. Uh that was
that was great stuff. Seriously. Um,
yeah, I'll scan through our chat here
and make sure uh we we capture all of
the the questions and feedback. Um, I
see we have some from through our
channels
um and some internal as well. Uh, you
know, why don't we kick things off with
uh this is a good question. you ever,
you know, with all of this with with all
of these war rooms and um, you know,
exploits that are going on, do you ever
consult with teams just to run dry
tabletop exercises involving uh, threat
or exploit response? Um, and and at what
cadence or how often should teams be
conducting these rehearsals internally?
>> Sure. Absolutely. Absolutely. We do so.
Now, unfortunately, it's not super
exciting to do what this is what we
would call a war game, right? or a
tabletop exercise where we go and
simulate it. These things are expensive.
You're going to spend a day or two just
doing this for a situation that hasn't
happened yet. So, you have to be very
proactive and forward thinking to invest
in it, but we do do it. There's
specifically a firm that we love to work
with on this called Shield 3 that has
done incredible war games across the
industry that we would really recommend
to other people. Uh, and this is this is
an approach we very much believe in, but
you have to have the bandwidth for it.
If you do have the bandwidth for it, I
would recommend you do it at least once
a year, if not twice a year.
>> Kind of cool.
>> Yeah.
>> But, you know, generally only the
largest projects get to that stage. When
you have like when you're at a smaller
scale and you have one security lead
who's responsible for every single
problem, you know, one does not easily
get permission to go and do a war game,
>> right? Yeah. And and I mean it might be
expensive. Um, but I mean even I would
say leveraging other builders in the
space um that you that you trust. Um,
and you don't have to give out sensitive
information or anything, but just kind
of doing a sort of a dry run with them
and bouncing ideas off of each other.
Even if you're a solar solo builder in
the space, you know, you you can still
take time to plan out the steps that
would be involved um with actually
triaging a situation like this. and
it'll just save that precious time in
those earlier critical hectic chaotic
moments of a of an exploit. You know,
out of curiosity, because I think you
know the data a lot better than I do,
let's let's envision maybe the top 10%
of onchain protocols that are out there
right now. Like what percentage would
you say are adequately prepared to
smoothly exe execute in line with the
playbook that you've kind of just laid
out here?
>> Sure. And you know your your last
comment on how wise it is even for
solabilities. This is totally true,
right? This is the definition of an
ounce of prevention is worth more than a
pound of cure kind of situation. But you
know, everybody's just like, I just
don't want vitamins. I want a pill.
[laughter]
And you'll see this like when you look
at the numbers, I I would bet that less
than 10% of projects are well prepared
for this. Okay.
>> Interesting. Yeah.
>> And I think 10% is a glorious number
compared to where we used to be. When I
was starting this stuff, I had to deal
with a war room every single week. And
they almost always happen on the
weekends for reasons I'm sure you can
guess if you think about it, right?
Think about it. And you like, "Oh, okay.
So that's who the attackers are for one
thing and why they're doing their
attacks."
>> Yeah.
>> It almost always happened on the
weekends. But now these things have
gotten more rare. We've made a lot more
progress. So 10% is way better things
used to be. I can imagine a world over
the next few years if we work really
hard with you know educational stuff
like this and people really see maybe we
can even get to like 20 or 25 that would
be crazy but everything is possible
right
>> I think so yeah
so um you know with uh with sort of in
line with this theme of lean teams and
you know we're not talking the top 10%
protocols that do have uh engineers that
are internal uh like a a massive team to
bounce ideas off of let's say or either
solo builders or maybe a founder and a
co-founder. Um, and that's pretty much
all running all of the wearing all of
the hats. Um, are there any like
templates or channels that they can like
in terms of communication templates um
things that they resources that are
available anywhere online for
communicating with impacted users,
partners, some cases media um things
like that?
>> Sure. two places to look and we have to
caveat this with the fact that every
hack is unique and every protocol is
unique. So there aren't good cookie
cutter templates for approaching this
because they won't match yours. But we
do have some pretty good things in the
form of postmortems themselves
>> and guides on how to deal with worms. So
for example, this presentation here
that'll be made available to people is a
great uh resource that they should check
out for how they think about it. I
really recommend reading postmortems. We
put out a lot of postmortems at immunifi
which we call bug fix reviews for kind
of almost hacks that we manage to
prevent and many projects do put out
postmortems these days. I recommend you
read that to understand how things are
framed after you read you know five or
10 of those. You kind of understand how
it goes and what wiggle room you have.
And then I really recommend you you you
do some crisis training if you're lucky
enough to have a PR firm or or somewhere
similar who can provide media training
to you. do a little bit of crisis
training uh to understand what it's like
and and how you're going to be squeezed
and and understand how they respond to
things because these
are you know hacks onchain hacks are
unique to us and our situation but
crises are not okay we know how to
communicate and manage these difficult
situations your media experts have
already gone through this I mean but I I
was prepared for this because I I had
already managed dozens of crises before
I started running you know defi war
rooms like it was all old hat to me at
the end of the day. And you already know
people who are experts in this. You just
have to ask them for advice and for
training and they can come and help you
with that and they can come and show you
how to frame that messaging. And maybe
if you're lucky and you bully them
enough, they can even come into the war
room and do that messaging for you. And
that is a much better solution than
relying on a template. And speaking of
which, right, I'm being prepared. Like
the best preparation is just to know who
you're going to call and have that set
up in advance. Yeah. Yeah. That even in
those moments of panic, your brain's not
not thinking. It's thinking about a
thousand different things and even
anything that can simplify the immediate
response to that is gives you great
momentum that is very important uh to
handle things uh the right way. So, all
right. Uh and then I think we'll have
time for one last multiplepart question.
This one I admit I I I wrote this one.
So, uh, this one's for me. I'm just
curious your take, uh, and what your
answer is. Again, kind of databased. Um,
but what's your best guess and the in
terms of the percentage breakdown of,
uh, you know, white hats, black hats, or
gray hat actors in the space? Um, and do
you think it's common for uh, you know,
a hacker to change the color of their
hat? Then I have one followup as well.
>> That's a scary question, Dan.
You know, we were trying to derisk it,
make people feel a little bit more
comfortable with the situation. Now,
we're going to blow it back up and spook
them all. That's how it's going to go.
Look, I think that crypto has an
incredible community of white hats.
Like, really, the best people in crypto
are the White Hat security community in
my view. I've been lucky enough to spend
years with them. They're by far my
favorite people in crypto. I don't know,
like I think what we do in the industry
is incredible, but really these people
are are are kind of shining lights in so
many ways. And unfortunately on the
downside, while we do have these literal
Batman's among us, I think they probably
make up 10 to 20% of the hacker
community.
>> Okay,
>> sounds about right.
>> Right now, I don't think that the rest
of it is black hats. I think, you know,
black hats are probably sub 10%.
>> An even smaller.
>> Makes me feel better.
>> Right. But the challenge is almost
everybody in between is what we call a
gray hat, which means that, you know,
they're going to have their own motives
and they're going to do what is in
typically the primary financial
self-interest. And the thing with these,
you know, hacker types in my experience,
and I'm sure, you know, Dan, you've kind
of seen the same is these are very smart
people. They will all have kind of their
own codes of honor or ethics, right? But
the challenge is whatever that code is,
it may not be the same as yours and it
can be very very gray. And a lot of
these guys are very very gray, right? So
do I see them changing the color of
their hat? Yeah, I've seen it many times
and that's the scary part, right? That's
why, you know, atmunifi we did so much
work on bug bounties in in years ago to
make those multi-million. It's to deal
with this part. Okay. Let's always make
sure that the incentives make people go
white by default and not black.
>> Yeah. Yeah. I guess that you I mean you
kind of answered my my final followup
there which is uh you know are there
situations that you've seen where you
know someone probably had black hat
intentions at the start but then had
that remorse afterwards or the fear of
consequences that
>> after having done so. Yeah.
>> Many times usually after the fear of
consequences is you know nudged in their
way often via email to their personal
email. That's usually when they're like,
"Oh, suddenly I'm very remorseful about
my life choices,
>> right? Found me.
>> Yeah. Yeah. Watch your back. Look over
your shoulder. Build your own paranoia
because Yeah. I mean, it's
>> it it's it's not going to be worth the
the hassle like you said throughout this
presentation. Uh it's and it's feels a
lot a lot more rewarding, I would
imagine, to to be be on the good side of
things. So, um, and feel like you're
actually doing the community, you know,
a greater good. Uh, so yeah, I mean,
this was this was an awesome chat and
and I really appreciate you taking your
time out of your busy day to educate the
rest of the builders in the space um on
this super important topic. Um, you
know, we have another session coming up
in two weeks, everybody. Um, we're going
to be talking about setting up bug
bounty programs and we'll have our
special guest Cantina join us for that
session. And so make sure you tune in in
two weeks from now on the 19th. Um but
thank you so much Mitchell and the
entire Immunifi team uh for for joining
us on today's session and educating the
community.
>> U later
>> be well.
>> All right. Thanks guys.
These workshops are run by our internal security team, designed to teach you best practices coming from real-world Coinbase experience. #OnchainAppInsights #web3 #accountabstraction #erc4337 #onchain #onchainApps #gasless #base #basechain #coinbasewallet #crypto #web3 #apps #blockchainrewards