Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/2513 Thanks!
[Music]
hello and good evening on day two of the
chaos communication Camp
2023 it's late in the evening this is
milway stage in case you're
wondering and the next talk is going to
be about incident responses so if you're
curious about how do you even get there
to have an incident response how you
could prepare for an incidental response
and how you could support in your
organization uh the incident Response
Team uh in doing their job and uh trying
to fix whatever broke let's put it that
way um we have the right talk for you
this is stories from the life of an
incident from incident responders Harry
and Chris please a very warm Round of
[Applause]
Applause so good good evening thank you
for joining us today um we will uh tell
you a little bit of our life as incident
responders and I'm Chris uh I did my uh
computer science studies at the
University of alang and nurg I do this
uh security stuff for over 10 years now
so my CV is a little bit longer uh at
the moment I'm a detection engineer um
before that I was a long time uh working
in DFI so digital forensic incident
response uh in different
organizations
and yeah I'm har I studied electrical
and computer engineering at rw2 aren
University and I pled a lot of CTF and
did some hacking stuff at Kos computer
club aren during my masters I worked at
x41 DC doing pen testing and Patch
analysis so I also have some kind of
offensive security background on for
around one year now I'm working at Gata
Advanced analytics doing digital
forensics and incident
handling first Christian will give you a
short introduction and then he will tell
you how a classical ransomware attack
looks like and in the second part of the
talk I will tell you how the incident
responders work and what you can do in
advance to make it go as smooth as
possible and support the incident
response team
so as Harry told you um I will probably
we'll talk about ransomware because um
the customers we usually have are small
and mediumsized businesses uh
universities and hospitals and those are
regularly unfortunately regularly hit by
um
um uh ransomware gangs um the main
reason for this and that's if you heard
the last
talk um why they maybe not that
responsive and are not so interested in
they just lack the resources so the
manpower to do uh proper security
measurements to secure their systems
especially in in uh situations where you
are for example in a hospital have
medical devices um which where you
cannot simply install an AV on or even
patch the system
uh because you lose the certification as
a medical device then um but also in in
um companies manufacturing companies on
the shop floor um we're talking about
systems uh that have run times of 25
plus years so if you look back now
2023 um we're talking about XP and older
systems fun fact I was in a in a
ransomware case in a Warner cry in 2017
um when I got a call from from a a
person from the shop floor asking me if
we have an N4 expert um uh that can tell
us if Warner cry is uh um affecting an4
of course you don't need to be a expert
on an4 in this Wonder cry is of course
not uh affecting N4
systems so due to the time uh slot we
thought memes are the best way to to
tell you those stories and we have a lot
of
them so in the first uh um section I
tell you a little bit of how an attack
Works um there are a lot of different
possibilities how you can describe and
how to structure the how an attack Works
um there's the miter attack framework
for example there was for examp a talk
Yesterday by m Sao um here on the stage
there's the original cyber kill chain
from from loit Martin um you have uh um
stuff from from companies like Mand and
their targeted the tech life cycle but
that's all in my opinion too too fine
grain it's uh that's the reason I just
take three simple steps yeah get a
foothold in the door look uh move play
around and cash out uh those three uh I
will just go
over so start with uh get foot in the
door so normally we see three ways how
attackers get can get into the the
environment uh in ransomware cases you
have vulnerabilities in uh remote uh or
Internet facing systems you have the
remote Services uh itself and you have
malware starting with the with the the
vulnerability
and um I just looked uh up the last four
years and uh maybe somebody remembers
net scaler the the so-called shitx
vulnerability in December
2019 um it was released mid of uh
2019 uh December 2019 um the P first POC
publicly available PC was in beginning
of January and the patch was available
in middle of January so so there was
around one week to one and a half weeks
between a public uh proof of concept for
the vulnerability and a patch for the
vulnerability and what we saw during
2020 a lot of companies patched but the
patch didn't remove the the the
compromise so they were already
compromised and um
yeah with the with the patch they didn't
remove the
compromise so what we found what we
could provable see or prove evidence uh
for uh was a 9mon uh customer was
breached after 9 months using this this
vulnerability and we had other customers
where we could see that the net scaler
was uh affected after two years uh but
we couldn't prove that this uh this
compromise was the reason for the uh
actual ransomware case
and of course such vulnerabilities
happen not that
often yeah so
20121 gave us a haum exchange
vulnerability also uh similar situation
uh the patch appeared as an outof band
patch from Microsoft on a Tuesday
evening 10:00 in German time um we saw
during our uh incidents or the the
assessments we did uh that um uh the
first exploit exploitation attempts uh
were seen on Wednesday in the morning at
5:00 a.m. so around 7 8 hours later um
with I know one guy who could patch
because he was online when the patch was
released otherwise Germany was unable to
patch in
time and of course we can go on with
2021 proxy shell also exchange
vulnerability proxy not shell also
exchange
vulnerability we have uh in 20202 VMware
Horizon the the virtual desktop
infrastructure from
VMware just to name also open source uh
stuff Zimbra uh a collaboration platform
including a email server uh has had an
uh vulnerability actually the
vulnerability was in cpio uh from 2015 I
think which uh led to a compromise um
using via email so you send an email
with a cpio um with a with a specially
crafted um archive file and you could
drop a webshell in one of the the
directories yeah you have of course uh
40 o which is the 40 gate um VPN and uh
um firewall operating
system and if you read the
news we start at the beginning again uh
net scaler had some issues uh several
weeks ago um according to Fox it we have
1,900 still unpatched uh net scalers
worldwide um how many
patched was net scalers exist that um
have not been checked for compromise we
don't know of course so
that will be a nice year
probably um so what can you can you do
against this kind of of attack Vector
patch your systems is one thing as you
see this that doesn't uh lead to the the
the um uh or what you need to do
afterwards in such cases you need to
check
your um systems for possible
compromise that is important important
to reduce this I highly suggest put your
uh Services behind some VPN so that only
people who already uh have connection to
the VPN um can access your services or
the services they
need and that would reduce the attack
surface at least to the VPN
server so but I of course we can also uh
think about remote services without
vulnerabilities um there can be uh
configuration uh mistakes so the admin
do does something wrong there can be um
insecure default configurations like
this um I don't know if you know it but
the the local admins or the
administrators on the Windows system are
are uh automatically in the remote
desktop users group
in a uh and so we had several cases
especially in the beginning of the
pandemic when everybody moved from uh to
to the home offices and they needed to
put people fast uh in the position to to
access their the the the internal
systems again they just put a RDP server
on the internet and hoped for the
best
um additionally if you put services on
the internet of course Brute Force and
uh credential uh stuffing are attacks
that are possible um so brute forcing
just trying the the username and
password combinations uh credential
stuffing using already leaked passwords
or credentials from leaks you find on
the
internet what you can do about this kind
of of attack Vector is uh just as I said
use uh multiactor Au authentification
and reduce use the attack surface as in
in the point with the vulnerabilities
before by moving the surfaces behind a
VPN and then use multiactor
authentication on VPN of
course the last Vector that we see
normally that attackers can get in the
net Network smw um we all know this uh
about uh those funny uh emails you get
uh with the attachments
um include uh that uh have either Word
documents attached either uh zip files
uh with with Visual Basic scripts
javascripts and what you can get isos um
you you see uh a lot these
days um or what you can also have that
you can have just a link inside the
email and you download the the
respective file from some some shady uh
um file sharing uh
website um what we saw over the last
year was uh USB sticks again
finally um I'm not sure if you have
heard about raspberry Robin which is a
uh malware that that warms via USB
sticks um but uh I haven't seen it as a
vector for uh ransomware yet on my own
but there are people who said that it's
uh an initial access broker for some of
the ransomware
gangs so what can you do about this if
you think um
the you can of course ban simply some
file extensions in your um mail server
or uh you um change the file Association
types in um your operating system
meaning that you don't open the
JavaScript and Visual Basic script files
using for example the windows scripting
host but open it with notepad and that
will of course uh some people will be
um uh some people will will think about
what this this is then and ask the it
guys but it's better than running the
the the script
itself one thing
I I don't like to to say it but keep
your AV updated um uh this is one thing
keep it updated and read the locks we
see a lot of incidents where we see that
that the uh
already days or weeks before we you
could have seen that there's something
going on in your
network and if you see mware in your AV
locks then react to it just check it you
don't know how long this AV uh the smw
has been on your
system the thing
is that just because your AV detected it
now it might have been getting uh
received an an update for its signatures
and uh the M was active for days or
weeks
before
so when they are inside
then they usually look move and play
around a little
bit
so when they look around what they do is
they they enumerate ad they do ports
against they you they search for
vulnerabilities they try uh check uh
what they how they can um escalate their
privileges they try to find credentials
cerber roasting we heard in the talk
before for example is this one
thing
um they try to identify accounts you run
have running on your systems they can
use they can get the credentials from
and
reuse and for that reason one of the
most important things I think is that
you have a uh principle of leas
privileges in your environment so only
what a account needs he should be able
to
do um you should use dedicated service
accounts for your services of course and
just for your information um service
account is not account that has a SVC
underscore in front of it and is
otherwise a normal user account um there
are exist edicated service accounts in
Windows environments so so use them use
strong passwords I still I today I can I
know companies who still use eight
character long passwords um I think
that's 20 minutes on a on a decent
graphics card today
so use strong passwords length matters
12 plus uh um characters is a minimum in
my
opinion and don't reuse password
especially on your uh uh systems the
local
administrator especially in small and
mediumsized businesses you see a lot uh
that uh people use the same password for
all local administrators so if I have it
on one system I have the whole
company don't reuse
it
um
so when they they move around in your
network using either uh password hashes
they found um valid credentials they
they they discovered somewhere um
vulnerabilities are also used to to move
around in your environment um they try
to to uh establish persistence
mechanisms yeah we heard also in the
talk before about the C2 uh um channels
command and control channels they they
use
um they install in some cases directly
any desk team viewer or other remote uh
Control
software or sometimes they also use
tunneling softwares like enr or uh
recently they started using Cloud flare
G um and this is what you need to do is
a prop have a proper Network
segmentation um and with the proper
Network segmentation I don't talk about
subnetting um subnetting
means you just have different subnets
you need to have a firewall between them
and you have need to have rules between
them that uh um restrict exess between
your your your
subnets and one thing is especially uh
important
please keep uh or use Network
segmentation to restrict the access to
your backup and your Management Systems
as far as possible we see a lot
especially as I said in small and
mediumsized businesses or in the other
organizations we we have as as customers
that yeah they have they tell us in in
in in in workshops yeah we have uh um
Network segmentation every building is
one segment and uh on the question yeah
you can move uh between in a segment you
can uh access everything yes you can and
between the the the buildings you you
have also uh firewall and you cannot
access anything no you can access
everything and also your your V uh
VMware uh Management console oh yes yes
we can so everybody can access it yes of
course and that's doesn't
work so
um when they play around they normally
get try to to gain more privileges so
privilege escalation is thing local
privilege escalations uh normally but
also using
vulnerabilities um or misconfigurations
insecure default configurations um my
personal favorites are uh Group Policy
preferences or passwords in group policy
preferences um this is no longer
possible since I think 2014 to put
passwords in group policy preferences uh
however if you had your um password
stored in those preferences before the
patch in
2015
2014 then they're still there and yeah
they are AES encrypted but the
encryption key is on the Microsoft
website um so you can uh just download
it and just take it and decrypt the keys
um
they during that phase they also try to
um disable your security
measures the thing you can do is of
course patch your systems so try to get
your vulnerabilities out of of of this
equation con try to configure your
systems in a secure way this is not
always possible due to some shitty uh
third party
software and keep your AV um updated and
please please as I said already check
the locks and act
accordingly so in the last phase they
cash out that's when when they uh start
using a uh uh being your backup service
so they copy data from your your uh
environment using um file sharing
platforms for example yeah Mega and set
was was once the thing uh we fa we had
already uh every every other file
sharing platform you you can think about
is a possible uh way to exfiltrate data
they also use their their C2
Communication channel so sometimes you
they also uh they just use the the
possibilities in uh any desk or in in
RDP uh
clients or they use uh file transfer
protocols like um uh SS
SFTP um um
we saw for example in one case that they
tried to install filezilla on every
machine they had access to um because on
the first one it didn't work on the
second it didn't work on the third it
didn't work yet because SFTP was blocked
uh uh outgo and that is one of the
things you can do to to prevent uh
exfiltration block um at least
uh um protocols you know that you don't
need uh in your environment
uh and proper Network segmentation of
course uh is a general uh a
thing so in the last uh uh step that's
when they start your
encryption
um they running the
ransomware normally they are have uh
domain admins at that point so they can
run it on all domain connected
systems um they can also disable of
course when they when they are domain
admin they can disable the the AV before
they they start the
ransomware um ransomwares today disable
services like databases and uh such
things so that they have the full power
of the machine for um
the uh for the encryption um if you get
lucky not everything works perfectly
because they use uh um uh group names
and windows is uh especially picky when
you uh have a non-english uh
uh uh windows installed um for example
in Germany the the group everybody is
called yea and we had um cases where the
ransom didn't really work that well
because they couldn't change the
permissions of the files
first
um they use different encryption schemas
um normally they they come with the
asymmetric and a symmetric encryption
type the asymmetric or public key
cryptography the public key comes with
the ransomware and is used to encrypt uh
the symmetric keys they generate uh on
in your environment depending on the
ransomware they they generate one key
for each system or even one key for each
file that depends a little bit on the on
the ransomware how it works but that's
the the usual thing they use
um I would never count on the the fact
that there are
possibly maybe there could
be decryptable uh things
um in in my op in my uh in my
world The ransomware Gangs have learned
and use the standard Microsoft Windows
uh or some other publicly available
Library to to do their
encryption um they executed via remote
uh um uh tools like PS xac uh Powershell
or some use gpos or group policies they
uh to execute the ransomware on every
machines they they connected to the
domain and what can you do about this
it's it's hard but the the most
important thing is have online backups
uh offline backups sorry
you you see you see off online backups
are not that are great but not that
great offline backups is the the most is
the most important thing so um don't
have it connected to your
environment the the the USB disc on a
system is not uh offline
backup
um in my opinion if you see that
something is is still
encrypting I I'm I'm always hesitant to
say shut down the system because you can
break the encryption and maybe the file
that is currently in under encryption or
the files uh will never be decryptable
if you want to buy a uh
decryptor um or get a
decryptor through
some discussions with the with the
ransomware guy
um if it's a VM just suspend it and
that's it and if everything is already
encrypted keep cool and call your
incident
responder so now let's talk about
incident responds what happens when it's
already too late and what can can you do
to support your incident response team
at first the things I'll say in this
chapter are for our company and how we
work so other companies might work a
little bit different than
that first for some reason incidents
always come on Friday
afternoon so some customers think it is
a good idea to try to solve a case by
themselves maybe until the end of the
week and and if they didn't solve it
until the end of the week they call the
incident response team please don't do
that it doesn't help your company and it
doesn't make your incident Response Team
happy to have to work on the
weekend and in addition the longer you
wait with calling the incident Response
Team the longer the incident response
will take and the more complicated
forensics will be because you have lock
retention times while trying to do stuff
by yourself maybe you modify some of the
systems and it becomes much more harder
to do precise
forensics so what happens on our side
when such a new incident ticket arrives
the first thing we do is team internal
coordination so we discuss do we have
enough people do we have a person for
each role in our team we have three
roads incident handling forensics
analyst and mayware
analyst so first let's talk about
incident
handling incident Handler is responsible
for all the tasks that our customer
facing and the first point is always get
the customer out of that headless
chicken mode like we call it because
when an incident comes at our customer
side everyone is like running around in
circles like like headless chicken doing
something but not doing anything helpful
so this is always the first task for the
incident Handler handlers structure the
customer do meetings and then do all the
relevant decisions leading to a secure
emergency operation mode that means in
this case that you have working core
infrastructure so working domain
controller maybe a working email server
and whatever um you need or whatever you
define as very business critical
systems let's go a little bit more in
detail probably the first measure will
be to cut off the internet connection
because you just buy you a lot of time
with doing that no matter how many back
do the attack has placed in the network
if you cut off the internet connection
the attackers can't access their back
doors
anymore and then you will start to
rebuild your network you will Define
everything in your current
infrastructure as red Network and then
start building up a green network with
clean systems maybe you will start with
some um admin workstations so that the
administrators can work
properly and then we go through a
prioritized system list and build up the
most important systems this can be of
course like I said domain controllers or
email servers but also whatever is
important for your current um company
and for the business this is of course
in a hospital something very different
from a small business or maybe a
university and please have such a
prioritized list before your first
incident because when you start
discussing in an incident which system
is the most important then everyone in
the company will tell you something else
and everyone will tell you that their
system is the most important and has to
be yeah migrated and checked and
analyzed at first and this isn't really
helpful so during incident handling
there might also be obstr for example if
you have backup ups and you encrypted
your backups this is great but you
should think about where to store that
encryption key because when you store it
in your password manager and the
password manager database is a in a VM
which then gets encrypted by the
ransomware well you have backups but you
don't have your key to decrypt the
backups so this doesn't help you very
much and this is not only true for
password this is also true for like
everything else which you could need
during an incident like contact lists or
network lists and so
on
so working with third parties is also a
task for the incident
Handler this could be a data Protection
Agency or the customer's customers
because they also often yeah have some
panic and have questions which the
customer maybe can't answer so this is
also a talk
task for the incident Handler and of
course also working with law enforcement
but there's one thing to keep in mind
Law Enforcement wants to do criminal
investigation incident response wants to
bring you back to business as soon as
possible this is not necessarily the
same goal but it defitely makes sense to
allow your incident responders to share
all information with law enforcement
because this is beneficial for
everyone so why do I do forensics at all
one could ask well the less precise the
forensic results are the more
conservative the rebuild needs to be so
if you don't know anything about the
attack you have to rebuild everything
from scratch so let's say you have maybe
two week old backups and you know the
initial access of the attackers was only
one week ago then you could use that two
weeks um old backups for rebuilding
which is really great but if you don't
know how long the attackers were in the
network then it's really hard to to
say a second Point why you should do
forensics is to estimate the impact
especially in an early um phase of an
attack it is important to know did the
attackers only compromise some user
devices or did they already gain domain
administrative privileges because then
the incident handling is completely
different and of course it is also
relevant to know which attack path the
eack is used so you can Harden your
network and improve in the
future how should incident how should
forensics be it should of course be
correct it should be non-disruptive for
the customer so that they can
concentrate on rebuilding their
infrastructure and it should be fast
what means correct you want to have a
dedicated forensics analyst so you want
to have incident response team which is
doing incident response in their
day-to-day
business and you want in your forensics
teams one person who can really
concentrate on forensics and is not also
doing the incident handling and the
meare analysis and so on because then
you can't really concentrate on doing
the technical
work it also should be non-disruptive as
I said that means for our company we do
remote triage collection we use the tool
video Sur Raptor for it this is an
inofficial logo of Il Raptor um it's a
really nice tool because we just send
our customers an exe file the customer
just has to execute the exe file and
then it collects all the locks we need
all the um yeah information we need and
up uploads it to our um to our
infrastructure so we don't need to drive
to the customer and like start copying
hard diss for week for weeks because
this doesn't help
anyone I should also said it should be
fast so we have a lot of Automation and
tooling to make the work really fast
automatically pushing our the uploaded
files to analysis infrastructure
automatically doing reporting and so on
I won't go in detail here because this
is enough for a single talk about it
maybe maybe on Congress let's
see so what happens when we do manual
for IC
analysis there are some questions you
normally want to answer one is of course
about the vors you want to find all
vectors the attack is placed in the
network of course uh you want to have
General ioc so indicators of compromise
so maybe which IP addresses did the
attackers use so you can block them in
the firewall or which tools did they use
so you can scan over the whole it
infrastructure and find further
compromised systems and if you have the
ioc you can also cross correlate between
cases or sh share them with law
enforcement another important thing is
of course lary movement because you can
use letter movement to build a timeline
of the attack so the methodology here is
to look where the attas came from and
where the attas went so if you have a
system a and you see the attas came from
system B then you analyze system B and
go back in time and the same you could
do forward in time until you found yeah
all relevant systems which were
compromised and have a proper timeline
of the
case another common thing in the life of
an incident responder is to fall in a
rabbit hole because it can be really
really interesting to do forensics but
yeah you need to check what the customer
really wants to know or what your
incident Handler really wants to know
and not just analy ize something and
fall into the rabbit hole because this
yeah really doesn't help anyone here
also the 8020 rule applies so I'd say
you get like 80% of the relevant
forensics results and 20% of the time
and often this is enough for the
incident Handler to do the right
decisions to rebuild the customer net
customers networks as fast as
possible
so let's go into reporting nobody really
likes
reporting but producing fancy forensics
results doing fancy technical stuff is
worthless if you can can't explain it to
a manager nobody pays us to do fancy
technical stuff nobody understands but
we are paid to help our customers and
explain the situation to them and to do
the right decisions so yeah nobody likes
reporting but it is a really really
important
thing so after delivering the report
the incident is over and all work is
done right no not really because such a
report normally contains a lot of
recommendations what to do for the
customers and as I already said incident
response more or less stops when the
customer is in an emergency operation
mode with the core infrastructure and
the most critical systems working but
there's still a lot to do and of course
security is a process so there's always
something to improve to stay ahead of
the
attackers so now let's go into a quick
recap so so how um to protect against
those kind of attacks yeah we had this
uh the topics just as a recap um patch
your systems it's important keep them
updated um vulnerability ities come and
go uh that's one thing have a sane
privilege account management um so use
the correct um user accounts uh for the
correct for the the right uh
task use second two- Factor
authentication for rem remote
Services have a proper Network
segmentation uh so put firewalls and
access rules between your networks um
check your AV locks uh regularly and act
accordingly and not really protecting
your you against an attack but keeping
the the symptoms a little bit um U not
not that that uh devastating have
offline
backups so how to make incident response
easier for you for yourself and for the
incident response team first thing is
don't try to solve the stuff by yourself
just call the incident Response Team off
your trust as soon as you know you have
an
incident the second thing is logging
policy because if you have a very short
lock retention for example and you only
have lcks for the last like two days
then it might be hard to find out what
happened and if you don't lck stuff at
all especially in the cloud environments
this is a thing where you need to to
look at then it might also be hard to
get proper forensic
results then as I said have offline
contact lists because your May server is
down your accounts are compromised so
you can't use like your Microsoft teams
or whatever you use in your company you
need an alternative way to communicate
to your colleagues and or
employees then as I said prioritized
asset list and network plans and it isas
recovery plan in best case and please
all have all that in a secure place in
best case on paper well nobody likes
doing stuff with dead trees I know but
dead trees don't get ransomware
right so that's it from our side more or
less normally this is the point where
you can ask us questions but we want to
spin that around and give you do some
questions maybe someone can guess the
right
answer what do you think is the shortest
time to domain admin we saw an incident
between initial compromise and the
attack is gaining domain
admin yeah it's not it somebody said 17
seconds 2 minutes well it's not that bad
six minutes is is what we have seen an
incident what do you think think is the
shortest lock retention on a domain
controller so how long Z zero Z 10 10
hours are
here well it is 2.5
minutes what what do you think is the
highest number of domain administrator
accounts we saw in an
incident 200 200 N
50 60 four with 120 people working in
it what do you think is a longest dwell
time so the longest the longest time
between being initially compromised and
realizing that you are
compromised 60 nah it's it's not that
bad I hit hear a lot of years well it is
like around 2
years so now that that's really it from
our side no time for questions thank
you if
you if you have questions we will be
there outside waiting come join
us thank
you thank you thank you her and Chris
warm Round of Applause thank
you
[Music]