Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/322 Thanks!
Hello, everybody, I'm Nick.
I'm the security engineering lead at
CloudFlare, and I'm going to
talk to you today about Heartbleed
if you guys aren't sick of it already
after this long year.
So specifically,
I'm going to talk about my personal
experience or my experience at CloudFlare
with what happened during
during Heartbleed, when it came out
and what happened after the fact.
So has anybody read this
article?
This is this is a snippet from
The Verge, and it
is kind of semi fictional account
of what happened during the disclosure
of of Heartbleed.
I can kind of paraphrase a little bit
how that conversation went, but
it kind of kind of went a little bit like
this, like.
So do you use details?
No, not that I know of.
Does anybody use details?
And that's for websites.
Nobody use data.
Grumped details.
How about topics?
Well, while my answer was,
well, what's Attila's heartbeat?
I mean, at this point, barely anybody
knew what this was. It was very obscure
feature.
And the answer
here was, was, oh, there's the stupid
and and there's a bug.
You should turn them off.
Right. So recompile open SSL
with openness to sell no heartbeat's.
And that was that was kind of that
was that. So heartbeat's were off
for CloudFlare. We have a pretty simple
architecture, so deployed it really
quickly and the answer was,
OK, looks good.
Public disclosure should be around April
9th.
So I guess a lot of people had this
question. But why tell CloudFlare?
Well, let me just go real quick
and describe what CloudFlare does.
So it's a reverse proxy.
So if you have a website,
CloudFlare can sit in front of it and
block malicious traffic.
That's the sort of red X up there,
as well as send cached content,
static content. That's the bright orange.
So it reduces anybody getting
to your website that's malicious
and reduces the load on your website.
And for this to work, this cloud
has to be closer to the visitor than it
does have to be to your website.
So we have this global network.
So are our nodes are closer to
the visitors and that's that's how it
works. And there's over a million
sites on CloudFlare, including banks,
government websites, Bitcoin exchanges,
almost every Bitcoin exchanges on
CloudFlare, the ETFs
website, Reddit.
I can go on and on and on.
So lots of sites.
But what CloudFlare does is very simple.
It's essentially three
services or two DNS,
HTP and ETPs,
which is powered by open SSL
and Engine X.
So the architecture
is very simple in that every machine that
we have can serve every site.
So thinking back to
Heartbleed, this is actually
you can see why this would be a really,
really bad situation for Heartbleed
to hit anyways.
So it happened early
April 7th.
Ten, twenty seven open SSL publish
their advisory and that hit
Hacker News really quickly after that.
Within half an hour it hit the front page
and about an hour
later we posted our standard
CloudFlare customer sites are patched.
You don't have to worry about it sort of
post. And this is this
was you know, it was
it was a thing. It was it was a bug and
it was starting to gain some steam.
And then about an hour after that,
there was a tweet from code name Con.
And I think everyone can
know what this is.
This is that's the next site.
This Heartbleed itself, it was branded
and it came out to mass media.
So this became a really
big deal. Heartbleed Dotcom had a
logo at the mainstream press
Heartbleed virus.
I don't know if you guys remember that,
but people were saying, oh, there's a
Heartbleed virus out there.
And I knew I knew
it got really bad when my my
mother called me and said, what?
What's going on?
So this was kind of a big deal and,
well, we were finished patching so
well, we had some time to kill.
What are we going to do
at this point?
There is we decided on three things.
And one was to help
keep this scanner that Flippo had from
falling over.
The second was to turn our network into a
large honeypot to see what type
of attacks or what type of scans are
happening. And then the third was to
decide or to figure out what we're going
to do about our certificate's.
We have a quite a few websites that
use our service and many of them use SSL
and we had about one hundred thousand
certificates and revoking them was not
really at this point.
The day after disclosure, it wasn't
absolutely clear that this was something
that you had to do.
So first, let's talk a little bit
about this Heartbleed scanner.
So Filippo, who's now a CloudFlare
engineer, wrote this server and go
and you type in your hostname and it
scans it for her for
whether or not it answers Heartbleed
heartbeat requests that are malformed.
So these are small ones around one
hundred bytes so that it doesn't leak
anything beyond your standard
frame. It shouldn't leak any any
information.
You put it on us and then put it behind
CloudFlare and shout out to
Kyle Eisen from our team for helping
keep this up.
But this this is kind of what from
Philippos server, what it looked like
there was up April
8th, up to two thousand requests per
minute. So this was a very
highly used tool.
And that's nothing because
this is the next two weeks.
Two thousand is the the bottom tick right
there. So up up to ten,
ten to twenty thousand a day,
a ten to twenty thousand
scans per minute for the next
two weeks. And it we held it up to
two hundred million tests in the first
two weeks.
So with the scanner up and up and
running. Yeah.
And thank you to Filipa, wherever he is,
he's somewhere in the room stand up.
This is thanks for the tool.
I don't I don't see him, he's somewhere.
But in any case, this is this is what
what he found in terms
of Domain's, it
was really bad at first.
This is the night.
This is two days after Heartbleed was
originally announced.
Up to 30 percent of the sites
that he scanned were vulnerable.
And this luckily cut down.
And a lot of people use this in their
automated testing to validate
sites. But, yeah, it got down to a low,
low number pretty pretty quickly.
So now that that was up and running back
at CloudFlare, we decide,
well, what can we do?
Well, we can log every heartbeat
that we see that comes in with with a bad
size. And while we can
put that data in the shelf until now,
and that's that's kind of what we did.
So here are logs from the
9th and sixty
nine percent of them had a message size
of sixteen three, eighty four,
which you might
know as the largest power of two,
you can fit into assigned sixteen bit
integer.
But you might also recognize it as
the hard coded value in the
SSL test python tool that came
out the first day, 20 percent were
one twenty one and that was actually from
Flippo site and University of Michigan
maybe where you guys scanning on April
9th or
in any case there are a lot of requests
that used the
zero length packet, which is another way
to just check to see if your site is is
is vulnerable.
But
about a week later it was around the
same.
So if there were people who
are mass exploiting this against sites,
they were probably using just the basic
SSL test Python script
and around 20 percent were still
Philippos tests.
So if you can, you can do the math
here.
It's it seems that a lot of people
were just scanning with Lebas test.
It wasn't a lot of mass exploitation.
And flipping the
numbers back, you see that one percent
of flippo scans were actually against
sites on CloudFlare.
So this is what the map looks like
for where the attacks were coming from.
I know IP maps are not really that
interesting. They don't tell you that
much. But this is
this is where it is. There's some some
strange spikes like in Iceland, but
don't read too much into it.
Now, the question is, and this is
this is what we were thinking when it
first came out was why was it
really so dangerous?
Why was Heartbleed so bad?
Well, it's a kind of a layer six
request that doesn't necessarily
get logged. People don't log
parts of handshake's very often unless
you have a specific ID's rule
or something like that. And it's really
bad in that. Sixty four K of server
memory can be filtrated
by one request, and this has
longer login info, session cookies
and perhaps Tila's
private keys.
We didn't know.
And if you kind of look
at the diagram here, this is that's the
heap right there.
Everything that's above the request that
if you have a request comes in, it gets
put on the heap and anything that was
previously removed from the heap is
still sitting there.
So we know that there are passwords,
cookies. People were finding this right
away. And the question
was, would the key be there?
Is the key going to set up above one of
these requests?
So we looked at the code.
Right.
And what did the code say?
Well, it said this can't happen, at
least not in Engine X.
The key gets loaded right away and
therefore gets at the bottom of the
stack. And any
time that you do applications or
for requests coming in, they're going to
be higher up on the stack and they're not
going be able to read the original key.
Right.
And Engine X itself was single threaded.
So if you have, it's not going to be
able to catch something halfway in
the middle of another thread doing
an operation.
So Open SSL
has a big number library that they use
and they clear the memory when they're
done. So if you're doing a handshake for
TLS, all the cryptographic
material is going to be cleared by open
SSL.
At least that's what we thought.
Right. And we
we weren't sure. I mean, I just looked at
some code and what do I know?
So we launched the CloudFlare
Heartbleed challenge.
So it was a this was something that we
did to crowdsource an answer.
So we set up a standard Engine X,
which was outside of CloudFlare.
It was on a third party VPs, and
it had the vulnerable version of open
SSL.
And we said to all of you guys,
come and find it, see what you can do and
to show proof, give
us a message signed with that private
key.
What did we find?
Well, for the first couple of hours,
there was there was trolling and
as you can see, somebody clapping
because they did this.
But basically
anything that you post onto the page is
going to be put into memory and the
Ingenix. So people were
posting private keys in there and they
were posting what looked like
you can see my name there, Nick, what
looked like a password's file.
So everyone was getting really confused
getting all this. There's a private key
in my Heartbleed question, but nobody was
actually getting the key until we saw
this tweet from a fettah
and.
We took a look, this is
our office.
It's me pointing out a television screen
and yeah, he
solved it. So congrats to or.
And so he wasn't the only one.
This was in the first twenty four hours
or so, there were 12 people in the first
48. About twenty five people had had
solved it and got the real key and
sent proof.
So can you still
private keys?
Absolutely. Yes.
And it was solved in under 10 hours
at private keys can definitely be
vulnerable. But another thing we did was
we logged where in memory the
Heartbleed requests were coming in and
we compared that relative to where the
private key was initially allocated
and then they never overlapped.
So how did this how did it actually
get solved?
Well, there
was yeah, there was a second bug and open
SSL who would have known
looking through that code.
If you dump the memory of of the request,
all the places in red are where
key private keys did exist at one
point and it turns out some temporary
variables were not wiped.
This is the this is the code to clean up
the mess. There's big name
free versus big, dumb, clear, free.
This is just yeah.
It was just in certain cases in the
Montgomery multiplication, they didn't
clear up. They didn't clean the partial
pieces.
We can do a little bit of math just to
show how people actually
solved it with this.
But RSA, you have a
couple of different things.
You have a public exponent e
you have two primes multiplied
together. They make the public key and
the private exponent D and
any one of the if you get any one
of P, Q or or D,
you get the whole private key.
So what people did was they took
every one hundred and twenty eight bit
block that they saw in
X filtrated sharply data and they just
tried to divide it into the modulus.
And if it divided, there
you go, it's factored.
And this is how nine out of the ten
people solved it turns out that one of
the prime factors is just sitting there
on the heap.
After tens of thousands of requests,
you might look upon it.
But one enterprising
gentleman, Ruben Tsou, who's
at University of Cambridge, he
used a much cleverer method, which is
coppersmiths attack, which is a Latisse
reduction attack, where you only
really need about 60 percent of
one of the private keys to to to
find it out. And this depends on the fact
that the public exponent is small.
So for performance
reasons, the public exponent in RSA is is
small so that any public operation
is is really fast.
But he solved it in only 50
requests. So this was
that was actually really interesting.
So private keys are gone, right?
What does that mean,
revocation time?
You know, the Internet was built for
this, right? I mean, people who designed
the peak, they said, yeah, you know,
people are going to revoke 100 hundred
thousand certificates in twenty four
hours. This is just this
this is how we designed the system.
Right. And this is
what sounds kind of reported.
That's that's mostly us, actually.
That's that's all the Internet.
But that's mostly CloudFlare.
And this is this is what it look like.
The blue line is revoke certificates.
And as you can see, that's April 7th.
This is after Heartbleed.
This is when everybody was revoking and
then that green spike is is Clouser
revoking. But once we
revoked all these certificates, we
found out that, well, it didn't
really mean that browsers wouldn't accept
these certificates anymore.
And I can go into that
really quickly. There are three methods
to do this that were
built for, you know, handling
revocation and certificates and five or
nine. And first is the certificate
revocation list.
And this is just a
flat file with a list of certificates
that are revoked and
did as revoking one hundred thousand
certificates, breaks the URLs
Hecuba did.
So the CRL from Global
Sign grew from twenty two
to five megs.
Yeah.
This basically does the CRL
server, and
lucky for this, lucky for global sign,
CloudFlare was in front of their KRELLE
server, but unlucky for us.
Where we're used to that kind of traffic,
but yeah, you can see here every three
hours there are waves and this had to do
with the cycles in which scrolls were
updated and Microsoft Internet Explorer
downloading them.
And that was pretty rough.
Anyways, Earl's broken.
How about OSPI XP?
Well, this is the online certificate
status protocol.
This is a kind of question answer.
Is this certificate revoked, yes or
no? And well,
it's really broken.
And Chrome has kind of known this
for a while and stopped checking.
But if you have hard fails, it does
disallow you from using certain networks,
especially with captive portals.
And if you have a soft fail,
somebody on the network can just kind of
drop the ISP response and voila,
the site is no longer revoked.
So it really doesn't work
or scale to the
degree that we'd want it to in plain
being requested from the browser.
In that sense, I would say
URL sets. This is Google's proprietary
method that they
basically collect all the
kraals that they can from different
certificate authorities and put them
together and install them in the browser.
So you get updates with the browser
of these sets of scrolls.
I shouldn't say all the scrolls, but
it's specific certificates
and this is kind of what we found out.
They only do EV search and special of
sorts, and
if the browser doesn't get updated, then
you're not going to get an update on your
URL set, which is it's
kind of bad.
So CloudFlare challenge dot com.
Once it was solved, we revoked it and the
way it was, it was
not an easy search. But Chrome did market
as revoked because, well, they added it
manually into JSON file.
So none of the hundred thousand
CloudFlare certificates were
being marked as revoked by
Google Chrome.
So we we basically
made a hack.
And if you can read this, but it's
the most efficient for line of
C++ revocation.
This isn't chromium.
It revokes all of our certificates.
This is a hackish this is not scalable.
You shouldn't do it this way.
But it was it was how
it had to get done because there were no
valid ways of of doing revocation.
So, yeah, revocation is pretty
much broken.
And what can we do?
Well, there's Schauder certificate
expiration periods that could help that
would at least help with the cartels
because you don't have to be holding on
to old certificates for very long.
You can sort of shrink the size of the
sets pretty quickly.
Hoxby must staple is an
extension that requires you to
send the server to send the Hoxby
response in the handshake
that can help to certificate
transparency, something else that's
thrown out there to solve this.
But none of these are
widespread and none of these have been
implemented. So something has to work
for revocation.
So I guess in summary, there are three
things that we did after after
Heartbleed.
We kept the scanner from falling over,
turned our site into a honeypot.
And while we definitively
answered that, yes, you have to revoke
your certificates and there's no excuse.
So there's a lot of takeaways from
Heartbleed, right?
I don't want to be the one to sort
of tell everybody how to take away what
to learn from this. But open
source disclosure is hard.
And this this really was the first
one of the year of
what turned out to many be many open
source disclosures.
And we did learn a lot
of lessons of how to do that correctly.
Other things that pointed out, which seem
obvious to people, but they weren't
obvious in or weren't in open SSL,
which is features should be disabled by
default. Nobody who installed open SSL
one oh one wanted
heartbeat's necessarily so
turn-off features by default.
Another thing is, while expect
the unexpected, that's sort of obvious
in computer security, that we didn't
really learn that from Heartbleed, but it
was it was definitely a shock when it
came to
other things, these attacks, a
lot of the attacks on real sites like
CloudFlare. So we saw quite a few attacks
as as I mentioned, but a lot of them were
still we're just scans from people just
trying to see if their sites were alive.
So that's that's a reassuring sign.
Crowdsourcing was effective for the
Heartbleed challenge. I couldn't find the
private keys. And luckily, there are
very smart people out there who were able
to find it. Is anybody here in the in
the auditorium winner of the
CloudFlare challenge?
Is anybody here? I don't see any hands,
but anyways.
I don't have my glasses, so
congratulations wherever you are,
and the last thing is, is revocation
needs a solution.
And the last
conclusion from this is, is really,
you know, support open SSL.
It's it's.
I messed up my microphone there, but
thanks.
No, really, I mean, this is this is part
of critical infrastructure for
the world and for not only websites,
but as Zach here was telling
embedded devices, and
these guys need support. So please
support open and sell and I'm done.
Thank you.
OK, quick announcement before we start
the Q&A, if you are going to leave
the room, please get up now, go
out quietly without talking so we can do
the Q&A nice and quickly.
Also, you will be only able to leave
the room at this point.
Yes. If you have any questions, please
line up at the microphones.
We'll start with microphone number one.
Thanks. Not really a question.
Thanks. Thanks. OK, we really, really
followed the progress
even before the talk with your paper
republished.
And we just want to contribute a
piece of missing information.
So the first
attacks, as you call them, 24 hours
after the disclosure that were
us, we weren't really attacking.
We started working on the scanner
about 20 minutes after the public
information, and it was a top
scanner in the region.
So it was a scanner and
used multiple different methods.
We just as we went, we
came to the conclusion that there
might be firewalls to be deployed
in between which
which may stop some
of the packets. So we use many
different packets in one in
every request. So that's also a scanner
there,
is there?
No.
That quick interruption, if you're
leaving on the ground
floor, please only use the front left
and the front door.
If you're up there, you can use any
door you want. But if you're down here,
please only use this door and
this door to leave the room.
Thank you.
Does your bike work out well for
you?
Jim, thanks for the comment.
I don't think I saw anything specific
to you to to your your account, but
yeah, it's great to know.
So, I mean, it's one of those things it's
really hard to tell from our perspective
about the attack, whether someone's
malicious or whether someone just
scanning a lot of the
the holes that we see don't have any
information that identify and
what the intent is.
Microphone number two, please.
Hello.
I have a question to CloudFlare.
Could you stop looking for
users, please?
You make the Internet more central every
day, all the fancy home, but you switch
there and come on to the network
is so small you could handle it with
that thing asleep, all of the bandwidth
and you're blocking all the notes all the
time.
We're looking into solutions for
the main problem is that
there is a lot of spam
on tour for Web sites.
And right now filtering
through the good and the bad is
something that we're working on and we
are focusing on that this year.
So look for something coming up this year
for Tor users like.
Next up, we have a question from our
signal Angel asked on IFC.
Thank you.
So given that maximum TLC record
length is 16 kilobyte, how
is it possible to even get back 64
kilobytes?
It's you can split you
can split a request over multiple
records, I believe.
I think I'm pretty sure that's how it
works.
Microphone number one, please.
What is your opinion on.
I said
go for it.
I'm not sure for the most qualified
person to answer that.
To be honest.
I think there's well intentioned.
I think the community acknowledges that
there are a lot of problems, us not
necessarily in terms of maintainability.
It's one solution.
I'm not sure it's the correct one.
Yeah, I think I think Libros itself works
for its goal,
which is OpenBSD.
They have a portable version by now
as well as they did with
all of our other folks.
And it's
much cleaner and still
open as the cell tries to support the
way old systems
like Windows, 16 feet and bullshit, you
really don't need so down
maintainability, you will remain for a
while.
Yeah, I agree that openness
itself does support quite a lot of
different platforms and some people need
that. But the different forms
of openness as well that have come
recently, Libras and sell boring SSL
are taking patches from
each other. So I think the
more people looking at this project,
the better.
We have time for one more question.
Microphone number three, please.
Hello. And both your talks, you showed
numbers of vulnerable sites
decreasing and increasing again
afterwards.
Can you explain that?
So I think there are small bumps
that were just people coming and going.
I don't think we saw a lot of
websites that became vulnerable.
I think a lot of it is just pieces
of measurement websites that came and go
between different scans.
I don't think there are any large jumps
that we saw
when it comes to the data that I was
showing.
This is from Philippos scanner and
not everybody was scanning the same
domains every day. So that's just
standard variance.
Thank you very much.
Psychotronic, please give our
speakers a warm round of applause.